[R] install Netcdf library (linux)
Hi All, i'm on a debian linux 64bit, i'm tying to install the netcdf intraface, i tried both ncdf and ncdf4 but trying to build i received the error : (i have necdf installed on my machine and it is able to fiund it .. no missed .h) epy@epinux:~$ sudo R CMD INSTALL --configure-args=-with-netcdf_incdir=/usr/include -with-netcdf_libdir=/usr/lib ncdf4_1.8.tar.gz * installing to library ‘/usr/local/lib/R/site-library’ * installing *source* package ‘ncdf4’ ... checking for nc-config... yes Using nc-config: nc-config Output of nc-config --all: This netCDF 4.2.1.1 has been built with the following features: --cc- gcc --cflags- -I/usr/local/include -I/usr/local/include --libs - -L/usr/local/lib -lnetcdf --has-c++ - no --cxx - --has-c++4 - no --cxx4 - --fc- --fflags- --flibs - --has-f90 - no --has-dap - yes --has-nc2 - yes --has-nc4 - yes --has-hdf5 - yes --has-hdf4 - no --has-pnetcdf- no --has-szlib - --prefix- /usr/local --includedir- /usr/local/include --version - netCDF 4.2.1.1 --- netcdf.m4: about to set rpath, here is source string: -L/usr/local/lib -lnetcdf netcdf.m4: final rpath: -Wl,-rpath,/usr/local/lib Netcdf library version: netCDF 4.2.1.1 Netcdf library has version 4 interface present: yes Netcdf library was compiled with C compiler: gcc configure: creating ./config.status config.status: creating R/load.R config.status: creating src/Makevars ** Results of ncdf4 package configure *** netCDF v4 CPP flags = -I/usr/local/include -I/usr/local/include netCDF v4 LD flags = -Wl,-rpath,/usr/local/lib -L/usr/local/lib -lnetcdf netCDF v4 runtime path = -Wl,-rpath,/usr/local/lib ** ** libs gcc -std=gnu99 -I/usr/share/R/include -DNDEBUG -I/usr/local/include -I/usr/local/include -fpic -O2 -pipe -g -c ncdf.c -o ncdf.o ncdf.c: In function ‘R_nc4_nctype_to_Rtypecode’: ncdf.c:40:18: error: ‘NC_INT’ undeclared (first use in this function) ncdf.c:40:18: note: each undeclared identifier is reported only once for each function it appears in ncdf.c:49:18: error: ‘NC_UBYTE’ undeclared (first use in this function) ncdf.c:51:18: error: ‘NC_USHORT’ undeclared (first use in this function) ncdf.c:53:18: error: ‘NC_UINT’ undeclared (first use in this function) ncdf.c:55:18: error: ‘NC_INT64’ undeclared (first use in this function) ncdf.c:57:18: error: ‘NC_UINT64’ undeclared (first use in this function) ncdf.c: In function ‘R_nc4_varsize’: ncdf.c:69:28: error: ‘NC_MAX_DIMS’ undeclared (first use in this function) ncdf.c:75:2: warning: implicit declaration of function ‘nc_inq_varndims’ [-Wimplicit-function-declaration] ncdf.c:78:4: warning: implicit declaration of function ‘nc_strerror’ [-Wimplicit-function-declaration] ncdf.c:84:2: warning: implicit declaration of function ‘nc_inq_vardimid’ [-Wimplicit-function-declaration] ncdf.c:94:3: warning: implicit declaration of function ‘nc_inq_dimlen’ [-Wimplicit-function-declaration] ncdf.c: In function ‘R_nc4_inq_varunlim’: ncdf.c:112:2: warning: implicit declaration of function ‘nc_inq_unlimdim’ [-Wimplicit-function-declaration] ncdf.c: In function ‘R_nc4_inq_var’: ncdf.c:152:2: warning: implicit declaration of function ‘nc_inq_var’ [-Wimplicit-function-declaration] ncdf.c: In function ‘R_nc4_inq_vartype’: ncdf.c:168:2: warning: implicit declaration of function ‘nc_inq_vartype’ [-Wimplicit-function-declaration] ncdf.c: In function ‘R_nc4_inq_varname’: ncdf.c:181:2: warning: implicit declaration of function ‘nc_inq_varname’ [-Wimplicit-function-declaration] ncdf.c: In function ‘R_nc4_get_vara_double’: ncdf.c:214:2: warning: implicit declaration of function ‘nc_get_vara_double’ [-Wimplicit-function-declaration] ncdf.c: In function ‘R_nc4_get_vara_int’: ncdf.c:257:2: warning: implicit declaration of function ‘nc_get_vara_int’ [-Wimplicit-function-declaration] ncdf.c: In function ‘R_nc4_get_vara_text’: ncdf.c:313:2: warning: implicit declaration of function ‘nc_get_vara_text’ [-Wimplicit-function-declaration] ncdf.c: In function ‘R_nc4_inq_dimid’: ncdf.c:345:2: warning: implicit declaration of function ‘nc_inq_dimid’ [-Wimplicit-function-declaration] ncdf.c: In function ‘R_nc4_inq_varid’: ncdf.c:355:2: warning: implicit declaration of function ‘nc_inq_varid’ [-Wimplicit-function-declaration] ncdf.c: In function ‘R_nc4_inq_dimids’: ncdf.c:377:9: warning: implicit declaration of function ‘nc_inq_dimids’ [-Wimplicit-function-declaration] ncdf.c: In function ‘R_nc4_inq_dim’: ncdf.c:387:12: error: ‘NC_MAX_NAME’ undeclared (first use in this function) ncdf.c:391:2: warning: implicit declaration of function ‘nc_inq_dim’ [-Wimplicit-function-declaration] ncdf.c:408:2: warning: implicit declaration of function ‘nc_inq_unlimdims’ [-Wimplicit-function-declaration] ncdf.c: In function ‘R_nc4_inq’: ncdf.c:451:2:
Re: [R] Help! Please recommend good books/resources on visualizing data and understanding multivariate relations...
I found this [1] book interesting. About big data It really depends from a number of things... if can help, I know hdf5 work pretty Well with huge dataset . [1] http://www.ggobi.org/book/index.html On Jul 3, 2012 7:14 PM, Michael comtech@gmail.com wrote: Hi all, Could you please help me? I am looking for books/pointers/resources/tutorials on visualizing complex/big data and on understanding multivariate relations in complicated data. More specifically, we have categorical variables and are interested in how to visualize the categorical data and visualize data conditioned upon categorical values. Could anybody please give me some pointers? Thanks a lot! [[alternative HTML version deleted]] __ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code. [[alternative HTML version deleted]] __ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
Re: [R] hierarchical clustering of large dataset
Peter,
really thanks for your answer.
install.packages(flashClust)
library(flashClust)
data - read.csv('/Users/epifanio/Desktop/cluster/x.txt')
data - na.omit(data)
data - scale(data)
mydata
a bc d e
1 -0.207709346 -6.618558e-01 0.481413046 0.7761133 0.96473124
2 -0.207709346 -6.618558e-01 0.481413046 0.7761133 0.96473124
3 -0.256330843 -6.618558e-01 -0.352285877 0.7761133 0.96473124
4 -0.289039851 -6.618558e-01 -0.370032451 -0.2838308 0.96473124
my target is to group my observation by 'speciesID'
the speciesID is the last column : 'e'
Before to go ahead, i should understand how to tell R that the he has to
generate the groups using the column 'e' as variable,
so to have the groups by speciesID.
using this instruction :
d - dist(data)
clust - hclust(d)
is not clear to me how R will understand to use the column 'e' as label.
Sarah said :
Yes. Identical rows have a distance of 0, so they're clustered
together immediately,
so a dendrogram that includes them is identical to one that has only
unique rows.
in this way i will lose a lot informations!
seems relevant for me that a species is found 4 times instead of 1 with a
specific combination of environmental parameters.
no?
Maybe a way to Try to decrease the size of my dataset can be :
convert my multiple rows to abundance values, i means :
if a species occurs four times with exactly the same environmental parameters
i'll add a column for abundance and fill in a 4. and then remove three rows
?
in this way i can decrease the size of my dataset (in rows) but i'll add a
column.
make sense ?
Thanks a lot for your help (and patience),
Massimo.
Il giorno Mar 9, 2012, alle ore 3:54 PM, Peter Langfelder ha scritto:
On Thu, Mar 8, 2012 at 4:41 AM, Massimo Di Stefano
massimodisa...@gmail.com wrote:
Hello All,
i've a set of observations that is in the form :
a,b,c,d,e,f
67.12,4.28,1.7825,30,3,16001
67.12,4.28,1.7825,30,3,16001
66.57,4.28,1.355,30,3,16001
66.2,4.28,1.3459,13,3,16001
66.2,4.28,1.3459,13,3,16001
66.2,4.28,1.3459,13,3,16001
66.2,4.28,1.3459,13,3,16001
66.2,4.28,1.3459,13,3,16001
66.2,4.28,1.3459,13,3,16001
63.64,9.726,1.3004,6,3,11012
63.28,9.725,1.2755,6,3,11012
63.28,9.725,1.2755,6,3,11012
63.28,9.725,1.2755,6,3,11012
63.28,9.725,1.2755,6,3,11012
63.28,9.725,1.2755,6,3,11012
…
….
55.000 observation in total.
Hi Massimo,
you don't want to use the entire matrix to calculate the distance. You
will want to select the environmental columns and you may want to
standardize them to prevent one of them having more influence than
others.
Second, if you want to cluster such a huge data set using hierarchical
clustering, you need a lot of memory, at least 32GB but preferably
64GB. If you don't have that much, you cannot use hierarchical
clustering.
Third, if you do have enough memory, use package flashClust or
fastcluster (I am the maintainer of flashClust.)
For flashClust, you can install it using
install.packages(flashClust) and load it using library(flashClust).
The standard R implementation of hclust is unnecessarily slow (order
n^3). flashClust provides a replacement (function hclust) that is
approximately n^2. I have clustered data sets of 3 variables in a
minute or two, so 55000 shouldn't take more than 4-5 minutes, again
assuming your computer has enough memory.
HTH,
Peter
__
R-help@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
Re: [R] hierarchical clustering of large dataset
i'll try to describe the data,
here [1] there is a subdatset (255 rows) 6 columns (a to f)
the last columns contains the Identification Number (ID) for a particular
species.
the ID in f are 20 different species and it should be my 'label':
16001
11012
25011
13011
11029
11027
10022
10024
20009
11016
20002
13001
11010
22037
15001
30016
21005
11028
15002
20008
the other vars (from 'a' to 'e') are :
depth
temperature
salinity
substrate-class
morphology-class
my target is to have 'groups of species' based on the similarity of theyr
environmental parameters, and build a dendrogram like [2]
the full dataset (1,5 mb) is available here [3]
[1] http://massimo-timecapsule.whoi.edu//data/img/subdataset.txt
[2] http://massimo-timecapsule.whoi.edu//data/img/manova_clust_matlab.png
[3] http://massimo-timecapsule.whoi.edu//data/img/x.txt
Il giorno Mar 9, 2012, alle ore 7:18 PM, Peter Langfelder ha scritto:
On Fri, Mar 9, 2012 at 1:50 PM, Massimo Di Stefano
massimodisa...@gmail.com wrote:
Peter,
really thanks for your answer.
install.packages(flashClust)
library(flashClust)
data - read.csv('/Users/epifanio/Desktop/cluster/x.txt')
data - na.omit(data)
data - scale(data)
mydata
a bc d e
1 -0.207709346 -6.618558e-01 0.481413046 0.7761133 0.96473124
2 -0.207709346 -6.618558e-01 0.481413046 0.7761133 0.96473124
3 -0.256330843 -6.618558e-01 -0.352285877 0.7761133 0.96473124
4 -0.289039851 -6.618558e-01 -0.370032451 -0.2838308 0.96473124
my target is to group my observation by 'speciesID'
the speciesID is the last column : 'e'
Before to go ahead, i should understand how to tell R that the he has to
generate the groups using the column 'e' as variable,
so to have the groups by speciesID.
using this instruction :
d - dist(data)
clust - hclust(d)
is not clear to me how R will understand to use the column 'e' as label.
Well, you didn't say that column e was a label that you wanted to keep
separate. Any other labels in the data? You may not want to use labels
in the distance calculation.
Do I understand right that you want to cluster each species separately?
Peter
__
R-help@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
[R] hierarchical clustering of large dataset
Hello All, i've a set of observations that is in the form : a,b,c,d,e,f 67.12,4.28,1.7825,30,3,16001 67.12,4.28,1.7825,30,3,16001 66.57,4.28,1.355,30,3,16001 66.2,4.28,1.3459,13,3,16001 66.2,4.28,1.3459,13,3,16001 66.2,4.28,1.3459,13,3,16001 66.2,4.28,1.3459,13,3,16001 66.2,4.28,1.3459,13,3,16001 66.2,4.28,1.3459,13,3,16001 63.64,9.726,1.3004,6,3,11012 63.28,9.725,1.2755,6,3,11012 63.28,9.725,1.2755,6,3,11012 63.28,9.725,1.2755,6,3,11012 63.28,9.725,1.2755,6,3,11012 63.28,9.725,1.2755,6,3,11012 … …. 55.000 observation in total. where : a,b,c,d,e are environmental parameters and f is a label. as you can see some rows are duplicated, this means that the observation occurred more times (in my use cases the observation is the presence of a specific biological specie in a photo, if in the photo there are more than one individual of the same species i have a duplicated row ) i'm trying to learn how to use R in order to build a dendrogram that will help me to 'group' several species in communities, based on the similarity of the env. parameters. i tried with d - diet(as.matrix(my data)) hc - hclust(d) but it doesn't works. is the 'redundancy' of my data (multiple rows with same information) a problem? should i remove all the rows that are exactly the same ? in this way how to take care about the fact that for the same environmental parameters i've multiple observation ? maybe this information is not relevant in order to build the dendrogram ? Please, can you suggest me a valid approach in order to cluster a such dataset ? forgive me, i've an evident lack of statistic knowledge, thank you very mach for you help! __ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
Re: [R] use R from python
Hi Barry,
thanks for your preciouse help!
Please apologize my newbie questions.
i'm able to go ahead tring to port the R code to run under python,
now i'm blocked on the plot function.
maybe i'm doing the same error but i haven't yet find a way to debug
my code line.
this what i tried :
# import library (rpy2 + spgrass6)
import rpy2.rinterface as rinterface
import rpy2.robjects as robjects
robjects.r.require('spgrass6', quietly = True)[0]
# def some functions :
rsort = robjects.r['sort']
rseq = robjects.r['seq']
rlength = robjects.r['length']
rdiv = robjects.r['/']
rmin = robjects.r['min']
rmax = robjects.r['max']
rdiff = robjects.r['-']
rget = robjects.r['get']
rslot = robjects.r['slot']
rpng = robjects.r['png']
rplot = robjects.r['plot']
df = robjects.r['data.frame']
# input data and output image
inputmap = 'elevation.dem'
output_image = '/Users/sasha/Desktop/file.png'
# code
mymap = robjects.r.readRAST6(inputmap)
elev = rget($)(rslot(mymap,data),elevation.dem)
r_sorted = rsort(elev, decreasing=True)
p_seq = rseq(1,rlength(r_sorted))
x = rdiv(p_seq,rlength(r_sorted))
x = rdiv( ( rdiff( x, rmin(x) ) ) , ( rdiff( rmax(x) , rmin(x) ) ) )
y = rdiv( ( rdiff( r_sorted, rmin(r_sorted) ) ) , ( rdiff( rmax(r_sorted) ,
rmin(r_sorted) ) ) )
rpng(output_image)
result = df(x,y)
rplot(result,type = 'l',x)
File ipython console, line 1
SyntaxError: non-keyword arg after keyword arg (ipython console, line 1)
my rplot actions seems to be wrong :-(
while if i try :
args = [x, y]
kwargs = {'ylab':foo/bar, 'type':b, 'col':blue, 'log':x}
robjects.r.plot(*args, **kwargs)
i have a png file, but to see the file on my desktop
i had to type twice the command :
robjects.r.plot(*args, **kwargs)
thanks!!!
Massimo.
Il giorno 22/gen/2010, alle ore 00.38, Barry Rowlingson ha scritto:
On Thu, Jan 21, 2010 at 8:35 PM, Massimo Di Stefano
Hi All,
please aplogize me if my qustion is a bit OT here,
but maybe is there someone that uses R from inside python
using rpy or rpy2 interface.
In [54]: x = rdiv( ( rdiff( x, rmin(x) ) ) , ( rdiff( rmax(x) , rmin(x) ) ) )
In [55]: y = rdiv( ( rdiff( r_sorted, rmin(r_sorted) ) ) , ( rdiff(
rmax(r_sorted) , rmin(r_sorted) ) ) )
Errore in .Primitive(-)(my...@data$elevation.dem,
my...@data$elevation.dem) :
argomento non numerico trasformato in operatore binario
Traceback (most recent call last):
File ipython console, line 1, in module
File build/bdist.macosx-10.6-universal/egg/rpy2/robjects/__init__.py,
line 423, in __call__
RRuntimeError: Errore in .Primitive(-)(my...@data$elevation.dem,
my...@data$elevation.dem) :
argomento non numerico trasformato in operatore binario
My non-existent Italian is telling me this is non-numeric argument in
binary operator. Something like:
hello - goodbye
Error in hello - goodbye : non-numeric argument to binary operator
- because you are subtracting the strings my...@data$elevation.dem.
Tracking back, those strings come from:
r_sorted = rsort('my...@data$elevation.dem', decreasing=True)
- which is sorting the string vector! Like this:
sort('my...@data$elevation.dem', decreasing=TRUE)
[1] my...@data$elevation.dem
You want to sort the *value* of that object. You want to sort the
$elevation.dem column of the @data slot of the python R object mymap.
In a functional form which will translate to your style of rpy2 would
be this in R:
get($)(slot(mymap,data),elevation.dem)
You may need to get 'get' and 'slot' from r.robjects in the way you
do other functions. This looks a bit weird to me, but I'm used to
[[elided Yahoo spam]]
Hope that points you in the right direction.
Barry
--
blog: http://geospaced.blogspot.com/
web: http://www.maths.lancs.ac.uk/~rowlings
web: http://www.rowlingson.com/
twitter: http://twitter.com/geospacedman
pics: http://www.flickr.com/photos/spacedman
__
R-help@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
[R] use R from python
Hi All,
please aplogize me if my qustion is a bit OT here,
but maybe is there someone that uses R from inside python
using rpy or rpy2 interface.
i'm at my first experience using rpy2.
i need to port torpy some R code .. but i'm having little problems
i hope yopu can help me.
The R code is :
grass65
R
library(spgrass6)
r=readRAST6(elevation.10m)
r_sorted=sort(r...@data$elevation.10m,decreasing=T)
p_seq=seq(1,length(r_sorted)) #area
x=p_seq/length(r_sorted)
x=(x-min(x))/(max(x)-min(x))
y=(r_sorted-min(r_sorted))/(max(r_sorted)-min(r_sorted))
result=data.frame(x,y)
png(graphic.png) # oppure (graphic.pdf)
plot(result,type=l,x)
plot(result,type=l, xlab=area, ylab=elevation)
dev.off()
i tried to write it in python-rpy2 :
# import library (rpy2 + spgrass6)
import rpy2.rinterface as rinterface
import rpy2.robjects as robjects
robjects.r.require('spgrass6', quietly = True)[0]
# def some functions :
rsort = robjects.r['sort']
rseq = robjects.r['seq']
rlength = robjects.r['length']
rdiv = robjects.r['/']
rmin = robjects.r['min']
rmax = robjects.r['max']
rdiff = robjects.r['-']
# read input file
mymap = robjects.r.readRAST6(inputmap)
elev = rsort('my...@data$inputmap', decreasing=True)
r_sorted = rsort('my...@data$elevation.dem', decreasing=True)
p_seq = rseq(1,rlength(r_sorted))
x = rdiv(p_seq,length(r_sorted))
x = rdiv( ( rdiff( x, rmin(x) ) ) , ( rdiff( rmax(x) , rmin(x) ) ) )
y = rdiv( ( rdiff( r_sorted, rmin(r_sorted) ) ) , ( rdiff( rmax(r_sorted) ,
rmin(r_sorted) ) ) )
errors
but maybe it is totally wrong...
this the same code running in ipython.
tring to run it i have :
In [38]: import rpy2.rinterface as rinterface
In [39]: import rpy2.robjects as robjects
In [40]: robjects.r.require('spgrass6', quietly = True)[0]
Out[40]: True
In [41]: rsort = robjects.r['sort']
In [42]: rseq = robjects.r['seq']
In [43]: rlength = robjects.r['length']
In [44]: rdiv = robjects.r['/']
In [45]: rmin = robjects.r['min']
In [46]: rmax = robjects.r['max']
In [47]: rdiff = robjects.r['-']
In [48]: inputmap = 'elevation.dem'
In [49]: mymap = robjects.r.readRAST6(inputmap)
raster map/current region mismatch detected in components:
colsrows origin.northing origin.easting
FALSE FALSE FALSE FALSE
set plugin=TRUE to override; continuing with plugin=FALSE
/Users/sasha/Downloads/spearfish60/PERMANENT/.tmp/MacBook-Pro-15-di-Massimo-Di-Stefano.local/elevation.dem
has GDAL driver GTiff
and has 140 rows and 190 columns
In [50]: elev = rsort('my...@data$inputmap', decreasing=True)
In [51]: r_sorted = rsort('my...@data$elevation.dem', decreasing=True)
In [52]: p_seq = rseq(1,rlength(r_sorted))
In [53]: x = rdiv(p_seq,length(r_sorted))
In [54]: x = rdiv( ( rdiff( x, rmin(x) ) ) , ( rdiff( rmax(x) , rmin(x) ) ) )
In [55]: y = rdiv( ( rdiff( r_sorted, rmin(r_sorted) ) ) , ( rdiff(
rmax(r_sorted) , rmin(r_sorted) ) ) )
Errore in .Primitive(-)(my...@data$elevation.dem,
my...@data$elevation.dem) :
argomento non numerico trasformato in operatore binario
Traceback (most recent call last):
File ipython console, line 1, in module
File build/bdist.macosx-10.6-universal/egg/rpy2/robjects/__init__.py, line
423, in __call__
RRuntimeError: Errore in .Primitive(-)(my...@data$elevation.dem,
my...@data$elevation.dem) :
argomento non numerico trasformato in operatore binario
[[elided Yahoo spam]]
regards,
Massimo.
[[alternative HTML version deleted]]
__
R-help@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
