Re: [datatable-help] setnames on a non-data.table object

Hi, There's no technical reason. I guess enough people realise now that the set* functions change the object by reference. So if setnames worked on data.frame : DF1 = data.frame(a=1:3, b=4:6) DF2 = DF1 setnames(DF2, "b", "B") This would change both DF1 and DF2. There might be some

Re: [datatable-help] setnames on a non-data.table object

On 02/10/13 12:50, Ricky Saporta wrote: This might be a topic to raise in a separate email: What do you think of adapting a naming convention where the name of the function indicates when a function will modify an object by reference? In my personal work, I have been trying to end such funct

Re: [datatable-help] setnames on a non-data.table object

Rick, Oh - setnames already does work on data.frame. That was a change in v1.8.4. Was the question more for lists and vectors then (anything that can have names), rather than just data.frame/data.table? Matt On 02/10/13 15:13, Matt Dowle wrote: On 02/10/13 12:50, Ricky Saporta wrote

Re: [datatable-help] Running on variables in data.table

Hi, Likely : dt[,lapply(.SD,sum),by=type] See the examples section of ?data.table for an example. `.SD` is explained on that page too. Matt On 03/10/13 09:25, Kostia wrote: Hi, I have a data table with a number of variables and I wish to do some function on each variable, my data tab

Re: [datatable-help] Flagging duplicate (non-unique) values based on specifications

It's more efficient to ask questions like this on Stack Overflow please : http://stackoverflow.com/questions/tagged/data.table You can edit the question there, and people can add or remove quick comments. In v1.8.10

[datatable-help] Cologne on Friday

Hi, If anyone is in or near Cologne on Friday, I'm presenting with Arun : http://www.meetup.com/KoelnRUG/ Hope to meet a few of you there. Regards, Matt ___ datatable-help mailing list datatable-help@lists.r-forge.r-project.org https://lists.r-for

Re: [datatable-help] Fail to add new columns within a function

Hi, This isn't a bug really. A documentation or too low default issue maybe. When all spare slots are used up, there is no choice but to make a shallow copy and create a new vector of column pointer slots. This is the pointer (address in RAM) which any variable names (symbols) point to.

[datatable-help] Latest presentation

The presentation Arun and I gave on Friday is now on the homepage : http://datatable.r-forge.r-project.org/CologneR_2013.pdf Matt ___ datatable-help mailing list datatable-help@lists.r-forge.r-project.org https://lists.r-forge.r-project.org/cgi-bin/

Re: [datatable-help] Fail to add new columns within a function

On 15/12/13 23:26, Huashan Chen wrote: Hi Matthew, Thank you for the thoughtful reply. It's from a real example I am using where a master dataset is to be merged with other datasets by selected rows and columns(may or may not exist in master dataset). This function is call multiple times. As can

Re: [datatable-help] Fail to add new columns within a function

Why are you doing this iteratively? Can't you load all the files into a list, rbindlist and then reshape? On 18/12/13 09:04, Huashan Chen wrote: OK, here is the complete code with some mock functions from my example. # data: data.table object # fn: a filename to read data from merge_data<-

Re: [datatable-help] Fail to add new columns within a function

Why are you doing this iteratively? Can't you load all the files into a list, rbindlist and then reshape? What kind of data is this e.g. which field? On 18/12/13 09:04, Huashan Chen wrote: OK, here is the complete code with some mock functions from my example. # data: data.table object #

Re: [datatable-help] fread crash

On 10/01/14 16:04, statquant3 wrote: R) sessionInfo() R version 3.0.1 (2013-05-16) Platform: x86_64-unknown-linux-gnu (64-bit) locale: [1] C attached base packages: [1] stats graphics grDevices datasets utils methods base other attached packages: [1] data.table_1.8.11 [**something I

Re: [datatable-help] setnames changes names of other data.table

Yes that's correct behaviour and desirable/crucial. The set* functions and := do not copy-on-write, unlike base. They work by reference. See ?copy and type example(copy) at the prompt. If you *really* want to copy a 20GB in RAM, use DT2 <- copy(DT) http://stackoverflow.com/questions/1

Re: [datatable-help] setnames changes names of other data.table

On 11/01/14 14:31, Arunkumar Srinivasan wrote: Thanks for reporting. That's expected behaviour. Use an explicit |copy|. In short, when you do: |DT1 <- DT2|, there's *|no copy|* being made. They still reference/point to the same location (try doing |tracemem(DT1)| and |tracemem(DT2)|). Just

Re: [datatable-help] Error in structure(ordered, dim = ns) : dims [product 1] do not match the length of object [0]

Hello, This mailing list is just for the data.table package. You could try the [plyr] tag on Stack Overflow or look at the documentation for those packages to find out where support is for those packages. You probably intended to post to r-help, which is the level up in Nabble. To post h

Re: [datatable-help] leading spaces on column names from CSV file

Hi Jim, Ok yes good idea. The roots of that are due to data.table allows leading and trailing spaces (and all special characters) in column names. The confusion that by="a, b, c" doesn't work due to the spaces, due to column name " b" being different to "b". But for fread that doesn't ma

Re: [datatable-help] R-devel breaks data.table

Hi Kevin, Yes R-devel has a new check as of Friday. No problem in data.table, just R getting stricter which is a good. Fixed and v1.8.11 (r1108) works again on latest R-devel 2014-02-01 r64910. Thanks, Matt On 01/02/14 19:50, Kevin Ushey wrote: Hi guys, See the commit here: https://git

Re: [datatable-help] datatable roll="next" takes 150 times longer than findInterval

But this is at the *micro* second level ?!! I confirm those results on my slow netbook but remember these are **micro** seconds i.e. 71,000 here is less than 0.1 of a second. > microbenchmark(flodel(X,Y), GG1(X,Y), GG2(X,Y)) Unit: microseconds expr minlq median

Re: [datatable-help] datatable roll="next" takes 150 times longer than findInterval

speed issue came up recently somewhere else as well which the change in default should help. Matt On 02/02/14 18:57, Matt Dowle wrote: But this is at the *micro* second level ?!! I confirm those results on my slow netbook but remember these are **micro** seconds i.e. 71,000 here is less than

Re: [datatable-help] integer64 group by doesn't find all groups

Hi, You're doing nothing wrong. Although you can load integer64 using fread and create them directly, data.table's grouping and keys don't work on them yet. Sorry, just not yet implemented. Because integer64 are internally stored as type double (a good idea by package bit64), data.tabl

Re: [datatable-help] integer64 group by doesn't find all groups

g coerced. In my situations, if I am grouping by a int64, it is usually either an ID so I can just make it a character vector instead, or it is something where I don't mind lost precision so I just make it numeric. On Wed Feb 12 2014 at 11:22:40 AM, Matt Dowle <mailto:mdo...@mdowle.plus

Re: [datatable-help] Force evaluation of first argument to [

Hi John, In examples like this I'd use set() and [[, since it's a bit easier to write but memory efficient too. for (x in c("foo", "bar", "qux")) set(fbq, is.na(fbq[[x]]), x, FALSE) [untested] A downside here is one repetition of the "fbq" symbol, but can live with that. If

Re: [datatable-help] Force evaluation of first argument to [

t;)) set(fbq, which(is.na <http://is.na>(fbq[[x]])), x, FALSE) Which highlights an opportunity to say that I really appreciate the unusually helpful error messages in this package. -John On Wed, Feb 12, 2014 at 12:44 PM, Matt Dowle <mailto:mdo...@mdowle.plus.com>> wrote:

Re: [datatable-help] Merging strings claim that the encodings don't match

Think you may have ended up with some strings internally marked ASCII by R, which Encoding() returns as "unknown". That shouldn't be a problem and they should join fine. I've change the new warning in v1.8.11 so if it was that, it should be ok now (commit 1153), please confirm. Matt On

Re: [datatable-help] Problem with data.table and FastRWeb

Hi Mel, Thanks for the info. It's likely related to cedta() and we can handle it from data.table's side as follows. Background : http://stackoverflow.com/a/10529888/403310 Type "data.table:::cedta" so you can see the rules. I guess FastWeb is running your code in its own environment.

Re: [datatable-help] Obtain data.table_1.8.11.tar.gz source?

Sure, now in the homepage directory. That takes an hour or so to update, so this link should work now : https://r-forge.r-project.org/scm/viewvc.php/*checkout*/www/data.table_1.8.11.tar.gz?root=datatable if not, try here and browse from there : https://r-forge.r-project.org/scm/viewvc.php/

[datatable-help] v1.9.2 is now on CRAN

It usually takes a few days for binaries to make their way to all mirrors and all platforms. You can install now from source from CRAN, or within an hour the data.table homepage should refresh with the Windows .zip for v1.9.2 (Ctrl-F5 and even clearing the browser cache may be required to

Re: [datatable-help] Possible bug in 1.9.x versions

From those messages, it looks like the install didn't work properly. This can happen on Windows if another process is still using the older .dll. On every release we usually do get reports like this. Since it is Windows, let's try overkill first : 1. Close all R sessions 2. To be sure, r

Re: [datatable-help] Possible bug in 1.9.x versions

Thanks. Yes, I see the same. 1,220 tests plus tests from 37 dependent packages still isn't enough is it. Sigh. Will fix. Matt On 27/02/14 15:26, Michele wrote: Hi, thanks for the quick response. Still nothing though. Using the .zip form r-forge, at http://datatable.r-forge.r-project.org

Re: [datatable-help] Possible bug in 1.9.x versions

On 27/02/14 15:55, Michele wrote: :-) it happens to best ones as well! May I ask if you are using R-devel for your development? Not only Rdevel, but Rdevel compiled with ASAN, 2.14.0, 3.0.2, and some 3.0.3beta as well for good measure. Here was the CRAN submission covering email : I hav

Re: [datatable-help] Possible bug in 1.9.x versions

Now fixed and new Windows .zip for R-release uploaded to data.table homepage. We'll aim to release to CRAN fairly soon. Thanks again Michele. From NEWS : o When joining to fewer columns than the key has, using one of the later key columns explicitly in j repeated the first value. A problem

Re: [datatable-help] integer64 group by doesn't find all groups

on C I don't mind guiding you through. Matt On Wed, Feb 12, 2014 at 11:17 AM, can...@gmail.com <mailto:can...@gmail.com> mailto:can...@gmail.com>> wrote: Yes this isn't a data.table criticism, just a bit64 one in general. On Wed Feb 12 2014 at 11:39:47 AM

Re: [datatable-help] integer64 group by doesn't find all groups

27;s the other parts I'm not as familiar with: what's the SCM procedure, what's the build procedure going to be? On Sun, Mar 2, 2014 at 6:26 AM, Matt Dowle <mailto:mdo...@mdowle.plus.com>> wrote: On 14/02/14 15:07, Yike Lu wrote: Thanks for the info g

[datatable-help] More info on GForce

More background info and example of GForce : http://stackoverflow.com/a/22146905/403310 Matt ___ datatable-help mailing list datatable-help@lists.r-forge.r-project.org https://lists.r-forge.r-project.org/cgi-bin/mailman/listinfo/datatable-help

Re: [datatable-help] using a UPC as identifier broken in 1.9.2 (related to 'tolerance of precision' NEWS item)

On 04/03/14 23:49, James Sams wrote: I suspect there are plenty of data.table users that use UPCs and other large integer-like doubles as identifiers in their data. Storing UPCs as character data takes up an order of magnitude more space compared to a double; not really an acceptable alternativ

Re: [datatable-help] Odd problem using fread to read in a csv file: no data, just headers

Works for me as well on linux, same output as Kevin's. I was perplexed as to why Farrel's output has : File opened, filesize is 6.2E-05B but we see : File opened, filesize is 0.000 GB That line is switched depending on Windows or not. Comparing them : // On Windows : if (verbose) Rprintf

Re: [datatable-help] Odd problem using fread to read in a csv file: no data, just headers

b) opened URL downloaded 64 Kb Empty data.table (0 rows) of 14 cols: Sex,Agemos,L,M,S,P3... On 6 March 2014 12:34, Matt Dowle <mailto:mdo...@mdowle.plus.com>> wrote: Works for me as well on linux, same output as Kevin's. I was perplexed as to why Farrel's output has

Re: [datatable-help] setnames on copy data.table also renames original data.table object

And more info here (see both answers) : http://stackoverflow.com/questions/10225098/understanding-exactly-when-a-data-table-is-a-reference-to-vs-a-copy-of-another On 10/03/14 05:20, Michael Nelson wrote: This is well explained in the help file for `copy` (and `setnames`) DT <- DT2 does *not*

Re: [datatable-help] what's wrong with my quote()-ing

Hi, It's easier to use get() to fetch column names dynamically. eval() is more for when you want to construct the 'j' or 'by' clause dynamically and you need to eval the whole argument; e.g. DT[, eval(...), eval(...)] In your example, i <- "east2a" temp1 <- paste(i) temp2 <- paste0("east2b.

[datatable-help] by=.EACHI and related - please check ok

Dear all, by=.EACHI is now implemented and available in v1.9.3 from R-Forge. Please take a look at NEWS and see what you think : https://r-forge.r-project.org/scm/viewvc.php/pkg/NEWS?view=markup&root=datatable Quite a few related bugs and feature requests get resolved by this, still going thr

Re: [datatable-help] by=.EACHI and related - please check ok

ll remain a fan of the "by-without-by" :-) Is there a option to make the future stable version 1.9.4 to behave exactly as the 1.8.10? (same script same results) Thanks, Michele On 13 Mar 2014 11:36, "Matt Dowle" <mailto:mdo...@mdowle.plus.com>> wrote: Dea

Re: [datatable-help] How to replace values in data.table conditionally

On 13/03/14 14:48, Manabu Sakamoto wrote: Dear list I'm trying to access values within a data.table column by matching to elements in a vector and replacing with corresponding elements in a second vector. But I want to loop through specific column names also stored as a character vector. So

Re: [datatable-help] Possible FR - but just checking opinons

Hi, It sounds like you mean 'foreign' key. This could be useful, yes. In simple cases, I've seen that used in SQL to do what R does automatically. A de-normalised database in SQL may have lookup tables with two columns mapping say country id to country name, to save storing long country

Re: [datatable-help] Is assignment such as DT[, a:=7] supposed to print DT when surrounded by braces?

Interesting. What's happening is due to the result of DT[,d:=7] being DT. That's so that compound statements can work e.g. DT[is.na(d),d:=0][,sum(a),by=d] If DT[,d:=7] is the last line of a function or last line inside braces, then R is printing the result. It's not DT[,:=] printing,

Re: [datatable-help] Weird error in package with older data.table version

What do you mean by older data.table version, which one? Can you upgrade to v1.9.2? > Therefore this column gets bumped and the program stops. It shouldn't stop. It should bump the column and continue. That's what happens for me. Could be a bug then, which is why it's confusing talking ab

Re: [datatable-help] Change in list( ) behavior inside join

Shir is using v1.9.2 though, although in R 3.2.0 apparently which might mean R 3.0.2 perhaps. Anyway regardless of versions, the example code results in errors. Shir? > # dates > dt.dateEx <- data.table(date = as.character(seq(as.Date('2014-04-01'), as.Date('2014-04-15'), by = 1))) > setkey

Re: [datatable-help] Change in list( ) behavior inside join

st(dt, hour_beginning)] # not expected outcome dt.hrEx[dt.dateEx, list(hour_beginning)] On Mar 25, 2014, at 9:45 AM, Matt Dowle <mailto:mdo...@mdowle.plus.com>> wrote: Shir is using v1.9.2 though, although in R 3.2.0 apparently which might mean R 3.0.2 perhaps. Anyway regardless

Re: [datatable-help] internal FALSE/TRUE value has been modified

Reproduced, thanks for nice example. Not sure yet but what R 3.1 now does is store length 1 logical vectors once only, globally, for efficiency to avoid many new allocations for the common case of single TRUE or FALSE values passed around at C or R level (a nice and welcome change). Since da

Re: [datatable-help] fread() coercion bug?

Reproduced, thanks. Can't think why that is, but will fix. Please file as a bug so it's not forgotten. In the meantime, setting the class manually for that column (colClasses argument) works in this example : fread( paste0( strData, collapse="\n" ), integer64="character", colClasses=list(

[datatable-help] R/Finance in Chicago on Friday

Looking forward to it. Spaces available. Tutorial on data.table at 8am. http://www.rinfinance.com/agenda/ Matt ___ datatable-help mailing list datatable-help@lists.r-forge.r-project.org https://lists.r-forge.r-project.org/cgi-bin/mailman/listinfo/da

Re: [datatable-help] internal FALSE/TRUE value has been modified

. The warning is issued by R and we have asked if it can be upgraded to error. Matt On 01/05/14 16:29, Matt Dowle wrote: Reproduced, thanks for nice example. Not sure yet but what R 3.1 now does is store length 1 logical vectors once only, globally, for efficiency to avoid many new a

[datatable-help] data.table has moved to GitHub

Dear all, Arun has done an amazing job in transferring everything from R-Forge to GitHub. This includes the full commit history and outstanding bug and feature requests. https://github.com/Rdatatable/datatable/ To install the latest version from now on it's : devtools:::install_git

[datatable-help] Slides for useR! data.table tutorial

Draft slides are now online for the 3 hour data.table tutorial at useR! on Monday 30 June. user2014.stat.ucla.edu/#tutorials Is there something fundamental that you wished had been explained in a tutorial like this? If so, please let me know. I'm doing another of these long tutorials

Re: [datatable-help] Slides for useR! data.table tutorial

or is it something the user should take care of? (I couldn't find it in the current docs.) Thanks, M On 06/12/2014 04:22 AM, Matt Dowle wrote: Draft slides are now online for the 3 hour data.table tutorial at useR! on Monday 30 June. user2014.stat.ucla.edu/#tutorials Is there

[datatable-help] We don't know why Stack Overflow data.table tag has just been renamed

Have asked on Meta : http://meta.stackoverflow.com/questions/260463/why-has-the-data-table-tag-just-been-renamed-r-data-table Matt ___ datatable-help mailing list datatable-help@lists.r-forge.r-project.org https://lists.r-forge.r-project.org/cgi-bin

Re: [datatable-help] Efficiently checking value of other row in data.table

Hi, Not sure exactly what you need but looks interesting. Something a bit like this ? DT[ word == "good", .SD[ lag(word, N) != "not" ], by=document] Your idea being you don't want to have to repeat all the pre and post words alongside each word but rather express it in the query. Makes sen

Re: [datatable-help] Efficiently checking value of other row in data.table

the rows. Hence the two-step process I have now where I key by document and position first, create the lag_word column, key by the word and lag_word columns and query by row. Matt On Fri, Jun 27, 2014 at 3:17 PM, Matt Dowle <mailto:mdo...@mdowle.plus.com>> wrote: Hi,

[datatable-help] useR2014

To catch those who maybe don't follow twitter or blogs, slides from the data.table talk and tutorial are now online : http://user2014.stat.ucla.edu/files/talk_Matt.pdf http://user2014.stat.ucla.edu/files/tutorial_Matt.pdf I've never been to useR! before so didn't know what to expect. It cert

Re: [datatable-help] data.table vs matrix speed

Nice example. Yes this is the way to use it and I agree more readable. But I fear it isn't actually working as you expected. Each component of `:=` doesn't see previous results, yet (not yet implemented). Easier to see that in a simple example : > DT = data.table(a=1:3,b=1:6) > DT a b

Re: [datatable-help] data.table vs matrix speed

g) that sum(b) then saw and returned 1 not 2. Ok, iterative RHS more pressing that I thought then. Thanks for highlighting. Matt On 09/07/14 17:52, Matt Dowle wrote: Nice example. Yes this is the way to use it and I agree more readable. But I fear it isn't actually working as yo

Re: [datatable-help] useR2014

/2014 03:38 AM, Matt Dowle wrote: To catch those who maybe don't follow twitter or blogs, slides from the data.table talk and tutorial are now online : http://user2014.stat.ucla.edu/files/talk_Matt.pdf http://user2014.stat.ucla.edu/files/tutorial_Matt.pdf I've never been to useR! before

[datatable-help] v1.9.4 is now on CRAN

Available on Linux now or a few hours/tomorrow for Windows and Mac binaries to make their way to all mirrors. NEWS is the README on CRAN : http://cran.r-project.org/web/packages/data.table/README.html or the formatting may be easier to read on GitHub : https://github.com/Rdatatable/

[datatable-help] Video of talk at H2O World

Hi, A video of my talk at H2O World in San Francisco recently : https://www.youtube.com/watch?v=MvH1eTdsekA 0:00 Examples from two insurance companies using data.table 12:00 What is data.table, benchmarks dplyr and pandas 16:55 Overlap joins 20:00 Rolling joins 22:30 data.table r

Re: [datatable-help] Video of talk at H2O World

As a few have asked already, will upload slides later. It was a collection of different files and part was just an R script. I'll need to merge together ... On 08/12/14 14:44, Matt Dowle wrote: Hi, A video of my talk at H2O World in San Francisco recently : https://www.youtub

Re: [datatable-help] Video of talk at H2O World

December 2014 at 15:03, Matt Dowle <mailto:mdo...@mdowle.plus.com>> wrote: As a few have asked already, will upload slides later. It was a collection of different files and part was just an R script. I'll need to merge together ... On 08/12/14 14:44, Matt Do

Re: [datatable-help] Any date for 1.9.6 push to CRAN?

Hi Damian, valgrind is reporting 3 issues and there's a https:// issue in the vignette when run on win-builder. But all cran + bioconductor packages downstream are checked ok and we're passing big endian as emulated thanks to qemu so we can be sure to pass solaris sparc. Hard to give an eta. Matt

Re: [datatable-help] Any date for 1.9.6 push to CRAN?

ific >> >> Contact valgr...@valgrind.org if you are happy to have your project added >> to this list. Please provide a URL and a short description of your project. > > > > On Thu, Jul 23, 2015 at 9:44 AM, Matt Dowle wrote: >> >> Hi Damian, >>

[datatable-help] Fwd: Tuesday: Join 79 R hackers at "data table discussion"

If you're in the area, meetup in Singapore tomorrow discussing data.table ... As a follow-up to our most recent session on dplyr, we shall have a session discussing the advantages/disadvantages of data table. We have two speakers lined up, but if you out there have been using data table and found

[datatable-help] Proposal to close this mailing list

All, This R-Forge mailing list has served us well but it now seems end of life. I looked through the last 200 posts back to Nov 2015. Seems low volume with very few actual data.table questions. The good ones I did see would have been better on Stack Overflow. We've now moved to 3 places, none of