Clayton, Thanks for posting it here. Here’s the first follow-up. Here’s an example:
require(data.table) ## 1.9.3 comm 1263 dt <- data.table(x=1:1e7, y=1:1e7) ## data.table optimisation removes names system.time(ans1 <- dt[, list(z=y), by=x]) # user system elapsed # 7.193 0.275 7.859 ## data.table can't optimise to remove names foo <- function(x) list(z=x) system.time(ans2 <- dt[, foo(y), by=x]) # user system elapsed # 16.020 0.179 16.411 > identical(ans1, ans2) [1] TRUE This is without checking for names, for each of the 1e7 groups. Arun From: Clayton Stanley [email protected] Reply: Clayton Stanley [email protected] Date: April 16, 2014 at 6:23:50 PM To: [email protected] [email protected] Subject: [datatable-help] data.table and aggregating out-of-order columns in result from by Copied from this SO post: http://stackoverflow.com/questions/23097461 Here's some interesting behavior that I noticed with data.table 1.9.2 > testFun <- function(val) { if (val == 'geteeee') return(data.table(x=4,y=3)) if (val == 'get') return(data.table(y=3,x=4)) } > tbl = data.table(val=c('geteeee', 'get')) > tbl[, testFun(val), by=val] val x y 1: geteeee 4 3 2: get 3 4 > When the column order of the data tables returned from each call to testFun are mixed (but have the same name and number of columns), data.table silently binds the tables together without taking into account that they are out of order. This was probably done for speed, but I found the behavior quite unexpected, and would have appreciated at least a warning. Is there a way that I can get data.table to warn or error when this situation happens? This happened in my analysis code and caused values for two DVs to be intermixed. The reason why it happened is that in the 'testFun' there is a branch and the returned data table is created within both sides of the branch. The branch is necessary to handle the case where the data table used to create the final returned data table is empty. So on one side of that branch I basically create an empty data table with the correct columns, and on the other side the data table is created from the first. The point is that the column order for the data tables returned from each side of the branch are different. Now this is certainly a bug on my part in 'testFun'. However I could have caught the issue much earlier if I had received a warning from data.table when the by operation completed and the resulting tables were bound together. Also since there isn't a check for column order, it does make me worry that there are other places in my analysis code where the same thing could be happening. What would be ideal is if there was some way for me to tell if that is the case. Perhaps a warning, temporarily increasing a 'safety' level as an options call, etc. Usually data.table is great at warning me when things are not quite right, so I was surprised when I noticed the current behavior. I understand that this was done for speed. So maybe temporarily increasing a 'safety' level is a way to keep things fast by default and have additional checks (for a speed cost) when the user wants them? This sort of mimics how compiler optimization declarations are done in common lisp. -Clayton _______________________________________________ datatable-help mailing list [email protected] https://lists.r-forge.r-project.org/cgi-bin/mailman/listinfo/datatable-help
_______________________________________________ datatable-help mailing list [email protected] https://lists.r-forge.r-project.org/cgi-bin/mailman/listinfo/datatable-help
