[ https://issues.apache.org/jira/browse/SPARK-10520?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14737757#comment-14737757 ]
Vincent Warmerdam commented on SPARK-10520: ------------------------------------------- It just occured to me that there is a very similar error with machine learning. In R you can pass a date/timestamp into a model and it will treat it as if it were a numeric. ``` > df <- data.frame(d = as.Date('2014-01-01') + 1:100, r = runif(100) + 0.5 * > 1:100) > lm(r ~ d, data = df) Call: lm(formula = r ~ d, data = df) Coefficients: (Intercept) d -7994.9971 0.4975 ``` I'm not sure if Spark wants to have similar support but it may be something to keep in mind; the problem seems similar. > dates cannot be summarised in SparkR > ------------------------------------ > > Key: SPARK-10520 > URL: https://issues.apache.org/jira/browse/SPARK-10520 > Project: Spark > Issue Type: Bug > Components: SparkR, SQL > Affects Versions: 1.5.0 > Reporter: Vincent Warmerdam > > I create a simple dataframe in R and call the summary function on it > (standard R, not SparkR). > {code} > > library(magrittr) > > df <- data.frame( > date = as.Date("2015-01-01") + 0:99, > r = runif(100) > ) > > df %>% summary > date r > Min. :2015-01-01 Min. :0.01221 > 1st Qu.:2015-01-25 1st Qu.:0.30003 > Median :2015-02-19 Median :0.46416 > Mean :2015-02-19 Mean :0.50350 > 3rd Qu.:2015-03-16 3rd Qu.:0.73361 > Max. :2015-04-10 Max. :0.99618 > {code} > Notice that the date can be summarised here. In SparkR; this will give an > error. > {code} > > ddf <- createDataFrame(sqlContext, df) > > ddf %>% summary > Error in invokeJava(isStatic = FALSE, objId$id, methodName, ...) : > org.apache.spark.sql.AnalysisException: cannot resolve 'avg(date)' due to > data type mismatch: function average requires numeric types, not DateType; > at > org.apache.spark.sql.catalyst.analysis.package$AnalysisErrorAt.failAnalysis(package.scala:42) > at > org.apache.spark.sql.catalyst.analysis.CheckAnalysis$$anonfun$checkAnalysis$1$$anonfun$apply$2.applyOrElse(CheckAnalysis.scala:61) > at > org.apache.spark.sql.catalyst.analysis.CheckAnalysis$$anonfun$checkAnalysis$1$$anonfun$apply$2.applyOrElse(CheckAnalysis.scala:53) > at > org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$transformUp$1.apply(TreeNode.scala:293) > at > org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$transformUp$1.apply(TreeNode.scala:293) > at > org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(TreeNode.scala:51) > at > org.apache.spark.sql.catalyst.trees.TreeNode.transformUp(TreeNode.scala:292) > at > org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$5.apply(TreeNode.scala:290) > at org.apache.spark.sql. > {code} > This is a rather annoying bug since the SparkR documentation currently > suggests that dates are now supported in SparkR. -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org