[
https://issues.apache.org/jira/browse/SPARK-6054?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Armbrust updated SPARK-6054:
------------------------------------
Description:
The following code fails with a stack trace beginning with:
{code}
15/02/26 23:21:32 ERROR Executor: Exception in task 2.0 in stage 7.0 (TID 422)
org.apache.spark.sql.catalyst.errors.package$TreeNodeException: makeCopy, tree:
scalaUDF(sales#2,discounts#3)
at
org.apache.spark.sql.catalyst.errors.package$.attachTree(package.scala:47)
at
org.apache.spark.sql.catalyst.trees.TreeNode.makeCopy(TreeNode.scala:309)
at
org.apache.spark.sql.catalyst.trees.TreeNode.transformChildrenDown(TreeNode.scala:237)
at
org.apache.spark.sql.catalyst.trees.TreeNode.transformDown(TreeNode.scala:192)
at
org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$4.apply(TreeNode.scala:207)
{code}
Here is the 1.3.0 version of the code:
{code}
case class SalesDisc(sales: Double, discounts: Double)
def makeStruct(sales: Double, disc:Double) = SalesDisc(sales, disc)
sqlContext.udf.register("makeStruct", makeStruct _)
val withStruct =
sqlContext.sql("SELECT id, sd.sales FROM (SELECT id, makeStruct(sales,
discounts) AS sd FROM customerTable) AS d")
withStruct.foreach(println)
{code}
This used to work in 1.2.0. Interestingly, the following simplified version
fails similarly, even though it seems to me to be VERY similar to the last test
in the UDFSuite:
{code}
SELECT makeStruct(sales, discounts) AS sd FROM customerTable
{code}
The data table is defined thus:
{code}
val custs = Seq(
Cust(1, "Widget Co", 120000.00, 0.00, "AZ"),
Cust(2, "Acme Widgets", 410500.00, 500.00, "CA"),
Cust(3, "Widgetry", 410500.00, 200.00, "CA"),
Cust(4, "Widgets R Us", 410500.00, 0.0, "CA"),
Cust(5, "Ye Olde Widgete", 500.00, 0.0, "MA")
)
val customerTable = sc.parallelize(custs, 4).toDF()
customerTable.registerTempTable("customerTable")
{code}
was:
The following code fails with a stack trace beginning with:
15/02/26 23:21:32 ERROR Executor: Exception in task 2.0 in stage 7.0 (TID 422)
org.apache.spark.sql.catalyst.errors.package$TreeNodeException: makeCopy, tree:
scalaUDF(sales#2,discounts#3)
at
org.apache.spark.sql.catalyst.errors.package$.attachTree(package.scala:47)
at
org.apache.spark.sql.catalyst.trees.TreeNode.makeCopy(TreeNode.scala:309)
at
org.apache.spark.sql.catalyst.trees.TreeNode.transformChildrenDown(TreeNode.scala:237)
at
org.apache.spark.sql.catalyst.trees.TreeNode.transformDown(TreeNode.scala:192)
at
org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$4.apply(TreeNode.scala:207)
Here is the 1.3.0 version of the code:
case class SalesDisc(sales: Double, discounts: Double)
def makeStruct(sales: Double, disc:Double) = SalesDisc(sales, disc)
sqlContext.udf.register("makeStruct", makeStruct _)
val withStruct =
sqlContext.sql("SELECT id, sd.sales FROM (SELECT id, makeStruct(sales,
discounts) AS sd FROM customerTable) AS d")
withStruct.foreach(println)
This used to work in 1.2.0. Interestingly, the following simplified version
fails similarly, even though it seems to me to be VERY similar to the last test
in the UDFSuite:
SELECT makeStruct(sales, discounts) AS sd FROM customerTable
The data table is defined thus:
val custs = Seq(
Cust(1, "Widget Co", 120000.00, 0.00, "AZ"),
Cust(2, "Acme Widgets", 410500.00, 500.00, "CA"),
Cust(3, "Widgetry", 410500.00, 200.00, "CA"),
Cust(4, "Widgets R Us", 410500.00, 0.0, "CA"),
Cust(5, "Ye Olde Widgete", 500.00, 0.0, "MA")
)
val customerTable = sc.parallelize(custs, 4).toDF()
customerTable.registerTempTable("customerTable")
> SQL UDF returning object of case class; regression from 1.2.0
> -------------------------------------------------------------
>
> Key: SPARK-6054
> URL: https://issues.apache.org/jira/browse/SPARK-6054
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 1.3.0
> Environment: Windows 8, Scala 2.11.2, Spark 1.3.0 RC1
> Reporter: Spiro Michaylov
>
> The following code fails with a stack trace beginning with:
> {code}
> 15/02/26 23:21:32 ERROR Executor: Exception in task 2.0 in stage 7.0 (TID 422)
> org.apache.spark.sql.catalyst.errors.package$TreeNodeException: makeCopy,
> tree: scalaUDF(sales#2,discounts#3)
> at
> org.apache.spark.sql.catalyst.errors.package$.attachTree(package.scala:47)
> at
> org.apache.spark.sql.catalyst.trees.TreeNode.makeCopy(TreeNode.scala:309)
> at
> org.apache.spark.sql.catalyst.trees.TreeNode.transformChildrenDown(TreeNode.scala:237)
> at
> org.apache.spark.sql.catalyst.trees.TreeNode.transformDown(TreeNode.scala:192)
> at
> org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$4.apply(TreeNode.scala:207)
> {code}
> Here is the 1.3.0 version of the code:
> {code}
> case class SalesDisc(sales: Double, discounts: Double)
> def makeStruct(sales: Double, disc:Double) = SalesDisc(sales, disc)
> sqlContext.udf.register("makeStruct", makeStruct _)
> val withStruct =
> sqlContext.sql("SELECT id, sd.sales FROM (SELECT id, makeStruct(sales,
> discounts) AS sd FROM customerTable) AS d")
> withStruct.foreach(println)
> {code}
> This used to work in 1.2.0. Interestingly, the following simplified version
> fails similarly, even though it seems to me to be VERY similar to the last
> test in the UDFSuite:
> {code}
> SELECT makeStruct(sales, discounts) AS sd FROM customerTable
> {code}
> The data table is defined thus:
> {code}
> val custs = Seq(
> Cust(1, "Widget Co", 120000.00, 0.00, "AZ"),
> Cust(2, "Acme Widgets", 410500.00, 500.00, "CA"),
> Cust(3, "Widgetry", 410500.00, 200.00, "CA"),
> Cust(4, "Widgets R Us", 410500.00, 0.0, "CA"),
> Cust(5, "Ye Olde Widgete", 500.00, 0.0, "MA")
> )
> val customerTable = sc.parallelize(custs, 4).toDF()
> customerTable.registerTempTable("customerTable")
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
