[GitHub] HyukjinKwon commented on issue #23349: [SPARK-26403][SQL] Support pivoting using array column for `pivot(column)` API
HyukjinKwon commented on issue #23349: [SPARK-26403][SQL] Support pivoting using array column for `pivot(column)` API URL: https://github.com/apache/spark/pull/23349#issuecomment-451045810 Haha, looks missed. I merged instead :D. Happy new year Wenchen. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] HyukjinKwon commented on issue #23349: [SPARK-26403][SQL] Support pivoting using array column for `pivot(column)` API
HyukjinKwon commented on issue #23349: [SPARK-26403][SQL] Support pivoting using array column for `pivot(column)` API URL: https://github.com/apache/spark/pull/23349#issuecomment-450537268 Hey, @cloud-fan, I think this change doesn't make `Literal` or `pivot` worse. If you worry about that we support `WrappedArray` in `Literal`, I can only make within `pivot`. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] HyukjinKwon commented on issue #23349: [SPARK-26403][SQL] Support pivoting using array column for `pivot(column)` API
HyukjinKwon commented on issue #23349: [SPARK-26403][SQL] Support pivoting using array column for `pivot(column)` API URL: https://github.com/apache/spark/pull/23349#issuecomment-448860344 BTW, if we give the `values` by `pivot(column, values)` API, it works as described in JIRA. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] HyukjinKwon commented on issue #23349: [SPARK-26403][SQL] Support pivoting using array column for `pivot(column)` API
HyukjinKwon commented on issue #23349: [SPARK-26403][SQL] Support pivoting using array column for `pivot(column)` API URL: https://github.com/apache/spark/pull/23349#issuecomment-448732387 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] HyukjinKwon commented on issue #23349: [SPARK-26403][SQL] Support pivoting using array column for `pivot(column)` API
HyukjinKwon commented on issue #23349: [SPARK-26403][SQL] Support pivoting using array column for `pivot(column)` API URL: https://github.com/apache/spark/pull/23349#issuecomment-448535079 cc @cloud-fan During checking the codes, I found two things below that I would like to mention. 1.. `Literal.create` supports this when input type is explicit. ```scala scala> Literal.create(collection.mutable.WrappedArray.make(Array(1))) res0: org.apache.spark.sql.catalyst.expressions.Literal = [1] scala> Literal.create(collection.mutable.WrappedArray.make(Array(1)): Seq[Int]) res1: org.apache.spark.sql.catalyst.expressions.Literal = [1] ``` However, currently not when we don't know the type: ```scala scala> Literal.create(collection.mutable.WrappedArray.make(Array(1)): Any) java.lang.RuntimeException: Unsupported literal type class scala.collection.mutable.WrappedArray$ofInt WrappedArray(1) at org.apache.spark.sql.catalyst.expressions.Literal$.apply(literals.scala:79) at org.apache.spark.sql.catalyst.expressions.Literal$.$anonfun$create$2(literals.scala:171) at scala.util.Failure.getOrElse(Try.scala:222) at org.apache.spark.sql.catalyst.expressions.Literal$.create(literals.scala:171) ... 49 elided ``` This PR also is going to fix it as below: ```scala scala> Literal.create(collection.mutable.WrappedArray.make(Array(1)): Any) res0: org.apache.spark.sql.catalyst.expressions.Literal = [1] ``` 2.. `Literal.apply(Any)` can't support `Seq[_]` for now: ```scala scala> Literal.apply(Seq(1)) java.lang.RuntimeException: Unsupported literal type class scala.collection.immutable.$colon$colon List(1) at org.apache.spark.sql.catalyst.expressions.Literal$.apply(literals.scala:79) ... 49 elided ``` I would like to leave these as a separate discussion out of this PR. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
