[jira] [Commented] (SPARK-11827) Support java.math.BigInteger in Type-Inference utilities for POJOs
[ https://issues.apache.org/jira/browse/SPARK-11827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15294219#comment-15294219 ] Apache Spark commented on SPARK-11827: -- User 'ted-yu' has created a pull request for this issue: https://github.com/apache/spark/pull/13233 > Support java.math.BigInteger in Type-Inference utilities for POJOs > -- > > Key: SPARK-11827 > URL: https://issues.apache.org/jira/browse/SPARK-11827 > Project: Spark > Issue Type: Improvement > Components: SQL >Affects Versions: 1.5.1 >Reporter: Abhilash Srimat Tirumala Pallerlamudi >Assignee: kevin yu >Priority: Minor > Fix For: 2.0.0 > > > I get the below exception when creating DataFrame using RDD of JavaBean > having a property of type java.math.BigInteger > scala.MatchError: class java.math.BigInteger (of class java.lang.Class) > at > org.apache.spark.sql.SQLContext$$anonfun$getSchema$1.apply(SQLContext.scala:1182) > at > org.apache.spark.sql.SQLContext$$anonfun$getSchema$1.apply(SQLContext.scala:1181) > at > scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244) > at > scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244) > at > scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33) > at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:108) > at > scala.collection.TraversableLike$class.map(TraversableLike.scala:244) > at scala.collection.mutable.ArrayOps$ofRef.map(ArrayOps.scala:108) > at org.apache.spark.sql.SQLContext.getSchema(SQLContext.scala:1181) > at > org.apache.spark.sql.SQLContext.createDataFrame(SQLContext.scala:419) > at > org.apache.spark.sql.SQLContext.createDataFrame(SQLContext.scala:447) > I don't see the support for java.math.BigInteger in > org.apache.spark.sql.catalyst.JavaTypeInference.scala -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-11827) Support java.math.BigInteger in Type-Inference utilities for POJOs
[ https://issues.apache.org/jira/browse/SPARK-11827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15293127#comment-15293127 ] Maciej Bryński commented on SPARK-11827: [~kevinyu98] Could you check https://issues.apache.org/jira/browse/SPARK-15445 ? > Support java.math.BigInteger in Type-Inference utilities for POJOs > -- > > Key: SPARK-11827 > URL: https://issues.apache.org/jira/browse/SPARK-11827 > Project: Spark > Issue Type: Improvement > Components: SQL >Affects Versions: 1.5.1 >Reporter: Abhilash Srimat Tirumala Pallerlamudi >Assignee: kevin yu >Priority: Minor > Fix For: 2.0.0 > > > I get the below exception when creating DataFrame using RDD of JavaBean > having a property of type java.math.BigInteger > scala.MatchError: class java.math.BigInteger (of class java.lang.Class) > at > org.apache.spark.sql.SQLContext$$anonfun$getSchema$1.apply(SQLContext.scala:1182) > at > org.apache.spark.sql.SQLContext$$anonfun$getSchema$1.apply(SQLContext.scala:1181) > at > scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244) > at > scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244) > at > scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33) > at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:108) > at > scala.collection.TraversableLike$class.map(TraversableLike.scala:244) > at scala.collection.mutable.ArrayOps$ofRef.map(ArrayOps.scala:108) > at org.apache.spark.sql.SQLContext.getSchema(SQLContext.scala:1181) > at > org.apache.spark.sql.SQLContext.createDataFrame(SQLContext.scala:419) > at > org.apache.spark.sql.SQLContext.createDataFrame(SQLContext.scala:447) > I don't see the support for java.math.BigInteger in > org.apache.spark.sql.catalyst.JavaTypeInference.scala -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-11827) Support java.math.BigInteger in Type-Inference utilities for POJOs
[ https://issues.apache.org/jira/browse/SPARK-11827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15038038#comment-15038038 ] Apache Spark commented on SPARK-11827: -- User 'kevinyu98' has created a pull request for this issue: https://github.com/apache/spark/pull/10125 > Support java.math.BigInteger in Type-Inference utilities for POJOs > -- > > Key: SPARK-11827 > URL: https://issues.apache.org/jira/browse/SPARK-11827 > Project: Spark > Issue Type: Improvement > Components: SQL >Affects Versions: 1.5.1 >Reporter: Abhilash Srimat Tirumala Pallerlamudi >Priority: Minor > > I get the below exception when creating DataFrame using RDD of JavaBean > having a property of type java.math.BigInteger > scala.MatchError: class java.math.BigInteger (of class java.lang.Class) > at > org.apache.spark.sql.SQLContext$$anonfun$getSchema$1.apply(SQLContext.scala:1182) > at > org.apache.spark.sql.SQLContext$$anonfun$getSchema$1.apply(SQLContext.scala:1181) > at > scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244) > at > scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244) > at > scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33) > at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:108) > at > scala.collection.TraversableLike$class.map(TraversableLike.scala:244) > at scala.collection.mutable.ArrayOps$ofRef.map(ArrayOps.scala:108) > at org.apache.spark.sql.SQLContext.getSchema(SQLContext.scala:1181) > at > org.apache.spark.sql.SQLContext.createDataFrame(SQLContext.scala:419) > at > org.apache.spark.sql.SQLContext.createDataFrame(SQLContext.scala:447) > I don't see the support for java.math.BigInteger in > org.apache.spark.sql.catalyst.JavaTypeInference.scala -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-11827) Support java.math.BigInteger in Type-Inference utilities for POJOs
[ https://issues.apache.org/jira/browse/SPARK-11827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15014042#comment-15014042 ] kevin yu commented on SPARK-11827: -- I think I can recreate the problem now. I am looking into the code. scala.MatchError: 1234567 (of class java.math.BigInteger) at org.apache.spark.sql.catalyst.CatalystTypeConverters$StructConverter.toCatalystImpl(CatalystTypeConverters.scala:255) at org.apache.spark.sql.catalyst.CatalystTypeConverters$StructConverter.toCatalystImpl(CatalystTypeConverters.scala:250) at org.apache.spark.sql.catalyst.CatalystTypeConverters$CatalystTypeConverter.toCatalyst(CatalystTypeConverters.scala:102) at org.apache.spark.sql.catalyst.CatalystTypeConverters$$anonfun$createToCatalystConverter$2.apply(CatalystTypeConverters.scala:401) at org.apache.spark.sql.SQLContext$$anonfun$org$apache$spark$sql$SQLContext$$beansToRows$1$$anonfun$apply$1.apply(SQLContext.scala:1314) at org.apache.spark.sql.SQLContext$$anonfun$org$apache$spark$sql$SQLContext$$beansToRows$1$$anonfun$apply$1.apply(SQLContext.scala:1314) at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244) at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244) at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33) > Support java.math.BigInteger in Type-Inference utilities for POJOs > -- > > Key: SPARK-11827 > URL: https://issues.apache.org/jira/browse/SPARK-11827 > Project: Spark > Issue Type: Improvement > Components: SQL >Affects Versions: 1.5.1 >Reporter: Abhilash Srimat Tirumala Pallerlamudi >Priority: Minor > > I get the below exception when creating DataFrame using RDD of JavaBean > having a property of type java.math.BigInteger > scala.MatchError: class java.math.BigInteger (of class java.lang.Class) > at > org.apache.spark.sql.SQLContext$$anonfun$getSchema$1.apply(SQLContext.scala:1182) > at > org.apache.spark.sql.SQLContext$$anonfun$getSchema$1.apply(SQLContext.scala:1181) > at > scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244) > at > scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244) > at > scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33) > at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:108) > at > scala.collection.TraversableLike$class.map(TraversableLike.scala:244) > at scala.collection.mutable.ArrayOps$ofRef.map(ArrayOps.scala:108) > at org.apache.spark.sql.SQLContext.getSchema(SQLContext.scala:1181) > at > org.apache.spark.sql.SQLContext.createDataFrame(SQLContext.scala:419) > at > org.apache.spark.sql.SQLContext.createDataFrame(SQLContext.scala:447) > I don't see the support for java.math.BigInteger in > org.apache.spark.sql.catalyst.JavaTypeInference.scala -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-11827) Support java.math.BigInteger in Type-Inference utilities for POJOs
[ https://issues.apache.org/jira/browse/SPARK-11827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15012165#comment-15012165 ] kevin yu commented on SPARK-11827: -- I will take a look at this one. Kevin > Support java.math.BigInteger in Type-Inference utilities for POJOs > -- > > Key: SPARK-11827 > URL: https://issues.apache.org/jira/browse/SPARK-11827 > Project: Spark > Issue Type: Improvement > Components: SQL >Affects Versions: 1.5.1 >Reporter: Abhilash Srimat Tirumala Pallerlamudi >Priority: Minor > > I get the below exception when creating DataFrame using RDD of JavaBean > having a property of type java.math.BigInteger > scala.MatchError: class java.math.BigInteger (of class java.lang.Class) > at > org.apache.spark.sql.SQLContext$$anonfun$getSchema$1.apply(SQLContext.scala:1182) > at > org.apache.spark.sql.SQLContext$$anonfun$getSchema$1.apply(SQLContext.scala:1181) > at > scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244) > at > scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244) > at > scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33) > at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:108) > at > scala.collection.TraversableLike$class.map(TraversableLike.scala:244) > at scala.collection.mutable.ArrayOps$ofRef.map(ArrayOps.scala:108) > at org.apache.spark.sql.SQLContext.getSchema(SQLContext.scala:1181) > at > org.apache.spark.sql.SQLContext.createDataFrame(SQLContext.scala:419) > at > org.apache.spark.sql.SQLContext.createDataFrame(SQLContext.scala:447) > I don't see the support for java.math.BigInteger in > org.apache.spark.sql.catalyst.JavaTypeInference.scala -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
