[jira] [Comment Edited] (SPARK-21391) Cannot convert a Seq of Map whose value type is again a seq, into a dataset
[
https://issues.apache.org/jira/browse/SPARK-21391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16085099#comment-16085099
]
Kazuaki Ishizaki edited comment on SPARK-21391 at 7/13/17 3:42 AM:
---
[~hyukjin.kwon] I think that SPARK-19254 and/or SPARK-19104 fixed this issue.
was (Author: kiszk):
[~hyukjin.kwon] I think that
[SPARK-19254|https://issues.apache.org/jira/browse/SPARK-19254] and/or
[SPARK-19104|https://issues.apache.org/jira/browse/SPARK-19104] fixed this
issue.
> Cannot convert a Seq of Map whose value type is again a seq, into a dataset
>
>
> Key: SPARK-21391
> URL: https://issues.apache.org/jira/browse/SPARK-21391
> Project: Spark
> Issue Type: Bug
> Components: SQL
>Affects Versions: 2.1.0
> Environment: Seen on mac OSX, scala 2.11, java 8
>Reporter: indraneel rao
>
> There is an error while trying to create a dataset from a sequence of Maps,
> whose values have any kind of collections. Even when they are wrapped in a
> case class.
> Eg : The following piece of code throws an error:
>
> {code:java}
> case class Values(values: Seq[Double])
> case class ItemProperties(properties:Map[String,Values])
> val l1 = List(ItemProperties(
> Map(
> "A1" -> Values(Seq(1.0,2.0)),
> "B1" -> Values(Seq(44.0,55.0))
> )
> ),
> ItemProperties(
> Map(
> "A2" -> Values(Seq(123.0,25.0)),
> "B2" -> Values(Seq(445.0,35.0))
> )
> )
> )
> l1.toDS().show()
> {code}
> Heres the error:
> 17/07/12 21:59:35 ERROR CodeGenerator: failed to compile:
> org.codehaus.commons.compiler.CompileException: File 'generated.java', Line
> 65, Column 46: Expression "ExternalMapToCatalyst_value_isNull0" is not an
> rvalue
> /* 001 */ public java.lang.Object generate(Object[] references) {
> /* 002 */ return new SpecificUnsafeProjection(references);
> /* 003 */ }
> /* 004 */
> /* 005 */ class SpecificUnsafeProjection extends
> org.apache.spark.sql.catalyst.expressions.UnsafeProjection {
> /* 006 */
> /* 007 */ private Object[] references;
> /* 008 */ private boolean resultIsNull;
> /* 009 */ private java.lang.String argValue;
> /* 010 */ private Object[] values;
> /* 011 */ private boolean resultIsNull1;
> /* 012 */ private scala.collection.Seq argValue1;
> /* 013 */ private boolean isNull11;
> /* 014 */ private boolean value11;
> /* 015 */ private boolean isNull12;
> /* 016 */ private InternalRow value12;
> /* 017 */ private boolean isNull13;
> /* 018 */ private InternalRow value13;
> /* 019 */ private UnsafeRow result;
> /* 020 */ private
> org.apache.spark.sql.catalyst.expressions.codegen.BufferHolder holder;
> /* 021 */ private
> org.apache.spark.sql.catalyst.expressions.codegen.UnsafeRowWriter rowWriter;
> /* 022 */ private
> org.apache.spark.sql.catalyst.expressions.codegen.UnsafeArrayWriter
> arrayWriter;
> /* 023 */ private
> org.apache.spark.sql.catalyst.expressions.codegen.UnsafeArrayWriter
> arrayWriter1;
> /* 024 */ private
> org.apache.spark.sql.catalyst.expressions.codegen.UnsafeRowWriter rowWriter1;
> /* 025 */ private
> org.apache.spark.sql.catalyst.expressions.codegen.UnsafeArrayWriter
> arrayWriter2;
> /* 026 */
> /* 027 */ public SpecificUnsafeProjection(Object[] references) {
> /* 028 */ this.references = references;
> /* 029 */
> /* 030 */
> /* 031 */ this.values = null;
> /* 032 */
> /* 033 */
> /* 034 */ isNull11 = false;
> /* 035 */ value11 = false;
> /* 036 */ isNull12 = false;
> /* 037 */ value12 = null;
> /* 038 */ isNull13 = false;
> /* 039 */ value13 = null;
> /* 040 */ result = new UnsafeRow(1);
> /* 041 */ this.holder = new
> org.apache.spark.sql.catalyst.expressions.codegen.BufferHolder(result, 32);
> /* 042 */ this.rowWriter = new
> org.apache.spark.sql.catalyst.expressions.codegen.UnsafeRowWriter(holder, 1);
> /* 043 */ this.arrayWriter = new
> org.apache.spark.sql.catalyst.expressions.codegen.UnsafeArrayWriter();
> /* 044 */ this.arrayWriter1 = new
> org.apache.spark.sql.catalyst.expressions.codegen.UnsafeArrayWriter();
> /* 045 */ this.rowWriter1 = new
> org.apache.spark.sql.catalyst.expressions.codegen.UnsafeRowWriter(holder, 1);
> /* 046 */ this.arrayWriter2 = new
> org.apache.spark.sql.catalyst.expressions.codegen.UnsafeArrayWriter();
> /* 047 */
> /* 048 */ }
> /* 049 */
> /* 050 */ public void initialize(int partitionIndex) {
> /* 051 */
> /* 052 */ }
> /* 053 */
> /* 054 */
> /* 055 */ private void evalIfTrueExpr(InternalRow i) {
> /* 056 */ final InternalRow value7 = null;
> /* 057 */ isNull12 = true;
> /* 058 */ value12 = value7;
> /* 059 */ }
> /* 060 */
> /* 061 */
> /* 062 */
[jira] [Comment Edited] (SPARK-21391) Cannot convert a Seq of Map whose value type is again a seq, into a dataset
[
https://issues.apache.org/jira/browse/SPARK-21391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16084333#comment-16084333
]
Kazuaki Ishizaki edited comment on SPARK-21391 at 7/12/17 5:19 PM:
---
This program works with the master or Spark 2.2. Would it be possible to use
Spark 2.2?
{code}
++
| properties|
++
|Map(A1 -> [Wrappe...|
|Map(A2 -> [Wrappe...|
++
{code}
was (Author: kiszk):
This program works with the master and Spark 2.2. Would it be possible to use
Spark 2.2?
{code}
++
| properties|
++
|Map(A1 -> [Wrappe...|
|Map(A2 -> [Wrappe...|
++
{code}
> Cannot convert a Seq of Map whose value type is again a seq, into a dataset
>
>
> Key: SPARK-21391
> URL: https://issues.apache.org/jira/browse/SPARK-21391
> Project: Spark
> Issue Type: Bug
> Components: SQL
>Affects Versions: 2.1.0
> Environment: Seen on mac OSX, scala 2.11, java 8
>Reporter: indraneel rao
>
> There is an error while trying to create a dataset from a sequence of Maps,
> whose values have any kind of collections. Even when they are wrapped in a
> case class.
> Eg : The following piece of code throws an error:
>
> {code:java}
> case class Values(values: Seq[Double])
> case class ItemProperties(properties:Map[String,Values])
> val l1 = List(ItemProperties(
> Map(
> "A1" -> Values(Seq(1.0,2.0)),
> "B1" -> Values(Seq(44.0,55.0))
> )
> ),
> ItemProperties(
> Map(
> "A2" -> Values(Seq(123.0,25.0)),
> "B2" -> Values(Seq(445.0,35.0))
> )
> )
> )
> l1.toDS().show()
> {code}
> Heres the error:
> 17/07/12 21:59:35 ERROR CodeGenerator: failed to compile:
> org.codehaus.commons.compiler.CompileException: File 'generated.java', Line
> 65, Column 46: Expression "ExternalMapToCatalyst_value_isNull0" is not an
> rvalue
> /* 001 */ public java.lang.Object generate(Object[] references) {
> /* 002 */ return new SpecificUnsafeProjection(references);
> /* 003 */ }
> /* 004 */
> /* 005 */ class SpecificUnsafeProjection extends
> org.apache.spark.sql.catalyst.expressions.UnsafeProjection {
> /* 006 */
> /* 007 */ private Object[] references;
> /* 008 */ private boolean resultIsNull;
> /* 009 */ private java.lang.String argValue;
> /* 010 */ private Object[] values;
> /* 011 */ private boolean resultIsNull1;
> /* 012 */ private scala.collection.Seq argValue1;
> /* 013 */ private boolean isNull11;
> /* 014 */ private boolean value11;
> /* 015 */ private boolean isNull12;
> /* 016 */ private InternalRow value12;
> /* 017 */ private boolean isNull13;
> /* 018 */ private InternalRow value13;
> /* 019 */ private UnsafeRow result;
> /* 020 */ private
> org.apache.spark.sql.catalyst.expressions.codegen.BufferHolder holder;
> /* 021 */ private
> org.apache.spark.sql.catalyst.expressions.codegen.UnsafeRowWriter rowWriter;
> /* 022 */ private
> org.apache.spark.sql.catalyst.expressions.codegen.UnsafeArrayWriter
> arrayWriter;
> /* 023 */ private
> org.apache.spark.sql.catalyst.expressions.codegen.UnsafeArrayWriter
> arrayWriter1;
> /* 024 */ private
> org.apache.spark.sql.catalyst.expressions.codegen.UnsafeRowWriter rowWriter1;
> /* 025 */ private
> org.apache.spark.sql.catalyst.expressions.codegen.UnsafeArrayWriter
> arrayWriter2;
> /* 026 */
> /* 027 */ public SpecificUnsafeProjection(Object[] references) {
> /* 028 */ this.references = references;
> /* 029 */
> /* 030 */
> /* 031 */ this.values = null;
> /* 032 */
> /* 033 */
> /* 034 */ isNull11 = false;
> /* 035 */ value11 = false;
> /* 036 */ isNull12 = false;
> /* 037 */ value12 = null;
> /* 038 */ isNull13 = false;
> /* 039 */ value13 = null;
> /* 040 */ result = new UnsafeRow(1);
> /* 041 */ this.holder = new
> org.apache.spark.sql.catalyst.expressions.codegen.BufferHolder(result, 32);
> /* 042 */ this.rowWriter = new
> org.apache.spark.sql.catalyst.expressions.codegen.UnsafeRowWriter(holder, 1);
> /* 043 */ this.arrayWriter = new
> org.apache.spark.sql.catalyst.expressions.codegen.UnsafeArrayWriter();
> /* 044 */ this.arrayWriter1 = new
> org.apache.spark.sql.catalyst.expressions.codegen.UnsafeArrayWriter();
> /* 045 */ this.rowWriter1 = new
> org.apache.spark.sql.catalyst.expressions.codegen.UnsafeRowWriter(holder, 1);
> /* 046 */ this.arrayWriter2 = new
> org.apache.spark.sql.catalyst.expressions.codegen.UnsafeArrayWriter();
> /* 047 */
> /* 048 */ }
> /* 049 */
> /* 050 */ public void initialize(int partitionIndex) {
> /* 051 */
> /* 052 */ }
> /* 053 */
> /* 054 */
> /* 055
[jira] [Comment Edited] (SPARK-21391) Cannot convert a Seq of Map whose value type is again a seq, into a dataset
[
https://issues.apache.org/jira/browse/SPARK-21391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16084333#comment-16084333
]
Kazuaki Ishizaki edited comment on SPARK-21391 at 7/12/17 5:19 PM:
---
This program works with the master and Spark 2.2. Would it be possible to use
Spark 2.2?
{code}
++
| properties|
++
|Map(A1 -> [Wrappe...|
|Map(A2 -> [Wrappe...|
++
{code}
was (Author: kiszk):
This program works with the master.
{code}
++
| properties|
++
|Map(A1 -> [Wrappe...|
|Map(A2 -> [Wrappe...|
++
{code}
> Cannot convert a Seq of Map whose value type is again a seq, into a dataset
>
>
> Key: SPARK-21391
> URL: https://issues.apache.org/jira/browse/SPARK-21391
> Project: Spark
> Issue Type: Bug
> Components: SQL
>Affects Versions: 2.1.0
> Environment: Seen on mac OSX, scala 2.11, java 8
>Reporter: indraneel rao
>
> There is an error while trying to create a dataset from a sequence of Maps,
> whose values have any kind of collections. Even when they are wrapped in a
> case class.
> Eg : The following piece of code throws an error:
>
> {code:java}
> case class Values(values: Seq[Double])
> case class ItemProperties(properties:Map[String,Values])
> val l1 = List(ItemProperties(
> Map(
> "A1" -> Values(Seq(1.0,2.0)),
> "B1" -> Values(Seq(44.0,55.0))
> )
> ),
> ItemProperties(
> Map(
> "A2" -> Values(Seq(123.0,25.0)),
> "B2" -> Values(Seq(445.0,35.0))
> )
> )
> )
> l1.toDS().show()
> {code}
> Heres the error:
> 17/07/12 21:59:35 ERROR CodeGenerator: failed to compile:
> org.codehaus.commons.compiler.CompileException: File 'generated.java', Line
> 65, Column 46: Expression "ExternalMapToCatalyst_value_isNull0" is not an
> rvalue
> /* 001 */ public java.lang.Object generate(Object[] references) {
> /* 002 */ return new SpecificUnsafeProjection(references);
> /* 003 */ }
> /* 004 */
> /* 005 */ class SpecificUnsafeProjection extends
> org.apache.spark.sql.catalyst.expressions.UnsafeProjection {
> /* 006 */
> /* 007 */ private Object[] references;
> /* 008 */ private boolean resultIsNull;
> /* 009 */ private java.lang.String argValue;
> /* 010 */ private Object[] values;
> /* 011 */ private boolean resultIsNull1;
> /* 012 */ private scala.collection.Seq argValue1;
> /* 013 */ private boolean isNull11;
> /* 014 */ private boolean value11;
> /* 015 */ private boolean isNull12;
> /* 016 */ private InternalRow value12;
> /* 017 */ private boolean isNull13;
> /* 018 */ private InternalRow value13;
> /* 019 */ private UnsafeRow result;
> /* 020 */ private
> org.apache.spark.sql.catalyst.expressions.codegen.BufferHolder holder;
> /* 021 */ private
> org.apache.spark.sql.catalyst.expressions.codegen.UnsafeRowWriter rowWriter;
> /* 022 */ private
> org.apache.spark.sql.catalyst.expressions.codegen.UnsafeArrayWriter
> arrayWriter;
> /* 023 */ private
> org.apache.spark.sql.catalyst.expressions.codegen.UnsafeArrayWriter
> arrayWriter1;
> /* 024 */ private
> org.apache.spark.sql.catalyst.expressions.codegen.UnsafeRowWriter rowWriter1;
> /* 025 */ private
> org.apache.spark.sql.catalyst.expressions.codegen.UnsafeArrayWriter
> arrayWriter2;
> /* 026 */
> /* 027 */ public SpecificUnsafeProjection(Object[] references) {
> /* 028 */ this.references = references;
> /* 029 */
> /* 030 */
> /* 031 */ this.values = null;
> /* 032 */
> /* 033 */
> /* 034 */ isNull11 = false;
> /* 035 */ value11 = false;
> /* 036 */ isNull12 = false;
> /* 037 */ value12 = null;
> /* 038 */ isNull13 = false;
> /* 039 */ value13 = null;
> /* 040 */ result = new UnsafeRow(1);
> /* 041 */ this.holder = new
> org.apache.spark.sql.catalyst.expressions.codegen.BufferHolder(result, 32);
> /* 042 */ this.rowWriter = new
> org.apache.spark.sql.catalyst.expressions.codegen.UnsafeRowWriter(holder, 1);
> /* 043 */ this.arrayWriter = new
> org.apache.spark.sql.catalyst.expressions.codegen.UnsafeArrayWriter();
> /* 044 */ this.arrayWriter1 = new
> org.apache.spark.sql.catalyst.expressions.codegen.UnsafeArrayWriter();
> /* 045 */ this.rowWriter1 = new
> org.apache.spark.sql.catalyst.expressions.codegen.UnsafeRowWriter(holder, 1);
> /* 046 */ this.arrayWriter2 = new
> org.apache.spark.sql.catalyst.expressions.codegen.UnsafeArrayWriter();
> /* 047 */
> /* 048 */ }
> /* 049 */
> /* 050 */ public void initialize(int partitionIndex) {
> /* 051 */
> /* 052 */ }
> /* 053 */
> /* 054 */
> /* 055 */ private void evalIfTrueExpr(InternalRow i) {
> /*
