[jira] [Updated] (SPARK-21393) spark (pyspark) crashes unpredictably when using show() or toPandas()

[Kazuaki Ishizaki (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SPARK-21393?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kazuaki Ishizaki updated SPARK-21393:
-
Affects Version/s: 2.2.0

> spark (pyspark) crashes unpredictably when using show() or toPandas()
> -
>
> Key: SPARK-21393
> URL: https://issues.apache.org/jira/browse/SPARK-21393
> Project: Spark
>  Issue Type: Bug
>  Components: PySpark
>Affects Versions: 2.1.1, 2.2.0
> Environment: Windows 10
> python 2.7
>Reporter: Zahra
> Attachments: Data.zip, working_ST_pyspark.py
>
>
> unpredictbly run into this error either when using 
> `pyspark.sql.DataFrame.show()` or `pyspark.sql.DataFrame.toPandas()`
> error log starts with  (truncated) :
> {noformat}
> 17/07/12 16:03:09 ERROR CodeGenerator: failed to compile: 
> org.codehaus.janino.JaninoRuntimeException: Code of method 
> "apply_47$(Lorg/apache/spark/sql/catalyst/expressions/GeneratedClass$SpecificUnsafeProjection;Lorg/apache/spark/sql/catalyst/InternalRow;)V"
>  of class 
> "org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificUnsafeProjection"
>  grows beyond 64 KB
> /* 001 */ public java.lang.Object generate(Object[] references) {
> /* 002 */   return new SpecificUnsafeProjection(references);
> /* 003 */ }
> /* 004 */
> /* 005 */ class SpecificUnsafeProjection extends 
> org.apache.spark.sql.catalyst.expressions.UnsafeProjection {
> /* 006 */
> /* 007 */   private Object[] references;
> /* 008 */   private scala.collection.immutable.Set set;
> /* 009 */   private scala.collection.immutable.Set set1;
> /* 010 */   private scala.collection.immutable.Set set2;
> /* 011 */   private scala.collection.immutable.Set set3;
> /* 012 */   private UTF8String.IntWrapper wrapper;
> /* 013 */   private UTF8String.IntWrapper wrapper1;
> /* 014 */   private scala.collection.immutable.Set set4;
> /* 015 */   private UTF8String.IntWrapper wrapper2;
> /* 016 */   private UTF8String.IntWrapper wrapper3;
> /* 017 */   private scala.collection.immutable.Set set5;
> /* 018 */   private scala.collection.immutable.Set set6;
> /* 019 */   private scala.collection.immutable.Set set7;
> /* 020 */   private UTF8String.IntWrapper wrapper4;
> /* 021 */   private UTF8String.IntWrapper wrapper5;
> /* 022 */   private scala.collection.immutable.Set set8;
> /* 023 */   private UTF8String.IntWrapper wrapper6;
> /* 024 */   private UTF8String.IntWrapper wrapper7;
> /* 025 */   private scala.collection.immutable.Set set9;
> /* 026 */   private scala.collection.immutable.Set set10;
> /* 027 */   private scala.collection.immutable.Set set11;
> /* 028 */   private UTF8String.IntWrapper wrapper8;
> /* 029 */   private UTF8String.IntWrapper wrapper9;
> /* 030 */   private scala.collection.immutable.Set set12;
> /* 031 */   private UTF8String.IntWrapper wrapper10;
> /* 032 */   private UTF8String.IntWrapper wrapper11;
> /* 033 */   private scala.collection.immutable.Set set13;
> /* 034 */   private scala.collection.immutable.Set set14;
> /* 035 */   private scala.collection.immutable.Set set15;
> /* 036 */   private UTF8String.IntWrapper wrapper12;
> /* 037 */   private UTF8String.IntWrapper wrapper13;
> /* 038 */   private scala.collection.immutable.Set set16;
> /* 039 */   private UTF8String.IntWrapper wrapper14;
> /* 040 */   private UTF8String.IntWrapper wrapper15;
> /* 041 */   private scala.collection.immutable.Set set17;
> /* 042 */   private scala.collection.immutable.Set set18;
> /* 043 */   private scala.collection.immutable.Set set19;
> /* 044 */   private UTF8String.IntWrapper wrapper16;
> /* 045 */   private UTF8String.IntWrapper wrapper17;
> /* 046 */   private scala.collection.immutable.Set set20;
> /* 047 */   private UTF8String.IntWrapper wrapper18;
> /* 048 */   private UTF8String.IntWrapper wrapper19;
> /* 049 */   private scala.collection.immutable.Set set21;
> /* 050 */   private scala.collection.immutable.Set set22;
> /* 051 */   private scala.collection.immutable.Set set23;
> /* 052 */   private UTF8String.IntWrapper wrapper20;
> /* 053 */   private UTF8String.IntWrapper wrapper21;
> /* 054 */   private scala.collection.immutable.Set set24;
> /* 055 */   private UTF8String.IntWrapper wrapper22;
> /* 056 */   private UTF8String.IntWrapper wrapper23;
> /* 057 */   private scala.collection.immutable.Set set25;
> /* 058 */   private scala.collection.immutable.Set set26;
> /* 059 */   private scala.collection.immutable.Set set27;
> /* 060 */   private UTF8String.IntWrapper wrapper24;
> /* 061 */   private UTF8String.IntWrapper wrapper25;
> /* 062 */   private scala.collection.immutable.Set set28;
> /* 063 */   private UTF8String.IntWrapper wrapper26;
> /* 064 */   private UTF8String.IntWrapper wrapper27;
> /* 065 */   private scala.collection.immutable.Set set29;
> /* 066 */   private scala.collection.immu

[jira] [Updated] (SPARK-21393) spark (pyspark) crashes unpredictably when using show() or toPandas()

[Zahra (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SPARK-21393?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Zahra updated SPARK-21393:
--
Attachment: Data.zip

Hi,

This folder contains the code and the data. Please ignore the old code.

Thanks,
Zahra

On Thu, Jul 13, 2017 at 9:57 AM, Kazuaki Ishizaki (JIRA) 



> spark (pyspark) crashes unpredictably when using show() or toPandas()
> -
>
> Key: SPARK-21393
> URL: https://issues.apache.org/jira/browse/SPARK-21393
> Project: Spark
>  Issue Type: Bug
>  Components: PySpark
>Affects Versions: 2.1.1
> Environment: Windows 10
> python 2.7
>Reporter: Zahra
> Attachments: Data.zip, working_ST_pyspark.py
>
>
> unpredictbly run into this error either when using 
> `pyspark.sql.DataFrame.show()` or `pyspark.sql.DataFrame.toPandas()`
> error log starts with  (truncated) :
> {noformat}
> 17/07/12 16:03:09 ERROR CodeGenerator: failed to compile: 
> org.codehaus.janino.JaninoRuntimeException: Code of method 
> "apply_47$(Lorg/apache/spark/sql/catalyst/expressions/GeneratedClass$SpecificUnsafeProjection;Lorg/apache/spark/sql/catalyst/InternalRow;)V"
>  of class 
> "org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificUnsafeProjection"
>  grows beyond 64 KB
> /* 001 */ public java.lang.Object generate(Object[] references) {
> /* 002 */   return new SpecificUnsafeProjection(references);
> /* 003 */ }
> /* 004 */
> /* 005 */ class SpecificUnsafeProjection extends 
> org.apache.spark.sql.catalyst.expressions.UnsafeProjection {
> /* 006 */
> /* 007 */   private Object[] references;
> /* 008 */   private scala.collection.immutable.Set set;
> /* 009 */   private scala.collection.immutable.Set set1;
> /* 010 */   private scala.collection.immutable.Set set2;
> /* 011 */   private scala.collection.immutable.Set set3;
> /* 012 */   private UTF8String.IntWrapper wrapper;
> /* 013 */   private UTF8String.IntWrapper wrapper1;
> /* 014 */   private scala.collection.immutable.Set set4;
> /* 015 */   private UTF8String.IntWrapper wrapper2;
> /* 016 */   private UTF8String.IntWrapper wrapper3;
> /* 017 */   private scala.collection.immutable.Set set5;
> /* 018 */   private scala.collection.immutable.Set set6;
> /* 019 */   private scala.collection.immutable.Set set7;
> /* 020 */   private UTF8String.IntWrapper wrapper4;
> /* 021 */   private UTF8String.IntWrapper wrapper5;
> /* 022 */   private scala.collection.immutable.Set set8;
> /* 023 */   private UTF8String.IntWrapper wrapper6;
> /* 024 */   private UTF8String.IntWrapper wrapper7;
> /* 025 */   private scala.collection.immutable.Set set9;
> /* 026 */   private scala.collection.immutable.Set set10;
> /* 027 */   private scala.collection.immutable.Set set11;
> /* 028 */   private UTF8String.IntWrapper wrapper8;
> /* 029 */   private UTF8String.IntWrapper wrapper9;
> /* 030 */   private scala.collection.immutable.Set set12;
> /* 031 */   private UTF8String.IntWrapper wrapper10;
> /* 032 */   private UTF8String.IntWrapper wrapper11;
> /* 033 */   private scala.collection.immutable.Set set13;
> /* 034 */   private scala.collection.immutable.Set set14;
> /* 035 */   private scala.collection.immutable.Set set15;
> /* 036 */   private UTF8String.IntWrapper wrapper12;
> /* 037 */   private UTF8String.IntWrapper wrapper13;
> /* 038 */   private scala.collection.immutable.Set set16;
> /* 039 */   private UTF8String.IntWrapper wrapper14;
> /* 040 */   private UTF8String.IntWrapper wrapper15;
> /* 041 */   private scala.collection.immutable.Set set17;
> /* 042 */   private scala.collection.immutable.Set set18;
> /* 043 */   private scala.collection.immutable.Set set19;
> /* 044 */   private UTF8String.IntWrapper wrapper16;
> /* 045 */   private UTF8String.IntWrapper wrapper17;
> /* 046 */   private scala.collection.immutable.Set set20;
> /* 047 */   private UTF8String.IntWrapper wrapper18;
> /* 048 */   private UTF8String.IntWrapper wrapper19;
> /* 049 */   private scala.collection.immutable.Set set21;
> /* 050 */   private scala.collection.immutable.Set set22;
> /* 051 */   private scala.collection.immutable.Set set23;
> /* 052 */   private UTF8String.IntWrapper wrapper20;
> /* 053 */   private UTF8String.IntWrapper wrapper21;
> /* 054 */   private scala.collection.immutable.Set set24;
> /* 055 */   private UTF8String.IntWrapper wrapper22;
> /* 056 */   private UTF8String.IntWrapper wrapper23;
> /* 057 */   private scala.collection.immutable.Set set25;
> /* 058 */   private scala.collection.immutable.Set set26;
> /* 059 */   private scala.collection.immutable.Set set27;
> /* 060 */   private UTF8String.IntWrapper wrapper24;
> /* 061 */   private UTF8String.IntWrapper wrapper25;
> /* 062 */   private scala.collection.immutable.Set set28;
> /* 063 */   private UTF8String.IntWrapper wrapper26;
> /* 064 */   private UTF8String.IntW

[jira] [Updated] (SPARK-21393) spark (pyspark) crashes unpredictably when using show() or toPandas()

[Zahra (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SPARK-21393?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Zahra updated SPARK-21393:
--
Attachment: working_ST_pyspark.py

Hi,

Sure. Attached.

Thanks,
Zahra

On Wed, Jul 12, 2017 at 6:13 PM, Hyukjin Kwon (JIRA) 



> spark (pyspark) crashes unpredictably when using show() or toPandas()
> -
>
> Key: SPARK-21393
> URL: https://issues.apache.org/jira/browse/SPARK-21393
> Project: Spark
>  Issue Type: Bug
>  Components: PySpark
>Affects Versions: 2.1.1
> Environment: Windows 10
> python 2.7
>Reporter: Zahra
> Attachments: working_ST_pyspark.py
>
>
> unpredictbly run into this error either when using 
> `pyspark.sql.DataFrame.show()` or `pyspark.sql.DataFrame.toPandas()`
> error log starts with  (truncated) :
> {noformat}
> 17/07/12 16:03:09 ERROR CodeGenerator: failed to compile: 
> org.codehaus.janino.JaninoRuntimeException: Code of method 
> "apply_47$(Lorg/apache/spark/sql/catalyst/expressions/GeneratedClass$SpecificUnsafeProjection;Lorg/apache/spark/sql/catalyst/InternalRow;)V"
>  of class 
> "org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificUnsafeProjection"
>  grows beyond 64 KB
> /* 001 */ public java.lang.Object generate(Object[] references) {
> /* 002 */   return new SpecificUnsafeProjection(references);
> /* 003 */ }
> /* 004 */
> /* 005 */ class SpecificUnsafeProjection extends 
> org.apache.spark.sql.catalyst.expressions.UnsafeProjection {
> /* 006 */
> /* 007 */   private Object[] references;
> /* 008 */   private scala.collection.immutable.Set set;
> /* 009 */   private scala.collection.immutable.Set set1;
> /* 010 */   private scala.collection.immutable.Set set2;
> /* 011 */   private scala.collection.immutable.Set set3;
> /* 012 */   private UTF8String.IntWrapper wrapper;
> /* 013 */   private UTF8String.IntWrapper wrapper1;
> /* 014 */   private scala.collection.immutable.Set set4;
> /* 015 */   private UTF8String.IntWrapper wrapper2;
> /* 016 */   private UTF8String.IntWrapper wrapper3;
> /* 017 */   private scala.collection.immutable.Set set5;
> /* 018 */   private scala.collection.immutable.Set set6;
> /* 019 */   private scala.collection.immutable.Set set7;
> /* 020 */   private UTF8String.IntWrapper wrapper4;
> /* 021 */   private UTF8String.IntWrapper wrapper5;
> /* 022 */   private scala.collection.immutable.Set set8;
> /* 023 */   private UTF8String.IntWrapper wrapper6;
> /* 024 */   private UTF8String.IntWrapper wrapper7;
> /* 025 */   private scala.collection.immutable.Set set9;
> /* 026 */   private scala.collection.immutable.Set set10;
> /* 027 */   private scala.collection.immutable.Set set11;
> /* 028 */   private UTF8String.IntWrapper wrapper8;
> /* 029 */   private UTF8String.IntWrapper wrapper9;
> /* 030 */   private scala.collection.immutable.Set set12;
> /* 031 */   private UTF8String.IntWrapper wrapper10;
> /* 032 */   private UTF8String.IntWrapper wrapper11;
> /* 033 */   private scala.collection.immutable.Set set13;
> /* 034 */   private scala.collection.immutable.Set set14;
> /* 035 */   private scala.collection.immutable.Set set15;
> /* 036 */   private UTF8String.IntWrapper wrapper12;
> /* 037 */   private UTF8String.IntWrapper wrapper13;
> /* 038 */   private scala.collection.immutable.Set set16;
> /* 039 */   private UTF8String.IntWrapper wrapper14;
> /* 040 */   private UTF8String.IntWrapper wrapper15;
> /* 041 */   private scala.collection.immutable.Set set17;
> /* 042 */   private scala.collection.immutable.Set set18;
> /* 043 */   private scala.collection.immutable.Set set19;
> /* 044 */   private UTF8String.IntWrapper wrapper16;
> /* 045 */   private UTF8String.IntWrapper wrapper17;
> /* 046 */   private scala.collection.immutable.Set set20;
> /* 047 */   private UTF8String.IntWrapper wrapper18;
> /* 048 */   private UTF8String.IntWrapper wrapper19;
> /* 049 */   private scala.collection.immutable.Set set21;
> /* 050 */   private scala.collection.immutable.Set set22;
> /* 051 */   private scala.collection.immutable.Set set23;
> /* 052 */   private UTF8String.IntWrapper wrapper20;
> /* 053 */   private UTF8String.IntWrapper wrapper21;
> /* 054 */   private scala.collection.immutable.Set set24;
> /* 055 */   private UTF8String.IntWrapper wrapper22;
> /* 056 */   private UTF8String.IntWrapper wrapper23;
> /* 057 */   private scala.collection.immutable.Set set25;
> /* 058 */   private scala.collection.immutable.Set set26;
> /* 059 */   private scala.collection.immutable.Set set27;
> /* 060 */   private UTF8String.IntWrapper wrapper24;
> /* 061 */   private UTF8String.IntWrapper wrapper25;
> /* 062 */   private scala.collection.immutable.Set set28;
> /* 063 */   private UTF8String.IntWrapper wrapper26;
> /* 064 */   private UTF8String.IntWrapper wrapper27;
> /* 065 */   private scala.collection.

[jira] [Updated] (SPARK-21393) spark (pyspark) crashes unpredictably when using show() or toPandas()

[Hyukjin Kwon (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SPARK-21393?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Hyukjin Kwon updated SPARK-21393:
-
Affects Version/s: (was: 2.2.1)

> spark (pyspark) crashes unpredictably when using show() or toPandas()
> -
>
> Key: SPARK-21393
> URL: https://issues.apache.org/jira/browse/SPARK-21393
> Project: Spark
>  Issue Type: Bug
>  Components: PySpark
>Affects Versions: 2.1.1
> Environment: Windows 10
> python 2.7
>Reporter: Zahra
>
> unpredictbly run into this error either when using 
> `pyspark.sql.DataFrame.show()` or `pyspark.sql.DataFrame.toPandas()`
> error log starts with  (truncated) :
> {noformat}
> 17/07/12 16:03:09 ERROR CodeGenerator: failed to compile: 
> org.codehaus.janino.JaninoRuntimeException: Code of method 
> "apply_47$(Lorg/apache/spark/sql/catalyst/expressions/GeneratedClass$SpecificUnsafeProjection;Lorg/apache/spark/sql/catalyst/InternalRow;)V"
>  of class 
> "org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificUnsafeProjection"
>  grows beyond 64 KB
> /* 001 */ public java.lang.Object generate(Object[] references) {
> /* 002 */   return new SpecificUnsafeProjection(references);
> /* 003 */ }
> /* 004 */
> /* 005 */ class SpecificUnsafeProjection extends 
> org.apache.spark.sql.catalyst.expressions.UnsafeProjection {
> /* 006 */
> /* 007 */   private Object[] references;
> /* 008 */   private scala.collection.immutable.Set set;
> /* 009 */   private scala.collection.immutable.Set set1;
> /* 010 */   private scala.collection.immutable.Set set2;
> /* 011 */   private scala.collection.immutable.Set set3;
> /* 012 */   private UTF8String.IntWrapper wrapper;
> /* 013 */   private UTF8String.IntWrapper wrapper1;
> /* 014 */   private scala.collection.immutable.Set set4;
> /* 015 */   private UTF8String.IntWrapper wrapper2;
> /* 016 */   private UTF8String.IntWrapper wrapper3;
> /* 017 */   private scala.collection.immutable.Set set5;
> /* 018 */   private scala.collection.immutable.Set set6;
> /* 019 */   private scala.collection.immutable.Set set7;
> /* 020 */   private UTF8String.IntWrapper wrapper4;
> /* 021 */   private UTF8String.IntWrapper wrapper5;
> /* 022 */   private scala.collection.immutable.Set set8;
> /* 023 */   private UTF8String.IntWrapper wrapper6;
> /* 024 */   private UTF8String.IntWrapper wrapper7;
> /* 025 */   private scala.collection.immutable.Set set9;
> /* 026 */   private scala.collection.immutable.Set set10;
> /* 027 */   private scala.collection.immutable.Set set11;
> /* 028 */   private UTF8String.IntWrapper wrapper8;
> /* 029 */   private UTF8String.IntWrapper wrapper9;
> /* 030 */   private scala.collection.immutable.Set set12;
> /* 031 */   private UTF8String.IntWrapper wrapper10;
> /* 032 */   private UTF8String.IntWrapper wrapper11;
> /* 033 */   private scala.collection.immutable.Set set13;
> /* 034 */   private scala.collection.immutable.Set set14;
> /* 035 */   private scala.collection.immutable.Set set15;
> /* 036 */   private UTF8String.IntWrapper wrapper12;
> /* 037 */   private UTF8String.IntWrapper wrapper13;
> /* 038 */   private scala.collection.immutable.Set set16;
> /* 039 */   private UTF8String.IntWrapper wrapper14;
> /* 040 */   private UTF8String.IntWrapper wrapper15;
> /* 041 */   private scala.collection.immutable.Set set17;
> /* 042 */   private scala.collection.immutable.Set set18;
> /* 043 */   private scala.collection.immutable.Set set19;
> /* 044 */   private UTF8String.IntWrapper wrapper16;
> /* 045 */   private UTF8String.IntWrapper wrapper17;
> /* 046 */   private scala.collection.immutable.Set set20;
> /* 047 */   private UTF8String.IntWrapper wrapper18;
> /* 048 */   private UTF8String.IntWrapper wrapper19;
> /* 049 */   private scala.collection.immutable.Set set21;
> /* 050 */   private scala.collection.immutable.Set set22;
> /* 051 */   private scala.collection.immutable.Set set23;
> /* 052 */   private UTF8String.IntWrapper wrapper20;
> /* 053 */   private UTF8String.IntWrapper wrapper21;
> /* 054 */   private scala.collection.immutable.Set set24;
> /* 055 */   private UTF8String.IntWrapper wrapper22;
> /* 056 */   private UTF8String.IntWrapper wrapper23;
> /* 057 */   private scala.collection.immutable.Set set25;
> /* 058 */   private scala.collection.immutable.Set set26;
> /* 059 */   private scala.collection.immutable.Set set27;
> /* 060 */   private UTF8String.IntWrapper wrapper24;
> /* 061 */   private UTF8String.IntWrapper wrapper25;
> /* 062 */   private scala.collection.immutable.Set set28;
> /* 063 */   private UTF8String.IntWrapper wrapper26;
> /* 064 */   private UTF8String.IntWrapper wrapper27;
> /* 065 */   private scala.collection.immutable.Set set29;
> /* 066 */   private scala.collection.immutable.Set set30;
> /* 067 */   private scala.collection.immut