[jira] [Comment Edited] (SPARK-19372) Code generation for Filter predicate including many OR conditions exceeds JVM method size limit
[ https://issues.apache.org/jira/browse/SPARK-19372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16124982#comment-16124982 ] Kazuaki Ishizaki edited comment on SPARK-19372 at 8/13/17 5:05 PM: --- [~srinivasanm] I can reproduce this issue by using the master branch. I think that this is another problem. Could you please create another JIRA entry to track this issue? I will work for this. was (Author: kiszk): [~srinivasanm] I can reproduce this issue by using the master branch. I think that this is another problem. Could you please create another JIRA entry to track this issue? > Code generation for Filter predicate including many OR conditions exceeds JVM > method size limit > > > Key: SPARK-19372 > URL: https://issues.apache.org/jira/browse/SPARK-19372 > Project: Spark > Issue Type: Bug >Affects Versions: 2.1.0 >Reporter: Jay Pranavamurthi >Assignee: Kazuaki Ishizaki > Fix For: 2.2.0, 2.3.0 > > Attachments: wide400cols.csv > > > For the attached csv file, the code below causes the exception > "org.codehaus.janino.JaninoRuntimeException: Code of method > "(Lorg/apache/spark/sql/catalyst/InternalRow;)Z" of class > "org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificPredicate" > grows beyond 64 KB > Code: > {code:borderStyle=solid} > val conf = new SparkConf().setMaster("local[1]") > val sqlContext = > SparkSession.builder().config(conf).getOrCreate().sqlContext > val dataframe = > sqlContext > .read > .format("com.databricks.spark.csv") > .load("wide400cols.csv") > val filter = (0 to 399) > .foldLeft(lit(false))((e, index) => > e.or(dataframe.col(dataframe.columns(index)) =!= s"column${index+1}")) > val filtered = dataframe.filter(filter) > filtered.show(100) > {code} -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Comment Edited] (SPARK-19372) Code generation for Filter predicate including many OR conditions exceeds JVM method size limit
[ https://issues.apache.org/jira/browse/SPARK-19372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16123029#comment-16123029 ] srinivasan edited comment on SPARK-19372 at 8/11/17 8:29 AM: - Hi [~kiszk], the fix does not work in 2.2.0 for select * from temp where not( Field1 = '' and Field2 = '' and Field3 = '' and Field4 = '' and Field5 = '' and BLANK_5 = '' and Field7 = '' and Field8 = '' and Field9 = '' and Field10 = '' and Field11 = '' and Field12 = '' and Field13 = '' and Field14 = '' and Field15 = '' and Field16 = '' and Field17 = '' and Field18 = '' and Field19 = '' and Field20 = '' and Field21 = '' and Field22 = '' and Field23 = '' and Field24 = '' and Field25 = '' and Field26 = '' and Field27 = '' and Field28 = '' and Field29 = '' and Field30 = '' and Field31 = '' and Field32 = '' and Field33 = '' and Field34 = '' and Field35 = '' and Field36 = '' and Field37 = '' and Field38 = '' and Field39 = '' and Field40 = '' and Field41 = '' and Field42 = '' and Field43 = '' and Field44 = '' and Field45 = '' and Field46 = '' and Field47 = '' and Field48 = '' and Field49 = '' and Field50 = '' and Field51 = '' and Field52 = '' and Field53 = '' and Field54 = '' and Field55 = '' and Field56 = '' and Field57 = '' and Field58 = '' and Field59 = '' and Field60 = '' and Field61 = '' and Field62 = '' and Field63 = '' and Field64 = '' and Field65 = '' and Field66 = '' and Field67 = '' and Field68 = '' and Field69 = '' and Field70 = '' and Field71 = '' and Field72 = '' and Field73 = '' and Field74 = '' and Field75 = '' and Field76 = '' and Field77 = '' and Field78 = '' and Field79 = '' and Field80 = '' and Field81 = '' and Field82 = '' and Field83 = '' and Field84 = '' and Field85 = '' and Field86 = '' and Field87 = '' and Field88 = '' and Field89 = '' and Field90 = '' and Field91 = '' and Field92 = '' and Field93 = '' and Field94 = '' and Field95 = '' and Field96 = '' and Field97 = '' and Field98 = '' and Field99 = '' and Field100 = '' and Field101 = '' and Field102 = '' and Field103 = '' and Field104 = '' and Field105 = '' and Field106 = '' and Field107 = '' and Field108 = '' and Field109 = '' and Field110 = '' and Field111 = '' and Field112 = '' and Field113 = '' and Field114 = '' and Field115 = '' and Field116 = '' and Field117 = '' and Field118 = '' and Field119 = '' and Field120 = '' and Field121 = '' and Field122 = '' and Field123 = '' and Field124 = '' and Field125 = '' and Field126 = '' and Field127 = '' and Field128 = '' and Field129 = '' and Field130 = '' and Field131 = '' and Field132 = '' and Field133 = '' and Field134 = '' and Field135 = '' and Field136 = '' and Field137 = '' and Field138 = '' and Field139 = '' and Field140 = '' and Field141 = '' and Field142 = '' and Field143 = '' and Field144 = '' and Field145 = '' and Field146 = '' and Field147 = '' and Field148 = '' and Field149 = '' and Field150 = '' and Field151 = '' and Field152 = '' and Field153 = '' and Field154 = '' and Field155 = '' and Field156 = '' and Field157 = '' and Field158 = '' and Field159 = '' and Field160 = '' and Field161 = '' and Field162 = '' and Field163 = '' and Field164 = '' and Field165 = '' and Field166 = '' and Field167 = '' and Field168 = '' and Field169 = '' and Field170 = '' and Field171 = '' and Field172 = '' and Field173 = '' and Field174 = '' and Field175 = '' and Field176 = '' and Field177 = '' and Field178 = '' and Field179 = '' and Field180 = '' and Field181 = '' and Field182 = '' and Field183 = '' and Field184 = '' and Field185 = '' and Field186 = '' and Field187 = '' and Field188 = '' and Field189 = '' and Field190 = '' and Field191 = '' and Field192 = '' and Field193 = '' and Field194 = '' and Field195 = '' and Field196 = '' and Field197 = '' and Field198 = '' and Field199 = '' and Field200 = '' and Field201 = '' and Field202 = '' and Field203 = '' and Field204 = '' and Field205 = '' and Field206 = '' and Field207 = '' and Field208 = '' and Field209 = '' and Field210 = '' and Field211 = '' and Field212 = '' and Field213 = '' and Field214 = '' and Field215 = '' and Field216 = '' and Field217 = '' and Field218 = '' and Field219 = '' and Field220 = '' and Field221 = '' and Field222 = '' and Field223 = '' and Field224 = '' and Field225 = '' and Field226 = '' and Field227 = '' and Field228 = '' and Field229 = '' and Field230 = '' and Field231 = '' and Field232 = '' and Field233 = '' and Field234 = '' and Field235 = '' and Field236 = '' and Field237 = '' and Field238 = '' and Field239 = '' and Field240 = '' and Field241 = '' and Field242 = '' and Field243 = '' and Field244 = '' and Field245 = '' and Field246 = '' and Field247
[jira] [Comment Edited] (SPARK-19372) Code generation for Filter predicate including many OR conditions exceeds JVM method size limit
[ https://issues.apache.org/jira/browse/SPARK-19372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16123029#comment-16123029 ] srinivasan edited comment on SPARK-19372 at 8/11/17 8:28 AM: - Hi [~kiszk], the fix does not work in 2.2.0 for select * from temp where not( Field1 = '' and Field2 = '' and Field3 = '' and Field4 = '' and Field5 = '' and BLANK_5 = '' and Field7 = '' and Field8 = '' and Field9 = '' and Field10 = '' and Field11 = '' and Field12 = '' and Field13 = '' and Field14 = '' and Field15 = '' and Field16 = '' and Field17 = '' and Field18 = '' and Field19 = '' and Field20 = '' and Field21 = '' and Field22 = '' and Field23 = '' and Field24 = '' and Field25 = '' and Field26 = '' and Field27 = '' and Field28 = '' and Field29 = '' and Field30 = '' and Field31 = '' and Field32 = '' and Field33 = '' and Field34 = '' and Field35 = '' and Field36 = '' and Field37 = '' and Field38 = '' and Field39 = '' and Field40 = '' and Field41 = '' and Field42 = '' and Field43 = '' and Field44 = '' and Field45 = '' and Field46 = '' and Field47 = '' and Field48 = '' and Field49 = '' and Field50 = '' and Field51 = '' and Field52 = '' and Field53 = '' and Field54 = '' and Field55 = '' and Field56 = '' and Field57 = '' and Field58 = '' and Field59 = '' and Field60 = '' and Field61 = '' and Field62 = '' and Field63 = '' and Field64 = '' and Field65 = '' and Field66 = '' and Field67 = '' and Field68 = '' and Field69 = '' and Field70 = '' and Field71 = '' and Field72 = '' and Field73 = '' and Field74 = '' and Field75 = '' and Field76 = '' and Field77 = '' and Field78 = '' and Field79 = '' and Field80 = '' and Field81 = '' and Field82 = '' and Field83 = '' and Field84 = '' and Field85 = '' and Field86 = '' and Field87 = '' and Field88 = '' and Field89 = '' and Field90 = '' and Field91 = '' and Field92 = '' and Field93 = '' and Field94 = '' and Field95 = '' and Field96 = '' and Field97 = '' and Field98 = '' and Field99 = '' and Field100 = '' and Field101 = '' and Field102 = '' and Field103 = '' and Field104 = '' and Field105 = '' and Field106 = '' and Field107 = '' and Field108 = '' and Field109 = '' and Field110 = '' and Field111 = '' and Field112 = '' and Field113 = '' and Field114 = '' and Field115 = '' and Field116 = '' and Field117 = '' and Field118 = '' and Field119 = '' and Field120 = '' and Field121 = '' and Field122 = '' and Field123 = '' and Field124 = '' and Field125 = '' and Field126 = '' and Field127 = '' and Field128 = '' and Field129 = '' and Field130 = '' and Field131 = '' and Field132 = '' and Field133 = '' and Field134 = '' and Field135 = '' and Field136 = '' and Field137 = '' and Field138 = '' and Field139 = '' and Field140 = '' and Field141 = '' and Field142 = '' and Field143 = '' and Field144 = '' and Field145 = '' and Field146 = '' and Field147 = '' and Field148 = '' and Field149 = '' and Field150 = '' and Field151 = '' and Field152 = '' and Field153 = '' and Field154 = '' and Field155 = '' and Field156 = '' and Field157 = '' and Field158 = '' and Field159 = '' and Field160 = '' and Field161 = '' and Field162 = '' and Field163 = '' and Field164 = '' and Field165 = '' and Field166 = '' and Field167 = '' and Field168 = '' and Field169 = '' and Field170 = '' and Field171 = '' and Field172 = '' and Field173 = '' and Field174 = '' and Field175 = '' and Field176 = '' and Field177 = '' and Field178 = '' and Field179 = '' and Field180 = '' and Field181 = '' and Field182 = '' and Field183 = '' and Field184 = '' and Field185 = '' and Field186 = '' and Field187 = '' and Field188 = '' and Field189 = '' and Field190 = '' and Field191 = '' and Field192 = '' and Field193 = '' and Field194 = '' and Field195 = '' and Field196 = '' and Field197 = '' and Field198 = '' and Field199 = '' and Field200 = '' and Field201 = '' and Field202 = '' and Field203 = '' and Field204 = '' and Field205 = '' and Field206 = '' and Field207 = '' and Field208 = '' and Field209 = '' and Field210 = '' and Field211 = '' and Field212 = '' and Field213 = '' and Field214 = '' and Field215 = '' and Field216 = '' and Field217 = '' and Field218 = '' and Field219 = '' and Field220 = '' and Field221 = '' and Field222 = '' and Field223 = '' and Field224 = '' and Field225 = '' and Field226 = '' and Field227 = '' and Field228 = '' and Field229 = '' and Field230 = '' and Field231 = '' and Field232 = '' and Field233 = '' and Field234 = '' and Field235 = '' and Field236 = '' and Field237 = '' and Field238 = '' and Field239 = '' and Field240 = '' and Field241 = '' and Field242 = '' and Field243 = '' and Field244 = '' and Field245 = '' and Field246 = '' and Field247