[ 
https://issues.apache.org/jira/browse/SPARK-19372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16123029#comment-16123029
 ] 

srinivasan edited comment on SPARK-19372 at 8/11/17 8:28 AM:
-------------------------------------------------------------

Hi [~kiszk], the fix does not work in 2.2.0 for 

select * from temp where not( Field1 = '' and  Field2 = '' and  Field3 = '' and 
 Field4 = '' and  Field5 = '' and  BLANK_5 = '' and  Field7 = '' and  Field8 = 
'' and  Field9 = '' and  Field10 = '' and  Field11 = '' and  Field12 = '' and  
Field13 = '' and  Field14 = '' and  Field15 = '' and  Field16 = '' and  Field17 
= '' and  Field18 = '' and  Field19 = '' and  Field20 = '' and  Field21 = '' 
and  Field22 = '' and  Field23 = '' and  Field24 = '' and  Field25 = '' and  
Field26 = '' and  Field27 = '' and  Field28 = '' and  Field29 = '' and  Field30 
= '' and  Field31 = '' and  Field32 = '' and  Field33 = '' and  Field34 = '' 
and  Field35 = '' and  Field36 = '' and  Field37 = '' and  Field38 = '' and  
Field39 = '' and  Field40 = '' and  Field41 = '' and  Field42 = '' and  Field43 
= '' and  Field44 = '' and  Field45 = '' and  Field46 = '' and  Field47 = '' 
and  Field48 = '' and  Field49 = '' and  Field50 = '' and  Field51 = '' and  
Field52 = '' and  Field53 = '' and  Field54 = '' and  Field55 = '' and  Field56 
= '' and  Field57 = '' and  Field58 = '' and  Field59 = '' and  Field60 = '' 
and  Field61 = '' and  Field62 = '' and  Field63 = '' and  Field64 = '' and  
Field65 = '' and  Field66 = '' and  Field67 = '' and  Field68 = '' and  Field69 
= '' and  Field70 = '' and  Field71 = '' and  Field72 = '' and  Field73 = '' 
and  Field74 = '' and  Field75 = '' and  Field76 = '' and  Field77 = '' and  
Field78 = '' and  Field79 = '' and  Field80 = '' and  Field81 = '' and  Field82 
= '' and  Field83 = '' and  Field84 = '' and  Field85 = '' and  Field86 = '' 
and  Field87 = '' and  Field88 = '' and  Field89 = '' and  Field90 = '' and  
Field91 = '' and  Field92 = '' and  Field93 = '' and  Field94 = '' and  Field95 
= '' and  Field96 = '' and  Field97 = '' and  Field98 = '' and  Field99 = '' 
and  Field100 = '' and  Field101 = '' and  Field102 = '' and  Field103 = '' and 
 Field104 = '' and  Field105 = '' and  Field106 = '' and  Field107 = '' and  
Field108 = '' and  Field109 = '' and  Field110 = '' and  Field111 = '' and  
Field112 = '' and  Field113 = '' and  Field114 = '' and  Field115 = '' and  
Field116 = '' and  Field117 = '' and  Field118 = '' and  Field119 = '' and  
Field120 = '' and  Field121 = '' and  Field122 = '' and  Field123 = '' and  
Field124 = '' and  Field125 = '' and  Field126 = '' and  Field127 = '' and  
Field128 = '' and  Field129 = '' and  Field130 = '' and  Field131 = '' and  
Field132 = '' and  Field133 = '' and  Field134 = '' and  Field135 = '' and  
Field136 = '' and  Field137 = '' and  Field138 = '' and  Field139 = '' and  
Field140 = '' and  Field141 = '' and  Field142 = '' and  Field143 = '' and  
Field144 = '' and  Field145 = '' and  Field146 = '' and  Field147 = '' and  
Field148 = '' and  Field149 = '' and  Field150 = '' and  Field151 = '' and  
Field152 = '' and  Field153 = '' and  Field154 = '' and  Field155 = '' and  
Field156 = '' and  Field157 = '' and  Field158 = '' and  Field159 = '' and  
Field160 = '' and  Field161 = '' and  Field162 = '' and  Field163 = '' and  
Field164 = '' and  Field165 = '' and  Field166 = '' and  Field167 = '' and  
Field168 = '' and  Field169 = '' and  Field170 = '' and  Field171 = '' and  
Field172 = '' and  Field173 = '' and  Field174 = '' and  Field175 = '' and  
Field176 = '' and  Field177 = '' and  Field178 = '' and  Field179 = '' and  
Field180 = '' and  Field181 = '' and  Field182 = '' and  Field183 = '' and  
Field184 = '' and  Field185 = '' and  Field186 = '' and  Field187 = '' and  
Field188 = '' and  Field189 = '' and  Field190 = '' and  Field191 = '' and  
Field192 = '' and  Field193 = '' and  Field194 = '' and  Field195 = '' and  
Field196 = '' and  Field197 = '' and  Field198 = '' and  Field199 = '' and  
Field200 = '' and  Field201 = '' and  Field202 = '' and  Field203 = '' and  
Field204 = '' and  Field205 = '' and  Field206 = '' and  Field207 = '' and  
Field208 = '' and  Field209 = '' and  Field210 = '' and  Field211 = '' and  
Field212 = '' and  Field213 = '' and  Field214 = '' and  Field215 = '' and  
Field216 = '' and  Field217 = '' and  Field218 = '' and  Field219 = '' and  
Field220 = '' and  Field221 = '' and  Field222 = '' and  Field223 = '' and  
Field224 = '' and  Field225 = '' and  Field226 = '' and  Field227 = '' and  
Field228 = '' and  Field229 = '' and  Field230 = '' and  Field231 = '' and  
Field232 = '' and  Field233 = '' and  Field234 = '' and  Field235 = '' and  
Field236 = '' and  Field237 = '' and  Field238 = '' and  Field239 = '' and  
Field240 = '' and  Field241 = '' and  Field242 = '' and  Field243 = '' and  
Field244 = '' and  Field245 = '' and  Field246 = '' and  Field247 = '' and  
Field248 = '' and  Field249 = '' and  Field250 = '' and  Field251 = '' and  
Field252 = '' and  Field253 = '' and  Field254 = '' and  Field255 = '' and  
Field256 = '' and  Field257 = '' and  Field258 = '' and  Field259 = '' and  
Field260 = '' and  Field261 = '' and  Field262 = '' and  Field263 = '' and  
Field264 = '' and  Field265 = '' and  Field266 = '' and  Field267 = '' and  
Field268 = '' and  Field269 = '' and  Field270 = '' and  Field271 = '' and  
Field272 = '' and  Field273 = '' and  Field274 = '' and  Field275 = '' and  
Field276 = '' and  Field277 = '' and  Field278 = '' and  Field279 = '' and  
Field280 = '' and  Field281 = '' and  Field282 = '' and  Field283 = '' and  
Field284 = '' and  Field285 = '' and  Field286 = '' and  Field287 = '' and  
Field288 = '' and  Field289 = '' and  Field290 = '' and  Field291 = '' and  
Field292 = '' and  Field293 = '' and  Field294 = '' and  Field295 = '' and  
Field296 = '' and  Field297 = '' and  Field298 = '' and  Field299 = '' and  
Field300 = '' and  Field301 = '' and  Field302 = '' and  Field303 = '' and  
Field304 = '' and  Field305 = '' and  Field306 = '' and  Field307 = '' and  
Field308 = '' and  Field309 = '' and  Field310 = '' and  Field311 = '' and  
Field312 = '' and  Field313 = '' and  Field314 = '' and  Field315 = '' and  
Field316 = '' and  Field317 = '' and  Field318 = '' and  Field319 = '' and  
Field320 = '' and  Field321 = '' and  Field322 = '' and  Field323 = '' and  
Field324 = '' and  Field325 = '' and  Field326 = '' and  Field327 = '' and  
Field328 = '' and  Field329 = '' and  Field330 = '' and  Field331 = '' and  
Field332 = '' and  Field333 = '' and  Field334 = '')


Any advice? Thanks


was (Author: srinivasanm):
Hi [~kiszk], the fix does not work for 
select * from temp where not( Field1 = '' and  Field2 = '' and  Field3 = '' and 
 Field4 = '' and  Field5 = '' and  BLANK_5 = '' and  Field7 = '' and  Field8 = 
'' and  Field9 = '' and  Field10 = '' and  Field11 = '' and  Field12 = '' and  
Field13 = '' and  Field14 = '' and  Field15 = '' and  Field16 = '' and  Field17 
= '' and  Field18 = '' and  Field19 = '' and  Field20 = '' and  Field21 = '' 
and  Field22 = '' and  Field23 = '' and  Field24 = '' and  Field25 = '' and  
Field26 = '' and  Field27 = '' and  Field28 = '' and  Field29 = '' and  Field30 
= '' and  Field31 = '' and  Field32 = '' and  Field33 = '' and  Field34 = '' 
and  Field35 = '' and  Field36 = '' and  Field37 = '' and  Field38 = '' and  
Field39 = '' and  Field40 = '' and  Field41 = '' and  Field42 = '' and  Field43 
= '' and  Field44 = '' and  Field45 = '' and  Field46 = '' and  Field47 = '' 
and  Field48 = '' and  Field49 = '' and  Field50 = '' and  Field51 = '' and  
Field52 = '' and  Field53 = '' and  Field54 = '' and  Field55 = '' and  Field56 
= '' and  Field57 = '' and  Field58 = '' and  Field59 = '' and  Field60 = '' 
and  Field61 = '' and  Field62 = '' and  Field63 = '' and  Field64 = '' and  
Field65 = '' and  Field66 = '' and  Field67 = '' and  Field68 = '' and  Field69 
= '' and  Field70 = '' and  Field71 = '' and  Field72 = '' and  Field73 = '' 
and  Field74 = '' and  Field75 = '' and  Field76 = '' and  Field77 = '' and  
Field78 = '' and  Field79 = '' and  Field80 = '' and  Field81 = '' and  Field82 
= '' and  Field83 = '' and  Field84 = '' and  Field85 = '' and  Field86 = '' 
and  Field87 = '' and  Field88 = '' and  Field89 = '' and  Field90 = '' and  
Field91 = '' and  Field92 = '' and  Field93 = '' and  Field94 = '' and  Field95 
= '' and  Field96 = '' and  Field97 = '' and  Field98 = '' and  Field99 = '' 
and  Field100 = '' and  Field101 = '' and  Field102 = '' and  Field103 = '' and 
 Field104 = '' and  Field105 = '' and  Field106 = '' and  Field107 = '' and  
Field108 = '' and  Field109 = '' and  Field110 = '' and  Field111 = '' and  
Field112 = '' and  Field113 = '' and  Field114 = '' and  Field115 = '' and  
Field116 = '' and  Field117 = '' and  Field118 = '' and  Field119 = '' and  
Field120 = '' and  Field121 = '' and  Field122 = '' and  Field123 = '' and  
Field124 = '' and  Field125 = '' and  Field126 = '' and  Field127 = '' and  
Field128 = '' and  Field129 = '' and  Field130 = '' and  Field131 = '' and  
Field132 = '' and  Field133 = '' and  Field134 = '' and  Field135 = '' and  
Field136 = '' and  Field137 = '' and  Field138 = '' and  Field139 = '' and  
Field140 = '' and  Field141 = '' and  Field142 = '' and  Field143 = '' and  
Field144 = '' and  Field145 = '' and  Field146 = '' and  Field147 = '' and  
Field148 = '' and  Field149 = '' and  Field150 = '' and  Field151 = '' and  
Field152 = '' and  Field153 = '' and  Field154 = '' and  Field155 = '' and  
Field156 = '' and  Field157 = '' and  Field158 = '' and  Field159 = '' and  
Field160 = '' and  Field161 = '' and  Field162 = '' and  Field163 = '' and  
Field164 = '' and  Field165 = '' and  Field166 = '' and  Field167 = '' and  
Field168 = '' and  Field169 = '' and  Field170 = '' and  Field171 = '' and  
Field172 = '' and  Field173 = '' and  Field174 = '' and  Field175 = '' and  
Field176 = '' and  Field177 = '' and  Field178 = '' and  Field179 = '' and  
Field180 = '' and  Field181 = '' and  Field182 = '' and  Field183 = '' and  
Field184 = '' and  Field185 = '' and  Field186 = '' and  Field187 = '' and  
Field188 = '' and  Field189 = '' and  Field190 = '' and  Field191 = '' and  
Field192 = '' and  Field193 = '' and  Field194 = '' and  Field195 = '' and  
Field196 = '' and  Field197 = '' and  Field198 = '' and  Field199 = '' and  
Field200 = '' and  Field201 = '' and  Field202 = '' and  Field203 = '' and  
Field204 = '' and  Field205 = '' and  Field206 = '' and  Field207 = '' and  
Field208 = '' and  Field209 = '' and  Field210 = '' and  Field211 = '' and  
Field212 = '' and  Field213 = '' and  Field214 = '' and  Field215 = '' and  
Field216 = '' and  Field217 = '' and  Field218 = '' and  Field219 = '' and  
Field220 = '' and  Field221 = '' and  Field222 = '' and  Field223 = '' and  
Field224 = '' and  Field225 = '' and  Field226 = '' and  Field227 = '' and  
Field228 = '' and  Field229 = '' and  Field230 = '' and  Field231 = '' and  
Field232 = '' and  Field233 = '' and  Field234 = '' and  Field235 = '' and  
Field236 = '' and  Field237 = '' and  Field238 = '' and  Field239 = '' and  
Field240 = '' and  Field241 = '' and  Field242 = '' and  Field243 = '' and  
Field244 = '' and  Field245 = '' and  Field246 = '' and  Field247 = '' and  
Field248 = '' and  Field249 = '' and  Field250 = '' and  Field251 = '' and  
Field252 = '' and  Field253 = '' and  Field254 = '' and  Field255 = '' and  
Field256 = '' and  Field257 = '' and  Field258 = '' and  Field259 = '' and  
Field260 = '' and  Field261 = '' and  Field262 = '' and  Field263 = '' and  
Field264 = '' and  Field265 = '' and  Field266 = '' and  Field267 = '' and  
Field268 = '' and  Field269 = '' and  Field270 = '' and  Field271 = '' and  
Field272 = '' and  Field273 = '' and  Field274 = '' and  Field275 = '' and  
Field276 = '' and  Field277 = '' and  Field278 = '' and  Field279 = '' and  
Field280 = '' and  Field281 = '' and  Field282 = '' and  Field283 = '' and  
Field284 = '' and  Field285 = '' and  Field286 = '' and  Field287 = '' and  
Field288 = '' and  Field289 = '' and  Field290 = '' and  Field291 = '' and  
Field292 = '' and  Field293 = '' and  Field294 = '' and  Field295 = '' and  
Field296 = '' and  Field297 = '' and  Field298 = '' and  Field299 = '' and  
Field300 = '' and  Field301 = '' and  Field302 = '' and  Field303 = '' and  
Field304 = '' and  Field305 = '' and  Field306 = '' and  Field307 = '' and  
Field308 = '' and  Field309 = '' and  Field310 = '' and  Field311 = '' and  
Field312 = '' and  Field313 = '' and  Field314 = '' and  Field315 = '' and  
Field316 = '' and  Field317 = '' and  Field318 = '' and  Field319 = '' and  
Field320 = '' and  Field321 = '' and  Field322 = '' and  Field323 = '' and  
Field324 = '' and  Field325 = '' and  Field326 = '' and  Field327 = '' and  
Field328 = '' and  Field329 = '' and  Field330 = '' and  Field331 = '' and  
Field332 = '' and  Field333 = '' and  Field334 = '')


Any advice? Thanks

> Code generation for Filter predicate including many OR conditions exceeds JVM 
> method size limit 
> ------------------------------------------------------------------------------------------------
>
>                 Key: SPARK-19372
>                 URL: https://issues.apache.org/jira/browse/SPARK-19372
>             Project: Spark
>          Issue Type: Bug
>    Affects Versions: 2.1.0
>            Reporter: Jay Pranavamurthi
>            Assignee: Kazuaki Ishizaki
>             Fix For: 2.2.0, 2.3.0
>
>         Attachments: wide400cols.csv
>
>
> For the attached csv file, the code below causes the exception 
> "org.codehaus.janino.JaninoRuntimeException: Code of method 
> "(Lorg/apache/spark/sql/catalyst/InternalRow;)Z" of class 
> "org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificPredicate" 
> grows beyond 64 KB
> Code:
> {code:borderStyle=solid}
>   val conf = new SparkConf().setMaster("local[1]")
>   val sqlContext = 
> SparkSession.builder().config(conf).getOrCreate().sqlContext
>   val dataframe =
>     sqlContext
>       .read
>       .format("com.databricks.spark.csv")
>       .load("wide400cols.csv")
>   val filter = (0 to 399)
>     .foldLeft(lit(false))((e, index) => 
> e.or(dataframe.col(dataframe.columns(index)) =!= s"column${index+1}"))
>   val filtered = dataframe.filter(filter)
>   filtered.show(100)
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

Reply via email to