[
https://issues.apache.org/jira/browse/SPARK-21720?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17870362#comment-17870362
]
Mithun Antony edited comment on SPARK-21720 at 8/2/24 5:07 AM:
---------------------------------------------------------------
[~abhiips07] I was able to overcome the issue using the below spark
configuration in Spark 2.4.7. With the below config the whole stage (of
multiple operators) will not be compiled into single java
{code:java}
--conf spark.sql.codegen.wholeStage=false{code}
was (Author: maxmithun):
[~abhiips07] I was able to overcome the issue using the below spark
configuration. With the below config the whole stage (of multiple operators)
will not be compiled into single java
{code:java}
--conf spark.sql.codegen.wholeStage=false{code}
> Filter predicate with many conditions throw stackoverflow error
> ---------------------------------------------------------------
>
> Key: SPARK-21720
> URL: https://issues.apache.org/jira/browse/SPARK-21720
> Project: Spark
> Issue Type: Sub-task
> Components: SQL
> Affects Versions: 2.2.0
> Reporter: srinivasan
> Assignee: Kazuaki Ishizaki
> Priority: Major
> Fix For: 2.2.1, 2.3.0
>
>
> When trying to filter on dataset with many predicate conditions on both spark
> sql and dataset filter transformation as described below, spark throws a
> stackoverflow exception
> Case 1: Filter Transformation on Data
> Dataset<Row> filter = sourceDataset.filter(String.format("not(%s)",
> buildQuery()));
> filter.show();
> where buildQuery() returns
> Field1 = "" and Field2 = "" and Field3 = "" and Field4 = "" and Field5 =
> "" and BLANK_5 = "" and Field7 = "" and Field8 = "" and Field9 = "" and
> Field10 = "" and Field11 = "" and Field12 = "" and Field13 = "" and
> Field14 = "" and Field15 = "" and Field16 = "" and Field17 = "" and
> Field18 = "" and Field19 = "" and Field20 = "" and Field21 = "" and
> Field22 = "" and Field23 = "" and Field24 = "" and Field25 = "" and
> Field26 = "" and Field27 = "" and Field28 = "" and Field29 = "" and
> Field30 = "" and Field31 = "" and Field32 = "" and Field33 = "" and
> Field34 = "" and Field35 = "" and Field36 = "" and Field37 = "" and
> Field38 = "" and Field39 = "" and Field40 = "" and Field41 = "" and
> Field42 = "" and Field43 = "" and Field44 = "" and Field45 = "" and
> Field46 = "" and Field47 = "" and Field48 = "" and Field49 = "" and
> Field50 = "" and Field51 = "" and Field52 = "" and Field53 = "" and
> Field54 = "" and Field55 = "" and Field56 = "" and Field57 = "" and
> Field58 = "" and Field59 = "" and Field60 = "" and Field61 = "" and
> Field62 = "" and Field63 = "" and Field64 = "" and Field65 = "" and
> Field66 = "" and Field67 = "" and Field68 = "" and Field69 = "" and
> Field70 = "" and Field71 = "" and Field72 = "" and Field73 = "" and
> Field74 = "" and Field75 = "" and Field76 = "" and Field77 = "" and
> Field78 = "" and Field79 = "" and Field80 = "" and Field81 = "" and
> Field82 = "" and Field83 = "" and Field84 = "" and Field85 = "" and
> Field86 = "" and Field87 = "" and Field88 = "" and Field89 = "" and
> Field90 = "" and Field91 = "" and Field92 = "" and Field93 = "" and
> Field94 = "" and Field95 = "" and Field96 = "" and Field97 = "" and
> Field98 = "" and Field99 = "" and Field100 = "" and Field101 = "" and
> Field102 = "" and Field103 = "" and Field104 = "" and Field105 = "" and
> Field106 = "" and Field107 = "" and Field108 = "" and Field109 = "" and
> Field110 = "" and Field111 = "" and Field112 = "" and Field113 = "" and
> Field114 = "" and Field115 = "" and Field116 = "" and Field117 = "" and
> Field118 = "" and Field119 = "" and Field120 = "" and Field121 = "" and
> Field122 = "" and Field123 = "" and Field124 = "" and Field125 = "" and
> Field126 = "" and Field127 = "" and Field128 = "" and Field129 = "" and
> Field130 = "" and Field131 = "" and Field132 = "" and Field133 = "" and
> Field134 = "" and Field135 = "" and Field136 = "" and Field137 = "" and
> Field138 = "" and Field139 = "" and Field140 = "" and Field141 = "" and
> Field142 = "" and Field143 = "" and Field144 = "" and Field145 = "" and
> Field146 = "" and Field147 = "" and Field148 = "" and Field149 = "" and
> Field150 = "" and Field151 = "" and Field152 = "" and Field153 = "" and
> Field154 = "" and Field155 = "" and Field156 = "" and Field157 = "" and
> Field158 = "" and Field159 = "" and Field160 = "" and Field161 = "" and
> Field162 = "" and Field163 = "" and Field164 = "" and Field165 = "" and
> Field166 = "" and Field167 = "" and Field168 = "" and Field169 = "" and
> Field170 = "" and Field171 = "" and Field172 = "" and Field173 = "" and
> Field174 = "" and Field175 = "" and Field176 = "" and Field177 = "" and
> Field178 = "" and Field179 = "" and Field180 = "" and Field181 = "" and
> Field182 = "" and Field183 = "" and Field184 = "" and Field185 = "" and
> Field186 = "" and Field187 = "" and Field188 = "" and Field189 = "" and
> Field190 = "" and Field191 = "" and Field192 = "" and Field193 = "" and
> Field194 = "" and Field195 = "" and Field196 = "" and Field197 = "" and
> Field198 = "" and Field199 = "" and Field200 = "" and Field201 = "" and
> Field202 = "" and Field203 = "" and Field204 = "" and Field205 = "" and
> Field206 = "" and Field207 = "" and Field208 = "" and Field209 = "" and
> Field210 = "" and Field211 = "" and Field212 = "" and Field213 = "" and
> Field214 = "" and Field215 = "" and Field216 = "" and Field217 = "" and
> Field218 = "" and Field219 = "" and Field220 = "" and Field221 = "" and
> Field222 = "" and Field223 = "" and Field224 = "" and Field225 = "" and
> Field226 = "" and Field227 = "" and Field228 = "" and Field229 = "" and
> Field230 = "" and Field231 = "" and Field232 = "" and Field233 = "" and
> Field234 = "" and Field235 = "" and Field236 = "" and Field237 = "" and
> Field238 = "" and Field239 = "" and Field240 = "" and Field241 = "" and
> Field242 = "" and Field243 = "" and Field244 = "" and Field245 = "" and
> Field246 = "" and Field247 = "" and Field248 = "" and Field249 = "" and
> Field250 = "" and Field251 = "" and Field252 = "" and Field253 = "" and
> Field254 = "" and Field255 = "" and Field256 = "" and Field257 = "" and
> Field258 = "" and Field259 = "" and Field260 = "" and Field261 = "" and
> Field262 = "" and Field263 = "" and Field264 = "" and Field265 = "" and
> Field266 = "" and Field267 = "" and Field268 = "" and Field269 = "" and
> Field270 = "" and Field271 = "" and Field272 = "" and Field273 = "" and
> Field274 = "" and Field275 = "" and Field276 = "" and Field277 = "" and
> Field278 = "" and Field279 = "" and Field280 = "" and Field281 = "" and
> Field282 = "" and Field283 = "" and Field284 = "" and Field285 = "" and
> Field286 = "" and Field287 = "" and Field288 = "" and Field289 = "" and
> Field290 = "" and Field291 = "" and Field292 = "" and Field293 = "" and
> Field294 = "" and Field295 = "" and Field296 = "" and Field297 = "" and
> Field298 = "" and Field299 = "" and Field300 = "" and Field301 = "" and
> Field302 = "" and Field303 = "" and Field304 = "" and Field305 = "" and
> Field306 = "" and Field307 = "" and Field308 = "" and Field309 = "" and
> Field310 = "" and Field311 = "" and Field312 = "" and Field313 = "" and
> Field314 = "" and Field315 = "" and Field316 = "" and Field317 = "" and
> Field318 = "" and Field319 = "" and Field320 = "" and Field321 = "" and
> Field322 = "" and Field323 = "" and Field324 = "" and Field325 = "" and
> Field326 = "" and Field327 = "" and Field328 = "" and Field329 = "" and
> Field330 = "" and Field331 = "" and Field332 = "" and Field333 = "" and
> Field334 = ""
> Case 2 : Using Spark SQL on temp view
> select * from temp where not( Field1 = '' and Field2 = '' and Field3 = ''
> and Field4 = '' and Field5 = '' and BLANK_5 = '' and Field7 = '' and
> Field8 = '' and Field9 = '' and Field10 = '' and Field11 = '' and Field12
> = '' and Field13 = '' and Field14 = '' and Field15 = '' and Field16 = ''
> and Field17 = '' and Field18 = '' and Field19 = '' and Field20 = '' and
> Field21 = '' and Field22 = '' and Field23 = '' and Field24 = '' and
> Field25 = '' and Field26 = '' and Field27 = '' and Field28 = '' and
> Field29 = '' and Field30 = '' and Field31 = '' and Field32 = '' and
> Field33 = '' and Field34 = '' and Field35 = '' and Field36 = '' and
> Field37 = '' and Field38 = '' and Field39 = '' and Field40 = '' and
> Field41 = '' and Field42 = '' and Field43 = '' and Field44 = '' and
> Field45 = '' and Field46 = '' and Field47 = '' and Field48 = '' and
> Field49 = '' and Field50 = '' and Field51 = '' and Field52 = '' and
> Field53 = '' and Field54 = '' and Field55 = '' and Field56 = '' and
> Field57 = '' and Field58 = '' and Field59 = '' and Field60 = '' and
> Field61 = '' and Field62 = '' and Field63 = '' and Field64 = '' and
> Field65 = '' and Field66 = '' and Field67 = '' and Field68 = '' and
> Field69 = '' and Field70 = '' and Field71 = '' and Field72 = '' and
> Field73 = '' and Field74 = '' and Field75 = '' and Field76 = '' and
> Field77 = '' and Field78 = '' and Field79 = '' and Field80 = '' and
> Field81 = '' and Field82 = '' and Field83 = '' and Field84 = '' and
> Field85 = '' and Field86 = '' and Field87 = '' and Field88 = '' and
> Field89 = '' and Field90 = '' and Field91 = '' and Field92 = '' and
> Field93 = '' and Field94 = '' and Field95 = '' and Field96 = '' and
> Field97 = '' and Field98 = '' and Field99 = '' and Field100 = '' and
> Field101 = '' and Field102 = '' and Field103 = '' and Field104 = '' and
> Field105 = '' and Field106 = '' and Field107 = '' and Field108 = '' and
> Field109 = '' and Field110 = '' and Field111 = '' and Field112 = '' and
> Field113 = '' and Field114 = '' and Field115 = '' and Field116 = '' and
> Field117 = '' and Field118 = '' and Field119 = '' and Field120 = '' and
> Field121 = '' and Field122 = '' and Field123 = '' and Field124 = '' and
> Field125 = '' and Field126 = '' and Field127 = '' and Field128 = '' and
> Field129 = '' and Field130 = '' and Field131 = '' and Field132 = '' and
> Field133 = '' and Field134 = '' and Field135 = '' and Field136 = '' and
> Field137 = '' and Field138 = '' and Field139 = '' and Field140 = '' and
> Field141 = '' and Field142 = '' and Field143 = '' and Field144 = '' and
> Field145 = '' and Field146 = '' and Field147 = '' and Field148 = '' and
> Field149 = '' and Field150 = '' and Field151 = '' and Field152 = '' and
> Field153 = '' and Field154 = '' and Field155 = '' and Field156 = '' and
> Field157 = '' and Field158 = '' and Field159 = '' and Field160 = '' and
> Field161 = '' and Field162 = '' and Field163 = '' and Field164 = '' and
> Field165 = '' and Field166 = '' and Field167 = '' and Field168 = '' and
> Field169 = '' and Field170 = '' and Field171 = '' and Field172 = '' and
> Field173 = '' and Field174 = '' and Field175 = '' and Field176 = '' and
> Field177 = '' and Field178 = '' and Field179 = '' and Field180 = '' and
> Field181 = '' and Field182 = '' and Field183 = '' and Field184 = '' and
> Field185 = '' and Field186 = '' and Field187 = '' and Field188 = '' and
> Field189 = '' and Field190 = '' and Field191 = '' and Field192 = '' and
> Field193 = '' and Field194 = '' and Field195 = '' and Field196 = '' and
> Field197 = '' and Field198 = '' and Field199 = '' and Field200 = '' and
> Field201 = '' and Field202 = '' and Field203 = '' and Field204 = '' and
> Field205 = '' and Field206 = '' and Field207 = '' and Field208 = '' and
> Field209 = '' and Field210 = '' and Field211 = '' and Field212 = '' and
> Field213 = '' and Field214 = '' and Field215 = '' and Field216 = '' and
> Field217 = '' and Field218 = '' and Field219 = '' and Field220 = '' and
> Field221 = '' and Field222 = '' and Field223 = '' and Field224 = '' and
> Field225 = '' and Field226 = '' and Field227 = '' and Field228 = '' and
> Field229 = '' and Field230 = '' and Field231 = '' and Field232 = '' and
> Field233 = '' and Field234 = '' and Field235 = '' and Field236 = '' and
> Field237 = '' and Field238 = '' and Field239 = '' and Field240 = '' and
> Field241 = '' and Field242 = '' and Field243 = '' and Field244 = '' and
> Field245 = '' and Field246 = '' and Field247 = '' and Field248 = '' and
> Field249 = '' and Field250 = '' and Field251 = '' and Field252 = '' and
> Field253 = '' and Field254 = '' and Field255 = '' and Field256 = '' and
> Field257 = '' and Field258 = '' and Field259 = '' and Field260 = '' and
> Field261 = '' and Field262 = '' and Field263 = '' and Field264 = '' and
> Field265 = '' and Field266 = '' and Field267 = '' and Field268 = '' and
> Field269 = '' and Field270 = '' and Field271 = '' and Field272 = '' and
> Field273 = '' and Field274 = '' and Field275 = '' and Field276 = '' and
> Field277 = '' and Field278 = '' and Field279 = '' and Field280 = '' and
> Field281 = '' and Field282 = '' and Field283 = '' and Field284 = '' and
> Field285 = '' and Field286 = '' and Field287 = '' and Field288 = '' and
> Field289 = '' and Field290 = '' and Field291 = '' and Field292 = '' and
> Field293 = '' and Field294 = '' and Field295 = '' and Field296 = '' and
> Field297 = '' and Field298 = '' and Field299 = '' and Field300 = '' and
> Field301 = '' and Field302 = '' and Field303 = '' and Field304 = '' and
> Field305 = '' and Field306 = '' and Field307 = '' and Field308 = '' and
> Field309 = '' and Field310 = '' and Field311 = '' and Field312 = '' and
> Field313 = '' and Field314 = '' and Field315 = '' and Field316 = '' and
> Field317 = '' and Field318 = '' and Field319 = '' and Field320 = '' and
> Field321 = '' and Field322 = '' and Field323 = '' and Field324 = '' and
> Field325 = '' and Field326 = '' and Field327 = '' and Field328 = '' and
> Field329 = '' and Field330 = '' and Field331 = '' and Field332 = '' and
> Field333 = '' and Field334 = '')
> in both cases, the error thrown is
> java.lang.StackOverflowError
> at
> org.codehaus.janino.CodeContext.extract16BitValue(CodeContext.java:679)
> at org.codehaus.janino.CodeContext.flowAnalysis(CodeContext.java:545)
> at org.codehaus.janino.CodeContext.flowAnalysis(CodeContext.java:541)
> at org.codehaus.janino.CodeContext.flowAnalysis(CodeContext.java:541)
> at org.codehaus.janino.CodeContext.flowAnalysis(CodeContext.java:541)
> at org.codehaus.janino.CodeContext.flowAnalysis(CodeContext.java:541)
> at org.codehaus.janino.CodeContext.flowAnalysis(CodeContext.java:541)
> at org.codehaus.janino.CodeContext.flowAnalysis(CodeContext.java:541)
> ......
> Thanks
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
