ammarchalifah commented on PR #55519: URL: https://github.com/apache/spark/pull/55519#issuecomment-4592436608
> This issue got fixed in Spark 4.2 by the SPJ refactor as a side effect due to extracting the SPJ logic from the scan into a new operator. As a workaround you can try disabling the config. Got it, thank you for checking it out. > Regarding the stacktrace/issue in https://github.com/apache/iceberg/issues/15349#issuecomment-3915636021, I still think [SPARK-54439](https://issues.apache.org/jira/browse/SPARK-54439) / https://github.com/apache/spark/pull/53142 is very likely the cure. If you need the fix in Spark 3.5 then we can try to backport it. I was using 3.5 in the beginning, but just recently I upgraded to 4.0.2. In that PR I can see that the fix is supposed to be shipped in 4.0 already. I'll try to talk to AWS support if they can patch Amazon's distro with latest patch from `4.0.x`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
