Abacn commented on PR #32723: URL: https://github.com/apache/beam/pull/32723#issuecomment-2403310482
tested `./gradlew :runners:spark:3:job-server:validatesPortableRunnerBatch --tests "*ParDoTest*TimerTests*"` added log to "setDefaultPipelineOptionsOnce" as well as inside "setDefaultPipelineOptions". Found - "setDefaultPipelineOptions" executed 14 times, that is one called from TestPipeline and 13 from https://github.com/apache/beam/blob/2ee6100980b4661a9db88d507c8b2c667f07b1d4/sdks/java/core/src/main/java/org/apache/beam/sdk/PipelineRunner.java#L47, that is each time pipeline run - "setDefaultPipelineOptionsOnce" called 256 times in total. Only one time goes to actual "setDefaultPipelineOptions" This suggests there was excessive call to "setDefaultPipelineOptions" which becomes source of Exception when there is lots of jars in classLoader (added by Spark) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
