ajithme commented on issue #27267: [WIP][SPARK-30556][SQL] Copy sparkContext.localproperties to child thread inSubqueryExec.executionContext URL: https://github.com/apache/spark/pull/27267#issuecomment-576351616 > IMO the thread pools should not be a big issue. The subquery is guaranteed to be executed on a different thread (you can even add an assert for this). You just set some unique property on the local properties (value should also be unique), construct the something that contains a broadcast join, use an accumulator that you modify using either a UDF (easy) or a dataset operation. Agree. But with a pool size of 16, i will have to ensure all 16 threads are used at-least once and are alive to reproduce this issue and not to make that test flaky. Its lot easier if i can set the pool size to 1 in test to reproduce.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
