ankurdave commented on PR #36425: URL: https://github.com/apache/spark/pull/36425#issuecomment-1115248094
As @HyukjinKwon had [noted in October 2020](https://github.com/apache/spark/pull/30177/files#r513898999), the `ContextAwareIterator` approach didn't fully solve the problem because the root issue was that there was a race between (1) the main task thread freeing an off-heap buffer, and (2) the Python WriterThread reading from that off-heap buffer. To fully solve that, it [required adding synchronization](https://github.com/apache/spark/pull/34245) between the two threads. If this issue is caused by a race between the main thread and some other thread, then it seems like the same reasoning applies. Do you have a crash file indicating which thread is causing the JVM crash? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
