laglangyue commented on PR #4128: URL: https://github.com/apache/incubator-seatunnel/pull/4128#issuecomment-1500916858
I studied the relevant implementation of Spark today and found that the checkpoint should be set during context initialization using getOrCreateContext(), so I think this setting is incorrect. It is reasonable to obtain the checkpoint path through SparkConf, which can be set at both task level and instance level through Spark lib. Directly setting /tmp will result in no read and write permissions, especially in Spark on Yarn mode, where there are basically no permissions. I will validate my ideas next week -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
