rangareddy commented on issue #18266: URL: https://github.com/apache/hudi/issues/18266#issuecomment-4012556004
Hi @Sahil333 InProcessLockProvider works only within the same JVM process, typically the Spark driver. In a Spark application with multiple executors, each executor runs in its own JVM, so the lock is not shared across them. Because of this, concurrent writes from different executors or queries can still lead to conflicts like HoodieConflictException. For such scenarios, it is recommended to use a distributed lock provider (e.g., Zookeeper or DynamoDB) instead of InProcessLockProvider. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
