cshuo commented on issue #12738: URL: https://github.com/apache/hudi/issues/12738#issuecomment-2686598529
> [@cshuo](https://github.com/cshuo) Since MDT is the default in 1.0 hudi and it requires the lock. The File system lock is not recommended for Production and S3. Also `org.apache.hudi.client.transaction.lock.InProcessLockProvider` is not working as expected so the remaining option is `zookeeper` and `dynamoDb`. > > > Another approach I could try is to completely remove it(MDT-related config), as you did(keep default), and then do the testing. > > Yes, it requires a lock if not provide any MDT config. > > ``` > Caused by: org.apache.hudi.exception.HoodieLockException: Unsupported scheme: s3a, since this fs can not support atomic creation > at org.apache.hudi.client.transaction.lock.FileSystemBasedLockProvider.<init>(FileSystemBasedLockProvider.java:89) ~[blob_p-eb02881ebbe30d32963c608fce24f92c3ebe2ecd-ab4882c7a4628598f7d5e2561bd74c02:?] > ``` > > Will keep you posted. @maheshguptags thks for updating, as discussed in the call, there are directions you can try: * Validate whether the source can be set to the last success ckp properly, i.e., the source should emit records after failover(maybe you can substitute the kafka source with datagen source, and check whether the problem still exits); * If the problem still exits, you can try reproduce the problem with a smaller flink parallelism, and provide all the taskmanager.log and jobmanager.log files here. (the whole file, including logs before and after ckp failover). > Could you try deleting the Task Manager while checkpointing is ongoing? btw, I tried this, and unfortunately, the problem can not be reproduced either.  -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
