prashantwason opened a new pull request, #20554:
URL: https://github.com/apache/hudi/pull/20554
### Describe the issue this Pull Request addresses
A metadata-only bootstrap fails with `Task not serializable` when optimistic
concurrency control is enabled with a lock provider that holds non-serializable
state. This was seen in production with the ZooKeeper lock provider:
```
org.apache.spark.SparkException: Task not serializable
at
org.apache.spark.util.ClosureCleaner$.ensureSerializable(ClosureCleaner.scala:444)
...
Caused by: java.io.NotSerializableException:
org.apache.hudi...HoodieInterProcessMutex
```
### Summary and Changelog
`SparkBootstrapCommitActionExecutor#runMetadataBootstrap` runs this closure
on the executors:
```java
.map(partitionFsPair -> getMetadataHandler(config, table,
...).runMetadataBootstrap(...))
```
`config` and `table` are instance fields, so the lambda captures `this`, the
executor. `BaseCommitActionExecutor` holds `txnManagerOption`, whose
`LockManager` holds the lock provider. If the lock provider is not
serializable, closure serialization fails.
- `SparkBootstrapCommitActionExecutor`: copy `config` and `table` into
locals before the closure and use the locals inside it. The other lambdas in
the metadata bootstrap path run on the driver (Java streams) or capture nothing
from the executor.
- `TestBootstrapRead#testMetadataBootstrapWithNonSerializableLockProvider`:
metadata-only bootstrap with `OPTIMISTIC_CONCURRENCY_CONTROL` and a test lock
provider (extends `InProcessLockProvider`) with an extra non-serializable field.
Verification: without the fix the new test fails with `SparkException: Task
not serializable`, and the serialization stack goes `SerializedLambda` ->
`SparkBootstrapCommitActionExecutor` -> `txnManagerOption` ->
`TransactionManager` -> `LockManager` -> `NonSerializableLockProvider` ->
`nonSerializableState`, which shows the transaction manager is built with the
lock provider. With the fix, all 4 tests in `TestBootstrapRead` pass. Spark's
`ClosureCleaner` checks serializability when the job is submitted, so this
reproduces in local mode.
### Impact
No public API or behavior change. Metadata-only bootstrap now works with
lock providers that are not serializable.
### Risk Level
Low. The change only replaces field reads with equivalent local variables in
one lambda.
### Documentation Update
None.
### Contributor's checklist
- [x] Read through [contributor's
guide](https://hudi.apache.org/contribute/how-to-contribute)
- [x] Enough context is provided in the sections above
- [x] Adequate tests were added if applicable
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
