rangareddy commented on issue #17348:
URL: https://github.com/apache/hudi/issues/17348#issuecomment-5351236496
This issue was reviewed as part of the JIRA-migrated backlog triage
(HUDI-8770).
**Findings: confirmed, with a clear root cause in `master`.**
`hudi-utilities/src/main/java/org/apache/hudi/utilities/streamer/HoodieMultiTableStreamer.java:465`:
```java
public void sync() {
for (TableExecutionContext context : tableExecutionContexts) {
HoodieStreamer streamer = null;
try {
streamer = new HoodieStreamer(context.getConfig(), jssc,
Option.ofNullable(context.getProperties()));
streamer.sync();
successTables.add(Helpers.getTableWithDatabase(context));
streamer.shutdownGracefully();
} catch (Exception e) {
...
```
The tables are processed **sequentially**, and `HoodieStreamer.sync()`
blocks for the lifetime of the job when continuous mode is set. Continuous mode
is propagated to every table at `:246`:
```java
tableConfig.continuousMode = globalConfig.continuousMode;
```
So with `--continuous`, the loop constructs the streamer for the first
table, enters its continuous ingestion loop, and never returns. Tables 2..N are
never even constructed. That is exactly the behaviour you reported, and it is
independent of your Postgres CDC setup - any multi-table configuration will hit
it.
In one-shot mode the same loop is correct, which is why the problem only
shows up with `--continuous`.
Two possible directions, and the choice is a design decision worth settling
in the ticket:
- Run each table's streamer on its own thread or executor when continuous
mode is on, with shared failure handling, a bounded pool, and graceful shutdown
across all of them. This is what users expect but it changes the failure
semantics (currently a failing table is recorded in `failedTables` and the loop
continues).
- Or reject `--continuous` at the `HoodieMultiTableStreamer` entry point
with a clear error, so the current silent single-table behaviour stops being a
trap.
Either is better than today, where the run appears healthy while N-1 tables
are silently not ingesting.
Keeping this open.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
