rangareddy commented on issue #9471: URL: https://github.com/apache/hudi/issues/9471#issuecomment-4841485671
`JdbcSource` incremental pull loses rows because it pages with `LIMIT source-limit` on a possibly **non-unique** incremental column and then advances the checkpoint with strict `>`, so any group of rows sharing the boundary value that straddles the limit is skipped forever (your `name=Spark`, incrColumn=1 row). Fix it by using a **unique, strictly-increasing** incremental column (making LIMIT pagination lossless), or drop `--source-limit` and bound source load with JDBC `fetch.size` + partitioned parallel reads instead — and for large high-churn MySQL tables, move to log-based CDC (`DebeziumSource`) rather than JDBC polling. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
