[
https://issues.apache.org/jira/browse/FLINK-40108?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated FLINK-40108:
-----------------------------------
Labels: pull-request-available (was: )
> Add Splitters for snapshot a table or database
> ----------------------------------------------
>
> Key: FLINK-40108
> URL: https://issues.apache.org/jira/browse/FLINK-40108
> Project: Flink
> Issue Type: New Feature
> Components: Connectors / JDBC
> Reporter: João Boto
> Priority: Major
> Labels: pull-request-available
>
> Historically, extracting large datasets or entire databases in Apache Flink
> required a fair amount of manual heavy lifting. Developers often had to write
> custom SQL queries and explicitly define chunking/partitioning strategies to
> prevent memory bottlenecks and ensure parallel execution.
> With the introduction of {*}{{SplitterEnumerator}} (FLINK-38733){*}, this
> paradigm shifts. We now have a more native, automated way to handle
> full-table and full-database extractions seamlessly.
> h3. The Old Way vs. The New Way
> |*Feature*|*The Old Approach*|*With SplitterEnumerator*|
> |*Query Definition*|Manual SQL statements with explicit {{WHERE}} clauses for
> ranges.|Automated metadata-driven table/database scanning.|
> |*Chunking Logic*|Custom-coded pagination or partitioning
> boundaries.|Built-in, dynamic split generation handled by the enumerator.|
> |*Maintenance*|High. Schema changes or data volume spikes required manual
> tuning.|Low. Adapts dynamically to the underlying data structure.|
> h3. Key Benefits
> * *Zero-Configuration Chunking:* You no longer need to guess the optimal
> chunk sizes or write complex boundary logic. The {{SplitterEnumerator}}
> automatically determines how to divide the table into manageable "splits" for
> parallel workers.
> * *Database-Wide Scaling:* Instead of writing dozens of individual source
> queries for a multi-table database, the enumerator can discover and
> orchestrate the extraction of all tables under the hood.
> * *Cleaner Codebases:* Removes hundreds of lines of boilerplate SQL and
> partitioning code, making your Flink jobs significantly easier to read,
> maintain, and audit.
> h3. How it Works (Under the Hood)
> Instead of the coordinator node just passing a static query to the source
> workers, the {{SplitterEnumerator}} acts as an intelligent traffic controller:
> # *Discovery:* It inspects the target database or table metadata.
> # *Splitting:* It breaks the dataset down into independent, parallelizable
> pieces (splits) based on primary keys, indices, or data volume.
> # *Distribution:* It dynamically assigns these splits to the source readers
> as they become available, ensuring an even workload distribution without
> manual intervention.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
