[
https://issues.apache.org/jira/browse/FLINK-40108?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
João Boto updated FLINK-40108:
------------------------------
Description:
Historically, extracting large datasets or entire databases in Apache Flink
required a fair amount of manual heavy lifting. Developers often had to write
custom SQL queries and explicitly define chunking/partitioning strategies to
prevent memory bottlenecks and ensure parallel execution.
With the introduction of {*}{{SplitterEnumerator}} (FLINK-38733){*}, this
paradigm shifts. We now have a more native, automated way to handle full-table
and full-database extractions seamlessly.
h3. The Old Way vs. The New Way
|*Feature*|*The Old Approach*|*With SplitterEnumerator*|
|*Query Definition*|Manual SQL statements with explicit {{WHERE}} clauses for
ranges.|Automated metadata-driven table/database scanning.|
|*Chunking Logic*|Custom-coded pagination or partitioning boundaries.|Built-in,
dynamic split generation handled by the enumerator.|
|*Maintenance*|High. Schema changes or data volume spikes required manual
tuning.|Low. Adapts dynamically to the underlying data structure.|
h3. Key Benefits
* *Zero-Configuration Chunking:* You no longer need to guess the optimal chunk
sizes or write complex boundary logic. The {{SplitterEnumerator}} automatically
determines how to divide the table into manageable "splits" for parallel
workers.
* *Database-Wide Scaling:* Instead of writing dozens of individual source
queries for a multi-table database, the enumerator can discover and orchestrate
the extraction of all tables under the hood.
* *Cleaner Codebases:* Removes hundreds of lines of boilerplate SQL and
partitioning code, making your Flink jobs significantly easier to read,
maintain, and audit.
h3. How it Works (Under the Hood)
Instead of the coordinator node just passing a static query to the source
workers, the {{SplitterEnumerator}} acts as an intelligent traffic controller:
# *Discovery:* It inspects the target database or table metadata.
# *Splitting:* It breaks the dataset down into independent, parallelizable
pieces (splits) based on primary keys, indices, or data volume.
# *Distribution:* It dynamically assigns these splits to the source readers as
they become available, ensuring an even workload distribution without manual
intervention.
was:
Historically, extracting large datasets or entire databases in Apache Flink
required a fair amount of manual heavy lifting. Developers often had to write
custom SQL queries and explicitly define chunking/partitioning strategies to
prevent memory bottlenecks and ensure parallel execution.
With the introduction of {*}{{SplitterEnumerator}} (FLINK-38733){*}, this
paradigm shifts. We now have a more native, automated way to handle full-table
and full-database extractions seamlessly.
h3. The Old Way vs. The New Way
|*Feature*|*The Old Approach*|*With SplitterEnumerator*|
|*Query Definition*|Manual SQL statements with explicit {{WHERE}} clauses for
ranges.|Automated metadata-driven table/database scanning.|
|*Chunking Logic*|Custom-coded pagination or partitioning boundaries.|Built-in,
dynamic split generation handled by the enumerator.|
|*Maintenance*|High. Schema changes or data volume spikes required manual
tuning.|Low. Adapts dynamically to the underlying data structure.|
> Add Splitters for snapshot a table or database
> ----------------------------------------------
>
> Key: FLINK-40108
> URL: https://issues.apache.org/jira/browse/FLINK-40108
> Project: Flink
> Issue Type: New Feature
> Components: Connectors / JDBC
> Reporter: João Boto
> Priority: Major
>
> Historically, extracting large datasets or entire databases in Apache Flink
> required a fair amount of manual heavy lifting. Developers often had to write
> custom SQL queries and explicitly define chunking/partitioning strategies to
> prevent memory bottlenecks and ensure parallel execution.
> With the introduction of {*}{{SplitterEnumerator}} (FLINK-38733){*}, this
> paradigm shifts. We now have a more native, automated way to handle
> full-table and full-database extractions seamlessly.
> h3. The Old Way vs. The New Way
> |*Feature*|*The Old Approach*|*With SplitterEnumerator*|
> |*Query Definition*|Manual SQL statements with explicit {{WHERE}} clauses for
> ranges.|Automated metadata-driven table/database scanning.|
> |*Chunking Logic*|Custom-coded pagination or partitioning
> boundaries.|Built-in, dynamic split generation handled by the enumerator.|
> |*Maintenance*|High. Schema changes or data volume spikes required manual
> tuning.|Low. Adapts dynamically to the underlying data structure.|
> h3. Key Benefits
> * *Zero-Configuration Chunking:* You no longer need to guess the optimal
> chunk sizes or write complex boundary logic. The {{SplitterEnumerator}}
> automatically determines how to divide the table into manageable "splits" for
> parallel workers.
> * *Database-Wide Scaling:* Instead of writing dozens of individual source
> queries for a multi-table database, the enumerator can discover and
> orchestrate the extraction of all tables under the hood.
> * *Cleaner Codebases:* Removes hundreds of lines of boilerplate SQL and
> partitioning code, making your Flink jobs significantly easier to read,
> maintain, and audit.
> h3. How it Works (Under the Hood)
> Instead of the coordinator node just passing a static query to the source
> workers, the {{SplitterEnumerator}} acts as an intelligent traffic controller:
> # *Discovery:* It inspects the target database or table metadata.
> # *Splitting:* It breaks the dataset down into independent, parallelizable
> pieces (splits) based on primary keys, indices, or data volume.
> # *Distribution:* It dynamically assigns these splits to the source readers
> as they become available, ensuring an even workload distribution without
> manual intervention.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
