João Boto created FLINK-40108:
---------------------------------
Summary: Add Splitters for snapshot a table or database
Key: FLINK-40108
URL: https://issues.apache.org/jira/browse/FLINK-40108
Project: Flink
Issue Type: New Feature
Components: Connectors / JDBC
Reporter: João Boto
Historically, extracting large datasets or entire databases in Apache Flink
required a fair amount of manual heavy lifting. Developers often had to write
custom SQL queries and explicitly define chunking/partitioning strategies to
prevent memory bottlenecks and ensure parallel execution.
With the introduction of {*}{{SplitterEnumerator}} (FLINK-38733){*}, this
paradigm shifts. We now have a more native, automated way to handle full-table
and full-database extractions seamlessly.
h3. The Old Way vs. The New Way
|*Feature*|*The Old Approach*|*With SplitterEnumerator*|
|*Query Definition*|Manual SQL statements with explicit {{WHERE}} clauses for
ranges.|Automated metadata-driven table/database scanning.|
|*Chunking Logic*|Custom-coded pagination or partitioning boundaries.|Built-in,
dynamic split generation handled by the enumerator.|
|*Maintenance*|High. Schema changes or data volume spikes required manual
tuning.|Low. Adapts dynamically to the underlying data structure.|
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
