Hi all,
Thanks Poorvank for your work.
Some quick comments:
*phase1 design:*
- config: usually in Flink we use factories to build from configuration
rather than a visitor pattern which is quite associated with tree/graph
or any collection to visit.
- parallel writes : what is the instanciation model of RecordWriter ?
one per subtask writing incoming record one by one?
- are semaphore permits (max number of inflight requests) ?
- does the cassandra client that writes include support for custom
retry/backoff policies (e.g. exponential delay etc...)
- there is no buffering of records, so when flink framework calls write,
an async request is done on the fly with each request, right ?
*phase2 design:*
- nice to see that the cassandra writes will now be per-partition,
leveraging the high write throughput that Cassandra can do.!
- need to migrate all the existing cassandra sources and sink to flink
2.1 before starting ?
- ElementConverter is not provided by the sink but by the user of the
sink, as a mapping function from stream elements to RequestEntryT
- when group size in bytes or in nb of records is exceeded, the records
are not added to the group (batch), make sure that these records are
part of the Collection<RequestEntryT> that is snapshoted in AsyncSinkWriter
Best
Etienne
Le 04/06/2025 à 16:40, Poorvank Bhatia a écrit :
Hi all,
Following the earlier discussion on modernizing the Cassandra Sink (initial
thread<https://lists.apache.org/thread/8v7wlx23svvwzb9568fos8lkxobcty7y>),
FLIP-533
<https://cwiki.apache.org/confluence/display/FLINK/FLIP-533%3A+Cassandra+Sink+V2>
has now been created:
This FLIP proposes a two-phase redesign:
-
*Phase 1* introduces a modular Sink V2-based Cassandra sink with async
write support, type-agnostic configuration, and better integration with
Flink's threading and checkpointing model.
-
*Phase 2* builds on top of Flink 2.1 (FLIP-509), adding partition-aware
batching using AsyncSinkWriter, enabling efficient Cassandra writes
while avoiding cross-partition batch anti-patterns.
Looking forward to hearing your thoughts and suggestions.
Thanks,
Poorvank
