mmmmxa opened a new issue, #3839: URL: https://github.com/apache/iggy/issues/3839
### Description The `delta_sink` connector serializes all writes through a single shared lock for which all partitions compete. `DeltaSink` holds one `Mutex<Option<SinkState>>` wrapping the Delta table handle, writer, and coercion tree. `consume()` acquires that lock and holds it across the actual write-to-object-store and commit-to-Delta-log calls (`writer.write()` and `writer.flush_and_commit()`). This limits the throughput of the sink, especially if the object store responds slowly. Relevant code: `core/connectors/sinks/delta_sink/src/sink.rs::consume` (see the `TODO` comment above the lock acquisition). Ref discussion: https://github.com/apache/iggy/pull/2889/#discussion_r2936719763 Another TODO in this algorithm is the implementation of retries for the parquet buffer writes. Since they are technically moving into a different place with this code, the retries have to be implemented here too. ### Affected area / component Connectors ### Proposed solution _This is still very much TBD, I am currently researching what could be the best solution. Once I am done with research, I will write my idea down as final and call up the contributors for a discussion._ Introduce a shared map keyed by partition ID where each partition's `consume()` call writes into its own buffer independently and concurrently. A separate piece of coordination logic could then periodically (time-based and/or write-count-based trigger — exact policy still TBD) combine buffers and flush/commit them to the Delta table. `consume()` would still only return once its partition's data has actually been durably committed — buffering into the shared map does not mean early-return/fire-and-forget; the caller blocks until its contribution is confirmed flushed, to preserve delivery guarantees. ### Alternatives considered Currently researching alternatives. ### Contribution - [x] I'm willing to submit a pull request to implement this feature ### Good first issue - [ ] I think this could be a good first issue for a new contributor -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
