HeartSaVioR commented on PR #59910: URL: https://github.com/apache/spark/pull/59910#issuecomment-6513750813
> [!NOTE] > **DISCLAIMER:** This comment was written by an LLM, Claude Opus 5.5 (`claude-opus-5-5`, 1M context), posting on behalf of @HeartSaVioR. A follow-up on the deduplication-window question, correcting part of my [earlier reply](https://github.com/apache/spark/pull/59910#issuecomment-6513271537). I said that one buffer per request keeps the buffered stream rows bounded by the request limit. That doesn't distinguish the two designs: your alternative keeps the same 1000-row bound. The reasons for keeping deduplication per request in this PR are: - **A simple join iterator.** Each batch has exactly one cursor, and its stream rows finish when that cursor ends. Splitting one buffer into several requests would mean running several cursors per batch, one after another, and finishing rows across them. - **A planned cache for repeated keys.** The task-scoped lookup cache planned in [SPARK-60693](https://issues.apache.org/jira/browse/SPARK-60693) is meant to be on by default. It answers a key from the cache before requesting it. In your example (request limit 1, 1000 rows, 10 distinct ids), the join would then send 10 requests instead of 1000. With the cache disabled, a connector with a small request limit still sends one request per batch. If that matters in practice, deduplicating across a larger buffer can be revisited later; the connector API wouldn't change. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
