Spenserrrr commented on PR #59976:
URL: https://github.com/apache/spark/pull/59976#issuecomment-6666783816
@Yicong-Huang Thanks for the review! I updated the init-response handling,
and right now:
- In default mode, the reader sends the buffered setup command and reads the
init response without producing row input. Scala records the timestamp before
normal row production resumes.
- In pipelined mode, setup reads and caches the init response and timestamp
before returning the output iterator. This excludes delays before the
consumer’s first pull; the writer thread still runs independently.
I also checked the TCP/ACK concern and didn’t observe the stall with the new
init reader in the tested Spark workloads, so I kept the flush without changing
socket options.
Hi @viirya! The current implementation changes pipelined startup to wait for
the init response during stream setup. Could you take a quick look when you
have time, especially at startup ordering and cancellation/cleanup?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
