Yuchen Liu created SPARK-51362:
----------------------------------
Summary: change toJSON to use NextIterator API to reduce latency
Key: SPARK-51362
URL: https://issues.apache.org/jira/browse/SPARK-51362
Project: Spark
Issue Type: Improvement
Components: SQL
Affects Versions: 4.0.0
Reporter: Yuchen Liu
The current toJSON operation uses the Iterator API where iter.hasNext is called
after iter.next, which means the return of current row depends on the next row
to arrive. If we change it to use the NextIterator API, iter.next will be
called after iter.hasNext, so the current row will return immediately. This
eliminates the dependency between adjacent rows, which reduces record-level
latency.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
