echo567 commented on issue #7235:
URL: https://github.com/apache/kyuubi/issues/7235#issuecomment-3707825594
The thread pool contains only one thread, and the retrieved data is stored
in a queue. kyuubi retrieves data from the queue, which can shorten the
retrieval time while ensuring that the data remains unchanged.
Below are two flowcharts.
```text
┌─────────────────────────────────────────────────────────────────────────┐
│ Open Source Implementation (Serial) │
└─────────────────────────────────────────────────────────────────────────┘
Spark Driver
│
├─ [Generate RowSet #n] ────→ [Send RowSet #n]
│ │ │
│ ▼ ▼
│ HDFS Download, Decompress, ┌───────────────┐
│ Deserialize, Format Convert │ Kyuubi Server │
│ │ Receive #n ──→ Send #n ──→ ...
│ └───────────────┘
│ │
│ ▼
│ ┌───────────────┐
│ │ Hive JDBC │
│ │ Client │
│ │ Receive #n │
│ └───────────────┘
│
└─ [Generate RowSet #n+1] ────→ [Send RowSet #n+1] → ... (Serial)
│ │
▼ ▼
(Same Process) (Same Process)
Time: ────────────────────────────────────────────────────────────────────►
↑↑↑ Single RowSet Transmission Latency ↑↑↑
```
```text
┌─────────────────────────────────────────────────────────────────────────┐
│ Parallel Optimization (Prefetch Enabled) │
└─────────────────────────────────────────────────────────────────────────┘
Spark Driver
│
├── [Prefetch Thread] ────────────────────────────────────────────────┐
│ │ │
│ ├─ [Generate RowSet #n] │
│ ├─ [Generate RowSet #n+1] ←──┐ │
│ └─ [Generate RowSet #n+2] ←──┼── Overlapping Generation │
│ │ │
│ ▼ │
└─ [Main Thread] ────────────────────────────────────────────────────┐
│ │
├─ [Send RowSet #n] ────────────────→ [Receive #n] ──→ [Send #n] │
│ │ │ │ │
│ ▼ ▼ ▼ │
│ ┌───────────────┐ ┌───────────────┐
┌───────────────┐
│ │ Kyuubi Server │ │ Kyuubi Server │ │ Hive JDBC
│
│ │ Receive #n │ │ Send #n │ │ Client
│
│ └───────────────┘ └───────────────┘ │ Receive #n
│
│
└───────────────┘
│
└─ [Send RowSet #n+1] ───────────────→ [Receive #n+1] → ...
(Overlap!)
│ │
▼ ▼
(Next RowSet Sent) (Client Receives Next)
Time: ────────────────────────────────────────────────────────────────────►
↑↑↑ Reduced Single RowSet Transmission Latency ↑↑↑
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
