[
https://issues.apache.org/jira/browse/IMPALA-4475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16804219#comment-16804219
]
Todd Lipcon commented on IMPALA-4475:
-------------------------------------
Perhaps at this point it would be better to move the appropriate
ExecQueryFInstances RPC to krpc, and then implement optional compression in
general for KRPC? Or if we end up using a sidecar to encapsulate the serialized
thrift plan (because converting it to protobuf is a ton of work) we can easily
compress just the sidecar.
> Compress ExecPlanFragment before shipping it to worker nodes to reduce
> network traffic
> --------------------------------------------------------------------------------------
>
> Key: IMPALA-4475
> URL: https://issues.apache.org/jira/browse/IMPALA-4475
> Project: IMPALA
> Issue Type: New Feature
> Components: Distributed Exec
> Affects Versions: Impala 2.6.0
> Reporter: Mostafa Mokhtar
> Assignee: Vuk Ercegovac
> Priority: Major
> Labels: ramp-up, scalability
> Attachments: count_store_returns.txt.zip,
> slow_query_start_250K_partitions_134nodes.txt
>
>
> Sending the ExecPlanFragment to remote nodes dominates the query startup time
> on clusters larger than 100 nodes, size of the ExecPlanFragment grows with
> number of tables, blocks and partitions in the table.
> On large cluster this is limits query throughput.
> From TPC-DS Q11 on 1K node cluster
> {code}
> Query Timeline: 5m6s
> - Query submitted: 75.256us (75.256us)
> - Planning finished: 1s580ms (1s580ms)
> - Submit for admission: 2s376ms (795.652ms)
> - Completed admission: 2s377ms (1.512ms)
> - Ready to start 15993 fragment instances: 2s458ms (80.378ms)
> - First dynamic filter received: 2m35s (2m33s)
> - All 15993 fragment instances started: 2m35s (40.934ms)
> - Rows available: 4m53s (2m17s)
> - First row fetched: 4m53s (176.254ms)
> - Unregister query: 4m58s (4s828ms)
> - ComputeScanRangeAssignmentTimer: 600.086ms
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
