[
https://issues.apache.org/jira/browse/IMPALA-3701?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tim Armstrong resolved IMPALA-3701.
-----------------------------------
Resolution: Won't Fix
We expect these to be generally incompressible, so not worth pursuing.
> Evaluate compressing Runtime filters to save coordinator network bandwidth
> --------------------------------------------------------------------------
>
> Key: IMPALA-3701
> URL: https://issues.apache.org/jira/browse/IMPALA-3701
> Project: IMPALA
> Issue Type: New Feature
> Components: Distributed Exec
> Affects Versions: Impala 2.5.0
> Reporter: Mostafa Mokhtar
> Assignee: Henry Robinson
> Priority: Major
> Labels: runtime-filters, scalability
> Attachments: image-2016-06-08-22-55-36-966.png, query17.sql.2.out
>
>
> When running complex queries on large clusters with lots of runtime filters
> the coordinator quickly becomes network bound due to the extra incoming and
> outgoing traffic for runtime filters, once the coordinator becomes network
> bound all other fragments in the cluster are negatively affected as they get
> blocked on shuffling/broadcasting data to the coordinator node.
> This bottleneck was identified when running large scale tests on EC2 nodes
> with less than ideal network throughput.
> In attached png is aggregate network throughput across the 32 nodes in the
> cluster with the coordinator in red.
> !image-2016-06-08-22-55-36-966.png|thumbnail!
> Compression should alleviate this bottleneck but we should consider other
> solutions
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
