Jihoon Son created TAJO-1996:
--------------------------------
Summary: Too many index lookups during range shuffle
Key: TAJO-1996
URL: https://issues.apache.org/jira/browse/TAJO-1996
Project: Tajo
Issue Type: Improvement
Components: Pull Server, Data Shuffle
Reporter: Jihoon Son
Priority: Critical
Fix For: 0.12.0
During range shuffle, there are a huge number of index lookups, thereby taking
at least the half time of the total shuffle time. This is because it is
difficult to know which data is stored on which machine, so each machine needs
to ask to all machines including itself.
Unfortunately, this problem is difficult to resolved using statistics. With an
assumption of even data distribution, every machine is expected to have the
data of the whole range.
So, we need to investigate an alternative solution to reduce the frequent index
lookups.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
