Hyoungjun Kim created TAJO-992:
----------------------------------
Summary: Reduce number of hash shuffle output file.
Key: TAJO-992
URL: https://issues.apache.org/jira/browse/TAJO-992
Project: Tajo
Issue Type: Sub-task
Reporter: Hyoungjun Kim
Currently Tajo creates too many intermediate files in the case of hash shuffle.
A execution block(SubQuery) on a TajoWorker creates intermediate files as
following rule:
# intermediate files in a worker = # tasks / # workers * # partitions
This may cause 'too many file opens' error and makes it difficult to scale out.
To solve this problem, We should reduce number of hash shuffle output file.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
