Sandeep Pal created HBASE-25226:
-----------------------------------
Summary: Optimize in-memory representation for HBase map reduce
table splits for MultiTableInputFormat
Key: HBASE-25226
URL: https://issues.apache.org/jira/browse/HBASE-25226
Project: HBase
Issue Type: Improvement
Reporter: Sandeep Pal
Assignee: Sandeep Pal
It has been observed that when the table has too many regions, MR jobs consume
a lot of memory in the client. This is because we keep the region level
information in memory and the memory heavy object is TableSplit because of the
Scan object as a part of it.
There is a jira
[HBASE-24859|https://issues.apache.org/jira/projects/HBASE/issues/HBASE-24859]
which fix this single table TableInputFormat because we do not use the scan
object from TableSplit in this case.
However, it looks like we can do some optimization in case of
MultiTableInputFormat as well since each split is not required to have memory
heavy scan object.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
