[
https://issues.apache.org/jira/browse/HBASE-28440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18047131#comment-18047131
]
Hudson commented on HBASE-28440:
--------------------------------
Results for branch branch-3
[build #503 on
builds.a.o|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-3/503/]:
(x) *{color:red}-1 overall{color}*
----
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-3/503/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk17 hadoop3 checks{color}
-- For more information [see jdk17
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-3/503/JDK17_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk17 hadoop 3.3.5 backward compatibility checks{color}
-- For more information [see jdk17
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-3/503/JDK17_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk17 hadoop 3.3.6 backward compatibility checks{color}
-- For more information [see jdk17
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-3/503/JDK17_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk17 hadoop 3.4.0 backward compatibility checks{color}
-- For more information [see jdk17
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-3/503/JDK17_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(x) {color:red}-1 jdk17 hadoop 3.4.1 backward compatibility checks{color}
-- For more information [see jdk17
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-3/503/JDK17_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test for 3.3.5 {color}
(/) {color:green}+1 client integration test for 3.3.6 {color}
(/) {color:green}+1 client integration test for 3.4.0 {color}
(/) {color:green}+1 client integration test for 3.4.1 {color}
(/) {color:green}+1 client integration test for 3.4.2 {color}
> Add support for using mapreduce sort in HFileOutputFormat2
> ----------------------------------------------------------
>
> Key: HBASE-28440
> URL: https://issues.apache.org/jira/browse/HBASE-28440
> Project: HBase
> Issue Type: Improvement
> Components: backup&restore
> Reporter: Bryan Beaudreault
> Assignee: Hernan Gelaf-Romer
> Priority: Major
> Labels: pull-request-available
> Fix For: 2.7.0, 3.0.0-beta-2, 2.6.4
>
>
> Currently HFileOutputFormat2 uses CellSortReducer, which attempts to sort all
> of the cells of a row in memory using a TreeSet. There is a warning in the
> javadoc "If lots of columns per row, it will use lots of memory sorting."
> This can be problematic for WALPlayer, which uses HFileOutputFormat2. You
> could have reasonably sized row which just gets lots of edits in the time
> period of WALs being replayed, and that would cause an OOM. We are seeing
> this in some cases with incremental backups.
> MapReduce has built-in sorting capabilities which are not limited to sorting
> in memory. It can spill to disk as necessary to sort very large datasets. We
> can get this capability in HFileOutputFormat2 with a couple changes:
> # Add support for a KeyOnlyCellComparable type as the map output key
> # When configured, use
> job.setSortComparatorClass(CellWritableComparator.class) and
> job.setReducerClass(PreSortedCellsReducer.class)
> # Update WALPlayer to have a mode which can output this new comparable
> instead of ImmutableBytesWritable
> CellWritableComparator exists already for the Import job, so there is some
> prior art.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
