[
https://issues.apache.org/jira/browse/GOBBLIN-1126?focusedWorklogId=430436&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-430436
]
ASF GitHub Bot logged work on GOBBLIN-1126:
-------------------------------------------
Author: ASF GitHub Bot
Created on: 04/May/20 21:47
Start Date: 04/May/20 21:47
Worklog Time Spent: 10m
Work Description: autumnust commented on a change in pull request #2966:
URL: https://github.com/apache/incubator-gobblin/pull/2966#discussion_r419748724
##########
File path:
gobblin-compaction/src/main/java/org/apache/gobblin/compaction/mapreduce/orc/OrcKeyDedupReducer.java
##########
@@ -37,6 +53,30 @@ protected void setOutKey(OrcValue valueToRetain) {
// do nothing since initReusableObject has assigned value for outKey.
}
+ @Override
+ protected void reduce(OrcKey key, Iterable<OrcValue> values, Context context)
+ throws IOException, InterruptedException {
+
+ /* Map from hash of value(Typed in OrcStruct) object to its times of
duplication*/
+ Map<Integer, Integer> valuesToRetain = new HashMap<>();
+ int valueHash = 0;
+
+ for (OrcValue value : values) {
+ valueHash = ((OrcStruct) value.value).hashCode();
Review comment:
Mentioned above: As long as the shuffle key is chosen properly, the cost
should not be high.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
-------------------
Worklog Id: (was: 430436)
Time Spent: 1h 20m (was: 1h 10m)
> Supporting Column Projection in OrcStruct in compaction for making shuffle
> key slimmer
> --------------------------------------------------------------------------------------
>
> Key: GOBBLIN-1126
> URL: https://issues.apache.org/jira/browse/GOBBLIN-1126
> Project: Apache Gobblin
> Issue Type: Bug
> Reporter: Lei Sun
> Priority: Major
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
