[
https://issues.apache.org/jira/browse/GOBBLIN-1190?focusedWorklogId=445240&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-445240
]
ASF GitHub Bot logged work on GOBBLIN-1190:
-------------------------------------------
Author: ASF GitHub Bot
Created on: 13/Jun/20 00:53
Start Date: 13/Jun/20 00:53
Worklog Time Spent: 10m
Work Description: sv2000 commented on a change in pull request #3038:
URL: https://github.com/apache/incubator-gobblin/pull/3038#discussion_r439692012
##########
File path:
gobblin-compaction/src/main/java/org/apache/gobblin/compaction/mapreduce/orc/OrcUtils.java
##########
@@ -482,4 +483,32 @@ public static WritableComparable
createValueRecursively(TypeDescription schema,
public static WritableComparable createValueRecursively(TypeDescription
schema) {
return createValueRecursively(schema, 1);
}
+
+ /**
+ * Check recursively if owning schema is eligible to be up-converted to
targetSchema:
+ * 1. TargetSchema is a subset of originalSchema.
+ * 2. TargetSchema is larger than originalSchema.
+ */
+ public static boolean eligibleForUpConvert(TypeDescription originalSchema,
TypeDescription targetSchema) {
+ if (!targetSchema.getCategory().isPrimitive()) {
+ if (originalSchema.getFieldNames().size() <
targetSchema.getFieldNames().size()) {
+ return true;
Review comment:
Why is targetSchema having more fields than original schema a sufficient
condition for up conversion?
##########
File path:
gobblin-compaction/src/main/java/org/apache/gobblin/compaction/mapreduce/orc/OrcUtils.java
##########
@@ -482,4 +483,32 @@ public static WritableComparable
createValueRecursively(TypeDescription schema,
public static WritableComparable createValueRecursively(TypeDescription
schema) {
return createValueRecursively(schema, 1);
}
+
+ /**
+ * Check recursively if owning schema is eligible to be up-converted to
targetSchema:
+ * 1. TargetSchema is a subset of originalSchema.
+ * 2. TargetSchema is larger than originalSchema.
+ */
+ public static boolean eligibleForUpConvert(TypeDescription originalSchema,
TypeDescription targetSchema) {
+ if (!targetSchema.getCategory().isPrimitive()) {
+ if (originalSchema.getFieldNames().size() <
targetSchema.getFieldNames().size()) {
+ return true;
Review comment:
Shouldn't we be checking if each field in the target Schema is present
in the record schema?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
-------------------
Worklog Id: (was: 445240)
Time Spent: 40m (was: 0.5h)
> Fallback to full schema as shuffle key is configured one is not available in
> ORC compaction
> -------------------------------------------------------------------------------------------
>
> Key: GOBBLIN-1190
> URL: https://issues.apache.org/jira/browse/GOBBLIN-1190
> Project: Apache Gobblin
> Issue Type: Improvement
> Reporter: Lei Sun
> Priority: Major
> Time Spent: 40m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
