[
https://issues.apache.org/jira/browse/GOBBLIN-1726?focusedWorklogId=818117&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-818117
]
ASF GitHub Bot logged work on GOBBLIN-1726:
-------------------------------------------
Author: ASF GitHub Bot
Created on: 18/Oct/22 19:07
Start Date: 18/Oct/22 19:07
Worklog Time Spent: 10m
Work Description: abhisheknath2011 commented on code in PR #3581:
URL: https://github.com/apache/gobblin/pull/3581#discussion_r998601818
##########
gobblin-core-base/src/main/java/org/apache/gobblin/converter/filter/AvroSchemaFieldRemover.java:
##########
@@ -119,14 +121,17 @@ private Schema removeFieldsFromRecords(Schema schema,
Map<String, Schema> schema
if (!this.shouldRemove(field)) {
Field newField;
if (this.children.containsKey(field.name())) {
- newField = new Field(field.name(),
this.children.get(field.name()).removeFields(field.schema(), schemaMap),
- field.doc(), field.defaultValue());
+ newField = AvroCompatibilityHelper.createSchemaField(field.name(),
+ this.children.get(field.name()).removeFields(field.schema(),
schemaMap),
+ field.doc(), AvroUtils.getCompatibleDefaultValue(field));
} else {
- newField = new Field(field.name(),
DO_NOTHING_INSTANCE.removeFields(field.schema(), schemaMap), field.doc(),
- field.defaultValue());
+ newField = AvroCompatibilityHelper.createSchemaField(field.name(),
+ DO_NOTHING_INSTANCE.removeFields(field.schema(), schemaMap),
field.doc(),
+ AvroUtils.getCompatibleDefaultValue(field));
}
- for (Map.Entry<String, JsonNode> stringJsonNodeEntry :
field.getJsonProps().entrySet()) {
- newField.addProp(stringJsonNodeEntry.getKey(),
stringJsonNodeEntry.getValue());
+ // Avro 1.9 compatible change - replaced deprecated public api
getJsonProps with getObjectProps
+ for (Map.Entry<String, Object> objectEntry :
field.getObjectProps().entrySet()) {
Review Comment:
Thanks. I have updated the PR with the changes.
##########
gobblin-core-base/src/main/java/org/apache/gobblin/converter/filter/AvroSchemaFieldRemover.java:
##########
@@ -119,14 +121,17 @@ private Schema removeFieldsFromRecords(Schema schema,
Map<String, Schema> schema
if (!this.shouldRemove(field)) {
Field newField;
if (this.children.containsKey(field.name())) {
- newField = new Field(field.name(),
this.children.get(field.name()).removeFields(field.schema(), schemaMap),
- field.doc(), field.defaultValue());
+ newField = AvroCompatibilityHelper.createSchemaField(field.name(),
+ this.children.get(field.name()).removeFields(field.schema(),
schemaMap),
+ field.doc(), AvroUtils.getCompatibleDefaultValue(field));
} else {
- newField = new Field(field.name(),
DO_NOTHING_INSTANCE.removeFields(field.schema(), schemaMap), field.doc(),
- field.defaultValue());
+ newField = AvroCompatibilityHelper.createSchemaField(field.name(),
+ DO_NOTHING_INSTANCE.removeFields(field.schema(), schemaMap),
field.doc(),
+ AvroUtils.getCompatibleDefaultValue(field));
}
- for (Map.Entry<String, JsonNode> stringJsonNodeEntry :
field.getJsonProps().entrySet()) {
- newField.addProp(stringJsonNodeEntry.getKey(),
stringJsonNodeEntry.getValue());
+ // Avro 1.9 compatible change - replaced deprecated public api
getJsonProps with getObjectProps
+ for (Map.Entry<String, Object> objectEntry :
field.getObjectProps().entrySet()) {
+ newField.addProp(objectEntry.getKey(), objectEntry.getValue());
Review Comment:
Done.
##########
gobblin-core/src/test/java/org/apache/gobblin/recordaccess/AvroGenericRecordAccessorTest.java:
##########
@@ -139,6 +139,11 @@ public void testGetStringArrayUtf8() throws IOException {
@Test
public void testGetMultiConvertsStrings() throws IOException {
+ // The below error is due to invalid avro data. Type with "null" union
must have "null" first and then
Review Comment:
Thanks for the detailed explanation. I have updated the comment.
Issue Time Tracking
-------------------
Worklog Id: (was: 818117)
Time Spent: 0.5h (was: 20m)
> Upgrade Avro version supported in Gobblin to 1.9.2
> --------------------------------------------------
>
> Key: GOBBLIN-1726
> URL: https://issues.apache.org/jira/browse/GOBBLIN-1726
> Project: Apache Gobblin
> Issue Type: Improvement
> Reporter: William Lo
> Priority: Major
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> Bumps avro version to 1.9.2
> Relies on a LinkedIn version of Hive in order to utilize
> AvroCompatibilityHelper as the version used in Gobblin is very out of date.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
