[
https://issues.apache.org/jira/browse/GOBBLIN-1006?focusedWorklogId=361131&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-361131
]
ASF GitHub Bot logged work on GOBBLIN-1006:
-------------------------------------------
Author: ASF GitHub Bot
Created on: 17/Dec/19 21:40
Start Date: 17/Dec/19 21:40
Worklog Time Spent: 10m
Work Description: autumnust commented on pull request #2851:
[GOBBLIN-1006] Enable configurable case-preserving and schema source-of-truth
in table level properties
URL: https://github.com/apache/incubator-gobblin/pull/2851#discussion_r359043176
##########
File path:
gobblin-data-management/src/main/java/org/apache/gobblin/data/management/conversion/hive/query/HiveAvroORCQueryGenerator.java
##########
@@ -160,21 +163,25 @@ public static String generateCreateTableDDL(Schema
schema,
// (evolution does not matter if its new destination table)
// 4. If evolution is disabled, and destination table does exists
// .. use columns from destination schema
+ // Make sure the schema attribute will be updated in source-of-truth
attribute.
+ // Or fall back to default attribute-pair used in Hive for ORC format.
+ if (tableProperties.containsKey(SCHEMA_SOURCE_OF_TRUTH)) {
+
tableProperties.setProperty(tableProperties.getProperty(SCHEMA_SOURCE_OF_TRUTH),
schema.toString());
+ tableProperties.remove(SCHEMA_SOURCE_OF_TRUTH);
Review comment:
So, usually this should be `avro.schema.literal`, which is what we did
internally for most of tables.
While there are cases where we cannot set `avro.schema.literal`: When there
were older files written in lower-case ORC but newly added
`avro.schema.literal` is case-preserved. Any mismatch between
`avro.schema.literal` and file-level schema will cause problem, if
`avro.schema.literal` is being set which gives offline system like Dali an
illusion that this is the correct schema.
But we still need something to inform pipelines like "view-creation" the
Avro schema of a table, since interfering it from Hive schema is lossy. When
this case happens, we will set this `SCHEMA_SOURCE_OF_TRUTH ` to be something
else (different from `avro.schema.literal`) as a workaround to provide Avro
schema but not messing with Dali.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 361131)
Time Spent: 50m (was: 40m)
> Configurable case-preserving in ORC-conversion
> ----------------------------------------------
>
> Key: GOBBLIN-1006
> URL: https://issues.apache.org/jira/browse/GOBBLIN-1006
> Project: Apache Gobblin
> Issue Type: Improvement
> Reporter: Lei Sun
> Priority: Major
> Time Spent: 50m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
