ZihanLi58 commented on a change in pull request #2753: [GOBBLIN-899]Add config
in replication config to determine wheter schema cehck ena…
URL: https://github.com/apache/incubator-gobblin/pull/2753#discussion_r332666140
##########
File path:
gobblin-data-management/src/main/java/org/apache/gobblin/data/management/copy/extractor/FileAwareInputStreamExtractorWithCheckSchema.java
##########
@@ -66,14 +67,19 @@ protected FileAwareInputStream buildStream(FileSystem
fsFromFile) throws DataRec
* @throws IOException
*/
protected boolean schemaChecking(FileSystem fsFromFile) throws IOException {
+ if(
!this.state.getPropAsBoolean(ReplicationConfiguration.COPY_SCHEMA_CHECK_ENABLED,
false) ) {
+ return true;
+ }
DatumReader<GenericRecord> datumReader = new GenericDatumReader<>();
DataFileReader<GenericRecord> dataFileReader =
new DataFileReader(new FsInput(this.file.getFileStatus().getPath(),
new Configuration()), datumReader);
Review comment:
There is no existing generic logic to read schema because the schema need to
be read by specific reader. And the next PR should enable schema check for ORC
schema as well. But it will be implemented by an if else statement which is
also not generic.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
