snleee commented on a change in pull request #3895: Fix the RecordReader to
pick incoming time column
URL: https://github.com/apache/incubator-pinot/pull/3895#discussion_r261494255
##########
File path:
pinot-core/src/main/java/org/apache/pinot/core/data/readers/AvroRecordReader.java
##########
@@ -60,26 +64,58 @@ public AvroRecordReader(File dataFile, Schema schema)
private void validateSchema() {
org.apache.avro.Schema avroSchema = _avroReader.getSchema();
for (FieldSpec fieldSpec : _schema.getAllFieldSpecs()) {
- String fieldName = fieldSpec.getName();
- Field avroField = avroSchema.getField(fieldName);
- if (avroField == null) {
- LOGGER.warn("Pinot field: {} does not exist in Avro Schema",
fieldName);
- } else {
- boolean isPinotFieldSingleValue = fieldSpec.isSingleValueField();
- boolean isAvroFieldSingleValue =
AvroUtils.isSingleValueField(avroField);
- if (isPinotFieldSingleValue != isAvroFieldSingleValue) {
- String errorMessage = "Pinot field: " + fieldName + " is " +
(isPinotFieldSingleValue ? "Single" : "Multi")
- + "-valued in Pinot schema but not in Avro schema";
+ if (fieldSpec.getFieldType() == FieldSpec.FieldType.TIME) {
Review comment:
Is there a way to avoid this repeating validation logic for each record
reader? This makes hard for people to add a new type of record reader. For
instance, outside contributor is working on adding a parquet reader
https://github.com/apache/incubator-pinot/pull/3852
One approach would be always try to read incoming & outgoing time column
values and do the validation based on `GenericRow`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
