Jackie-Jiang commented on a change in pull request #5238: Evaluate schema
transform expressions during ingestion
URL: https://github.com/apache/incubator-pinot/pull/5238#discussion_r407678994
##########
File path:
pinot-spi/src/main/java/org/apache/pinot/spi/data/readers/RecordReaderUtils.java
##########
@@ -67,31 +63,66 @@ public static InputStream getInputStream(File dataFile)
}
/**
- * Extracts all field specs from the given schema.
- * <p>For time field spec:
- * <ul>
- * <li>If incoming and outgoing time column name are the same, use
incoming time field spec</li>
- * <li>If incoming and outgoing time column name are different, put both
of them as time field spec</li>
- * <li>
- * We keep both incoming and outgoing time column to handle cases where
the input file contains time values that
- * are already converted
- * </li>
- * </ul>
+ * Converts the value to a multi-values value or a single values value
*/
- public static List<FieldSpec> extractFieldSpecs(Schema schema) {
- List<FieldSpec> fieldSpecs = new ArrayList<>();
- for (FieldSpec fieldSpec : schema.getAllFieldSpecs()) {
- if (fieldSpec.getFieldType() == FieldSpec.FieldType.TIME) {
- TimeFieldSpec timeFieldSpec = (TimeFieldSpec) fieldSpec;
- fieldSpecs.add(new
TimeFieldSpec(timeFieldSpec.getIncomingGranularitySpec()));
- if
(!timeFieldSpec.getOutgoingTimeColumnName().equals(timeFieldSpec.getIncomingTimeColumnName()))
{
- fieldSpecs.add(new
TimeFieldSpec(timeFieldSpec.getOutgoingGranularitySpec()));
- }
- } else {
- fieldSpecs.add(fieldSpec);
+ public static Object convert(Object value) {
Review comment:
I don't think this logic can be shared among all the record readers. For
different input format, we might need different utils for them
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
