npawar commented on a change in pull request #5597:
URL: https://github.com/apache/incubator-pinot/pull/5597#discussion_r444544479
##########
File path:
pinot-spi/src/main/java/org/apache/pinot/spi/data/readers/GenericRow.java
##########
@@ -49,7 +49,17 @@
*/
public class GenericRow {
+ /**
+ * This key is used by a Decoder/RecordReader to handle 1 record to many
records flattening.
+ * If a Decoder/RecordReader produces multiple GenericRows from the given
record, they must be put into the destination GenericRow as a List<GenericRow>
with this key
+ * The segment generation drivers handle this key as a special case and
process the multiple records
+ */
public static final String MULTIPLE_RECORDS_KEY = "$MULTIPLE_RECORDS_KEY$";
+ /**
+ * This key is used by the FilterTransformer to handle filtering out of
records during ingestion
+ * The FilterTransformer puts this key into the GenericRow with value true,
if the record matches the filtering out criteria, based on FilterConfig
+ */
+ public static final String FILTER_RECORD_KEY = "$FILTER_RECORD_KEY$";
Review comment:
Multiple records key feature didn't go into the release. It has been
documented here:
https://docs.pinot.apache.org/developers/advanced/ingestion-level-transformations#one-record-into-many
Renamed to `$SKIP_RECORDS_KEY$`. Also based on Jackie's comment, will avoid
making it long, so multiple key remains `$MULTIPLE_RECORDS_KEY$`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
