siddharthteotia commented on issue #4368: 3891: Check for validity of segment start/end time during segment generation URL: https://github.com/apache/incubator-pinot/pull/4368#issuecomment-506910360 > @siddharthteotia One work-around would be write some throw-away java code that reads in the avro in question and writes another avro with valid/allowed time stamps. Should be pretty straight forward to do that. @mayankshriv , thanks for the suggestion. Yes that is what I was doing (after intially trying to put the avro file in a hive table and then updating it via sql). After a brief discussion with @mcvsubbu , we also decided that it will be good to have an option in SegmentGeneratorConfig to enable/disable (if at all needed) this time validity check in segment generation code path. The check is enabled by default and we can see later if realtime code has any surprises in production -- in that case we can disable it. Secondly, we also discussed about the changes to failing unit tests. Very few of them were fixed by changing the data generation code. For rest of them, I was in the middle of changing the avro datasets. However, for current PR, I have just explicitly disabled the check for these unit tests that use avro file. I have another patch in progress that cleans up the avro data through a simple java program that updates the records. Also, a unit test has been added to test the exception code in SegmentGenerationWithTimeColumnTest.java
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
