ly923976094 commented on issue #3460: Who knows what these two fields mean? In the table segmentConfig ' URL: https://github.com/apache/incubator-pinot/issues/3460#issuecomment-438515162 > @ly923976094 The segmentPushFrequency is also used for realtime data as well. The value would be the same as the ones you mentioned, like `"segmentPushFrequency": "daily"`. > > So ValidationManager is run at the background when Pinot controller is up and running. It's configured in pinot controller config: `controller.validation.frequencyInSeconds`. Basically it gets the frequency of segment push defined in the table config and then searches for any gaps between data. E.g. if the frequency is set to `daily`, and the validation manager found out there's no data for 11/11, then validation manager would log the message saying `"Table: blabla has segments with invalid interval: blabla1-blabla2`, and emit any metric if possible. You can check the code in `ValidationManager.java` for more details on how Validation manager works. @jackjlli In my practice, kafka real-time data was first loaded into memory, and when I turned off the stored server, the data disappeared. When did the data in memory flush to disk? It has something to do with segmentPushFrequency
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
