npawar opened a new issue #6242: URL: https://github.com/apache/incubator-pinot/issues/6242
Currently when running a batch ingestion job, all the configs are provided via the ingestion-job-spec yaml file to the job. This will not be possible if we want to move to a pull-based ingestion model for segment creation. For example, in the pull-based model, we want to simply keep creating segments whenever a new file is provided into some fixed path. This means all the configs needed for segment generation should be included into the table config. The proposal is to introduce batchConfigs into the tableConfig. This would look very similar to the streamConfigs, but would be for batch sources. We will add this into the ingestionConfig. We will also take this opportunity to move streamConfigs into the ingestionConfigs. Please comment on the doc: https://docs.google.com/document/d/1PXtD2VFlWkInHpsmRvvgFwcCasWmjw0iM1YySy0eSIg/edit?usp=sharing ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
