fx19880617 opened a new pull request #5341: URL: https://github.com/apache/incubator-pinot/pull/5341
Per https://github.com/apache/incubator-pinot/issues/5330, we need to support Pinot Ingestion job take an ingestion job spec template file and passing thru values. Templating is based on [jinjava](https://github.com/HubSpot/jinjava) syntax. - Adding jinjava template render support for Pinot. - Update pinot ingestion launcher command to take a list of parameters which will be the context to set in the template. - The template ingestion spec file will be rendered to final ingestion spec with the given context values. - Adding support for default values `today`, `yesterday` into context for dev simplicity. Usage example: user can define `inputDirURI` as `'file:///path/to/input/{{year}}/{{month}}/{{day}}'` during job launcher, the new command line is : ``` bin/pinot-admin.sh LaunchDataIngestionJob -jobSpecFile ingestionJobSpec.yaml -values year=2020 month=05 day=06 ``` After that the real ingestion spec passed to ingestion job will have `inputDirURI` as `'file:///path/to/input/2020/05/06'` ``` ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
