fx19880617 opened a new issue #6115:
URL: https://github.com/apache/incubator-pinot/issues/6115
Pinot supports launch ingestion jobs with a template file and a command line
option to set the values.
This logic is handled in class:
`org.apache.pinot.spi.ingestion.batch.IngestionJobLauncher`.
Here we want to also support environment variables to set those values.
This will be convenient for users in k8s environment with dynamic configs
settings.
E.g. user can do dynamic job loading with k8s cronjob by setting:
- Set `inputDirURI` to `'s3://event-logs/offlinepinot/dt=${JOB_DATE}/'` in
`batch_job_spec.yaml` file.
- Set container `env` with `JOB_DATE` to `$(date -u +"%Y-%m-%d")`, so
`JOB_DATE` will be the runtime date.
- Set container `args` to
```
[
"LaunchDataIngestionJob",
"-jobSpecFile",
"/home/pinot/pinot-config/local_batch_job_spec.yaml"
]
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
