Will-Lo commented on pull request #3283: URL: https://github.com/apache/gobblin/pull/3283#issuecomment-843831577
Hi, I think this is really great stuff, especially as a PoC. Went over it a bit and a few thoughts, will look into this more too but wanted to brainstorm with you: 1) I think the user experience we want to submit jobs would be in some form of bash script (easy) or k8s operator (hard). This current work is closest to a script where we can submit jobs to k8s clusters with something like `./bin/gobblin-kubernetes.sh --cluster <cluster_id> --config <gobblin_config>` 2) Passing the job properties to be read by the job launcher would be the ideal state. Since Gobblin is very configurable this will be a must. I'll take a closer look at the code too, but this is a good stepping stone to reaching that point. 3) There should be a new JobLauncher which is independent from MRJobLauncher that extends from the k8s launcher. Passing the configurations here helps as you are no longer constrained to what the CLI options are. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected]
