oelesinsc24 commented on issue #3504: [AIRFLOW-2310]: Add AWS Glue Job Compatibility to Airflow URL: https://github.com/apache/incubator-airflow/pull/3504#issuecomment-431243451 > @OElesin Have you been able to make any progress on this? > There's one duplicate file in there `airflow/contrib/aws_glue_job_hook.py` vs `airflow/contrib/hook/aws_glue_job_hook.py`. > > [edit] From looking at the PR it seems it will both create a job (using boto3's `create_job()`) as well as run the job (using boto3's `start_job_run()`). IMHO at the very least these should be separate things so one can run jobs without creating them. > To expand on this, we create the jobs using a normal development workflow/CI and we just want to be able to run them from Airflow. IMHO it's not Airflow's responsibility to create resources, it should only invoke them. > > FYI All the unrelated changes are because you merged `master` into your branch instead of rebasing your branch on top of `master` This is absolutely correct. And also makes the implementation way easier. I would make the necessary changes and add the commit.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services
