[
https://issues.apache.org/jira/browse/AIRFLOW-4030?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17042866#comment-17042866
]
ASF subversion and git services commented on AIRFLOW-4030:
----------------------------------------------------------
Commit 0bb687990b94da7445f4ba081592de8cea73119e in airflow's branch
refs/heads/master from Vanessasaurus
[ https://gitbox.apache.org/repos/asf?p=airflow.git;h=0bb6879 ]
[AIRFLOW-4030] second attempt to add singularity to airflow (#7191)
* adding singularity operator and tests
Signed-off-by: Vanessa Sochat <[email protected]>
* removing encoding pragmas and fixing up dockerfile to pass linting
Signed-off-by: Vanessa Sochat <[email protected]>
* make workdir in /tmp because AIRFLOW_SOURCES not defined yet
Signed-off-by: Vanessa Sochat <[email protected]>
* curl needs to follow redirects with -L
Signed-off-by: Vanessa Sochat <[email protected]>
* moving files to where they are supposed to be, more changes to mock, no clue
Signed-off-by: vsoch <[email protected]>
* removing trailing whitespace, moving example_dag for singularity, adding
licenses to empty init files
Signed-off-by: vsoch <[email protected]>
* ran isort on example dags file
Signed-off-by: vsoch <[email protected]>
* adding missing init in example_dags folder for singularity
Signed-off-by: vsoch <[email protected]>
* removing code from __init__.py files for singularity operator to fix
documentation generation
Signed-off-by: vsoch <[email protected]>
* forgot to update link to singularity in operators and hooks ref
Signed-off-by: vsoch <[email protected]>
* command must have been provided on init of singularity operator instance
Signed-off-by: vsoch <[email protected]>
* I guess I'm required to have a task_id?
Signed-off-by: vsoch <[email protected]>
* try adding working_dir to singularity operator type definitions
Signed-off-by: vsoch <[email protected]>
* disable too many arguments for pylint of singularity operator init
Signed-off-by: vsoch <[email protected]>
* move pylint disable up to line 64 - doesnt catch at end of statement like
other examples
Signed-off-by: vsoch <[email protected]>
* two spaces before inline comment
Signed-off-by: vsoch <[email protected]>
* I dont see task_id as a param for other providers, removing for singularity
operator
Signed-off-by: vsoch <[email protected]>
* adding debug print
Signed-off-by: vsoch <[email protected]>
* allow for return of just image and/or lines
Signed-off-by: vsoch <[email protected]>
* dont understand how mock works, but the image should exist after its
pulled....
Signed-off-by: vsoch <[email protected]>
* try removing shutil, the client should handle pull folder instead
Signed-off-by: vsoch <[email protected]>
* try changing pull-file to same uri that is expected to be pulled
Signed-off-by: vsoch <[email protected]>
* import of AirflowException moved to exceptions
Signed-off-by: vsoch <[email protected]>
* DAG module was moved to airflow.models
Signed-off-by: vsoch <[email protected]>
* ensure pull is called with pull_folder
Signed-off-by: vsoch <[email protected]>
> Add Singularity Container Operator
> ----------------------------------
>
> Key: AIRFLOW-4030
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4030
> Project: Apache Airflow
> Issue Type: Improvement
> Components: operators
> Reporter: Vanessa
> Assignee: Vanessa
> Priority: Minor
> Fix For: 2.0.0
>
>
> Singularity containers are comparable to Docker in the level of operation -
> they provide an encapsulated environment with an OS, libraries, and custom
> software for the user to run. The key difference is that Docker is not
> optimized for scientific compute because it could never be installed on a
> shared research cluster. Singularity, on the other hand, does not have these
> issues and is installed across HCP centers internationally.
> This issue is to add Singularity containers as an operator to Apache Airflow,
> so that we can start to explore using airflow in an HPC environment. I work
> with Encode DCC at Stanford, and am hopeful to explore Airflow as an
> alternative to the workflow manager(s) we are using. I am one of the
> [original Singularity developers see
> |https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0177459]
> that manages the Singularity Python client (spython), Singularity Hub and
> Singularity Registry Server, and have started working on this issue here:
> [https://github.com/apache/airflow/pull/4846.] Looking forward to working
> with you!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
