vsoch commented on issue #7191: [AIRFLOW-4030] second attempt to add
singularity to airflow
URL: https://github.com/apache/airflow/pull/7191#issuecomment-590052010
Ah, I love the computer history museum! I’m actually a remote worker to
Stanford based out of Colorado, so there is still an
vsoch commented on issue #7191: [AIRFLOW-4030] second attempt to add
singularity to airflow
URL: https://github.com/apache/airflow/pull/7191#issuecomment-590050857
Thanks everyone :) It's just a little before 3:00am here but I still think
I'm going to have a tiny dance party before I slip
vsoch commented on issue #7191: [AIRFLOW-4030] second attempt to add
singularity to airflow
URL: https://github.com/apache/airflow/pull/7191#issuecomment-590049375
holy crap it passed! Woot!!
This is an automated message
vsoch commented on issue #7191: [AIRFLOW-4030] second attempt to add
singularity to airflow
URL: https://github.com/apache/airflow/pull/7191#issuecomment-590029643
Alright, rebase is just done! I think it's likely something transient,
because the rebase just had changes to UPDATING.md
vsoch commented on issue #7191: [AIRFLOW-4030] second attempt to add
singularity to airflow
URL: https://github.com/apache/airflow/pull/7191#issuecomment-590012811
@potiuk for the recent tests, I'm seeing a redirect (302) for an address
that (used to be?) 200? I don't see any singularity
vsoch commented on issue #7191: [AIRFLOW-4030] second attempt to add
singularity to airflow
URL: https://github.com/apache/airflow/pull/7191#issuecomment-589395121
looks like the error linked above was one off - a later build didn't trigger
it.
vsoch commented on issue #7191: [AIRFLOW-4030] second attempt to add
singularity to airflow
URL: https://github.com/apache/airflow/pull/7191#issuecomment-589285097
Quick question - I don't see any reference to the singularity operator
vsoch commented on issue #7191: [AIRFLOW-4030] second attempt to add
singularity to airflow
URL: https://github.com/apache/airflow/pull/7191#issuecomment-588482545
Hmm, so I definitely added the disable line, but it doesn't seem to be
taking:
vsoch commented on issue #7191: [AIRFLOW-4030] second attempt to add
singularity to airflow
URL: https://github.com/apache/airflow/pull/7191#issuecomment-588463321
@potiuk I tried this huge build multiple times before, and they all resulted
in errors. I'm glad to try it again, but I've
vsoch commented on issue #7191: [AIRFLOW-4030] second attempt to add
singularity to airflow
URL: https://github.com/apache/airflow/pull/7191#issuecomment-588448150
okay, I found plenty of examples and copied those, so I didn't need to run
it manually. Let's hope that works!
vsoch commented on issue #7191: [AIRFLOW-4030] second attempt to add
singularity to airflow
URL: https://github.com/apache/airflow/pull/7191#issuecomment-588442023
One quick note about the docs for pylint - the files are now changed to have
pylint_main.sh and pylint_tests.sh. Also, this
vsoch commented on issue #7191: [AIRFLOW-4030] second attempt to add
singularity to airflow
URL: https://github.com/apache/airflow/pull/7191#issuecomment-588437467
okay, I'll try that! It does seem like I'm between a rock and a hard place,
glad that you are okay with disabling pylint
vsoch commented on issue #7191: [AIRFLOW-4030] second attempt to add
singularity to airflow
URL: https://github.com/apache/airflow/pull/7191#issuecomment-588357686
How would you like me to address this?
```
vsoch commented on issue #7191: [AIRFLOW-4030] second attempt to add
singularity to airflow
URL: https://github.com/apache/airflow/pull/7191#issuecomment-587936273
Actually it looks like it wasn't happy with working_dir as a kwargs, I'll
try adding it to the type definitions to see if
vsoch commented on issue #7191: [AIRFLOW-4030] second attempt to add
singularity to airflow
URL: https://github.com/apache/airflow/pull/7191#issuecomment-587935001
Can somebody explain what the task_id (provided as task-id?) is expected to
work, be parsed, etc? @mik-laj added it to the
vsoch commented on issue #7191: [AIRFLOW-4030] second attempt to add
singularity to airflow
URL: https://github.com/apache/airflow/pull/7191#issuecomment-587823490
docs build passed! Tiny progress :)
This is an automated
vsoch commented on issue #7191: [AIRFLOW-4030] second attempt to add
singularity to airflow
URL: https://github.com/apache/airflow/pull/7191#issuecomment-587082403
Ah thank you! Just added.
This is an automated message from
vsoch commented on issue #7191: [AIRFLOW-4030] second attempt to add
singularity to airflow
URL: https://github.com/apache/airflow/pull/7191#issuecomment-586651037
Can you show me where to add in the toc tree? I thought I grepped for
similar providers and added it, but it's not happy:
vsoch commented on issue #7191: [AIRFLOW-4030] second attempt to add
singularity to airflow
URL: https://github.com/apache/airflow/pull/7191#issuecomment-586644781
> Sorry - devl...@airflow.apache.org - see CONTRIB.rst. I was typing it on
my tablet and autocomplete kicked in.
Haha,
vsoch commented on issue #7191: [AIRFLOW-4030] second attempt to add
singularity to airflow
URL: https://github.com/apache/airflow/pull/7191#issuecomment-586640931
Okay the changes I've just made:
- moved `airflow/contrib/operators/singularity_operator.py` to
vsoch commented on issue #7191: [AIRFLOW-4030] second attempt to add
singularity to airflow
URL: https://github.com/apache/airflow/pull/7191#issuecomment-586639114
Conflict resolved - it was the same line as before, another package was
added! I added a new line (it was getting too long)
vsoch commented on issue #7191: [AIRFLOW-4030] second attempt to add
singularity to airflow
URL: https://github.com/apache/airflow/pull/7191#issuecomment-586638581
Note that there were conflicts with setup.py before my set of last commits,
and I resolved them here.
vsoch commented on issue #7191: [AIRFLOW-4030] second attempt to add
singularity to airflow
URL: https://github.com/apache/airflow/pull/7191#issuecomment-586638274
I can try to address the conflicts, but I can't rebase it seems:
```
Auto-merging setup.py
CONFLICT (content):
vsoch commented on issue #7191: [AIRFLOW-4030] second attempt to add
singularity to airflow
URL: https://github.com/apache/airflow/pull/7191#issuecomment-586424350
Questions:
- where does example_singularity_operator.py go - it used to be under
contrib/example_dags but now, that
vsoch commented on issue #7191: [AIRFLOW-4030] second attempt to add
singularity to airflow
URL: https://github.com/apache/airflow/pull/7191#issuecomment-586178588
I totally agree on all points! I can’t imagine what managing a PR board like
this must be like.
I’m mostly happy that
vsoch commented on issue #7191: [AIRFLOW-4030] second attempt to add
singularity to airflow
URL: https://github.com/apache/airflow/pull/7191#issuecomment-586067049
High level feedback since this was a topic earlier:
- the extra linting for the Dockerfile, which comes down to fairly
vsoch commented on issue #7191: [AIRFLOW-4030] second attempt to add
singularity to airflow
URL: https://github.com/apache/airflow/pull/7191#issuecomment-586039637
okay I'm going to assume that this "no pragma" is new and just remove the
lines from the top of those files. I also used the
vsoch commented on issue #7191: [AIRFLOW-4030] second attempt to add
singularity to airflow
URL: https://github.com/apache/airflow/pull/7191#issuecomment-586036255
What are these new errors about encoding / pragma? I literally didn't
change any of these files
```
vsoch commented on issue #7191: [AIRFLOW-4030] second attempt to add
singularity to airflow
URL: https://github.com/apache/airflow/pull/7191#issuecomment-586021498
okay to so to update everyone following (and future me reading back to
remember this) we are first trying a simple approach
vsoch commented on issue #7191: [AIRFLOW-4030] second attempt to add
singularity to airflow
URL: https://github.com/apache/airflow/pull/7191#issuecomment-585967157
Perfecto! I'll give this a shot.
This is an automated
vsoch commented on issue #7191: [AIRFLOW-4030] second attempt to add
singularity to airflow
URL: https://github.com/apache/airflow/pull/7191#issuecomment-585946960
> I assume we can start singularity from the host and be able to forward
this connection to inside the airlfow-testing
vsoch commented on issue #7191: [AIRFLOW-4030] second attempt to add
singularity to airflow
URL: https://github.com/apache/airflow/pull/7191#issuecomment-585869248
ping...
This is an automated message from the Apache Git
vsoch commented on issue #7191: [AIRFLOW-4030] second attempt to add
singularity to airflow
URL: https://github.com/apache/airflow/pull/7191#issuecomment-579881380
okay just to clarify - you want a Singularity + Airflow container run via a
similar kind cluster? You said something about
vsoch commented on issue #7191: [AIRFLOW-4030] second attempt to add
singularity to airflow
URL: https://github.com/apache/airflow/pull/7191#issuecomment-579878775
Oh wait I see, the "deploy_airflow_to_kubernetes.sh" detects the runtime and
triggers the build to happen via:
vsoch commented on issue #7191: [AIRFLOW-4030] second attempt to add
singularity to airflow
URL: https://github.com/apache/airflow/pull/7191#issuecomment-579877224
okay so I'm tracing the kubernetes (runtime) as an example, and I have a
quick question. In
vsoch commented on issue #7191: [AIRFLOW-4030] second attempt to add
singularity to airflow
URL: https://github.com/apache/airflow/pull/7191#issuecomment-579359117
I think the third approach is best:
> If you need something pre-backed in the Docker image itself then I think
we need
vsoch commented on issue #7191: [AIRFLOW-4030] second attempt to add
singularity to airflow
URL: https://github.com/apache/airflow/pull/7191#issuecomment-577373099
okay, I've taken a look at the docker-compose approach. Specifically, I've
created a `integration-singularity.yml` file that
vsoch commented on issue #7191: [AIRFLOW-4030] second attempt to add
singularity to airflow
URL: https://github.com/apache/airflow/pull/7191#issuecomment-577337258
[Here are my
thoughts](https://gist.github.com/vsoch/599bd51246c2b2d3da350f97c0916efe)
@potiuk, I hope that they are
vsoch commented on issue #7191: [AIRFLOW-4030] second attempt to add
singularity to airflow
URL: https://github.com/apache/airflow/pull/7191#issuecomment-576851524
Writing up this robust feedback and researching the CI solutions is going to
take me some time, just to keep you updated I
vsoch commented on issue #7191: [AIRFLOW-4030] second attempt to add
singularity to airflow
URL: https://github.com/apache/airflow/pull/7191#issuecomment-576472830
But one quick note - regardless of the testing troubles, the support on this
PR has been absolutely excellent, night and day
vsoch commented on issue #7191: [AIRFLOW-4030] second attempt to add
singularity to airflow
URL: https://github.com/apache/airflow/pull/7191#issuecomment-576471260
You definitely can’t get a fully working Singularity from within Docker
(even with privileged it’s a bit janky) so I’ll need
vsoch commented on issue #7191: [AIRFLOW-4030] second attempt to add
singularity to airflow
URL: https://github.com/apache/airflow/pull/7191#issuecomment-576470774
Ah understood, I thought it was for the PR title, I’ll update the message
for the next fix I do.
I think the
vsoch commented on issue #7191: [AIRFLOW-4030] second attempt to add
singularity to airflow
URL: https://github.com/apache/airflow/pull/7191#issuecomment-576459284
Also, could you give feedback on:
> Title Validator — Wrong commit title: adding singularity operator and
tests
vsoch commented on issue #7191: [AIRFLOW-4030] second attempt to add
singularity to airflow
URL: https://github.com/apache/airflow/pull/7191#issuecomment-576459203
The linting appears to be all set, but since Singularity is installed, the
tests are obviously going to fail. Let me know how
44 matches
Mail list logo
