[
https://issues.apache.org/jira/browse/AIRFLOW-4722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16854681#comment-16854681
]
ASF GitHub Bot commented on AIRFLOW-4722:
-----------------------------------------
rummens commented on pull request #5366: [AIRFLOW-4722] Add third PV mount to
worker node to share data
URL: https://github.com/apache/airflow/pull/5366
For many use cases it is neccessary to write data from a worker back to a PV
to share it between other workers (e.g. for TFX). Therefore a new (third) PVC
should be optionaly mounted into the worker nodes (as well as the webserver).
The PVC has to be provided by the user (similiar to DAG and log PV). It has to
be mounted with the ReadOnly flag off.
As this is very new to me, I am not sure if I did everything correct. I was
not able to run all tests but it should not break anything as this is mainly a
copy of the log PV. Also I am not sure if the mount also works for the
webserver pod.
### Jira
- [ ] My PR addresses the following [Airflow
Jira](https://issues.apache.org/jira/browse/AIRFLOW-4722)
### Description
- Add a new parameter to add a new PV called airflow-data, which is mounted
as writeable into a worker to allow sharing of data between workers.
### Tests
- Sorry but I have no idea what to here :-/
### Documentation
Updaded the default config with the new param.
### Code Quality
- How can I run this?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
> Add third PV mount to worker node to share data
> -----------------------------------------------
>
> Key: AIRFLOW-4722
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4722
> Project: Apache Airflow
> Issue Type: Improvement
> Components: configuration, scheduler, worker
> Affects Versions: 1.10.2, 1.10.3
> Reporter: Marcel Rummens
> Priority: Major
> Original Estimate: 2m
> Remaining Estimate: 2m
>
> For many use cases it is neccessary to write data from a worker back to a PV
> to share it between other workers (e.g. for TFX). Therefore a new (third) PVC
> should be optionaly mounted into the worker nodes (as well as the webserver).
> The PVC has to be provided by the user (similiar to DAG and log PV). It has
> to be mounted with the ReadOnly flag off.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
