[jira] [Commented] (AIRFLOW-1319) Fix misleading SparkSubmitOperator and SparkSubmitHook docstring
[ https://issues.apache.org/jira/browse/AIRFLOW-1319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16053709#comment-16053709 ] Chris Sng commented on AIRFLOW-1319: PR here https://github.com/apache/incubator-airflow/pull/2377 > Fix misleading SparkSubmitOperator and SparkSubmitHook docstring > > > Key: AIRFLOW-1319 > URL: https://issues.apache.org/jira/browse/AIRFLOW-1319 > Project: Apache Airflow > Issue Type: Improvement > Components: contrib >Reporter: Chris Sng >Assignee: Chris Sng >Priority: Trivial > Labels: documentation > Original Estimate: 5m > Remaining Estimate: 5m > > In the community-contributed Spark submit hook and operator, it support > {{spark-submit}}'s {{\--files}} command line option. The {{\--files}} option > is used to submit file to each executor to be used. A good example of such > files are serialized objects. > However, in both docstrings > (https://github.com/apache/incubator-airflow/blob/master/airflow/contrib/operators/spark_submit_operator.py#L37 > and > https://github.com/apache/incubator-airflow/blob/master/airflow/contrib/hooks/spark_submit_hook.py#L36), > it provided {{hive-site.xml}} as an example. This may mislead less-informed > developers into assuming that Hive configuration files can be submitted to > the cluster in this manner. According to Apache Hive's documentation, hive > configuration files are located in the directory located in the > {{HIVE_CONF_DIR}} environment variable. > I propose excluding this example from the docstrings. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (AIRFLOW-1319) Fix misleading SparkSubmitOperator and SparkSubmitHook docstring
[ https://issues.apache.org/jira/browse/AIRFLOW-1319?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Sng updated AIRFLOW-1319: --- Description: In the community-contributed Spark submit hook and operator, it support {{spark-submit}} {{\--files}} command line option. The {{\--files}} option is used to submit file to each executor to be used. A good example of such files are serialized objects. However, in both docstrings (https://github.com/apache/incubator-airflow/blob/master/airflow/contrib/operators/spark_submit_operator.py#L37 and https://github.com/apache/incubator-airflow/blob/master/airflow/contrib/hooks/spark_submit_hook.py#L36), it provided {{hive-site.xml}} as an example. This may mislead less-informed developers into assuming that Hive configuration files can be submitted to the cluster in this manner. According to Apache Hive's documentation, hive configuration files are located in the directory located in the {{HIVE_CONF_DIR}} environment variable. I propose excluding this example from the docstrings. was: In the community-contributed Spark submit hook and operator, it support {{spark-submit}} {{--files}} command line option. The {{---files}} option is used to submit file to each executor to be used. A good example of such files are serialized objects. However, in both docstrings (https://github.com/apache/incubator-airflow/blob/master/airflow/contrib/operators/spark_submit_operator.py#L37 and https://github.com/apache/incubator-airflow/blob/master/airflow/contrib/hooks/spark_submit_hook.py#L36), it provided {{hive-site.xml}} as an example. This may mislead less-informed developers into assuming that Hive configuration files can be submitted to the cluster in this manner. According to Apache Hive's documentation, hive configuration files are located in the directory located in the {{HIVE_CONF_DIR}} environment variable. I propose excluding this example from the docstrings. > Fix misleading SparkSubmitOperator and SparkSubmitHook docstring > > > Key: AIRFLOW-1319 > URL: https://issues.apache.org/jira/browse/AIRFLOW-1319 > Project: Apache Airflow > Issue Type: Improvement > Components: contrib >Reporter: Chris Sng >Assignee: Chris Sng >Priority: Trivial > Labels: documentation > Original Estimate: 5m > Remaining Estimate: 5m > > In the community-contributed Spark submit hook and operator, it support > {{spark-submit}} {{\--files}} command line option. The {{\--files}} option is > used to submit file to each executor to be used. A good example of such files > are serialized objects. > However, in both docstrings > (https://github.com/apache/incubator-airflow/blob/master/airflow/contrib/operators/spark_submit_operator.py#L37 > and > https://github.com/apache/incubator-airflow/blob/master/airflow/contrib/hooks/spark_submit_hook.py#L36), > it provided {{hive-site.xml}} as an example. This may mislead less-informed > developers into assuming that Hive configuration files can be submitted to > the cluster in this manner. According to Apache Hive's documentation, hive > configuration files are located in the directory located in the > {{HIVE_CONF_DIR}} environment variable. > I propose excluding this example from the docstrings. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (AIRFLOW-1319) Fix misleading SparkSubmitOperator and SparkSubmitHook docstring
[ https://issues.apache.org/jira/browse/AIRFLOW-1319?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Sng updated AIRFLOW-1319: --- Description: In the community-contributed Spark submit hook and operator, it support {{spark-submit}}'s {{\--files}} command line option. The {{\--files}} option is used to submit file to each executor to be used. A good example of such files are serialized objects. However, in both docstrings (https://github.com/apache/incubator-airflow/blob/master/airflow/contrib/operators/spark_submit_operator.py#L37 and https://github.com/apache/incubator-airflow/blob/master/airflow/contrib/hooks/spark_submit_hook.py#L36), it provided {{hive-site.xml}} as an example. This may mislead less-informed developers into assuming that Hive configuration files can be submitted to the cluster in this manner. According to Apache Hive's documentation, hive configuration files are located in the directory located in the {{HIVE_CONF_DIR}} environment variable. I propose excluding this example from the docstrings. was: In the community-contributed Spark submit hook and operator, it support {{spark-submit}} {{\--files}} command line option. The {{\--files}} option is used to submit file to each executor to be used. A good example of such files are serialized objects. However, in both docstrings (https://github.com/apache/incubator-airflow/blob/master/airflow/contrib/operators/spark_submit_operator.py#L37 and https://github.com/apache/incubator-airflow/blob/master/airflow/contrib/hooks/spark_submit_hook.py#L36), it provided {{hive-site.xml}} as an example. This may mislead less-informed developers into assuming that Hive configuration files can be submitted to the cluster in this manner. According to Apache Hive's documentation, hive configuration files are located in the directory located in the {{HIVE_CONF_DIR}} environment variable. I propose excluding this example from the docstrings. > Fix misleading SparkSubmitOperator and SparkSubmitHook docstring > > > Key: AIRFLOW-1319 > URL: https://issues.apache.org/jira/browse/AIRFLOW-1319 > Project: Apache Airflow > Issue Type: Improvement > Components: contrib >Reporter: Chris Sng >Assignee: Chris Sng >Priority: Trivial > Labels: documentation > Original Estimate: 5m > Remaining Estimate: 5m > > In the community-contributed Spark submit hook and operator, it support > {{spark-submit}}'s {{\--files}} command line option. The {{\--files}} option > is used to submit file to each executor to be used. A good example of such > files are serialized objects. > However, in both docstrings > (https://github.com/apache/incubator-airflow/blob/master/airflow/contrib/operators/spark_submit_operator.py#L37 > and > https://github.com/apache/incubator-airflow/blob/master/airflow/contrib/hooks/spark_submit_hook.py#L36), > it provided {{hive-site.xml}} as an example. This may mislead less-informed > developers into assuming that Hive configuration files can be submitted to > the cluster in this manner. According to Apache Hive's documentation, hive > configuration files are located in the directory located in the > {{HIVE_CONF_DIR}} environment variable. > I propose excluding this example from the docstrings. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (AIRFLOW-1319) Fix misleading SparkSubmitOperator and SparkSubmitHook docstring
[ https://issues.apache.org/jira/browse/AIRFLOW-1319?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Sng updated AIRFLOW-1319: --- Description: In the community-contributed Spark submit hook and operator, it support {{spark-submit}}'s {{--files}} command line option. The {{--files}} option is used to submit file to each executor to be used. A good example of such files are serialized objects. However, in both docstrings (https://github.com/apache/incubator-airflow/blob/master/airflow/contrib/operators/spark_submit_operator.py#L37 and https://github.com/apache/incubator-airflow/blob/master/airflow/contrib/hooks/spark_submit_hook.py#L36), it provided {{hive-site.xml}} as an example. This may mislead less-informed developers into assuming that Hive configuration files can be submitted to the cluster in this manner. According to Apache Hive's documentation, hive configuration files are located in the directory located in the `HIVE_CONF_DIR` environment variable. I propose excluding this example from the docstrings. was: In the community-contributed Spark submit hook and operator, it support `spark-submit`'s `--files` command line option. The `--files` option is used to submit file to each executor to be used. A good example of such files are serialized objects. However, in both docstrings (https://github.com/apache/incubator-airflow/blob/master/airflow/contrib/operators/spark_submit_operator.py#L37 and https://github.com/apache/incubator-airflow/blob/master/airflow/contrib/hooks/spark_submit_hook.py#L36), it provided `hive-site.xml` as an example. This may mislead less-informed developers into assuming that Hive configuration files can be submitted to the cluster in this manner. According to Apache Hive's documentation, hive configuration files are located in the directory located in the `HIVE_CONF_DIR` environment variable. I propose excluding this example from the docstrings. > Fix misleading SparkSubmitOperator and SparkSubmitHook docstring > > > Key: AIRFLOW-1319 > URL: https://issues.apache.org/jira/browse/AIRFLOW-1319 > Project: Apache Airflow > Issue Type: Improvement > Components: contrib >Reporter: Chris Sng >Assignee: Chris Sng >Priority: Trivial > Labels: documentation > Original Estimate: 5m > Remaining Estimate: 5m > > In the community-contributed Spark submit hook and operator, it support > {{spark-submit}}'s {{--files}} command line option. The {{--files}} option is > used to submit file to each executor to be used. A good example of such files > are serialized objects. > However, in both docstrings > (https://github.com/apache/incubator-airflow/blob/master/airflow/contrib/operators/spark_submit_operator.py#L37 > and > https://github.com/apache/incubator-airflow/blob/master/airflow/contrib/hooks/spark_submit_hook.py#L36), > it provided {{hive-site.xml}} as an example. This may mislead less-informed > developers into assuming that Hive configuration files can be submitted to > the cluster in this manner. According to Apache Hive's documentation, hive > configuration files are located in the directory located in the > `HIVE_CONF_DIR` environment variable. > I propose excluding this example from the docstrings. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (AIRFLOW-1319) Fix misleading SparkSubmitOperator and SparkSubmitHook docstring
[ https://issues.apache.org/jira/browse/AIRFLOW-1319?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Sng updated AIRFLOW-1319: --- Description: In the community-contributed Spark submit hook and operator, it support {{spark-submit}} {{--files}} command line option. The {{---files}} option is used to submit file to each executor to be used. A good example of such files are serialized objects. However, in both docstrings (https://github.com/apache/incubator-airflow/blob/master/airflow/contrib/operators/spark_submit_operator.py#L37 and https://github.com/apache/incubator-airflow/blob/master/airflow/contrib/hooks/spark_submit_hook.py#L36), it provided {{hive-site.xml}} as an example. This may mislead less-informed developers into assuming that Hive configuration files can be submitted to the cluster in this manner. According to Apache Hive's documentation, hive configuration files are located in the directory located in the {{HIVE_CONF_DIR}} environment variable. I propose excluding this example from the docstrings. was: In the community-contributed Spark submit hook and operator, it support {{spark-submit}} {{--files}} command line option. The {{---files}} option is used to submit file to each executor to be used. A good example of such files are serialized objects. However, in both docstrings (https://github.com/apache/incubator-airflow/blob/master/airflow/contrib/operators/spark_submit_operator.py#L37 and https://github.com/apache/incubator-airflow/blob/master/airflow/contrib/hooks/spark_submit_hook.py#L36), it provided {{hive-site.xml}} as an example. This may mislead less-informed developers into assuming that Hive configuration files can be submitted to the cluster in this manner. According to Apache Hive's documentation, hive configuration files are located in the directory located in the `HIVE_CONF_DIR` environment variable. I propose excluding this example from the docstrings. > Fix misleading SparkSubmitOperator and SparkSubmitHook docstring > > > Key: AIRFLOW-1319 > URL: https://issues.apache.org/jira/browse/AIRFLOW-1319 > Project: Apache Airflow > Issue Type: Improvement > Components: contrib >Reporter: Chris Sng >Assignee: Chris Sng >Priority: Trivial > Labels: documentation > Original Estimate: 5m > Remaining Estimate: 5m > > In the community-contributed Spark submit hook and operator, it support > {{spark-submit}} {{--files}} command line option. The {{---files}} option is > used to submit file to each executor to be used. A good example of such files > are serialized objects. > However, in both docstrings > (https://github.com/apache/incubator-airflow/blob/master/airflow/contrib/operators/spark_submit_operator.py#L37 > and > https://github.com/apache/incubator-airflow/blob/master/airflow/contrib/hooks/spark_submit_hook.py#L36), > it provided {{hive-site.xml}} as an example. This may mislead less-informed > developers into assuming that Hive configuration files can be submitted to > the cluster in this manner. According to Apache Hive's documentation, hive > configuration files are located in the directory located in the > {{HIVE_CONF_DIR}} environment variable. > I propose excluding this example from the docstrings. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (AIRFLOW-1319) Fix misleading SparkSubmitOperator and SparkSubmitHook docstring
[ https://issues.apache.org/jira/browse/AIRFLOW-1319?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Sng updated AIRFLOW-1319: --- Description: In the community-contributed Spark submit hook and operator, it support {{spark-submit}} {{--files}} command line option. The {{---files}} option is used to submit file to each executor to be used. A good example of such files are serialized objects. However, in both docstrings (https://github.com/apache/incubator-airflow/blob/master/airflow/contrib/operators/spark_submit_operator.py#L37 and https://github.com/apache/incubator-airflow/blob/master/airflow/contrib/hooks/spark_submit_hook.py#L36), it provided {{hive-site.xml}} as an example. This may mislead less-informed developers into assuming that Hive configuration files can be submitted to the cluster in this manner. According to Apache Hive's documentation, hive configuration files are located in the directory located in the `HIVE_CONF_DIR` environment variable. I propose excluding this example from the docstrings. was: In the community-contributed Spark submit hook and operator, it support {{spark-submit}}'s {{files}} command line option. The {{---files}} option is used to submit file to each executor to be used. A good example of such files are serialized objects. However, in both docstrings (https://github.com/apache/incubator-airflow/blob/master/airflow/contrib/operators/spark_submit_operator.py#L37 and https://github.com/apache/incubator-airflow/blob/master/airflow/contrib/hooks/spark_submit_hook.py#L36), it provided {{hive-site.xml}} as an example. This may mislead less-informed developers into assuming that Hive configuration files can be submitted to the cluster in this manner. According to Apache Hive's documentation, hive configuration files are located in the directory located in the `HIVE_CONF_DIR` environment variable. I propose excluding this example from the docstrings. > Fix misleading SparkSubmitOperator and SparkSubmitHook docstring > > > Key: AIRFLOW-1319 > URL: https://issues.apache.org/jira/browse/AIRFLOW-1319 > Project: Apache Airflow > Issue Type: Improvement > Components: contrib >Reporter: Chris Sng >Assignee: Chris Sng >Priority: Trivial > Labels: documentation > Original Estimate: 5m > Remaining Estimate: 5m > > In the community-contributed Spark submit hook and operator, it support > {{spark-submit}} {{--files}} command line option. The {{---files}} option is > used to submit file to each executor to be used. A good example of such files > are serialized objects. > However, in both docstrings > (https://github.com/apache/incubator-airflow/blob/master/airflow/contrib/operators/spark_submit_operator.py#L37 > and > https://github.com/apache/incubator-airflow/blob/master/airflow/contrib/hooks/spark_submit_hook.py#L36), > it provided {{hive-site.xml}} as an example. This may mislead less-informed > developers into assuming that Hive configuration files can be submitted to > the cluster in this manner. According to Apache Hive's documentation, hive > configuration files are located in the directory located in the > `HIVE_CONF_DIR` environment variable. > I propose excluding this example from the docstrings. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (AIRFLOW-1319) Fix misleading SparkSubmitOperator and SparkSubmitHook docstring
[ https://issues.apache.org/jira/browse/AIRFLOW-1319?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Sng updated AIRFLOW-1319: --- Description: In the community-contributed Spark submit hook and operator, it support {{spark-submit}}'s {{files}} command line option. The {{---files}} option is used to submit file to each executor to be used. A good example of such files are serialized objects. However, in both docstrings (https://github.com/apache/incubator-airflow/blob/master/airflow/contrib/operators/spark_submit_operator.py#L37 and https://github.com/apache/incubator-airflow/blob/master/airflow/contrib/hooks/spark_submit_hook.py#L36), it provided {{hive-site.xml}} as an example. This may mislead less-informed developers into assuming that Hive configuration files can be submitted to the cluster in this manner. According to Apache Hive's documentation, hive configuration files are located in the directory located in the `HIVE_CONF_DIR` environment variable. I propose excluding this example from the docstrings. was: In the community-contributed Spark submit hook and operator, it support {{spark-submit}}'s {{---files}} command line option. The {{---files}} option is used to submit file to each executor to be used. A good example of such files are serialized objects. However, in both docstrings (https://github.com/apache/incubator-airflow/blob/master/airflow/contrib/operators/spark_submit_operator.py#L37 and https://github.com/apache/incubator-airflow/blob/master/airflow/contrib/hooks/spark_submit_hook.py#L36), it provided {{hive-site.xml}} as an example. This may mislead less-informed developers into assuming that Hive configuration files can be submitted to the cluster in this manner. According to Apache Hive's documentation, hive configuration files are located in the directory located in the `HIVE_CONF_DIR` environment variable. I propose excluding this example from the docstrings. > Fix misleading SparkSubmitOperator and SparkSubmitHook docstring > > > Key: AIRFLOW-1319 > URL: https://issues.apache.org/jira/browse/AIRFLOW-1319 > Project: Apache Airflow > Issue Type: Improvement > Components: contrib >Reporter: Chris Sng >Assignee: Chris Sng >Priority: Trivial > Labels: documentation > Original Estimate: 5m > Remaining Estimate: 5m > > In the community-contributed Spark submit hook and operator, it support > {{spark-submit}}'s {{files}} command line option. The {{---files}} option > is used to submit file to each executor to be used. A good example of such > files are serialized objects. > However, in both docstrings > (https://github.com/apache/incubator-airflow/blob/master/airflow/contrib/operators/spark_submit_operator.py#L37 > and > https://github.com/apache/incubator-airflow/blob/master/airflow/contrib/hooks/spark_submit_hook.py#L36), > it provided {{hive-site.xml}} as an example. This may mislead less-informed > developers into assuming that Hive configuration files can be submitted to > the cluster in this manner. According to Apache Hive's documentation, hive > configuration files are located in the directory located in the > `HIVE_CONF_DIR` environment variable. > I propose excluding this example from the docstrings. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (AIRFLOW-1319) Fix misleading SparkSubmitOperator and SparkSubmitHook docstring
Chris Sng created AIRFLOW-1319: -- Summary: Fix misleading SparkSubmitOperator and SparkSubmitHook docstring Key: AIRFLOW-1319 URL: https://issues.apache.org/jira/browse/AIRFLOW-1319 Project: Apache Airflow Issue Type: Improvement Components: contrib Reporter: Chris Sng Assignee: Chris Sng Priority: Trivial In the community-contributed Spark submit hook and operator, it support `spark-submit`'s `--files` command line option. The `--files` option is used to submit file to each executor to be used. A good example of such files are serialized objects. However, in both docstrings (https://github.com/apache/incubator-airflow/blob/master/airflow/contrib/operators/spark_submit_operator.py#L37 and https://github.com/apache/incubator-airflow/blob/master/airflow/contrib/hooks/spark_submit_hook.py#L36), it provided `hive-site.xml` as an example. This may mislead less-informed developers into assuming that Hive configuration files can be submitted to the cluster in this manner. According to Apache Hive's documentation, hive configuration files are located in the directory located in the `HIVE_CONF_DIR` environment variable. I propose excluding this example from the docstrings. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (AIRFLOW-979) Add GovTech GDS
[ https://issues.apache.org/jira/browse/AIRFLOW-979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15923829#comment-15923829 ] Chris Sng commented on AIRFLOW-979: --- Please review my Pull Request #2149: https://github.com/apache/incubator-airflow/pull/2149 > Add GovTech GDS > --- > > Key: AIRFLOW-979 > URL: https://issues.apache.org/jira/browse/AIRFLOW-979 > Project: Apache Airflow > Issue Type: Wish > Components: docs >Reporter: Chris Sng >Assignee: Chris Sng >Priority: Trivial > Labels: documentation > > Add to README.md: > ``` > 1. [GovTech GDS](https://gds-gov.tech) > [[@chrissng](https://github.com/chrissng) & > [@datagovsg](https://github.com/datagovsg)] > ``` -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (AIRFLOW-979) Add GovTech GDS
[ https://issues.apache.org/jira/browse/AIRFLOW-979?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Sng updated AIRFLOW-979: -- Issue Type: Wish (was: Bug) > Add GovTech GDS > --- > > Key: AIRFLOW-979 > URL: https://issues.apache.org/jira/browse/AIRFLOW-979 > Project: Apache Airflow > Issue Type: Wish > Components: docs >Reporter: Chris Sng >Assignee: Chris Sng >Priority: Trivial > Labels: documentation > > Add to README.md: > ``` > 1. [GovTech GDS](https://gds-gov.tech) > [[@chrissng](https://github.com/chrissng) & > [@datagovsg](https://github.com/datagovsg)] > ``` -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (AIRFLOW-979) Add GovTech GDS
[ https://issues.apache.org/jira/browse/AIRFLOW-979?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Sng updated AIRFLOW-979: -- Description: Add to README.md: ``` 1. [GovTech GDS](https://gds-gov.tech) [[@chrissng](https://github.com/chrissng) & [@datagovsg](https://github.com/datagovsg)] ``` was: Add to README.md: ``` 1. [GovTech GDS](https://gds-gov.tech) [[@chrissng](https://github.com/chrissng)] ``` > Add GovTech GDS > --- > > Key: AIRFLOW-979 > URL: https://issues.apache.org/jira/browse/AIRFLOW-979 > Project: Apache Airflow > Issue Type: Bug > Components: docs >Reporter: Chris Sng >Assignee: Chris Sng >Priority: Trivial > Labels: documentation > > Add to README.md: > ``` > 1. [GovTech GDS](https://gds-gov.tech) > [[@chrissng](https://github.com/chrissng) & > [@datagovsg](https://github.com/datagovsg)] > ``` -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (AIRFLOW-979) Add GovTech GDS
Chris Sng created AIRFLOW-979: - Summary: Add GovTech GDS Key: AIRFLOW-979 URL: https://issues.apache.org/jira/browse/AIRFLOW-979 Project: Apache Airflow Issue Type: Bug Components: docs Reporter: Chris Sng Assignee: Chris Sng Priority: Trivial Add to README.md: ``` 1. [GovTech GDS](https://gds-gov.tech) [[@chrissng](https://github.com/chrissng)] ``` -- This message was sent by Atlassian JIRA (v6.3.15#6346)