[jira] [Commented] (AIRFLOW-4289) spark_binary argument in SparkSubmitHook is ignored when building the connection_cmd
[ https://issues.apache.org/jira/browse/AIRFLOW-4289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16897020#comment-16897020 ] ASF subversion and git services commented on AIRFLOW-4289: -- Commit 8be59fb4edf0f2a132b13d0ffd1df0b8908191ab in airflow's branch refs/heads/v1-10-stable from eladkal [ https://gitbox.apache.org/repos/asf?p=airflow.git;h=8be59fb ] [AIRFLOW-4289] fix spark_binary argument being ignored in SparkSubmitHook (#5564) (cherry picked from commit 399ddb139652fee1bcefe2be09831e71bf5fa832) > spark_binary argument in SparkSubmitHook is ignored when building the > connection_cmd > > > Key: AIRFLOW-4289 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4289 > Project: Apache Airflow > Issue Type: Bug > Components: contrib, hooks >Affects Versions: 1.10.3 >Reporter: Luiz Svoboda >Assignee: Elad >Priority: Minor > Labels: usability > Fix For: 1.10.4 > > > When using the SparkSubmitOperator, although it is possible to specify the > parameter _spark_binary_, its value is ignored during the creation of the > _connection_cmd_. Instead, the value used for this property is extracted from > the connection parameters, or it defaults to _spark-submit_ as can be seen in > [spark_submit_hook > line:190|https://github.com/apache/airflow/blob/1.10.3/airflow/contrib/hooks/spark_submit_hook.py#L190] > Actually, this configuration is a bit confusing as the user can configure it > via _connection_ or directly when creating the operator instance. I suggest > keeping only one option, and in this case, [IMHO] the connection approach > seems to be better as it is already used to configure some other options. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Commented] (AIRFLOW-4289) spark_binary argument in SparkSubmitHook is ignored when building the connection_cmd
[ https://issues.apache.org/jira/browse/AIRFLOW-4289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16896919#comment-16896919 ] ASF subversion and git services commented on AIRFLOW-4289: -- Commit 8be59fb4edf0f2a132b13d0ffd1df0b8908191ab in airflow's branch refs/heads/v1-10-test from eladkal [ https://gitbox.apache.org/repos/asf?p=airflow.git;h=8be59fb ] [AIRFLOW-4289] fix spark_binary argument being ignored in SparkSubmitHook (#5564) (cherry picked from commit 399ddb139652fee1bcefe2be09831e71bf5fa832) > spark_binary argument in SparkSubmitHook is ignored when building the > connection_cmd > > > Key: AIRFLOW-4289 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4289 > Project: Apache Airflow > Issue Type: Bug > Components: contrib, hooks >Affects Versions: 1.10.3 >Reporter: Luiz Svoboda >Assignee: Elad >Priority: Minor > Labels: usability > Fix For: 1.10.4 > > > When using the SparkSubmitOperator, although it is possible to specify the > parameter _spark_binary_, its value is ignored during the creation of the > _connection_cmd_. Instead, the value used for this property is extracted from > the connection parameters, or it defaults to _spark-submit_ as can be seen in > [spark_submit_hook > line:190|https://github.com/apache/airflow/blob/1.10.3/airflow/contrib/hooks/spark_submit_hook.py#L190] > Actually, this configuration is a bit confusing as the user can configure it > via _connection_ or directly when creating the operator instance. I suggest > keeping only one option, and in this case, [IMHO] the connection approach > seems to be better as it is already used to configure some other options. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Commented] (AIRFLOW-4289) spark_binary argument in SparkSubmitHook is ignored when building the connection_cmd
[ https://issues.apache.org/jira/browse/AIRFLOW-4289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16896904#comment-16896904 ] ASF subversion and git services commented on AIRFLOW-4289: -- Commit 399ddb139652fee1bcefe2be09831e71bf5fa832 in airflow's branch refs/heads/master from eladkal [ https://gitbox.apache.org/repos/asf?p=airflow.git;h=399ddb1 ] [AIRFLOW-4289] fix spark_binary argument being ignored in SparkSubmitHook (#5564) > spark_binary argument in SparkSubmitHook is ignored when building the > connection_cmd > > > Key: AIRFLOW-4289 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4289 > Project: Apache Airflow > Issue Type: Bug > Components: contrib, hooks >Affects Versions: 1.10.3 >Reporter: Luiz Svoboda >Assignee: Elad >Priority: Minor > Labels: usability > Fix For: 1.10.4 > > > When using the SparkSubmitOperator, although it is possible to specify the > parameter _spark_binary_, its value is ignored during the creation of the > _connection_cmd_. Instead, the value used for this property is extracted from > the connection parameters, or it defaults to _spark-submit_ as can be seen in > [spark_submit_hook > line:190|https://github.com/apache/airflow/blob/1.10.3/airflow/contrib/hooks/spark_submit_hook.py#L190] > Actually, this configuration is a bit confusing as the user can configure it > via _connection_ or directly when creating the operator instance. I suggest > keeping only one option, and in this case, [IMHO] the connection approach > seems to be better as it is already used to configure some other options. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Commented] (AIRFLOW-4289) spark_binary argument in SparkSubmitHook is ignored when building the connection_cmd
[ https://issues.apache.org/jira/browse/AIRFLOW-4289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16896903#comment-16896903 ] ASF GitHub Bot commented on AIRFLOW-4289: - ashb commented on pull request #5564: [AIRFLOW-4289] fix spark_binary argument being ignored in SparkSubmit… URL: https://github.com/apache/airflow/pull/5564 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > spark_binary argument in SparkSubmitHook is ignored when building the > connection_cmd > > > Key: AIRFLOW-4289 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4289 > Project: Apache Airflow > Issue Type: Bug > Components: contrib, hooks >Affects Versions: 1.10.3 >Reporter: Luiz Svoboda >Assignee: Elad >Priority: Minor > Labels: usability > Fix For: 1.10.4 > > > When using the SparkSubmitOperator, although it is possible to specify the > parameter _spark_binary_, its value is ignored during the creation of the > _connection_cmd_. Instead, the value used for this property is extracted from > the connection parameters, or it defaults to _spark-submit_ as can be seen in > [spark_submit_hook > line:190|https://github.com/apache/airflow/blob/1.10.3/airflow/contrib/hooks/spark_submit_hook.py#L190] > Actually, this configuration is a bit confusing as the user can configure it > via _connection_ or directly when creating the operator instance. I suggest > keeping only one option, and in this case, [IMHO] the connection approach > seems to be better as it is already used to configure some other options. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Commented] (AIRFLOW-4289) spark_binary argument in SparkSubmitHook is ignored when building the connection_cmd
[ https://issues.apache.org/jira/browse/AIRFLOW-4289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16883319#comment-16883319 ] ASF GitHub Bot commented on AIRFLOW-4289: - eladkal commented on pull request #5564: [AIRFLOW-4289] fix spark_binary argument being ignored in SparkSubmit… URL: https://github.com/apache/airflow/pull/5564 …Hook ### Jira https://issues.apache.org/jira/browse/AIRFLOW-4289 ### Description Fix bug where the spark_binary argument in the constructor of SparkSubmitHook was overwritten by spark Connection. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > spark_binary argument in SparkSubmitHook is ignored when building the > connection_cmd > > > Key: AIRFLOW-4289 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4289 > Project: Apache Airflow > Issue Type: Bug > Components: contrib, hooks >Affects Versions: 1.10.3 >Reporter: Luiz Svoboda >Priority: Minor > Labels: usability > Fix For: 1.10.4 > > > When using the SparkSubmitOperator, although it is possible to specify the > parameter _spark_binary_, its value is ignored during the creation of the > _connection_cmd_. Instead, the value used for this property is extracted from > the connection parameters, or it defaults to _spark-submit_ as can be seen in > [spark_submit_hook > line:190|https://github.com/apache/airflow/blob/1.10.3/airflow/contrib/hooks/spark_submit_hook.py#L190] > Actually, this configuration is a bit confusing as the user can configure it > via _connection_ or directly when creating the operator instance. I suggest > keeping only one option, and in this case, [IMHO] the connection approach > seems to be better as it is already used to configure some other options. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
