[jira] [Updated] (AIRFLOW-1183) How to pass Spark URL for standalone cluster?
[ https://issues.apache.org/jira/browse/AIRFLOW-1183?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sam sen updated AIRFLOW-1183: - Description: How can I pass my Spark URL? When I look in the logs I see `--master` is pointed to "yarn." Also, the same thing for `cluster-mode`. I tried passing it within the function but I'm getting an error. *Error* {code} PendingDeprecationWarning: Invalid arguments were passed to SparkSubmitOperator. Support for passing such arguments will be dropped in Airflow 2.0. Invalid arguments were: *args: () **kwargs: {'deploy_mode': 'cluster', 'java_class': 'SimpleApp'} {code} *Code* {code} testSpark = SparkSubmitOperator( task_id='test-spark', deploy_mode='cluster', application='src/main/scala/target/scala-2.11/simple-project_2.11-1.0.jar', java_class='SimpleApp' dag=dag) {code} was: How can I pass my Spark URL? When I look in the logs I see `--master` is pointed to "yarn." Also, the same thing for `cluster-mode`. I tried passing it within the function but I'm getting an error. {quote} PendingDeprecationWarning: Invalid arguments were passed to SparkSubmitOperator. Support for passing such arguments will be dropped in Airflow 2.0. Invalid arguments were: *args: () **kwargs: {'deploy_mode': 'cluster', 'java_class': 'SimpleApp'} {quote} *Code* {code} testSpark = SparkSubmitOperator( task_id='test-spark', deploy_mode='cluster', application='src/main/scala/target/scala-2.11/simple-project_2.11-1.0.jar', java_class='SimpleApp' dag=dag) {code} > How to pass Spark URL for standalone cluster? > - > > Key: AIRFLOW-1183 > URL: https://issues.apache.org/jira/browse/AIRFLOW-1183 > Project: Apache Airflow > Issue Type: Bug > Components: operators >Affects Versions: Airflow 1.8 >Reporter: sam sen >Priority: Critical > > How can I pass my Spark URL? When I look in the logs I see `--master` is > pointed to "yarn." Also, the same thing for `cluster-mode`. I tried passing > it within the function but I'm getting an error. > *Error* > {code} > PendingDeprecationWarning: Invalid arguments were passed to > SparkSubmitOperator. Support for passing such arguments will be dropped in > Airflow 2.0. Invalid arguments were: > *args: () > **kwargs: {'deploy_mode': 'cluster', 'java_class': 'SimpleApp'} {code} > *Code* > {code} > testSpark = SparkSubmitOperator( >task_id='test-spark', > deploy_mode='cluster', > > application='src/main/scala/target/scala-2.11/simple-project_2.11-1.0.jar', > java_class='SimpleApp' >dag=dag) > {code} -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (AIRFLOW-1183) How to pass Spark URL for standalone cluster?
[ https://issues.apache.org/jira/browse/AIRFLOW-1183?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sam sen updated AIRFLOW-1183: - Description: How can I pass my Spark URL? When I look in the logs I see `--master` is pointed to "yarn." Also, the same thing for `cluster-mode`. I tried passing it within the function but I'm getting an error. *Error* {code} PendingDeprecationWarning: Invalid arguments were passed to SparkSubmitOperator. Support for passing such arguments will be dropped in Airflow 2.0. Invalid arguments were: *args: () **kwargs: {'deploy_mode': 'cluster', 'java_class': 'SimpleApp'} {code} *Code* {code} testSpark = SparkSubmitOperator( task_id='test-spark', deploy_mode='cluster', application='src/main/scala/target/scala-2.11/simple-project_2.11-1.0.jar', java_class='SimpleApp', deploy_mode='cluster', dag=dag) {code} was: How can I pass my Spark URL? When I look in the logs I see `--master` is pointed to "yarn." Also, the same thing for `cluster-mode`. I tried passing it within the function but I'm getting an error. *Error* {code} PendingDeprecationWarning: Invalid arguments were passed to SparkSubmitOperator. Support for passing such arguments will be dropped in Airflow 2.0. Invalid arguments were: *args: () **kwargs: {'deploy_mode': 'cluster', 'java_class': 'SimpleApp'} {code} *Code* {code} testSpark = SparkSubmitOperator( task_id='test-spark', deploy_mode='cluster', application='src/main/scala/target/scala-2.11/simple-project_2.11-1.0.jar', java_class='SimpleApp' dag=dag) {code} > How to pass Spark URL for standalone cluster? > - > > Key: AIRFLOW-1183 > URL: https://issues.apache.org/jira/browse/AIRFLOW-1183 > Project: Apache Airflow > Issue Type: Bug > Components: operators >Affects Versions: Airflow 1.8 >Reporter: sam sen >Priority: Critical > > How can I pass my Spark URL? When I look in the logs I see `--master` is > pointed to "yarn." Also, the same thing for `cluster-mode`. I tried passing > it within the function but I'm getting an error. > *Error* > {code} > PendingDeprecationWarning: Invalid arguments were passed to > SparkSubmitOperator. Support for passing such arguments will be dropped in > Airflow 2.0. Invalid arguments were: > *args: () > **kwargs: {'deploy_mode': 'cluster', 'java_class': 'SimpleApp'} {code} > *Code* > {code} > testSpark = SparkSubmitOperator( >task_id='test-spark', > deploy_mode='cluster', > > application='src/main/scala/target/scala-2.11/simple-project_2.11-1.0.jar', > java_class='SimpleApp', > deploy_mode='cluster', > dag=dag) > {code} -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (AIRFLOW-1183) How to pass Spark URL for standalone cluster?
[ https://issues.apache.org/jira/browse/AIRFLOW-1183?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sam sen updated AIRFLOW-1183: - Description: How can I pass my Spark URL? When I look in the logs I see `--master` is pointed to "yarn." Also, the same thing for `cluster-mode`. I tried passing it within the function but I'm getting an error. {quote} PendingDeprecationWarning: Invalid arguments were passed to SparkSubmitOperator. Support for passing such arguments will be dropped in Airflow 2.0. Invalid arguments were: *args: () **kwargs: {'deploy_mode': 'cluster', 'java_class': 'SimpleApp'} {quote} *Code* {code} testSpark = SparkSubmitOperator( task_id='test-spark', deploy_mode='cluster', application='src/main/scala/target/scala-2.11/simple-project_2.11-1.0.jar', java_class='SimpleApp' dag=dag) {code} was: How can I pass my Spark URL? When I look in the logs I see `--master` is pointed to "yarn." Also, the same thing for `cluster-mode`. I tried passing it within the function but I'm getting an error. {quote} PendingDeprecationWarning: Invalid arguments were passed to SparkSubmitOperator. Support for passing such arguments will be dropped in Airflow 2.0. Invalid arguments were: *args: () **kwargs: {'deploy_mode': 'cluster', 'java_class': 'SimpleApp'} {quote} *Code* {code} testSpark = SparkSubmitOperator( task_id='test-spark', deploy_mode='cluster', application='src/main/scala/target/scala-2.11/simple-project_2.11-1.0.jar', java_class='SimpleApp' dag=dag) {code} > How to pass Spark URL for standalone cluster? > - > > Key: AIRFLOW-1183 > URL: https://issues.apache.org/jira/browse/AIRFLOW-1183 > Project: Apache Airflow > Issue Type: Bug > Components: operators >Affects Versions: Airflow 1.8 >Reporter: sam sen >Priority: Critical > > How can I pass my Spark URL? When I look in the logs I see `--master` is > pointed to "yarn." Also, the same thing for `cluster-mode`. I tried passing > it within the function but I'm getting an error. > {quote} > PendingDeprecationWarning: Invalid arguments were passed to > SparkSubmitOperator. Support for passing such arguments will be dropped in > Airflow 2.0. Invalid arguments were: > *args: () > **kwargs: {'deploy_mode': 'cluster', 'java_class': 'SimpleApp'} {quote} > *Code* > {code} > testSpark = SparkSubmitOperator( >task_id='test-spark', > deploy_mode='cluster', > > application='src/main/scala/target/scala-2.11/simple-project_2.11-1.0.jar', > java_class='SimpleApp' >dag=dag) > {code} -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (AIRFLOW-1183) How to pass Spark URL for standalone cluster?
[ https://issues.apache.org/jira/browse/AIRFLOW-1183?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sam sen updated AIRFLOW-1183: - Description: How can I pass my Spark URL? When I look in the logs I see `--master` is pointed to "yarn." Also, the same thing for `cluster-mode`. I tried passing it within the function but I'm getting an error. {quote} PendingDeprecationWarning: Invalid arguments were passed to SparkSubmitOperator. Support for passing such arguments will be dropped in Airflow 2.0. Invalid arguments were: *args: () **kwargs: {'deploy_mode': 'cluster', 'java_class': 'SimpleApp'} {quote} *Code* {code} testSpark = SparkSubmitOperator( task_id='test-spark', deploy_mode='cluster', application='src/main/scala/target/scala-2.11/simple-project_2.11-1.0.jar', java_class='SimpleApp' dag=dag) {code} was: How can I pass my Spark URL? When I look in the logs I see `--master` is pointed to "yarn." Also, the same thing for `cluster-mode`. I tried passing it within the function but I'm getting an error. {quote} PendingDeprecationWarning: Invalid arguments were passed to SparkSubmitOperator. Support for passing such arguments will be dropped in Airflow 2.0. Invalid arguments were: *args: () **kwargs: {'deploy_mode': 'cluster', 'java_class': 'SimpleApp'} {quote} #Code {code} testSpark = SparkSubmitOperator( task_id='test-spark', deploy_mode='cluster', application='src/main/scala/target/scala-2.11/simple-project_2.11-1.0.jar', java_class='SimpleApp' dag=dag) {code} > How to pass Spark URL for standalone cluster? > - > > Key: AIRFLOW-1183 > URL: https://issues.apache.org/jira/browse/AIRFLOW-1183 > Project: Apache Airflow > Issue Type: Bug > Components: operators >Affects Versions: Airflow 1.8 >Reporter: sam sen >Priority: Critical > > How can I pass my Spark URL? When I look in the logs I see `--master` is > pointed to "yarn." Also, the same thing for `cluster-mode`. I tried passing > it within the function but I'm getting an error. > {quote} > PendingDeprecationWarning: Invalid arguments were passed to > SparkSubmitOperator. Support for passing such arguments will be dropped in > Airflow 2.0. Invalid arguments were: > *args: () > **kwargs: {'deploy_mode': 'cluster', 'java_class': 'SimpleApp'} > {quote} > *Code* > {code} > testSpark = SparkSubmitOperator( >task_id='test-spark', > deploy_mode='cluster', > > application='src/main/scala/target/scala-2.11/simple-project_2.11-1.0.jar', > java_class='SimpleApp' >dag=dag) > {code} -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (AIRFLOW-1183) How to pass Spark URL for standalone cluster?
[ https://issues.apache.org/jira/browse/AIRFLOW-1183?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sam sen updated AIRFLOW-1183: - Description: How can I pass my Spark URL? When I look in the logs I see `--master` is pointed to "yarn." Also, the same thing for `cluster-mode`. I tried passing it within the function but I'm getting an error. {quote} PendingDeprecationWarning: Invalid arguments were passed to SparkSubmitOperator. Support for passing such arguments will be dropped in Airflow 2.0. Invalid arguments were: *args: () **kwargs: {'deploy_mode': 'cluster', 'java_class': 'SimpleApp'} {quote} #Code {code} testSpark = SparkSubmitOperator( task_id='test-spark', deploy_mode='cluster', application='src/main/scala/target/scala-2.11/simple-project_2.11-1.0.jar', java_class='SimpleApp' dag=dag) {code} was: How can I pass my Spark URL? When I look in the logs I see `--master` is pointed to "yarn." Also, the same thing for `cluster-mode`. I tried passing it within the function but I'm getting an error. {quote} PendingDeprecationWarning: Invalid arguments were passed to SparkSubmitOperator. Support for passing such arguments will be dropped in Airflow 2.0. Invalid arguments were: *args: () **kwargs: {'deploy_mode': 'cluster', 'java_class': 'SimpleApp'} {quote} {code} testSpark = SparkSubmitOperator( task_id='test-spark', deploy_mode='cluster', application='src/main/scala/target/scala-2.11/simple-project_2.11-1.0.jar', java_class='SimpleApp' dag=dag) {code} > How to pass Spark URL for standalone cluster? > - > > Key: AIRFLOW-1183 > URL: https://issues.apache.org/jira/browse/AIRFLOW-1183 > Project: Apache Airflow > Issue Type: Bug > Components: operators >Affects Versions: Airflow 1.8 >Reporter: sam sen >Priority: Critical > > How can I pass my Spark URL? When I look in the logs I see `--master` is > pointed to "yarn." Also, the same thing for `cluster-mode`. I tried passing > it within the function but I'm getting an error. > {quote} > PendingDeprecationWarning: Invalid arguments were passed to > SparkSubmitOperator. Support for passing such arguments will be dropped in > Airflow 2.0. Invalid arguments were: > *args: () > **kwargs: {'deploy_mode': 'cluster', 'java_class': 'SimpleApp'} > {quote} > #Code > {code} > testSpark = SparkSubmitOperator( >task_id='test-spark', > deploy_mode='cluster', > > application='src/main/scala/target/scala-2.11/simple-project_2.11-1.0.jar', > java_class='SimpleApp' >dag=dag) > {code} -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (AIRFLOW-1183) How to pass Spark URL for standalone cluster?
[ https://issues.apache.org/jira/browse/AIRFLOW-1183?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sam sen updated AIRFLOW-1183: - Description: How can I pass my Spark URL? When I look in the logs I see `--master` is pointed to "yarn." Also, the same thing for `cluster-mode`. I tried passing it within the function but I'm getting an error. {quote} PendingDeprecationWarning: Invalid arguments were passed to SparkSubmitOperator. Support for passing such arguments will be dropped in Airflow 2.0. Invalid arguments were: *args: () **kwargs: {'deploy_mode': 'cluster', 'java_class': 'SimpleApp'} {quote} {code:python} testSpark = SparkSubmitOperator( task_id='test-spark', deploy_mode='cluster', application='src/main/scala/target/scala-2.11/simple-project_2.11-1.0.jar', java_class='SimpleApp' dag=dag) {code} was: How can I pass my Spark URL? When I look in the logs I see `--master` is pointed to "yarn." Also, the same thing for `cluster-mode`. I tried passing it within the function but I'm getting an error. {quote} PendingDeprecationWarning: Invalid arguments were passed to SparkSubmitOperator. Support for passing such arguments will be dropped in Airflow 2.0. Invalid arguments were: *args: () **kwargs: {'deploy_mode': 'cluster', 'java_class': 'SimpleApp'} {quote} #Code ``` testSpark = SparkSubmitOperator( task_id='test-spark', deploy_mode='cluster', application='src/main/scala/target/scala-2.11/simple-project_2.11-1.0.jar', java_class='SimpleApp' dag=dag) ``` > How to pass Spark URL for standalone cluster? > - > > Key: AIRFLOW-1183 > URL: https://issues.apache.org/jira/browse/AIRFLOW-1183 > Project: Apache Airflow > Issue Type: Bug > Components: operators >Affects Versions: Airflow 1.8 >Reporter: sam sen >Priority: Critical > > How can I pass my Spark URL? When I look in the logs I see `--master` is > pointed to "yarn." Also, the same thing for `cluster-mode`. I tried passing > it within the function but I'm getting an error. > {quote} > PendingDeprecationWarning: Invalid arguments were passed to > SparkSubmitOperator. Support for passing such arguments will be dropped in > Airflow 2.0. Invalid arguments were: > *args: () > **kwargs: {'deploy_mode': 'cluster', 'java_class': 'SimpleApp'} > {quote} > {code:python} > testSpark = SparkSubmitOperator( >task_id='test-spark', > deploy_mode='cluster', > > application='src/main/scala/target/scala-2.11/simple-project_2.11-1.0.jar', > java_class='SimpleApp' >dag=dag) > {code} -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (AIRFLOW-1183) How to pass Spark URL for standalone cluster?
[ https://issues.apache.org/jira/browse/AIRFLOW-1183?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sam sen updated AIRFLOW-1183: - Description: How can I pass my Spark URL? When I look in the logs I see `--master` is pointed to "yarn." Also, the same thing for `cluster-mode`. I tried passing it within the function but I'm getting an error. {quote} PendingDeprecationWarning: Invalid arguments were passed to SparkSubmitOperator. Support for passing such arguments will be dropped in Airflow 2.0. Invalid arguments were: *args: () **kwargs: {'deploy_mode': 'cluster', 'java_class': 'SimpleApp'} {quote} {code} testSpark = SparkSubmitOperator( task_id='test-spark', deploy_mode='cluster', application='src/main/scala/target/scala-2.11/simple-project_2.11-1.0.jar', java_class='SimpleApp' dag=dag) {code} was: How can I pass my Spark URL? When I look in the logs I see `--master` is pointed to "yarn." Also, the same thing for `cluster-mode`. I tried passing it within the function but I'm getting an error. {quote} PendingDeprecationWarning: Invalid arguments were passed to SparkSubmitOperator. Support for passing such arguments will be dropped in Airflow 2.0. Invalid arguments were: *args: () **kwargs: {'deploy_mode': 'cluster', 'java_class': 'SimpleApp'} {quote} {code:python} testSpark = SparkSubmitOperator( task_id='test-spark', deploy_mode='cluster', application='src/main/scala/target/scala-2.11/simple-project_2.11-1.0.jar', java_class='SimpleApp' dag=dag) {code} > How to pass Spark URL for standalone cluster? > - > > Key: AIRFLOW-1183 > URL: https://issues.apache.org/jira/browse/AIRFLOW-1183 > Project: Apache Airflow > Issue Type: Bug > Components: operators >Affects Versions: Airflow 1.8 >Reporter: sam sen >Priority: Critical > > How can I pass my Spark URL? When I look in the logs I see `--master` is > pointed to "yarn." Also, the same thing for `cluster-mode`. I tried passing > it within the function but I'm getting an error. > {quote} > PendingDeprecationWarning: Invalid arguments were passed to > SparkSubmitOperator. Support for passing such arguments will be dropped in > Airflow 2.0. Invalid arguments were: > *args: () > **kwargs: {'deploy_mode': 'cluster', 'java_class': 'SimpleApp'} > {quote} > {code} > testSpark = SparkSubmitOperator( >task_id='test-spark', > deploy_mode='cluster', > > application='src/main/scala/target/scala-2.11/simple-project_2.11-1.0.jar', > java_class='SimpleApp' >dag=dag) > {code} -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (AIRFLOW-1183) How to pass Spark URL for standalone cluster?
[ https://issues.apache.org/jira/browse/AIRFLOW-1183?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sam sen updated AIRFLOW-1183: - Description: How can I pass my Spark URL? When I look in the logs I see `--master` is pointed to "yarn." Also, the same thing for `cluster-mode`. I tried passing it within the function but I'm getting an error. bq. PendingDeprecationWarning: Invalid arguments were passed to SparkSubmitOperator. Support for passing such arguments will be dropped in Airflow 2.0. Invalid arguments were: *args: () **kwargs: {'deploy_mode': 'cluster', 'java_class': 'SimpleApp'} #Code ``` testSpark = SparkSubmitOperator( task_id='test-spark', deploy_mode='cluster', application='src/main/scala/target/scala-2.11/simple-project_2.11-1.0.jar', java_class='SimpleApp' dag=dag) ``` was: How can I pass my Spark URL? When I look in the logs I see `--master` is pointed to "yarn." Also, the same thing for `cluster-mode`. I tried passing it within the function but I'm getting an error. bq. PendingDeprecationWarning: Invalid arguments were passed to SparkSubmitOperator. Support for passing such arguments will be dropped in Airflow 2.0. Invalid arguments were: bq. *args: () bq. **kwargs: {'deploy_mode': 'cluster', 'java_class': 'SimpleApp'} #Code ``` testSpark = SparkSubmitOperator( task_id='test-spark', deploy_mode='cluster', application='src/main/scala/target/scala-2.11/simple-project_2.11-1.0.jar', java_class='SimpleApp' dag=dag) ``` > How to pass Spark URL for standalone cluster? > - > > Key: AIRFLOW-1183 > URL: https://issues.apache.org/jira/browse/AIRFLOW-1183 > Project: Apache Airflow > Issue Type: Bug > Components: operators >Affects Versions: Airflow 1.8 >Reporter: sam sen >Priority: Critical > > How can I pass my Spark URL? When I look in the logs I see `--master` is > pointed to "yarn." Also, the same thing for `cluster-mode`. I tried passing > it within the function but I'm getting an error. > bq. PendingDeprecationWarning: Invalid arguments were passed to > SparkSubmitOperator. Support for passing such arguments will be dropped in > Airflow 2.0. Invalid arguments were: > *args: () > **kwargs: {'deploy_mode': 'cluster', 'java_class': 'SimpleApp'} > #Code > ``` > testSpark = SparkSubmitOperator( >task_id='test-spark', > deploy_mode='cluster', > > application='src/main/scala/target/scala-2.11/simple-project_2.11-1.0.jar', > java_class='SimpleApp' >dag=dag) > ``` -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (AIRFLOW-1183) How to pass Spark URL for standalone cluster?
[ https://issues.apache.org/jira/browse/AIRFLOW-1183?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sam sen updated AIRFLOW-1183: - Description: How can I pass my Spark URL? When I look in the logs I see `--master` is pointed to "yarn." Also, the same thing for `cluster-mode`. I tried passing it within the function but I'm getting an error. {quote} PendingDeprecationWarning: Invalid arguments were passed to SparkSubmitOperator. Support for passing such arguments will be dropped in Airflow 2.0. Invalid arguments were: *args: () **kwargs: {'deploy_mode': 'cluster', 'java_class': 'SimpleApp'} {quote} #Code ``` testSpark = SparkSubmitOperator( task_id='test-spark', deploy_mode='cluster', application='src/main/scala/target/scala-2.11/simple-project_2.11-1.0.jar', java_class='SimpleApp' dag=dag) ``` was: How can I pass my Spark URL? When I look in the logs I see `--master` is pointed to "yarn." Also, the same thing for `cluster-mode`. I tried passing it within the function but I'm getting an error. bq. PendingDeprecationWarning: Invalid arguments were passed to SparkSubmitOperator. Support for passing such arguments will be dropped in Airflow 2.0. Invalid arguments were: *args: () **kwargs: {'deploy_mode': 'cluster', 'java_class': 'SimpleApp'} #Code ``` testSpark = SparkSubmitOperator( task_id='test-spark', deploy_mode='cluster', application='src/main/scala/target/scala-2.11/simple-project_2.11-1.0.jar', java_class='SimpleApp' dag=dag) ``` > How to pass Spark URL for standalone cluster? > - > > Key: AIRFLOW-1183 > URL: https://issues.apache.org/jira/browse/AIRFLOW-1183 > Project: Apache Airflow > Issue Type: Bug > Components: operators >Affects Versions: Airflow 1.8 >Reporter: sam sen >Priority: Critical > > How can I pass my Spark URL? When I look in the logs I see `--master` is > pointed to "yarn." Also, the same thing for `cluster-mode`. I tried passing > it within the function but I'm getting an error. > {quote} > PendingDeprecationWarning: Invalid arguments were passed to > SparkSubmitOperator. Support for passing such arguments will be dropped in > Airflow 2.0. Invalid arguments were: > *args: () > **kwargs: {'deploy_mode': 'cluster', 'java_class': 'SimpleApp'} > {quote} > #Code > ``` > testSpark = SparkSubmitOperator( >task_id='test-spark', > deploy_mode='cluster', > > application='src/main/scala/target/scala-2.11/simple-project_2.11-1.0.jar', > java_class='SimpleApp' >dag=dag) > ``` -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (AIRFLOW-1183) How to pass Spark URL for standalone cluster?
[ https://issues.apache.org/jira/browse/AIRFLOW-1183?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sam sen updated AIRFLOW-1183: - Description: How can I pass my Spark URL? When I look in the logs I see `--master` is pointed to "yarn." Also, the same thing for `cluster-mode`. I tried passing it within the function but I'm getting an error. bq. PendingDeprecationWarning: Invalid arguments were passed to SparkSubmitOperator. Support for passing such arguments will be dropped in Airflow 2.0. Invalid arguments were: *args: () **kwargs: {'deploy_mode': 'cluster', 'java_class': 'SimpleApp'} #Code ``` testSpark = SparkSubmitOperator( task_id='test-spark', deploy_mode='cluster', application='src/main/scala/target/scala-2.11/simple-project_2.11-1.0.jar', java_class='SimpleApp' dag=dag) ``` was: How can I pass my Spark URL? When I look in the logs I see `--master` is pointed to "yarn." Also, the same thing for `cluster-mode`. I tried passing it within the function but I'm getting an error. ``` PendingDeprecationWarning: Invalid arguments were passed to SparkSubmitOperator. Support for passing such arguments will be dropped in Airflow 2.0. Invalid arguments were: *args: () **kwargs: {'deploy_mode': 'cluster', 'java_class': 'SimpleApp'}``` #Code ``` testSpark = SparkSubmitOperator( task_id='test-spark', deploy_mode='cluster', application='src/main/scala/target/scala-2.11/simple-project_2.11-1.0.jar', java_class='SimpleApp' dag=dag) ``` > How to pass Spark URL for standalone cluster? > - > > Key: AIRFLOW-1183 > URL: https://issues.apache.org/jira/browse/AIRFLOW-1183 > Project: Apache Airflow > Issue Type: Bug > Components: operators >Affects Versions: Airflow 1.8 >Reporter: sam sen >Priority: Critical > > How can I pass my Spark URL? When I look in the logs I see `--master` is > pointed to "yarn." Also, the same thing for `cluster-mode`. I tried passing > it within the function but I'm getting an error. > bq. PendingDeprecationWarning: Invalid arguments were passed to > SparkSubmitOperator. Support for passing such arguments will be dropped in > Airflow 2.0. Invalid arguments were: > *args: () > **kwargs: {'deploy_mode': 'cluster', 'java_class': 'SimpleApp'} > #Code > ``` > testSpark = SparkSubmitOperator( >task_id='test-spark', > deploy_mode='cluster', > > application='src/main/scala/target/scala-2.11/simple-project_2.11-1.0.jar', > java_class='SimpleApp' >dag=dag) > ``` -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (AIRFLOW-1183) How to pass Spark URL for standalone cluster?
[ https://issues.apache.org/jira/browse/AIRFLOW-1183?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sam sen updated AIRFLOW-1183: - Description: How can I pass my Spark URL? When I look in the logs I see `--master` is pointed to "yarn." Also, the same thing for `cluster-mode`. I tried passing it within the function but I'm getting an error. bq. PendingDeprecationWarning: Invalid arguments were passed to SparkSubmitOperator. Support for passing such arguments will be dropped in Airflow 2.0. Invalid arguments were: bq. *args: () bq. **kwargs: {'deploy_mode': 'cluster', 'java_class': 'SimpleApp'} #Code ``` testSpark = SparkSubmitOperator( task_id='test-spark', deploy_mode='cluster', application='src/main/scala/target/scala-2.11/simple-project_2.11-1.0.jar', java_class='SimpleApp' dag=dag) ``` was: How can I pass my Spark URL? When I look in the logs I see `--master` is pointed to "yarn." Also, the same thing for `cluster-mode`. I tried passing it within the function but I'm getting an error. bq. PendingDeprecationWarning: Invalid arguments were passed to SparkSubmitOperator. Support for passing such arguments will be dropped in Airflow 2.0. Invalid arguments were: *args: () **kwargs: {'deploy_mode': 'cluster', 'java_class': 'SimpleApp'} #Code ``` testSpark = SparkSubmitOperator( task_id='test-spark', deploy_mode='cluster', application='src/main/scala/target/scala-2.11/simple-project_2.11-1.0.jar', java_class='SimpleApp' dag=dag) ``` > How to pass Spark URL for standalone cluster? > - > > Key: AIRFLOW-1183 > URL: https://issues.apache.org/jira/browse/AIRFLOW-1183 > Project: Apache Airflow > Issue Type: Bug > Components: operators >Affects Versions: Airflow 1.8 >Reporter: sam sen >Priority: Critical > > How can I pass my Spark URL? When I look in the logs I see `--master` is > pointed to "yarn." Also, the same thing for `cluster-mode`. I tried passing > it within the function but I'm getting an error. > bq. PendingDeprecationWarning: Invalid arguments were passed to > SparkSubmitOperator. Support for passing such arguments will be dropped in > Airflow 2.0. Invalid arguments were: > bq. *args: () > bq. **kwargs: {'deploy_mode': 'cluster', 'java_class': 'SimpleApp'} > #Code > ``` > testSpark = SparkSubmitOperator( >task_id='test-spark', > deploy_mode='cluster', > > application='src/main/scala/target/scala-2.11/simple-project_2.11-1.0.jar', > java_class='SimpleApp' >dag=dag) > ``` -- This message was sent by Atlassian JIRA (v6.3.15#6346)