[jira] [Created] (AIRFLOW-3235) Add list function in AzureDataLakeHook
Brandon Kvarda created AIRFLOW-3235: --- Summary: Add list function in AzureDataLakeHook Key: AIRFLOW-3235 URL: https://issues.apache.org/jira/browse/AIRFLOW-3235 Project: Apache Airflow Issue Type: Improvement Components: hooks Reporter: Brandon Kvarda Assignee: Brandon Kvarda AzureDataLakeHook could use a list function similar to those of both the GCS and S3 hooks. This could be the basis for an ADLS list operator (again, which both GCS and S3 have). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (AIRFLOW-3233) Dag deletion in the UI doesn't work
[ https://issues.apache.org/jira/browse/AIRFLOW-3233?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dan Davydov updated AIRFLOW-3233: - Affects Version/s: 1.10.0 > Dag deletion in the UI doesn't work > --- > > Key: AIRFLOW-3233 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3233 > Project: Apache Airflow > Issue Type: Bug > Components: webserver >Affects Versions: 1.10.0 >Reporter: Dan Davydov >Assignee: Dan Davydov >Priority: Major > > Dag deletion in the UI doesn't work, DAGs can only be deleted if the DAG > doesn't exist in the DAGBag, but if the DAG doesn't exist in the DAGBag the > deletion URL gets passed an empty DAG id. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] codecov-io commented on issue #4069: [AIRFLOW-3233] Fix deletion of DAGs in the UI
codecov-io commented on issue #4069: [AIRFLOW-3233] Fix deletion of DAGs in the UI URL: https://github.com/apache/incubator-airflow/pull/4069#issuecomment-431490567 # [Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4069?src=pr=h1) Report > Merging [#4069](https://codecov.io/gh/apache/incubator-airflow/pull/4069?src=pr=desc) into [master](https://codecov.io/gh/apache/incubator-airflow/commit/f5e3b03aa6b902898148e88fb9d90ecebcadc226?src=pr=desc) will **not change** coverage. > The diff coverage is `n/a`. [](https://codecov.io/gh/apache/incubator-airflow/pull/4069?src=pr=tree) ```diff @@ Coverage Diff @@ ## master#4069 +/- ## === Coverage 75.91% 75.91% === Files 199 199 Lines 1596115961 === Hits1211712117 Misses 3844 3844 ``` -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4069?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/4069?src=pr=footer). Last update [f5e3b03...71bb5c2](https://codecov.io/gh/apache/incubator-airflow/pull/4069?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-3233) Dag deletion in the UI doesn't work
[
https://issues.apache.org/jira/browse/AIRFLOW-3233?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16657329#comment-16657329
]
ASF GitHub Bot commented on AIRFLOW-3233:
-
aoen opened a new pull request #4069: [AIRFLOW-3233] Fix deletion of DAGs in
the UI
URL: https://github.com/apache/incubator-airflow/pull/4069
Make sure you have checked _all_ steps below.
### Jira
- [X] My PR addresses the following [Airflow
Jira](https://issues.apache.org/jira/browse/AIRFLOW-3233) issues and references
them in the PR title.
### Description
- [X] Here are some details about my PR, including screenshots of any UI
changes:
Currently deleting DAGs that exist on the scheduler but not the webserver is
not possible because the DAG id in the deletion URL comes from the webserver
DAG object.
e.g. http://localhost:8080/delete?dag_id= instead of
http://localhost:8080/delete?dag_id=example_kubernetes_executor
### Tests
- [X] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
UI Fix, tested both RBAC and regular.
### Commits
- [X] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [X] In case of new functionality, my PR adds documentation that describes
how to use it.
- When adding new operators/hooks/sensors, the autoclass documentation
generation needs to be added.
### Code Quality
- [X] Passes `flake8`
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Dag deletion in the UI doesn't work
> ---
>
> Key: AIRFLOW-3233
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3233
> Project: Apache Airflow
> Issue Type: Bug
> Components: webserver
>Reporter: Dan Davydov
>Assignee: Dan Davydov
>Priority: Major
>
> Dag deletion in the UI doesn't work, DAGs can only be deleted if the DAG
> doesn't exist in the DAGBag, but if the DAG doesn't exist in the DAGBag the
> deletion URL gets passed an empty DAG id.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] aoen opened a new pull request #4069: [AIRFLOW-3233] Fix deletion of DAGs in the UI
aoen opened a new pull request #4069: [AIRFLOW-3233] Fix deletion of DAGs in
the UI
URL: https://github.com/apache/incubator-airflow/pull/4069
Make sure you have checked _all_ steps below.
### Jira
- [X] My PR addresses the following [Airflow
Jira](https://issues.apache.org/jira/browse/AIRFLOW-3233) issues and references
them in the PR title.
### Description
- [X] Here are some details about my PR, including screenshots of any UI
changes:
Currently deleting DAGs that exist on the scheduler but not the webserver is
not possible because the DAG id in the deletion URL comes from the webserver
DAG object.
e.g. http://localhost:8080/delete?dag_id= instead of
http://localhost:8080/delete?dag_id=example_kubernetes_executor
### Tests
- [X] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
UI Fix, tested both RBAC and regular.
### Commits
- [X] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [X] In case of new functionality, my PR adds documentation that describes
how to use it.
- When adding new operators/hooks/sensors, the autoclass documentation
generation needs to be added.
### Code Quality
- [X] Passes `flake8`
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Commented] (AIRFLOW-1788) EMR hook doesn't pass through SecurityConfiguration parameter
[
https://issues.apache.org/jira/browse/AIRFLOW-1788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16657234#comment-16657234
]
ASF GitHub Bot commented on AIRFLOW-1788:
-
ashb closed pull request #2760: [AIRFLOW-1788] Support all boto kwargs
URL: https://github.com/apache/incubator-airflow/pull/2760
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git a/airflow/contrib/hooks/emr_hook.py
b/airflow/contrib/hooks/emr_hook.py
index cee239867c..2acb33451e 100644
--- a/airflow/contrib/hooks/emr_hook.py
+++ b/airflow/contrib/hooks/emr_hook.py
@@ -56,7 +56,8 @@ def create_job_flow(self, job_flow_overrides):
VisibleToAllUsers=config.get('VisibleToAllUsers'),
JobFlowRole=config.get('JobFlowRole'),
ServiceRole=config.get('ServiceRole'),
-Tags=config.get('Tags')
+Tags=config.get('Tags'),
+SecurityConfiguration=config.get('SecurityConfiguration'),
)
return response
diff --git a/tests/contrib/hooks/test_emr_hook.py
b/tests/contrib/hooks/test_emr_hook.py
index 119df99a7c..ab1387e52a 100644
--- a/tests/contrib/hooks/test_emr_hook.py
+++ b/tests/contrib/hooks/test_emr_hook.py
@@ -48,6 +48,19 @@ def
test_create_job_flow_uses_the_emr_config_to_create_a_cluster(self):
cluster = hook.create_job_flow({'Name': 'test_cluster'})
self.assertEqual(client.list_clusters()['Clusters'][0]['Id'],
cluster['JobFlowId'])
+
+@unittest.skipIf(mock_emr is None, 'mock_emr package not present')
+@mock_emr
+def test_create_job_flow_passes_through_boto_kwargs(self):
+client = boto3.client('emr', region_name='us-east-1')
+if len(client.list_clusters()['Clusters']):
+raise ValueError('AWS not properly mocked')
+
+hook = EmrHook(aws_conn_id='aws_default', emr_conn_id='emr_default')
+cluster = hook.create_job_flow({'Name': 'test_cluster',
+'SecurityConfiguration':
'test_config'})
+
+self.assertEqual(client.list_clusters()['Clusters'][0]['Id'],
cluster['JobFlowId'])
if __name__ == '__main__':
unittest.main()
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> EMR hook doesn't pass through SecurityConfiguration parameter
> -
>
> Key: AIRFLOW-1788
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1788
> Project: Apache Airflow
> Issue Type: Bug
>Reporter: Sam Kitajima-Kimbrel
>Assignee: Sam Kitajima-Kimbrel
>Priority: Major
>
> Expected: `SecurityConfiguration` argument to boto3's run_job_flow() method
> are passed through via the EMR hook in Airflow.
> Actual: Not passed through because hard-coded parameter names.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] ashb closed pull request #2760: [AIRFLOW-1788] Support all boto kwargs
ashb closed pull request #2760: [AIRFLOW-1788] Support all boto kwargs
URL: https://github.com/apache/incubator-airflow/pull/2760
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git a/airflow/contrib/hooks/emr_hook.py
b/airflow/contrib/hooks/emr_hook.py
index cee239867c..2acb33451e 100644
--- a/airflow/contrib/hooks/emr_hook.py
+++ b/airflow/contrib/hooks/emr_hook.py
@@ -56,7 +56,8 @@ def create_job_flow(self, job_flow_overrides):
VisibleToAllUsers=config.get('VisibleToAllUsers'),
JobFlowRole=config.get('JobFlowRole'),
ServiceRole=config.get('ServiceRole'),
-Tags=config.get('Tags')
+Tags=config.get('Tags'),
+SecurityConfiguration=config.get('SecurityConfiguration'),
)
return response
diff --git a/tests/contrib/hooks/test_emr_hook.py
b/tests/contrib/hooks/test_emr_hook.py
index 119df99a7c..ab1387e52a 100644
--- a/tests/contrib/hooks/test_emr_hook.py
+++ b/tests/contrib/hooks/test_emr_hook.py
@@ -48,6 +48,19 @@ def
test_create_job_flow_uses_the_emr_config_to_create_a_cluster(self):
cluster = hook.create_job_flow({'Name': 'test_cluster'})
self.assertEqual(client.list_clusters()['Clusters'][0]['Id'],
cluster['JobFlowId'])
+
+@unittest.skipIf(mock_emr is None, 'mock_emr package not present')
+@mock_emr
+def test_create_job_flow_passes_through_boto_kwargs(self):
+client = boto3.client('emr', region_name='us-east-1')
+if len(client.list_clusters()['Clusters']):
+raise ValueError('AWS not properly mocked')
+
+hook = EmrHook(aws_conn_id='aws_default', emr_conn_id='emr_default')
+cluster = hook.create_job_flow({'Name': 'test_cluster',
+'SecurityConfiguration':
'test_config'})
+
+self.assertEqual(client.list_clusters()['Clusters'][0]['Id'],
cluster['JobFlowId'])
if __name__ == '__main__':
unittest.main()
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] ashb commented on issue #2760: [AIRFLOW-1788] Support all boto kwargs
ashb commented on issue #2760: [AIRFLOW-1788] Support all boto kwargs URL: https://github.com/apache/incubator-airflow/pull/2760#issuecomment-431451902 Closed by #4044 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Resolved] (AIRFLOW-1788) EMR hook doesn't pass through SecurityConfiguration parameter
[ https://issues.apache.org/jira/browse/AIRFLOW-1788?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ash Berlin-Taylor resolved AIRFLOW-1788. Resolution: Fixed AIRFLOW-3197 allows this (and any other params in the future) > EMR hook doesn't pass through SecurityConfiguration parameter > - > > Key: AIRFLOW-1788 > URL: https://issues.apache.org/jira/browse/AIRFLOW-1788 > Project: Apache Airflow > Issue Type: Bug >Reporter: Sam Kitajima-Kimbrel >Assignee: Sam Kitajima-Kimbrel >Priority: Major > > Expected: `SecurityConfiguration` argument to boto3's run_job_flow() method > are passed through via the EMR hook in Airflow. > Actual: Not passed through because hard-coded parameter names. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] wmorris75 commented on issue #3828: [AIRFLOW-2993] s3_to_sftp and sftp_to_s3 operators
wmorris75 commented on issue #3828: [AIRFLOW-2993] s3_to_sftp and sftp_to_s3 operators URL: https://github.com/apache/incubator-airflow/pull/3828#issuecomment-431442228 Changes have been made as recommended and now awaiting final review. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Assigned] (AIRFLOW-3234) Enable alerting for dagbag import errors
[ https://issues.apache.org/jira/browse/AIRFLOW-3234?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Adam Boscarino reassigned AIRFLOW-3234: --- Assignee: Adam Boscarino > Enable alerting for dagbag import errors > > > Key: AIRFLOW-3234 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3234 > Project: Apache Airflow > Issue Type: Improvement >Reporter: Adam Boscarino >Assignee: Adam Boscarino >Priority: Minor > > If a task fails due to being unable to load the DagBag it is set to `failed` > without being able to use callbacks or retries. This creates the possibility > for "silent" failures. We should have the ability to handle these failures > with some other functionality (similar to the SLA callbacks). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (AIRFLOW-3234) Enable alerting for dagbag import errors
Adam Boscarino created AIRFLOW-3234: --- Summary: Enable alerting for dagbag import errors Key: AIRFLOW-3234 URL: https://issues.apache.org/jira/browse/AIRFLOW-3234 Project: Apache Airflow Issue Type: Improvement Reporter: Adam Boscarino If a task fails due to being unable to load the DagBag it is set to `failed` without being able to use callbacks or retries. This creates the possibility for "silent" failures. We should have the ability to handle these failures with some other functionality (similar to the SLA callbacks). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (AIRFLOW-3233) Dag deletion in the UI doesn't work
Dan Davydov created AIRFLOW-3233: Summary: Dag deletion in the UI doesn't work Key: AIRFLOW-3233 URL: https://issues.apache.org/jira/browse/AIRFLOW-3233 Project: Apache Airflow Issue Type: Bug Components: webserver Reporter: Dan Davydov Assignee: Dan Davydov Dag deletion in the UI doesn't work, DAGs can only be deleted if the DAG doesn't exist in the DAGBag, but if the DAG doesn't exist in the DAGBag the deletion URL gets passed an empty DAG id. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] oelesinsc24 commented on issue #4068: [AIRFLOW-2310]: Add AWS Glue Job Compatibility to Airflow
oelesinsc24 commented on issue #4068: [AIRFLOW-2310]: Add AWS Glue Job Compatibility to Airflow URL: https://github.com/apache/incubator-airflow/pull/4068#issuecomment-431382565 @Fokko, can you please help look at this? Some Kubernetes tests seem to be failing and they are not related to my commits, precisely, I have not had anything to do with them This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] codecov-io edited a comment on issue #3650: [AIRFLOW-2703] exceptions from scheduler's heartbeat is handled so that scheduler does not crash
codecov-io edited a comment on issue #3650: [AIRFLOW-2703] exceptions from scheduler's heartbeat is handled so that scheduler does not crash URL: https://github.com/apache/incubator-airflow/pull/3650#issuecomment-408336130 # [Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/3650?src=pr=h1) Report > Merging [#3650](https://codecov.io/gh/apache/incubator-airflow/pull/3650?src=pr=desc) into [master](https://codecov.io/gh/apache/incubator-airflow/commit/b8be322d3badfeadfa8f08e0bf92a12a6cd26418?src=pr=desc) will **increase** coverage by `1.87%`. > The diff coverage is `85%`. [](https://codecov.io/gh/apache/incubator-airflow/pull/3650?src=pr=tree) ```diff @@Coverage Diff @@ ## master#3650 +/- ## == + Coverage 75.79% 77.67% +1.87% == Files 199 204 +5 Lines 1594615849 -97 == + Hits1208612310 +224 + Misses 3860 3539 -321 ``` | [Impacted Files](https://codecov.io/gh/apache/incubator-airflow/pull/3650?src=pr=tree) | Coverage Δ | | |---|---|---| | [airflow/jobs.py](https://codecov.io/gh/apache/incubator-airflow/pull/3650/diff?src=pr=tree#diff-YWlyZmxvdy9qb2JzLnB5) | `82.63% <85%> (+0.23%)` | :arrow_up: | | [airflow/sensors/s3\_key\_sensor.py](https://codecov.io/gh/apache/incubator-airflow/pull/3650/diff?src=pr=tree#diff-YWlyZmxvdy9zZW5zb3JzL3MzX2tleV9zZW5zb3IucHk=) | `31.03% <0%> (-68.97%)` | :arrow_down: | | [airflow/sensors/s3\_prefix\_sensor.py](https://codecov.io/gh/apache/incubator-airflow/pull/3650/diff?src=pr=tree#diff-YWlyZmxvdy9zZW5zb3JzL3MzX3ByZWZpeF9zZW5zb3IucHk=) | `41.17% <0%> (-58.83%)` | :arrow_down: | | [airflow/utils/helpers.py](https://codecov.io/gh/apache/incubator-airflow/pull/3650/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9oZWxwZXJzLnB5) | `71.34% <0%> (-13.04%)` | :arrow_down: | | [airflow/hooks/mysql\_hook.py](https://codecov.io/gh/apache/incubator-airflow/pull/3650/diff?src=pr=tree#diff-YWlyZmxvdy9ob29rcy9teXNxbF9ob29rLnB5) | `78% <0%> (-12%)` | :arrow_down: | | [airflow/sensors/sql\_sensor.py](https://codecov.io/gh/apache/incubator-airflow/pull/3650/diff?src=pr=tree#diff-YWlyZmxvdy9zZW5zb3JzL3NxbF9zZW5zb3IucHk=) | `90.47% <0%> (-9.53%)` | :arrow_down: | | [airflow/configuration.py](https://codecov.io/gh/apache/incubator-airflow/pull/3650/diff?src=pr=tree#diff-YWlyZmxvdy9jb25maWd1cmF0aW9uLnB5) | `84.07% <0%> (-5.19%)` | :arrow_down: | | [airflow/utils/state.py](https://codecov.io/gh/apache/incubator-airflow/pull/3650/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9zdGF0ZS5weQ==) | `93.33% <0%> (-3.34%)` | :arrow_down: | | [airflow/models.py](https://codecov.io/gh/apache/incubator-airflow/pull/3650/diff?src=pr=tree#diff-YWlyZmxvdy9tb2RlbHMucHk=) | `88.78% <0%> (-2.93%)` | :arrow_down: | | [airflow/utils/email.py](https://codecov.io/gh/apache/incubator-airflow/pull/3650/diff?src=pr=tree#diff-YWlyZmxvdy91dGlscy9lbWFpbC5weQ==) | `97.4% <0%> (-2.6%)` | :arrow_down: | | ... and [61 more](https://codecov.io/gh/apache/incubator-airflow/pull/3650/diff?src=pr=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/3650?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/incubator-airflow/pull/3650?src=pr=footer). Last update [b8be322...644f6bb](https://codecov.io/gh/apache/incubator-airflow/pull/3650?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (AIRFLOW-3215) Creating EMR using python from airflow
[
https://issues.apache.org/jira/browse/AIRFLOW-3215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16656467#comment-16656467
]
Ash Berlin-Taylor commented on AIRFLOW-3215:
That is an error from a Cloudformation stack provided by AWS, not Airflow, so
it isn't something we can help with.
> Creating EMR using python from airflow
> --
>
> Key: AIRFLOW-3215
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3215
> Project: Apache Airflow
> Issue Type: Task
> Components: aws, DAG
>Affects Versions: 1.7.0
> Environment: Airflow with boto3 - connecting AWS -configure with
> access and security
>Reporter: Pandu
>Priority: Major
>
> I have problem with imports while creating EMR.
> import boto3
> connection = boto3.client(
> 'emr'
> )
> cluster_id = connection.run_job_flow(
> Name='emr123',
> LogUri='s3://emr-spark-application/log.txt',
> ReleaseLabel='emr-4.1.0',
> Instances={
> 'InstanceGroups': [
> {
> 'Name': "Master nodes",
> 'Market': 'ON_DEMAND',
> 'InstanceRole': 'MASTER',
> 'InstanceType': 'm1.large',
> 'InstanceCount': 1
> },
> {
> 'Name': "Slave nodes",
> 'Market': 'ON_DEMAND',
> 'InstanceRole': 'CORE',
> 'InstanceType': 'm1.large',
> 'InstanceCount': 1
> }
> ],
> 'KeepJobFlowAliveWhenNoSteps': True,
> 'TerminationProtected': False
> },
> Applications=[{
> 'Name': 'Hadoop'
> }, {
> 'Name': 'Spark'
> }],
> JobFlowRole='EMR_EC2_DefaultRole',
> ServiceRole='EMR_DefaultRole'
> )
> print (cluster_id['JobFlowId'])
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
