Pandu created AIRFLOW-3215:
------------------------------
Summary: Creating EMR using python from airflow
Key: AIRFLOW-3215
URL: https://issues.apache.org/jira/browse/AIRFLOW-3215
Project: Apache Airflow
Issue Type: Task
Components: aws, DAG
Affects Versions: 1.7.0
Environment: Airflow with boto3 - connecting AWS -configure with
access and security
Reporter: Pandu
I have problem with imports while creating EMR.
import boto3
connection = boto3.client(
'emr'
)
cluster_id = connection.run_job_flow(
Name='emr123',
LogUri='s3://emr-spark-application/log.txt',
ReleaseLabel='emr-4.1.0',
Instances={
'InstanceGroups': [
{
'Name': "Master nodes",
'Market': 'ON_DEMAND',
'InstanceRole': 'MASTER',
'InstanceType': 'm1.large',
'InstanceCount': 1
},
{
'Name': "Slave nodes",
'Market': 'ON_DEMAND',
'InstanceRole': 'CORE',
'InstanceType': 'm1.large',
'InstanceCount': 1
}
],
'KeepJobFlowAliveWhenNoSteps': True,
'TerminationProtected': False
},
Applications=[{
'Name': 'Hadoop'
}, {
'Name': 'Spark'
}],
JobFlowRole='EMR_EC2_DefaultRole',
ServiceRole='EMR_DefaultRole'
)
print (cluster_id['JobFlowId'])
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
