[
https://issues.apache.org/jira/browse/AIRFLOW-3215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16653465#comment-16653465
]
Pandu commented on AIRFLOW-3215:
--------------------------------
DeprecationWarning)
[2018-10-17 12:38:28,414] [12519] {models.py:266} ERROR - Failed to import:
/root/airflow/dags/emr_create_job_flow_operator.py
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/airflow/models.py", line 263, in
process_file
m = imp.load_source(mod_name, filepath)
File "/root/airflow/dags/emr_create_job_flow_operator.py", line 7, in <module>
from telemetry_pipeline_utils import *
ImportError: No module named telemetry_pipeline_utils
[2018-10-17 12:38:28,573] [12520] {models.py:167} INFO - Filling up the DagBag
from /root/airflow/dags
/usr/local/lib/python2.7/dist-packages/airflow/utils/helpers.py:406:
DeprecationWarning: Importing BashOperator directly from <module
'airflow.operators' from
'/usr/local/lib/python2.7/dist-packages/airflow/operators/__init__.pyc'> has
been deprecated. Please import from '<module 'airflow.operators' from
'/usr/local/lib/python2.7/dist-packages/airflow/operators/__init__.pyc'>.[operator_module]'
instead. Support for direct imports will be dropped entirely in Airflow 2.0.
DeprecationWarning)
/usr/local/lib/python2.7/dist-packages/airflow/utils/helpers.py:406:
DeprecationWarning: Importing ExternalTaskSensor directly from <module
'airflow.operators' from
'/usr/local/lib/python2.7/dist-packages/airflow/operators/__init__.pyc'> has
been deprecated. Please import from '<module 'airflow.operators' from
'/usr/local/lib/python2.7/dist-packages/airflow/operators/__init__.pyc'>.[operator_module]'
instead. Support for direct imports will be dropped entirely in Airflow 2.0.
DeprecationWarning)
[2018-10-17 12:38:28,590] [12520] {models.py:266} ERROR - Failed to import:
/root/airflow/dags/emr_create_job_flow_operator.py
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/airflow/models.py", line 263, in
process_file
m = imp.load_source(mod_name, filepath)
File "/root/airflow/dags/emr_create_job_flow_operator.py", line 7, in <module>
from telemetry_pipeline_utils import *
ImportError: No module named telemetry_pipeline_utils
[2018-10-17 12:38:28,637] [12521] {models.py:167} INFO - Filling up the DagBag
from /root/airflow/dags
/usr/local/lib/python2.7/dist-packages/airflow/utils/helpers.py:406:
DeprecationWarning: Importing BashOperator directly from <module
'airflow.operators' from
'/usr/local/lib/python2.7/dist-packages/airflow/operators/__init__.pyc'> has
been deprecated. Please import from '<module 'airflow.operators' from
'/usr/local/lib/python2.7/dist-packages/airflow/operators/__init__.pyc'>.[operator_module]'
instead. Support for direct imports will be dropped entirely in Airflow 2.0.
DeprecationWarning)
/usr/local/lib/python2.7/dist-packages/airflow/utils/helpers.py:406:
DeprecationWarning: Importing ExternalTaskSensor directly from <module
'airflow.operators' from
'/usr/local/lib/python2.7/dist-packages/airflow/operators/__init__.pyc'> has
been deprecated. Please import from '<module 'airflow.operators' from
'/usr/local/lib/python2.7/dist-packages/airflow/operators/__init__.pyc'>.[operator_module]'
instead. Support for direct imports will be dropped entirely in Airflow 2.0.
DeprecationWarning)
[2018-10-17 12:38:28,651] [12521] {models.py:266} ERROR - Failed to import:
/root/airflow/dags/emr_create_job_flow_operator.py
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/airflow/models.py", line 263, in
process_file
m = imp.load_source(mod_name, filepath)
File "/root/airflow/dags/emr_create_job_flow_operator.py", line 7, in <module>
from telemetry_pipeline_utils import *
ImportError: No module named telemetry_pipeline_utils
[2018-10-17 12:38:28,751] [12522] {models.py:167} INFO - Filling up the DagBag
from /root/airflow/dags
/usr/local/lib/python2.7/dist-packages/airflow/utils/helpers.py:406:
DeprecationWarning: Importing BashOperator directly from <module
'airflow.operators' from
'/usr/local/lib/python2.7/dist-packages/airflow/operators/__init__.pyc'> has
been deprecated. Please import from '<module 'airflow.operators' from
'/usr/local/lib/python2.7/dist-packages/airflow/operators/__init__.pyc'>.[operator_module]'
instead. Support for direct imports will be dropped entirely in Airflow 2.0.
DeprecationWarning)
/usr/local/lib/python2.7/dist-packages/airflow/utils/helpers.py:406:
DeprecationWarning: Importing ExternalTaskSensor directly from <module
'airflow.operators' from
'/usr/local/lib/python2.7/dist-packages/airflow/operators/__init__.pyc'> has
been deprecated. Please import from '<module 'airflow.operators' from
'/usr/local/lib/python2.7/dist-packages/airflow/operators/__init__.pyc'>.[operator_module]'
instead. Support for direct imports will be dropped entirely in Airflow 2.0.
DeprecationWarning)
[2018-10-17 12:38:28,766] [12522] {models.py:266} ERROR - Failed to import:
/root/airflow/dags/emr_create_job_flow_operator.py
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/airflow/models.py", line 263, in
process_file
m = imp.load_source(mod_name, filepath)
File "/root/airflow/dags/emr_create_job_flow_operator.py", line 7, in <module>
from telemetry_pipeline_utils import *
ImportError: No module named telemetry_pipeline_utils
Here is the error i get when it is trying to import telemetry_pipeline_utils
But, I have updated airflow to 1.7.0 to 1.8.0
> Creating EMR using python from airflow
> --------------------------------------
>
> Key: AIRFLOW-3215
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3215
> Project: Apache Airflow
> Issue Type: Task
> Components: aws, DAG
> Affects Versions: 1.7.0
> Environment: Airflow with boto3 - connecting AWS -configure with
> access and security
> Reporter: Pandu
> Priority: Major
>
> I have problem with imports while creating EMR.
> import boto3
> connection = boto3.client(
> 'emr'
> )
> cluster_id = connection.run_job_flow(
> Name='emr123',
> LogUri='s3://emr-spark-application/log.txt',
> ReleaseLabel='emr-4.1.0',
> Instances={
> 'InstanceGroups': [
> {
> 'Name': "Master nodes",
> 'Market': 'ON_DEMAND',
> 'InstanceRole': 'MASTER',
> 'InstanceType': 'm1.large',
> 'InstanceCount': 1
> },
> {
> 'Name': "Slave nodes",
> 'Market': 'ON_DEMAND',
> 'InstanceRole': 'CORE',
> 'InstanceType': 'm1.large',
> 'InstanceCount': 1
> }
> ],
> 'KeepJobFlowAliveWhenNoSteps': True,
> 'TerminationProtected': False
> },
> Applications=[{
> 'Name': 'Hadoop'
> }, {
> 'Name': 'Spark'
> }],
> JobFlowRole='EMR_EC2_DefaultRole',
> ServiceRole='EMR_DefaultRole'
> )
> print (cluster_id['JobFlowId'])
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
