[
https://issues.apache.org/jira/browse/AIRFLOW-892?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Morel resolved AIRFLOW-892.
---------------------------------
Resolution: Fixed
Fix Version/s: Airflow 1.8
> HIvePartitionSensor and HiveNamedPartitionSensor raise AttributeError
> ---------------------------------------------------------------------
>
> Key: AIRFLOW-892
> URL: https://issues.apache.org/jira/browse/AIRFLOW-892
> Project: Apache Airflow
> Issue Type: Bug
> Components: core
> Affects Versions: Airflow 1.8
> Environment: Virtualenv 2.7.13 on Centos 6 / Centos 6 + Docker
> Reporter: David Morel
> Priority: Critical
> Labels: metastore
> Fix For: Airflow 1.8
>
> Attachments: airflow-hivesensor.diff
>
>
> All HiveParititionSensor and friends using the Meatastore hook throw this
> exception in 2 installs I've done, one on a clean docker to rule out other
> causes. install has been done using pip install
> git+https://github.com/apache/incubator-airflow.git@v1-8-stable
> {noformat}
> [2017-02-22 08:44:49,079] {base_task_runner.py:95} INFO - Subtask:
> --------------------------------------------------------------------------------
> [2017-02-22 08:44:49,079] {base_task_runner.py:95} INFO - Subtask: Starting
> attempt 1 of 1
> [2017-02-22 08:44:49,079] {base_task_runner.py:95} INFO - Subtask:
> --------------------------------------------------------------------------------
> [2017-02-22 08:44:49,079] {base_task_runner.py:95} INFO - Subtask:
> [2017-02-22 08:44:49,111] {base_task_runner.py:95} INFO - Subtask:
> [2017-02-22 08:44:49,110] {models.py:1342} INFO - Executing
> <Task(HivePartitionSensor): wait_for_***> on 2017-02-18 00:00:00
> [2017-02-22 08:44:49,124] {base_task_runner.py:95} INFO - Subtask:
> [2017-02-22 08:44:49,123] {sensors.py:373} INFO - Poking for table ****,
> partition d_date='2017-02-18'
> [2017-02-22 08:44:49,125] {base_task_runner.py:95} INFO - Subtask:
> [2017-02-22 08:44:49,124] {models.py:1417} ERROR -
> [2017-02-22 08:44:49,125] {base_task_runner.py:95} INFO - Subtask: Traceback
> (most recent call last):
> [2017-02-22 08:44:49,125] {base_task_runner.py:95} INFO - Subtask: File
> "/home/airflow/airflow2/lib/python2.7/site-packages/airflow/models.py", line
> 1374, in run
> [2017-02-22 08:44:49,125] {base_task_runner.py:95} INFO - Subtask: result
> = task_copy.execute(context=context)
> [2017-02-22 08:44:49,125] {base_task_runner.py:95} INFO - Subtask: File
> "/home/airflow/airflow2/lib/python2.7/site-packages/airflow/operators/sensors.py",
> line 76, in execute
> [2017-02-22 08:44:49,125] {base_task_runner.py:95} INFO - Subtask: while
> not self.poke(context):
> [2017-02-22 08:44:49,125] {base_task_runner.py:95} INFO - Subtask: File
> "/home/airflow/airflow2/lib/python2.7/site-packages/airflow/operators/sensors.py",
> line 375, in poke
> [2017-02-22 08:44:49,126] {base_task_runner.py:95} INFO - Subtask:
> self.hook = airflow.hooks.hive_hooks.HiveMetastoreHook(
> [2017-02-22 08:44:49,126] {base_task_runner.py:95} INFO - Subtask: File
> "/home/airflow/airflow2/lib/python2.7/site-packages/airflow/utils/helpers.py",
> line 436, in __getattr__
> [2017-02-22 08:44:49,126] {base_task_runner.py:95} INFO - Subtask: raise
> AttributeError
> [2017-02-22 08:44:49,126] {base_task_runner.py:95} INFO - Subtask:
> AttributeError
> [2017-02-22 08:44:49,126] {base_task_runner.py:95} INFO - Subtask:
> [2017-02-22 08:44:49,125] {models.py:1441} INFO - Marking task as FAILED.
> [2017-02-22 08:44:49,163] {base_task_runner.py:95} INFO - Subtask:
> [2017-02-22 08:44:49,163] {models.py:1462} ERROR -
> [2017-02-22 08:44:49,163] {base_task_runner.py:95} INFO - Subtask: Traceback
> (most recent call last):
> [2017-02-22 08:44:49,164] {base_task_runner.py:95} INFO - Subtask: File
> "/home/airflow/airflow2/bin/airflow", line 28, in <module>
> [2017-02-22 08:44:49,164] {base_task_runner.py:95} INFO - Subtask:
> args.func(args)
> [2017-02-22 08:44:49,164] {base_task_runner.py:95} INFO - Subtask: File
> "/home/airflow/airflow2/lib/python2.7/site-packages/airflow/bin/cli.py", line
> 427, in run
> [2017-02-22 08:44:49,164] {base_task_runner.py:95} INFO - Subtask:
> pool=args.pool,
> [2017-02-22 08:44:49,164] {base_task_runner.py:95} INFO - Subtask: File
> "/home/airflow/airflow2/lib/python2.7/site-packages/airflow/utils/db.py",
> line 54, in wrapper
> [2017-02-22 08:44:49,164] {base_task_runner.py:95} INFO - Subtask: result
> = func(*args, **kwargs)
> [2017-02-22 08:44:49,164] {base_task_runner.py:95} INFO - Subtask: File
> "/home/airflow/airflow2/lib/python2.7/site-packages/airflow/models.py", line
> 1374, in run
> [2017-02-22 08:44:49,165] {base_task_runner.py:95} INFO - Subtask: result
> = task_copy.execute(context=context)
> [2017-02-22 08:44:49,165] {base_task_runner.py:95} INFO - Subtask: File
> "/home/airflow/airflow2/lib/python2.7/site-packages/airflow/operators/sensors.py",
> line 76, in execute
> [2017-02-22 08:44:49,165] {base_task_runner.py:95} INFO - Subtask: while
> not self.poke(context):
> [2017-02-22 08:44:49,165] {base_task_runner.py:95} INFO - Subtask: File
> "/home/airflow/airflow2/lib/python2.7/site-packages/airflow/operators/sensors.py",
> line 375, in poke
> [2017-02-22 08:44:49,165] {base_task_runner.py:95} INFO - Subtask:
> self.hook = airflow.hooks.hive_hooks.HiveMetastoreHook(
> [2017-02-22 08:44:49,165] {base_task_runner.py:95} INFO - Subtask: File
> "/home/airflow/airflow2/lib/python2.7/site-packages/airflow/utils/helpers.py",
> line 436, in __getattr__
> [2017-02-22 08:44:49,165] {base_task_runner.py:95} INFO - Subtask: raise
> AttributeError
> {noformat}
> I've worked around the issue adding a 'import airflow.hooks.hive_hooks' in
> class HiveParititionSensor (because I don't really know what this is about,
> there surely is a real fix to be done). It fixes the issue for me.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
