[
https://issues.apache.org/jira/browse/AIRFLOW-2751?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
juhwi lee updated AIRFLOW-2751:
-------------------------------
Description:
There is a jackson.datatype.guava library conflict problem in
HiveToDruidTransfer when hive and druid has different library.
In my case
hdp version: 2.6.3.0
druid version : druid-0.10.1( jackson-datatype-guava-2.4.6.jar)
We should be able to set this property to solve that problem.
"mapreduce.job.user.classpath.first" : "true"
It means "prefer loading Druid's version of a library when there is a conflict".
reference. [http://druid.io/docs/latest/operations/other-hadoop.html] (Tip #2)
But Job properties is hard coded in HiveToDruidTransfer.
This is the error message I got.
2018-06-22T01:37:11,190 INFO [task-runner-0-priority-0]
org.apache.hadoop.mapreduce.Job - Task Id :
attempt_1525761839652_85792_m_000000_0, Status : FAILED
Error: class
com.fasterxml.jackson.datatype.guava.deser.HostAndPortDeserializer overrides
final method
deserialize.(Lcom/fasterxml/jackson/core/JsonParser;Lcom/fasterxml/jackson/databind/DeserializationContext;)Ljava/lang/Object;
2018-06-22T01:37:15,216 INFO [task-runner-0-priority-0]
org.apache.hadoop.mapreduce.Job - Task Id :
attempt_1525761839652_85792_m_000000_1, Status : FAILED
Error: class
com.fasterxml.jackson.datatype.guava.deser.HostAndPortDeserializer overrides
final method
deserialize.(Lcom/fasterxml/jackson/core/JsonParser;Lcom/fasterxml/jackson/databind/DeserializationContext;)Ljava/lang/Object;
2018-06-22T01:37:20,231 INFO [task-runner-0-priority-0]
org.apache.hadoop.mapreduce.Job - Task Id :
attempt_1525761839652_85792_m_000000_2, Status : FAILED
Error: class
com.fasterxml.jackson.datatype.guava.deser.HostAndPortDeserializer overrides
final method
deserialize.(Lcom/fasterxml/jackson/core/JsonParser;Lcom/fasterxml/jackson/databind/DeserializationContext;)Ljava/lang/Object;
2018-06-22T01:37:26,248 INFO [task-runner-0-priority-0]
org.apache.hadoop.mapreduce.Job - map 100% reduce 100%
2018-06-22T01:37:26,252 INFO [task-runner-0-priority-0]
org.apache.hadoop.mapreduce.Job - Job job_1525761839652_85792 failed with state
FAILED due to: Task failed task_1525761839652_85792_m_000000
Job failed as tasks failed. failedMaps:1 failedReduces:0
was:
There is a jackson.datatype.guava library conflict problem in
HiveToDruidTransfer when hive and druid has different library.
In my case
hdp version: 2.6.3.0
druid version : druid-0.10.1( jackson-datatype-guava-2.4.6.jar)
This is the error message I got.
2018-06-22T01:37:11,190 INFO [task-runner-0-priority-0]
org.apache.hadoop.mapreduce.Job - Task Id :
attempt_1525761839652_85792_m_000000_0, Status : FAILED
Error: class com.fasterxml.jackson.datatype.guava.deser.HostAndPortDeserializer
overrides final method
deserialize.(Lcom/fasterxml/jackson/core/JsonParser;Lcom/fasterxml/jackson/databind/DeserializationContext;)Ljava/lang/Object;
2018-06-22T01:37:15,216 INFO [task-runner-0-priority-0]
org.apache.hadoop.mapreduce.Job - Task Id :
attempt_1525761839652_85792_m_000000_1, Status : FAILED
Error: class com.fasterxml.jackson.datatype.guava.deser.HostAndPortDeserializer
overrides final method
deserialize.(Lcom/fasterxml/jackson/core/JsonParser;Lcom/fasterxml/jackson/databind/DeserializationContext;)Ljava/lang/Object;
2018-06-22T01:37:20,231 INFO [task-runner-0-priority-0]
org.apache.hadoop.mapreduce.Job - Task Id :
attempt_1525761839652_85792_m_000000_2, Status : FAILED
Error: class com.fasterxml.jackson.datatype.guava.deser.HostAndPortDeserializer
overrides final method
deserialize.(Lcom/fasterxml/jackson/core/JsonParser;Lcom/fasterxml/jackson/databind/DeserializationContext;)Ljava/lang/Object;
2018-06-22T01:37:26,248 INFO [task-runner-0-priority-0]
org.apache.hadoop.mapreduce.Job - map 100% reduce 100%
2018-06-22T01:37:26,252 INFO [task-runner-0-priority-0]
org.apache.hadoop.mapreduce.Job - Job job_1525761839652_85792 failed with state
FAILED due to: Task failed task_1525761839652_85792_m_000000
Job failed as tasks failed. failedMaps:1 failedReduces:0
We should be able to set this property to solve that problem.
"mapreduce.job.user.classpath.first" : "true"
It means "prefer loading Druid's version of a library when there is a conflict".
reference. http://druid.io/docs/latest/operations/other-hadoop.html (Tip #2)
But Job properties is hard coded in HiveToDruidTransfer.
> hive to druid library conflicts
> -------------------------------
>
> Key: AIRFLOW-2751
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2751
> Project: Apache Airflow
> Issue Type: Bug
> Components: operators
> Affects Versions: 1.10
> Reporter: juhwi lee
> Priority: Major
>
> There is a jackson.datatype.guava library conflict problem in
> HiveToDruidTransfer when hive and druid has different library.
> In my case
> hdp version: 2.6.3.0
> druid version : druid-0.10.1( jackson-datatype-guava-2.4.6.jar)
> We should be able to set this property to solve that problem.
> "mapreduce.job.user.classpath.first" : "true"
> It means "prefer loading Druid's version of a library when there is a
> conflict".
> reference. [http://druid.io/docs/latest/operations/other-hadoop.html] (Tip
> #2)
> But Job properties is hard coded in HiveToDruidTransfer.
>
> This is the error message I got.
> 2018-06-22T01:37:11,190 INFO [task-runner-0-priority-0]
> org.apache.hadoop.mapreduce.Job - Task Id :
> attempt_1525761839652_85792_m_000000_0, Status : FAILED
> Error: class
> com.fasterxml.jackson.datatype.guava.deser.HostAndPortDeserializer overrides
> final method
> deserialize.(Lcom/fasterxml/jackson/core/JsonParser;Lcom/fasterxml/jackson/databind/DeserializationContext;)Ljava/lang/Object;
> 2018-06-22T01:37:15,216 INFO [task-runner-0-priority-0]
> org.apache.hadoop.mapreduce.Job - Task Id :
> attempt_1525761839652_85792_m_000000_1, Status : FAILED
> Error: class
> com.fasterxml.jackson.datatype.guava.deser.HostAndPortDeserializer overrides
> final method
> deserialize.(Lcom/fasterxml/jackson/core/JsonParser;Lcom/fasterxml/jackson/databind/DeserializationContext;)Ljava/lang/Object;
> 2018-06-22T01:37:20,231 INFO [task-runner-0-priority-0]
> org.apache.hadoop.mapreduce.Job - Task Id :
> attempt_1525761839652_85792_m_000000_2, Status : FAILED
> Error: class
> com.fasterxml.jackson.datatype.guava.deser.HostAndPortDeserializer overrides
> final method
> deserialize.(Lcom/fasterxml/jackson/core/JsonParser;Lcom/fasterxml/jackson/databind/DeserializationContext;)Ljava/lang/Object;
> 2018-06-22T01:37:26,248 INFO [task-runner-0-priority-0]
> org.apache.hadoop.mapreduce.Job - map 100% reduce 100%
> 2018-06-22T01:37:26,252 INFO [task-runner-0-priority-0]
> org.apache.hadoop.mapreduce.Job - Job job_1525761839652_85792 failed with
> state FAILED due to: Task failed task_1525761839652_85792_m_000000
> Job failed as tasks failed. failedMaps:1 failedReduces:0
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
