[ https://issues.apache.org/jira/browse/AIRFLOW-2751?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16601327#comment-16601327 ]
Apache Spark commented on AIRFLOW-2751: --------------------------------------- User 'happyjulie' has created a pull request for this issue: https://github.com/apache/incubator-airflow/pull/3600 > hive to druid library conflicts > ------------------------------- > > Key: AIRFLOW-2751 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2751 > Project: Apache Airflow > Issue Type: Bug > Components: operators > Affects Versions: 1.10 > Reporter: juhwi lee > Assignee: juhwi lee > Priority: Major > Fix For: 2.0.0 > > > There is a jackson.datatype.guava library conflict problem in > HiveToDruidTransfer when hive and druid has different library version. > In my case > hdp version: 2.6.3.0 > druid version : druid-0.10.1( jackson-datatype-guava-2.4.6.jar) > We should be able to set this property to solve that problem. > "mapreduce.job.user.classpath.first" : "true" > It means "prefer loading Druid's version of a library when there is a > conflict". > reference. [http://druid.io/docs/latest/operations/other-hadoop.html] (Tip > #2) > But Job properties is hard coded in HiveToDruidTransfer. > > This is the error message I got. > 2018-06-22T01:37:11,190 INFO [task-runner-0-priority-0] > org.apache.hadoop.mapreduce.Job - Task Id : > attempt_1525761839652_85792_m_000000_0, Status : FAILED > Error: class > com.fasterxml.jackson.datatype.guava.deser.HostAndPortDeserializer overrides > final method > deserialize.(Lcom/fasterxml/jackson/core/JsonParser;Lcom/fasterxml/jackson/databind/DeserializationContext;)Ljava/lang/Object; > 2018-06-22T01:37:15,216 INFO [task-runner-0-priority-0] > org.apache.hadoop.mapreduce.Job - Task Id : > attempt_1525761839652_85792_m_000000_1, Status : FAILED > Error: class > com.fasterxml.jackson.datatype.guava.deser.HostAndPortDeserializer overrides > final method > deserialize.(Lcom/fasterxml/jackson/core/JsonParser;Lcom/fasterxml/jackson/databind/DeserializationContext;)Ljava/lang/Object; > 2018-06-22T01:37:20,231 INFO [task-runner-0-priority-0] > org.apache.hadoop.mapreduce.Job - Task Id : > attempt_1525761839652_85792_m_000000_2, Status : FAILED > Error: class > com.fasterxml.jackson.datatype.guava.deser.HostAndPortDeserializer overrides > final method > deserialize.(Lcom/fasterxml/jackson/core/JsonParser;Lcom/fasterxml/jackson/databind/DeserializationContext;)Ljava/lang/Object; > 2018-06-22T01:37:26,248 INFO [task-runner-0-priority-0] > org.apache.hadoop.mapreduce.Job - map 100% reduce 100% > 2018-06-22T01:37:26,252 INFO [task-runner-0-priority-0] > org.apache.hadoop.mapreduce.Job - Job job_1525761839652_85792 failed with > state FAILED due to: Task failed task_1525761839652_85792_m_000000 > Job failed as tasks failed. failedMaps:1 failedReduces:0 > -- This message was sent by Atlassian JIRA (v7.6.3#76005)