[
https://issues.apache.org/jira/browse/HIVE-26939?focusedWorklogId=839034&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-839034
]
ASF GitHub Bot logged work on HIVE-26939:
-----------------------------------------
Author: ASF GitHub Bot
Created on: 13/Jan/23 09:14
Start Date: 13/Jan/23 09:14
Worklog Time Spent: 10m
Work Description: amanraj2520 commented on PR #3941:
URL: https://github.com/apache/hive/pull/3941#issuecomment-1381518433
Hi @abstractdog Actually this error does not come up in a test until we
manually bring up a cluster with Hive master and Hadoop 3.3.4. This error did
not happen with hadoop 3.1.1 since at that time Hadoop was using Jackson 2.7.8
for deserializing this placement_policy yarnfile. Jackson 3.7.8 was able to
deserialize it from "NODE" to "node" since it was case insensitive. Thus is
found the [node] enum in the Hadoop PlacementScope class which in turn brought
up the AM successfully. But Jackson 2.12.7 (Hadoop 3.3.4) is doing case
sensitive parsing thereby not finding the relevant enum. Either we have to
disable this configuration from jackson side
2.3/com/fasterxml/jackson/databind/MapperFeature.html#ACCEPT_CASE_INSENSITIVE_ENUMS
or from Hive side we will have to send "node" as the placement policy. We can
discuss on this Lazlo if you have doubts.
Issue Time Tracking
-------------------
Worklog Id: (was: 839034)
Time Spent: 50m (was: 40m)
> Hive LLAP Application Master fails to come up with Hadoop 3.3.4
> ---------------------------------------------------------------
>
> Key: HIVE-26939
> URL: https://issues.apache.org/jira/browse/HIVE-26939
> Project: Hive
> Issue Type: Bug
> Components: Hive
> Reporter: Aman Raj
> Assignee: Aman Raj
> Priority: Major
> Labels: pull-request-available
> Time Spent: 50m
> Remaining Estimate: 0h
>
> When current oss master hive tries to bring up the LLAP Application Master,
> it fails with this issue :
> {code:java}
> Executing the launch command\nINFO client.ServiceClient: Loading service
> definition from local FS:
> /var/lib/ambari-agent/tmp/llap-yarn-service_2023-01-10_07-56-46/Yarnfile\nERROR
> utils.JsonSerDeser: Exception while parsing json input
> stream\ncom.fasterxml.jackson.databind.exc.InvalidFormatException: Cannot
> deserialize value of type
> `org.apache.hadoop.yarn.service.api.records.PlacementScope` from String
> \"NODE\": not one of the values accepted for Enum class: [node, rack]\n at
> [Source: (org.apache.hadoop.fs.ChecksumFileSystem$FSDataBoundedInputStream);
> line: 31, column: 22] (through reference chain:
> org.apache.hadoop.yarn.service.api.records.Service[\"components\"]->java.util.ArrayList[0]->org.apache.hadoop.yarn.service.api.records.Component[\"placement_policy\"]->org.apache.hadoop.yarn.service.api.records.PlacementPolicy[\"constraints\"]->java.util.ArrayList[0]->org.apache.hadoop.yarn.service.api.records.PlacementConstraint[\"scope\"])\n\tat
>
> com.fasterxml.jackson.databind.exc.InvalidFormatException.from(InvalidFormatException.java:67)
> ~[jackson-databind-2.12.7.jar:2.12.7]\n\tat
> com.fasterxml.jackson.databind.DeserializationContext.weirdStringException(DeserializationContext.java:1851)
> ~[jackson-databind-2.12.7.jar:2.12.7]\n\tat
> com.fasterxml.jackson.databind.DeserializationContext.handleWeirdStringValue(DeserializationContext.java:1079)
> ~[jackson-databind-2.12.7.jar:2.12.7]\n\tat
> com.fasterxml.jackson.databind.deser.std.EnumDeserializer._deserializeAltString(EnumDeserializer.java:339)
> ~[jackson-databind-2.12.7.jar:2.12.7]\n\tat
> com.fasterxml.jackson.databind.deser.std.EnumDeserializer._fromString(EnumDeserializer.java:214)
> ~[jackson-databind-2.12.7.jar:2.12.7]\n\tat
> com.fasterxml.jackson.databind.deser.std.EnumDeserializer.deserialize(EnumDeserializer.java:188)
> ~[jackson-databind-2.12.7.jar:2.12.7]\n\tat
> com.fasterxml.jackson.databind.deser.impl.MethodProperty.deserializeAndSet(MethodProperty.java:129)
> ~[jackson-databind-2.12.7.jar:2.12.7]\n\tat
> com.fasterxml.jackson.databind.deser.BeanDeserializer.vanillaDeserialize(BeanDeserializer.java:324)
> ~[jackson-databind-2.12.7.jar:2.12.7]\n\tat
> com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:187)
> ~[jackson-databind-2.12.7.jar:2.12.7]\n\tat
> com.fasterxml.jackson.databind.deser.std.CollectionDeserializer._deserializeFromArray(CollectionDeserializer.java:355)
> ~[jackson-databind-2.12.7.jar:2.12.7]\n\tat
> com.fasterxml.jackson.databind.deser.std.CollectionDeserializer.deserialize(CollectionDeserializer.java:244)
> ~[jackson-databind-2.12.7.jar:2.12.7]\n\tat
> com.fasterxml.jackson.databind.deser.std.CollectionDeserializer.deserialize(CollectionDeserializer.java:28)
> ~[jackson-databind-2.12.7.jar:2.12.7]\n\tat
> com.fasterxml.jackson.databind.deser.impl.MethodProperty.deserializeAndSet(MethodProperty.java:129)
> ~[jackson-databind-2.12.7.jar:2.12.7]\n\tat
> com.fasterxml.jackson.databind.deser.BeanDeserializer.vanillaDeserialize(BeanDeserializer.java:324)
> ~[jackson-databind-2.12.7.jar:2.12.7]\n\tat
> com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:187)
> ~[jackson-databind-2.12.7.jar:2.12.7]\n\tat
> com.fasterxml.jackson.databind.deser.impl.MethodProperty.deserializeAndSet(MethodProperty.java:129)
> ~[jackson-databind-2.12.7.jar:2.12.7]\n\tat
> com.fasterxml.jackson.databind.deser.BeanDeserializer.vanillaDeserialize(BeanDeserializer.java:324)
> ~[jackson-databind-2.12.7.jar:2.12.7]\n\tat
> com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:187)
> ~[jackson-databind-2.12.7.jar:2.12.7]\n\tat
> com.fasterxml.jackson.databind.deser.std.CollectionDeserializer._deserializeFromArray(CollectionDeserializer.java:355)
> ~[jackson-databind-2.12.7.jar:2.12.7]\n\tat
> com.fasterxml.jackson.databind.deser.std.CollectionDeserializer.deserialize(CollectionDeserializer.java:244)
> ~[jackson-databind-2.12.7.jar:2.12.7]\n\tat
> com.fasterxml.jackson.databind.deser.std.CollectionDeserializer.deserialize(CollectionDeserializer.java:28)
> ~[jackson-databind-2.12.7.jar:2.12.7]\n\tat
> com.fasterxml.jackson.databind.deser.impl.MethodProperty.deserializeAndSet(MethodProperty.java:129)
> ~[jackson-databind-2.12.7.jar:2.12.7]\n\tat
> com.fasterxml.jackson.databind.deser.BeanDeserializer.vanillaDeserialize(BeanDeserializer.java:324)
> ~[jackson-databind-2.12.7.jar:2.12.7]\n\tat
> com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:187)
> ~[jackson-databind-2.12.7.jar:2.12.7]\n\tat
> com.fasterxml.jackson.databind.deser.DefaultDeserializationContext.readRootValue(DefaultDeserializationContext.java:322)
> ~[jackson-databind-2.12.7.jar:2.12.7]\n\tat
> com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:4593)
> ~[jackson-databind-2.12.7.jar:2.12.7]\n\tat
> com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:3585)
> ~[jackson-databind-2.12.7.jar:2.12.7]\n\tat
> org.apache.hadoop.yarn.service.utils.JsonSerDeser.fromStream(JsonSerDeser.java:137)
> [hadoop-yarn-services-core-3.3.4.5.1-SNAPSHOT.jar:?]\n\tat
> org.apache.hadoop.yarn.service.utils.JsonSerDeser.load(JsonSerDeser.java:179)
> [hadoop-yarn-services-core-3.3.4.5.1-SNAPSHOT.jar:?]\n\tat
> org.apache.hadoop.yarn.service.client.ServiceClient.loadAppJsonFromLocalFS(ServiceClient.java:208)
> [hadoop-yarn-services-core-3.3.4.5.1-SNAPSHOT.jar:?]\n\tat
> org.apache.hadoop.yarn.service.client.ServiceClient.actionLaunch(ServiceClient.java:546)
> [hadoop-yarn-services-core-3.3.4.5.1-SNAPSHOT.jar:?]\n\tat
> org.apache.hadoop.hive.llap.cli.LlapSliderUtils.startCluster(LlapSliderUtils.java:102)
> [hive-llap-server-3.1.2.5.1-SNAPSHOT.jar:3.1.2.5.1-SNAPSHOT]\n\tat
> org.apache.hadoop.hive.llap.cli.LlapServiceDriver.run(LlapServiceDriver.java:601)
> [hive-llap-server-3.1.2.5.1-SNAPSHOT.jar:3.1.2.5.1-SNAPSHOT]\n\tat
> org.apache.hadoop.hive.llap.cli.LlapServiceDriver.main(LlapServiceDriver.java:119)
> [hive-llap-server-3.1.2.5.1-SNAPSHOT.jar:3.1.2.5.1-SNAPSHOT]\n {code}
> This is because llap_server uses templates.py yarnfile to bringup the LLAP
> Application Master which has the placement_policy as "NODE". Hadoop has
> started using 2.12.7 jackson version which has issues in parsing NODE to
> either [node, rack]. Therefore it is not able to cast the placement policy
> correctly.
>
> Resolution :
> We will have to send placement_policy as "node" from the templates.py of Hive
> Llap Server module. Will raise a PR for this issue :
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
