virajjasani opened a new pull request #3802:
URL: https://github.com/apache/hadoop/pull/3802
### Description of PR
Yarn ResourceManager and NodeManager in the current state are not able to
interact with HBase cluster because of the incompatible guava and guice
dependencies being used in the classpath.
```
2021-12-14 19:26:30,511 INFO [IPC Server listener on 8020] ipc.Server
(Server.java:run(1479)) - IPC Server listener on 8020: starting
2021-12-14 19:26:30,960 INFO [Listener at 0.0.0.0/8020] webproxy.ProxyCA
(ProxyCA.java:createCert(183)) - Created Certificate for
OU=YARN-3eb03ade-472e-45ec-a274-7a11be9c791e
2021-12-14 19:26:30,992 INFO [Listener at 0.0.0.0/8020]
recovery.RMStateStore (RMStateStore.java:transition(647)) - Storing CA
Certificate and Private Key
2021-12-14 19:26:30,992 INFO [Listener at 0.0.0.0/8020]
resourcemanager.ResourceManager (ResourceManager.java:transitionToActive(1525))
- Transitioned to active state
2021-12-14 19:26:33,345 WARN [pool-28-thread-1]
storage.TimelineStorageMonitor (TimelineStorageMonitor.java:run(95)) - Got
failure attempting to read from HBase, assuming Storage is down
java.lang.RuntimeException: org.apache.hadoop.hbase.DoNotRetryIOException:
java.lang.NoSuchMethodError:
com.google.common.net.HostAndPort.getHostText()Ljava/lang/String;
at
org.apache.hadoop.hbase.client.AbstractClientScanner$1.hasNext(AbstractClientScanner.java:95)
at
org.apache.hadoop.yarn.server.timelineservice.storage.reader.TimelineEntityReader.readEntities(TimelineEntityReader.java:283)
at
org.apache.hadoop.yarn.server.timelineservice.storage.HBaseStorageMonitor.healthCheck(HBaseStorageMonitor.java:77)
at
org.apache.hadoop.yarn.server.timelineservice.storage.TimelineStorageMonitor$MonitorThread.run(TimelineStorageMonitor.java:89)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.hadoop.hbase.DoNotRetryIOException:
java.lang.NoSuchMethodError:
com.google.common.net.HostAndPort.getHostText()Ljava/lang/String;
at
org.apache.hadoop.hbase.client.RpcRetryingCaller.translateException(RpcRetryingCaller.java:260)
at
org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:233)
at
org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:394)
at
org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:368)
at
org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:143)
at
org.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture.run(ResultBoundedCompletionService.java:80)
... 3 more
Caused by: java.lang.NoSuchMethodError:
com.google.common.net.HostAndPort.getHostText()Ljava/lang/String;
at org.apache.hadoop.hbase.net.Address.getHostName(Address.java:72)
at
org.apache.hadoop.hbase.net.Address.toSocketAddress(Address.java:57)
at
org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:576)
at
org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$BlockingStub.scan(ClientProtos.java:37250)
at
org.apache.hadoop.hbase.client.ScannerCallable.openScanner(ScannerCallable.java:405)
at
org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:274)
at
org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:62)
at
org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:231)
... 7 more
```
Once we allow RM and NM to use Timeline service specific guava version, it
fails due to incompatibilities b/ guava and guice versions because hbase
compatible guava is not guice compatible guava for Yarn:
```
2021-12-14 20:28:10,405 ERROR [main] resourcemanager.ResourceManager
(MarkerIgnoringBase.java:error(159)) - Error starting ResourceManager
java.lang.NoSuchMethodError:
com.google.common.base.Preconditions.checkArgument(ZLjava/lang/String;Ljava/lang/Object;Ljava/lang/Object;)V
at com.google.inject.TypeLiteral.getParameterTypes(TypeLiteral.java:269)
at
com.google.inject.spi.InjectionPoint.forMember(InjectionPoint.java:113)
at com.google.inject.spi.InjectionPoint.<init>(InjectionPoint.java:72)
at
com.google.inject.spi.InjectionPoint.forMethod(InjectionPoint.java:316)
at
com.google.inject.internal.ProviderMethodsModule.createProviderMethod(ProviderMethodsModule.java:293)
at
com.google.inject.internal.ProviderMethodsModule.getProviderMethods(ProviderMethodsModule.java:135)
at
com.google.inject.internal.ProviderMethodsModule.configure(ProviderMethodsModule.java:105)
at
com.google.inject.spi.Elements$RecordingBinder.install(Elements.java:347)
at
com.google.inject.spi.Elements$RecordingBinder.install(Elements.java:356)
at com.google.inject.AbstractModule.install(AbstractModule.java:103)
at
com.google.inject.servlet.ServletModule.configure(ServletModule.java:49)
at com.google.inject.AbstractModule.configure(AbstractModule.java:61)
at
com.google.inject.spi.Elements$RecordingBinder.install(Elements.java:347)
at com.google.inject.spi.Elements.getElements(Elements.java:104)
at
com.google.inject.internal.InjectorShell$Builder.build(InjectorShell.java:137)
at
com.google.inject.internal.InternalInjectorCreator.build(InternalInjectorCreator.java:105)
at com.google.inject.Guice.createInjector(Guice.java:87)
at com.google.inject.Guice.createInjector(Guice.java:69)
at com.google.inject.Guice.createInjector(Guice.java:59)
at org.apache.hadoop.yarn.webapp.WebApps$Builder.build(WebApps.java:420)
at org.apache.hadoop.yarn.webapp.WebApps$Builder.start(WebApps.java:468)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.startWepApp(ResourceManager.java:1443)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceStart(ResourceManager.java:1552)
at
org.apache.hadoop.service.AbstractService.start(AbstractService.java:195)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(ResourceManager.java:1753)
```
Hence, we need to keep both guice and guava compatible with each other and
ensure guava is also compatible with HBase.
Changes on this PR would allow Yarn to use guice and guava versions
compatible with HBase and not the ones being derived from hadoop-project.
### How was this patch tested?
Tested locally. Triggered MapReduce sample WordCount job. No issues
encountered with RM and NM.
Data is also flowing from Yarn to HBase tables:
<img width="1783" alt="Screenshot 2021-12-14 at 10 32 28 PM"
src="https://user-images.githubusercontent.com/34790606/146157552-648e66be-2c42-4f18-bfde-bbf555b199ad.png";>
<img width="1030" alt="Screenshot 2021-12-14 at 10 32 59 PM"
src="https://user-images.githubusercontent.com/34790606/146157575-8839788d-8cb6-40a2-ae67-0e8b8250a53b.png";>
<img width="1357" alt="Screenshot 2021-12-14 at 10 33 25 PM"
src="https://user-images.githubusercontent.com/34790606/146157578-8dfd44cd-1bc1-49ba-b893-c8fda4010097.png";>
### For code changes:
- [X] Does the title or this PR starts with the corresponding JIRA issue id
(e.g. 'HADOOP-17799. Your PR title ...')?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
![https://user-images.githubusercontent.com/34790606/146157552-648e66be-2c42-4f18-bfde-bbf555b199ad.png"](https://user-images.githubusercontent.com/34790606/146157552-648e66be-2c42-4f18-bfde-bbf555b199ad.png"){kind=link}
![https://user-images.githubusercontent.com/34790606/146157575-8839788d-8cb6-40a2-ae67-0e8b8250a53b.png"](https://user-images.githubusercontent.com/34790606/146157575-8839788d-8cb6-40a2-ae67-0e8b8250a53b.png"){kind=link}
![https://user-images.githubusercontent.com/34790606/146157578-8dfd44cd-1bc1-49ba-b893-c8fda4010097.png"](https://user-images.githubusercontent.com/34790606/146157578-8dfd44cd-1bc1-49ba-b893-c8fda4010097.png"){kind=link}