[jira] [Created] (HADOOP-19174) Tez and hive jobs fail due to google's protobuf 2.5.0 in classpath
Bilwa S T created HADOOP-19174: -- Summary: Tez and hive jobs fail due to google's protobuf 2.5.0 in classpath Key: HADOOP-19174 URL: https://issues.apache.org/jira/browse/HADOOP-19174 Project: Hadoop Common Issue Type: Bug Reporter: Bilwa S T Assignee: Bilwa S T There are two issues here: 1. We are running tez 0.10.3 which uses hadoop 3.3.6 version. Tez has protobuf version 3.21.1 Below is the exception we get. This is due to protobuf-2.5.0 in our hadoop classpath java.lang.IllegalAccessError: class org.apache.tez.dag.api.records.DAGProtos$ConfigurationProto tried to access private field com.google.protobuf.AbstractMessage.memoizedSize (org.apache.tez.dag.api.records.DAGProtos$ConfigurationProto and com.google.protobuf.AbstractMessage are in unnamed module of loader 'app') at org.apache.tez.dag.api.records.DAGProtos$ConfigurationProto.getSerializedSize(DAGProtos.java:21636) at com.google.protobuf.AbstractMessageLite.writeTo(AbstractMessageLite.java:75) at org.apache.tez.common.TezUtils.writeConfInPB(TezUtils.java:170) at org.apache.tez.common.TezUtils.createByteStringFromConf(TezUtils.java:83) at org.apache.tez.common.TezUtils.createUserPayloadFromConf(TezUtils.java:101) at org.apache.tez.dag.app.DAGAppMaster.serviceInit(DAGAppMaster.java:436) at org.apache.hadoop.service.AbstractService.init(AbstractService.java:164) at org.apache.tez.dag.app.DAGAppMaster$9.run(DAGAppMaster.java:2600) at java.base/java.security.AccessController.doPrivileged(AccessController.java:712) at java.base/javax.security.auth.Subject.doAs(Subject.java:439) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1899) at org.apache.tez.dag.app.DAGAppMaster.initAndStartAppMaster(DAGAppMaster.java:2597) at org.apache.tez.dag.app.DAGAppMaster.main(DAGAppMaster.java:2384) 2024-04-18 16:27:54,741 [INFO] [shutdown-hook-0] |app.DAGAppMaster|: DAGAppMasterShutdownHook invoked 2024-04-18 16:27:54,743 [INFO] [shutdown-hook-0] |service.AbstractService|: Service org.apache.tez.dag.app.DAGAppMaster failed in state STOPPED java.lang.NullPointerException: Cannot invoke "org.apache.tez.dag.app.rm.TaskSchedulerManager.initiateStop()" because "this.taskSchedulerManager" is null at org.apache.tez.dag.app.DAGAppMaster.initiateStop(DAGAppMaster.java:2111) at org.apache.tez.dag.app.DAGAppMaster.serviceStop(DAGAppMaster.java:2126) at org.apache.hadoop.service.AbstractService.stop(AbstractService.java:220) at org.apache.tez.dag.app.DAGAppMaster$DAGAppMasterShutdownHook.run(DAGAppMaster.java:2432) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) at java.base/java.lang.Thread.run(Thread.java:840) 2024-04-18 16:27:54,744 [WARN] [Thread-2] |util.ShutdownHookManager|: ShutdownHook 'DAGAppMasterShutdownHook' failed, java.util.concurrent.ExecutionException: java.lang.NullPointerException: Cannot invoke "org.apache.tez.dag.app.rm.TaskSchedulerManager.initiateStop()" because "this.taskSchedulerManager" is null java.util.concurrent.ExecutionException: java.lang.NullPointerException: Cannot invoke "org.apache.tez.dag.app.rm.TaskSchedulerManager.initiateStop()" because "this.taskSchedulerManager" is null at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:205) at org.apache.hadoop.util.ShutdownHookManager.executeShutdown(ShutdownHookManager.java:124) at org.apache.hadoop.util.ShutdownHookManager$1.run(ShutdownHookManager.java:95) Caused by: java.lang.NullPointerException: Cannot invoke "org.apache.tez.dag.app.rm.TaskSchedulerManager.initiateStop()" because "this.taskSchedulerManager" is null at org.apache.tez.dag.app.DAGAppMaster.initiateStop(DAGAppMaster.java:2111) at org.apache.tez.dag.app.DAGAppMaster.serviceStop(DAGAppMaster.java:2126) at org.apache.hadoop.service.AbstractService.stop(AbstractService.java:220) at org.apache.tez.dag.app.DAGAppMaster$DAGAppMasterShutdownHook.run(DAGAppMaster.java:2432) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) at java.base/java.lang.Thread.run(Thread.java:840) 2. Run Hive having protobuf 3.24.4 with hadoop 3.3.6 Containers fail with
[jira] [Created] (HADOOP-19163) Upgrade protobuf version to 3.24.4
Bilwa S T created HADOOP-19163: -- Summary: Upgrade protobuf version to 3.24.4 Key: HADOOP-19163 URL: https://issues.apache.org/jira/browse/HADOOP-19163 Project: Hadoop Common Issue Type: Bug Components: hadoop-thirdparty Reporter: Bilwa S T Assignee: Bilwa S T -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-19058) [JDK-17] TestCryptoOutputStreamClosing#testUnderlyingOutputStreamClosedWhenExceptionClosing fails
Bilwa S T created HADOOP-19058: -- Summary: [JDK-17] TestCryptoOutputStreamClosing#testUnderlyingOutputStreamClosedWhenExceptionClosing fails Key: HADOOP-19058 URL: https://issues.apache.org/jira/browse/HADOOP-19058 Project: Hadoop Common Issue Type: Sub-task Reporter: Bilwa S T Assignee: Bilwa S T -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-18986) Upgrade Zookeeper to 3.8.2
Bilwa S T created HADOOP-18986: -- Summary: Upgrade Zookeeper to 3.8.2 Key: HADOOP-18986 URL: https://issues.apache.org/jira/browse/HADOOP-18986 Project: Hadoop Common Issue Type: Improvement Reporter: Bilwa S T Assignee: Bilwa S T -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-18775) Add instrumentation access check for /logs servlet
Bilwa S T created HADOOP-18775: -- Summary: Add instrumentation access check for /logs servlet Key: HADOOP-18775 URL: https://issues.apache.org/jira/browse/HADOOP-18775 Project: Hadoop Common Issue Type: Bug Reporter: Bilwa S T Assignee: Bilwa S T Currently AdminAuthorizedServlet is configured for /logs which would always check if user has admin access. Just like other servelets, log should also have instrumentation check before the admin access check. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-18603) NPE in LdapAuthenticationHandler as disableHostNameVerification is never initialized
Bilwa S T created HADOOP-18603: -- Summary: NPE in LdapAuthenticationHandler as disableHostNameVerification is never initialized Key: HADOOP-18603 URL: https://issues.apache.org/jira/browse/HADOOP-18603 Project: Hadoop Common Issue Type: Bug Reporter: Bilwa S T Assignee: Bilwa S T Steps to reproduce this issue: Enable ldap auth with tls by configuring these in core-site.xml 1. hadoop.http.authentication.multi-scheme-auth-handler.schemes = basic 2. hadoop.http.authentication.multi-scheme-auth-handler.schemes.basic.handler = ldap 3. hadoop.http.authentication.ldap.enablestarttls = true -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-18566) Set Cache-Control no-store max-age as 0 header on all dynamic content
Bilwa S T created HADOOP-18566: -- Summary: Set Cache-Control no-store max-age as 0 header on all dynamic content Key: HADOOP-18566 URL: https://issues.apache.org/jira/browse/HADOOP-18566 Project: Hadoop Common Issue Type: Bug Reporter: Bilwa S T Assignee: Bilwa S T -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-17763) DistCp job fails when AM is killed
Bilwa S T created HADOOP-17763:
--
Summary: DistCp job fails when AM is killed
Key: HADOOP-17763
URL: https://issues.apache.org/jira/browse/HADOOP-17763
Project: Hadoop Common
Issue Type: Bug
Reporter: Bilwa S T
Assignee: Bilwa S T
Job fails as tasks fail with below exception
{code:java}
2021-06-11 18:48:47,047 | ERROR | IPC Server handler 0 on 27101 | Task:
attempt_1623387358383_0006_m_00_1000 - exited :
java.io.FileNotFoundException: File does not exist:
hdfs://hacluster/staging-dir/dsperf/.staging/_distcp-646531269/fileList.seq
at
org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1637)
at
org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1630)
at
org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at
org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1645)
at org.apache.hadoop.io.SequenceFile$Reader.(SequenceFile.java:1863)
at org.apache.hadoop.io.SequenceFile$Reader.(SequenceFile.java:1886)
at
org.apache.hadoop.mapreduce.lib.input.SequenceFileRecordReader.initialize(SequenceFileRecordReader.java:54)
at
org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.initialize(MapTask.java:560)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:798)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:347)
at org.apache.hadoop.mapred.YarnChild$1.run(YarnChild.java:183)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1761)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:177)
| TaskAttemptListenerImpl.java:304{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-17119) Netty upgrade to 9.4.x causes MR app fail with IOException
Bilwa S T created HADOOP-17119:
--
Summary: Netty upgrade to 9.4.x causes MR app fail with IOException
Key: HADOOP-17119
URL: https://issues.apache.org/jira/browse/HADOOP-17119
Project: Hadoop Common
Issue Type: Bug
Reporter: Bilwa S T
Assignee: Bilwa S T
I think we should catch IOException here instead of BindException in
HttpServer2#bindForPortRange
{code:java}
for(Integer port : portRanges) {
if (port == startPort) {
continue;
}
Thread.sleep(100);
listener.setPort(port);
try {
bindListener(listener);
return;
} catch (BindException ex) {
// Ignore exception. Move to next port.
ioException = ex;
}
}
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-16391) Duplicate values in rpcDetailedMetrics
Bilwa S T created HADOOP-16391: -- Summary: Duplicate values in rpcDetailedMetrics Key: HADOOP-16391 URL: https://issues.apache.org/jira/browse/HADOOP-16391 Project: Hadoop Common Issue Type: Bug Reporter: Bilwa S T Attachments: image-2019-06-25-20-30-15-395.png In RpcDetailedMetrics init is called two times . Once for deferredRpcrates and other one rates metrics which causes duplicate values in RM and NM metrics. !image-2019-06-25-20-30-15-395.png! -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-15523) Shell command timeout given is in seconds whereas it is taken as millisec while scheduling
Bilwa S T created HADOOP-15523:
--
Summary: Shell command timeout given is in seconds whereas it is
taken as millisec while scheduling
Key: HADOOP-15523
URL: https://issues.apache.org/jira/browse/HADOOP-15523
Project: Hadoop Common
Issue Type: Bug
Reporter: Bilwa S T
ShellBasedUnixGroupsMapping has a property
{{hadoop.security.groups.shell.command.timeout}} to control how long to wait
for the fetch groups command which can be configured in seconds. but while
scheduling the time taken is millisecs. so currently if u give value as 60s, it
is taken as 60ms.
{code:java}
timeout = conf.getTimeDuration(
CommonConfigurationKeys.
HADOOP_SECURITY_GROUP_SHELL_COMMAND_TIMEOUT_SECS,
CommonConfigurationKeys.
HADOOP_SECURITY_GROUP_SHELL_COMMAND_TIMEOUT_SECS_DEFAULT,
TimeUnit.SECONDS);{code}
Time unit given is in seconds but it should be millisecs
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
