[
https://issues.apache.org/jira/browse/TEZ-4032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16779529#comment-16779529
]
Jonathan Eagles edited comment on TEZ-4032 at 2/27/19 5:24 PM:
---------------------------------------------------------------
The jenkins report did not get posted to this jira.
{code}
-1 overall
| Vote | Subsystem | Runtime | Comment
============================================================================
| 0 | reexec | 0m 19s | Docker mode activated.
| | | | Prechecks
| +1 | @author | 0m 0s | The patch does not contain any @author
| | | | tags.
| +1 | test4tests | 0m 0s | The patch appears to include 1 new or
| | | | modified test files.
| | | | master Compile Tests
| 0 | mvndep | 0m 19s | Maven dependency ordering for branch
| +1 | mvninstall | 4m 6s | master passed
| +1 | compile | 0m 49s | master passed
| +1 | checkstyle | 0m 41s | master passed
| -1 | findbugs | 0m 37s | tez-mapreduce in master has 2 extant
| | | | Findbugs warnings.
| +1 | javadoc | 0m 53s | master passed
| | | | Patch Compile Tests
| 0 | mvndep | 0m 17s | Maven dependency ordering for patch
| +1 | mvninstall | 0m 44s | the patch passed
| +1 | compile | 0m 44s | the patch passed
| +1 | javac | 0m 44s | the patch passed
| -0 | checkstyle | 0m 21s | tez-api: The patch generated 5 new + 73
| | | | unchanged - 0 fixed = 78 total (was 73)
| -0 | checkstyle | 0m 22s | tez-mapreduce: The patch generated 1 new
| | | | + 334 unchanged - 0 fixed = 335 total
| | | | (was 334)
| +1 | whitespace | 0m 0s | The patch has no whitespace issues.
| +1 | findbugs | 1m 41s | the patch passed
| +1 | javadoc | 0m 52s | the patch passed
| | | | Other Tests
| +1 | unit | 1m 47s | tez-api in the patch passed.
| +1 | unit | 1m 8s | tez-mapreduce in the patch passed.
| +1 | asflicense | 0m 32s | The patch does not generate ASF License
| | | | warnings.
| | | 18m 8s |
|| Subsystem || Report/Notes ||
============================================================================
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/tez:d4a62de |
| JIRA Issue | TEZ-4032 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12960354/TEZ-4032.003.patch |
| Optional Tests | dupname asflicense javac javadoc unit findbugs
checkstyle compile |
| uname | Linux e3b133699a67 3.13.0-153-generic #203-Ubuntu SMP Thu Jun 14
08:52:28 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | master / 0093f8b |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_191 |
| findbugs | v3.0.1 |
| findbugs |
https://builds.apache.org/job/PreCommit-TEZ-Build/104/artifact/out/branch-findbugs-tez-mapreduce-warnings.html
|
| checkstyle |
https://builds.apache.org/job/PreCommit-TEZ-Build/104/artifact/out/diff-checkstyle-tez-api.txt
|
| checkstyle |
https://builds.apache.org/job/PreCommit-TEZ-Build/104/artifact/out/diff-checkstyle-tez-mapreduce.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-TEZ-Build/104/testReport/ |
| Max. process+thread count | 224 (vs. ulimit of 10000) |
| modules | C: tez-api tez-mapreduce U: . |
| Console output |
https://builds.apache.org/job/PreCommit-TEZ-Build/104/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
{code}
https://builds.apache.org/job/PreCommit-TEZ-Build/104/artifact/out/diff-checkstyle-tez-mapreduce.txt
{code}
./tez-mapreduce/src/main/java/org/apache/tez/mapreduce/hadoop/MRJobConfig.java:305:
public static final String JOB_NAMENODES_TOKEN_RENEWAL_EXCLUDE =
"mapreduce.job.hdfs-servers.token-renewal.exclude";:3: Redundant 'public'
modifier. [RedundantModifier]
{code}
https://builds.apache.org/job/PreCommit-TEZ-Build/104/artifact/out/diff-checkstyle-tez-api.txt
{code}
./tez-api/src/main/java/org/apache/tez/client/TezClientUtils.java:499:
populateTokenCache(conf,sessionCreds);:28: ',' is not followed by whitespace.
[WhitespaceAfter]
./tez-api/src/main/java/org/apache/tez/client/TezClientUtils.java:727:
String [] nameNodes = conf.getStrings(TezConfiguration.TEZ_JOB_FS_SERVERS);:12:
'String' is followed by whitespace. [NoWhitespaceAfter]
./tez-api/src/main/java/org/apache/tez/client/TezClientUtils.java:731:
Path [] ps = new Path[nameNodes.length];:12: 'Path' is followed by whitespace.
[NoWhitespaceAfter]
./tez-api/src/main/java/org/apache/tez/common/security/TokenCache.java:106:
String [] nns =:12: 'String' is followed by whitespace. [NoWhitespaceAfter]
./tez-api/src/test/java/org/apache/tez/common/security/TestTokenCache.java:118:
conf.set("tez.job.fs-servers.token-renewal.exclude","dir");:56: ',' is not
followed by whitespace. [WhitespaceAfter]
{code}
https://builds.apache.org/job/PreCommit-TEZ-Build/104/artifact/out/branch-findbugs-tez-mapreduce-warnings.html
{code}
DLS Dead store to unknownAppId in
org.apache.tez.mapreduce.client.NotRunningJob.getUnknownApplicationReport()
Bug type DLS_DEAD_LOCAL_STORE (click for details)
In class org.apache.tez.mapreduce.client.NotRunningJob
In method
org.apache.tez.mapreduce.client.NotRunningJob.getUnknownApplicationReport()
Local variable named unknownAppId
At NotRunningJob.java:[line 83]
DLS Dead store to unknownAttemptId in
org.apache.tez.mapreduce.client.NotRunningJob.getUnknownApplicationReport()
Bug type DLS_DEAD_LOCAL_STORE (click for details)
In class org.apache.tez.mapreduce.client.NotRunningJob
In method
org.apache.tez.mapreduce.client.NotRunningJob.getUnknownApplicationReport()
Local variable named unknownAttemptId
At NotRunningJob.java:[line 85]
Details
DLS_DEAD_LOCAL_STORE: Dead store to local variable
This instruction assigns a value to a local variable, but the value is not read
or used in any subsequent instruction. Often, this indicates an error, because
the value computed is never used.
Note that Sun's javac compiler often generates dead stores for final local
variables. Because FindBugs is a bytecode-based tool, there is no easy way to
eliminate these false positives.
{code}
was (Author: jeagles):
The jenkins report did not get posted to this jira.
{code}
-1 overall
| Vote | Subsystem | Runtime | Comment
============================================================================
| 0 | reexec | 0m 19s | Docker mode activated.
| | | | Prechecks
| +1 | @author | 0m 0s | The patch does not contain any @author
| | | | tags.
| +1 | test4tests | 0m 0s | The patch appears to include 1 new or
| | | | modified test files.
| | | | master Compile Tests
| 0 | mvndep | 0m 19s | Maven dependency ordering for branch
| +1 | mvninstall | 4m 6s | master passed
| +1 | compile | 0m 49s | master passed
| +1 | checkstyle | 0m 41s | master passed
| -1 | findbugs | 0m 37s | tez-mapreduce in master has 2 extant
| | | | Findbugs warnings.
| +1 | javadoc | 0m 53s | master passed
| | | | Patch Compile Tests
| 0 | mvndep | 0m 17s | Maven dependency ordering for patch
| +1 | mvninstall | 0m 44s | the patch passed
| +1 | compile | 0m 44s | the patch passed
| +1 | javac | 0m 44s | the patch passed
| -0 | checkstyle | 0m 21s | tez-api: The patch generated 5 new + 73
| | | | unchanged - 0 fixed = 78 total (was 73)
| -0 | checkstyle | 0m 22s | tez-mapreduce: The patch generated 1 new
| | | | + 334 unchanged - 0 fixed = 335 total
| | | | (was 334)
| +1 | whitespace | 0m 0s | The patch has no whitespace issues.
| +1 | findbugs | 1m 41s | the patch passed
| +1 | javadoc | 0m 52s | the patch passed
| | | | Other Tests
| +1 | unit | 1m 47s | tez-api in the patch passed.
| +1 | unit | 1m 8s | tez-mapreduce in the patch passed.
| +1 | asflicense | 0m 32s | The patch does not generate ASF License
| | | | warnings.
| | | 18m 8s |
|| Subsystem || Report/Notes ||
============================================================================
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/tez:d4a62de |
| JIRA Issue | TEZ-4032 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12960354/TEZ-4032.003.patch |
| Optional Tests | dupname asflicense javac javadoc unit findbugs
checkstyle compile |
| uname | Linux e3b133699a67 3.13.0-153-generic #203-Ubuntu SMP Thu Jun 14
08:52:28 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | master / 0093f8b |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_191 |
| findbugs | v3.0.1 |
| findbugs |
https://builds.apache.org/job/PreCommit-TEZ-Build/104/artifact/out/branch-findbugs-tez-mapreduce-warnings.html
|
| checkstyle |
https://builds.apache.org/job/PreCommit-TEZ-Build/104/artifact/out/diff-checkstyle-tez-api.txt
|
| checkstyle |
https://builds.apache.org/job/PreCommit-TEZ-Build/104/artifact/out/diff-checkstyle-tez-mapreduce.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-TEZ-Build/104/testReport/ |
| Max. process+thread count | 224 (vs. ulimit of 10000) |
| modules | C: tez-api tez-mapreduce U: . |
| Console output |
https://builds.apache.org/job/PreCommit-TEZ-Build/104/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
{code}
https://builds.apache.org/job/PreCommit-TEZ-Build/104/artifact/out/diff-checkstyle-tez-mapreduce.txt
{code}
./tez-mapreduce/src/main/java/org/apache/tez/mapreduce/hadoop/MRJobConfig.java:305:
public static final String JOB_NAMENODES_TOKEN_RENEWAL_EXCLUDE =
"mapreduce.job.hdfs-servers.token-renewal.exclude";:3: Redundant 'public'
modifier. [RedundantModifier]
{code}
https://builds.apache.org/job/PreCommit-TEZ-Build/104/artifact/out/diff-checkstyle-tez-api.txt
{code}
./tez-api/src/main/java/org/apache/tez/client/TezClientUtils.java:499:
populateTokenCache(conf,sessionCreds);:28: ',' is not followed by whitespace.
[WhitespaceAfter]
./tez-api/src/main/java/org/apache/tez/client/TezClientUtils.java:727:
String [] nameNodes = conf.getStrings(TezConfiguration.TEZ_JOB_FS_SERVERS);:12:
'String' is followed by whitespace. [NoWhitespaceAfter]
./tez-api/src/main/java/org/apache/tez/client/TezClientUtils.java:731:
Path [] ps = new Path[nameNodes.length];:12: 'Path' is followed by whitespace.
[NoWhitespaceAfter]
./tez-api/src/main/java/org/apache/tez/common/security/TokenCache.java:106:
String [] nns =:12: 'String' is followed by whitespace. [NoWhitespaceAfter]
./tez-api/src/test/java/org/apache/tez/common/security/TestTokenCache.java:118:
conf.set("tez.job.fs-servers.token-renewal.exclude","dir");:56: ',' is not
followed by whitespace. [WhitespaceAfter]
{code}
https://builds.apache.org/job/PreCommit-TEZ-Build/104/artifact/out/branch-findbugs-tez-mapreduce-warnings.html
{code}
{code}
> TEZ will throw "Client cannot authenticate via:[TOKEN, KERBEROS]" when used
> with HDFS federation(non viewfs, only hdfs schema used).
> --------------------------------------------------------------------------------------------------------------------------------------
>
> Key: TEZ-4032
> URL: https://issues.apache.org/jira/browse/TEZ-4032
> Project: Apache Tez
> Issue Type: Bug
> Affects Versions: 0.9.1
> Reporter: zhangbutao
> Priority: Major
> Attachments: TEZ-4032.001.patch, TEZ-4032.002.patch,
> TEZ-4032.003.patch
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> I execute hive tez job in HDFS federation and kerberos. The hadoop cluster
> has multiple namespace (hdfs://ns1,hdfs://ns2,hdfs://ns3 ...)and we don't
> use viewfs schema. Hive tez job will throw error as follows when the table
> is created in hdfs://ns2 (default configuration fs.defaluFS=hdfs://ns1)：
> {code:java}
> 2019-01-21 15:43:46,507 [WARN] [TezChild] |ipc.Client|: Exception encountered
> while connecting to the server :
> org.apache.hadoop.security.AccessControlException: Client cannot authenticate
> via:[TOKEN, KERBEROS]
> 2019-01-21 15:43:46,507 [INFO] [TezChild] |retry.RetryInvocationHandler|:
> java.io.IOException: DestHost:destPort docker5.cmss.com:8020 ,
> LocalHost:localPort docker1.cmss.com/10.254.10.116:0. Failed on local
> exception: java.io.IOException:
> org.apache.hadoop.security.AccessControlException: Client cannot authenticate
> via:[TOKEN, KERBEROS], while invoking
> ClientNamenodeProtocolTranslatorPB.getFileInfo over
> docker5.cmss.com/10.254.2.106:8020 after 14 failover attempts. Trying to
> failover after sleeping for 10827ms.
> 2019-01-21 15:43:57,338 [WARN] [TezChild] |ipc.Client|: Exception encountered
> while connecting to the server :
> org.apache.hadoop.security.AccessControlException: Client cannot authenticate
> via:[TOKEN, KERBEROS]
> 2019-01-21 15:43:57,363 [ERROR] [TezChild] |tez.MapRecordSource|:
> org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while
> processing writable (null)
> at
> org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:568)
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.processRow(MapRecordSource.java:92)
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.pushRecord(MapRecordSource.java:76)
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.run(MapRecordProcessor.java:419)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:267)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:250)
> at
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:374)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:73)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:61)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1682)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:61)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:37)
> at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
> at
> com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:108)
> at
> com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:41)
> at
> com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:77)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
> org.apache.hadoop.hive.ql.metadata.HiveException: java.io.IOException:
> DestHost:destPort docker4.cmss.com:8020 , LocalHost:localPort
> docker1.cmss.com/10.254.10.116:0. Failed on local exception:
> java.io.IOException: org.apache.hadoop.security.AccessControlException:
> Client cannot authenticate via:[TOKEN, KERBEROS]
> at
> org.apache.hadoop.hive.ql.exec.FileSinkOperator.createBucketFiles(FileSinkOperator.java:742)
> at
> org.apache.hadoop.hive.ql.exec.FileSinkOperator.process(FileSinkOperator.java:897)
> at
> org.apache.hadoop.hive.ql.exec.Operator.baseForward(Operator.java:995)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:941)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:928)
> at
> org.apache.hadoop.hive.ql.exec.SelectOperator.process(SelectOperator.java:95)
> at
> org.apache.hadoop.hive.ql.exec.Operator.baseForward(Operator.java:995)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:941)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:928)
> at
> org.apache.hadoop.hive.ql.exec.UDTFOperator.forwardUDTFOutput(UDTFOperator.java:133)
> at
> org.apache.hadoop.hive.ql.udf.generic.UDTFCollector.collect(UDTFCollector.java:45)
> at
> org.apache.hadoop.hive.ql.udf.generic.GenericUDTF.forward(GenericUDTF.java:110)
> at
> org.apache.hadoop.hive.ql.udf.generic.GenericUDTFInline.process(GenericUDTFInline.java:64)
> at
> org.apache.hadoop.hive.ql.exec.UDTFOperator.process(UDTFOperator.java:116)
> at
> org.apache.hadoop.hive.ql.exec.Operator.baseForward(Operator.java:995)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:941)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:928)
> at
> org.apache.hadoop.hive.ql.exec.SelectOperator.process(SelectOperator.java:95)
> at
> org.apache.hadoop.hive.ql.exec.Operator.baseForward(Operator.java:995)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:941)
> at
> org.apache.hadoop.hive.ql.exec.TableScanOperator.process(TableScanOperator.java:125)
> at
> org.apache.hadoop.hive.ql.exec.MapOperator$MapOpCtx.forward(MapOperator.java:153)
> at
> org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:555)
> ... 20 more
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
