[
https://issues.apache.org/jira/browse/HIVE-18327?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16374169#comment-16374169
]
Daniel Voros commented on HIVE-18327:
-------------------------------------
Attached patch #2. This fixes the checkstyle issues and does not upgrade hadoop
dependency.
The affected tests have all passed with Hadoop 3.0.0. The new failures in these
files all fail with similar errors ({{output/file.out.index does not exist}}):
* TestMetaStoreLimitPartitionRequest
* TestAutoPurgeTables
* TestEmbeddedThriftBinaryCLIService
* TestJdbcWithMiniHS2
An example stack trace from {{TestMetaStoreLimitPartitionRequest}}:
{code}
2018-02-22T15:12:54,852 WARN [Thread-1485] mapred.LocalJobRunner:
job_local1550446283_0016
java.lang.Exception:
org.apache.hadoop.mapreduce.task.reduce.Shuffle$ShuffleError: error in shuffle
in localfetcher#8
at
org.apache.hadoop.mapred.LocalJobRunner$Job.runTasks(LocalJobRunner.java:492)
~[hadoop-mapreduce-client-common-3.0.0.jar:?]
at
org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:559)
[hadoop-mapreduce-client-common-3.0.0.jar:?]
Caused by: org.apache.hadoop.mapreduce.task.reduce.Shuffle$ShuffleError: error
in shuffle in localfetcher#8
at
org.apache.hadoop.mapreduce.task.reduce.Shuffle.run(Shuffle.java:134)
~[hadoop-mapreduce-client-core-3.0.0.jar:?]
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:377)
~[hadoop-mapreduce-client-core-3.0.0.jar:?]
at
org.apache.hadoop.mapred.LocalJobRunner$Job$ReduceTaskRunnable.run(LocalJobRunner.java:347)
~[hadoop-mapreduce-client-common-3.0.0.jar:?]
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
~[?:1.8.0_102]
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
~[?:1.8.0_102]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
~[?:1.8.0_102]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
~[?:1.8.0_102]
at java.lang.Thread.run(Thread.java:745) ~[?:1.8.0_102]
Caused by: java.io.FileNotFoundException: File
/home/hiveptest/35.188.186.120-hiveptest-0/apache-github-source-source/itests/hive-unit/$%7Btest.tmp.dir%7D/hadoop-tmp/mapred/local/localRunner/hiveptest/jobcache/job_local1550446283_0016/attempt_local1550446283_0016_m_000000_0/output/file.out.index
does not exist
at
org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:641)
~[hadoop-common-3.0.0.jar:?]
at
org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:867)
~[hadoop-common-3.0.0.jar:?]
at
org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:631)
~[hadoop-common-3.0.0.jar:?]
at
org.apache.hadoop.fs.RawLocalFileSystem.open(RawLocalFileSystem.java:211)
~[hadoop-common-3.0.0.jar:?]
at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:949)
~[hadoop-common-3.0.0.jar:?]
at
org.apache.hadoop.io.SecureIOUtils.openFSDataInputStream(SecureIOUtils.java:152)
~[hadoop-common-3.0.0.jar:?]
at org.apache.hadoop.mapred.SpillRecord.<init>(SpillRecord.java:71)
~[hadoop-mapreduce-client-core-3.0.0.jar:?]
at org.apache.hadoop.mapred.SpillRecord.<init>(SpillRecord.java:62)
~[hadoop-mapreduce-client-core-3.0.0.jar:?]
at org.apache.hadoop.mapred.SpillRecord.<init>(SpillRecord.java:57)
~[hadoop-mapreduce-client-core-3.0.0.jar:?]
at
org.apache.hadoop.mapreduce.task.reduce.LocalFetcher.copyMapOutput(LocalFetcher.java:125)
~[hadoop-mapreduce-client-core-3.0.0.jar:?]
at
org.apache.hadoop.mapreduce.task.reduce.LocalFetcher.doCopy(LocalFetcher.java:103)
~[hadoop-mapreduce-client-core-3.0.0.jar:?]
at
org.apache.hadoop.mapreduce.task.reduce.LocalFetcher.run(LocalFetcher.java:86)
~[hadoop-mapreduce-client-core-3.0.0.jar:?]
{code}
There's no MiniHiveKdc involved in these tests, so I believe this must be a
separate issue (might be caused by the same Hadoop commit tho, I'm not sure
yet). I'll continue investigating and will probably open a new ticket to deal
with these failures.
> Remove the unnecessary HiveConf dependency for MiniHiveKdc
> ----------------------------------------------------------
>
> Key: HIVE-18327
> URL: https://issues.apache.org/jira/browse/HIVE-18327
> Project: Hive
> Issue Type: Test
> Components: Test
> Affects Versions: 3.0.0
> Reporter: Aihua Xu
> Assignee: Daniel Voros
> Priority: Major
> Attachments: HIVE-18327.1.patch, HIVE-18327.2.patch
>
>
> MiniHiveKdc takes HiveConf as input parameter while it's not needed. Remove
> the unnecessary HiveConf.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
