[
https://issues.apache.org/jira/browse/HIVE-3519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13469962#comment-13469962
]
Thejas M Nair commented on HIVE-3519:
-------------------------------------
The previous patch exposed another problem with related code on windows.
TestSymlinkTextInputFormat throws the following exception -
{code}
java.lang.NullPointerException
at
org.apache.hadoop.hive.ql.io.HiveFileFormatUtils.doGetWorksFromPath(HiveFileFormatUtils.java:420)
at
org.apache.hadoop.hive.ql.io.CombineHiveInputFormat.getSplits(CombineHiveInputFormat.java:349)
at
org.apache.hadoop.hive.ql.io.TestSymlinkTextInputFormat.testCombine(TestSymlinkTextInputFormat.java:187)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at junit.framework.TestCase.runTest(TestCase.java:154)
at junit.framework.TestCase.runBare(TestCase.java:127)
at junit.framework.TestResult$1.protect(TestResult.java:106)
at junit.framework.TestResult.runProtected(TestResult.java:124)
at junit.framework.TestResult.run(TestResult.java:109)
at junit.framework.TestCase.run(TestCase.java:118)
at junit.framework.TestSuite.runTest(TestSuite.java:208)
at junit.framework.TestSuite.run(TestSuite.java:203)
at
org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRunner.java:520)
at
org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.launch(JUnitTestRunner.java:1060)
at
org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.main(JUnitTestRunner.java:911)
PREHOOK: query: drop table text_symlink_text
{code}
This is because in HiveFileFormatUtils.populateNewPartitionDesc it was looking
up a path whose string representation has an additional "/" before the drive
letter. While the strings that were keys in the Map that did not have the "/C".
Eg - It was doing lookup on map-
{C:/Users/daijy/hive-monarch_btest/ql/TestSymlinkTextInputFormat/datadir1/combinefile1_1=[text_symlink_text],
C:/Users/daijy/hive-monarch_btest/ql/TestSymlinkTextInputFormat/datadir2/combinefile2_1=[text_symlink_text]}
with
with
file:/C:/Users/daijy/hive-monarch_btest/ql/TestSymlinkTextInputFormat/datadir1/combinefile1_1
with
/C:/Users/daijy/hive-monarch_btest/ql/TestSymlinkTextInputFormat/datadir1/combinefile1_1
> partition to directory comparison in CombineHiveInputFormat needs to accept
> partitions dir without scheme
> ---------------------------------------------------------------------------------------------------------
>
> Key: HIVE-3519
> URL: https://issues.apache.org/jira/browse/HIVE-3519
> Project: Hive
> Issue Type: Bug
> Reporter: Thejas M Nair
> Assignee: Thejas M Nair
> Fix For: 0.10.0
>
> Attachments: HIVE-3519.1.patch
>
>
> TestSymlinkTextInputFormat.testCombine throws following exception. The test
> case is just printing out the stacktrace when that happens instead of failing.
> {code}
> java.io.IOException: cannot find dir =
> file:/Users/thejas/hive-trunk/ql/TestSymlinkTextInputFormat/datadir1/combinefile1_1
> in pathToPartitionInfo:
> [/Users/thejas/hive-trunk/ql/TestSymlinkTextInputFormat/datadir2/combinefile2_1,
>
> /Users/thejas/hive-trunk/ql/TestSymlinkTextInputFormat/datadir1/combinefile1_1]
> at
> org.apache.hadoop.hive.ql.io.HiveFileFormatUtils.getPartitionDescFromPathRecursively(HiveFileFormatUtils.java:288)
> at
> org.apache.hadoop.hive.ql.io.HiveFileFormatUtils.getPartitionDescFromPathRecursively(HiveFileFormatUtils.java:256)
> at
> org.apache.hadoop.hive.ql.io.CombineHiveInputFormat.getSplits(CombineHiveInputFormat.java:289)
> at
> org.apache.hadoop.hive.ql.io.TestSymlinkTextInputFormat.testCombine(TestSymlinkTextInputFormat.java:186)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at junit.framework.TestCase.runTest(TestCase.java:168)
> at junit.framework.TestCase.runBare(TestCase.java:134)
> at junit.framework.TestResult$1.protect(TestResult.java:110)
> at junit.framework.TestResult.runProtected(TestResult.java:128)
> at junit.framework.TestResult.run(TestResult.java:113)
> at junit.framework.TestCase.run(TestCase.java:124)
> at junit.framework.TestSuite.runTest(TestSuite.java:232)
> at junit.framework.TestSuite.run(TestSuite.java:227)
> at
> org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRunner.java:520)
> at
> org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.launch(JUnitTestRunner.java:1060)
> at
> org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.main(JUnitTestRunner.java:911)
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
