[jira] [Commented] (HIVE-4448) Fix metastore warehouse incorrect location on Windows in unit tests
[ https://issues.apache.org/jira/browse/HIVE-4448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13853448#comment-13853448 ] Jason Dere commented on HIVE-4448: -- The same changes are also a part of HIVE-5176, trying to get that one pushed through. > Fix metastore warehouse incorrect location on Windows in unit tests > --- > > Key: HIVE-4448 > URL: https://issues.apache.org/jira/browse/HIVE-4448 > Project: Hive > Issue Type: Bug > Components: Testing Infrastructure >Affects Versions: 0.11.0 > Environment: Windows >Reporter: Shuaishuai Nie >Assignee: Shuaishuai Nie > Attachments: HIVE-4448.1.patch > > > Unit test cases which not using QTestUtil will pass incompatible Windows path > of METASTOREWAREHOUSE to HiveConf which result in creating the > /test/data/warehouse folder in the wrong location in Windows. This folder > will not be deleted at the beginning of the unit test and the content will > cause failure of unit tests if run the same test case repeatedly. The root > cause of this problem is for path like this > "pfile://C:\hive\build\ql/test/data/warehouse", the "C:\hive\build\" part > will be parsed as authority of the path and removed from the path string. The > patch will fix this problem and make the unit test result consistent between > Windows and Linux. -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Commented] (HIVE-4448) Fix metastore warehouse incorrect location on Windows in unit tests
[ https://issues.apache.org/jira/browse/HIVE-4448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13691542#comment-13691542 ] Shuaishuai Nie commented on HIVE-4448: -- Hi [~ashutoshgupt...@gmail.com] Thanks for the comments. You are right, since HiveConf.ConfVars.METASTOREWAREHOUSE is a path on hdfs, so the "convertPathsFromWindowsToHdfs" not only change the file separator, it also remove the drive name in Windows path. Here is an example for HiveConf.ConfVars.METASTOREWAREHOUSE before and after the conversion: before: "pfile://E:\HIVE-4448\hive-trunk\build\ql/test/data/warehouse" (this path will cause confusion and "E:\HIVE-4448\hive-trunk\build\ql" will be removes as authority of the path. We need the /HIVE-4448/hive-trunk/build/ql folder structure when creating the warehouse folder) after conversion: "pfile:///HIVE-4448/hive-trunk/build/ql/test/data/warehouse" (valid hdfs path which will create the /test/data/warehouse folder in the correct location on windows machine) by only use the proper file separators, the driver name in Windows path like E:\ will still cause confusion in HDFS path. In addition, the reason I use the function "convertPathsFromWindowsToHdfs" is because the same function is in the class QTestUtil. This insure the same behavior of path conversion in all unit tests (a few unit test does not use QTestUtil which make the fix like this easier to apply to all of the tests) > Fix metastore warehouse incorrect location on Windows in unit tests > --- > > Key: HIVE-4448 > URL: https://issues.apache.org/jira/browse/HIVE-4448 > Project: Hive > Issue Type: Bug > Components: Testing Infrastructure >Affects Versions: 0.11.0 > Environment: Windows >Reporter: Shuaishuai Nie >Assignee: Shuaishuai Nie > Attachments: HIVE-4448.1.patch > > > Unit test cases which not using QTestUtil will pass incompatible Windows path > of METASTOREWAREHOUSE to HiveConf which result in creating the > /test/data/warehouse folder in the wrong location in Windows. This folder > will not be deleted at the beginning of the unit test and the content will > cause failure of unit tests if run the same test case repeatedly. The root > cause of this problem is for path like this > "pfile://C:\hive\build\ql/test/data/warehouse", the "C:\hive\build\" part > will be parsed as authority of the path and removed from the path string. The > patch will fix this problem and make the unit test result consistent between > Windows and Linux. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4448) Fix metastore warehouse incorrect location on Windows in unit tests
[
https://issues.apache.org/jira/browse/HIVE-4448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13690046#comment-13690046
]
Ashutosh Chauhan commented on HIVE-4448:
You are changing following four strings, which I think is not necessary.
* HiveConf.ConfVars.METASTOREWAREHOUSE : This is a path on hdfs.
C:\\user\hive\warehouse won't make sense here.
* System.getProperty("test.tmp.dir") This is coming from build-common.xml. We
should rather fix it there, by properly using file seprators ${file.separator}
More info: http://ant.apache.org/manual/Types/mapper.html
* System.getProperty("test.src.data.dir") : Same as above.
* HiveConf.ConfVars.SCRATCHDIR : This fixed via HIVE-4609
> Fix metastore warehouse incorrect location on Windows in unit tests
> ---
>
> Key: HIVE-4448
> URL: https://issues.apache.org/jira/browse/HIVE-4448
> Project: Hive
> Issue Type: Bug
> Components: Testing Infrastructure
>Affects Versions: 0.11.0
> Environment: Windows
>Reporter: Shuaishuai Nie
>Assignee: Shuaishuai Nie
> Attachments: HIVE-4448.1.patch
>
>
> Unit test cases which not using QTestUtil will pass incompatible Windows path
> of METASTOREWAREHOUSE to HiveConf which result in creating the
> /test/data/warehouse folder in the wrong location in Windows. This folder
> will not be deleted at the beginning of the unit test and the content will
> cause failure of unit tests if run the same test case repeatedly. The root
> cause of this problem is for path like this
> "pfile://C:\hive\build\ql/test/data/warehouse", the "C:\hive\build\" part
> will be parsed as authority of the path and removed from the path string. The
> patch will fix this problem and make the unit test result consistent between
> Windows and Linux.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
