[jira] [Resolved] (HIVE-3197) Hive compile errors under Java 7 (JDBC 4.1)
[
https://issues.apache.org/jira/browse/HIVE-3197?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ashutosh Chauhan resolved HIVE-3197.
Resolution: Fixed
Release Note: Dupe of HIVE-3384
Hive compile errors under Java 7 (JDBC 4.1)
---
Key: HIVE-3197
URL: https://issues.apache.org/jira/browse/HIVE-3197
Project: Hive
Issue Type: Bug
Environment: Ubuntu 12.04
Reporter: Brian Bloniarz
Hi, I've been trying to compile Hive trunk from source and getting failures:
{code}
[javac]
hive-svn/jdbc/src/java/org/apache/hadoop/hive/jdbc/HiveCallableStatement.java:48:
error: HiveCallableStatement is not abstract and does not override abstract
method TgetObject(String,ClassT) in CallableStatement
[javac] public class HiveCallableStatement implements
java.sql.CallableStatement {
[javac]^
[javac] where T is a type-variable:
[javac] T extends Object declared in method
TgetObject(String,ClassT)
{code}
I think this is because JDBC 4.1 is part of Java 7, and is not
source-compatible with older JDBC versions. Any chance you guys could add
JDBC 4.1 support?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3384) HIVE JDBC module won't compile under JDK1.7 as new methods added in JDBC specification
[ https://issues.apache.org/jira/browse/HIVE-3384?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13505283#comment-13505283 ] Phabricator commented on HIVE-3384: --- ashutoshc has requested changes to the revision [jira] [HIVE-3384] HIVE JDBC module won't compile under JDK1.7 as new methods added in JDBC specification. Patch looks good. Couple of minor comments. INLINE COMMENTS jdbc/src/java/org/apache/hadoop/hive/jdbc/HiveConnection.java:49 Unneeded import. jdbc/src/java/org/apache/hadoop/hive/jdbc/HiveDataSource.java:29 Duplicate import metastore/build.xml:106 Why UseSplitVerifier is required? Unless there is a strong reason for this I will recommend to get rid of this, since this flag is specific to Oracle JVM, we should avoid vendor specific flag for better portability. REVISION DETAIL https://reviews.facebook.net/D6873 BRANCH trunk To: cdrome, weidongbian, JIRA, ashutoshc, mbautin HIVE JDBC module won't compile under JDK1.7 as new methods added in JDBC specification -- Key: HIVE-3384 URL: https://issues.apache.org/jira/browse/HIVE-3384 Project: Hive Issue Type: Bug Components: JDBC Reporter: Weidong Bian Priority: Minor Attachments: D6873-0.9.1.patch, D6873.1.patch, HIVE-3384.2.patch, HIVE-3384.patch, HIVE-JDK7-JDBC.patch jdbc module couldn't be compiled with jdk7 as it adds some abstract method in the JDBC specification some error info: error: HiveCallableStatement is not abstract and does not override abstract method TgetObject(String,ClassT) in CallableStatement . . . -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3748) QTestUtil should correctly find data files when running in the build directory
[ https://issues.apache.org/jira/browse/HIVE-3748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13505286#comment-13505286 ] Ashutosh Chauhan commented on HIVE-3748: From description, I am not able to figure out in what scenarios test cases fail. I always run test cases from src root dir, why would you want to run them from build dir? QTestUtil should correctly find data files when running in the build directory -- Key: HIVE-3748 URL: https://issues.apache.org/jira/browse/HIVE-3748 Project: Hive Issue Type: Improvement Reporter: Mikhail Bautin Priority: Minor Attachments: D7005.1.patch Some parts of the the TestCliDriver test suite (i.e. some jar lookups) require that the current directory is set to the build directory. This makes QTestUtil correctly find data files when running either in the Hive source root or in the build directory. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-3748) QTestUtil should correctly find data files when running in the build directory
[ https://issues.apache.org/jira/browse/HIVE-3748?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-3748: --- Assignee: Mikhail Bautin QTestUtil should correctly find data files when running in the build directory -- Key: HIVE-3748 URL: https://issues.apache.org/jira/browse/HIVE-3748 Project: Hive Issue Type: Improvement Reporter: Mikhail Bautin Assignee: Mikhail Bautin Priority: Minor Attachments: D7005.1.patch Some parts of the the TestCliDriver test suite (i.e. some jar lookups) require that the current directory is set to the build directory. This makes QTestUtil correctly find data files when running either in the Hive source root or in the build directory. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-3552) HIVE-3552 performant manner for performing cubes/rollups/grouping sets for a high number of grouping set keys
[ https://issues.apache.org/jira/browse/HIVE-3552?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Namit Jain updated HIVE-3552: - Summary: HIVE-3552 performant manner for performing cubes/rollups/grouping sets for a high number of grouping set keys (was: performant manner for performing cubes and rollups in case of less aggregation) HIVE-3552 performant manner for performing cubes/rollups/grouping sets for a high number of grouping set keys - Key: HIVE-3552 URL: https://issues.apache.org/jira/browse/HIVE-3552 Project: Hive Issue Type: New Feature Components: Query Processor Reporter: Namit Jain Assignee: Namit Jain This is a follow up for HIVE-3433. Had a offline discussion with Sambavi - she pointed out a scenario where the implementation in HIVE-3433 will not scale. Assume that the user is performing a cube on many columns, say '8' columns. So, each row would generate 256 rows for the hash table, which may kill the current group by implementation. A better implementation would be to add an additional stage - in the first stage perform the group by assuming there was no cube. Ad another stage, where you would perform the cube. The assumption is that the group by would have decreased the output data significantly. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3709) Stop storing default ConfVars in temp file
[
https://issues.apache.org/jira/browse/HIVE-3709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13505292#comment-13505292
]
Carl Steinbach commented on HIVE-3709:
--
Another option for fixing this problem is to give administrators the ability to
specify the location where the temporary file should be written (for setuid
processes running on POSIX systems the conventional location is somewhere under
/var). For example we could add a configuration property named
hive.process.local.temporary.dir and have it default to
${user.home}/.hive/${process_id}.
Stop storing default ConfVars in temp file
--
Key: HIVE-3709
URL: https://issues.apache.org/jira/browse/HIVE-3709
Project: Hive
Issue Type: Improvement
Components: Configuration
Affects Versions: 0.10.0
Reporter: Kevin Wilfong
Assignee: Kevin Wilfong
Attachments: HIVE-3709.1.patch.txt, HIVE-3709.2.patch.txt,
HIVE-3709.3.patch.txt
To work around issues with Hadoop's Configuration object, specifically it's
addResource(InputStream), default configurations are written to a temp file
(I think HIVE-2362 introduced this).
This, however, introduces the problem that once that file is deleted from
/tmp the client crashes. This is particularly problematic for long running
services like the metastore server.
Writing a custom InputStream to deal with the problems in the Configuration
object should provide a work around, which does not introduce a time bomb
into Hive.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-3552) HIVE-3552 performant manner for performing cubes/rollups/grouping sets for a high number of grouping set keys
[ https://issues.apache.org/jira/browse/HIVE-3552?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Namit Jain updated HIVE-3552: - Attachment: hive.3552.1.patch HIVE-3552 performant manner for performing cubes/rollups/grouping sets for a high number of grouping set keys - Key: HIVE-3552 URL: https://issues.apache.org/jira/browse/HIVE-3552 Project: Hive Issue Type: New Feature Components: Query Processor Reporter: Namit Jain Assignee: Namit Jain Attachments: hive.3552.1.patch This is a follow up for HIVE-3433. Had a offline discussion with Sambavi - she pointed out a scenario where the implementation in HIVE-3433 will not scale. Assume that the user is performing a cube on many columns, say '8' columns. So, each row would generate 256 rows for the hash table, which may kill the current group by implementation. A better implementation would be to add an additional stage - in the first stage perform the group by assuming there was no cube. Ad another stage, where you would perform the cube. The assumption is that the group by would have decreased the output data significantly. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-3552) HIVE-3552 performant manner for performing cubes/rollups/grouping sets for a high number of grouping set keys
[ https://issues.apache.org/jira/browse/HIVE-3552?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Namit Jain updated HIVE-3552: - Status: Patch Available (was: Open) HIVE-3552 performant manner for performing cubes/rollups/grouping sets for a high number of grouping set keys - Key: HIVE-3552 URL: https://issues.apache.org/jira/browse/HIVE-3552 Project: Hive Issue Type: New Feature Components: Query Processor Reporter: Namit Jain Assignee: Namit Jain Attachments: hive.3552.1.patch This is a follow up for HIVE-3433. Had a offline discussion with Sambavi - she pointed out a scenario where the implementation in HIVE-3433 will not scale. Assume that the user is performing a cube on many columns, say '8' columns. So, each row would generate 256 rows for the hash table, which may kill the current group by implementation. A better implementation would be to add an additional stage - in the first stage perform the group by assuming there was no cube. Ad another stage, where you would perform the cube. The assumption is that the group by would have decreased the output data significantly. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-3552) HIVE-3552 performant manner for performing cubes/rollups/grouping sets for a high number of grouping set keys
[ https://issues.apache.org/jira/browse/HIVE-3552?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Namit Jain updated HIVE-3552: - Description: This is a follow up for HIVE-3433. Had a offline discussion with Sambavi - she pointed out a scenario where the implementation in HIVE-3433 will not scale. Assume that the user is performing a cube on many columns, say '8' columns. So, each row would generate 256 rows for the hash table, which may kill the current group by implementation. A better implementation would be to add an additional mr job - in the first mr job perform the group by assuming there was no cube. Add another mr job, where you would perform the cube. The assumption is that the group by would have decreased the output data significantly, and the rows would appear in the order of grouping keys which has a higher probability of hitting the hash table. was: This is a follow up for HIVE-3433. Had a offline discussion with Sambavi - she pointed out a scenario where the implementation in HIVE-3433 will not scale. Assume that the user is performing a cube on many columns, say '8' columns. So, each row would generate 256 rows for the hash table, which may kill the current group by implementation. A better implementation would be to add an additional stage - in the first stage perform the group by assuming there was no cube. Ad another stage, where you would perform the cube. The assumption is that the group by would have decreased the output data significantly. HIVE-3552 performant manner for performing cubes/rollups/grouping sets for a high number of grouping set keys - Key: HIVE-3552 URL: https://issues.apache.org/jira/browse/HIVE-3552 Project: Hive Issue Type: New Feature Components: Query Processor Reporter: Namit Jain Assignee: Namit Jain Attachments: hive.3552.1.patch This is a follow up for HIVE-3433. Had a offline discussion with Sambavi - she pointed out a scenario where the implementation in HIVE-3433 will not scale. Assume that the user is performing a cube on many columns, say '8' columns. So, each row would generate 256 rows for the hash table, which may kill the current group by implementation. A better implementation would be to add an additional mr job - in the first mr job perform the group by assuming there was no cube. Add another mr job, where you would perform the cube. The assumption is that the group by would have decreased the output data significantly, and the rows would appear in the order of grouping keys which has a higher probability of hitting the hash table. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HIVE-3531) Simple lock manager for dedicated hive server
[ https://issues.apache.org/jira/browse/HIVE-3531?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Carl Steinbach resolved HIVE-3531. -- Resolution: Fixed Fix Version/s: 0.11 Hadoop Flags: Reviewed Committed to trunk. Thanks Navis! Simple lock manager for dedicated hive server - Key: HIVE-3531 URL: https://issues.apache.org/jira/browse/HIVE-3531 Project: Hive Issue Type: Improvement Components: Locking, Server Infrastructure Reporter: Navis Assignee: Navis Priority: Trivial Fix For: 0.11 Attachments: HIVE-3531.D5871.1.patch, HIVE-3531.D5871.2.patch, HIVE-3531.D5871.3.patch In many cases, we uses hive server as a sole proxy for executing all the queries. For that, current default lock manager based on zookeeper seemed a little heavy. Simple in-memory lock manager could be enough. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3729) Error in groupSetExpression rule in Hive grammar
[ https://issues.apache.org/jira/browse/HIVE-3729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13505312#comment-13505312 ] Namit Jain commented on HIVE-3729: -- +1 Error in groupSetExpression rule in Hive grammar Key: HIVE-3729 URL: https://issues.apache.org/jira/browse/HIVE-3729 Project: Hive Issue Type: Bug Components: Query Processor Affects Versions: 0.10.0 Environment: All Reporter: Harish Butani Assignee: Harish Butani Priority: Minor Fix For: 0.10.0 Attachments: HIVE-3729.1.patch.txt Original Estimate: 5m Remaining Estimate: 5m Here is the error: Hive.g:1902:38: reference to rewrite element groupByExpression without reference on left of - -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3729) Error in groupSetExpression rule in Hive grammar
[ https://issues.apache.org/jira/browse/HIVE-3729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13505313#comment-13505313 ] Namit Jain commented on HIVE-3729: -- https://reviews.facebook.net/D7047 Error in groupSetExpression rule in Hive grammar Key: HIVE-3729 URL: https://issues.apache.org/jira/browse/HIVE-3729 Project: Hive Issue Type: Bug Components: Query Processor Affects Versions: 0.10.0 Environment: All Reporter: Harish Butani Assignee: Harish Butani Priority: Minor Fix For: 0.10.0 Attachments: HIVE-3729.1.patch.txt Original Estimate: 5m Remaining Estimate: 5m Here is the error: Hive.g:1902:38: reference to rewrite element groupByExpression without reference on left of - -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3531) Simple lock manager for dedicated hive server
[ https://issues.apache.org/jira/browse/HIVE-3531?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13505315#comment-13505315 ] Phabricator commented on HIVE-3531: --- navis has closed the revision HIVE-3531 [jira] Simple lock manager for dedicated hive server. Closed by cws. CHANGED PRIOR TO COMMIT https://reviews.facebook.net/D5871?vs=22401id=22593#differential-review-toc REVISION DETAIL https://reviews.facebook.net/D5871 COMMIT https://reviews.facebook.net/rHIVE1414590 To: JIRA, cwsteinbach, navis Simple lock manager for dedicated hive server - Key: HIVE-3531 URL: https://issues.apache.org/jira/browse/HIVE-3531 Project: Hive Issue Type: Improvement Components: Locking, Server Infrastructure Reporter: Navis Assignee: Navis Priority: Trivial Fix For: 0.11 Attachments: HIVE-3531.D5871.1.patch, HIVE-3531.D5871.2.patch, HIVE-3531.D5871.3.patch In many cases, we uses hive server as a sole proxy for executing all the queries. For that, current default lock manager based on zookeeper seemed a little heavy. Simple in-memory lock manager could be enough. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
Review Request: Performance regression introduced by HIVE-3483
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/8246/ --- Review request for hive, Ashutosh Chauhan and namit jain. Description --- Solves the issue described in HIVE-3483 without adding any performance overhead. It does not attempt to use complete path (including scheme) for comparison like earlier solution. It has fix to use consistent way of converting path to string, before paths are compared. Diffs - ql/src/java/org/apache/hadoop/hive/ql/io/CombineHiveInputFormat.java b774a95 Diff: https://reviews.apache.org/r/8246/diff/ Testing --- Thanks, Thejas Nair
[jira] [Updated] (HIVE-3693) Performance regression introduced by HIVE-3483
[ https://issues.apache.org/jira/browse/HIVE-3693?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thejas M Nair updated HIVE-3693: Assignee: Thejas M Nair Status: Patch Available (was: Open) Review board request - https://reviews.apache.org/r/8246/ Performance regression introduced by HIVE-3483 -- Key: HIVE-3693 URL: https://issues.apache.org/jira/browse/HIVE-3693 Project: Hive Issue Type: Bug Reporter: Gang Tim Liu Assignee: Thejas M Nair Priority: Minor Attachments: HIVE-3693.1.patch https://issues.apache.org/jira/browse/HIVE-3483 introduced a performance regression in the client side during split computation. The client side spends a lot more time in the split computation phase. The problem is checkFilterPathContains method. While investigating, can you create a config to disable it by default? thanks -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: hive 0.10 release
Hi Ashutosh, Can you please include patch in https://issues.apache.org/jira/browse/HIVE-3693 in 0.10 if possible ? It fixes test case failures on windows, and it is a small low risk change. Thanks, Thejas On 11/19/12 3:26 PM, kulkarni.swar...@gmail.com wrote: There are couple of enhancements that I have been working on mainly related to the hive/hbase integration. It would be awesome if it is possible at all to include them in this release. None of them should really be high risk. I have patches submitted for few of them. Will try to get for others submitted in next couple of days. Any specific deadline that I should be looking forward to? [1] https://issues.apache.org/jira/browse/HIVE-2599 (Patch Available) [2] https://issues.apache.org/jira/browse/HIVE-3553 (Patch Available) [3] https://issues.apache.org/jira/browse/HIVE-3211 [4] https://issues.apache.org/jira/browse/HIVE-3555 [5] https://issues.apache.org/jira/browse/HIVE-3725 On Mon, Nov 19, 2012 at 4:55 PM, Ashutosh Chauhan hashut...@apache.orgwrote: Another quick update. I have created a hive-0.10 branch. At this point, HIVE-3678 is a blocker to do a 0.10 release. There are few others nice to have which were there in my previous email. I will be happy to merge new patches between now and RC if folks request for it and are low risk. Thanks, Ashutosh On Thu, Nov 15, 2012 at 2:29 PM, Ashutosh Chauhan hashut...@apache.org wrote: Good progress. Looks like folks are on board. I propose to cut the branch in next couple of days. There are few jiras which are patch ready which I want to get into the hive-0.10 release, including HIVE-3255 HIVE-2517 HIVE-3400 HIVE-3678 Ed has already made a request for HIVE-3083. If folks have other patches they want see in 0.10, please chime in. Also, request to other committers to help in review patches. There are quite a few in Patch Available state. Thanks, Ashutosh On Thu, Nov 8, 2012 at 3:22 PM, Owen O'Malley omal...@apache.org wrote: +1 On Thu, Nov 8, 2012 at 3:18 PM, Carl Steinbach c...@cloudera.com wrote: +1 On Wed, Nov 7, 2012 at 11:23 PM, Alexander Lorenz wget.n...@gmail.com wrote: +1, good karma On Nov 8, 2012, at 4:58 AM, Namit Jain nj...@fb.com wrote: +1 to the idea On 11/8/12 6:33 AM, Edward Capriolo edlinuxg...@gmail.com wrote: That sounds good. I think this issue needs to be solved as well as anything else that produces a bugus query result. https://issues.apache.org/jira/browse/HIVE-3083 Edward On Wed, Nov 7, 2012 at 7:50 PM, Ashutosh Chauhan hashut...@apache.org wrote: Hi, Its been a while since we released 0.10 more than six months ago. All this while, lot of action has happened with various cool features landing in trunk. Additionally, I am looking forward to HiveServer2 landing in trunk. So, I propose that we cut the branch for 0.10 soon afterwards and than release it. Thoughts? Thanks, Ashutosh -- Alexander Alten-Lorenz http://mapredit.blogspot.com German Hadoop LinkedIn Group: http://goo.gl/N8pCF
[jira] [Updated] (HIVE-3729) Error in groupSetExpression rule in Hive grammar
[ https://issues.apache.org/jira/browse/HIVE-3729?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Namit Jain updated HIVE-3729: - Resolution: Fixed Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Committed. Thanks Harish Error in groupSetExpression rule in Hive grammar Key: HIVE-3729 URL: https://issues.apache.org/jira/browse/HIVE-3729 Project: Hive Issue Type: Bug Components: Query Processor Affects Versions: 0.10.0 Environment: All Reporter: Harish Butani Assignee: Harish Butani Priority: Minor Fix For: 0.10.0 Attachments: HIVE-3729.1.patch.txt Original Estimate: 5m Remaining Estimate: 5m Here is the error: Hive.g:1902:38: reference to rewrite element groupByExpression without reference on left of - -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3723) Hive Driver leaks ZooKeeper connections
[ https://issues.apache.org/jira/browse/HIVE-3723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13505338#comment-13505338 ] Hudson commented on HIVE-3723: -- Integrated in Hive-trunk-h0.21 #1821 (See [https://builds.apache.org/job/Hive-trunk-h0.21/1821/]) HIVE-3723 : Hive Driver leaks ZooKeeper connections (Gunther Hagleitner via Ashutosh Chauhan) (Revision 1414278) Result = FAILURE hashutosh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1414278 Files : * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/Driver.java Hive Driver leaks ZooKeeper connections --- Key: HIVE-3723 URL: https://issues.apache.org/jira/browse/HIVE-3723 Project: Hive Issue Type: Bug Reporter: Gunther Hagleitner Assignee: Gunther Hagleitner Fix For: 0.10.0 Attachments: HIVE-3723.1-r1411423.patch In certain error cases (i.e.: statement fails to compile, semantic errors) the hive driver leaks zookeeper connections. This can be seen in the TestNegativeCliDriver test which accumulates a large number of open file handles and fails if the max allowed number of file handles isn't at least 2048. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3749) New test cases added by HIVE-3676 in insert1.q is not deterministic
[ https://issues.apache.org/jira/browse/HIVE-3749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13505339#comment-13505339 ] Hudson commented on HIVE-3749: -- Integrated in Hive-trunk-h0.21 #1821 (See [https://builds.apache.org/job/Hive-trunk-h0.21/1821/]) HIVE-3749 : New test cases added by HIVE-3676 in insert1.q is not deterministic (Navis via Ashutosh Chauhan) (Revision 1414488) Result = FAILURE hashutosh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1414488 Files : * /hive/trunk/ql/src/test/queries/clientpositive/insert1.q * /hive/trunk/ql/src/test/results/clientpositive/insert1.q.out New test cases added by HIVE-3676 in insert1.q is not deterministic --- Key: HIVE-3749 URL: https://issues.apache.org/jira/browse/HIVE-3749 Project: Hive Issue Type: Test Components: Tests Reporter: Navis Assignee: Navis Fix For: 0.10.0 Attachments: HIVE-3749.D7011.1.patch The test case inserts two row and selects those all. But the displaying order can be different from env to env. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3234) getting the reporter in the recordwriter
[ https://issues.apache.org/jira/browse/HIVE-3234?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13505340#comment-13505340 ] Hudson commented on HIVE-3234: -- Integrated in Hive-trunk-h0.21 #1821 (See [https://builds.apache.org/job/Hive-trunk-h0.21/1821/]) HIVE-3234 : getting the reporter in the recordwriter (Owen Omalley via Ashutosh Chauhan) (Revision 1414266) Result = FAILURE hashutosh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1414266 Files : * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/AbstractMapJoinOperator.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/CommonJoinOperator.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/JoinUtil.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/SMBMapJoinOperator.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/RowContainer.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/HiveFileFormatUtils.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/rcfile/merge/BlockMergeTask.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/rcfile/merge/RCFileMergeMapper.java * /hive/trunk/ql/src/test/org/apache/hadoop/hive/ql/io/udf * /hive/trunk/ql/src/test/org/apache/hadoop/hive/ql/io/udf/Rot13InputFormat.java * /hive/trunk/ql/src/test/org/apache/hadoop/hive/ql/io/udf/Rot13OutputFormat.java * /hive/trunk/ql/src/test/queries/clientpositive/custom_input_output_format.q * /hive/trunk/ql/src/test/results/clientpositive/custom_input_output_format.q.out getting the reporter in the recordwriter Key: HIVE-3234 URL: https://issues.apache.org/jira/browse/HIVE-3234 Project: Hive Issue Type: Improvement Components: Serializers/Deserializers Affects Versions: 0.9.1 Environment: any Reporter: Jimmy Hu Assignee: Owen O'Malley Labels: newbie Fix For: 0.10.0 Attachments: HIVE-3234.D6699.1.patch, HIVE-3234.D6699.2.patch, HIVE-3234.D6987.1.patch Original Estimate: 48h Remaining Estimate: 48h We would like to generate some custom statistics and report back to map/reduce later wen implement the FileSinkOperator.RecordWriter interface. However, the current interface design doesn't allow us to get the map reduce reporter object. Please extend the current FileSinkOperator.RecordWriter interface so that it's close() method passes in a map reduce reporter object. For the same reason, please also extend the RecordReader interface too to include a reporter object so that users can passes in custom map reduce counters. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3742) The derby metastore schema script for 0.10.0 doesn't run
[ https://issues.apache.org/jira/browse/HIVE-3742?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13505342#comment-13505342 ] Hudson commented on HIVE-3742: -- Integrated in Hive-trunk-h0.21 #1821 (See [https://builds.apache.org/job/Hive-trunk-h0.21/1821/]) HIVE-3742 : The derby metastore schema script for 0.10.0 doesn't run (Prasad Mujumdar via Ashutosh Chauhan) (Revision 1414292) Result = FAILURE hashutosh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1414292 Files : * /hive/trunk/metastore/scripts/upgrade/derby/hive-schema-0.10.0.derby.sql The derby metastore schema script for 0.10.0 doesn't run Key: HIVE-3742 URL: https://issues.apache.org/jira/browse/HIVE-3742 Project: Hive Issue Type: Bug Affects Versions: 0.10.0 Reporter: Prasad Mujumdar Assignee: Prasad Mujumdar Fix For: 0.10.0 Attachments: HIVE-3742-2.patch, HIVE-3742.patch The hive-schema-0.10.0.derby.sql contains incorrect alter statement for SKEWED_STRING_LIST which causes the script execution to fail -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3648) HiveMetaStoreFsImpl is not compatible with hadoop viewfs
[ https://issues.apache.org/jira/browse/HIVE-3648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13505341#comment-13505341 ] Hudson commented on HIVE-3648: -- Integrated in Hive-trunk-h0.21 #1821 (See [https://builds.apache.org/job/Hive-trunk-h0.21/1821/]) HIVE-3648 : HiveMetaStoreFsImpl is not compatible with hadoop viewfs (Arup Malakar via Ashutosh Chauhan) (Revision 1414353) Result = FAILURE hashutosh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1414353 Files : * /hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStoreFsImpl.java * /hive/trunk/shims/src/0.20/java/org/apache/hadoop/hive/shims/Hadoop20Shims.java * /hive/trunk/shims/src/0.20S/java/org/apache/hadoop/hive/shims/Hadoop20SShims.java * /hive/trunk/shims/src/0.23/java/org/apache/hadoop/hive/shims/Hadoop23Shims.java * /hive/trunk/shims/src/common-secure/java/org/apache/hadoop/hive/shims/HadoopShimsSecure.java * /hive/trunk/shims/src/common/java/org/apache/hadoop/hive/shims/HadoopShims.java HiveMetaStoreFsImpl is not compatible with hadoop viewfs Key: HIVE-3648 URL: https://issues.apache.org/jira/browse/HIVE-3648 Project: Hive Issue Type: Bug Components: Metastore Affects Versions: 0.9.0, 0.10.0 Reporter: Kihwal Lee Assignee: Arup Malakar Fix For: 0.11 Attachments: HIVE_3648_branch_0.patch, HIVE-3648-trunk-0.patch, HIVE_3648_trunk_1.patch, HIVE-3648-trunk-1.patch HiveMetaStoreFsImpl#deleteDir() method calls Trash#moveToTrash(). This may not work when viewfs is used. It needs to call Trash#moveToAppropriateTrash() instead. Please note that this method is not available in hadoop versions earlier than 0.23. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3645) RCFileWriter does not implement the right function to support Federation
[
https://issues.apache.org/jira/browse/HIVE-3645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13505343#comment-13505343
]
Hudson commented on HIVE-3645:
--
Integrated in Hive-trunk-h0.21 #1821 (See
[https://builds.apache.org/job/Hive-trunk-h0.21/1821/])
HIVE-3645 : RCFileWriter does not implement the right function to support
Federation (Arup Malakar via Ashutosh Chauhan) (Revision 1414286)
Result = FAILURE
hashutosh :
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1414286
Files :
* /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/io/RCFile.java
*
/hive/trunk/shims/src/0.20/java/org/apache/hadoop/hive/shims/Hadoop20Shims.java
*
/hive/trunk/shims/src/0.20S/java/org/apache/hadoop/hive/shims/Hadoop20SShims.java
*
/hive/trunk/shims/src/0.23/java/org/apache/hadoop/hive/shims/Hadoop23Shims.java
*
/hive/trunk/shims/src/common-secure/java/org/apache/hadoop/hive/shims/HadoopShimsSecure.java
*
/hive/trunk/shims/src/common/java/org/apache/hadoop/hive/shims/HadoopShims.java
RCFileWriter does not implement the right function to support Federation
Key: HIVE-3645
URL: https://issues.apache.org/jira/browse/HIVE-3645
Project: Hive
Issue Type: Bug
Components: Serializers/Deserializers
Affects Versions: 0.9.0, 0.10.0
Environment: Hadoop 0.23.3 federation, Hive 0.9 and Pig 0.10
Reporter: Viraj Bhat
Assignee: Arup Malakar
Fix For: 0.11
Attachments: HIVE_3645_branch_0.patch, HIVE_3645_trunk_0.patch
Create a table using Hive DDL
{code}
CREATE TABLE tmp_hcat_federated_numbers_part_1 (
id int,
intnum int,
floatnum float
)partitioned by (
part1string,
part2string
)
STORED AS rcfile
LOCATION 'viewfs:///database/tmp_hcat_federated_numbers_part_1';
{code}
Populate it using Pig:
{code}
A = load 'default.numbers_pig' using org.apache.hcatalog.pig.HCatLoader();
B = filter A by id = 500;
C = foreach B generate (int)id, (int)intnum, (float)floatnum;
store C into
'default.tmp_hcat_federated_numbers_part_1'
using org.apache.hcatalog.pig.HCatStorer
('part1=pig, part2=hcat_pig_insert',
'id: int,intnum: int,floatnum: float');
{code}
Generates the following error when running on a Federated Cluster:
{quote}
2012-10-29 20:40:25,011 [main] ERROR
org.apache.pig.tools.pigstats.SimplePigStats - ERROR 2997: Unable to recreate
exception from backed error: AttemptID:attempt_1348522594824_0846_m_00_3
Info:Error: org.apache.hadoop.fs.viewfs.NotInMountpointException:
getDefaultReplication on empty path is invalid
at
org.apache.hadoop.fs.viewfs.ViewFileSystem.getDefaultReplication(ViewFileSystem.java:479)
at org.apache.hadoop.hive.ql.io.RCFile$Writer.init(RCFile.java:723)
at org.apache.hadoop.hive.ql.io.RCFile$Writer.init(RCFile.java:705)
at
org.apache.hadoop.hive.ql.io.RCFileOutputFormat.getRecordWriter(RCFileOutputFormat.java:86)
at
org.apache.hcatalog.mapreduce.FileOutputFormatContainer.getRecordWriter(FileOutputFormatContainer.java:100)
at
org.apache.hcatalog.mapreduce.HCatOutputFormat.getRecordWriter(HCatOutputFormat.java:228)
at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat.getRecordWriter(PigOutputFormat.java:84)
at
org.apache.hadoop.mapred.MapTask$NewDirectOutputCollector.init(MapTask.java:587)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:706)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:332)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:157)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1212)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:152)
{quote}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3676) INSERT INTO regression caused by HIVE-3465
[ https://issues.apache.org/jira/browse/HIVE-3676?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13505344#comment-13505344 ] Hudson commented on HIVE-3676: -- Integrated in Hive-trunk-h0.21 #1821 (See [https://builds.apache.org/job/Hive-trunk-h0.21/1821/]) HIVE-3749 : New test cases added by HIVE-3676 in insert1.q is not deterministic (Navis via Ashutosh Chauhan) (Revision 1414488) Result = FAILURE hashutosh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1414488 Files : * /hive/trunk/ql/src/test/queries/clientpositive/insert1.q * /hive/trunk/ql/src/test/results/clientpositive/insert1.q.out INSERT INTO regression caused by HIVE-3465 -- Key: HIVE-3676 URL: https://issues.apache.org/jira/browse/HIVE-3676 Project: Hive Issue Type: Bug Components: Query Processor Reporter: Carl Steinbach Assignee: Navis Fix For: 0.10.0 Attachments: HIVE-3676.D6741.1.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
Hive-trunk-h0.21 - Build # 1821 - Still Failing
Changes for Build #1777 [kevinwilfong] HIVE-3524. Storing certain Exception objects thrown in HiveMetaStore.java in MetaStoreEndFunctionContext. (Maheshwaran Srinivasan via kevinwilfong) [cws] HIVE-1977. DESCRIBE TABLE syntax doesn't support specifying a database qualified table name (Zhenxiao Luo via cws) [cws] HIVE-3674. Test case TestParse broken after recent checkin (Sambavi Muthukrishnan via cws) Changes for Build #1778 [cws] HIVE-1362. Column level scalar valued statistics on Tables and Partitions (Shreepadma Venugopalan via cws) Changes for Build #1779 Changes for Build #1780 [kevinwilfong] HIVE-3686. Fix compile errors introduced by the interaction of HIVE-1362 and HIVE-3524. (Shreepadma Venugopalan via kevinwilfong) Changes for Build #1781 [namit] HIVE-3687 smb_mapjoin_13.q is nondeterministic (Kevin Wilfong via namit) Changes for Build #1782 [hashutosh] HIVE-2715: Upgrade Thrift dependency to 0.9.0 (Ashutosh Chauhan) Changes for Build #1783 [kevinwilfong] HIVE-3654. block relative path access in hive. (njain via kevinwilfong) [hashutosh] HIVE-3658 : Unable to generate the Hbase related unit tests using velocity templates on Windows (Kanna Karanam via Ashutosh Chauhan) [hashutosh] HIVE-3661 : Remove the Windows specific = related swizzle path changes from Proxy FileSystems (Kanna Karanam via Ashutosh Chauhan) [hashutosh] HIVE-3480 : Resource leak: Fix the file handle leaks in Symbolic Symlink related input formats. (Kanna Karanam via Ashutosh Chauhan) Changes for Build #1784 [kevinwilfong] HIVE-3675. NaN does not work correctly for round(n). (njain via kevinwilfong) [cws] HIVE-3651. bucketmapjoin?.q tests fail with hadoop 0.23 (Prasad Mujumdar via cws) Changes for Build #1785 [namit] HIVE-3613 Implement grouping_id function (Ian Gorbachev via namit) [namit] HIVE-3692 Update parallel test documentation (Ivan Gorbachev via namit) [namit] HIVE-3649 Hive List Bucketing - enhance DDL to specify list bucketing table (Gang Tim Liu via namit) Changes for Build #1786 [namit] HIVE-3696 Revert HIVE-3483 which causes performance regression (Gang Tim Liu via namit) Changes for Build #1787 [kevinwilfong] HIVE-3621. Make prompt in Hive CLI configurable. (Jingwei Lu via kevinwilfong) [kevinwilfong] HIVE-3695. TestParse breaks due to HIVE-3675. (njain via kevinwilfong) Changes for Build #1788 [kevinwilfong] HIVE-3557. Access to external URLs in hivetest.py. (Ivan Gorbachev via kevinwilfong) Changes for Build #1789 [hashutosh] HIVE-3662 : TestHiveServer: testScratchDirShouldClearWhileStartup is failing on Windows (Kanna Karanam via Ashutosh Chauhan) [hashutosh] HIVE-3659 : TestHiveHistory::testQueryloglocParentDirNotExist Test fails on Windows because of some resource leaks in ZK (Kanna Karanam via Ashutosh Chauhan) [hashutosh] HIVE-3663 Unable to display the MR Job file path on Windows in case of MR job failures. (Kanna Karanam via Ashutosh Chauhan) Changes for Build #1790 Changes for Build #1791 Changes for Build #1792 Changes for Build #1793 [hashutosh] HIVE-3704 : name of some metastore scripts are not per convention (Ashutosh Chauhan) Changes for Build #1794 [hashutosh] HIVE-3243 : ignore white space between entries of hive/hbase table mapping (Shengsheng Huang via Ashutosh Chauhan) [hashutosh] HIVE-3215 : JobDebugger should use RunningJob.getTrackingURL (Bhushan Mandhani via Ashutosh Chauhan) Changes for Build #1795 [cws] HIVE-3437. 0.23 compatibility: fix unit tests when building against 0.23 (Chris Drome via cws) [hashutosh] HIVE-3626 : RetryingHMSHandler should wrap JDOException inside MetaException (Bhushan Mandhani via Ashutosh Chauhan) [hashutosh] HIVE-3560 : Hive always prints a warning message when using remote metastore (Travis Crawford via Ashutosh Chauhan) Changes for Build #1796 Changes for Build #1797 [hashutosh] HIVE-3664 : Avoid to create a symlink for hive-contrib.jar file in dist\lib folder. (Kanna Karanam via Ashutosh Chauhan) Changes for Build #1798 [namit] HIVE-3706 getBoolVar in FileSinkOperator can be optimized (Kevin Wilfong via namit) [namit] HIVE-3707 Round map/reduce progress down when it is in the range [99.5, 100) (Kevin Wilfong via namit) [namit] HIVE-3471 Implement grouping sets in hive (Ivan Gorbachev via namit) Changes for Build #1799 [hashutosh] HIVE-3291 : fix fs resolvers (Ashish Singh via Ashutosh Chauhan) [hashutosh] HIVE-3680 : Include Table information in Hive's AddPartitionEvent. (Mithun Radhakrishnan via Ashutosh Chauhan) Changes for Build #1800 [hashutosh] HIVE-3520 : ivysettings.xml does not let you override .m2/repository (Raja Aluri via Ashutosh Chauhan) [hashutosh] HIVE-3435 : Get pdk pluginTest passed when triggered from both builtin tests and pdk tests on hadoop23 (Zhenxiao Luo via Ashutosh Chauhan) Changes for Build #1801 Changes for Build #1802 Changes for Build #1803 Changes for Build #1804 Changes for Build #1805 Changes for Build #1806 [hashutosh] Preparing
[jira] [Commented] (HIVE-3729) Error in groupSetExpression rule in Hive grammar
[ https://issues.apache.org/jira/browse/HIVE-3729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13505432#comment-13505432 ] Hudson commented on HIVE-3729: -- Integrated in Hive-trunk-h0.21 #1822 (See [https://builds.apache.org/job/Hive-trunk-h0.21/1822/]) HIVE-3729 Error in groupSetExpression rule in Hive grammar (Harish Butani via namit) (Revision 1414608) Result = ABORTED namit : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1414608 Files : * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/Hive.g Error in groupSetExpression rule in Hive grammar Key: HIVE-3729 URL: https://issues.apache.org/jira/browse/HIVE-3729 Project: Hive Issue Type: Bug Components: Query Processor Affects Versions: 0.10.0 Environment: All Reporter: Harish Butani Assignee: Harish Butani Priority: Minor Fix For: 0.10.0 Attachments: HIVE-3729.1.patch.txt Original Estimate: 5m Remaining Estimate: 5m Here is the error: Hive.g:1902:38: reference to rewrite element groupByExpression without reference on left of - -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3531) Simple lock manager for dedicated hive server
[ https://issues.apache.org/jira/browse/HIVE-3531?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13505431#comment-13505431 ] Hudson commented on HIVE-3531: -- Integrated in Hive-trunk-h0.21 #1822 (See [https://builds.apache.org/job/Hive-trunk-h0.21/1822/]) HIVE-3531 [jira] Simple lock manager for dedicated hive server (Navis Ryu via Carl Steinbach) Summary: DPAL-1906 Implement simple lock manager for hive server In many cases, we uses hive server as a sole proxy for executing all the queries. For that, current default lock manager based on zookeeper seemed a little heavy. Simple in-memory lock manager could be enough. Test Plan: TestDedicatedLockManager Reviewers: JIRA, cwsteinbach Reviewed By: cwsteinbach Differential Revision: https://reviews.facebook.net/D5871 (Revision 1414590) Result = ABORTED cws : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1414590 Files : * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/lockmgr/EmbeddedLockManager.java * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/lockmgr/HiveLockObject.java * /hive/trunk/ql/src/test/org/apache/hadoop/hive/ql/lockmgr * /hive/trunk/ql/src/test/org/apache/hadoop/hive/ql/lockmgr/TestEmbeddedLockManager.java Simple lock manager for dedicated hive server - Key: HIVE-3531 URL: https://issues.apache.org/jira/browse/HIVE-3531 Project: Hive Issue Type: Improvement Components: Locking, Server Infrastructure Reporter: Navis Assignee: Navis Priority: Trivial Fix For: 0.11 Attachments: HIVE-3531.D5871.1.patch, HIVE-3531.D5871.2.patch, HIVE-3531.D5871.3.patch In many cases, we uses hive server as a sole proxy for executing all the queries. For that, current default lock manager based on zookeeper seemed a little heavy. Simple in-memory lock manager could be enough. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3665) Allow URIs without port to be specified in metatool
[ https://issues.apache.org/jira/browse/HIVE-3665?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13505433#comment-13505433 ] Hudson commented on HIVE-3665: -- Integrated in Hive-trunk-h0.21 #1822 (See [https://builds.apache.org/job/Hive-trunk-h0.21/1822/]) HIVE-3665: Allow URIs without port to be specified in metatool (Shreepadma via Ashutosh Chauhan) (Revision 1414512) Result = ABORTED hashutosh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1414512 Files : * /hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/tools/HiveMetaTool.java Allow URIs without port to be specified in metatool --- Key: HIVE-3665 URL: https://issues.apache.org/jira/browse/HIVE-3665 Project: Hive Issue Type: Bug Components: Metastore Affects Versions: 0.10.0 Reporter: Shreepadma Venugopalan Assignee: Shreepadma Venugopalan Fix For: 0.11 Attachments: HIVE-3665.1.patch.txt Metatool should accept input URIs where one URI contains a port and the other doesn't. While metatool today accepts input URIs without the port when both the input URIs (oldLoc and newLoc) don't contain the port, we should make the tool a little more flexible to allow for the case where one URI contains a valid port and the other input URI doesn't. This makes more sense when transitioning to HA and a user chooses to specify the port as part of the oldLoc, but the port doesn't mean much for the newLoc. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3678) Add metastore upgrade scripts for column stats schema changes
[ https://issues.apache.org/jira/browse/HIVE-3678?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13505430#comment-13505430 ] Hudson commented on HIVE-3678: -- Integrated in Hive-trunk-h0.21 #1822 (See [https://builds.apache.org/job/Hive-trunk-h0.21/1822/]) HIVE-3678 : Add metastore upgrade scripts for column stats schema changes (Shreepadma via Ashutosh Chauhan) (Revision 1414509) Result = ABORTED hashutosh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1414509 Files : * /hive/trunk/metastore/scripts/upgrade/derby/012-HIVE-1362.derby.sql * /hive/trunk/metastore/scripts/upgrade/derby/hive-schema-0.10.0.derby.sql * /hive/trunk/metastore/scripts/upgrade/derby/upgrade-0.9.0-to-0.10.0.derby.sql * /hive/trunk/metastore/scripts/upgrade/mysql/012-HIVE-1362.mysql.sql * /hive/trunk/metastore/scripts/upgrade/mysql/hive-schema-0.10.0.mysql.sql * /hive/trunk/metastore/scripts/upgrade/mysql/upgrade-0.9.0-to-0.10.0.mysql.sql * /hive/trunk/metastore/scripts/upgrade/oracle/012-HIVE-1362.oracle.sql * /hive/trunk/metastore/scripts/upgrade/oracle/hive-schema-0.10.0.oracle.sql * /hive/trunk/metastore/scripts/upgrade/postgres/012-HIVE-1362.postgres.sql * /hive/trunk/metastore/scripts/upgrade/postgres/hive-schema-0.10.0.postgres.sql * /hive/trunk/metastore/scripts/upgrade/postgres/upgrade-0.9.0-to-0.10.0.postgres.sql * /hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java * /hive/trunk/metastore/src/model/org/apache/hadoop/hive/metastore/model/MPartitionColumnStatistics.java * /hive/trunk/metastore/src/model/org/apache/hadoop/hive/metastore/model/MTableColumnStatistics.java * /hive/trunk/metastore/src/model/package.jdo Add metastore upgrade scripts for column stats schema changes - Key: HIVE-3678 URL: https://issues.apache.org/jira/browse/HIVE-3678 Project: Hive Issue Type: Bug Components: Metastore Reporter: Shreepadma Venugopalan Assignee: Shreepadma Venugopalan Fix For: 0.10.0 Attachments: HIVE-3678.1.patch.txt, HIVE-3678.2.patch.txt, HIVE-3678.3.patch.txt, HIVE-3678.4.patch.txt Add upgrade script for column statistics schema changes for Postgres/MySQL/Oracle/Derby -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3709) Stop storing default ConfVars in temp file
[ https://issues.apache.org/jira/browse/HIVE-3709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13505455#comment-13505455 ] Chris McConnell commented on HIVE-3709: --- I was also looking into this with 3596, I was able to fix utilizing a location similar to the suggestion Carl made above, however I think that pushes the problem to another location, rather than addressing the actual issue. I like where Kevin is going with this fix, I had thought about the possibility of checking the confVarURL in the copy constructor, removing and re-creating if it did not exist, but even that would not be perfect depending upon timing. Stop storing default ConfVars in temp file -- Key: HIVE-3709 URL: https://issues.apache.org/jira/browse/HIVE-3709 Project: Hive Issue Type: Improvement Components: Configuration Affects Versions: 0.10.0 Reporter: Kevin Wilfong Assignee: Kevin Wilfong Attachments: HIVE-3709.1.patch.txt, HIVE-3709.2.patch.txt, HIVE-3709.3.patch.txt To work around issues with Hadoop's Configuration object, specifically it's addResource(InputStream), default configurations are written to a temp file (I think HIVE-2362 introduced this). This, however, introduces the problem that once that file is deleted from /tmp the client crashes. This is particularly problematic for long running services like the metastore server. Writing a custom InputStream to deal with the problems in the Configuration object should provide a work around, which does not introduce a time bomb into Hive. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2206) add a new optimizer for query correlation discovery and optimization
[ https://issues.apache.org/jira/browse/HIVE-2206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13505495#comment-13505495 ] Yin Huai commented on HIVE-2206: [~cwsteinbach] I am not sure if unit tests in Hive are comprehensive enough. If not, it might be better that we turn on this optimizer by default in future after we can use more queries to test it. I just tested all unit tests with an enabled correlation optimizer. Because, if map side aggregation is on, correlation optimizer also requires regular reduce side aggregation to be generated, if cube or rollup is used in the query, error message 10209 (org.apache.hadoop.hive.ql.ErrorMsg.HIVE_GROUPING_SETS_AGGR_NOMAPAGGR) will be thrown. Seems HIVE-3508 can solve this issue. Except this issue, a few query plans need to be re-generated because of changing operator ids. This jira has taken a long time. Can we wrap it up and I will start to work on follow-up jiras. add a new optimizer for query correlation discovery and optimization Key: HIVE-2206 URL: https://issues.apache.org/jira/browse/HIVE-2206 Project: Hive Issue Type: New Feature Components: Query Processor Affects Versions: 0.10.0 Reporter: He Yongqiang Assignee: Yin Huai Attachments: HIVE-2206.10-r1384442.patch.txt, HIVE-2206.11-r1385084.patch.txt, HIVE-2206.12-r1386996.patch.txt, HIVE-2206.13-r1389072.patch.txt, HIVE-2206.14-r1389704.patch.txt, HIVE-2206.15-r1392491.patch.txt, HIVE-2206.16-r1399936.patch.txt, HIVE-2206.17-r1404933.patch.txt, HIVE-2206.18-r1407720.patch.txt, HIVE-2206.19-r1410581.patch.txt, HIVE-2206.1.patch.txt, HIVE-2206.2.patch.txt, HIVE-2206.3.patch.txt, HIVE-2206.4.patch.txt, HIVE-2206.5-1.patch.txt, HIVE-2206.5.patch.txt, HIVE-2206.6.patch.txt, HIVE-2206.7.patch.txt, HIVE-2206.8.r1224646.patch.txt, HIVE-2206.8-r1237253.patch.txt, testQueries.2.q, YSmartPatchForHive.patch This issue proposes a new logical optimizer called Correlation Optimizer, which is used to merge correlated MapReduce jobs (MR jobs) into a single MR job. The idea is based on YSmart (http://ysmart.cse.ohio-state.edu/).The paper and slides of YSmart are linked at the bottom. Since Hive translates queries in a sentence by sentence fashion, for every operation which may need to shuffle the data (e.g. join and aggregation operations), Hive will generate a MapReduce job for that operation. However, for those operations which may need to shuffle the data, they may involve correlations explained below and thus can be executed in a single MR job. # Input Correlation: Multiple MR jobs have input correlation (IC) if their input relation sets are not disjoint; # Transit Correlation: Multiple MR jobs have transit correlation (TC) if they have not only input correlation, but also the same partition key; # Job Flow Correlation: An MR has job flow correlation (JFC) with one of its child nodes if it has the same partition key as that child node. The current implementation of correlation optimizer only detect correlations among MR jobs for reduce-side join operators and reduce-side aggregation operators (not map only aggregation). A query will be optimized if it satisfies following conditions. # There exists a MR job for reduce-side join operator or reduce side aggregation operator which have JFC with all of its parents MR jobs (TCs will be also exploited if JFC exists); # All input tables of those correlated MR job are original input tables (not intermediate tables generated by sub-queries); and # No self join is involved in those correlated MR jobs. Correlation optimizer is implemented as a logical optimizer. The main reasons are that it only needs to manipulate the query plan tree and it can leverage the existing component on generating MR jobs. Current implementation can serve as a framework for correlation related optimizations. I think that it is better than adding individual optimizers. There are several work that can be done in future to improve this optimizer. Here are three examples. # Support queries only involve TC; # Support queries in which input tables of correlated MR jobs involves intermediate tables; and # Optimize queries involving self join. References: Paper and presentation of YSmart. Paper: http://www.cse.ohio-state.edu/hpcs/WWW/HTML/publications/papers/TR-11-7.pdf Slides: http://sdrv.ms/UpwJJc -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: hive 0.10 release
When will hive 0.10 be released? 2012/11/28 Thejas Nair the...@hortonworks.com Hi Ashutosh, Can you please include patch in https://issues.apache.org/** jira/browse/HIVE-3693 https://issues.apache.org/jira/browse/HIVE-3693in 0.10 if possible ? It fixes test case failures on windows, and it is a small low risk change. Thanks, Thejas On 11/19/12 3:26 PM, kulkarni.swar...@gmail.com wrote: There are couple of enhancements that I have been working on mainly related to the hive/hbase integration. It would be awesome if it is possible at all to include them in this release. None of them should really be high risk. I have patches submitted for few of them. Will try to get for others submitted in next couple of days. Any specific deadline that I should be looking forward to? [1] https://issues.apache.org/**jira/browse/HIVE-2599https://issues.apache.org/jira/browse/HIVE-2599(Patch Available) [2] https://issues.apache.org/**jira/browse/HIVE-3553https://issues.apache.org/jira/browse/HIVE-3553(Patch Available) [3] https://issues.apache.org/**jira/browse/HIVE-3211https://issues.apache.org/jira/browse/HIVE-3211 [4] https://issues.apache.org/**jira/browse/HIVE-3555https://issues.apache.org/jira/browse/HIVE-3555 [5] https://issues.apache.org/**jira/browse/HIVE-3725https://issues.apache.org/jira/browse/HIVE-3725 On Mon, Nov 19, 2012 at 4:55 PM, Ashutosh Chauhan hashut...@apache.org wrote: Another quick update. I have created a hive-0.10 branch. At this point, HIVE-3678 is a blocker to do a 0.10 release. There are few others nice to have which were there in my previous email. I will be happy to merge new patches between now and RC if folks request for it and are low risk. Thanks, Ashutosh On Thu, Nov 15, 2012 at 2:29 PM, Ashutosh Chauhan hashut...@apache.org wrote: Good progress. Looks like folks are on board. I propose to cut the branch in next couple of days. There are few jiras which are patch ready which I want to get into the hive-0.10 release, including HIVE-3255 HIVE-2517 HIVE-3400 HIVE-3678 Ed has already made a request for HIVE-3083. If folks have other patches they want see in 0.10, please chime in. Also, request to other committers to help in review patches. There are quite a few in Patch Available state. Thanks, Ashutosh On Thu, Nov 8, 2012 at 3:22 PM, Owen O'Malley omal...@apache.org wrote: +1 On Thu, Nov 8, 2012 at 3:18 PM, Carl Steinbach c...@cloudera.com wrote: +1 On Wed, Nov 7, 2012 at 11:23 PM, Alexander Lorenz wget.n...@gmail.com wrote: +1, good karma On Nov 8, 2012, at 4:58 AM, Namit Jain nj...@fb.com wrote: +1 to the idea On 11/8/12 6:33 AM, Edward Capriolo edlinuxg...@gmail.com wrote: That sounds good. I think this issue needs to be solved as well as anything else that produces a bugus query result. https://issues.apache.org/**jira/browse/HIVE-3083https://issues.apache.org/jira/browse/HIVE-3083 Edward On Wed, Nov 7, 2012 at 7:50 PM, Ashutosh Chauhan hashut...@apache.org wrote: Hi, Its been a while since we released 0.10 more than six months ago. All this while, lot of action has happened with various cool features landing in trunk. Additionally, I am looking forward to HiveServer2 landing in trunk. So, I propose that we cut the branch for 0.10 soon afterwards and than release it. Thoughts? Thanks, Ashutosh -- Alexander Alten-Lorenz http://mapredit.blogspot.com German Hadoop LinkedIn Group: http://goo.gl/N8pCF
[jira] [Created] (HIVE-3751) Handle enclosing characters in delimited fields
Mark Grover created HIVE-3751:
-
Summary: Handle enclosing characters in delimited fields
Key: HIVE-3751
URL: https://issues.apache.org/jira/browse/HIVE-3751
Project: Hive
Issue Type: New Feature
Components: Serializers/Deserializers
Affects Versions: 0.9.0
Reporter: Mark Grover
People have requested in the past to have Hive support enclosing characters in
delimited row format.
An example of such data:
{code}
1, 123456, , Charlotte, NC
{code}
The format of the create table statement may look like:
{{DELIMITED [FIELDS TERMINATED BY char] [COLLECTION ITEMS TERMINATED BY char]
[FIELDS ENCLOSED BY char]}}
The first pass solution to this JIRA may disallow inclusion of enclosing
characters within the data. A latter improvement could be to allow escaping of
enclosing characters within the data.
Reference mailing list thread:
http://mail-archives.apache.org/mod_mbox/hive-user/201204.mbox/%3CCAENxBwwrZrqBSJXtJHpqc_FfcZvwRMoaT9W7dR=jgtyjoxp...@mail.gmail.com%3E
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-3751) Handle enclosing characters in delimited fields
[
https://issues.apache.org/jira/browse/HIVE-3751?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mark Grover updated HIVE-3751:
--
Description:
People have requested in the past to have Hive support enclosing characters in
delimited row format.
An example of such data:
{code}
1, 123456, , Charlotte, NC
{code}
The format of the create table statement may look like:
{code}
DELIMITED [FIELDS TERMINATED BY char] [COLLECTION ITEMS TERMINATED BY char]
[FIELDS ENCLOSED BY char]
{code}
The first pass solution to this JIRA may disallow inclusion of enclosing
characters within the data. A latter improvement could be to allow escaping of
enclosing characters within the data.
Reference mailing list thread:
http://mail-archives.apache.org/mod_mbox/hive-user/201204.mbox/%3CCAENxBwwrZrqBSJXtJHpqc_FfcZvwRMoaT9W7dR=jgtyjoxp...@mail.gmail.com%3E
was:
People have requested in the past to have Hive support enclosing characters in
delimited row format.
An example of such data:
{code}
1, 123456, , Charlotte, NC
{code}
The format of the create table statement may look like:
{{DELIMITED [FIELDS TERMINATED BY char] [COLLECTION ITEMS TERMINATED BY char]
[FIELDS ENCLOSED BY char]}}
The first pass solution to this JIRA may disallow inclusion of enclosing
characters within the data. A latter improvement could be to allow escaping of
enclosing characters within the data.
Reference mailing list thread:
http://mail-archives.apache.org/mod_mbox/hive-user/201204.mbox/%3CCAENxBwwrZrqBSJXtJHpqc_FfcZvwRMoaT9W7dR=jgtyjoxp...@mail.gmail.com%3E
Handle enclosing characters in delimited fields
---
Key: HIVE-3751
URL: https://issues.apache.org/jira/browse/HIVE-3751
Project: Hive
Issue Type: New Feature
Components: Serializers/Deserializers
Affects Versions: 0.9.0
Reporter: Mark Grover
People have requested in the past to have Hive support enclosing characters
in delimited row format.
An example of such data:
{code}
1, 123456, , Charlotte, NC
{code}
The format of the create table statement may look like:
{code}
DELIMITED [FIELDS TERMINATED BY char] [COLLECTION ITEMS TERMINATED BY char]
[FIELDS ENCLOSED BY char]
{code}
The first pass solution to this JIRA may disallow inclusion of enclosing
characters within the data. A latter improvement could be to allow escaping
of enclosing characters within the data.
Reference mailing list thread:
http://mail-archives.apache.org/mod_mbox/hive-user/201204.mbox/%3CCAENxBwwrZrqBSJXtJHpqc_FfcZvwRMoaT9W7dR=jgtyjoxp...@mail.gmail.com%3E
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
Build failed in Jenkins: Hive-0.9.1-SNAPSHOT-h0.21-keepgoing=false #213
See https://builds.apache.org/job/Hive-0.9.1-SNAPSHOT-h0.21-keepgoing=false/213/ -- [...truncated 9912 lines...] compile-test: [echo] Project: serde [javac] Compiling 26 source files to https://builds.apache.org/job/Hive-0.9.1-SNAPSHOT-h0.21-keepgoing=false/213/artifact/hive/build/serde/test/classes [javac] Note: Some input files use or override a deprecated API. [javac] Note: Recompile with -Xlint:deprecation for details. [javac] Note: Some input files use unchecked or unsafe operations. [javac] Note: Recompile with -Xlint:unchecked for details. create-dirs: [echo] Project: service [copy] Warning: https://builds.apache.org/job/Hive-0.9.1-SNAPSHOT-h0.21-keepgoing=false/ws/hive/service/src/test/resources does not exist. init: [echo] Project: service ivy-init-settings: [echo] Project: service ivy-resolve: [echo] Project: service [ivy:resolve] :: loading settings :: file = https://builds.apache.org/job/Hive-0.9.1-SNAPSHOT-h0.21-keepgoing=false/ws/hive/ivy/ivysettings.xml [ivy:report] Processing https://builds.apache.org/job/Hive-0.9.1-SNAPSHOT-h0.21-keepgoing=false/213/artifact/hive/build/ivy/resolution-cache/org.apache.hive-hive-service-default.xml to https://builds.apache.org/job/Hive-0.9.1-SNAPSHOT-h0.21-keepgoing=false/213/artifact/hive/build/ivy/report/org.apache.hive-hive-service-default.html ivy-retrieve: [echo] Project: service compile: [echo] Project: service ivy-resolve-test: [echo] Project: service ivy-retrieve-test: [echo] Project: service compile-test: [echo] Project: service [javac] Compiling 2 source files to https://builds.apache.org/job/Hive-0.9.1-SNAPSHOT-h0.21-keepgoing=false/213/artifact/hive/build/service/test/classes test: [echo] Project: hive test-shims: [echo] Project: hive test-conditions: [echo] Project: shims gen-test: [echo] Project: shims create-dirs: [echo] Project: shims [copy] Warning: https://builds.apache.org/job/Hive-0.9.1-SNAPSHOT-h0.21-keepgoing=false/ws/hive/shims/src/test/resources does not exist. init: [echo] Project: shims ivy-init-settings: [echo] Project: shims ivy-resolve: [echo] Project: shims [ivy:resolve] :: loading settings :: file = https://builds.apache.org/job/Hive-0.9.1-SNAPSHOT-h0.21-keepgoing=false/ws/hive/ivy/ivysettings.xml [ivy:report] Processing https://builds.apache.org/job/Hive-0.9.1-SNAPSHOT-h0.21-keepgoing=false/213/artifact/hive/build/ivy/resolution-cache/org.apache.hive-hive-shims-default.xml to https://builds.apache.org/job/Hive-0.9.1-SNAPSHOT-h0.21-keepgoing=false/213/artifact/hive/build/ivy/report/org.apache.hive-hive-shims-default.html ivy-retrieve: [echo] Project: shims compile: [echo] Project: shims [echo] Building shims 0.20 build_shims: [echo] Project: shims [echo] Compiling https://builds.apache.org/job/Hive-0.9.1-SNAPSHOT-h0.21-keepgoing=false/ws/hive/shims/src/common/java;/home/jenkins/jenkins-slave/workspace/Hive-0.9.1-SNAPSHOT-h0.21-keepgoing=false/hive/shims/src/0.20/java against hadoop 0.20.2 (https://builds.apache.org/job/Hive-0.9.1-SNAPSHOT-h0.21-keepgoing=false/213/artifact/hive/build/hadoopcore/hadoop-0.20.2) ivy-init-settings: [echo] Project: shims ivy-resolve-hadoop-shim: [echo] Project: shims [ivy:resolve] :: loading settings :: file = https://builds.apache.org/job/Hive-0.9.1-SNAPSHOT-h0.21-keepgoing=false/ws/hive/ivy/ivysettings.xml ivy-retrieve-hadoop-shim: [echo] Project: shims [echo] Building shims 0.20S build_shims: [echo] Project: shims [echo] Compiling https://builds.apache.org/job/Hive-0.9.1-SNAPSHOT-h0.21-keepgoing=false/ws/hive/shims/src/common/java;/home/jenkins/jenkins-slave/workspace/Hive-0.9.1-SNAPSHOT-h0.21-keepgoing=false/hive/shims/src/common-secure/java;/home/jenkins/jenkins-slave/workspace/Hive-0.9.1-SNAPSHOT-h0.21-keepgoing=false/hive/shims/src/0.20S/java against hadoop 1.0.0 (https://builds.apache.org/job/Hive-0.9.1-SNAPSHOT-h0.21-keepgoing=false/213/artifact/hive/build/hadoopcore/hadoop-1.0.0) ivy-init-settings: [echo] Project: shims ivy-resolve-hadoop-shim: [echo] Project: shims [ivy:resolve] :: loading settings :: file = https://builds.apache.org/job/Hive-0.9.1-SNAPSHOT-h0.21-keepgoing=false/ws/hive/ivy/ivysettings.xml ivy-retrieve-hadoop-shim: [echo] Project: shims [echo] Building shims 0.23 build_shims: [echo] Project: shims [echo] Compiling https://builds.apache.org/job/Hive-0.9.1-SNAPSHOT-h0.21-keepgoing=false/ws/hive/shims/src/common/java;/home/jenkins/jenkins-slave/workspace/Hive-0.9.1-SNAPSHOT-h0.21-keepgoing=false/hive/shims/src/common-secure/java;/home/jenkins/jenkins-slave/workspace/Hive-0.9.1-SNAPSHOT-h0.21-keepgoing=false/hive/shims/src/0.23/java against hadoop 0.23.3 (https://builds.apache.org/job/Hive-0.9.1-SNAPSHOT-h0.21-keepgoing=false/213/artifact/hive/build/hadoopcore/hadoop-0.23.3)
[jira] [Commented] (HIVE-3646) Add 'IGNORE PROTECTION' predicate for dropping partitions
[ https://issues.apache.org/jira/browse/HIVE-3646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13505697#comment-13505697 ] Andrew Chalfant commented on HIVE-3646: --- Namit should I document before or after the change has been pushed? Add 'IGNORE PROTECTION' predicate for dropping partitions - Key: HIVE-3646 URL: https://issues.apache.org/jira/browse/HIVE-3646 Project: Hive Issue Type: New Feature Components: CLI Affects Versions: 0.9.0 Reporter: Andrew Chalfant Assignee: Andrew Chalfant Priority: Minor Attachments: HIVE-3646.1.patch.txt Original Estimate: 1m Remaining Estimate: 1m There are cases where it is desirable to move partitions between clusters. Having to undo protection and then re-protect tables in order to delete partitions from a source are multi-step and can leave us in a failed open state where partition and table metadata is dirty. By implementing an 'rm -rf'-like functionality, we can perform these operations atomically. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3646) Add 'IGNORE PROTECTION' predicate for dropping partitions
[ https://issues.apache.org/jira/browse/HIVE-3646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13505713#comment-13505713 ] Andrew Chalfant commented on HIVE-3646: --- I spoke with Kevin Wilfong and he said for me to just go ahead and make the changes to the docs, but I need a confluence account and he doesn't have permissions to make the account for me. Can you please make a confluence account with edit permissions for me please? Add 'IGNORE PROTECTION' predicate for dropping partitions - Key: HIVE-3646 URL: https://issues.apache.org/jira/browse/HIVE-3646 Project: Hive Issue Type: New Feature Components: CLI Affects Versions: 0.9.0 Reporter: Andrew Chalfant Assignee: Andrew Chalfant Priority: Minor Attachments: HIVE-3646.1.patch.txt Original Estimate: 1m Remaining Estimate: 1m There are cases where it is desirable to move partitions between clusters. Having to undo protection and then re-protect tables in order to delete partitions from a source are multi-step and can leave us in a failed open state where partition and table metadata is dirty. By implementing an 'rm -rf'-like functionality, we can perform these operations atomically. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3384) HIVE JDBC module won't compile under JDK1.7 as new methods added in JDBC specification
[ https://issues.apache.org/jira/browse/HIVE-3384?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13505722#comment-13505722 ] Phabricator commented on HIVE-3384: --- mbautin has commented on the revision [jira] [HIVE-3384] HIVE JDBC module won't compile under JDK1.7 as new methods added in JDBC specification. INLINE COMMENTS jdbc/src/java/org/apache/hadoop/hive/jdbc/HiveConnection.java:49 Removed. jdbc/src/java/org/apache/hadoop/hive/jdbc/HiveDataSource.java:29 Removed. metastore/build.xml:106 Unfortunately, this seems to be required in order to run unit tests under JDK 7. Without this, verification errors such as P87 start happening, presumably because some of the dependencies have been compiled using an earlier version of javac and do not pass the more strict verification guidelines Java 7 uses by default. REVISION DETAIL https://reviews.facebook.net/D6873 BRANCH trunk To: cdrome, weidongbian, JIRA, ashutoshc, mbautin HIVE JDBC module won't compile under JDK1.7 as new methods added in JDBC specification -- Key: HIVE-3384 URL: https://issues.apache.org/jira/browse/HIVE-3384 Project: Hive Issue Type: Bug Components: JDBC Reporter: Weidong Bian Priority: Minor Attachments: D6873-0.9.1.patch, D6873.1.patch, HIVE-3384.2.patch, HIVE-3384.patch, HIVE-JDK7-JDBC.patch jdbc module couldn't be compiled with jdk7 as it adds some abstract method in the JDBC specification some error info: error: HiveCallableStatement is not abstract and does not override abstract method TgetObject(String,ClassT) in CallableStatement . . . -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-3384) HIVE JDBC module won't compile under JDK1.7 as new methods added in JDBC specification
[ https://issues.apache.org/jira/browse/HIVE-3384?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Phabricator updated HIVE-3384: -- Attachment: D6873.2.patch mbautin updated the revision [jira] [HIVE-3384] HIVE JDBC module won't compile under JDK1.7 as new methods added in JDBC specification. Reviewers: cdrome, weidongbian, JIRA, ashutoshc Addressing Ashutosh's comments. REVISION DETAIL https://reviews.facebook.net/D6873 AFFECTED FILES jdbc/src/java/org/apache/hadoop/hive/jdbc/HiveBaseResultSet.java jdbc/src/java/org/apache/hadoop/hive/jdbc/HiveCallableStatement.java jdbc/src/java/org/apache/hadoop/hive/jdbc/HiveConnection.java jdbc/src/java/org/apache/hadoop/hive/jdbc/HiveDataSource.java jdbc/src/java/org/apache/hadoop/hive/jdbc/HiveDatabaseMetaData.java jdbc/src/java/org/apache/hadoop/hive/jdbc/HiveDriver.java jdbc/src/java/org/apache/hadoop/hive/jdbc/HivePreparedStatement.java jdbc/src/java/org/apache/hadoop/hive/jdbc/HiveQueryResultSet.java jdbc/src/java/org/apache/hadoop/hive/jdbc/HiveStatement.java metastore/build.xml To: cdrome, weidongbian, JIRA, ashutoshc, mbautin HIVE JDBC module won't compile under JDK1.7 as new methods added in JDBC specification -- Key: HIVE-3384 URL: https://issues.apache.org/jira/browse/HIVE-3384 Project: Hive Issue Type: Bug Components: JDBC Reporter: Weidong Bian Priority: Minor Attachments: D6873-0.9.1.patch, D6873.1.patch, D6873.2.patch, HIVE-3384.2.patch, HIVE-3384.patch, HIVE-JDK7-JDBC.patch jdbc module couldn't be compiled with jdk7 as it adds some abstract method in the JDBC specification some error info: error: HiveCallableStatement is not abstract and does not override abstract method TgetObject(String,ClassT) in CallableStatement . . . -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-3646) Add 'IGNORE PROTECTION' predicate for dropping partitions
[ https://issues.apache.org/jira/browse/HIVE-3646?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Chalfant updated HIVE-3646: -- Attachment: HIVE-3646.2.patch.txt New version which includes requested comments Add 'IGNORE PROTECTION' predicate for dropping partitions - Key: HIVE-3646 URL: https://issues.apache.org/jira/browse/HIVE-3646 Project: Hive Issue Type: New Feature Components: CLI Affects Versions: 0.9.0 Reporter: Andrew Chalfant Assignee: Andrew Chalfant Priority: Minor Attachments: HIVE-3646.1.patch.txt, HIVE-3646.2.patch.txt Original Estimate: 1m Remaining Estimate: 1m There are cases where it is desirable to move partitions between clusters. Having to undo protection and then re-protect tables in order to delete partitions from a source are multi-step and can leave us in a failed open state where partition and table metadata is dirty. By implementing an 'rm -rf'-like functionality, we can perform these operations atomically. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3705) Adding authorization capability to the metastore
[ https://issues.apache.org/jira/browse/HIVE-3705?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13505754#comment-13505754 ] Phabricator commented on HIVE-3705: --- khorgath has commented on the revision HIVE-3705 [jira] Adding authorization capability to the metastore. INLINE COMMENTS ql/src/java/org/apache/hadoop/hive/ql/security/authorization/StorageBasedAuthorizationProvider.java:49 Good point. Thought about it a bit, and decided that the best place for this constant was in MetaStoreUtils along with some other similar constants there. Have refactored to push it there. ql/src/java/org/apache/hadoop/hive/ql/security/authorization/StorageBasedAuthorizationProvider.java:77 Agreed. Removing. I was mimicing existing code in HCatalog's HDFS Auth Provider, but you're right, we need to be stricter. ql/src/java/org/apache/hadoop/hive/ql/security/authorization/StorageBasedAuthorizationProvider.java:89 Agreed, same response. ql/src/java/org/apache/hadoop/hive/ql/security/authorization/StorageBasedAuthorizationProvider.java:111 Agreed, same response. ql/src/java/org/apache/hadoop/hive/ql/security/authorization/StorageBasedAuthorizationProvider.java:95 getPath is equivalent to a path constructed on table.getTTable().getSd().getLocation() only if it is nonEmpty, which it is in the else segment. Have made it clearer by avoiding referencing .getPath altogether. ql/src/java/org/apache/hadoop/hive/ql/security/authorization/StorageBasedAuthorizationProvider.java:117 Done. ql/src/java/org/apache/hadoop/hive/ql/security/authorization/StorageBasedAuthorizationProvider.java:122 There isn't a case with partition being null, removing that bit as with 77,89 and 111 above. But it is possible for the location to be null as with the case of creating a partition - when the PreEventListener is triggered, it's possible for the part location to be null, in which case the correct behaviour is to check the table's permissions. ql/src/java/org/apache/hadoop/hive/ql/security/authorization/StorageBasedAuthorizationProvider.java:165 Interesting question - I assumed it was for creating an Index. That said, this is currently unused in Hive - there's no reference to this that I find in the codebase. ql/src/java/org/apache/hadoop/hive/ql/security/authorization/StorageBasedAuthorizationProvider.java:168 Ditto as with 165, but with the exception that HiveOperation does define them as read privileges for both LOCKTABLE and UNLOCKTABLE. That doesn't sound terribly right to me, as I don't think read privileges are enough to be able to perform either of these operations. I'm going to leave this as-is, and ditto with the INDEX case above unless you think we should change it. At any rate, these are not privileges currently in use, even if LOCK is partially defined. ql/src/java/org/apache/hadoop/hive/ql/security/authorization/StorageBasedAuthorizationProvider.java:218 Agreed, changing across all files in this patch. ql/src/java/org/apache/hadoop/hive/ql/security/authorization/StorageBasedAuthorizationProvider.java:301 I was trying to keep changes minimal and not change HiveMetaStore too much, but yes, okay, refactoring, moving over to Warehouse ql/src/java/org/apache/hadoop/hive/ql/security/authorization/StorageBasedAuthorizationProvider.java:308 This does not reimplement Warehouse::getDatabasePath, it extends it(it calls it) by providing a default path if the location was null. ql/src/test/org/apache/hadoop/hive/ql/security/TestAuthorizationPreEventListener.java:44 Agreed, done across tests. ql/src/test/org/apache/hadoop/hive/ql/security/TestDefaultHiveMetastoreAuthorizationProvider.java:137 Added as requested. REVISION DETAIL https://reviews.facebook.net/D6681 BRANCH HIVE-3705 To: JIRA, ashutoshc, khorgath Adding authorization capability to the metastore Key: HIVE-3705 URL: https://issues.apache.org/jira/browse/HIVE-3705 Project: Hive Issue Type: New Feature Components: Authorization, Metastore Reporter: Sushanth Sowmyan Assignee: Sushanth Sowmyan Attachments: HIVE-3705.D6681.1.patch, HIVE-3705.D6681.2.patch, HIVE-3705.D6681.3.patch, hive-backend-auth.2.git.patch, hive-backend-auth.git.patch, hivesec_investigation.pdf In an environment where multiple clients access a single metastore, and we want to evolve hive security to a point where it's no longer simply preventing users from shooting their own foot, we need to be able to authorize metastore calls as well, instead of simply performing every metastore api call that's made. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see:
[jira] [Created] (HIVE-3752) Add a non-sql API in hive to access data.
Nitay Joffe created HIVE-3752: - Summary: Add a non-sql API in hive to access data. Key: HIVE-3752 URL: https://issues.apache.org/jira/browse/HIVE-3752 Project: Hive Issue Type: Improvement Reporter: Nitay Joffe We would like to add an input/output format for accessing Hive data in Hadoop directly without having to use e.g. a transform. Using a transform means having to do a whole map-reduce step with its own disk accesses and its imposed structure. It also means needing to have Hive be the base infrastructure for the entire system being developed which is not the right fit as we only need a small part of it (access to the data). So we propose adding an API level InputFormat and OutputFormat to Hive that will make it trivially easy to select a table with partition spec and read from / write to it. We chose this design to make it compatible with Hadoop so that existing systems that work with Hadoop's IO API will just work out of the box. We need this system for the Giraph graph processing (http://giraph.apache.org/) as running graph jobs which read/write from Hive is a common use case. [~namitjain] [~aching] [~kevinwilfong] [~apresta] -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-3752) Add a non-sql API in hive to access data.
[ https://issues.apache.org/jira/browse/HIVE-3752?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nitay Joffe updated HIVE-3752: -- Description: We would like to add an input/output format for accessing Hive data in Hadoop directly without having to use e.g. a transform. Using a transform means having to do a whole map-reduce step with its own disk accesses and its imposed structure. It also means needing to have Hive be the base infrastructure for the entire system being developed which is not the right fit as we only need a small part of it (access to the data). So we propose adding an API level InputFormat and OutputFormat to Hive that will make it trivially easy to select a table with partition spec and read from / write to it. We chose this design to make it compatible with Hadoop so that existing systems that work with Hadoop's IO API will just work out of the box. We need this system for the Giraph graph processing system (http://giraph.apache.org/) as running graph jobs which read/write from Hive is a common use case. [~namitjain] [~aching] [~kevinwilfong] [~apresta] was: We would like to add an input/output format for accessing Hive data in Hadoop directly without having to use e.g. a transform. Using a transform means having to do a whole map-reduce step with its own disk accesses and its imposed structure. It also means needing to have Hive be the base infrastructure for the entire system being developed which is not the right fit as we only need a small part of it (access to the data). So we propose adding an API level InputFormat and OutputFormat to Hive that will make it trivially easy to select a table with partition spec and read from / write to it. We chose this design to make it compatible with Hadoop so that existing systems that work with Hadoop's IO API will just work out of the box. We need this system for the Giraph graph processing (http://giraph.apache.org/) as running graph jobs which read/write from Hive is a common use case. [~namitjain] [~aching] [~kevinwilfong] [~apresta] Add a non-sql API in hive to access data. - Key: HIVE-3752 URL: https://issues.apache.org/jira/browse/HIVE-3752 Project: Hive Issue Type: Improvement Reporter: Nitay Joffe We would like to add an input/output format for accessing Hive data in Hadoop directly without having to use e.g. a transform. Using a transform means having to do a whole map-reduce step with its own disk accesses and its imposed structure. It also means needing to have Hive be the base infrastructure for the entire system being developed which is not the right fit as we only need a small part of it (access to the data). So we propose adding an API level InputFormat and OutputFormat to Hive that will make it trivially easy to select a table with partition spec and read from / write to it. We chose this design to make it compatible with Hadoop so that existing systems that work with Hadoop's IO API will just work out of the box. We need this system for the Giraph graph processing system (http://giraph.apache.org/) as running graph jobs which read/write from Hive is a common use case. [~namitjain] [~aching] [~kevinwilfong] [~apresta] -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-3705) Adding authorization capability to the metastore
[ https://issues.apache.org/jira/browse/HIVE-3705?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Phabricator updated HIVE-3705: -- Attachment: HIVE-3705.D6681.4.patch khorgath updated the revision HIVE-3705 [jira] Adding authorization capability to the metastore. Reviewers: JIRA, ashutoshc Updates per review comments REVISION DETAIL https://reviews.facebook.net/D6681 AFFECTED FILES common/src/java/org/apache/hadoop/hive/conf/HiveConf.java conf/hive-default.xml.template metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreUtils.java metastore/src/java/org/apache/hadoop/hive/metastore/Warehouse.java ql/src/java/org/apache/hadoop/hive/ql/metadata/HiveUtils.java ql/src/java/org/apache/hadoop/hive/ql/security/HadoopDefaultAuthenticator.java ql/src/java/org/apache/hadoop/hive/ql/security/HadoopDefaultMetastoreAuthenticator.java ql/src/java/org/apache/hadoop/hive/ql/security/HiveMetastoreAuthenticationProvider.java ql/src/java/org/apache/hadoop/hive/ql/security/authorization/AuthorizationPreEventListener.java ql/src/java/org/apache/hadoop/hive/ql/security/authorization/BitSetCheckedAuthorizationProvider.java ql/src/java/org/apache/hadoop/hive/ql/security/authorization/DefaultHiveAuthorizationProvider.java ql/src/java/org/apache/hadoop/hive/ql/security/authorization/DefaultHiveMetastoreAuthorizationProvider.java ql/src/java/org/apache/hadoop/hive/ql/security/authorization/HiveAuthorizationProviderBase.java ql/src/java/org/apache/hadoop/hive/ql/security/authorization/HiveMetastoreAuthorizationProvider.java ql/src/java/org/apache/hadoop/hive/ql/security/authorization/StorageBasedAuthorizationProvider.java ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java ql/src/test/org/apache/hadoop/hive/ql/security/DummyHiveMetastoreAuthorizationProvider.java ql/src/test/org/apache/hadoop/hive/ql/security/InjectableDummyAuthenticator.java ql/src/test/org/apache/hadoop/hive/ql/security/TestAuthorizationPreEventListener.java ql/src/test/org/apache/hadoop/hive/ql/security/TestDefaultHiveMetastoreAuthorizationProvider.java To: JIRA, ashutoshc, khorgath Adding authorization capability to the metastore Key: HIVE-3705 URL: https://issues.apache.org/jira/browse/HIVE-3705 Project: Hive Issue Type: New Feature Components: Authorization, Metastore Reporter: Sushanth Sowmyan Assignee: Sushanth Sowmyan Attachments: HIVE-3705.D6681.1.patch, HIVE-3705.D6681.2.patch, HIVE-3705.D6681.3.patch, HIVE-3705.D6681.4.patch, hive-backend-auth.2.git.patch, hive-backend-auth.git.patch, hivesec_investigation.pdf In an environment where multiple clients access a single metastore, and we want to evolve hive security to a point where it's no longer simply preventing users from shooting their own foot, we need to be able to authorize metastore calls as well, instead of simply performing every metastore api call that's made. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-3384) HIVE JDBC module won't compile under JDK1.7 as new methods added in JDBC specification
[ https://issues.apache.org/jira/browse/HIVE-3384?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Phabricator updated HIVE-3384: -- Attachment: D6873.3.patch mbautin updated the revision [jira] [HIVE-3384] HIVE JDBC module won't compile under JDK1.7 as new methods added in JDBC specification. Reviewers: cdrome, weidongbian, JIRA, ashutoshc Adding another -XX:-UseSplitVerifier for unit tests to work on JDK7. REVISION DETAIL https://reviews.facebook.net/D6873 AFFECTED FILES build-common.xml jdbc/src/java/org/apache/hadoop/hive/jdbc/HiveBaseResultSet.java jdbc/src/java/org/apache/hadoop/hive/jdbc/HiveCallableStatement.java jdbc/src/java/org/apache/hadoop/hive/jdbc/HiveConnection.java jdbc/src/java/org/apache/hadoop/hive/jdbc/HiveDataSource.java jdbc/src/java/org/apache/hadoop/hive/jdbc/HiveDatabaseMetaData.java jdbc/src/java/org/apache/hadoop/hive/jdbc/HiveDriver.java jdbc/src/java/org/apache/hadoop/hive/jdbc/HivePreparedStatement.java jdbc/src/java/org/apache/hadoop/hive/jdbc/HiveQueryResultSet.java jdbc/src/java/org/apache/hadoop/hive/jdbc/HiveStatement.java metastore/build.xml To: cdrome, weidongbian, JIRA, ashutoshc, mbautin HIVE JDBC module won't compile under JDK1.7 as new methods added in JDBC specification -- Key: HIVE-3384 URL: https://issues.apache.org/jira/browse/HIVE-3384 Project: Hive Issue Type: Bug Components: JDBC Reporter: Weidong Bian Priority: Minor Attachments: D6873-0.9.1.patch, D6873.1.patch, D6873.2.patch, D6873.3.patch, HIVE-3384.2.patch, HIVE-3384.patch, HIVE-JDK7-JDBC.patch jdbc module couldn't be compiled with jdk7 as it adds some abstract method in the JDBC specification some error info: error: HiveCallableStatement is not abstract and does not override abstract method TgetObject(String,ClassT) in CallableStatement . . . -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3709) Stop storing default ConfVars in temp file
[ https://issues.apache.org/jira/browse/HIVE-3709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13505838#comment-13505838 ] Kevin Wilfong commented on HIVE-3709: - I had considered the solution Carl mentioned, but I didn't go that route because it could result in clutter in the directory specified, in particular we can't guarantee the file will be deleted in the presence of catastrophic failures. To solve this the user would need to set up some sort of periodic cleanup which puts us back in the same position. We might be able to work around this by regularly touching the file, but I'm not 100% sure. Carl, are you still seeing threading problems with the most recent patch? TestHiveServerSessions has been succeeding for me consistently. Stop storing default ConfVars in temp file -- Key: HIVE-3709 URL: https://issues.apache.org/jira/browse/HIVE-3709 Project: Hive Issue Type: Improvement Components: Configuration Affects Versions: 0.10.0 Reporter: Kevin Wilfong Assignee: Kevin Wilfong Attachments: HIVE-3709.1.patch.txt, HIVE-3709.2.patch.txt, HIVE-3709.3.patch.txt To work around issues with Hadoop's Configuration object, specifically it's addResource(InputStream), default configurations are written to a temp file (I think HIVE-2362 introduced this). This, however, introduces the problem that once that file is deleted from /tmp the client crashes. This is particularly problematic for long running services like the metastore server. Writing a custom InputStream to deal with the problems in the Configuration object should provide a work around, which does not introduce a time bomb into Hive. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
Hive-trunk-h0.21 - Build # 1823 - Still Failing
Changes for Build #1779 Changes for Build #1780 [kevinwilfong] HIVE-3686. Fix compile errors introduced by the interaction of HIVE-1362 and HIVE-3524. (Shreepadma Venugopalan via kevinwilfong) Changes for Build #1781 [namit] HIVE-3687 smb_mapjoin_13.q is nondeterministic (Kevin Wilfong via namit) Changes for Build #1782 [hashutosh] HIVE-2715: Upgrade Thrift dependency to 0.9.0 (Ashutosh Chauhan) Changes for Build #1783 [kevinwilfong] HIVE-3654. block relative path access in hive. (njain via kevinwilfong) [hashutosh] HIVE-3658 : Unable to generate the Hbase related unit tests using velocity templates on Windows (Kanna Karanam via Ashutosh Chauhan) [hashutosh] HIVE-3661 : Remove the Windows specific = related swizzle path changes from Proxy FileSystems (Kanna Karanam via Ashutosh Chauhan) [hashutosh] HIVE-3480 : Resource leak: Fix the file handle leaks in Symbolic Symlink related input formats. (Kanna Karanam via Ashutosh Chauhan) Changes for Build #1784 [kevinwilfong] HIVE-3675. NaN does not work correctly for round(n). (njain via kevinwilfong) [cws] HIVE-3651. bucketmapjoin?.q tests fail with hadoop 0.23 (Prasad Mujumdar via cws) Changes for Build #1785 [namit] HIVE-3613 Implement grouping_id function (Ian Gorbachev via namit) [namit] HIVE-3692 Update parallel test documentation (Ivan Gorbachev via namit) [namit] HIVE-3649 Hive List Bucketing - enhance DDL to specify list bucketing table (Gang Tim Liu via namit) Changes for Build #1786 [namit] HIVE-3696 Revert HIVE-3483 which causes performance regression (Gang Tim Liu via namit) Changes for Build #1787 [kevinwilfong] HIVE-3621. Make prompt in Hive CLI configurable. (Jingwei Lu via kevinwilfong) [kevinwilfong] HIVE-3695. TestParse breaks due to HIVE-3675. (njain via kevinwilfong) Changes for Build #1788 [kevinwilfong] HIVE-3557. Access to external URLs in hivetest.py. (Ivan Gorbachev via kevinwilfong) Changes for Build #1789 [hashutosh] HIVE-3662 : TestHiveServer: testScratchDirShouldClearWhileStartup is failing on Windows (Kanna Karanam via Ashutosh Chauhan) [hashutosh] HIVE-3659 : TestHiveHistory::testQueryloglocParentDirNotExist Test fails on Windows because of some resource leaks in ZK (Kanna Karanam via Ashutosh Chauhan) [hashutosh] HIVE-3663 Unable to display the MR Job file path on Windows in case of MR job failures. (Kanna Karanam via Ashutosh Chauhan) Changes for Build #1790 Changes for Build #1791 Changes for Build #1792 Changes for Build #1793 [hashutosh] HIVE-3704 : name of some metastore scripts are not per convention (Ashutosh Chauhan) Changes for Build #1794 [hashutosh] HIVE-3243 : ignore white space between entries of hive/hbase table mapping (Shengsheng Huang via Ashutosh Chauhan) [hashutosh] HIVE-3215 : JobDebugger should use RunningJob.getTrackingURL (Bhushan Mandhani via Ashutosh Chauhan) Changes for Build #1795 [cws] HIVE-3437. 0.23 compatibility: fix unit tests when building against 0.23 (Chris Drome via cws) [hashutosh] HIVE-3626 : RetryingHMSHandler should wrap JDOException inside MetaException (Bhushan Mandhani via Ashutosh Chauhan) [hashutosh] HIVE-3560 : Hive always prints a warning message when using remote metastore (Travis Crawford via Ashutosh Chauhan) Changes for Build #1796 Changes for Build #1797 [hashutosh] HIVE-3664 : Avoid to create a symlink for hive-contrib.jar file in dist\lib folder. (Kanna Karanam via Ashutosh Chauhan) Changes for Build #1798 [namit] HIVE-3706 getBoolVar in FileSinkOperator can be optimized (Kevin Wilfong via namit) [namit] HIVE-3707 Round map/reduce progress down when it is in the range [99.5, 100) (Kevin Wilfong via namit) [namit] HIVE-3471 Implement grouping sets in hive (Ivan Gorbachev via namit) Changes for Build #1799 [hashutosh] HIVE-3291 : fix fs resolvers (Ashish Singh via Ashutosh Chauhan) [hashutosh] HIVE-3680 : Include Table information in Hive's AddPartitionEvent. (Mithun Radhakrishnan via Ashutosh Chauhan) Changes for Build #1800 [hashutosh] HIVE-3520 : ivysettings.xml does not let you override .m2/repository (Raja Aluri via Ashutosh Chauhan) [hashutosh] HIVE-3435 : Get pdk pluginTest passed when triggered from both builtin tests and pdk tests on hadoop23 (Zhenxiao Luo via Ashutosh Chauhan) Changes for Build #1801 Changes for Build #1802 Changes for Build #1803 Changes for Build #1804 Changes for Build #1805 Changes for Build #1806 [hashutosh] Preparing for 0.11 development [kevinwilfong] HIVE-3647. map-side groupby wrongly due to HIVE-3432. (njain via kevinwilfong) Changes for Build #1807 [kevinwilfong] HIVE-3679. Unescape partition names returned by show partitions. (Wonho Kim via kevinwilfong) Changes for Build #1808 [kevinwilfong] Remove extra files accidentally committed with HIVE-3679. (kevinwilfong) Changes for Build #1809 Changes for Build #1810 Changes for Build #1811 [namit] HIVE-3735 PTest doesn't work due to hive snapshot version upgrade to 11 (Gang Tim Liu via namit)
Build failed in Jenkins: Hive-0.9.1-SNAPSHOT-h0.21 #213
See https://builds.apache.org/job/Hive-0.9.1-SNAPSHOT-h0.21/213/ -- [...truncated 36521 lines...] [junit] POSTHOOK: Input: default@testhivedrivertable [junit] POSTHOOK: Output: file:/tmp/hudson/hive_2012-11-28_14-14-24_382_6161782169221319366/-mr-1 [junit] OK [junit] PREHOOK: query: drop table testhivedrivertable [junit] PREHOOK: type: DROPTABLE [junit] PREHOOK: Input: default@testhivedrivertable [junit] PREHOOK: Output: default@testhivedrivertable [junit] POSTHOOK: query: drop table testhivedrivertable [junit] POSTHOOK: type: DROPTABLE [junit] POSTHOOK: Input: default@testhivedrivertable [junit] POSTHOOK: Output: default@testhivedrivertable [junit] OK [junit] Hive history file=https://builds.apache.org/job/Hive-0.9.1-SNAPSHOT-h0.21/213/artifact/hive/build/service/tmp/hive_job_log_hudson_201211281414_657969550.txt [junit] PREHOOK: query: drop table testhivedrivertable [junit] PREHOOK: type: DROPTABLE [junit] POSTHOOK: query: drop table testhivedrivertable [junit] POSTHOOK: type: DROPTABLE [junit] OK [junit] PREHOOK: query: create table testhivedrivertable (num int) [junit] PREHOOK: type: DROPTABLE [junit] Copying file: https://builds.apache.org/job/Hive-0.9.1-SNAPSHOT-h0.21/ws/hive/data/files/kv1.txt [junit] POSTHOOK: query: create table testhivedrivertable (num int) [junit] POSTHOOK: type: DROPTABLE [junit] POSTHOOK: Output: default@testhivedrivertable [junit] OK [junit] PREHOOK: query: load data local inpath 'https://builds.apache.org/job/Hive-0.9.1-SNAPSHOT-h0.21/ws/hive/data/files/kv1.txt' into table testhivedrivertable [junit] PREHOOK: type: DROPTABLE [junit] PREHOOK: Output: default@testhivedrivertable [junit] Copying data from https://builds.apache.org/job/Hive-0.9.1-SNAPSHOT-h0.21/ws/hive/data/files/kv1.txt [junit] Loading data to table default.testhivedrivertable [junit] POSTHOOK: query: load data local inpath 'https://builds.apache.org/job/Hive-0.9.1-SNAPSHOT-h0.21/ws/hive/data/files/kv1.txt' into table testhivedrivertable [junit] POSTHOOK: type: DROPTABLE [junit] POSTHOOK: Output: default@testhivedrivertable [junit] OK [junit] PREHOOK: query: select * from testhivedrivertable limit 10 [junit] PREHOOK: type: DROPTABLE [junit] PREHOOK: Input: default@testhivedrivertable [junit] PREHOOK: Output: file:/tmp/hudson/hive_2012-11-28_14-14-28_795_3773609989986/-mr-1 [junit] POSTHOOK: query: select * from testhivedrivertable limit 10 [junit] POSTHOOK: type: DROPTABLE [junit] POSTHOOK: Input: default@testhivedrivertable [junit] POSTHOOK: Output: file:/tmp/hudson/hive_2012-11-28_14-14-28_795_3773609989986/-mr-1 [junit] OK [junit] PREHOOK: query: drop table testhivedrivertable [junit] PREHOOK: type: DROPTABLE [junit] PREHOOK: Input: default@testhivedrivertable [junit] PREHOOK: Output: default@testhivedrivertable [junit] POSTHOOK: query: drop table testhivedrivertable [junit] POSTHOOK: type: DROPTABLE [junit] POSTHOOK: Input: default@testhivedrivertable [junit] POSTHOOK: Output: default@testhivedrivertable [junit] OK [junit] Hive history file=https://builds.apache.org/job/Hive-0.9.1-SNAPSHOT-h0.21/213/artifact/hive/build/service/tmp/hive_job_log_hudson_201211281414_1311558659.txt [junit] PREHOOK: query: drop table testhivedrivertable [junit] PREHOOK: type: DROPTABLE [junit] POSTHOOK: query: drop table testhivedrivertable [junit] POSTHOOK: type: DROPTABLE [junit] OK [junit] PREHOOK: query: create table testhivedrivertable (num int) [junit] PREHOOK: type: DROPTABLE [junit] POSTHOOK: query: create table testhivedrivertable (num int) [junit] POSTHOOK: type: DROPTABLE [junit] POSTHOOK: Output: default@testhivedrivertable [junit] OK [junit] PREHOOK: query: drop table testhivedrivertable [junit] PREHOOK: type: DROPTABLE [junit] PREHOOK: Input: default@testhivedrivertable [junit] PREHOOK: Output: default@testhivedrivertable [junit] POSTHOOK: query: drop table testhivedrivertable [junit] POSTHOOK: type: DROPTABLE [junit] POSTHOOK: Input: default@testhivedrivertable [junit] POSTHOOK: Output: default@testhivedrivertable [junit] OK [junit] Hive history file=https://builds.apache.org/job/Hive-0.9.1-SNAPSHOT-h0.21/213/artifact/hive/build/service/tmp/hive_job_log_hudson_201211281414_697160228.txt [junit] Hive history file=https://builds.apache.org/job/Hive-0.9.1-SNAPSHOT-h0.21/213/artifact/hive/build/service/tmp/hive_job_log_hudson_201211281414_1440676227.txt [junit] Copying file: https://builds.apache.org/job/Hive-0.9.1-SNAPSHOT-h0.21/ws/hive/data/files/kv1.txt [junit] PREHOOK: query: drop table testhivedrivertable [junit] PREHOOK: type: DROPTABLE [junit] POSTHOOK: query: drop table testhivedrivertable [junit] POSTHOOK: type:
[jira] [Created] (HIVE-3754) Trunk hadoop 23 build fails
Gang Tim Liu created HIVE-3754:
--
Summary: Trunk hadoop 23 build fails
Key: HIVE-3754
URL: https://issues.apache.org/jira/browse/HIVE-3754
Project: Hive
Issue Type: Bug
Components: Build Infrastructure
Reporter: Gang Tim Liu
check out the latest code from trunk
svn info
Path: .
URL: http://svn.apache.org/repos/asf/hive/trunk
Repository Root: http://svn.apache.org/repos/asf
Repository UUID: 13f79535-47bb-0310-9956-ffa450edef68
Revision: 1415005
Node Kind: directory
Schedule: normal
Last Changed Author: namit
Last Changed Rev: 1414608
Last Changed Date: 2012-11-28 01:36:27 -0800 (Wed, 28 Nov 2012)
ant clean package -Dhadoop.version=0.23.1 -Dhadoop-0.23.version=0.23.1
-Dhadoop.mr.rev=23
..
{quote}
ivy-retrieve-hadoop-shim:
[echo] Project: shims
[javac] Compiling 2 source files to
/Users/gang/hive-trunk-11-28/build/shims/classes
[javac]
/Users/gang/hive-trunk-11-28/shims/src/0.23/java/org/apache/hadoop/hive/shims/Hadoop23Shims.java:122:
getDefaultBlockSize() in org.apache.hadoop.fs.FileSystem cannot be applied to
(org.apache.hadoop.fs.Path)
[javac] return fs.getDefaultBlockSize(path);
[javac] ^
[javac]
/Users/gang/hive-trunk-11-28/shims/src/0.23/java/org/apache/hadoop/hive/shims/Hadoop23Shims.java:127:
getDefaultReplication() in org.apache.hadoop.fs.FileSystem cannot be applied
to (org.apache.hadoop.fs.Path)
[javac] return fs.getDefaultReplication(path);
[javac] ^
[javac] 2 errors
BUILD FAILED
/Users/gang/hive-trunk-11-28/build.xml:302: The following error occurred while
executing this line:
{quote}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-3754) Trunk hadoop 23 build fails
[
https://issues.apache.org/jira/browse/HIVE-3754?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gang Tim Liu updated HIVE-3754:
---
Description:
check out the latest code from trunk
svn info
{quote}
Path: .
URL: http://svn.apache.org/repos/asf/hive/trunk
Repository Root: http://svn.apache.org/repos/asf
Repository UUID: 13f79535-47bb-0310-9956-ffa450edef68
Revision: 1415005
Node Kind: directory
Schedule: normal
Last Changed Author: namit
Last Changed Rev: 1414608
Last Changed Date: 2012-11-28 01:36:27 -0800 (Wed, 28 Nov 2012)
{quote}
ant clean package -Dhadoop.version=0.23.1 -Dhadoop-0.23.version=0.23.1
-Dhadoop.mr.rev=23
{quote}
ivy-retrieve-hadoop-shim:
[echo] Project: shims
[javac] Compiling 2 source files to
/Users/gang/hive-trunk-11-28/build/shims/classes
[javac]
/Users/gang/hive-trunk-11-28/shims/src/0.23/java/org/apache/hadoop/hive/shims/Hadoop23Shims.java:122:
getDefaultBlockSize() in org.apache.hadoop.fs.FileSystem cannot be applied to
(org.apache.hadoop.fs.Path)
[javac] return fs.getDefaultBlockSize(path);
[javac] ^
[javac]
/Users/gang/hive-trunk-11-28/shims/src/0.23/java/org/apache/hadoop/hive/shims/Hadoop23Shims.java:127:
getDefaultReplication() in org.apache.hadoop.fs.FileSystem cannot be applied
to (org.apache.hadoop.fs.Path)
[javac] return fs.getDefaultReplication(path);
[javac] ^
[javac] 2 errors
BUILD FAILED
/Users/gang/hive-trunk-11-28/build.xml:302: The following error occurred while
executing this line:
{quote}
was:
check out the latest code from trunk
svn info
Path: .
URL: http://svn.apache.org/repos/asf/hive/trunk
Repository Root: http://svn.apache.org/repos/asf
Repository UUID: 13f79535-47bb-0310-9956-ffa450edef68
Revision: 1415005
Node Kind: directory
Schedule: normal
Last Changed Author: namit
Last Changed Rev: 1414608
Last Changed Date: 2012-11-28 01:36:27 -0800 (Wed, 28 Nov 2012)
ant clean package -Dhadoop.version=0.23.1 -Dhadoop-0.23.version=0.23.1
-Dhadoop.mr.rev=23
..
{quote}
ivy-retrieve-hadoop-shim:
[echo] Project: shims
[javac] Compiling 2 source files to
/Users/gang/hive-trunk-11-28/build/shims/classes
[javac]
/Users/gang/hive-trunk-11-28/shims/src/0.23/java/org/apache/hadoop/hive/shims/Hadoop23Shims.java:122:
getDefaultBlockSize() in org.apache.hadoop.fs.FileSystem cannot be applied to
(org.apache.hadoop.fs.Path)
[javac] return fs.getDefaultBlockSize(path);
[javac] ^
[javac]
/Users/gang/hive-trunk-11-28/shims/src/0.23/java/org/apache/hadoop/hive/shims/Hadoop23Shims.java:127:
getDefaultReplication() in org.apache.hadoop.fs.FileSystem cannot be applied
to (org.apache.hadoop.fs.Path)
[javac] return fs.getDefaultReplication(path);
[javac] ^
[javac] 2 errors
BUILD FAILED
/Users/gang/hive-trunk-11-28/build.xml:302: The following error occurred while
executing this line:
{quote}
Trunk hadoop 23 build fails
---
Key: HIVE-3754
URL: https://issues.apache.org/jira/browse/HIVE-3754
Project: Hive
Issue Type: Bug
Components: Build Infrastructure
Reporter: Gang Tim Liu
check out the latest code from trunk
svn info
{quote}
Path: .
URL: http://svn.apache.org/repos/asf/hive/trunk
Repository Root: http://svn.apache.org/repos/asf
Repository UUID: 13f79535-47bb-0310-9956-ffa450edef68
Revision: 1415005
Node Kind: directory
Schedule: normal
Last Changed Author: namit
Last Changed Rev: 1414608
Last Changed Date: 2012-11-28 01:36:27 -0800 (Wed, 28 Nov 2012)
{quote}
ant clean package -Dhadoop.version=0.23.1 -Dhadoop-0.23.version=0.23.1
-Dhadoop.mr.rev=23
{quote}
ivy-retrieve-hadoop-shim:
[echo] Project: shims
[javac] Compiling 2 source files to
/Users/gang/hive-trunk-11-28/build/shims/classes
[javac]
/Users/gang/hive-trunk-11-28/shims/src/0.23/java/org/apache/hadoop/hive/shims/Hadoop23Shims.java:122:
getDefaultBlockSize() in org.apache.hadoop.fs.FileSystem cannot be applied
to (org.apache.hadoop.fs.Path)
[javac] return fs.getDefaultBlockSize(path);
[javac] ^
[javac]
/Users/gang/hive-trunk-11-28/shims/src/0.23/java/org/apache/hadoop/hive/shims/Hadoop23Shims.java:127:
getDefaultReplication() in org.apache.hadoop.fs.FileSystem cannot be applied
to (org.apache.hadoop.fs.Path)
[javac] return fs.getDefaultReplication(path);
[javac] ^
[javac] 2 errors
BUILD FAILED
/Users/gang/hive-trunk-11-28/build.xml:302: The following error occurred
while executing this line:
{quote}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-3754) Trunk hadoop 23 build fails
[
https://issues.apache.org/jira/browse/HIVE-3754?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gang Tim Liu updated HIVE-3754:
---
Description:
check out the latest code from trunk
{code}
svn info
{code}
{quote}
Path: .
URL: http://svn.apache.org/repos/asf/hive/trunk
Repository Root: http://svn.apache.org/repos/asf
Repository UUID: 13f79535-47bb-0310-9956-ffa450edef68
Revision: 1415005
Node Kind: directory
Schedule: normal
Last Changed Author: namit
Last Changed Rev: 1414608
Last Changed Date: 2012-11-28 01:36:27 -0800 (Wed, 28 Nov 2012)
{quote}
{code}
ant clean package -Dhadoop.version=0.23.1 -Dhadoop-0.23.version=0.23.1
-Dhadoop.mr.rev=23
{code}
{quote}
ivy-retrieve-hadoop-shim:
[echo] Project: shims
[javac] Compiling 2 source files to
/Users/gang/hive-trunk-11-28/build/shims/classes
[javac]
/Users/gang/hive-trunk-11-28/shims/src/0.23/java/org/apache/hadoop/hive/shims/Hadoop23Shims.java:122:
getDefaultBlockSize() in org.apache.hadoop.fs.FileSystem cannot be applied to
(org.apache.hadoop.fs.Path)
[javac] return fs.getDefaultBlockSize(path);
[javac] ^
[javac]
/Users/gang/hive-trunk-11-28/shims/src/0.23/java/org/apache/hadoop/hive/shims/Hadoop23Shims.java:127:
getDefaultReplication() in org.apache.hadoop.fs.FileSystem cannot be applied
to (org.apache.hadoop.fs.Path)
[javac] return fs.getDefaultReplication(path);
[javac] ^
[javac] 2 errors
BUILD FAILED
/Users/gang/hive-trunk-11-28/build.xml:302: The following error occurred while
executing this line:
{quote}
was:
check out the latest code from trunk
{code}
svn info
{code}
{quote}
Path: .
URL: http://svn.apache.org/repos/asf/hive/trunk
Repository Root: http://svn.apache.org/repos/asf
Repository UUID: 13f79535-47bb-0310-9956-ffa450edef68
Revision: 1415005
Node Kind: directory
Schedule: normal
Last Changed Author: namit
Last Changed Rev: 1414608
Last Changed Date: 2012-11-28 01:36:27 -0800 (Wed, 28 Nov 2012)
{quote}
ant clean package -Dhadoop.version=0.23.1 -Dhadoop-0.23.version=0.23.1
-Dhadoop.mr.rev=23
{quote}
ivy-retrieve-hadoop-shim:
[echo] Project: shims
[javac] Compiling 2 source files to
/Users/gang/hive-trunk-11-28/build/shims/classes
[javac]
/Users/gang/hive-trunk-11-28/shims/src/0.23/java/org/apache/hadoop/hive/shims/Hadoop23Shims.java:122:
getDefaultBlockSize() in org.apache.hadoop.fs.FileSystem cannot be applied to
(org.apache.hadoop.fs.Path)
[javac] return fs.getDefaultBlockSize(path);
[javac] ^
[javac]
/Users/gang/hive-trunk-11-28/shims/src/0.23/java/org/apache/hadoop/hive/shims/Hadoop23Shims.java:127:
getDefaultReplication() in org.apache.hadoop.fs.FileSystem cannot be applied
to (org.apache.hadoop.fs.Path)
[javac] return fs.getDefaultReplication(path);
[javac] ^
[javac] 2 errors
BUILD FAILED
/Users/gang/hive-trunk-11-28/build.xml:302: The following error occurred while
executing this line:
{quote}
Trunk hadoop 23 build fails
---
Key: HIVE-3754
URL: https://issues.apache.org/jira/browse/HIVE-3754
Project: Hive
Issue Type: Bug
Components: Build Infrastructure
Reporter: Gang Tim Liu
check out the latest code from trunk
{code}
svn info
{code}
{quote}
Path: .
URL: http://svn.apache.org/repos/asf/hive/trunk
Repository Root: http://svn.apache.org/repos/asf
Repository UUID: 13f79535-47bb-0310-9956-ffa450edef68
Revision: 1415005
Node Kind: directory
Schedule: normal
Last Changed Author: namit
Last Changed Rev: 1414608
Last Changed Date: 2012-11-28 01:36:27 -0800 (Wed, 28 Nov 2012)
{quote}
{code}
ant clean package -Dhadoop.version=0.23.1 -Dhadoop-0.23.version=0.23.1
-Dhadoop.mr.rev=23
{code}
{quote}
ivy-retrieve-hadoop-shim:
[echo] Project: shims
[javac] Compiling 2 source files to
/Users/gang/hive-trunk-11-28/build/shims/classes
[javac]
/Users/gang/hive-trunk-11-28/shims/src/0.23/java/org/apache/hadoop/hive/shims/Hadoop23Shims.java:122:
getDefaultBlockSize() in org.apache.hadoop.fs.FileSystem cannot be applied
to (org.apache.hadoop.fs.Path)
[javac] return fs.getDefaultBlockSize(path);
[javac] ^
[javac]
/Users/gang/hive-trunk-11-28/shims/src/0.23/java/org/apache/hadoop/hive/shims/Hadoop23Shims.java:127:
getDefaultReplication() in org.apache.hadoop.fs.FileSystem cannot be applied
to (org.apache.hadoop.fs.Path)
[javac] return fs.getDefaultReplication(path);
[javac] ^
[javac] 2 errors
BUILD FAILED
/Users/gang/hive-trunk-11-28/build.xml:302: The following error occurred
while executing this line:
{quote}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see:
[jira] [Commented] (HIVE-3754) Trunk hadoop 23 build fails
[
https://issues.apache.org/jira/browse/HIVE-3754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13506092#comment-13506092
]
Gang Tim Liu commented on HIVE-3754:
[~ashutoshc] [~cwsteinbach], are you aware of any build command change? Is this
still the right way to build hadoop 23? It has been working well except recent
trunk. Just ensure it's right way. thanks
{code}
ant clean package -Dhadoop.version=0.23.1 -Dhadoop-0.23.version=0.23.1
-Dhadoop.mr.rev=23
{code}
Trunk hadoop 23 build fails
---
Key: HIVE-3754
URL: https://issues.apache.org/jira/browse/HIVE-3754
Project: Hive
Issue Type: Bug
Components: Build Infrastructure
Reporter: Gang Tim Liu
check out the latest code from trunk
{code}
svn info
{code}
{quote}
Path: .
URL: http://svn.apache.org/repos/asf/hive/trunk
Repository Root: http://svn.apache.org/repos/asf
Repository UUID: 13f79535-47bb-0310-9956-ffa450edef68
Revision: 1415005
Node Kind: directory
Schedule: normal
Last Changed Author: namit
Last Changed Rev: 1414608
Last Changed Date: 2012-11-28 01:36:27 -0800 (Wed, 28 Nov 2012)
{quote}
{code}
ant clean package -Dhadoop.version=0.23.1 -Dhadoop-0.23.version=0.23.1
-Dhadoop.mr.rev=23
{code}
{quote}
ivy-retrieve-hadoop-shim:
[echo] Project: shims
[javac] Compiling 2 source files to
/Users/gang/hive-trunk-11-28/build/shims/classes
[javac]
/Users/gang/hive-trunk-11-28/shims/src/0.23/java/org/apache/hadoop/hive/shims/Hadoop23Shims.java:122:
getDefaultBlockSize() in org.apache.hadoop.fs.FileSystem cannot be applied
to (org.apache.hadoop.fs.Path)
[javac] return fs.getDefaultBlockSize(path);
[javac] ^
[javac]
/Users/gang/hive-trunk-11-28/shims/src/0.23/java/org/apache/hadoop/hive/shims/Hadoop23Shims.java:127:
getDefaultReplication() in org.apache.hadoop.fs.FileSystem cannot be applied
to (org.apache.hadoop.fs.Path)
[javac] return fs.getDefaultReplication(path);
[javac] ^
[javac] 2 errors
BUILD FAILED
/Users/gang/hive-trunk-11-28/build.xml:302: The following error occurred
while executing this line:
{quote}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2800) NPE in create index without comment clause in external metastore
[ https://issues.apache.org/jira/browse/HIVE-2800?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13506120#comment-13506120 ] Phabricator commented on HIVE-2800: --- khorgath has abandoned the revision HIVE-2800 [jira] NPE in create index without comment clause in external metastore. Same change implemented by HIVE-3722, redundant now. REVISION DETAIL https://reviews.facebook.net/D3519 To: JIRA, ashutoshc, khorgath NPE in create index without comment clause in external metastore -- Key: HIVE-2800 URL: https://issues.apache.org/jira/browse/HIVE-2800 Project: Hive Issue Type: Bug Components: Metastore Affects Versions: 0.7.1, 0.8.1, 0.9.0 Reporter: Daniel Dai Assignee: Sushanth Sowmyan Priority: Minor Fix For: 0.10.0 This happens only when using external metastore (with --hiveconf hive.metastore.uris=thrift://localhost:8088 --hiveconf hive.metastore.local=false). Also if I gave a comment in the statement, this exception go away. Here is the statement: create index test111 on table hcat_test(name) as 'compact' with deferred rebuild; Here is the stack: 2012-02-10 17:07:42,612 ERROR exec.Task (SessionState.java:printError(380)) - FAILED: Error in metadata: java.lang.NullPointerException org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.NullPointerException at org.apache.hadoop.hive.ql.metadata.Hive.createIndex(Hive.java:725) at org.apache.hadoop.hive.ql.exec.DDLTask.createIndex(DDLTask.java:822) at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:231) at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:134) at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:57) at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1291) at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1082) at org.apache.hadoop.hive.ql.Driver.run(Driver.java:933) at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:255) at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:212) at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:403) at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:671) at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:554) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.util.RunJar.main(RunJar.java:156) Caused by: java.lang.NullPointerException at org.apache.thrift.protocol.TBinaryProtocol.writeString(TBinaryProtocol.java:185) at org.apache.hadoop.hive.metastore.api.Index.write(Index.java:1032) at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$add_index_args.write(ThriftHiveMetastore.java:47518) at org.apache.thrift.TServiceClient.sendBase(TServiceClient.java:63) at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.send_add_index(ThriftHiveMetastore.java:1675) at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.add_index(ThriftHiveMetastore.java:1666) at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.createIndex(HiveMetaStoreClient.java:853) at org.apache.hadoop.hive.ql.metadata.Hive.createIndex(Hive.java:722) ... 17 more -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3754) Trunk hadoop 23 build fails
[
https://issues.apache.org/jira/browse/HIVE-3754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13506141#comment-13506141
]
Ashutosh Chauhan commented on HIVE-3754:
CCing: [~amalakar] This seems to be related to HIVE-3645
Trunk hadoop 23 build fails
---
Key: HIVE-3754
URL: https://issues.apache.org/jira/browse/HIVE-3754
Project: Hive
Issue Type: Bug
Components: Build Infrastructure
Reporter: Gang Tim Liu
check out the latest code from trunk
{code}
svn info
{code}
{quote}
Path: .
URL: http://svn.apache.org/repos/asf/hive/trunk
Repository Root: http://svn.apache.org/repos/asf
Repository UUID: 13f79535-47bb-0310-9956-ffa450edef68
Revision: 1415005
Node Kind: directory
Schedule: normal
Last Changed Author: namit
Last Changed Rev: 1414608
Last Changed Date: 2012-11-28 01:36:27 -0800 (Wed, 28 Nov 2012)
{quote}
{code}
ant clean package -Dhadoop.version=0.23.1 -Dhadoop-0.23.version=0.23.1
-Dhadoop.mr.rev=23
{code}
{quote}
ivy-retrieve-hadoop-shim:
[echo] Project: shims
[javac] Compiling 2 source files to
/Users/gang/hive-trunk-11-28/build/shims/classes
[javac]
/Users/gang/hive-trunk-11-28/shims/src/0.23/java/org/apache/hadoop/hive/shims/Hadoop23Shims.java:122:
getDefaultBlockSize() in org.apache.hadoop.fs.FileSystem cannot be applied
to (org.apache.hadoop.fs.Path)
[javac] return fs.getDefaultBlockSize(path);
[javac] ^
[javac]
/Users/gang/hive-trunk-11-28/shims/src/0.23/java/org/apache/hadoop/hive/shims/Hadoop23Shims.java:127:
getDefaultReplication() in org.apache.hadoop.fs.FileSystem cannot be applied
to (org.apache.hadoop.fs.Path)
[javac] return fs.getDefaultReplication(path);
[javac] ^
[javac] 2 errors
BUILD FAILED
/Users/gang/hive-trunk-11-28/build.xml:302: The following error occurred
while executing this line:
{quote}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3754) Trunk hadoop 23 build fails
[
https://issues.apache.org/jira/browse/HIVE-3754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13506149#comment-13506149
]
Gang Tim Liu commented on HIVE-3754:
[~ashutoshc] thank you very much Tim
[~amalakar] [~viraj] would you please check if it is introduced by HIVE-3645?
If yes, would you please fix it in trunk? It's blocking me. thanks a lot Tim
Trunk hadoop 23 build fails
---
Key: HIVE-3754
URL: https://issues.apache.org/jira/browse/HIVE-3754
Project: Hive
Issue Type: Bug
Components: Build Infrastructure
Reporter: Gang Tim Liu
check out the latest code from trunk
{code}
svn info
{code}
{quote}
Path: .
URL: http://svn.apache.org/repos/asf/hive/trunk
Repository Root: http://svn.apache.org/repos/asf
Repository UUID: 13f79535-47bb-0310-9956-ffa450edef68
Revision: 1415005
Node Kind: directory
Schedule: normal
Last Changed Author: namit
Last Changed Rev: 1414608
Last Changed Date: 2012-11-28 01:36:27 -0800 (Wed, 28 Nov 2012)
{quote}
{code}
ant clean package -Dhadoop.version=0.23.1 -Dhadoop-0.23.version=0.23.1
-Dhadoop.mr.rev=23
{code}
{quote}
ivy-retrieve-hadoop-shim:
[echo] Project: shims
[javac] Compiling 2 source files to
/Users/gang/hive-trunk-11-28/build/shims/classes
[javac]
/Users/gang/hive-trunk-11-28/shims/src/0.23/java/org/apache/hadoop/hive/shims/Hadoop23Shims.java:122:
getDefaultBlockSize() in org.apache.hadoop.fs.FileSystem cannot be applied
to (org.apache.hadoop.fs.Path)
[javac] return fs.getDefaultBlockSize(path);
[javac] ^
[javac]
/Users/gang/hive-trunk-11-28/shims/src/0.23/java/org/apache/hadoop/hive/shims/Hadoop23Shims.java:127:
getDefaultReplication() in org.apache.hadoop.fs.FileSystem cannot be applied
to (org.apache.hadoop.fs.Path)
[javac] return fs.getDefaultReplication(path);
[javac] ^
[javac] 2 errors
BUILD FAILED
/Users/gang/hive-trunk-11-28/build.xml:302: The following error occurred
while executing this line:
{quote}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3693) Performance regression introduced by HIVE-3483
[ https://issues.apache.org/jira/browse/HIVE-3693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13506158#comment-13506158 ] Ashutosh Chauhan commented on HIVE-3693: +1. Will commit if tests pass. Performance regression introduced by HIVE-3483 -- Key: HIVE-3693 URL: https://issues.apache.org/jira/browse/HIVE-3693 Project: Hive Issue Type: Bug Reporter: Gang Tim Liu Assignee: Thejas M Nair Priority: Minor Attachments: HIVE-3693.1.patch https://issues.apache.org/jira/browse/HIVE-3483 introduced a performance regression in the client side during split computation. The client side spends a lot more time in the split computation phase. The problem is checkFilterPathContains method. While investigating, can you create a config to disable it by default? thanks -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3754) Trunk hadoop 23 build fails
[
https://issues.apache.org/jira/browse/HIVE-3754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13506161#comment-13506161
]
Viraj Bhat commented on HIVE-3754:
--
Arup Malakar recently checked this in to enable Federation capability.
Viraj
Trunk hadoop 23 build fails
---
Key: HIVE-3754
URL: https://issues.apache.org/jira/browse/HIVE-3754
Project: Hive
Issue Type: Bug
Components: Build Infrastructure
Reporter: Gang Tim Liu
check out the latest code from trunk
{code}
svn info
{code}
{quote}
Path: .
URL: http://svn.apache.org/repos/asf/hive/trunk
Repository Root: http://svn.apache.org/repos/asf
Repository UUID: 13f79535-47bb-0310-9956-ffa450edef68
Revision: 1415005
Node Kind: directory
Schedule: normal
Last Changed Author: namit
Last Changed Rev: 1414608
Last Changed Date: 2012-11-28 01:36:27 -0800 (Wed, 28 Nov 2012)
{quote}
{code}
ant clean package -Dhadoop.version=0.23.1 -Dhadoop-0.23.version=0.23.1
-Dhadoop.mr.rev=23
{code}
{quote}
ivy-retrieve-hadoop-shim:
[echo] Project: shims
[javac] Compiling 2 source files to
/Users/gang/hive-trunk-11-28/build/shims/classes
[javac]
/Users/gang/hive-trunk-11-28/shims/src/0.23/java/org/apache/hadoop/hive/shims/Hadoop23Shims.java:122:
getDefaultBlockSize() in org.apache.hadoop.fs.FileSystem cannot be applied
to (org.apache.hadoop.fs.Path)
[javac] return fs.getDefaultBlockSize(path);
[javac] ^
[javac]
/Users/gang/hive-trunk-11-28/shims/src/0.23/java/org/apache/hadoop/hive/shims/Hadoop23Shims.java:127:
getDefaultReplication() in org.apache.hadoop.fs.FileSystem cannot be applied
to (org.apache.hadoop.fs.Path)
[javac] return fs.getDefaultReplication(path);
[javac] ^
[javac] 2 errors
BUILD FAILED
/Users/gang/hive-trunk-11-28/build.xml:302: The following error occurred
while executing this line:
{quote}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3753) 'CTAS' and INSERT OVERWRITE send different column names to the underlying SerDe
[
https://issues.apache.org/jira/browse/HIVE-3753?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13506162#comment-13506162
]
Mark Grover commented on HIVE-3753:
---
Thanks for reporting this Matthew. This looks like a bug. Do you think a query
like
{code}
CREATE TABLE foo ROW FORMAT SERDE 'JsonSerDe' SELECT host as host from
table1;
{code}
could serve as a work around for this?
'CTAS' and INSERT OVERWRITE send different column names to the underlying
SerDe
---
Key: HIVE-3753
URL: https://issues.apache.org/jira/browse/HIVE-3753
Project: Hive
Issue Type: Bug
Components: Serializers/Deserializers
Affects Versions: 0.9.0
Reporter: Matthew Rathbone
Original Estimate: 24h
Remaining Estimate: 24h
A good example is with a JSON serde
(https://github.com/rathboma/Hive-JSON-Serde-1)
Here is a simple example of how the two results differ:
CREATE TABLE foo ROW FORMAT SERDE 'JsonSerDe' SELECT host from table1;
generates = {_col0: localhost}
CREATE TABLE foo(host string) ROW FORMAT SERDE 'JsonSerDe';
INSERT OVERWRITE TABLE FOO SELECT host FROM table;
generates = {host: localhost}
The SerDe gets passed column names in two places:
1) The property Constants.LIST_COLUMNS
2) It gets passed a StructObjectInspector on serialize
In the CTAS example above, both of these contain '_col0' as the column name.
This is not true in the second example, as the LIST_COLUMNS property contains
the real column names.
I'd be happy to help out with this change, but I fear that the solution lies
somewhere in SemanticAnalyser.java, and I'm having a hard time finding my way
around.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3754) Trunk hadoop 23 build fails
[
https://issues.apache.org/jira/browse/HIVE-3754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13506166#comment-13506166
]
Gang Tim Liu commented on HIVE-3754:
Viraj can you check if it breaks Hadoop 23 build? You can use the following to
check
Sent from my iPhone
On Nov 28, 2012, at 6:25 PM, Viraj Bhat (JIRA) j...@apache.org wrote:
Trunk hadoop 23 build fails
---
Key: HIVE-3754
URL: https://issues.apache.org/jira/browse/HIVE-3754
Project: Hive
Issue Type: Bug
Components: Build Infrastructure
Reporter: Gang Tim Liu
check out the latest code from trunk
{code}
svn info
{code}
{quote}
Path: .
URL: http://svn.apache.org/repos/asf/hive/trunk
Repository Root: http://svn.apache.org/repos/asf
Repository UUID: 13f79535-47bb-0310-9956-ffa450edef68
Revision: 1415005
Node Kind: directory
Schedule: normal
Last Changed Author: namit
Last Changed Rev: 1414608
Last Changed Date: 2012-11-28 01:36:27 -0800 (Wed, 28 Nov 2012)
{quote}
{code}
ant clean package -Dhadoop.version=0.23.1 -Dhadoop-0.23.version=0.23.1
-Dhadoop.mr.rev=23
{code}
{quote}
ivy-retrieve-hadoop-shim:
[echo] Project: shims
[javac] Compiling 2 source files to
/Users/gang/hive-trunk-11-28/build/shims/classes
[javac]
/Users/gang/hive-trunk-11-28/shims/src/0.23/java/org/apache/hadoop/hive/shims/Hadoop23Shims.java:122:
getDefaultBlockSize() in org.apache.hadoop.fs.FileSystem cannot be applied
to (org.apache.hadoop.fs.Path)
[javac] return fs.getDefaultBlockSize(path);
[javac] ^
[javac]
/Users/gang/hive-trunk-11-28/shims/src/0.23/java/org/apache/hadoop/hive/shims/Hadoop23Shims.java:127:
getDefaultReplication() in org.apache.hadoop.fs.FileSystem cannot be applied
to (org.apache.hadoop.fs.Path)
[javac] return fs.getDefaultReplication(path);
[javac] ^
[javac] 2 errors
BUILD FAILED
/Users/gang/hive-trunk-11-28/build.xml:302: The following error occurred
while executing this line:
{quote}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3754) Trunk hadoop 23 build fails
[
https://issues.apache.org/jira/browse/HIVE-3754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13506174#comment-13506174
]
Arup Malakar commented on HIVE-3754:
Hi Gang the API getDefaultBlockSize(Path) and getDefaultReplication(Path) in
FileSystem is not available on 0.23.1 they are in 0.23.3 though or 2.0.0-alpha.
That is why they are building fine with default configuration , but won't
compile when you use Hadoop 0.23.1. Is this a concern?
Trunk hadoop 23 build fails
---
Key: HIVE-3754
URL: https://issues.apache.org/jira/browse/HIVE-3754
Project: Hive
Issue Type: Bug
Components: Build Infrastructure
Reporter: Gang Tim Liu
check out the latest code from trunk
{code}
svn info
{code}
{quote}
Path: .
URL: http://svn.apache.org/repos/asf/hive/trunk
Repository Root: http://svn.apache.org/repos/asf
Repository UUID: 13f79535-47bb-0310-9956-ffa450edef68
Revision: 1415005
Node Kind: directory
Schedule: normal
Last Changed Author: namit
Last Changed Rev: 1414608
Last Changed Date: 2012-11-28 01:36:27 -0800 (Wed, 28 Nov 2012)
{quote}
{code}
ant clean package -Dhadoop.version=0.23.1 -Dhadoop-0.23.version=0.23.1
-Dhadoop.mr.rev=23
{code}
{quote}
ivy-retrieve-hadoop-shim:
[echo] Project: shims
[javac] Compiling 2 source files to
/Users/gang/hive-trunk-11-28/build/shims/classes
[javac]
/Users/gang/hive-trunk-11-28/shims/src/0.23/java/org/apache/hadoop/hive/shims/Hadoop23Shims.java:122:
getDefaultBlockSize() in org.apache.hadoop.fs.FileSystem cannot be applied
to (org.apache.hadoop.fs.Path)
[javac] return fs.getDefaultBlockSize(path);
[javac] ^
[javac]
/Users/gang/hive-trunk-11-28/shims/src/0.23/java/org/apache/hadoop/hive/shims/Hadoop23Shims.java:127:
getDefaultReplication() in org.apache.hadoop.fs.FileSystem cannot be applied
to (org.apache.hadoop.fs.Path)
[javac] return fs.getDefaultReplication(path);
[javac] ^
[javac] 2 errors
BUILD FAILED
/Users/gang/hive-trunk-11-28/build.xml:302: The following error occurred
while executing this line:
{quote}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-3709) Stop storing default ConfVars in temp file
[ https://issues.apache.org/jira/browse/HIVE-3709?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Carl Steinbach updated HIVE-3709: - Resolution: Fixed Fix Version/s: 0.11 Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Committed to trunk. Thanks Kevin! Stop storing default ConfVars in temp file -- Key: HIVE-3709 URL: https://issues.apache.org/jira/browse/HIVE-3709 Project: Hive Issue Type: Improvement Components: Configuration Affects Versions: 0.10.0 Reporter: Kevin Wilfong Assignee: Kevin Wilfong Fix For: 0.11 Attachments: HIVE-3709.1.patch.txt, HIVE-3709.2.patch.txt, HIVE-3709.3.patch.txt To work around issues with Hadoop's Configuration object, specifically it's addResource(InputStream), default configurations are written to a temp file (I think HIVE-2362 introduced this). This, however, introduces the problem that once that file is deleted from /tmp the client crashes. This is particularly problematic for long running services like the metastore server. Writing a custom InputStream to deal with the problems in the Configuration object should provide a work around, which does not introduce a time bomb into Hive. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3709) Stop storing default ConfVars in temp file
[ https://issues.apache.org/jira/browse/HIVE-3709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13506216#comment-13506216 ] Carl Steinbach commented on HIVE-3709: -- @Kevin: I tried running TestHiveServerSessions again and wasn't able to provoke the failure I saw before. Stop storing default ConfVars in temp file -- Key: HIVE-3709 URL: https://issues.apache.org/jira/browse/HIVE-3709 Project: Hive Issue Type: Improvement Components: Configuration Affects Versions: 0.10.0 Reporter: Kevin Wilfong Assignee: Kevin Wilfong Fix For: 0.11 Attachments: HIVE-3709.1.patch.txt, HIVE-3709.2.patch.txt, HIVE-3709.3.patch.txt To work around issues with Hadoop's Configuration object, specifically it's addResource(InputStream), default configurations are written to a temp file (I think HIVE-2362 introduced this). This, however, introduces the problem that once that file is deleted from /tmp the client crashes. This is particularly problematic for long running services like the metastore server. Writing a custom InputStream to deal with the problems in the Configuration object should provide a work around, which does not introduce a time bomb into Hive. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HIVE-3596) Regression - HiveConf static variable causes issues in long running JVM instances with /tmp/ data
[ https://issues.apache.org/jira/browse/HIVE-3596?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Carl Steinbach resolved HIVE-3596. -- Resolution: Duplicate Resolving this is a duplicate of HIVE-3709. Regression - HiveConf static variable causes issues in long running JVM instances with /tmp/ data - Key: HIVE-3596 URL: https://issues.apache.org/jira/browse/HIVE-3596 Project: Hive Issue Type: Bug Components: Configuration Affects Versions: 0.8.0, 0.8.1, 0.9.0 Reporter: Chris McConnell Assignee: Chris McConnell Fix For: 0.10.0, 0.9.0, 0.8.1 Attachments: HIVE-3596.patch With Hive 0.8.x, HiveConf was changed to utilize the private, static member confVarURL which points to /tmp/hive-user-tmp_number.xml for job configuration settings. During long running JVMs, such as a Beeswax server, which creates multiple HiveConf objects over time this variable does not properly get updated between jobs and can cause job failure if the OS cleans /tmp/ during a cron job. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
