[jira] [Commented] (HIVE-2781) HBaseSerDe should allow users to specify the timestamp passed to Puts
[ https://issues.apache.org/jira/browse/HIVE-2781?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13213431#comment-13213431 ] Phabricator commented on HIVE-2781: --- navis has commented on the revision "HIVE-2781 [jira] HBaseSerDe should allow users to specify the timestamp passed to Puts". Added special column :timestamp to interchange timestamp with hbase value. This is only a suggestion and should do more works including test and javadocs. REVISION DETAIL https://reviews.facebook.net/D1863 > HBaseSerDe should allow users to specify the timestamp passed to Puts > -- > > Key: HIVE-2781 > URL: https://issues.apache.org/jira/browse/HIVE-2781 > Project: Hive > Issue Type: Improvement >Reporter: Francis Liu >Assignee: Navis > Attachments: HIVE-2781.D1863.1.patch > > > Users may want to specify the timestamp used for Put requests to hbase. Thus > enabling users to have the same timestamp for a single batch of writes. Which > would be useful for a number of things. HCatalog's HBase storageHandler > implementation makes use of this feature to provide users with snapshot > isolation and write transactions. My proposal is to add the timestamp option > as a final static member: > public static final long HBASE_PUT_TIMESTAMP = "hbase.put_timestamp" > And passing this value to all the Puts created by serialize() -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HIVE-2781) HBaseSerDe should allow users to specify the timestamp passed to Puts
[ https://issues.apache.org/jira/browse/HIVE-2781?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Navis reassigned HIVE-2781: --- Assignee: Navis > HBaseSerDe should allow users to specify the timestamp passed to Puts > -- > > Key: HIVE-2781 > URL: https://issues.apache.org/jira/browse/HIVE-2781 > Project: Hive > Issue Type: Improvement >Reporter: Francis Liu >Assignee: Navis > Attachments: HIVE-2781.D1863.1.patch > > > Users may want to specify the timestamp used for Put requests to hbase. Thus > enabling users to have the same timestamp for a single batch of writes. Which > would be useful for a number of things. HCatalog's HBase storageHandler > implementation makes use of this feature to provide users with snapshot > isolation and write transactions. My proposal is to add the timestamp option > as a final static member: > public static final long HBASE_PUT_TIMESTAMP = "hbase.put_timestamp" > And passing this value to all the Puts created by serialize() -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2781) HBaseSerDe should allow users to specify the timestamp passed to Puts
[ https://issues.apache.org/jira/browse/HIVE-2781?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Phabricator updated HIVE-2781: -- Attachment: HIVE-2781.D1863.1.patch navis requested code review of "HIVE-2781 [jira] HBaseSerDe should allow users to specify the timestamp passed to Puts". Reviewers: JIRA DPAL-861 HBaseSerDe should allow users to specify the timestamp passed to Puts Users may want to specify the timestamp used for Put requests to hbase. Thus enabling users to have the same timestamp for a single batch of writes. Which would be useful for a number of things. HCatalog's HBase storageHandler implementation makes use of this feature to provide users with snapshot isolation and write transactions. My proposal is to add the timestamp option as a final static member: public static final long HBASE_PUT_TIMESTAMP = "hbase.put_timestamp" And passing this value to all the Puts created by serialize() TEST PLAN EMPTY REVISION DETAIL https://reviews.facebook.net/D1863 AFFECTED FILES hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSerDe.java hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseStorageHandler.java hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableInputFormat.java hbase-handler/src/java/org/apache/hadoop/hive/hbase/LazyHBaseRow.java hbase-handler/src/test/org/apache/hadoop/hive/hbase/TestLazyHBaseObject.java hbase-handler/src/test/queries/hbase_timestamp.q hbase-handler/src/test/results/hbase_timestamp.q.out MANAGE HERALD DIFFERENTIAL RULES https://reviews.facebook.net/herald/view/differential/ WHY DID I GET THIS EMAIL? https://reviews.facebook.net/herald/transcript/3957/ Tip: use the X-Herald-Rules header to filter Herald messages in your client. > HBaseSerDe should allow users to specify the timestamp passed to Puts > -- > > Key: HIVE-2781 > URL: https://issues.apache.org/jira/browse/HIVE-2781 > Project: Hive > Issue Type: Improvement >Reporter: Francis Liu > Attachments: HIVE-2781.D1863.1.patch > > > Users may want to specify the timestamp used for Put requests to hbase. Thus > enabling users to have the same timestamp for a single batch of writes. Which > would be useful for a number of things. HCatalog's HBase storageHandler > implementation makes use of this feature to provide users with snapshot > isolation and write transactions. My proposal is to add the timestamp option > as a final static member: > public static final long HBASE_PUT_TIMESTAMP = "hbase.put_timestamp" > And passing this value to all the Puts created by serialize() -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2810) Implement NULL-safe equality operator <=>
[ https://issues.apache.org/jira/browse/HIVE-2810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13213412#comment-13213412 ] Phabricator commented on HIVE-2810: --- cwsteinbach has commented on the revision "HIVE-2810 [jira] Implement NULL-safe equality operator <=>". Yup, exactly. I tried running the query from HIVE-741: https://issues.apache.org/jira/browse/HIVE-741?focusedCommentId=12896789&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-12896789 This works: select * from input1 a join input1 b on a.key=b.value; But I get the following error when I try to run this: select * from input1 a join input1 b on a.key <=> b.value; FAILED: Error in semantic analysis: Line 1:40 Both left and right aliases encountered in JOIN 'value' Looks like this may be a parsing issue? HIVE-741 added a join_nulls.q test to clientpositive. I think it would be a good idea to augment this testcase with null-safe equality queries. REVISION DETAIL https://reviews.facebook.net/D1791 BRANCH DPAL-843 > Implement NULL-safe equality operator <=> > - > > Key: HIVE-2810 > URL: https://issues.apache.org/jira/browse/HIVE-2810 > Project: Hive > Issue Type: New Feature > Components: Query Processor, UDF >Affects Versions: 0.9.0 >Reporter: Carl Steinbach >Assignee: Navis > Fix For: 0.9.0 > > Attachments: HIVE-2810.D1791.1.patch > > > Ref: > http://dev.mysql.com/doc/refman/5.0/en/comparison-operators.html#operator_equal-to -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2810) Implement NULL-safe equality operator <=>
[ https://issues.apache.org/jira/browse/HIVE-2810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13213306#comment-13213306 ] Phabricator commented on HIVE-2810: --- navis has commented on the revision "HIVE-2810 [jira] Implement NULL-safe equality operator <=>". You mean something like 'SELECT * FROM a JOIN b ON a.key <=> b.key' ? That seemed to be a huge work. (for me) REVISION DETAIL https://reviews.facebook.net/D1791 BRANCH DPAL-843 > Implement NULL-safe equality operator <=> > - > > Key: HIVE-2810 > URL: https://issues.apache.org/jira/browse/HIVE-2810 > Project: Hive > Issue Type: New Feature > Components: Query Processor, UDF >Affects Versions: 0.9.0 >Reporter: Carl Steinbach >Assignee: Navis > Fix For: 0.9.0 > > Attachments: HIVE-2810.D1791.1.patch > > > Ref: > http://dev.mysql.com/doc/refman/5.0/en/comparison-operators.html#operator_equal-to -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2810) Implement NULL-safe equality operator <=>
[ https://issues.apache.org/jira/browse/HIVE-2810?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Carl Steinbach updated HIVE-2810: - Status: Open (was: Patch Available) > Implement NULL-safe equality operator <=> > - > > Key: HIVE-2810 > URL: https://issues.apache.org/jira/browse/HIVE-2810 > Project: Hive > Issue Type: New Feature > Components: Query Processor, UDF >Affects Versions: 0.9.0 >Reporter: Carl Steinbach >Assignee: Navis > Fix For: 0.9.0 > > Attachments: HIVE-2810.D1791.1.patch > > > Ref: > http://dev.mysql.com/doc/refman/5.0/en/comparison-operators.html#operator_equal-to -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2716) Move retry logic in HiveMetaStore to a separe class
[ https://issues.apache.org/jira/browse/HIVE-2716?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Phabricator updated HIVE-2716: -- Attachment: HIVE-2716.D1227.4.patch enis updated the revision "HIVE-2716 [jira] Move retry logic in HiveMetaStore to a separe class". Reviewers: JIRA, cwsteinbach, ashutoshc The test was failing due to the change in handling checked exceptions. Updated the patch to chang the auth-related methods to throw MetaExceptions, and convert to RuntimeExceptions to keep the current behavior unchanged. Running the tests now, will update with the results. REVISION DETAIL https://reviews.facebook.net/D1227 AFFECTED FILES metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java metastore/src/java/org/apache/hadoop/hive/metastore/RetryingRawStore.java metastore/src/java/org/apache/hadoop/hive/metastore/events/EventCleanerTask.java > Move retry logic in HiveMetaStore to a separe class > --- > > Key: HIVE-2716 > URL: https://issues.apache.org/jira/browse/HIVE-2716 > Project: Hive > Issue Type: Sub-task > Components: Metastore >Affects Versions: 0.9.0 >Reporter: Enis Soztutar >Assignee: Enis Soztutar > Attachments: HIVE-2716.D1227.1.patch, HIVE-2716.D1227.2.patch, > HIVE-2716.D1227.3.patch, HIVE-2716.D1227.4.patch > > > In HIVE-1219, method retrying for raw store operation are introduced to > handle jdo operations more robustly. However, the abstraction for the > RawStore operations can be moved to a separate class implementing RawStore, > which should clean up the code base for HiveMetaStore. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2810) Implement NULL-safe equality operator <=>
[ https://issues.apache.org/jira/browse/HIVE-2810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13213224#comment-13213224 ] Phabricator commented on HIVE-2810: --- cwsteinbach has requested changes to the revision "HIVE-2810 [jira] Implement NULL-safe equality operator <=>". Looks good. However, I have a hunch that this doesn't work with JOINs (e.g. see HIVE-741). Can you please add a testcase that uses the NULL-safe comparison operator in a JOIN condition? Thanks. REVISION DETAIL https://reviews.facebook.net/D1791 BRANCH DPAL-843 > Implement NULL-safe equality operator <=> > - > > Key: HIVE-2810 > URL: https://issues.apache.org/jira/browse/HIVE-2810 > Project: Hive > Issue Type: New Feature > Components: Query Processor, UDF >Affects Versions: 0.9.0 >Reporter: Carl Steinbach >Assignee: Navis > Fix For: 0.9.0 > > Attachments: HIVE-2810.D1791.1.patch > > > Ref: > http://dev.mysql.com/doc/refman/5.0/en/comparison-operators.html#operator_equal-to -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-1444) "hdfs" is hardcoded in few places in the code which inhibits use of other file systems
[
https://issues.apache.org/jira/browse/HIVE-1444?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Phabricator updated HIVE-1444:
--
Attachment: HIVE-1444.D1839.1.patch
edwardcapriolo requested code review of "HIVE-1444 [jira] "hdfs" is hardcoded
in few places in the code which inhibits use of other file systems".
Reviewers: JIRA
https://issues.apache.org/jira/browse/HIVE-1444
Allows other fs implementations then HDFS
In quite a few places "hdfs" is hardcoded, which is OK for majority of the
cases, except when it is not really hdfs, but s3 or any other file system.
The place where it really breaks is:
in ql/src/java/org/apache/hadoop/hive/ql/parse/LoadSemanticAnalyzer.java :
method: private void applyConstraints(URI fromURI, URI toURI, Tree ast,
boolean isLocal)
First few lines are check for file system:
if (!fromURI.getScheme().equals("file")
&& !fromURI.getScheme().equals("hdfs")) {
throw new SemanticException(ErrorMsg.INVALID_PATH.getMsg(ast,
"only \"file\" or \"hdfs\" file systems accepted"));
}
"hdfs" is hardcoded.
I don't think you need to have this check at all as you are checking whether
filesystem is local or not later on anyway and in regards to non locla file
system - if one would be bad one you would get problems or have it look like
local before you even come to "applyConstraints" method.
TEST PLAN
EMPTY
REVISION DETAIL
https://reviews.facebook.net/D1839
AFFECTED FILES
ql/src/java/org/apache/hadoop/hive/ql/parse/LoadSemanticAnalyzer.java
MANAGE HERALD DIFFERENTIAL RULES
https://reviews.facebook.net/herald/view/differential/
WHY DID I GET THIS EMAIL?
https://reviews.facebook.net/herald/transcript/3909/
Tip: use the X-Herald-Rules header to filter Herald messages in your client.
> "hdfs" is hardcoded in few places in the code which inhibits use of other
> file systems
> --
>
> Key: HIVE-1444
> URL: https://issues.apache.org/jira/browse/HIVE-1444
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
>Affects Versions: 0.3.0, 0.4.0, 0.4.1, 0.5.0, 0.6.0, 0.7.0
> Environment: any
>Reporter: Yuliya Feldman
>Assignee: Edward Capriolo
>Priority: Minor
> Fix For: 0.9.0
>
> Attachments: HIVE-1444.D1839.1.patch, hive-1444.patch.txt
>
>
> In quite a few places "hdfs" is hardcoded, which is OK for majority of the
> cases, except when it is not really hdfs, but s3 or any other file system.
> The place where it really breaks is:
> in ql/src/java/org/apache/hadoop/hive/ql/parse/LoadSemanticAnalyzer.java :
> method: private void applyConstraints(URI fromURI, URI toURI, Tree ast,
> boolean isLocal)
> First few lines are check for file system:
> if (!fromURI.getScheme().equals("file")
> && !fromURI.getScheme().equals("hdfs")) {
> throw new SemanticException(ErrorMsg.INVALID_PATH.getMsg(ast,
> "only \"file\" or \"hdfs\" file systems accepted"));
> }
> "hdfs" is hardcoded.
> I don't think you need to have this check at all as you are checking whether
> filesystem is local or not later on anyway and in regards to non locla file
> system - if one would be bad one you would get problems or have it look like
> local before you even come to "applyConstraints" method.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2810) Implement NULL-safe equality operator <=>
[ https://issues.apache.org/jira/browse/HIVE-2810?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Navis updated HIVE-2810: Fix Version/s: 0.9.0 Affects Version/s: 0.9.0 Status: Patch Available (was: Open) > Implement NULL-safe equality operator <=> > - > > Key: HIVE-2810 > URL: https://issues.apache.org/jira/browse/HIVE-2810 > Project: Hive > Issue Type: New Feature > Components: Query Processor, UDF >Affects Versions: 0.9.0 >Reporter: Carl Steinbach >Assignee: Navis > Fix For: 0.9.0 > > Attachments: HIVE-2810.D1791.1.patch > > > Ref: > http://dev.mysql.com/doc/refman/5.0/en/comparison-operators.html#operator_equal-to -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HIVE-2810) Implement NULL-safe equality operator <=>
[ https://issues.apache.org/jira/browse/HIVE-2810?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Navis reassigned HIVE-2810: --- Assignee: Navis > Implement NULL-safe equality operator <=> > - > > Key: HIVE-2810 > URL: https://issues.apache.org/jira/browse/HIVE-2810 > Project: Hive > Issue Type: New Feature > Components: Query Processor, UDF >Affects Versions: 0.9.0 >Reporter: Carl Steinbach >Assignee: Navis > Fix For: 0.9.0 > > Attachments: HIVE-2810.D1791.1.patch > > > Ref: > http://dev.mysql.com/doc/refman/5.0/en/comparison-operators.html#operator_equal-to -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2813) Throw a Hive error if we're in strict mode and a the types of a partition comparison do not match.
[ https://issues.apache.org/jira/browse/HIVE-2813?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Phabricator updated HIVE-2813: -- Attachment: HIVE-2813.D1803.4.patch dmitrys updated the revision "HIVE-2813 [jira] Throw a Hive error if we're in strict mode and a the types of a partition comparison do not match.". Reviewers: kevinwilfong, JIRA, njain, cwsteinbach Yep, thanks; fixed And yes, perhaps it's another bug with reducing the 'date' to int; not sure since don't know the full specific (I thought '17' is error code of argument, not the reduced value). REVISION DETAIL https://reviews.facebook.net/D1803 AFFECTED FILES common/src/java/org/apache/hadoop/hive/conf/HiveConf.java conf/hive-default.xml.template ql/src/java/org/apache/hadoop/hive/ql/parse/ErrorMsg.java ql/src/java/org/apache/hadoop/hive/ql/plan/ExprNodeGenericFuncDesc.java ql/src/test/queries/clientnegative/select_partcol_diff_types.q ql/src/test/queries/clientpositive/select_partcol_diff_types.q ql/src/test/results/clientnegative/select_partcol_diff_types.q.out ql/src/test/results/clientpositive/select_partcol_diff_types.q.out > Throw a Hive error if we're in strict mode and a the types of a partition > comparison do not match. > -- > > Key: HIVE-2813 > URL: https://issues.apache.org/jira/browse/HIVE-2813 > Project: Hive > Issue Type: Improvement > Components: Query Processor >Affects Versions: 0.8.1, 0.9.0 >Reporter: Dmitry Soshnikov >Priority: Minor > Labels: newbie, patch > Fix For: 0.9.0 > > Attachments: HIVE-2813.D1803.2.patch, HIVE-2813.D1803.3.patch, > HIVE-2813.D1803.4.patch > > Original Estimate: 24h > Remaining Estimate: 24h > > Oftentimes people try to write queries like > SELECT * > FROM table > WHERE ds = 2011-08-03 > This won't work because quotes are missing, and it'll actually try to filter > on ds = "2000". > People run into this pretty regularly; a simple check on whether the types > match exactly in partition predicates would make this a lot less likely to > happen. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2813) Throw a Hive error if we're in strict mode and a the types of a partition comparison do not match.
[ https://issues.apache.org/jira/browse/HIVE-2813?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13213011#comment-13213011 ] Phabricator commented on HIVE-2813: --- cwsteinbach has commented on the revision "HIVE-2813 [jira] Throw a Hive error if we're in strict mode and a the types of a partition comparison do not match.". INLINE COMMENTS ql/src/test/results/clientnegative/select_partcol_diff_types.q.out:1 Looks like there's actually another bug here. Shouldn't the argument be 2012-2-17 = 1993? REVISION DETAIL https://reviews.facebook.net/D1803 BRANCH trunk > Throw a Hive error if we're in strict mode and a the types of a partition > comparison do not match. > -- > > Key: HIVE-2813 > URL: https://issues.apache.org/jira/browse/HIVE-2813 > Project: Hive > Issue Type: Improvement > Components: Query Processor >Affects Versions: 0.8.1, 0.9.0 >Reporter: Dmitry Soshnikov >Priority: Minor > Labels: newbie, patch > Fix For: 0.9.0 > > Attachments: HIVE-2813.D1803.2.patch, HIVE-2813.D1803.3.patch > > Original Estimate: 24h > Remaining Estimate: 24h > > Oftentimes people try to write queries like > SELECT * > FROM table > WHERE ds = 2011-08-03 > This won't work because quotes are missing, and it'll actually try to filter > on ds = "2000". > People run into this pretty regularly; a simple check on whether the types > match exactly in partition predicates would make this a lot less likely to > happen. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2813) Throw a Hive error if we're in strict mode and a the types of a partition comparison do not match.
[ https://issues.apache.org/jira/browse/HIVE-2813?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13213010#comment-13213010 ] Phabricator commented on HIVE-2813: --- cwsteinbach has requested changes to the revision "HIVE-2813 [jira] Throw a Hive error if we're in strict mode and a the types of a partition comparison do not match.". INLINE COMMENTS common/src/java/org/apache/hadoop/hive/conf/HiveConf.java:342 Please add this to conf/hive-default.xml.template along with a description of what it does, and the allowed values. ql/src/java/org/apache/hadoop/hive/ql/parse/ErrorMsg.java:59 spelling: "nonstric" ql/src/java/org/apache/hadoop/hive/ql/plan/ExprNodeGenericFuncDesc.java:215 spelling ql/src/test/queries/clientnegative/select_partcol_diff_types.q:1 Please add a positive testcase that explicitly sets hive.partition.comparison.mode=nonstrict so that we can test the WARNING message. REVISION DETAIL https://reviews.facebook.net/D1803 BRANCH trunk > Throw a Hive error if we're in strict mode and a the types of a partition > comparison do not match. > -- > > Key: HIVE-2813 > URL: https://issues.apache.org/jira/browse/HIVE-2813 > Project: Hive > Issue Type: Improvement > Components: Query Processor >Affects Versions: 0.8.1, 0.9.0 >Reporter: Dmitry Soshnikov >Priority: Minor > Labels: newbie, patch > Fix For: 0.9.0 > > Attachments: HIVE-2813.D1803.2.patch, HIVE-2813.D1803.3.patch > > Original Estimate: 24h > Remaining Estimate: 24h > > Oftentimes people try to write queries like > SELECT * > FROM table > WHERE ds = 2011-08-03 > This won't work because quotes are missing, and it'll actually try to filter > on ds = "2000". > People run into this pretty regularly; a simple check on whether the types > match exactly in partition predicates would make this a lot less likely to > happen. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2792) SUBSTR(CAST( AS BINARY)) produces unexpected results
[ https://issues.apache.org/jira/browse/HIVE-2792?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13212985#comment-13212985 ] Hudson commented on HIVE-2792: -- Integrated in Hive-trunk-h0.21 #1268 (See [https://builds.apache.org/job/Hive-trunk-h0.21/1268/]) HIVE-2792: SUBSTR(CAST( AS BINARY)) produces unexpected results (navis via hashutosh) (Revision 1291633) Result = SUCCESS hashutosh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1291633 Files : * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFSubstr.java * /hive/trunk/ql/src/test/queries/clientpositive/ba_table_udfs.q * /hive/trunk/ql/src/test/queries/clientpositive/udf_substr.q * /hive/trunk/ql/src/test/results/clientpositive/ba_table_udfs.q.out * /hive/trunk/ql/src/test/results/clientpositive/udf_substr.q.out > SUBSTR(CAST( AS BINARY)) produces unexpected results > > > Key: HIVE-2792 > URL: https://issues.apache.org/jira/browse/HIVE-2792 > Project: Hive > Issue Type: Bug > Components: UDF >Affects Versions: 0.8.0, 0.8.1 >Reporter: Carl Steinbach >Assignee: Navis > Fix For: 0.9.0 > > Attachments: HIVE-2792.D1797.1.patch, HIVE-2792.D1797.2.patch, > HIVE-2792.D1797.2.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2344) filter is removed due to regression of HIVE-1538
[ https://issues.apache.org/jira/browse/HIVE-2344?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13212986#comment-13212986 ] Hudson commented on HIVE-2344: -- Integrated in Hive-trunk-h0.21 #1268 (See [https://builds.apache.org/job/Hive-trunk-h0.21/1268/]) HIVE-2791: filter is still removed due to regression of HIVE-1538 althougth HIVE-2344 (binlijin via hashutosh) (Revision 1291916) Result = SUCCESS hashutosh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1291916 Files : * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/ppd/OpProcFactory.java * /hive/trunk/ql/src/test/queries/clientpositive/ppd2.q * /hive/trunk/ql/src/test/results/clientpositive/ppd2.q.out > filter is removed due to regression of HIVE-1538 > > > Key: HIVE-2344 > URL: https://issues.apache.org/jira/browse/HIVE-2344 > Project: Hive > Issue Type: Bug >Affects Versions: 0.8.0 >Reporter: He Yongqiang >Assignee: Amareshwari Sriramadasu > Fix For: 0.8.0 > > Attachments: hive-patch-2344-2.txt, hive-patch-2344.txt, > ppd_udf_col.q.out.txt > > > select * from > ( > select type_bucket,randum123 > from (SELECT *, cast(rand() as double) AS randum123 FROM tbl where ds = ...) > a > where randum123 <=0.1)s where s.randum123>0.1 limit 20; > This is returning results... > and > explain > select type_bucket,randum123 > from (SELECT *, cast(rand() as double) AS randum123 FROM tbl where ds = ...) > a > where randum123 <=0.1 > shows that there is no filter. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2791) filter is still removed due to regression of HIVE-1538 althougth HIVE-2344
[ https://issues.apache.org/jira/browse/HIVE-2791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13212984#comment-13212984 ] Hudson commented on HIVE-2791: -- Integrated in Hive-trunk-h0.21 #1268 (See [https://builds.apache.org/job/Hive-trunk-h0.21/1268/]) HIVE-2791: filter is still removed due to regression of HIVE-1538 althougth HIVE-2344 (binlijin via hashutosh) (Revision 1291916) Result = SUCCESS hashutosh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1291916 Files : * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/ppd/OpProcFactory.java * /hive/trunk/ql/src/test/queries/clientpositive/ppd2.q * /hive/trunk/ql/src/test/results/clientpositive/ppd2.q.out > filter is still removed due to regression of HIVE-1538 althougth HIVE-2344 > -- > > Key: HIVE-2791 > URL: https://issues.apache.org/jira/browse/HIVE-2791 > Project: Hive > Issue Type: Bug >Reporter: binlijin >Assignee: binlijin > Fix For: 0.9.0 > > Attachments: HIVE-2791.2.patch, HIVE-2791.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-1538) FilterOperator is applied twice with ppd on.
[ https://issues.apache.org/jira/browse/HIVE-1538?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13212987#comment-13212987 ] Hudson commented on HIVE-1538: -- Integrated in Hive-trunk-h0.21 #1268 (See [https://builds.apache.org/job/Hive-trunk-h0.21/1268/]) HIVE-2791: filter is still removed due to regression of HIVE-1538 althougth HIVE-2344 (binlijin via hashutosh) (Revision 1291916) Result = SUCCESS hashutosh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1291916 Files : * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/ppd/OpProcFactory.java * /hive/trunk/ql/src/test/queries/clientpositive/ppd2.q * /hive/trunk/ql/src/test/results/clientpositive/ppd2.q.out > FilterOperator is applied twice with ppd on. > > > Key: HIVE-1538 > URL: https://issues.apache.org/jira/browse/HIVE-1538 > Project: Hive > Issue Type: Bug > Components: Query Processor >Reporter: Amareshwari Sriramadasu >Assignee: Amareshwari Sriramadasu > Fix For: 0.8.0 > > Attachments: patch-1538-1.txt, patch-1538-2.txt, patch-1538-3.txt, > patch-1538-4.txt, patch-1538.txt > > > With hive.optimize.ppd set to true, FilterOperator is applied twice. And it > seems second operator is always filtering zero rows. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2812) Hive multi group by single reducer optimization fails when aggregation with no keys followed by query with no aggregations
[ https://issues.apache.org/jira/browse/HIVE-2812?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kevin Wilfong updated HIVE-2812: Status: Patch Available (was: Open) > Hive multi group by single reducer optimization fails when aggregation with > no keys followed by query with no aggregations > -- > > Key: HIVE-2812 > URL: https://issues.apache.org/jira/browse/HIVE-2812 > Project: Hive > Issue Type: Bug >Affects Versions: 0.9.0 >Reporter: Kevin Wilfong >Assignee: Kevin Wilfong > Attachments: HIVE-2812.D1821.1.patch > > > In multi insert queries where one subquery involves an aggregation with no > distinct or group by keys and is followed by a query without any > aggregations, like the following, Hive will attempt to add a group by > operator for the query without aggregations, causing semantic analysis to > fail. > FROM src > INSERT OVERWRITE TABLE table1 SELECT count(*) > INSERT OVERWRITE TABLE table2 SELECT key; -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2812) Hive multi group by single reducer optimization fails when aggregation with no keys followed by query with no aggregations
[ https://issues.apache.org/jira/browse/HIVE-2812?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Phabricator updated HIVE-2812: -- Attachment: HIVE-2812.D1821.1.patch kevinwilfong requested code review of "HIVE-2812 [jira] Hive multi group by single reducer optimization fails when aggregation with no keys followed by query with no aggregations". Reviewers: JIRA https://issues.apache.org/jira/browse/HIVE-2812 In multi insert queries, subqueries with aggregations but no group by keys were being grouped with subqueries without any aggregations. This meant if a subquery without aggregations came first, the subqueries without group by keys were not benefitting from the optimization. More imporantly, if a subquery without group by keys came first, the Semantic Analyzer tried to add group by operators for queries without group by clauses resulting in an error during semantic analysis. This patch fixes this by ensuring the two types of subqueries are grouped separately. In multi insert queries where one subquery involves an aggregation with no distinct or group by keys and is followed by a query without any aggregations, like the following, Hive will attempt to add a group by operator for the query without aggregations, causing semantic analysis to fail. FROM src INSERT OVERWRITE TABLE table1 SELECT count INSERT OVERWRITE TABLE table2 SELECT key; TEST PLAN EMPTY REVISION DETAIL https://reviews.facebook.net/D1821 AFFECTED FILES ql/src/test/results/clientpositive/groupby_multi_single_reducer3.q.out ql/src/test/queries/clientpositive/groupby_multi_single_reducer3.q ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java MANAGE HERALD DIFFERENTIAL RULES https://reviews.facebook.net/herald/view/differential/ WHY DID I GET THIS EMAIL? https://reviews.facebook.net/herald/transcript/3879/ Tip: use the X-Herald-Rules header to filter Herald messages in your client. > Hive multi group by single reducer optimization fails when aggregation with > no keys followed by query with no aggregations > -- > > Key: HIVE-2812 > URL: https://issues.apache.org/jira/browse/HIVE-2812 > Project: Hive > Issue Type: Bug >Affects Versions: 0.9.0 >Reporter: Kevin Wilfong >Assignee: Kevin Wilfong > Attachments: HIVE-2812.D1821.1.patch > > > In multi insert queries where one subquery involves an aggregation with no > distinct or group by keys and is followed by a query without any > aggregations, like the following, Hive will attempt to add a group by > operator for the query without aggregations, causing semantic analysis to > fail. > FROM src > INSERT OVERWRITE TABLE table1 SELECT count(*) > INSERT OVERWRITE TABLE table2 SELECT key; -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2813) Throw a Hive error if we're in strict mode and a the types of a partition comparison do not match.
[ https://issues.apache.org/jira/browse/HIVE-2813?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Phabricator updated HIVE-2813: -- Attachment: HIVE-2813.D1803.3.patch dmitrys updated the revision "HIVE-2813 [jira] Throw a Hive error if we're in strict mode and a the types of a partition comparison do not match.". Reviewers: kevinwilfong, JIRA, njain Yes, good points; fixed REVISION DETAIL https://reviews.facebook.net/D1803 AFFECTED FILES common/src/java/org/apache/hadoop/hive/conf/HiveConf.java ql/src/java/org/apache/hadoop/hive/ql/parse/ErrorMsg.java ql/src/java/org/apache/hadoop/hive/ql/plan/ExprNodeGenericFuncDesc.java ql/src/test/queries/clientnegative/select_partcol_diff_types.q ql/src/test/results/clientnegative/select_partcol_diff_types.q.out > Throw a Hive error if we're in strict mode and a the types of a partition > comparison do not match. > -- > > Key: HIVE-2813 > URL: https://issues.apache.org/jira/browse/HIVE-2813 > Project: Hive > Issue Type: Improvement > Components: Query Processor >Affects Versions: 0.8.1, 0.9.0 >Reporter: Dmitry Soshnikov >Priority: Minor > Labels: newbie, patch > Fix For: 0.9.0 > > Attachments: HIVE-2813.D1803.2.patch, HIVE-2813.D1803.3.patch > > Original Estimate: 24h > Remaining Estimate: 24h > > Oftentimes people try to write queries like > SELECT * > FROM table > WHERE ds = 2011-08-03 > This won't work because quotes are missing, and it'll actually try to filter > on ds = "2000". > People run into this pretty regularly; a simple check on whether the types > match exactly in partition predicates would make this a lot less likely to > happen. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2813) Throw a Hive error if we're in strict mode and a the types of a partition comparison do not match.
[ https://issues.apache.org/jira/browse/HIVE-2813?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13212847#comment-13212847 ] Phabricator commented on HIVE-2813: --- kevinwilfong has commented on the revision "HIVE-2813 [jira] Throw a Hive error if we're in strict mode and a the types of a partition comparison do not match.". INLINE COMMENTS ql/src/java/org/apache/hadoop/hive/ql/plan/ExprNodeGenericFuncDesc.java:235-239 It would be good to mention here as well that this only applies to partition columns. REVISION DETAIL https://reviews.facebook.net/D1803 > Throw a Hive error if we're in strict mode and a the types of a partition > comparison do not match. > -- > > Key: HIVE-2813 > URL: https://issues.apache.org/jira/browse/HIVE-2813 > Project: Hive > Issue Type: Improvement > Components: Query Processor >Affects Versions: 0.8.1, 0.9.0 >Reporter: Dmitry Soshnikov >Priority: Minor > Labels: newbie, patch > Fix For: 0.9.0 > > Attachments: HIVE-2813.D1803.2.patch > > Original Estimate: 24h > Remaining Estimate: 24h > > Oftentimes people try to write queries like > SELECT * > FROM table > WHERE ds = 2011-08-03 > This won't work because quotes are missing, and it'll actually try to filter > on ds = "2000". > People run into this pretty regularly; a simple check on whether the types > match exactly in partition predicates would make this a lot less likely to > happen. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2813) Throw a Hive error if we're in strict mode and a the types of a partition comparison do not match.
[ https://issues.apache.org/jira/browse/HIVE-2813?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13212780#comment-13212780 ] Phabricator commented on HIVE-2813: --- kevinwilfong has commented on the revision "HIVE-2813 [jira] Throw a Hive error if we're in strict mode and a the types of a partition comparison do not match.". INLINE COMMENTS ql/src/java/org/apache/hadoop/hive/ql/parse/ErrorMsg.java:58 This should mention the variable that needs to be set to strict/nonstrict to turn this error on/off, and should mention it only applies to comparisons where one side is a partition column. ql/src/java/org/apache/hadoop/hive/ql/plan/ExprNodeGenericFuncDesc.java:223-230 If the child is a column you should make sure that child is a partition column. ql/src/java/org/apache/hadoop/hive/ql/plan/ExprNodeGenericFuncDesc.java:234 It might be better to add a new variable to the HiveConf.ConfVars, something like PARTITION_COMPARISON_MODE, which only controls whether or not an error is thrown here or not. We have too many things which are controlled by HIVEMAPREDMODE, setting it to nonstrict for this could turn on other things the user doesn't want. REVISION DETAIL https://reviews.facebook.net/D1803 > Throw a Hive error if we're in strict mode and a the types of a partition > comparison do not match. > -- > > Key: HIVE-2813 > URL: https://issues.apache.org/jira/browse/HIVE-2813 > Project: Hive > Issue Type: Improvement > Components: Query Processor >Affects Versions: 0.8.1, 0.9.0 >Reporter: Dmitry Soshnikov >Priority: Minor > Labels: newbie, patch > Fix For: 0.9.0 > > Attachments: HIVE-2813.D1803.2.patch > > Original Estimate: 24h > Remaining Estimate: 24h > > Oftentimes people try to write queries like > SELECT * > FROM table > WHERE ds = 2011-08-03 > This won't work because quotes are missing, and it'll actually try to filter > on ds = "2000". > People run into this pretty regularly; a simple check on whether the types > match exactly in partition predicates would make this a lot less likely to > happen. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HIVE-2743) Enable SASL mode for HiveServer
[ https://issues.apache.org/jira/browse/HIVE-2743?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Carl Steinbach reassigned HIVE-2743: Assignee: (was: Carl Steinbach) I'm not going to have time to work on this as quickly as I expected. If someone else wants to pick this up that would be great. > Enable SASL mode for HiveServer > --- > > Key: HIVE-2743 > URL: https://issues.apache.org/jira/browse/HIVE-2743 > Project: Hive > Issue Type: New Feature > Components: Authentication, Security, Server Infrastructure >Reporter: Carl Steinbach > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2791) filter is still removed due to regression of HIVE-1538 althougth HIVE-2344
[ https://issues.apache.org/jira/browse/HIVE-2791?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-2791: --- Resolution: Fixed Status: Resolved (was: Patch Available) Committed to trunk. Thanks, binlijin! > filter is still removed due to regression of HIVE-1538 althougth HIVE-2344 > -- > > Key: HIVE-2791 > URL: https://issues.apache.org/jira/browse/HIVE-2791 > Project: Hive > Issue Type: Bug >Reporter: binlijin >Assignee: binlijin > Fix For: 0.9.0 > > Attachments: HIVE-2791.2.patch, HIVE-2791.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
HIVE-2773 is ready for review
Hi, HIVE-2773 is ready for review. Would be great to get feedback on this. It's part of the effort in Hcatalog to use Hive's storageHandler and make the written storageHandlers interoperable. -Francis
[jira] [Commented] (HIVE-2792) SUBSTR(CAST( AS BINARY)) produces unexpected results
[ https://issues.apache.org/jira/browse/HIVE-2792?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13212572#comment-13212572 ] Hudson commented on HIVE-2792: -- Integrated in Hive-trunk-h0.21 #1267 (See [https://builds.apache.org/job/Hive-trunk-h0.21/1267/]) HIVE-2792: SUBSTR(CAST( AS BINARY)) produces unexpected results (navis via hashutosh) (Revision 1291633) Result = SUCCESS hashutosh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1291633 Files : * /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFSubstr.java * /hive/trunk/ql/src/test/queries/clientpositive/ba_table_udfs.q * /hive/trunk/ql/src/test/queries/clientpositive/udf_substr.q * /hive/trunk/ql/src/test/results/clientpositive/ba_table_udfs.q.out * /hive/trunk/ql/src/test/results/clientpositive/udf_substr.q.out > SUBSTR(CAST( AS BINARY)) produces unexpected results > > > Key: HIVE-2792 > URL: https://issues.apache.org/jira/browse/HIVE-2792 > Project: Hive > Issue Type: Bug > Components: UDF >Affects Versions: 0.8.0, 0.8.1 >Reporter: Carl Steinbach >Assignee: Navis > Fix For: 0.9.0 > > Attachments: HIVE-2792.D1797.1.patch, HIVE-2792.D1797.2.patch, > HIVE-2792.D1797.2.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2773) HiveStorageHandler.configureTableJobProperites() should let the handler know wether it is configuration for input or output
[ https://issues.apache.org/jira/browse/HIVE-2773?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Phabricator updated HIVE-2773: -- Attachment: HIVE-2773.D1815.1.patch toffer requested code review of "HIVE-2773 [jira] HiveStorageHandler.configureTableJobProperites() should let the handler know wether it is configuration for input or output". Reviewers: JIRA, cwsteinbach HIVE-2773 HiveStorageHandler.configureTableJobProperites() should let the handler know wether it is configuration for input or output TEST PLAN Empty REVISION DETAIL https://reviews.facebook.net/D1815 AFFECTED FILES hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseStorageHandler.java ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java ql/src/java/org/apache/hadoop/hive/ql/metadata/DefaultStorageHandler.java ql/src/java/org/apache/hadoop/hive/ql/metadata/HiveStorageHandler.java ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java ql/src/java/org/apache/hadoop/hive/ql/plan/MapredLocalWork.java ql/src/java/org/apache/hadoop/hive/ql/plan/PartitionDesc.java ql/src/java/org/apache/hadoop/hive/ql/plan/PlanUtils.java MANAGE HERALD DIFFERENTIAL RULES https://reviews.facebook.net/herald/view/differential/ WHY DID I GET THIS EMAIL? https://reviews.facebook.net/herald/transcript/3867/ Tip: use the X-Herald-Rules header to filter Herald messages in your client. > HiveStorageHandler.configureTableJobProperites() should let the handler know > wether it is configuration for input or output > --- > > Key: HIVE-2773 > URL: https://issues.apache.org/jira/browse/HIVE-2773 > Project: Hive > Issue Type: Improvement >Reporter: Francis Liu > Labels: hcatalog, storage_handler > Attachments: HIVE-2773.D1815.1.patch, HIVE-2773.patch > > > HiveStorageHandler.configureTableJobProperties() is called to allow the > storage handler to setup any properties that the underlying > inputformat/outputformat/serde may need. But the handler implementation does > not know whether it is being called for configuring input or output. This > makes it a problem for handlers which sets an external state. In the case of > HCatalog's HBase storageHandler, whenever a write needs to be configured we > create a write transaction which needs to be committed or aborted later on. > In this case configuring for both input and output each time > configureTableJobProperties() is called would not be desirable. This has > become an issue since HCatalog is dropping storageDrivers for SerDe and > StorageHandler (see HCATALOG-237). > My proposal is to replace configureTableJobProperties() with two methods: > configureInputJobProperties() > configureOutputJobProperties() > Each method will have the same signature. I cursory look at the code and I > believe changes should be straighforward also given that we are not really > changing anything just splitting responsibility. If the community is fine > with this approach I will go ahead and create a aptch. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: HIVE-2773 is ready for review
Hi Francis, Can you please submit a review request on Phabricator? Instructions are located here: https://cwiki.apache.org/Hive/phabricatorcodereview.html Thanks. Carl On Mon, Feb 20, 2012 at 11:38 PM, Francis Liu wrote: > Hi, > > HIVE-2773 is ready for review. Would be great to get feedback on this. It's > part of the effort in Hcatalog to use Hive's storageHandler and make the > written storageHandlers interoperable. > > -Francis > > >
