[jira] [Commented] (HIVE-2134) Remove System.exit
[ https://issues.apache.org/jira/browse/HIVE-2134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13026879#comment-13026879 ] Amareshwari Sriramadasu commented on HIVE-2134: --- Patch removes System.exit calls and as logs warning saying it is exiting. Shouldn't it throw out the exceptions for exit? Remove System.exit --- Key: HIVE-2134 URL: https://issues.apache.org/jira/browse/HIVE-2134 Project: Hive Issue Type: Improvement Reporter: Ashutosh Chauhan Assignee: Ashutosh Chauhan Fix For: 0.8.0 Attachments: no_sysexit.patch System.exit exists in various places in code. Its good to get rid of it. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2133) DROP TABLE IF EXISTS should not fail if a view of that name exists
[ https://issues.apache.org/jira/browse/HIVE-2133?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Carl Steinbach updated HIVE-2133: - Resolution: Fixed Hadoop Flags: [Reviewed] Status: Resolved (was: Patch Available) Committed to trunk. Thanks John! DROP TABLE IF EXISTS should not fail if a view of that name exists -- Key: HIVE-2133 URL: https://issues.apache.org/jira/browse/HIVE-2133 Project: Hive Issue Type: Improvement Components: Query Processor Reporter: John Sichi Assignee: John Sichi Fix For: 0.8.0 Attachments: HIVE-2133.1.patch We should match MySQL behavior on this. Likewise for DROP VIEW IF EXISTS when a table of that name exists. Note that without IF EXISTS, we still want the statement to fail when the existing object type does not match. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2121) Input Sampling By Splits
[ https://issues.apache.org/jira/browse/HIVE-2121?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siying Dong updated HIVE-2121: -- Attachment: HIVE-2121.8.patch fix tests or test outputs. Input Sampling By Splits Key: HIVE-2121 URL: https://issues.apache.org/jira/browse/HIVE-2121 Project: Hive Issue Type: New Feature Reporter: Siying Dong Assignee: Siying Dong Attachments: HIVE-2121.1.patch, HIVE-2121.2.patch, HIVE-2121.3.patch, HIVE-2121.4.patch, HIVE-2121.5.patch, HIVE-2121.6.patch, HIVE-2121.7.patch, HIVE-2121.8.patch We need a better input sampling to serve at least two purposes: 1. test their queries against a smaller data set 2. understand more about how the data look like without scanning the whole table. A simple function that gives a subset splits will help in those cases. It doesn't have to be strict sampling. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-1644) use filter pushdown for automatically accessing indexes
[
https://issues.apache.org/jira/browse/HIVE-1644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Russell Melick updated HIVE-1644:
-
Attachment: hive.log
I've attached the hive.log I get from running
{{{
ant test -Dtestcase=TestCliDriver -Dqfile=index_auto_test_if_used.q
}}}
I still am only seeing input formats of
org.apache.hadoop.hive.ql.io.HiveInputFormat and
org.apache.hadoop.hive.ql.io.CombineHiveInputFormat
But, the good news (I guess) is that the splits exception also happens when
using the index manually
use filter pushdown for automatically accessing indexes
---
Key: HIVE-1644
URL: https://issues.apache.org/jira/browse/HIVE-1644
Project: Hive
Issue Type: Improvement
Components: Indexing
Affects Versions: 0.8.0
Reporter: John Sichi
Assignee: Russell Melick
Attachments: HIVE-1644.1.patch, HIVE-1644.10.patch,
HIVE-1644.11.patch, HIVE-1644.12.patch, HIVE-1644.13.patch,
HIVE-1644.14.patch, HIVE-1644.15.patch, HIVE-1644.16.patch,
HIVE-1644.17.patch, HIVE-1644.18.patch, HIVE-1644.2.patch, HIVE-1644.3.patch,
HIVE-1644.4.patch, HIVE-1644.5.patch, HIVE-1644.6.patch, HIVE-1644.7.patch,
HIVE-1644.8.patch, HIVE-1644.9.patch, hive.log
HIVE-1226 provides utilities for analyzing filters which have been pushed
down to a table scan. The next step is to use these for selecting available
indexes and generating access plans for those indexes.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2134) Remove System.exit
[ https://issues.apache.org/jira/browse/HIVE-2134?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Namit Jain updated HIVE-2134: - Status: Open (was: Patch Available) Remove System.exit --- Key: HIVE-2134 URL: https://issues.apache.org/jira/browse/HIVE-2134 Project: Hive Issue Type: Improvement Reporter: Ashutosh Chauhan Assignee: Ashutosh Chauhan Fix For: 0.8.0 Attachments: no_sysexit.patch System.exit exists in various places in code. Its good to get rid of it. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2121) Input Sampling By Splits
[ https://issues.apache.org/jira/browse/HIVE-2121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13027125#comment-13027125 ] Namit Jain commented on HIVE-2121: -- running tests Input Sampling By Splits Key: HIVE-2121 URL: https://issues.apache.org/jira/browse/HIVE-2121 Project: Hive Issue Type: New Feature Reporter: Siying Dong Assignee: Siying Dong Attachments: HIVE-2121.1.patch, HIVE-2121.2.patch, HIVE-2121.3.patch, HIVE-2121.4.patch, HIVE-2121.5.patch, HIVE-2121.6.patch, HIVE-2121.7.patch, HIVE-2121.8.patch We need a better input sampling to serve at least two purposes: 1. test their queries against a smaller data set 2. understand more about how the data look like without scanning the whole table. A simple function that gives a subset splits will help in those cases. It doesn't have to be strict sampling. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-1644) use filter pushdown for automatically accessing indexes
[ https://issues.apache.org/jira/browse/HIVE-1644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13027141#comment-13027141 ] John Sichi commented on HIVE-1644: -- Oh, I see...the reason it didn't work for you is that your setInputAttributes method is working on the job object. For MapRedTask, it needs to work on the conf object instead. So make it take an input parameter and pass in job from ExecDriver, and conf from MapRedTask. Since the splits exception happens for both manual/auto, we don't need to try to address it as part of this JIRA, so you can open a followup for that. But it means you won't be able to check in a meaningful test case, so better if you have a fix. When newInputPaths.toString() == , you could try calling FileInputFormat.setInputPaths(job, new Path[0]). I'm not sure whether that will work. use filter pushdown for automatically accessing indexes --- Key: HIVE-1644 URL: https://issues.apache.org/jira/browse/HIVE-1644 Project: Hive Issue Type: Improvement Components: Indexing Affects Versions: 0.8.0 Reporter: John Sichi Assignee: Russell Melick Attachments: HIVE-1644.1.patch, HIVE-1644.10.patch, HIVE-1644.11.patch, HIVE-1644.12.patch, HIVE-1644.13.patch, HIVE-1644.14.patch, HIVE-1644.15.patch, HIVE-1644.16.patch, HIVE-1644.17.patch, HIVE-1644.18.patch, HIVE-1644.2.patch, HIVE-1644.3.patch, HIVE-1644.4.patch, HIVE-1644.5.patch, HIVE-1644.6.patch, HIVE-1644.7.patch, HIVE-1644.8.patch, HIVE-1644.9.patch, hive.log HIVE-1226 provides utilities for analyzing filters which have been pushed down to a table scan. The next step is to use these for selecting available indexes and generating access plans for those indexes. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2135) Refactor HiveMetaStore.java
[ https://issues.apache.org/jira/browse/HIVE-2135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13027144#comment-13027144 ] Carl Steinbach commented on HIVE-2135: -- Changes look good to me. Nice to see some cleanup work! @John and Paul: Can one of you take a look at this and make sure it doesn't break anything on the FB side? Refactor HiveMetaStore.java --- Key: HIVE-2135 URL: https://issues.apache.org/jira/browse/HIVE-2135 Project: Hive Issue Type: Improvement Components: Metastore Reporter: Ashutosh Chauhan Assignee: Ashutosh Chauhan Fix For: 0.8.0 Attachments: refactor.patch HiveMetaStore.java is more then 3000 LOC which contains wide variety of code. It will be good to refactor it so that its easier to maintain. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
Jenkins build is back to normal : Hive-trunk-h0.20 #700
See https://builds.apache.org/hudson/job/Hive-trunk-h0.20/700/
[jira] [Commented] (HIVE-1731) Improve miscellaneous error messages
[ https://issues.apache.org/jira/browse/HIVE-1731?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13027186#comment-13027186 ] Syed S. Albiz commented on HIVE-1731: - Will take a look Improve miscellaneous error messages Key: HIVE-1731 URL: https://issues.apache.org/jira/browse/HIVE-1731 Project: Hive Issue Type: Improvement Components: Diagnosability, Query Processor Reporter: John Sichi This is a place for accumulating error message improvements so that we can update a bunch in batch. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HIVE-1731) Improve miscellaneous error messages
[ https://issues.apache.org/jira/browse/HIVE-1731?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Sichi reassigned HIVE-1731: Assignee: Syed S. Albiz Improve miscellaneous error messages Key: HIVE-1731 URL: https://issues.apache.org/jira/browse/HIVE-1731 Project: Hive Issue Type: Improvement Components: Diagnosability, Query Processor Reporter: John Sichi Assignee: Syed S. Albiz This is a place for accumulating error message improvements so that we can update a bunch in batch. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2029) MetaStore ConnectionURL updates need to trigger creation of Default DB if it doesn't exist
[ https://issues.apache.org/jira/browse/HIVE-2029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13027219#comment-13027219 ] Ashutosh Chauhan commented on HIVE-2029: +1 for removing 'hive.metastore.force.reload.conf'. Tests should not be written in such a manner that they need to modify the code. This has opened up multiple paths in Metastore code which needs to be reasoned about. Is there an agreement on whether connectionURL hook should look for default DB and create it in case it doesn't exist? MetaStore ConnectionURL updates need to trigger creation of Default DB if it doesn't exist -- Key: HIVE-2029 URL: https://issues.apache.org/jira/browse/HIVE-2029 Project: Hive Issue Type: Bug Components: Metastore Affects Versions: 0.7.0 Reporter: Carl Steinbach HIVE-1219 defined the JDOConnectionURLHook plugin, and integrated this feature into HiveMetaStore. On MetaStore operation failures, this plugin is used to update the metastore ConnectionURL configuration property. Currently this update triggers the reinitialization of the underlying JDO PersistenceManager, but it does not trigger checks to see if the default database exists, nor will it create the default database if it does not exist. It needs to do both. This ticket also covers removing the 'hive.metastore.force.reload.conf' property from HiveConf and HiveMetaStore. This property should not have been added in the first place since its sole purpose is to facilitate testing of the JDOConnectionURLHook mechanism by unnaturally forcing reinitialization of the PersistenceManager. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2121) Input Sampling By Splits
[ https://issues.apache.org/jira/browse/HIVE-2121?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Namit Jain updated HIVE-2121: - Resolution: Fixed Hadoop Flags: [Reviewed] Status: Resolved (was: Patch Available) Committed. Thanks Siying Input Sampling By Splits Key: HIVE-2121 URL: https://issues.apache.org/jira/browse/HIVE-2121 Project: Hive Issue Type: New Feature Reporter: Siying Dong Assignee: Siying Dong Attachments: HIVE-2121.1.patch, HIVE-2121.2.patch, HIVE-2121.3.patch, HIVE-2121.4.patch, HIVE-2121.5.patch, HIVE-2121.6.patch, HIVE-2121.7.patch, HIVE-2121.8.patch We need a better input sampling to serve at least two purposes: 1. test their queries against a smaller data set 2. understand more about how the data look like without scanning the whole table. A simple function that gives a subset splits will help in those cases. It doesn't have to be strict sampling. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2123) CommandNeedRetryException needs release locks
[ https://issues.apache.org/jira/browse/HIVE-2123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13027229#comment-13027229 ] He Yongqiang commented on HIVE-2123: looks good to me. will commit after tests pass. CommandNeedRetryException needs release locks - Key: HIVE-2123 URL: https://issues.apache.org/jira/browse/HIVE-2123 Project: Hive Issue Type: Bug Reporter: Siying Dong Assignee: Siying Dong Attachments: HIVE-2123.1.patch, HIVE-2123.2.patch now when CommandNeedRetryException is thrown, locks are not released. Not sure whether it will cause problem, since the same locks will be acquired when retrying it. It is anyway something we need to fix. Also we can do some little code cleaning up to make future mistakes less likely. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2038) Metastore listener
[ https://issues.apache.org/jira/browse/HIVE-2038?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13027239#comment-13027239 ] Carl Steinbach commented on HIVE-2038: -- @Ashutosh: Apologies for the delay in getting this committed. I've been having problems with my test setup lately, but I think I've got the issue resolved. Expect to see this go in sometime in the next day or so. Metastore listener -- Key: HIVE-2038 URL: https://issues.apache.org/jira/browse/HIVE-2038 Project: Hive Issue Type: Improvement Components: Metastore Reporter: Ashutosh Chauhan Assignee: Ashutosh Chauhan Fix For: 0.8.0 Attachments: hive-2038.patch, hive_2038_3.patch, metastore_listener.patch, metastore_listener.patch, metastore_listener.patch Provide to way to observe changes happening on Metastore -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2134) Remove System.exit
[ https://issues.apache.org/jira/browse/HIVE-2134?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-2134: --- Attachment: hive_2134.patch Now throws the exception instead of doing System.exit. Didn't do it in MetaStoreUtils since there it is launched in a thread and run() doesn't allow to throw checked exceptions. I could have casted exception into RuntimeException but even that would not make sense because once exception is thrown in thread, there is no one who can catch it and do something with it. Remove System.exit --- Key: HIVE-2134 URL: https://issues.apache.org/jira/browse/HIVE-2134 Project: Hive Issue Type: Improvement Reporter: Ashutosh Chauhan Assignee: Ashutosh Chauhan Fix For: 0.8.0 Attachments: hive_2134.patch, no_sysexit.patch System.exit exists in various places in code. Its good to get rid of it. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2134) Remove System.exit
[ https://issues.apache.org/jira/browse/HIVE-2134?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-2134: --- Status: Patch Available (was: Open) Remove System.exit --- Key: HIVE-2134 URL: https://issues.apache.org/jira/browse/HIVE-2134 Project: Hive Issue Type: Improvement Reporter: Ashutosh Chauhan Assignee: Ashutosh Chauhan Fix For: 0.8.0 Attachments: hive_2134.patch, no_sysexit.patch System.exit exists in various places in code. Its good to get rid of it. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
Fwd: HCatalogJournal now available on wiki
-- Forwarded message -- From: Alan Gates ga...@yahoo-inc.com Date: Fri, Apr 29, 2011 at 5:24 PM Subject: HCatalogJournal now available on wiki To: hcatalog-...@incubator.apache.org As requested at the Hive contributor meetup, the HCatalog roadmap is now available: https://cwiki.apache.org/confluence/display/HCATALOG/HCatalogJournal Currently this is updated through version 0.2, which we are currently doing development on. Alan.
[jira] [Updated] (HIVE-2029) MetaStore ConnectionURL updates need to trigger creation of Default DB if it doesn't exist
[ https://issues.apache.org/jira/browse/HIVE-2029?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-2029: --- Attachment: hive_2029.patch Patch which removes reload.conf property. MetaStore ConnectionURL updates need to trigger creation of Default DB if it doesn't exist -- Key: HIVE-2029 URL: https://issues.apache.org/jira/browse/HIVE-2029 Project: Hive Issue Type: Bug Components: Metastore Affects Versions: 0.7.0 Reporter: Carl Steinbach Attachments: hive_2029.patch HIVE-1219 defined the JDOConnectionURLHook plugin, and integrated this feature into HiveMetaStore. On MetaStore operation failures, this plugin is used to update the metastore ConnectionURL configuration property. Currently this update triggers the reinitialization of the underlying JDO PersistenceManager, but it does not trigger checks to see if the default database exists, nor will it create the default database if it does not exist. It needs to do both. This ticket also covers removing the 'hive.metastore.force.reload.conf' property from HiveConf and HiveMetaStore. This property should not have been added in the first place since its sole purpose is to facilitate testing of the JDOConnectionURLHook mechanism by unnaturally forcing reinitialization of the PersistenceManager. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-1644) use filter pushdown for automatically accessing indexes
[
https://issues.apache.org/jira/browse/HIVE-1644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13027283#comment-13027283
]
Russell Melick commented on HIVE-1644:
--
I can send it back out of the index handler, and that seems to function
correctly.
{{{
if (newInputPaths.length() == 0) {
return super.getSplits(job, numSplits);
} else {
FileInputFormat.setInputPaths(job, newInputPaths.toString());
}
}}}
But, this means that we won't use the index to get the splits, so I don't think
our test will work anymore. It will return results from the base table. This
feels like the correct behavior in the long term, even though it breaks the
test.
use filter pushdown for automatically accessing indexes
---
Key: HIVE-1644
URL: https://issues.apache.org/jira/browse/HIVE-1644
Project: Hive
Issue Type: Improvement
Components: Indexing
Affects Versions: 0.8.0
Reporter: John Sichi
Assignee: Russell Melick
Attachments: HIVE-1644.1.patch, HIVE-1644.10.patch,
HIVE-1644.11.patch, HIVE-1644.12.patch, HIVE-1644.13.patch,
HIVE-1644.14.patch, HIVE-1644.15.patch, HIVE-1644.16.patch,
HIVE-1644.17.patch, HIVE-1644.18.patch, HIVE-1644.2.patch, HIVE-1644.3.patch,
HIVE-1644.4.patch, HIVE-1644.5.patch, HIVE-1644.6.patch, HIVE-1644.7.patch,
HIVE-1644.8.patch, HIVE-1644.9.patch, hive.log
HIVE-1226 provides utilities for analyzing filters which have been pushed
down to a table scan. The next step is to use these for selecting available
indexes and generating access plans for those indexes.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-1644) use filter pushdown for automatically accessing indexes
[
https://issues.apache.org/jira/browse/HIVE-1644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13027294#comment-13027294
]
Russell Melick commented on HIVE-1644:
--
Unfortunately, having an empty array gives out of bounds exceptions.
{{
java.lang.ArrayIndexOutOfBoundsException: 0
[junit] at
org.apache.hadoop.mapred.FileInputFormat.setInputPaths(FileInputFormat.java:309)
}}
I tried looking through the code at other places we call setInputPaths. In a
few places, we create a directory by calling
{{
Path dir = new Path(System.getProperty(test.data.dir, .));
FileInputFormat.setInputPaths(job, dir);
}}
I tried using this, but unfortunately, this also gives an exception.
{{
java.io.IOException: cannot find dir = file:/Users/rmelick/hive/ql/mapred in
pathToPartitionInfo:
[pfile:/Users/rmelick/hive/build/ql/test/data/warehouse/temp]
}}
The ql/mapred directory does not exist on my computer, but I also tried
changing it to just hive/ql, and it also failed with the same exception. I'm
not sure if TestFlatFileInputFormat:146 is creating the temp file like you were
thinking. I haven't tried doing it like that.
use filter pushdown for automatically accessing indexes
---
Key: HIVE-1644
URL: https://issues.apache.org/jira/browse/HIVE-1644
Project: Hive
Issue Type: Improvement
Components: Indexing
Affects Versions: 0.8.0
Reporter: John Sichi
Assignee: Russell Melick
Attachments: HIVE-1644.1.patch, HIVE-1644.10.patch,
HIVE-1644.11.patch, HIVE-1644.12.patch, HIVE-1644.13.patch,
HIVE-1644.14.patch, HIVE-1644.15.patch, HIVE-1644.16.patch,
HIVE-1644.17.patch, HIVE-1644.18.patch, HIVE-1644.2.patch, HIVE-1644.3.patch,
HIVE-1644.4.patch, HIVE-1644.5.patch, HIVE-1644.6.patch, HIVE-1644.7.patch,
HIVE-1644.8.patch, HIVE-1644.9.patch, hive.log
HIVE-1226 provides utilities for analyzing filters which have been pushed
down to a table scan. The next step is to use these for selecting available
indexes and generating access plans for those indexes.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
