[jira] [Commented] (HIVE-1803) Implement bitmap indexing in Hive
[ https://issues.apache.org/jira/browse/HIVE-1803?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14042024#comment-14042024 ] xuanjinlee commented on HIVE-1803: -- Hello everyone ，how to use this page？I am a new learner！ Thanks Implement bitmap indexing in Hive - Key: HIVE-1803 URL: https://issues.apache.org/jira/browse/HIVE-1803 Project: Hive Issue Type: New Feature Components: Indexing Reporter: Marquis Wang Assignee: Marquis Wang Fix For: 0.8.0 Attachments: HIVE-1803.1.patch, HIVE-1803.10.patch, HIVE-1803.11.patch, HIVE-1803.12.patch, HIVE-1803.13.patch, HIVE-1803.14.patch, HIVE-1803.14.patch, HIVE-1803.15.patch, HIVE-1803.15.patch, HIVE-1803.2.patch, HIVE-1803.3.patch, HIVE-1803.4.patch, HIVE-1803.5.patch, HIVE-1803.6.patch, HIVE-1803.7.patch, HIVE-1803.8.patch, HIVE-1803.9.patch, JavaEWAH_20110304.zip, bitmap_index_1.png, bitmap_index_2.png, javaewah.jar, javaewah.jar, unit-tests.2.patch, unit-tests.3.patch, unit-tests.patch Implement bitmap index handler to complement compact indexing. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HIVE-1803) Implement bitmap indexing in Hive
[ https://issues.apache.org/jira/browse/HIVE-1803?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13066011#comment-13066011 ] siddharth ramanan commented on HIVE-1803: - Hi, I am trying to put an index for a string column in a table, I am getting NumberFormatException with respect to _offsets in the index table when I run the query? Can you give me suggestions as to how to do this? Thanks, Siddharth Implement bitmap indexing in Hive - Key: HIVE-1803 URL: https://issues.apache.org/jira/browse/HIVE-1803 Project: Hive Issue Type: New Feature Components: Indexing Reporter: Marquis Wang Assignee: Marquis Wang Fix For: 0.8.0 Attachments: HIVE-1803.1.patch, HIVE-1803.10.patch, HIVE-1803.11.patch, HIVE-1803.12.patch, HIVE-1803.13.patch, HIVE-1803.14.patch, HIVE-1803.14.patch, HIVE-1803.15.patch, HIVE-1803.15.patch, HIVE-1803.2.patch, HIVE-1803.3.patch, HIVE-1803.4.patch, HIVE-1803.5.patch, HIVE-1803.6.patch, HIVE-1803.7.patch, HIVE-1803.8.patch, HIVE-1803.9.patch, JavaEWAH_20110304.zip, bitmap_index_1.png, bitmap_index_2.png, javaewah.jar, javaewah.jar, unit-tests.2.patch, unit-tests.3.patch, unit-tests.patch Implement bitmap index handler to complement compact indexing. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-1803) Implement bitmap indexing in Hive
[ https://issues.apache.org/jira/browse/HIVE-1803?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13066147#comment-13066147 ] John Sichi commented on HIVE-1803: -- @Siddharth: please open a new JIRA issue, including exact steps to reproduce your problem. Implement bitmap indexing in Hive - Key: HIVE-1803 URL: https://issues.apache.org/jira/browse/HIVE-1803 Project: Hive Issue Type: New Feature Components: Indexing Reporter: Marquis Wang Assignee: Marquis Wang Fix For: 0.8.0 Attachments: HIVE-1803.1.patch, HIVE-1803.10.patch, HIVE-1803.11.patch, HIVE-1803.12.patch, HIVE-1803.13.patch, HIVE-1803.14.patch, HIVE-1803.14.patch, HIVE-1803.15.patch, HIVE-1803.15.patch, HIVE-1803.2.patch, HIVE-1803.3.patch, HIVE-1803.4.patch, HIVE-1803.5.patch, HIVE-1803.6.patch, HIVE-1803.7.patch, HIVE-1803.8.patch, HIVE-1803.9.patch, JavaEWAH_20110304.zip, bitmap_index_1.png, bitmap_index_2.png, javaewah.jar, javaewah.jar, unit-tests.2.patch, unit-tests.3.patch, unit-tests.patch Implement bitmap index handler to complement compact indexing. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-1803) Implement bitmap indexing in Hive
[ https://issues.apache.org/jira/browse/HIVE-1803?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13024638#comment-13024638 ] John Sichi commented on HIVE-1803: -- With patch 15, everything passed for me *except* the groupby* tests, so I think it's something about the Java version you're using (unrelated to your change). When I revert those .q.xml files, TestParse passes. I'm going to re-run the full ant test with those reverted, and assuming that passes, +1. Implement bitmap indexing in Hive - Key: HIVE-1803 URL: https://issues.apache.org/jira/browse/HIVE-1803 Project: Hive Issue Type: New Feature Components: Indexing Reporter: Marquis Wang Assignee: Marquis Wang Attachments: HIVE-1803.1.patch, HIVE-1803.10.patch, HIVE-1803.11.patch, HIVE-1803.12.patch, HIVE-1803.13.patch, HIVE-1803.14.patch, HIVE-1803.14.patch, HIVE-1803.15.patch, HIVE-1803.15.patch, HIVE-1803.2.patch, HIVE-1803.3.patch, HIVE-1803.4.patch, HIVE-1803.5.patch, HIVE-1803.6.patch, HIVE-1803.7.patch, HIVE-1803.8.patch, HIVE-1803.9.patch, JavaEWAH_20110304.zip, bitmap_index_1.png, bitmap_index_2.png, javaewah.jar, javaewah.jar, unit-tests.2.patch, unit-tests.3.patch, unit-tests.patch Implement bitmap index handler to complement compact indexing. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-1803) Implement bitmap indexing in Hive
[
https://issues.apache.org/jira/browse/HIVE-1803?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13023519#comment-13023519
]
John Sichi commented on HIVE-1803:
--
Everything passed...eeexcept TestParse, where I got 44 failures. Example diff
below; somehow the output seems to have gotten reordered, and it's in the area
of virtual columns. Best is to find out why and fix it, otherwise make sure
it's deterministic and then update .q.out.
{noformat}
[junit] diff -b -I'\(\(java version=.* class=java.beans.XMLDecoder\)\|
\(string.*/tmp/.*/string\)\|\(stringfile:.*/string\)\|\(stringpfile:.*
/string\)\|\(string[0-9]\{10\}/string\)\|\(string/.*/warehouse/.*/strin
g\)\)' /data/users/jsichi/open/test-trunk/build/ql/test/logs/positive/case_sens
itivity.q.xml /data/users/jsichi/open/test-trunk/ql/src/test/results/compiler/pl
an/case_sensitivity.q.xml
[junit] 1209c1209
[junit] stringINPUT__FILE__NAME/string
[junit] ---
[junit] stringBLOCK__OFFSET__INSIDE__FILE/string
[junit] 1215c1215,1219
[junit] object idref=PrimitiveTypeInfo0/
[junit] ---
[junit] object class=org.apache.hadoop.hive.serde2.ty
peinfo.PrimitiveTypeInfo
[junit] void property=typeName
[junit] stringbigint/string
[junit] /void
[junit] /object
[junit] 1225c1229
[junit] stringBLOCK__OFFSET__INSIDE__FILE/string
[junit] ---
[junit] stringINPUT__FILE__NAME/string
[junit] 1231,1235c1235
[junit] object class=org.apache.hadoop.hive.serde2.ty
peinfo.PrimitiveTypeInfo
[junit] void property=typeName
[junit] stringbigint/string
{noformat}
Implement bitmap indexing in Hive
-
Key: HIVE-1803
URL: https://issues.apache.org/jira/browse/HIVE-1803
Project: Hive
Issue Type: New Feature
Components: Indexing
Reporter: Marquis Wang
Assignee: Marquis Wang
Attachments: HIVE-1803.1.patch, HIVE-1803.10.patch,
HIVE-1803.11.patch, HIVE-1803.12.patch, HIVE-1803.13.patch,
HIVE-1803.2.patch, HIVE-1803.3.patch, HIVE-1803.4.patch, HIVE-1803.5.patch,
HIVE-1803.6.patch, HIVE-1803.7.patch, HIVE-1803.8.patch, HIVE-1803.9.patch,
JavaEWAH_20110304.zip, bitmap_index_1.png, bitmap_index_2.png, javaewah.jar,
javaewah.jar, unit-tests.2.patch, unit-tests.3.patch, unit-tests.patch
Implement bitmap index handler to complement compact indexing.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-1803) Implement bitmap indexing in Hive
[ https://issues.apache.org/jira/browse/HIVE-1803?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13023634#comment-13023634 ] John Sichi commented on HIVE-1803: -- The number comes from the operator ID; I'm not sure why those would have shifted, but if it's consistent, then give me a patch that just updates the .q.out files and if that passes for me, we'll call it a day. Implement bitmap indexing in Hive - Key: HIVE-1803 URL: https://issues.apache.org/jira/browse/HIVE-1803 Project: Hive Issue Type: New Feature Components: Indexing Reporter: Marquis Wang Assignee: Marquis Wang Attachments: HIVE-1803.1.patch, HIVE-1803.10.patch, HIVE-1803.11.patch, HIVE-1803.12.patch, HIVE-1803.13.patch, HIVE-1803.14.patch, HIVE-1803.14.patch, HIVE-1803.2.patch, HIVE-1803.3.patch, HIVE-1803.4.patch, HIVE-1803.5.patch, HIVE-1803.6.patch, HIVE-1803.7.patch, HIVE-1803.8.patch, HIVE-1803.9.patch, JavaEWAH_20110304.zip, bitmap_index_1.png, bitmap_index_2.png, javaewah.jar, javaewah.jar, unit-tests.2.patch, unit-tests.3.patch, unit-tests.patch Implement bitmap index handler to complement compact indexing. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-1803) Implement bitmap indexing in Hive
[ https://issues.apache.org/jira/browse/HIVE-1803?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13023355#comment-13023355 ] John Sichi commented on HIVE-1803: -- Meh, I'm still getting numRows failures myself. I noticed that your patch includes some changes to existing test outputs (e.g. bucketmapjoin1.q.out) where it is setting the expected numRows to 0; you should have reverted those before generating the patch. But the failure I got was in another existing test (filter_join_breaktask). I'm trying again after reverting the ones you changed (in case the failure I saw was a side effect), but I'm pessimistic; I'm wondering if something innocuous about the change is somehow exposing some existing non-determinism. Implement bitmap indexing in Hive - Key: HIVE-1803 URL: https://issues.apache.org/jira/browse/HIVE-1803 Project: Hive Issue Type: New Feature Components: Indexing Reporter: Marquis Wang Assignee: Marquis Wang Attachments: HIVE-1803.1.patch, HIVE-1803.10.patch, HIVE-1803.11.patch, HIVE-1803.12.patch, HIVE-1803.2.patch, HIVE-1803.3.patch, HIVE-1803.4.patch, HIVE-1803.5.patch, HIVE-1803.6.patch, HIVE-1803.7.patch, HIVE-1803.8.patch, HIVE-1803.9.patch, JavaEWAH_20110304.zip, bitmap_index_1.png, bitmap_index_2.png, javaewah.jar, javaewah.jar, unit-tests.2.patch, unit-tests.3.patch, unit-tests.patch Implement bitmap index handler to complement compact indexing. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-1803) Implement bitmap indexing in Hive
[ https://issues.apache.org/jira/browse/HIVE-1803?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13023095#comment-13023095 ] John Sichi commented on HIVE-1803: -- I noticed that in one of my attempts with your previous patches. I think it's the exact same problem you were hitting very early on in the clinic in the Hudson setup, having to do with the stats temp database not getting deleted? Implement bitmap indexing in Hive - Key: HIVE-1803 URL: https://issues.apache.org/jira/browse/HIVE-1803 Project: Hive Issue Type: New Feature Components: Indexing Reporter: Marquis Wang Assignee: Marquis Wang Attachments: HIVE-1803.1.patch, HIVE-1803.10.patch, HIVE-1803.11.patch, HIVE-1803.12.patch, HIVE-1803.2.patch, HIVE-1803.3.patch, HIVE-1803.4.patch, HIVE-1803.5.patch, HIVE-1803.6.patch, HIVE-1803.7.patch, HIVE-1803.8.patch, HIVE-1803.9.patch, JavaEWAH_20110304.zip, bitmap_index_1.png, bitmap_index_2.png, javaewah.jar, javaewah.jar, unit-tests.2.patch, unit-tests.3.patch, unit-tests.patch Implement bitmap index handler to complement compact indexing. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-1803) Implement bitmap indexing in Hive
[ https://issues.apache.org/jira/browse/HIVE-1803?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13023097#comment-13023097 ] Marquis Wang commented on HIVE-1803: I don't see anything that needs to be deleted in my checkout. Where is the stats temp database? Also, if you think it might just be something on our side, can you just run the tests and see if it passes for you? When I ran them I didn't see any other issues besides those, I don't think. Implement bitmap indexing in Hive - Key: HIVE-1803 URL: https://issues.apache.org/jira/browse/HIVE-1803 Project: Hive Issue Type: New Feature Components: Indexing Reporter: Marquis Wang Assignee: Marquis Wang Attachments: HIVE-1803.1.patch, HIVE-1803.10.patch, HIVE-1803.11.patch, HIVE-1803.12.patch, HIVE-1803.2.patch, HIVE-1803.3.patch, HIVE-1803.4.patch, HIVE-1803.5.patch, HIVE-1803.6.patch, HIVE-1803.7.patch, HIVE-1803.8.patch, HIVE-1803.9.patch, JavaEWAH_20110304.zip, bitmap_index_1.png, bitmap_index_2.png, javaewah.jar, javaewah.jar, unit-tests.2.patch, unit-tests.3.patch, unit-tests.patch Implement bitmap index handler to complement compact indexing. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-1803) Implement bitmap indexing in Hive
[ https://issues.apache.org/jira/browse/HIVE-1803?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13023121#comment-13023121 ] John Sichi commented on HIVE-1803: -- Yeah, I can give it a try and see if it passes. Implement bitmap indexing in Hive - Key: HIVE-1803 URL: https://issues.apache.org/jira/browse/HIVE-1803 Project: Hive Issue Type: New Feature Components: Indexing Reporter: Marquis Wang Assignee: Marquis Wang Attachments: HIVE-1803.1.patch, HIVE-1803.10.patch, HIVE-1803.11.patch, HIVE-1803.12.patch, HIVE-1803.2.patch, HIVE-1803.3.patch, HIVE-1803.4.patch, HIVE-1803.5.patch, HIVE-1803.6.patch, HIVE-1803.7.patch, HIVE-1803.8.patch, HIVE-1803.9.patch, JavaEWAH_20110304.zip, bitmap_index_1.png, bitmap_index_2.png, javaewah.jar, javaewah.jar, unit-tests.2.patch, unit-tests.3.patch, unit-tests.patch Implement bitmap index handler to complement compact indexing. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-1803) Implement bitmap indexing in Hive
[ https://issues.apache.org/jira/browse/HIVE-1803?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13018988#comment-13018988 ] John Sichi commented on HIVE-1803: -- I'm already seeing at least one test failures: auto_join28.q, which recently got its log updated (Friday) so you probably rebased before that. It's really difficult to get something like this through without locking down svn for a few days... You have a point about the unit tests; we would have to special-case the new column and leaving the existing ones as is. Ugly any way you slice it. Implement bitmap indexing in Hive - Key: HIVE-1803 URL: https://issues.apache.org/jira/browse/HIVE-1803 Project: Hive Issue Type: New Feature Components: Indexing Reporter: Marquis Wang Assignee: Marquis Wang Attachments: HIVE-1803.1.patch, HIVE-1803.10.patch, HIVE-1803.11.patch, HIVE-1803.2.patch, HIVE-1803.3.patch, HIVE-1803.4.patch, HIVE-1803.5.patch, HIVE-1803.6.patch, HIVE-1803.7.patch, HIVE-1803.8.patch, HIVE-1803.9.patch, JavaEWAH_20110304.zip, bitmap_index_1.png, bitmap_index_2.png, javaewah.jar, javaewah.jar, unit-tests.2.patch, unit-tests.3.patch, unit-tests.patch Implement bitmap index handler to complement compact indexing. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-1803) Implement bitmap indexing in Hive
[ https://issues.apache.org/jira/browse/HIVE-1803?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13017066#comment-13017066 ] John Sichi commented on HIVE-1803: -- OK, maybe the TestMTQueries failure was a side-effect of the other failures...I'll retry with your latest patch. Implement bitmap indexing in Hive - Key: HIVE-1803 URL: https://issues.apache.org/jira/browse/HIVE-1803 Project: Hive Issue Type: New Feature Components: Indexing Reporter: Marquis Wang Assignee: Marquis Wang Attachments: HIVE-1803.1.patch, HIVE-1803.10.patch, HIVE-1803.11.patch, HIVE-1803.2.patch, HIVE-1803.3.patch, HIVE-1803.4.patch, HIVE-1803.5.patch, HIVE-1803.6.patch, HIVE-1803.7.patch, HIVE-1803.8.patch, HIVE-1803.9.patch, JavaEWAH_20110304.zip, bitmap_index_1.png, bitmap_index_2.png, javaewah.jar, javaewah.jar, unit-tests.2.patch, unit-tests.patch Implement bitmap index handler to complement compact indexing. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-1803) Implement bitmap indexing in Hive
[ https://issues.apache.org/jira/browse/HIVE-1803?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13017069#comment-13017069 ] Marquis Wang commented on HIVE-1803: I re-pulled from trunk and made a new patch and there was no difference between the two. If you have the original unit-tests.patch applied then this patch will fail. Can you try patching HIVE-1803.11.patch followed by unit-tests.2.patch on a clean checkout? Implement bitmap indexing in Hive - Key: HIVE-1803 URL: https://issues.apache.org/jira/browse/HIVE-1803 Project: Hive Issue Type: New Feature Components: Indexing Reporter: Marquis Wang Assignee: Marquis Wang Attachments: HIVE-1803.1.patch, HIVE-1803.10.patch, HIVE-1803.11.patch, HIVE-1803.2.patch, HIVE-1803.3.patch, HIVE-1803.4.patch, HIVE-1803.5.patch, HIVE-1803.6.patch, HIVE-1803.7.patch, HIVE-1803.8.patch, HIVE-1803.9.patch, JavaEWAH_20110304.zip, bitmap_index_1.png, bitmap_index_2.png, javaewah.jar, javaewah.jar, unit-tests.2.patch, unit-tests.patch Implement bitmap index handler to complement compact indexing. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-1803) Implement bitmap indexing in Hive
[ https://issues.apache.org/jira/browse/HIVE-1803?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13017088#comment-13017088 ] John Sichi commented on HIVE-1803: -- That's what I did, and the conflicts match files which were in very recent commits. Are you sure you did svn update? If you're using git, there may be some lag in the replica. Implement bitmap indexing in Hive - Key: HIVE-1803 URL: https://issues.apache.org/jira/browse/HIVE-1803 Project: Hive Issue Type: New Feature Components: Indexing Reporter: Marquis Wang Assignee: Marquis Wang Attachments: HIVE-1803.1.patch, HIVE-1803.10.patch, HIVE-1803.11.patch, HIVE-1803.2.patch, HIVE-1803.3.patch, HIVE-1803.4.patch, HIVE-1803.5.patch, HIVE-1803.6.patch, HIVE-1803.7.patch, HIVE-1803.8.patch, HIVE-1803.9.patch, JavaEWAH_20110304.zip, bitmap_index_1.png, bitmap_index_2.png, javaewah.jar, javaewah.jar, unit-tests.2.patch, unit-tests.patch Implement bitmap index handler to complement compact indexing. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-1803) Implement bitmap indexing in Hive
[ https://issues.apache.org/jira/browse/HIVE-1803?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13011380#comment-13011380 ] John Sichi commented on HIVE-1803: -- Right, without row-level skipping, the main use case is AND/OR for block filtering. I'd suggest we get this committed without row-level skipping, and then create a followup for that. Besides AND/OR, having the bitmap index build/access code committed will be useful for others working on related issues such as automatic usage in the WHERE clause. Implement bitmap indexing in Hive - Key: HIVE-1803 URL: https://issues.apache.org/jira/browse/HIVE-1803 Project: Hive Issue Type: New Feature Components: Indexing Reporter: Marquis Wang Assignee: Marquis Wang Attachments: HIVE-1803.1.patch, HIVE-1803.2.patch, HIVE-1803.3.patch, HIVE-1803.4.patch, HIVE-1803.5.patch, HIVE-1803.6.patch, HIVE-1803.7.patch, JavaEWAH_20110304.zip, bitmap_index_1.png, bitmap_index_2.png, javaewah.jar, javaewah.jar Implement bitmap index handler to complement compact indexing. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-1803) Implement bitmap indexing in Hive
[ https://issues.apache.org/jira/browse/HIVE-1803?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13010471#comment-13010471 ] He Yongqiang commented on HIVE-1803: I did not do a detailed look, overall the patch looks good. It follows the steps of existing compact index. But can you do some tests on the javaewah stuff to see its performance? And what features in javawah are used by this jira? For compression, if we can just compress the bitmap using simple RLE, how much differences are there? What i mean, also not sure about, is javawah may have a lot features that we do not need, but that add some overhead to the code. Implement bitmap indexing in Hive - Key: HIVE-1803 URL: https://issues.apache.org/jira/browse/HIVE-1803 Project: Hive Issue Type: New Feature Components: Indexing Reporter: Marquis Wang Assignee: Marquis Wang Attachments: HIVE-1803.1.patch, HIVE-1803.2.patch, HIVE-1803.3.patch, HIVE-1803.4.patch, HIVE-1803.5.patch, HIVE-1803.6.patch, HIVE-1803.7.patch, JavaEWAH_20110304.zip, bitmap_index_1.png, bitmap_index_2.png, javaewah.jar, javaewah.jar Implement bitmap index handler to complement compact indexing. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HIVE-1803) Implement bitmap indexing in Hive
[ https://issues.apache.org/jira/browse/HIVE-1803?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13003735#comment-13003735 ] John Sichi commented on HIVE-1803: -- Table src has two columns (key and value). The value is equivalent to the key. For srcbucket, the value is one plus the key, so at least they're not exactly the same. Implement bitmap indexing in Hive - Key: HIVE-1803 URL: https://issues.apache.org/jira/browse/HIVE-1803 Project: Hive Issue Type: New Feature Components: Indexing Reporter: Marquis Wang Assignee: Marquis Wang Attachments: HIVE-1803.1.patch, HIVE-1803.2.patch, HIVE-1803.3.patch, HIVE-1803.4.patch, HIVE-1803.5.patch, bitmap_index_1.png, bitmap_index_2.png, javaewah.jar, javaewah.jar Implement bitmap index handler to complement compact indexing. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HIVE-1803) Implement bitmap indexing in Hive
[ https://issues.apache.org/jira/browse/HIVE-1803?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12999656#comment-12999656 ] Jeff Hammerbacher commented on HIVE-1803: - Hey, I came across a Daniel Lemire project recently that may be of use here: http://code.google.com/p/javaewah. Later, Jeff Implement bitmap indexing in Hive - Key: HIVE-1803 URL: https://issues.apache.org/jira/browse/HIVE-1803 Project: Hive Issue Type: New Feature Reporter: Marquis Wang Assignee: Marquis Wang Attachments: HIVE-1803.1.patch, HIVE-1803.2.patch, HIVE-1803.3.patch, bitmap_index_1.png, bitmap_index_2.png Implement bitmap index handler to complement compact indexing. -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HIVE-1803) Implement bitmap indexing in Hive
[ https://issues.apache.org/jira/browse/HIVE-1803?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12999657#comment-12999657 ] Marquis Wang commented on HIVE-1803: Thanks Jeff. We've actually seen this and have a patch in the works (next couple days) that uses it. Implement bitmap indexing in Hive - Key: HIVE-1803 URL: https://issues.apache.org/jira/browse/HIVE-1803 Project: Hive Issue Type: New Feature Reporter: Marquis Wang Assignee: Marquis Wang Attachments: HIVE-1803.1.patch, HIVE-1803.2.patch, HIVE-1803.3.patch, bitmap_index_1.png, bitmap_index_2.png Implement bitmap index handler to complement compact indexing. -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HIVE-1803) Implement bitmap indexing in Hive
[
https://issues.apache.org/jira/browse/HIVE-1803?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12991563#comment-12991563
]
John Sichi commented on HIVE-1803:
--
I tried it, and I did get that error, but not in the REBUILD, which worked
fine. So you may be misreading the output.
The error I'm getting is
{noformat}
FAILED: Error in semantic analysis: line 2:65 Expression Not In Group By Key
`_bucketname`
{noformat}
This error is correct since the statement below has `_bucketname` in the SELECT
list but not in the GROUP BY.
{noformat}
INSERT OVERWRITE DIRECTORY /tmp/index_test_index_result SELECT `_bucketname`,
COLLECT_BITMAP_SET(`_offset`, `_bitmaps`) as `_offsets` FROM
default__srcpart_srcpart_index_proj__ x WHERE x.key=100 AND x.ds = '2008-04-08'
GROUP BY x.key, x.ds;
{noformat}
As a result, the test bails out in the middle (without doing any of its DROP
INDEX cleanup). After that, what happens is that the test framework tries to
clean up by dropping all existing tables, but it isn't smart enough to know
that it shouldn't try to drop index tables directly. (We could use a separate
patch for that.)
{noformat}
[junit] org.apache.hadoop.hive.ql.metadata.HiveException:
java.lang.RuntimeException: The table default__srcpart_srcpart_index_proj__ is
an index table. Please do drop index instead.
[junit] at
org.apache.hadoop.hive.ql.metadata.Hive.dropTable(Hive.java:739)
[junit] at
org.apache.hadoop.hive.ql.metadata.Hive.dropTable(Hive.java:716)
[junit] at
org.apache.hadoop.hive.ql.QTestUtil.clearTestSideEffects(QTestUtil.java:333)
[junit] at
org.apache.hadoop.hive.cli.TestCliDriver.setUp(TestCliDriver.java:55)
[junit] at junit.framework.TestCase.runBare(TestCase.java:125)
...
{noformat}
Implement bitmap indexing in Hive
-
Key: HIVE-1803
URL: https://issues.apache.org/jira/browse/HIVE-1803
Project: Hive
Issue Type: New Feature
Reporter: Marquis Wang
Assignee: Marquis Wang
Attachments: HIVE-1803.1.patch, HIVE-1803.2.patch,
bitmap_index_1.png, bitmap_index_2.png
Implement bitmap index handler to complement compact indexing.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HIVE-1803) Implement bitmap indexing in Hive
[ https://issues.apache.org/jira/browse/HIVE-1803?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12988914#comment-12988914 ] John Sichi commented on HIVE-1803: -- Could you describe the problem you're seeing and provide an example of how to reproduce it? Implement bitmap indexing in Hive - Key: HIVE-1803 URL: https://issues.apache.org/jira/browse/HIVE-1803 Project: Hive Issue Type: New Feature Reporter: Marquis Wang Assignee: Marquis Wang Attachments: HIVE-1803.1.patch, bitmap_index_1.png, bitmap_index_2.png Implement bitmap index handler to complement compact indexing. -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HIVE-1803) Implement bitmap indexing in Hive
[ https://issues.apache.org/jira/browse/HIVE-1803?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12934088#action_12934088 ] Marquis Wang commented on HIVE-1803: Added a proposed design document on Hive wiki at http://wiki.apache.org/hadoop/Hive/IndexDev/Bitmap Implement bitmap indexing in Hive - Key: HIVE-1803 URL: https://issues.apache.org/jira/browse/HIVE-1803 Project: Hive Issue Type: New Feature Reporter: Marquis Wang Assignee: Marquis Wang Attachments: bitmap_index_1.png, bitmap_index_2.png Implement bitmap index handler to complement compact indexing. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
