[jira] [Commented] (HIVE-4825) Separate MapredWork into MapWork and ReduceWork
[
https://issues.apache.org/jira/browse/HIVE-4825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13721565#comment-13721565
]
Hive QA commented on HIVE-4825:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12594510/HIVE-4825.5.patch
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 2653 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_dynamic_partition_skip_default
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/205/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/205/console
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.CleanupPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests failed with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
Separate MapredWork into MapWork and ReduceWork
---
Key: HIVE-4825
URL: https://issues.apache.org/jira/browse/HIVE-4825
Project: Hive
Issue Type: Improvement
Reporter: Gunther Hagleitner
Assignee: Gunther Hagleitner
Priority: Minor
Attachments: HIVE-4825.1.patch, HIVE-4825.2.code.patch,
HIVE-4825.2.testfiles.patch, HIVE-4825.3.testfiles.patch, HIVE-4825.4.patch,
HIVE-4825.5.patch
Right now all the information needed to run an MR job is captured in
MapredWork. This class has aliases, tagging info, table descriptors etc.
For Tez and MRR it will be useful to break this into map and reduce specific
pieces. The separation is natural and I think has value in itself, it makes
the code easier to understand. However, it will also allow us to reuse these
abstractions in Tez where you'll have a graph of these instead of just 1M and
0-1R.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-4825) Separate MapredWork into MapWork and ReduceWork
[ https://issues.apache.org/jira/browse/HIVE-4825?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gunther Hagleitner updated HIVE-4825: - Attachment: HIVE-4825.6.patch Unfortunately, I missed updating the golden file for the new skip default partition testcase. Patch .6 has the update for the golden file. Separate MapredWork into MapWork and ReduceWork --- Key: HIVE-4825 URL: https://issues.apache.org/jira/browse/HIVE-4825 Project: Hive Issue Type: Improvement Reporter: Gunther Hagleitner Assignee: Gunther Hagleitner Priority: Minor Attachments: HIVE-4825.1.patch, HIVE-4825.2.code.patch, HIVE-4825.2.testfiles.patch, HIVE-4825.3.testfiles.patch, HIVE-4825.4.patch, HIVE-4825.5.patch, HIVE-4825.6.patch Right now all the information needed to run an MR job is captured in MapredWork. This class has aliases, tagging info, table descriptors etc. For Tez and MRR it will be useful to break this into map and reduce specific pieces. The separation is natural and I think has value in itself, it makes the code easier to understand. However, it will also allow us to reuse these abstractions in Tez where you'll have a graph of these instead of just 1M and 0-1R. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-3255) Add DBTokenStore to store Delegation Tokens in DB
[
https://issues.apache.org/jira/browse/HIVE-3255?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gunther Hagleitner updated HIVE-3255:
-
Attachment: HIVE-3255.6.patch
Re-uploading Ashutosh' patch with different name to trigger pre-commit build.
(.6 not _6)
Add DBTokenStore to store Delegation Tokens in DB
-
Key: HIVE-3255
URL: https://issues.apache.org/jira/browse/HIVE-3255
Project: Hive
Issue Type: New Feature
Components: Metastore, Security
Affects Versions: 0.9.0
Reporter: Ashutosh Chauhan
Assignee: Deepesh Khandelwal
Attachments: hive-3255_2.patch, hive-3255_3.patch, hive-3255_4.patch,
hive-3255_5.patch, HIVE-3255_6.patch, HIVE-3255.6.patch, hive-3255.patch
Before HIVE-1696 metastore was completely stateless and all the data is in
backend db. HIVE-1696 added delegation tokens for metastore which metastore
needs to keep in memory. HIVE-2467 added support for storing delegation
tokens in ZooKeeper via an interface {{DelegationTokenStore}}. This jira is
about using DB backend for storing tokens which metastore uses for storing
data.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4051) Hive's metastore suffers from 1+N queries when querying partitions is slow
[
https://issues.apache.org/jira/browse/HIVE-4051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13721584#comment-13721584
]
Hive QA commented on HIVE-4051:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12594511/HIVE-4051.D11805.2.patch
{color:red}ERROR:{color} -1 due to 59 failed/errored test(s), 2653 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ppr_pushdown2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_select_unquote_or
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_index_bitmap_rc
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_insert_into2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_index_auto_partitioned
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_index_compact_2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_select_unquote_and
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_partition_wise_fileformat11
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_explain_dependency2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_view
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_sample10
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_partition_special_char
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_load_dyn_part12
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_index_bitmap_auto_partitioned
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_load_dyn_part3
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_index_compact
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_inputddl7
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_archive_multi
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_mi
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_combine2_hadoop20
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_alter_merge_2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_merge_dynamic_partition3
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_load_dyn_part4
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_columnstats_partlvl
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_index_bitmap
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats4
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_merge3
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_partition_decode_name
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_index_auto_mult_tables
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_partition_wise_fileformat12
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ql_rewrite_gbtoidx
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats_noscan_2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_load_dyn_part10
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats12
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ppd2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby_sort_6
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_index_auto_unused
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_alter_partition_coltype
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_index_stale_partitioned
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucket_groupby
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_explain_dependency
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_concatenate_inherit_table_location
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_index_auth
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_partition_wise_fileformat5
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_input24
org.apache.hcatalog.api.TestHCatClient.testGetPartitionsWithPartialSpec
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_load_dyn_part8
org.apache.hcatalog.api.TestHCatClient.testPartitionsHCatClientImpl
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_explain_logical
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_rename_partition_location
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_select_unquote_not
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_load_dyn_part9
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_updateAccessTime
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats7
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_index_auto_mult_tables_compact
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_database_drop
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ppd_constant_where
org.apache.hcatalog.api.TestHCatClient.testDropPartitionsWithPartialSpec
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/206/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/206/console
Messages:
{noformat}
Executing
[jira] [Commented] (HIVE-3255) Add DBTokenStore to store Delegation Tokens in DB
[
https://issues.apache.org/jira/browse/HIVE-3255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13721617#comment-13721617
]
Hive QA commented on HIVE-3255:
---
{color:green}Overall{color}: +1 all checks pass
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12594528/HIVE-3255.6.patch
{color:green}SUCCESS:{color} +1 2646 tests passed
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/207/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/207/console
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.CleanupPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
Add DBTokenStore to store Delegation Tokens in DB
-
Key: HIVE-3255
URL: https://issues.apache.org/jira/browse/HIVE-3255
Project: Hive
Issue Type: New Feature
Components: Metastore, Security
Affects Versions: 0.9.0
Reporter: Ashutosh Chauhan
Assignee: Deepesh Khandelwal
Attachments: hive-3255_2.patch, hive-3255_3.patch, hive-3255_4.patch,
hive-3255_5.patch, HIVE-3255_6.patch, HIVE-3255.6.patch, hive-3255.patch
Before HIVE-1696 metastore was completely stateless and all the data is in
backend db. HIVE-1696 added delegation tokens for metastore which metastore
needs to keep in memory. HIVE-2467 added support for storing delegation
tokens in ZooKeeper via an interface {{DelegationTokenStore}}. This jira is
about using DB backend for storing tokens which metastore uses for storing
data.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: [Discuss] project chop up
Or feel free to suggest different approach. I am used to managing software as multi-module maven projects. From a development standpoint if I was working on beeline, it would be nice to only require some of the sub-projects to be open in my IDE to do that. Also managing everything globally is not ideal. Hive's project layout, build, and test infrastructure is just funky. It has to do a few interesting things (shims, testing), but I do not think what we are doing justifies the massive ant build system we have. Ant is so ten years ago. On Sat, Jul 27, 2013 at 12:04 AM, Alan Gates ga...@hortonworks.com wrote: But I assume they'd still be a part of targets like package, tar, and binary? Making them compile and test separately and explicitly load the core Hive jars from maven/ivy seems reasonable. Alan. On Jul 26, 2013, at 8:40 PM, Brock Noland wrote: Hi, I think thats part of it but I'd like to decouple the downstream projects even further so that the only connection is the dependency on the hive jars. Brock On Jul 26, 2013 10:10 PM, Alan Gates ga...@hortonworks.com wrote: I'm not sure how this is different from what hcat does today. It needs Hive's jars to compile, so it's one of the last things in the compile step. Would moving the other modules you note to be in the same category be enough? Did you want to also make it so that the default ant target doesn't compile those? Alan. On Jul 26, 2013, at 4:09 PM, Edward Capriolo wrote: My mistake on saying hcat was a fork metastore. I had a brain fart for a moment. One way we could do this is create a folder called downstream. In our release step we can execute the downstream builds and then copy the files we need back. So nothing downstream will be on the classpath of the main project. This could help us breakup ql as well. Things like exotic file formats , and things that are pluggable like zk locking can go here. That might be overkill. For now we can focus on building downstream and hivethrift1might be the first thing to try to downstream. On Friday, July 26, 2013, Thejas Nair the...@hortonworks.com wrote: +1 to the idea of making the build of core hive and other downstream components independent. bq. I was under the impression that Hcat and hive-metastore was supposed to merge up somehow. The metastore code was never forked. Hcat was just using hive-metastore and making the metadata available to rest of hadoop (pig, java MR..). A lot of the changes that were driven by hcat goals were being made in hive-metastore. You can think of hcat as set of libraries that let pig and java MR use hive metastore. Since hcat is closely tied to hive-metastore, it makes sense to have them in same project. On Fri, Jul 26, 2013 at 6:33 AM, Edward Capriolo edlinuxg...@gmail.com wrote: Also i believe hcatalog web can fall into the same designation. Question , hcatalog was initily a big hive-metastore fork. I was under the impression that Hcat and hive-metastore was supposed to merge up somehow. What is the status on that? I remember that was one of the core reasons we brought it in. On Friday, July 26, 2013, Edward Capriolo edlinuxg...@gmail.com wrote: I prefer option 3 as well. On Fri, Jul 26, 2013 at 12:52 AM, Brock Noland br...@cloudera.com wrote: On Thu, Jul 25, 2013 at 9:48 PM, Edward Capriolo edlinuxg...@gmail.com wrote: I have been developing my laptop on a duel core 2 GB Ram laptop for years now. With the addition of hcatalog, hive-thrift2, and some other growth trying to develop hive in a eclipse on this machine craws, especially if 'build automatically' is turned on. As we look to add on more things this is only going to get worse. I am also noticing issues like this: https://issues.apache.org/jira/browse/HIVE-4849 What I think we should do is strip down/out optional parts of hive. 1) Hive Hbase This should really be it's own project to do this right we really have to have multiple branches since hbase is not backwards compatible. 2) Hive Web Interface Now really a big project but not really critical can be just as easily be build separately 3) hive thrift 1 We have hive thrift 2 now, it is time for the sun to set on hivethrift1, 4) odbc Not entirely convinced about this one but it is really not critical to running hive. What I think we should do is create sub-projects for the above things or simply move them into directories that do not build with hive. Ideally they would use maven to pull dependencies. What does everyone think? I agree that projects like the HBase handler and probably others as well should somehow be downstream projects which simply depend on the hive jars. I see a couple alternatives for this: * Take the module in question to the Apache Incubator * Move the
[jira] [Commented] (HIVE-2905) Desc table can't show non-ascii comments
[ https://issues.apache.org/jira/browse/HIVE-2905?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13721641#comment-13721641 ] Edward Capriolo commented on HIVE-2905: --- +1 Desc table can't show non-ascii comments Key: HIVE-2905 URL: https://issues.apache.org/jira/browse/HIVE-2905 Project: Hive Issue Type: Bug Components: CLI Affects Versions: 0.7.0, 0.10.0 Environment: hive 0.7.0, mysql 5.1.45 hive 0.10.0, mysql 5.5.30 Reporter: Sheng Zhou Labels: patch Attachments: HIVE-2905.D11487.1.patch, utf8-desc-comment.patch When desc a table with command line or hive jdbc way, the table's comment can't be read. 1. I have updated javax.jdo.option.ConnectionURL parameter in hive-site.xml file. jdbc:mysql://*.*.*.*:3306/hive?characterEncoding=UTF-8 2. In mysql database, the comment field of COLUMNS table can be read normally. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2905) Desc table can't show non-ascii comments
[ https://issues.apache.org/jira/browse/HIVE-2905?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Edward Capriolo updated HIVE-2905: -- Resolution: Fixed Fix Version/s: 0.12.0 Status: Resolved (was: Patch Available) Thank you everyone involved. Great to see hive adoption in languages I do not understand! Desc table can't show non-ascii comments Key: HIVE-2905 URL: https://issues.apache.org/jira/browse/HIVE-2905 Project: Hive Issue Type: Bug Components: CLI Affects Versions: 0.7.0, 0.10.0 Environment: hive 0.7.0, mysql 5.1.45 hive 0.10.0, mysql 5.5.30 Reporter: Sheng Zhou Assignee: Navis Labels: patch Fix For: 0.12.0 Attachments: HIVE-2905.D11487.1.patch, utf8-desc-comment.patch When desc a table with command line or hive jdbc way, the table's comment can't be read. 1. I have updated javax.jdo.option.ConnectionURL parameter in hive-site.xml file. jdbc:mysql://*.*.*.*:3306/hive?characterEncoding=UTF-8 2. In mysql database, the comment field of COLUMNS table can be read normally. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4586) [HCatalog] WebHCat should return 404 error for undefined resource
[ https://issues.apache.org/jira/browse/HIVE-4586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13721650#comment-13721650 ] Alan Gates commented on HIVE-4586: -- When I run the unit tests on this I see Test org.apache.hcatalog.templeton.TestWebHCatE2e failing. Other than that everything looks good. [HCatalog] WebHCat should return 404 error for undefined resource - Key: HIVE-4586 URL: https://issues.apache.org/jira/browse/HIVE-4586 Project: Hive Issue Type: Bug Components: HCatalog Affects Versions: 0.11.0 Reporter: Daniel Dai Assignee: Daniel Dai Fix For: 0.12.0 Attachments: HIVE-4586-1.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-4551) HCatLoader smallint/tinyint promotions to Int have issues with ORC integration
[
https://issues.apache.org/jira/browse/HIVE-4551?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alan Gates updated HIVE-4551:
-
Resolution: Fixed
Status: Resolved (was: Patch Available)
Patch checked in.
HCatLoader smallint/tinyint promotions to Int have issues with ORC integration
--
Key: HIVE-4551
URL: https://issues.apache.org/jira/browse/HIVE-4551
Project: Hive
Issue Type: Bug
Components: HCatalog
Reporter: Sushanth Sowmyan
Assignee: Sushanth Sowmyan
Attachments: 4551.patch
This was initially reported from an e2e test run, with the following E2E test:
{code}
{
'name' = 'Hadoop_ORC_Write',
'tests' = [
{
'num' = 1
,'hcat_prep'=q\
drop table if exists hadoop_orc;
create table hadoop_orc (
t tinyint,
si smallint,
i int,
b bigint,
f float,
d double,
s string)
stored as orc;\
,'hadoop' = q\
jar :FUNCPATH:/testudf.jar org.apache.hcatalog.utils.WriteText -libjars
:HCAT_JAR: :THRIFTSERVER: all100k hadoop_orc\,
,'result_table' = 'hadoop_orc'
,'sql' = q\select * from all100k;\
,'floatpostprocess' = 1
,'delimiter' = ' '
},
],
},
{code}
This fails with the following error:
{code}
2013-04-26 00:26:07,437 WARN org.apache.hadoop.mapred.Child: Error running
child
org.apache.pig.backend.executionengine.ExecException: ERROR 6018: Error
converting read value to tuple
at
org.apache.hcatalog.pig.HCatBaseLoader.getNext(HCatBaseLoader.java:76)
at org.apache.hcatalog.pig.HCatLoader.getNext(HCatLoader.java:53)
at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigRecordReader.nextKeyValue(PigRecordReader.java:211)
at
org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.nextKeyValue(MapTask.java:532)
at
org.apache.hadoop.mapreduce.MapContext.nextKeyValue(MapContext.java:67)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:143)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:765)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:369)
at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1195)
at org.apache.hadoop.mapred.Child.main(Child.java:249)
Caused by: java.lang.ClassCastException:
org.apache.hadoop.hive.serde2.io.ByteWritable cannot be cast to
org.apache.hadoop.io.IntWritable
at
org.apache.hadoop.hive.serde2.objectinspector.primitive.WritableIntObjectInspector.getPrimitiveJavaObject(WritableIntObjectInspector.java:45)
at
org.apache.hcatalog.data.HCatRecordSerDe.serializePrimitiveField(HCatRecordSerDe.java:290)
at
org.apache.hcatalog.data.HCatRecordSerDe.serializeField(HCatRecordSerDe.java:192)
at org.apache.hcatalog.data.LazyHCatRecord.get(LazyHCatRecord.java:53)
at org.apache.hcatalog.data.LazyHCatRecord.get(LazyHCatRecord.java:97)
at
org.apache.hcatalog.mapreduce.HCatRecordReader.nextKeyValue(HCatRecordReader.java:203)
at
org.apache.hcatalog.pig.HCatBaseLoader.getNext(HCatBaseLoader.java:63)
... 12 more
2013-04-26 00:26:07,440 INFO org.apache.hadoop.mapred.Task: Runnning cleanup
for the task
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2906) Support providing some table properties by user via SQL
[
https://issues.apache.org/jira/browse/HIVE-2906?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13721656#comment-13721656
]
Edward Capriolo commented on HIVE-2906:
---
+1. I think this is a very clean way to handle this. This opens up a lot of
different ways to use input formats and serde
Support providing some table properties by user via SQL
---
Key: HIVE-2906
URL: https://issues.apache.org/jira/browse/HIVE-2906
Project: Hive
Issue Type: Improvement
Components: Query Processor
Reporter: Navis
Assignee: Navis
Attachments: ASF.LICENSE.NOT.GRANTED--HIVE-2906.D2499.1.patch,
ASF.LICENSE.NOT.GRANTED--HIVE-2906.D2499.2.patch,
ASF.LICENSE.NOT.GRANTED--HIVE-2906.D2499.3.patch,
ASF.LICENSE.NOT.GRANTED--HIVE-2906.D2499.4.patch,
ASF.LICENSE.NOT.GRANTED--HIVE-2906.D2499.5.patch, HIVE-2906.D2499.6.patch,
HIVE-2906.D2499.7.patch
Some properties are needed to be provided to StorageHandler by user in
runtime. It might be an address for remote resource or retry count for access
or maximum version count(for hbase), etc.
For example,
{code}
select emp.empno, emp.ename from hbase_emp ('max.version'='3') emp;
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: [Discuss] project chop up
I am open to any approach to making the build and project more modular. Maven is no silver bullet but it does have some big positives. Most importantly I have been a little distressed with Ivy when changing library versions as it caused hundreds of tests to fail for no good reason in HIVE-3632. I have hit this several times on Ivy projects. As has been iterated before offline and eclipse support are built in. IMO alignment with the other Hadoop ecosystem projects is another win. On Sat, Jul 27, 2013 at 9:03 AM, Edward Capriolo edlinuxg...@gmail.comwrote: Or feel free to suggest different approach. I am used to managing software as multi-module maven projects. From a development standpoint if I was working on beeline, it would be nice to only require some of the sub-projects to be open in my IDE to do that. Also managing everything globally is not ideal. Hive's project layout, build, and test infrastructure is just funky. It has to do a few interesting things (shims, testing), but I do not think what we are doing justifies the massive ant build system we have. Ant is so ten years ago. On Sat, Jul 27, 2013 at 12:04 AM, Alan Gates ga...@hortonworks.com wrote: But I assume they'd still be a part of targets like package, tar, and binary? Making them compile and test separately and explicitly load the core Hive jars from maven/ivy seems reasonable. Alan. On Jul 26, 2013, at 8:40 PM, Brock Noland wrote: Hi, I think thats part of it but I'd like to decouple the downstream projects even further so that the only connection is the dependency on the hive jars. Brock On Jul 26, 2013 10:10 PM, Alan Gates ga...@hortonworks.com wrote: I'm not sure how this is different from what hcat does today. It needs Hive's jars to compile, so it's one of the last things in the compile step. Would moving the other modules you note to be in the same category be enough? Did you want to also make it so that the default ant target doesn't compile those? Alan. On Jul 26, 2013, at 4:09 PM, Edward Capriolo wrote: My mistake on saying hcat was a fork metastore. I had a brain fart for a moment. One way we could do this is create a folder called downstream. In our release step we can execute the downstream builds and then copy the files we need back. So nothing downstream will be on the classpath of the main project. This could help us breakup ql as well. Things like exotic file formats , and things that are pluggable like zk locking can go here. That might be overkill. For now we can focus on building downstream and hivethrift1might be the first thing to try to downstream. On Friday, July 26, 2013, Thejas Nair the...@hortonworks.com wrote: +1 to the idea of making the build of core hive and other downstream components independent. bq. I was under the impression that Hcat and hive-metastore was supposed to merge up somehow. The metastore code was never forked. Hcat was just using hive-metastore and making the metadata available to rest of hadoop (pig, java MR..). A lot of the changes that were driven by hcat goals were being made in hive-metastore. You can think of hcat as set of libraries that let pig and java MR use hive metastore. Since hcat is closely tied to hive-metastore, it makes sense to have them in same project. On Fri, Jul 26, 2013 at 6:33 AM, Edward Capriolo edlinuxg...@gmail.com wrote: Also i believe hcatalog web can fall into the same designation. Question , hcatalog was initily a big hive-metastore fork. I was under the impression that Hcat and hive-metastore was supposed to merge up somehow. What is the status on that? I remember that was one of the core reasons we brought it in. On Friday, July 26, 2013, Edward Capriolo edlinuxg...@gmail.com wrote: I prefer option 3 as well. On Fri, Jul 26, 2013 at 12:52 AM, Brock Noland br...@cloudera.com wrote: On Thu, Jul 25, 2013 at 9:48 PM, Edward Capriolo edlinuxg...@gmail.com wrote: I have been developing my laptop on a duel core 2 GB Ram laptop for years now. With the addition of hcatalog, hive-thrift2, and some other growth trying to develop hive in a eclipse on this machine craws, especially if 'build automatically' is turned on. As we look to add on more things this is only going to get worse. I am also noticing issues like this: https://issues.apache.org/jira/browse/HIVE-4849 What I think we should do is strip down/out optional parts of hive. 1) Hive Hbase This should really be it's own project to do this right we really have to have multiple branches since hbase is not backwards compatible. 2) Hive Web Interface Now really a big project but not really critical can be just as easily be build
[jira] [Commented] (HIVE-4947) Add myself to committer list
[ https://issues.apache.org/jira/browse/HIVE-4947?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13721685#comment-13721685 ] Edward Capriolo commented on HIVE-4947: --- +1. Can you please set my company to 'm6d' while you are at it? thanks Add myself to committer list Key: HIVE-4947 URL: https://issues.apache.org/jira/browse/HIVE-4947 Project: Hive Issue Type: Task Reporter: Gunther Hagleitner Assignee: Gunther Hagleitner Priority: Minor Attachments: HIVE-4947.patch Update website to include myself. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4931) MAX/MIN aggregates yield incorrect results
[ https://issues.apache.org/jira/browse/HIVE-4931?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13721686#comment-13721686 ] Edward Capriolo commented on HIVE-4931: --- Why no tests? MAX/MIN aggregates yield incorrect results -- Key: HIVE-4931 URL: https://issues.apache.org/jira/browse/HIVE-4931 Project: Hive Issue Type: Sub-task Components: Query Processor Affects Versions: vectorization-branch Reporter: Remus Rusanu Assignee: Remus Rusanu Priority: Minor Fix For: vectorization-branch Attachments: HIVE-4931.1.patch With the dataset attached by Tony, running: select max(ctinyint), max(csmallint), max(cint), max(cbigint), max(cfloat), max(cdouble) from alltypes_orc; this is the result in rowmode: OK 62 16379 1073737829 2146661184 62.016376.0 This is the result in vector mode: OK 1 16379 1071926788 -106115134 62.016361.0 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Comment Edited] (HIVE-4931) MAX/MIN aggregates yield incorrect results
[ https://issues.apache.org/jira/browse/HIVE-4931?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13721686#comment-13721686 ] Edward Capriolo edited comment on HIVE-4931 at 7/27/13 5:08 PM: Why no tests? What is to prevent a regression? was (Author: appodictic): Why no tests? MAX/MIN aggregates yield incorrect results -- Key: HIVE-4931 URL: https://issues.apache.org/jira/browse/HIVE-4931 Project: Hive Issue Type: Sub-task Components: Query Processor Affects Versions: vectorization-branch Reporter: Remus Rusanu Assignee: Remus Rusanu Priority: Minor Fix For: vectorization-branch Attachments: HIVE-4931.1.patch With the dataset attached by Tony, running: select max(ctinyint), max(csmallint), max(cint), max(cbigint), max(cfloat), max(cdouble) from alltypes_orc; this is the result in rowmode: OK 62 16379 1073737829 2146661184 62.016376.0 This is the result in vector mode: OK 1 16379 1071926788 -106115134 62.016361.0 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4931) MAX/MIN aggregates yield incorrect results
[
https://issues.apache.org/jira/browse/HIVE-4931?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13721688#comment-13721688
]
Edward Capriolo commented on HIVE-4931:
---
{code}
-@Override
+@Override
{code}
And ^ means the last commit on this file went in without check-style.
MAX/MIN aggregates yield incorrect results
--
Key: HIVE-4931
URL: https://issues.apache.org/jira/browse/HIVE-4931
Project: Hive
Issue Type: Sub-task
Components: Query Processor
Affects Versions: vectorization-branch
Reporter: Remus Rusanu
Assignee: Remus Rusanu
Priority: Minor
Fix For: vectorization-branch
Attachments: HIVE-4931.1.patch
With the dataset attached by Tony, running:
select max(ctinyint), max(csmallint), max(cint), max(cbigint), max(cfloat),
max(cdouble) from alltypes_orc;
this is the result in rowmode:
OK
62 16379 1073737829 2146661184 62.016376.0
This is the result in vector mode:
OK
1 16379 1071926788 -106115134 62.016361.0
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Reopened] (HIVE-4931) MAX/MIN aggregates yield incorrect results
[ https://issues.apache.org/jira/browse/HIVE-4931?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Edward Capriolo reopened HIVE-4931: --- MAX/MIN aggregates yield incorrect results -- Key: HIVE-4931 URL: https://issues.apache.org/jira/browse/HIVE-4931 Project: Hive Issue Type: Sub-task Components: Query Processor Affects Versions: vectorization-branch Reporter: Remus Rusanu Assignee: Remus Rusanu Priority: Minor Fix For: vectorization-branch Attachments: HIVE-4931.1.patch With the dataset attached by Tony, running: select max(ctinyint), max(csmallint), max(cint), max(cbigint), max(cfloat), max(cdouble) from alltypes_orc; this is the result in rowmode: OK 62 16379 1073737829 2146661184 62.016376.0 This is the result in vector mode: OK 1 16379 1071926788 -106115134 62.016361.0 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HIVE-4936) PTest increase performance by parallelizing rsync
[ https://issues.apache.org/jira/browse/HIVE-4936?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brock Noland resolved HIVE-4936. Resolution: Duplicate PTest increase performance by parallelizing rsync - Key: HIVE-4936 URL: https://issues.apache.org/jira/browse/HIVE-4936 Project: Hive Issue Type: Bug Reporter: Brock Noland Assignee: Brock Noland Currently the bottom neck in PTest2 when adding hosts is the rsync to the slaves. Today if we have 3 threads which will execute on a slave we rsync the source from the master to the slave 3 times. This causes the process to be CPU bound on the master. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-4920) PTest2 handle Spot Price increases gracefully and improve rsync paralllelsim
[ https://issues.apache.org/jira/browse/HIVE-4920?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brock Noland updated HIVE-4920: --- Description: We should handle spot price increases more gracefully and parallelize rsync to slaves better NO PRECOMMIT TESTS was:We should handle spot price increases more gracefully and parallelize rsync to slaves better PTest2 handle Spot Price increases gracefully and improve rsync paralllelsim Key: HIVE-4920 URL: https://issues.apache.org/jira/browse/HIVE-4920 Project: Hive Issue Type: Improvement Reporter: Brock Noland Assignee: Brock Noland Priority: Critical Attachments: Screen Shot 2013-07-23 at 3.35.00 PM.png We should handle spot price increases more gracefully and parallelize rsync to slaves better NO PRECOMMIT TESTS -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-4920) PTest2 handle Spot Price increases gracefully and improve rsync paralllelsim
[ https://issues.apache.org/jira/browse/HIVE-4920?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brock Noland updated HIVE-4920: --- Description: We should handle spot price increases more gracefully and parallelize rsync to slaves better (was: Today the price for m1.xlarge instances has been varying dramatically. We should fall back on c1.xlarge (which is more powerful and is cheaper at present) and then on on-demand instances.) Summary: PTest2 handle Spot Price increases gracefully and improve rsync paralllelsim (was: PTest2 spot instances should fall back on c1.xlarge and then on-demand instances) PTest2 handle Spot Price increases gracefully and improve rsync paralllelsim Key: HIVE-4920 URL: https://issues.apache.org/jira/browse/HIVE-4920 Project: Hive Issue Type: Improvement Reporter: Brock Noland Assignee: Brock Noland Priority: Critical Attachments: Screen Shot 2013-07-23 at 3.35.00 PM.png We should handle spot price increases more gracefully and parallelize rsync to slaves better -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4920) PTest2 handle Spot Price increases gracefully and improve rsync paralllelsim
[ https://issues.apache.org/jira/browse/HIVE-4920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13721694#comment-13721694 ] Brock Noland commented on HIVE-4920: Also this change allows us to go much wider (16 slaves hosts) with linear increases in test time (2 hours - 1 hour). PTest2 handle Spot Price increases gracefully and improve rsync paralllelsim Key: HIVE-4920 URL: https://issues.apache.org/jira/browse/HIVE-4920 Project: Hive Issue Type: Improvement Reporter: Brock Noland Assignee: Brock Noland Priority: Critical Attachments: HIVE-4920.patch, Screen Shot 2013-07-23 at 3.35.00 PM.png We should handle spot price increases more gracefully and parallelize rsync to slaves better NO PRECOMMIT TESTS -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-4920) PTest2 handle Spot Price increases gracefully and improve rsync paralllelsim
[ https://issues.apache.org/jira/browse/HIVE-4920?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brock Noland updated HIVE-4920: --- Status: Patch Available (was: Open) PTest2 handle Spot Price increases gracefully and improve rsync paralllelsim Key: HIVE-4920 URL: https://issues.apache.org/jira/browse/HIVE-4920 Project: Hive Issue Type: Improvement Reporter: Brock Noland Assignee: Brock Noland Priority: Critical Attachments: HIVE-4920.patch, Screen Shot 2013-07-23 at 3.35.00 PM.png We should handle spot price increases more gracefully and parallelize rsync to slaves better NO PRECOMMIT TESTS -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-4920) PTest2 handle Spot Price increases gracefully and improve rsync paralllelsim
[ https://issues.apache.org/jira/browse/HIVE-4920?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brock Noland updated HIVE-4920: --- Attachment: HIVE-4920.patch Alright last non-critical fix to PTest2 :) I have tested this code a number of times including terminating all spot instances during a test and it worked quite well. PTest2 handle Spot Price increases gracefully and improve rsync paralllelsim Key: HIVE-4920 URL: https://issues.apache.org/jira/browse/HIVE-4920 Project: Hive Issue Type: Improvement Reporter: Brock Noland Assignee: Brock Noland Priority: Critical Attachments: HIVE-4920.patch, Screen Shot 2013-07-23 at 3.35.00 PM.png We should handle spot price increases more gracefully and parallelize rsync to slaves better NO PRECOMMIT TESTS -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4051) Hive's metastore suffers from 1+N queries when querying partitions is slow
[
https://issues.apache.org/jira/browse/HIVE-4051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13721696#comment-13721696
]
Phabricator commented on HIVE-4051:
---
brock has commented on the revision HIVE-4051 [jira] Hive's metastore suffers
from 1+N queries when querying partitions is slow.
Comments about handling of error.
INLINE COMMENTS
metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java:2074 In
general I am supportive of this patch but we should not be catching Throwable
here. We should at most catch Exceptuon.
Additionally log4j now supports passing exceptions directly, the
stringifyException is no longer needed, you can pass the exception as the
second arg of the log method.
REVISION DETAIL
https://reviews.facebook.net/D11805
To: JIRA, sershe
Cc: brock
Hive's metastore suffers from 1+N queries when querying partitions is slow
Key: HIVE-4051
URL: https://issues.apache.org/jira/browse/HIVE-4051
Project: Hive
Issue Type: Bug
Components: Clients, Metastore
Environment: RHEL 6.3 / EC2 C1.XL
Reporter: Gopal V
Assignee: Sergey Shelukhin
Attachments: HIVE-4051.D11805.1.patch, HIVE-4051.D11805.2.patch
Hive's query client takes a long time to initialize start planning queries
because of delays in creating all the MTable/MPartition objects.
For a hive db with 1800 partitions, the metastore took 6-7 seconds to
initialize - firing approximately 5900 queries to the mysql database.
Several of those queries fetch exactly one row to create a single object on
the client.
The following 12 queries were repeated for each partition, generating a storm
of SQL queries
{code}
4 Query SELECT
`A0`.`SD_ID`,`B0`.`INPUT_FORMAT`,`B0`.`IS_COMPRESSED`,`B0`.`IS_STOREDASSUBDIRECTORIES`,`B0`.`LOCATION`,`B0`.`NUM_BUCKETS`,`B0`.`OUTPUT_FORMAT`,`B0`.`SD_ID`
FROM `PARTITIONS` `A0` LEFT OUTER JOIN `SDS` `B0` ON `A0`.`SD_ID` =
`B0`.`SD_ID` WHERE `A0`.`PART_ID` = 3945
4 Query SELECT `A0`.`CD_ID`,`B0`.`CD_ID` FROM `SDS` `A0` LEFT OUTER JOIN
`CDS` `B0` ON `A0`.`CD_ID` = `B0`.`CD_ID` WHERE `A0`.`SD_ID` =4871
4 Query SELECT COUNT(*) FROM `COLUMNS_V2` THIS WHERE THIS.`CD_ID`=1546
AND THIS.`INTEGER_IDX`=0
4 Query SELECT
`A0`.`COMMENT`,`A0`.`COLUMN_NAME`,`A0`.`TYPE_NAME`,`A0`.`INTEGER_IDX` AS
NUCORDER0 FROM `COLUMNS_V2` `A0` WHERE `A0`.`CD_ID` = 1546 AND
`A0`.`INTEGER_IDX` = 0 ORDER BY NUCORDER0
4 Query SELECT `A0`.`SERDE_ID`,`B0`.`NAME`,`B0`.`SLIB`,`B0`.`SERDE_ID`
FROM `SDS` `A0` LEFT OUTER JOIN `SERDES` `B0` ON `A0`.`SERDE_ID` =
`B0`.`SERDE_ID` WHERE `A0`.`SD_ID` =4871
4 Query SELECT COUNT(*) FROM `SORT_COLS` THIS WHERE THIS.`SD_ID`=4871 AND
THIS.`INTEGER_IDX`=0
4 Query SELECT `A0`.`COLUMN_NAME`,`A0`.`ORDER`,`A0`.`INTEGER_IDX` AS
NUCORDER0 FROM `SORT_COLS` `A0` WHERE `A0`.`SD_ID` =4871 AND
`A0`.`INTEGER_IDX` = 0 ORDER BY NUCORDER0
4 Query SELECT COUNT(*) FROM `SKEWED_VALUES` THIS WHERE
THIS.`SD_ID_OID`=4871 AND THIS.`INTEGER_IDX`=0
4 Query SELECT 'org.apache.hadoop.hive.metastore.model.MStringList' AS
NUCLEUS_TYPE,`A1`.`STRING_LIST_ID`,`A0`.`INTEGER_IDX` AS NUCORDER0 FROM
`SKEWED_VALUES` `A0` INNER JOIN `SKEWED_STRING_LIST` `A1` ON
`A0`.`STRING_LIST_ID_EID` = `A1`.`STRING_LIST_ID` WHERE `A0`.`SD_ID_OID`
=4871 AND `A0`.`INTEGER_IDX` = 0 ORDER BY NUCORDER0
4 Query SELECT COUNT(*) FROM `SKEWED_COL_VALUE_LOC_MAP` WHERE `SD_ID`
=4871 AND `STRING_LIST_ID_KID` IS NOT NULL
4 Query SELECT 'org.apache.hadoop.hive.metastore.model.MStringList' AS
NUCLEUS_TYPE,`A0`.`STRING_LIST_ID` FROM `SKEWED_STRING_LIST` `A0` INNER JOIN
`SKEWED_COL_VALUE_LOC_MAP` `B0` ON `A0`.`STRING_LIST_ID` =
`B0`.`STRING_LIST_ID_KID` WHERE `B0`.`SD_ID` =4871
4 Query SELECT `A0`.`STRING_LIST_ID_KID`,`A0`.`LOCATION` FROM
`SKEWED_COL_VALUE_LOC_MAP` `A0` WHERE `A0`.`SD_ID` =4871 AND NOT
(`A0`.`STRING_LIST_ID_KID` IS NULL)
{code}
This data is not detached or cached, so this operation is performed during
every query plan for the partitions, even in the same hive client.
The queries are automatically generated by JDO/DataNucleus which makes it
nearly impossible to rewrite it into a single denormalized join operation
process it locally.
Attempts to optimize this with JDO fetch-groups did not bear fruit in
improving the query count.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HIVE-4338) Create Table fails after upgrade from 0.9 to 0.10
[ https://issues.apache.org/jira/browse/HIVE-4338?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brock Noland resolved HIVE-4338. Resolution: Not A Problem Agreed with Harsh J, it looks like the upgrade script was not run. Closing. Create Table fails after upgrade from 0.9 to 0.10 - Key: HIVE-4338 URL: https://issues.apache.org/jira/browse/HIVE-4338 Project: Hive Issue Type: Bug Components: Database/Schema Affects Versions: 0.10.0 Environment: Ubuntu 3.2.0-23-generic #36-Ubuntu on AMD Reporter: Geula Vainappel I ran apt-get upgrade on a relatively old cdh installation. Many things were upgraded, among them hadoop, hdfs and hive (from 0.9 to 0.10). After the upgrade, CREATE TABLE started failing. I rebooted the machine, and it is still not working. The error I am receiving is: hive create table ttt(line string); FAILED: Error in metadata: MetaException(message:javax.jdo.JDODataStoreException: Error(s) were found while auto-creating/validating the datastore for classes. The errors are printed in the log, and are attached to this exception. NestedThrowables: java.sql.SQLSyntaxErrorException: In an ALTER TABLE statement, the column 'IS_STOREDASSUBDIRECTORIES' has been specified as NOT NULL and either the DEFAULT clause was not specified or was specified as DEFAULT NULL.) FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4941) PTest2 Investigate Ignores
[ https://issues.apache.org/jira/browse/HIVE-4941?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13721697#comment-13721697 ] Brock Noland commented on HIVE-4941: TestJdbcDriver2 ran just fine, I have enabled it. PTest2 Investigate Ignores -- Key: HIVE-4941 URL: https://issues.apache.org/jira/browse/HIVE-4941 Project: Hive Issue Type: Task Reporter: Brock Noland Assignee: Brock Noland Priority: Minor Currently we excluding the following tests: unitTests.exclude = TestHiveMetaStore TestSerDe TestBeeLineDriver TestHiveServer2Concurrency TestJdbcDriver2 TestHiveServer2Concurrency TestBeeLineDriver some of them we got from the build files but I am not sure about TestJdbcDriver2 for example. We should investigate why these are excluded. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-4947) Add myself to committer list
[ https://issues.apache.org/jira/browse/HIVE-4947?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gunther Hagleitner updated HIVE-4947: - Attachment: HIVE-4947.2.patch Sure. Added m6d as your organization. Add myself to committer list Key: HIVE-4947 URL: https://issues.apache.org/jira/browse/HIVE-4947 Project: Hive Issue Type: Task Reporter: Gunther Hagleitner Assignee: Gunther Hagleitner Priority: Minor Attachments: HIVE-4947.2.patch, HIVE-4947.patch Update website to include myself. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4947) Add myself to committer list
[ https://issues.apache.org/jira/browse/HIVE-4947?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13721700#comment-13721700 ] Gunther Hagleitner commented on HIVE-4947: -- [~brocknoland] I've added you too, since I didn't see you in there. Can you confirm I got the entry right? Add myself to committer list Key: HIVE-4947 URL: https://issues.apache.org/jira/browse/HIVE-4947 Project: Hive Issue Type: Task Reporter: Gunther Hagleitner Assignee: Gunther Hagleitner Priority: Minor Attachments: HIVE-4947.2.patch, HIVE-4947.patch Update website to include myself. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: [Discuss] project chop up
Yes hive build and test cases got convoluted as the project scope gradually increased. This is the time to take action! Based on my other Apache experiences, I prefer the option #3 Breakup the projects within our own source tree. Make multiple modules or sub-projects. By default, only key modules will be built. Maven could be a possible candidate. Regards, Mohammad From: Edward Capriolo edlinuxg...@gmail.com To: dev@hive.apache.org dev@hive.apache.org Sent: Saturday, July 27, 2013 7:03 AM Subject: Re: [Discuss] project chop up Or feel free to suggest different approach. I am used to managing software as multi-module maven projects. From a development standpoint if I was working on beeline, it would be nice to only require some of the sub-projects to be open in my IDE to do that. Also managing everything globally is not ideal. Hive's project layout, build, and test infrastructure is just funky. It has to do a few interesting things (shims, testing), but I do not think what we are doing justifies the massive ant build system we have. Ant is so ten years ago. On Sat, Jul 27, 2013 at 12:04 AM, Alan Gates ga...@hortonworks.com wrote: But I assume they'd still be a part of targets like package, tar, and binary? Making them compile and test separately and explicitly load the core Hive jars from maven/ivy seems reasonable. Alan. On Jul 26, 2013, at 8:40 PM, Brock Noland wrote: Hi, I think thats part of it but I'd like to decouple the downstream projects even further so that the only connection is the dependency on the hive jars. Brock On Jul 26, 2013 10:10 PM, Alan Gates ga...@hortonworks.com wrote: I'm not sure how this is different from what hcat does today. It needs Hive's jars to compile, so it's one of the last things in the compile step. Would moving the other modules you note to be in the same category be enough? Did you want to also make it so that the default ant target doesn't compile those? Alan. On Jul 26, 2013, at 4:09 PM, Edward Capriolo wrote: My mistake on saying hcat was a fork metastore. I had a brain fart for a moment. One way we could do this is create a folder called downstream. In our release step we can execute the downstream builds and then copy the files we need back. So nothing downstream will be on the classpath of the main project. This could help us breakup ql as well. Things like exotic file formats , and things that are pluggable like zk locking can go here. That might be overkill. For now we can focus on building downstream and hivethrift1might be the first thing to try to downstream. On Friday, July 26, 2013, Thejas Nair the...@hortonworks.com wrote: +1 to the idea of making the build of core hive and other downstream components independent. bq. I was under the impression that Hcat and hive-metastore was supposed to merge up somehow. The metastore code was never forked. Hcat was just using hive-metastore and making the metadata available to rest of hadoop (pig, java MR..). A lot of the changes that were driven by hcat goals were being made in hive-metastore. You can think of hcat as set of libraries that let pig and java MR use hive metastore. Since hcat is closely tied to hive-metastore, it makes sense to have them in same project. On Fri, Jul 26, 2013 at 6:33 AM, Edward Capriolo edlinuxg...@gmail.com wrote: Also i believe hcatalog web can fall into the same designation. Question , hcatalog was initily a big hive-metastore fork. I was under the impression that Hcat and hive-metastore was supposed to merge up somehow. What is the status on that? I remember that was one of the core reasons we brought it in. On Friday, July 26, 2013, Edward Capriolo edlinuxg...@gmail.com wrote: I prefer option 3 as well. On Fri, Jul 26, 2013 at 12:52 AM, Brock Noland br...@cloudera.com wrote: On Thu, Jul 25, 2013 at 9:48 PM, Edward Capriolo edlinuxg...@gmail.com wrote: I have been developing my laptop on a duel core 2 GB Ram laptop for years now. With the addition of hcatalog, hive-thrift2, and some other growth trying to develop hive in a eclipse on this machine craws, especially if 'build automatically' is turned on. As we look to add on more things this is only going to get worse. I am also noticing issues like this: https://issues.apache.org/jira/browse/HIVE-4849 What I think we should do is strip down/out optional parts of hive. 1) Hive Hbase This should really be it's own project to do this right we really have to have multiple branches since hbase is not backwards compatible. 2) Hive Web Interface Now really a big project but not really critical can be just as easily be build separately 3) hive thrift 1 We have hive thrift 2 now, it is time for the sun to set on hivethrift1, 4) odbc Not entirely
[jira] [Updated] (HIVE-3325) serde-reported partition cols should not be persisted in metastore
[ https://issues.apache.org/jira/browse/HIVE-3325?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Edward Capriolo updated HIVE-3325: -- Resolution: Fixed Status: Resolved (was: Patch Available) This was committed on r1505418 | ecapriolo | 2013-07-21 12:50:25 -0400 (Sun, 21 Jul 2013) | 5 lines Serde-reported partition cols should not be persisted in metastore (Travis Crawford via egc) Sorry that I did not close out this ticket already. Thanks Travis. serde-reported partition cols should not be persisted in metastore -- Key: HIVE-3325 URL: https://issues.apache.org/jira/browse/HIVE-3325 Project: Hive Issue Type: Bug Components: Metastore Affects Versions: 0.10.0 Reporter: Travis Crawford Assignee: Travis Crawford Attachments: HIVE-3325.patch.txt, HIVE-3325_serde_reported_part_cols.1.patch In HIVE-3279 an issue was fixed where serde-reported columns were not added to partitions. However, the fix in that issue caused serde-reported columns to be stored in the partition storage descriptor. Serde-reported columns should be dynamic and only reported at runtime (not stored in the partition storage descriptor). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4920) PTest2 handle Spot Price increases gracefully and improve rsync paralllelsim
[ https://issues.apache.org/jira/browse/HIVE-4920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13721732#comment-13721732 ] Edward Capriolo commented on HIVE-4920: --- +1. I will review a bit more and commit in 24 hours. PTest2 handle Spot Price increases gracefully and improve rsync paralllelsim Key: HIVE-4920 URL: https://issues.apache.org/jira/browse/HIVE-4920 Project: Hive Issue Type: Improvement Reporter: Brock Noland Assignee: Brock Noland Priority: Critical Attachments: HIVE-4920.patch, Screen Shot 2013-07-23 at 3.35.00 PM.png We should handle spot price increases more gracefully and parallelize rsync to slaves better NO PRECOMMIT TESTS -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HIVE-4299) exported metadata by HIVE-3068 cannot be imported because of wrong file name
[
https://issues.apache.org/jira/browse/HIVE-4299?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Edward Capriolo reassigned HIVE-4299:
-
Assignee: Edward Capriolo (was: Sho Shimauchi)
exported metadata by HIVE-3068 cannot be imported because of wrong file name
Key: HIVE-4299
URL: https://issues.apache.org/jira/browse/HIVE-4299
Project: Hive
Issue Type: Bug
Components: Query Processor
Affects Versions: 0.11.0
Reporter: Sho Shimauchi
Assignee: Edward Capriolo
Attachments: HIVE-4299.1.patch.txt, HIVE-4299.patch
h2. Symptom
When DROP TABLE a table, metadata of the table is generated to be able to
import the dropped table again.
However, the exported metadata name is 'table name.metadata'.
Since ImportSemanticAnalyzer allows only '_metadata' as metadata filename,
user have to rename the metadata file to import the table.
h2. How to reproduce
Set the following setting to hive-site.xml:
{code}
property
namehive.metastore.pre.event.listeners/name
valueorg.apache.hadoop.hive.ql.parse.MetaDataExportListener/value
/property
{code}
Then run the following queries:
{code}
CREATE TABLE test_table (id INT, name STRING);
DROP TABLE test_table;
IMPORT TABLE test_table_imported FROM '/path/to/metadata/file';
FAILED: SemanticException [Error 10027]: Invalid path
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-4299) exported metadata by HIVE-3068 cannot be imported because of wrong file name
[
https://issues.apache.org/jira/browse/HIVE-4299?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Edward Capriolo updated HIVE-4299:
--
Attachment: HIVE-4299.4.patch.txt
Changed name to . from _. Also date format is not thread safe, moved it inside
the method just to be safe.
exported metadata by HIVE-3068 cannot be imported because of wrong file name
Key: HIVE-4299
URL: https://issues.apache.org/jira/browse/HIVE-4299
Project: Hive
Issue Type: Bug
Components: Query Processor
Affects Versions: 0.11.0
Reporter: Sho Shimauchi
Assignee: Edward Capriolo
Attachments: HIVE-4299.1.patch.txt, HIVE-4299.4.patch.txt,
HIVE-4299.patch
h2. Symptom
When DROP TABLE a table, metadata of the table is generated to be able to
import the dropped table again.
However, the exported metadata name is 'table name.metadata'.
Since ImportSemanticAnalyzer allows only '_metadata' as metadata filename,
user have to rename the metadata file to import the table.
h2. How to reproduce
Set the following setting to hive-site.xml:
{code}
property
namehive.metastore.pre.event.listeners/name
valueorg.apache.hadoop.hive.ql.parse.MetaDataExportListener/value
/property
{code}
Then run the following queries:
{code}
CREATE TABLE test_table (id INT, name STRING);
DROP TABLE test_table;
IMPORT TABLE test_table_imported FROM '/path/to/metadata/file';
FAILED: SemanticException [Error 10027]: Invalid path
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
hive windowing should not have been committed in it's current state
I was looking through the function registery today and I noticed this:
UDAFS_IMPLY_ORDER.add(rank);
UDAFS_IMPLY_ORDER.add(dense_rank);
UDAFS_IMPLY_ORDER.add(percent_rank);
UDAFS_IMPLY_ORDER.add(cume_dist);
UDAFS_IMPLY_ORDER.add(LEAD_FUNC_NAME);
UDAFS_IMPLY_ORDER.add(LAG_FUNC_NAME);
UDAFS_IMPLY_ORDER.add(first_value);
UDAFS_IMPLY_ORDER.add(last_value);
There are several things to point out first.
1) I can not follow why some method names are constant and some are not.
2) A hard coded list of UDAFS_IMPLY_ORDER seems to indicate that a window
function of this type can not be added.
3) Every other container in function registry is a synchronized collection,
but this is not
- /*
- * UDAFS that only work when the input rows have an order.
- */
- public static final HashSetString UDAFS_IMPLY_ORDER = new
HashSetString();
4) Our style rules were not followed '{' sometimes on the line
sometimes tab, sometimes two spaces, sometimes 8 tabs,
public static void registerTableFunction(String name, Class? extends
TableFunctionResolver tFnCls)
{
FunctionInfo tInfo = new FunctionInfo(name, tFnCls);
mFunctions.put(name.toLowerCase(), tInfo);
}
I have opened up https://issues.apache.org/jira/browse/HIVE-4879 to address
these things.
Personally I am not a crazy stickler for check-style and I rarely kick a
patch back at someone for a couple of style violations, but this really
needed the once over. Especially since we just spent a lot of effort
cleaning up un thread safe things so hiveserver2 can be happy.
Keep in mind that patches with flagrant check style violations, no tests,
and little javadoc should be a no go.
I am adding an annotation for the classes so we do not need to maintain a
static list.
Edward
[jira] [Commented] (HIVE-4586) [HCatalog] WebHCat should return 404 error for undefined resource
[ https://issues.apache.org/jira/browse/HIVE-4586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13721754#comment-13721754 ] Eugene Koifman commented on HIVE-4586: -- That test suite was added after Daniel made the patch [HCatalog] WebHCat should return 404 error for undefined resource - Key: HIVE-4586 URL: https://issues.apache.org/jira/browse/HIVE-4586 Project: Hive Issue Type: Bug Components: HCatalog Affects Versions: 0.11.0 Reporter: Daniel Dai Assignee: Daniel Dai Fix For: 0.12.0 Attachments: HIVE-4586-1.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HIVE-4055) add Date data type
[ https://issues.apache.org/jira/browse/HIVE-4055?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan resolved HIVE-4055. Resolution: Fixed Fix Version/s: 0.12.0 Committed to trunk. Thanks, Sun for initial work and Jason for following it up to see through this completion. add Date data type -- Key: HIVE-4055 URL: https://issues.apache.org/jira/browse/HIVE-4055 Project: Hive Issue Type: Sub-task Components: JDBC, Query Processor, Serializers/Deserializers, UDF Reporter: Sun Rui Assignee: Jason Dere Fix For: 0.12.0 Attachments: Date.pdf, HIVE-4055.1.patch.txt, HIVE-4055.2.patch.txt, HIVE-4055.3.patch.txt, HIVE-4055.4.patch, HIVE-4055.4.patch.txt, HIVE-4055.D11547.1.patch Add Date data type, a new primitive data type which supports the standard SQL date type. Basically, the implementation can take HIVE-2272 and HIVE-2957 as references. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4299) exported metadata by HIVE-3068 cannot be imported because of wrong file name
[
https://issues.apache.org/jira/browse/HIVE-4299?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13721768#comment-13721768
]
Hive QA commented on HIVE-4299:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12594554/HIVE-4299.4.patch.txt
{color:red}ERROR:{color} -1 due to 50 failed/errored test(s), 2723 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_exim_02_all_part_over_overlap
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_exim_09_part_spec_nonoverlap
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_exim_23_import_exist_authfail
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_exim_04_nonpart_noncompat_colnumber
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_exim_00_nonpart_empty
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_exim_17_part_spec_underspec
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_exim_16_part_noncompat_schema
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_exim_18_part_spec_missing
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_exim_16_part_external
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_exim_06_nonpart_noncompat_storage
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_exim_17_part_managed
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_exim_10_nonpart_noncompat_bucketing
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_exim_19_00_part_external_location
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_exim_23_import_part_authsuccess
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_exim_13_nonnative_import
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_exim_10_external_managed
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_exim_02_part
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_exim_22_import_exist_authsuccess
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_exim_25_import_nonexist_authfail
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_exim_14_nonpart_part
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_exim_07_all_part_over_nonoverlap
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_exim_14_managed_location_over_existing
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_exim_12_external_location
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_exim_02_00_part_empty
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_exim_08_nonpart_noncompat_serde
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_exim_09_nonpart_noncompat_serdeparam
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_exim_24_import_part_authfail
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_exim_20_managed_location_over_existing
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_exim_19_part_external_location
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_exim_15_part_nonpart
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_exim_03_nonpart_noncompat_colschema
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_exim_07_nonpart_noncompat_ifof
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_exim_11_nonpart_noncompat_sorting
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_exim_20_part_managed_location
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_exim_03_nonpart_over_compat
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_exim_01_nonpart_over_loaded
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_exim_04_evolved_parts
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_exim_21_part_managed_external
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_exim_05_nonpart_noncompat_coltype
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_exim_18_part_external
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_exim_04_all_part
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_exim_06_one_part
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_exim_24_import_nonexist_authsuccess
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_exim_01_nonpart
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_exim_13_managed_location
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_exim_11_managed_external
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_exim_15_external_part
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_exim_08_nonpart_rename
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_exim_05_some_part
[jira] [Commented] (HIVE-4941) PTest2 Investigate Ignores
[ https://issues.apache.org/jira/browse/HIVE-4941?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13721772#comment-13721772 ] Ashutosh Chauhan commented on HIVE-4941: In recent runs, I noticed Hive QA runs around ~2653 tests while when I run my tests old style (ie via ant test) it reports running ~2692 tests. It will be good to verify the same number of tests are run in either cases. PTest2 Investigate Ignores -- Key: HIVE-4941 URL: https://issues.apache.org/jira/browse/HIVE-4941 Project: Hive Issue Type: Task Reporter: Brock Noland Assignee: Brock Noland Priority: Minor Currently we excluding the following tests: unitTests.exclude = TestHiveMetaStore TestSerDe TestBeeLineDriver TestHiveServer2Concurrency TestJdbcDriver2 TestHiveServer2Concurrency TestBeeLineDriver some of them we got from the build files but I am not sure about TestJdbcDriver2 for example. We should investigate why these are excluded. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4941) PTest2 Investigate Ignores
[ https://issues.apache.org/jira/browse/HIVE-4941?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13721775#comment-13721775 ] Brock Noland commented on HIVE-4941: Will do PTest2 Investigate Ignores -- Key: HIVE-4941 URL: https://issues.apache.org/jira/browse/HIVE-4941 Project: Hive Issue Type: Task Reporter: Brock Noland Assignee: Brock Noland Priority: Minor Currently we excluding the following tests: unitTests.exclude = TestHiveMetaStore TestSerDe TestBeeLineDriver TestHiveServer2Concurrency TestJdbcDriver2 TestHiveServer2Concurrency TestBeeLineDriver some of them we got from the build files but I am not sure about TestJdbcDriver2 for example. We should investigate why these are excluded. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4299) exported metadata by HIVE-3068 cannot be imported because of wrong file name
[
https://issues.apache.org/jira/browse/HIVE-4299?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13721774#comment-13721774
]
Brock Noland commented on HIVE-4299:
At least one failure is:
{noformat}
[junit] junit.framework.AssertionFailedError: Client Execution failed with
error code = 10027
{noformat}
which is invalid path.
exported metadata by HIVE-3068 cannot be imported because of wrong file name
Key: HIVE-4299
URL: https://issues.apache.org/jira/browse/HIVE-4299
Project: Hive
Issue Type: Bug
Components: Query Processor
Affects Versions: 0.11.0
Reporter: Sho Shimauchi
Assignee: Edward Capriolo
Attachments: HIVE-4299.1.patch.txt, HIVE-4299.4.patch.txt,
HIVE-4299.patch
h2. Symptom
When DROP TABLE a table, metadata of the table is generated to be able to
import the dropped table again.
However, the exported metadata name is 'table name.metadata'.
Since ImportSemanticAnalyzer allows only '_metadata' as metadata filename,
user have to rename the metadata file to import the table.
h2. How to reproduce
Set the following setting to hive-site.xml:
{code}
property
namehive.metastore.pre.event.listeners/name
valueorg.apache.hadoop.hive.ql.parse.MetaDataExportListener/value
/property
{code}
Then run the following queries:
{code}
CREATE TABLE test_table (id INT, name STRING);
DROP TABLE test_table;
IMPORT TABLE test_table_imported FROM '/path/to/metadata/file';
FAILED: SemanticException [Error 10027]: Invalid path
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-4929) the type of all numeric constants is changed to double in the plan
[ https://issues.apache.org/jira/browse/HIVE-4929?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-4929: --- Resolution: Fixed Fix Version/s: 0.12.0 Status: Resolved (was: Patch Available) Committed to trunk. Thanks, Sergey! the type of all numeric constants is changed to double in the plan -- Key: HIVE-4929 URL: https://issues.apache.org/jira/browse/HIVE-4929 Project: Hive Issue Type: Bug Reporter: Sergey Shelukhin Assignee: Sergey Shelukhin Fix For: 0.12.0 Attachments: HIVE-4929.patch There's code which, after the numeric type for a constant in where clause has been chosen as the most restricted one or based on suffix, tries to change the type to match the numeric column which the constant is being compared with. However, due to a hack from HIVE-3059 every column type shows up as string in that code, causing it to always change the constant type to double. This should not be done (regardless of the hack). Spinoff from HIVE-2702, large number of query outputs change so it will be a big patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4825) Separate MapredWork into MapWork and ReduceWork
[ https://issues.apache.org/jira/browse/HIVE-4825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13721779#comment-13721779 ] Ashutosh Chauhan commented on HIVE-4825: +1 Separate MapredWork into MapWork and ReduceWork --- Key: HIVE-4825 URL: https://issues.apache.org/jira/browse/HIVE-4825 Project: Hive Issue Type: Improvement Reporter: Gunther Hagleitner Assignee: Gunther Hagleitner Priority: Minor Attachments: HIVE-4825.1.patch, HIVE-4825.2.code.patch, HIVE-4825.2.testfiles.patch, HIVE-4825.3.testfiles.patch, HIVE-4825.4.patch, HIVE-4825.5.patch, HIVE-4825.6.patch Right now all the information needed to run an MR job is captured in MapredWork. This class has aliases, tagging info, table descriptors etc. For Tez and MRR it will be useful to break this into map and reduce specific pieces. The separation is natural and I think has value in itself, it makes the code easier to understand. However, it will also allow us to reuse these abstractions in Tez where you'll have a graph of these instead of just 1M and 0-1R. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4520) java.lang.NegativeArraySizeException when query on hive-0.11.0, hbase-0.94.6.1
[ https://issues.apache.org/jira/browse/HIVE-4520?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13721783#comment-13721783 ] Swarnim Kulkarni commented on HIVE-4520: [~mayanhui] This bug seems to be a duplicate of HIVE-4515. Do you agree to resolve this one as duplicate? java.lang.NegativeArraySizeException when query on hive-0.11.0, hbase-0.94.6.1 -- Key: HIVE-4520 URL: https://issues.apache.org/jira/browse/HIVE-4520 Project: Hive Issue Type: Bug Components: HBase Handler Affects Versions: 0.11.0 Environment: hive-0.11.0 hbase-0.94.6.1 zookeeper-3.4.3 hadoop-1.0.4 centos-5.7 Reporter: Yanhui Ma Priority: Critical After integration hive-0.11.0+hbase-0.94.6.1, these commands could be executed sucessfully: create table insert overwrite table select * from table However, when execute select count(*) from table, throws exception: hive select count(*) from test; Total MapReduce jobs = 1 Launching Job 1 out of 1 Number of reduce tasks determined at compile time: 1 In order to change the average load for a reducer (in bytes): set hive.exec.reducers.bytes.per.reducer=number In order to limit the maximum number of reducers: set hive.exec.reducers.max=number In order to set a constant number of reducers: set mapred.reduce.tasks=number Starting Job = job_201305061042_0028, Tracking URL = http://master0:50030/jobdetails.jsp?jobid=job_201305061042_0028 Kill Command = /opt/modules/hadoop/hadoop-1.0.4/libexec/../bin/hadoop job -kill job_201305061042_0028 Hadoop job information for Stage-1: number of mappers: 1; number of reducers: 1 2013-05-07 18:41:42,649 Stage-1 map = 0%, reduce = 0% 2013-05-07 18:42:14,789 Stage-1 map = 100%, reduce = 100% Ended Job = job_201305061042_0028 with errors Error during job, obtaining debugging information... Job Tracking URL: http://master0:50030/jobdetails.jsp?jobid=job_201305061042_0028 Examining task ID: task_201305061042_0028_m_02 (and more) from job job_201305061042_0028 Task with the most failures(4): - Task ID: task_201305061042_0028_m_00 URL: http://master0:50030/taskdetails.jsp?jobid=job_201305061042_0028tipid=task_201305061042_0028_m_00 - Diagnostic Messages for this Task: java.lang.NegativeArraySizeException: -1 at org.apache.hadoop.hbase.util.Bytes.readByteArray(Bytes.java:148) at org.apache.hadoop.hbase.mapreduce.TableSplit.readFields(TableSplit.java:133) at org.apache.hadoop.hive.hbase.HBaseSplit.readFields(HBaseSplit.java:53) at org.apache.hadoop.hive.ql.io.HiveInputFormat$HiveInputSplit.readFields(HiveInputFormat.java:150) at org.apache.hadoop.io.serializer.WritableSerialization$WritableDeserializer.deserialize(WritableSerialization.java:67) at org.apache.hadoop.io.serializer.WritableSerialization$WritableDeserializer.deserialize(WritableSerialization.java:40) at org.apache.hadoop.mapred.MapTask.getSplitDetails(MapTask.java:396) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:412) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:372) at org.apache.hadoop.mapred.Child$4.run(Child.java:255) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1121) at org.apache.hadoop.mapred.Child.main(Child.java:249) FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.MapRedTask MapReduce Jobs Launched: Job 0: Map: 1 Reduce: 1 HDFS Read: 0 HDFS Write: 0 FAIL Total MapReduce CPU Time Spent: 0 msec == The log of tasktracker: stderr logs 13/05/07 18:43:20 INFO util.NativeCodeLoader: Loaded the native-hadoop library 13/05/07 18:43:20 INFO mapred.TaskRunner: Creating symlink: /tmp/hadoop-hadoop/mapred/local/taskTracker/distcache/107328478296390_-1298160740_2123690974/master0/tmp/hive-hadoop/hive_2013-05-07_18-41-30_290_832140779606816147/-mr-10003/fd22448b-e923-498c-bc00-2164ca68447d - /tmp/hadoop-hadoop/mapred/local/taskTracker/hadoop/jobcache/job_201305061042_0028/attempt_201305061042_0028_m_00_0/work/HIVE_PLANfd22448b-e923-498c-bc00-2164ca68447d 13/05/07 18:43:20 INFO filecache.TrackerDistributedCacheManager: Creating symlink: /tmp/hadoop-hadoop/mapred/local/taskTracker/hadoop/jobcache/job_201305061042_0028/jars/javolution - /tmp/hadoop-hadoop/mapred/local/taskTracker/hadoop/jobcache/job_201305061042_0028/attempt_201305061042_0028_m_00_0/work/javolution 13/05/07 18:43:20 INFO filecache.TrackerDistributedCacheManager:
[jira] [Resolved] (HIVE-4947) Add myself to committer list
[ https://issues.apache.org/jira/browse/HIVE-4947?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gunther Hagleitner resolved HIVE-4947. -- Resolution: Fixed Add myself to committer list Key: HIVE-4947 URL: https://issues.apache.org/jira/browse/HIVE-4947 Project: Hive Issue Type: Task Reporter: Gunther Hagleitner Assignee: Gunther Hagleitner Priority: Minor Attachments: HIVE-4947.2.patch, HIVE-4947.patch Update website to include myself. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4947) Add myself to committer list
[ https://issues.apache.org/jira/browse/HIVE-4947?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13721785#comment-13721785 ] Gunther Hagleitner commented on HIVE-4947: -- Committed to hive-site. Thanks! Add myself to committer list Key: HIVE-4947 URL: https://issues.apache.org/jira/browse/HIVE-4947 Project: Hive Issue Type: Task Reporter: Gunther Hagleitner Assignee: Gunther Hagleitner Priority: Minor Attachments: HIVE-4947.2.patch, HIVE-4947.patch Update website to include myself. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4299) exported metadata by HIVE-3068 cannot be imported because of wrong file name
[
https://issues.apache.org/jira/browse/HIVE-4299?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13721787#comment-13721787
]
Edward Capriolo commented on HIVE-4299:
---
Lets try again with the _ then.
exported metadata by HIVE-3068 cannot be imported because of wrong file name
Key: HIVE-4299
URL: https://issues.apache.org/jira/browse/HIVE-4299
Project: Hive
Issue Type: Bug
Components: Query Processor
Affects Versions: 0.11.0
Reporter: Sho Shimauchi
Assignee: Edward Capriolo
Attachments: HIVE-4299.1.patch.txt, HIVE-4299.4.patch.txt,
HIVE-4299.patch
h2. Symptom
When DROP TABLE a table, metadata of the table is generated to be able to
import the dropped table again.
However, the exported metadata name is 'table name.metadata'.
Since ImportSemanticAnalyzer allows only '_metadata' as metadata filename,
user have to rename the metadata file to import the table.
h2. How to reproduce
Set the following setting to hive-site.xml:
{code}
property
namehive.metastore.pre.event.listeners/name
valueorg.apache.hadoop.hive.ql.parse.MetaDataExportListener/value
/property
{code}
Then run the following queries:
{code}
CREATE TABLE test_table (id INT, name STRING);
DROP TABLE test_table;
IMPORT TABLE test_table_imported FROM '/path/to/metadata/file';
FAILED: SemanticException [Error 10027]: Invalid path
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-4299) exported metadata by HIVE-3068 cannot be imported because of wrong file name
[
https://issues.apache.org/jira/browse/HIVE-4299?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Edward Capriolo updated HIVE-4299:
--
Attachment: HIVE-4299.5.patch.txt
exported metadata by HIVE-3068 cannot be imported because of wrong file name
Key: HIVE-4299
URL: https://issues.apache.org/jira/browse/HIVE-4299
Project: Hive
Issue Type: Bug
Components: Query Processor
Affects Versions: 0.11.0
Reporter: Sho Shimauchi
Assignee: Edward Capriolo
Attachments: HIVE-4299.1.patch.txt, HIVE-4299.4.patch.txt,
HIVE-4299.5.patch.txt, HIVE-4299.patch
h2. Symptom
When DROP TABLE a table, metadata of the table is generated to be able to
import the dropped table again.
However, the exported metadata name is 'table name.metadata'.
Since ImportSemanticAnalyzer allows only '_metadata' as metadata filename,
user have to rename the metadata file to import the table.
h2. How to reproduce
Set the following setting to hive-site.xml:
{code}
property
namehive.metastore.pre.event.listeners/name
valueorg.apache.hadoop.hive.ql.parse.MetaDataExportListener/value
/property
{code}
Then run the following queries:
{code}
CREATE TABLE test_table (id INT, name STRING);
DROP TABLE test_table;
IMPORT TABLE test_table_imported FROM '/path/to/metadata/file';
FAILED: SemanticException [Error 10027]: Invalid path
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4299) exported metadata by HIVE-3068 cannot be imported because of wrong file name
[
https://issues.apache.org/jira/browse/HIVE-4299?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13721808#comment-13721808
]
Hive QA commented on HIVE-4299:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12594564/HIVE-4299.5.patch.txt
{color:red}ERROR:{color} -1 due to 56 failed/errored test(s), 2736 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.jdbc.TestJdbcDriver.testDriverProperties
org.apache.hadoop.hive.jdbc.TestJdbcDriver.testDatabaseMetaData
org.apache.hive.jdbc.TestJdbcDriver2.testSelectAll
org.apache.hive.jdbc.TestJdbcDriver2.testResultSetMetaData
org.apache.hive.jdbc.TestJdbcDriver2.testMetaDataGetColumnsMetaData
org.apache.hive.jdbc.TestJdbcDriver2.testDescribeTable
org.apache.hadoop.hive.jdbc.TestJdbcDriver.testSelectAllFetchSize
org.apache.hive.jdbc.TestJdbcDriver2.testNullType
org.apache.hive.jdbc.TestJdbcDriver2.testDuplicateColumnNameOrder
org.apache.hive.jdbc.TestJdbcDriver2.testProccedures
org.apache.hadoop.hive.jdbc.TestJdbcDriver.testSelectAllMaxRows
org.apache.hadoop.hive.jdbc.TestJdbcDriver.testErrorMessages
org.apache.hive.jdbc.TestJdbcDriver2.testSelectAllFetchSize
org.apache.hive.jdbc.TestJdbcDriver2.testDataTypes2
org.apache.hive.jdbc.TestJdbcDriver2.testOutOfBoundCols
org.apache.hadoop.hive.jdbc.TestJdbcDriver.testSelectAll
org.apache.hive.jdbc.TestJdbcDriver2.testDriverProperties
org.apache.hive.jdbc.TestJdbcDriver2.testErrorDiag
org.apache.hadoop.hive.jdbc.TestJdbcDriver.testNullType
org.apache.hive.jdbc.TestJdbcDriver2.testProcCols
org.apache.hive.jdbc.TestJdbcDriver2.testBuiltInUDFCol
org.apache.hive.jdbc.TestJdbcDriver2.testPostClose
org.apache.hadoop.hive.jdbc.TestJdbcDriver.testMetaDataGetColumnsMetaData
org.apache.hadoop.hive.jdbc.TestJdbcDriver.testMetaDataGetTableTypes
org.apache.hive.jdbc.TestJdbcDriver2.testErrorMessages
org.apache.hadoop.hive.jdbc.TestJdbcDriver.testSelectAllPartioned
org.apache.hive.jdbc.TestJdbcDriver2.testMetaDataGetColumns
org.apache.hive.jdbc.TestJdbcDriver2.testPrimaryKeys
org.apache.hive.jdbc.TestJdbcDriver2.testShowTables
org.apache.hadoop.hive.jdbc.TestJdbcDriver.testMetaDataGetTables
org.apache.hadoop.hive.jdbc.TestJdbcDriver.testResultSetMetaData
org.apache.hive.jdbc.TestJdbcDriver2.testExplainStmt
org.apache.hive.jdbc.TestJdbcDriver2.testBadURL
org.apache.hive.jdbc.TestJdbcDriver2.testMetaDataGetCatalogs
org.apache.hive.jdbc.TestJdbcDriver2.testImportedKeys
org.apache.hadoop.hive.jdbc.TestJdbcDriver.testShowTables
org.apache.hive.jdbc.TestJdbcDriver2.testPrepareStatement
org.apache.hadoop.hive.jdbc.TestJdbcDriver.testMetaDataGetCatalogs
org.apache.hive.jdbc.TestJdbcDriver2.testInvalidURL
org.apache.hive.jdbc.TestJdbcDriver2.testSetCommand
org.apache.hive.jdbc.TestJdbcDriver2.testSelectAllMaxRows
org.apache.hadoop.hive.jdbc.TestJdbcDriver.testExplainStmt
org.apache.hadoop.hive.jdbc.TestJdbcDriver.testDescribeTable
org.apache.hadoop.hive.jdbc.TestJdbcDriver.testDataTypes
org.apache.hive.jdbc.TestJdbcDriver2.testDataTypes
org.apache.hive.jdbc.TestJdbcDriver2.testSelectAllPartioned
org.apache.hadoop.hive.jdbc.TestJdbcDriver.testSetCommand
org.apache.hive.jdbc.TestJdbcDriver2.testMetaDataGetTables
org.apache.hadoop.hive.jdbc.TestJdbcDriver.testMetaDataGetColumns
org.apache.hadoop.hive.jdbc.TestJdbcDriver.testPrepareStatement
org.apache.hive.jdbc.TestJdbcDriver2.testDatabaseMetaData
org.apache.hive.jdbc.TestJdbcDriver2.testMetaDataGetSchemas
org.apache.hadoop.hive.jdbc.TestJdbcDriver.testMetaDataGetSchemas
org.apache.hadoop.hive.jdbc.TestJdbcDriver.testConversionsBaseResultSet
org.apache.hive.jdbc.TestJdbcDriver2.testExprCol
org.apache.hive.jdbc.TestJdbcDriver2.testMetaDataGetTableTypes
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/211/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/211/console
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests failed with: TestsFailedException: 56 tests failed
{noformat}
This message is automatically generated.
exported metadata by HIVE-3068 cannot be imported because of wrong file name
Key: HIVE-4299
URL: https://issues.apache.org/jira/browse/HIVE-4299
Project: Hive
Issue Type: Bug
Components: Query Processor
Affects Versions: 0.11.0
Reporter: Sho Shimauchi
Assignee: Edward Capriolo
Attachments: HIVE-4299.1.patch.txt, HIVE-4299.4.patch.txt,
HIVE-4299.5.patch.txt, HIVE-4299.patch
h2. Symptom
When DROP TABLE a table, metadata of the table is generated to be able to
import the dropped table
[jira] [Updated] (HIVE-4879) Window functions that imply order can only be registered at compile time
[
https://issues.apache.org/jira/browse/HIVE-4879?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Edward Capriolo updated HIVE-4879:
--
Description: Adding an annotation for impliesOrder (was: It seems like the
windowing (PTF) commit never used the apache code style. Curly braces in
particular are not formatted according to standards. Maybe we can bundle a
style commit up
{code}
for (OrderExpression oExpr : spec.getExpressions())
{
OrderExpressionDef oexpDef = translate(inpShape, oExpr);
def.addExpression(oexpDef);
}
{code})
Assignee: Edward Capriolo
Issue Type: Improvement (was: Task)
Summary: Window functions that imply order can only be registered at
compile time (was: Windowing commit is need of a code-style cleanup)
Window functions that imply order can only be registered at compile time
Key: HIVE-4879
URL: https://issues.apache.org/jira/browse/HIVE-4879
Project: Hive
Issue Type: Improvement
Reporter: Edward Capriolo
Assignee: Edward Capriolo
Adding an annotation for impliesOrder
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-4879) Window functions that imply order can only be registered at compile time
[ https://issues.apache.org/jira/browse/HIVE-4879?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Edward Capriolo updated HIVE-4879: -- Attachment: HIVE-4879.1.patch.txt Window functions that imply order can only be registered at compile time Key: HIVE-4879 URL: https://issues.apache.org/jira/browse/HIVE-4879 Project: Hive Issue Type: Improvement Reporter: Edward Capriolo Assignee: Edward Capriolo Attachments: HIVE-4879.1.patch.txt Adding an annotation for impliesOrder -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-4879) Window functions that imply order can only be registered at compile time
[ https://issues.apache.org/jira/browse/HIVE-4879?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Edward Capriolo updated HIVE-4879: -- Fix Version/s: 0.12.0 Affects Version/s: 0.11.0 Status: Patch Available (was: Open) Window functions that imply order can only be registered at compile time Key: HIVE-4879 URL: https://issues.apache.org/jira/browse/HIVE-4879 Project: Hive Issue Type: Improvement Affects Versions: 0.11.0 Reporter: Edward Capriolo Assignee: Edward Capriolo Fix For: 0.12.0 Attachments: HIVE-4879.1.patch.txt Adding an annotation for impliesOrder -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4299) exported metadata by HIVE-3068 cannot be imported because of wrong file name
[
https://issues.apache.org/jira/browse/HIVE-4299?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13721816#comment-13721816
]
Brock Noland commented on HIVE-4299:
The tests failed with
{noformat}
DATE and DATETIME types aren't supported yet. Please use TIMESTAMP instead at
{noformat}
which is believe is a problem on trunk. Running again to see.
exported metadata by HIVE-3068 cannot be imported because of wrong file name
Key: HIVE-4299
URL: https://issues.apache.org/jira/browse/HIVE-4299
Project: Hive
Issue Type: Bug
Components: Query Processor
Affects Versions: 0.11.0
Reporter: Sho Shimauchi
Assignee: Edward Capriolo
Attachments: HIVE-4299.1.patch.txt, HIVE-4299.4.patch.txt,
HIVE-4299.5.patch.txt, HIVE-4299.patch
h2. Symptom
When DROP TABLE a table, metadata of the table is generated to be able to
import the dropped table again.
However, the exported metadata name is 'table name.metadata'.
Since ImportSemanticAnalyzer allows only '_metadata' as metadata filename,
user have to rename the metadata file to import the table.
h2. How to reproduce
Set the following setting to hive-site.xml:
{code}
property
namehive.metastore.pre.event.listeners/name
valueorg.apache.hadoop.hive.ql.parse.MetaDataExportListener/value
/property
{code}
Then run the following queries:
{code}
CREATE TABLE test_table (id INT, name STRING);
DROP TABLE test_table;
IMPORT TABLE test_table_imported FROM '/path/to/metadata/file';
FAILED: SemanticException [Error 10027]: Invalid path
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3926) PPD on virtual column of partitioned table is not working
[
https://issues.apache.org/jira/browse/HIVE-3926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13721815#comment-13721815
]
Gunther Hagleitner commented on HIVE-3926:
--
[~navis] sorry patch doesn't apply cleanly anymore. I believe the conflict is
with HIVE-4929. Can you please rebase?
PPD on virtual column of partitioned table is not working
-
Key: HIVE-3926
URL: https://issues.apache.org/jira/browse/HIVE-3926
Project: Hive
Issue Type: Bug
Components: Query Processor
Reporter: Navis
Assignee: Navis
Priority: Minor
Attachments: HIVE-3926.D8121.1.patch, HIVE-3926.D8121.2.patch,
HIVE-3926.D8121.3.patch, HIVE-3926.D8121.4.patch, HIVE-3926.D8121.5.patch
{code}
select * from src where BLOCK__OFFSET__INSIDE__FILE100;
{code}
is working, but
{code}
select * from srcpart where BLOCK__OFFSET__INSIDE__FILE100;
{code}
throws SemanticException. Disabling PPD makes it work.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4055) add Date data type
[
https://issues.apache.org/jira/browse/HIVE-4055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13721817#comment-13721817
]
Brock Noland commented on HIVE-4055:
We appear to be seeing the following failures on trunk
{noformat}
DATE and DATETIME types aren't supported yet. Please use TIMESTAMP instead at
{noformat}
for TestJdbcDriver\{,2\}
add Date data type
--
Key: HIVE-4055
URL: https://issues.apache.org/jira/browse/HIVE-4055
Project: Hive
Issue Type: Sub-task
Components: JDBC, Query Processor, Serializers/Deserializers, UDF
Reporter: Sun Rui
Assignee: Jason Dere
Fix For: 0.12.0
Attachments: Date.pdf, HIVE-4055.1.patch.txt, HIVE-4055.2.patch.txt,
HIVE-4055.3.patch.txt, HIVE-4055.4.patch, HIVE-4055.4.patch.txt,
HIVE-4055.D11547.1.patch
Add Date data type, a new primitive data type which supports the standard SQL
date type.
Basically, the implementation can take HIVE-2272 and HIVE-2957 as references.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-3255) Add DBTokenStore to store Delegation Tokens in DB
[
https://issues.apache.org/jira/browse/HIVE-3255?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gunther Hagleitner updated HIVE-3255:
-
Resolution: Fixed
Fix Version/s: 0.12.0
Status: Resolved (was: Patch Available)
Add DBTokenStore to store Delegation Tokens in DB
-
Key: HIVE-3255
URL: https://issues.apache.org/jira/browse/HIVE-3255
Project: Hive
Issue Type: New Feature
Components: Metastore, Security
Affects Versions: 0.9.0
Reporter: Ashutosh Chauhan
Assignee: Deepesh Khandelwal
Fix For: 0.12.0
Attachments: hive-3255_2.patch, hive-3255_3.patch, hive-3255_4.patch,
hive-3255_5.patch, HIVE-3255_6.patch, HIVE-3255.6.patch, hive-3255.patch
Before HIVE-1696 metastore was completely stateless and all the data is in
backend db. HIVE-1696 added delegation tokens for metastore which metastore
needs to keep in memory. HIVE-2467 added support for storing delegation
tokens in ZooKeeper via an interface {{DelegationTokenStore}}. This jira is
about using DB backend for storing tokens which metastore uses for storing
data.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3255) Add DBTokenStore to store Delegation Tokens in DB
[
https://issues.apache.org/jira/browse/HIVE-3255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13721819#comment-13721819
]
Gunther Hagleitner commented on HIVE-3255:
--
Committed to trunk. Thanks Ashutosh and Deepesh!
Add DBTokenStore to store Delegation Tokens in DB
-
Key: HIVE-3255
URL: https://issues.apache.org/jira/browse/HIVE-3255
Project: Hive
Issue Type: New Feature
Components: Metastore, Security
Affects Versions: 0.9.0
Reporter: Ashutosh Chauhan
Assignee: Deepesh Khandelwal
Attachments: hive-3255_2.patch, hive-3255_3.patch, hive-3255_4.patch,
hive-3255_5.patch, HIVE-3255_6.patch, HIVE-3255.6.patch, hive-3255.patch
Before HIVE-1696 metastore was completely stateless and all the data is in
backend db. HIVE-1696 added delegation tokens for metastore which metastore
needs to keep in memory. HIVE-2467 added support for storing delegation
tokens in ZooKeeper via an interface {{DelegationTokenStore}}. This jira is
about using DB backend for storing tokens which metastore uses for storing
data.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3926) PPD on virtual column of partitioned table is not working
[
https://issues.apache.org/jira/browse/HIVE-3926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13721822#comment-13721822
]
Gunther Hagleitner commented on HIVE-3926:
--
Correction - I meant HIVE-4878.
PPD on virtual column of partitioned table is not working
-
Key: HIVE-3926
URL: https://issues.apache.org/jira/browse/HIVE-3926
Project: Hive
Issue Type: Bug
Components: Query Processor
Reporter: Navis
Assignee: Navis
Priority: Minor
Attachments: HIVE-3926.D8121.1.patch, HIVE-3926.D8121.2.patch,
HIVE-3926.D8121.3.patch, HIVE-3926.D8121.4.patch, HIVE-3926.D8121.5.patch
{code}
select * from src where BLOCK__OFFSET__INSIDE__FILE100;
{code}
is working, but
{code}
select * from srcpart where BLOCK__OFFSET__INSIDE__FILE100;
{code}
throws SemanticException. Disabling PPD makes it work.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-3926) PPD on virtual column of partitioned table is not working
[
https://issues.apache.org/jira/browse/HIVE-3926?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gunther Hagleitner updated HIVE-3926:
-
Attachment: HIVE-3926.6.patch
PPD on virtual column of partitioned table is not working
-
Key: HIVE-3926
URL: https://issues.apache.org/jira/browse/HIVE-3926
Project: Hive
Issue Type: Bug
Components: Query Processor
Reporter: Navis
Assignee: Navis
Priority: Minor
Attachments: HIVE-3926.6.patch, HIVE-3926.D8121.1.patch,
HIVE-3926.D8121.2.patch, HIVE-3926.D8121.3.patch, HIVE-3926.D8121.4.patch,
HIVE-3926.D8121.5.patch
{code}
select * from src where BLOCK__OFFSET__INSIDE__FILE100;
{code}
is working, but
{code}
select * from srcpart where BLOCK__OFFSET__INSIDE__FILE100;
{code}
throws SemanticException. Disabling PPD makes it work.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4055) add Date data type
[ https://issues.apache.org/jira/browse/HIVE-4055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13721829#comment-13721829 ] Brock Noland commented on HIVE-4055: Something quite strange is going on with the TestJdbcDriver* tests. I cannot TestJdbcDriver2 to fail locally but it does fail on ptest: https://builds.apache.org/user/brock/my-views/view/hive/job/Hive-trunk-hadoop1-ptest/103/ I'll take a look tomorrow unless someone else figures it out first. add Date data type -- Key: HIVE-4055 URL: https://issues.apache.org/jira/browse/HIVE-4055 Project: Hive Issue Type: Sub-task Components: JDBC, Query Processor, Serializers/Deserializers, UDF Reporter: Sun Rui Assignee: Jason Dere Fix For: 0.12.0 Attachments: Date.pdf, HIVE-4055.1.patch.txt, HIVE-4055.2.patch.txt, HIVE-4055.3.patch.txt, HIVE-4055.4.patch, HIVE-4055.4.patch.txt, HIVE-4055.D11547.1.patch Add Date data type, a new primitive data type which supports the standard SQL date type. Basically, the implementation can take HIVE-2272 and HIVE-2957 as references. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3926) PPD on virtual column of partitioned table is not working
[
https://issues.apache.org/jira/browse/HIVE-3926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13721861#comment-13721861
]
Hive QA commented on HIVE-3926:
---
{color:green}Overall{color}: +1 all checks pass
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12594569/HIVE-3926.6.patch
{color:green}SUCCESS:{color} +1 2730 tests passed
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/214/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/214/console
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
PPD on virtual column of partitioned table is not working
-
Key: HIVE-3926
URL: https://issues.apache.org/jira/browse/HIVE-3926
Project: Hive
Issue Type: Bug
Components: Query Processor
Reporter: Navis
Assignee: Navis
Priority: Minor
Attachments: HIVE-3926.6.patch, HIVE-3926.D8121.1.patch,
HIVE-3926.D8121.2.patch, HIVE-3926.D8121.3.patch, HIVE-3926.D8121.4.patch,
HIVE-3926.D8121.5.patch
{code}
select * from src where BLOCK__OFFSET__INSIDE__FILE100;
{code}
is working, but
{code}
select * from srcpart where BLOCK__OFFSET__INSIDE__FILE100;
{code}
throws SemanticException. Disabling PPD makes it work.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4520) java.lang.NegativeArraySizeException when query on hive-0.11.0, hbase-0.94.6.1
[ https://issues.apache.org/jira/browse/HIVE-4520?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13721865#comment-13721865 ] Bryan Peterson commented on HIVE-4520: -- Seems so. The only difference is version numbers for both hive and hbase. java.lang.NegativeArraySizeException when query on hive-0.11.0, hbase-0.94.6.1 -- Key: HIVE-4520 URL: https://issues.apache.org/jira/browse/HIVE-4520 Project: Hive Issue Type: Bug Components: HBase Handler Affects Versions: 0.11.0 Environment: hive-0.11.0 hbase-0.94.6.1 zookeeper-3.4.3 hadoop-1.0.4 centos-5.7 Reporter: Yanhui Ma Priority: Critical After integration hive-0.11.0+hbase-0.94.6.1, these commands could be executed sucessfully: create table insert overwrite table select * from table However, when execute select count(*) from table, throws exception: hive select count(*) from test; Total MapReduce jobs = 1 Launching Job 1 out of 1 Number of reduce tasks determined at compile time: 1 In order to change the average load for a reducer (in bytes): set hive.exec.reducers.bytes.per.reducer=number In order to limit the maximum number of reducers: set hive.exec.reducers.max=number In order to set a constant number of reducers: set mapred.reduce.tasks=number Starting Job = job_201305061042_0028, Tracking URL = http://master0:50030/jobdetails.jsp?jobid=job_201305061042_0028 Kill Command = /opt/modules/hadoop/hadoop-1.0.4/libexec/../bin/hadoop job -kill job_201305061042_0028 Hadoop job information for Stage-1: number of mappers: 1; number of reducers: 1 2013-05-07 18:41:42,649 Stage-1 map = 0%, reduce = 0% 2013-05-07 18:42:14,789 Stage-1 map = 100%, reduce = 100% Ended Job = job_201305061042_0028 with errors Error during job, obtaining debugging information... Job Tracking URL: http://master0:50030/jobdetails.jsp?jobid=job_201305061042_0028 Examining task ID: task_201305061042_0028_m_02 (and more) from job job_201305061042_0028 Task with the most failures(4): - Task ID: task_201305061042_0028_m_00 URL: http://master0:50030/taskdetails.jsp?jobid=job_201305061042_0028tipid=task_201305061042_0028_m_00 - Diagnostic Messages for this Task: java.lang.NegativeArraySizeException: -1 at org.apache.hadoop.hbase.util.Bytes.readByteArray(Bytes.java:148) at org.apache.hadoop.hbase.mapreduce.TableSplit.readFields(TableSplit.java:133) at org.apache.hadoop.hive.hbase.HBaseSplit.readFields(HBaseSplit.java:53) at org.apache.hadoop.hive.ql.io.HiveInputFormat$HiveInputSplit.readFields(HiveInputFormat.java:150) at org.apache.hadoop.io.serializer.WritableSerialization$WritableDeserializer.deserialize(WritableSerialization.java:67) at org.apache.hadoop.io.serializer.WritableSerialization$WritableDeserializer.deserialize(WritableSerialization.java:40) at org.apache.hadoop.mapred.MapTask.getSplitDetails(MapTask.java:396) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:412) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:372) at org.apache.hadoop.mapred.Child$4.run(Child.java:255) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1121) at org.apache.hadoop.mapred.Child.main(Child.java:249) FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.MapRedTask MapReduce Jobs Launched: Job 0: Map: 1 Reduce: 1 HDFS Read: 0 HDFS Write: 0 FAIL Total MapReduce CPU Time Spent: 0 msec == The log of tasktracker: stderr logs 13/05/07 18:43:20 INFO util.NativeCodeLoader: Loaded the native-hadoop library 13/05/07 18:43:20 INFO mapred.TaskRunner: Creating symlink: /tmp/hadoop-hadoop/mapred/local/taskTracker/distcache/107328478296390_-1298160740_2123690974/master0/tmp/hive-hadoop/hive_2013-05-07_18-41-30_290_832140779606816147/-mr-10003/fd22448b-e923-498c-bc00-2164ca68447d - /tmp/hadoop-hadoop/mapred/local/taskTracker/hadoop/jobcache/job_201305061042_0028/attempt_201305061042_0028_m_00_0/work/HIVE_PLANfd22448b-e923-498c-bc00-2164ca68447d 13/05/07 18:43:20 INFO filecache.TrackerDistributedCacheManager: Creating symlink: /tmp/hadoop-hadoop/mapred/local/taskTracker/hadoop/jobcache/job_201305061042_0028/jars/javolution - /tmp/hadoop-hadoop/mapred/local/taskTracker/hadoop/jobcache/job_201305061042_0028/attempt_201305061042_0028_m_00_0/work/javolution 13/05/07 18:43:20 INFO filecache.TrackerDistributedCacheManager: Creating symlink:
[jira] [Updated] (HIVE-4879) Window functions that imply order can only be registered at compile time
[ https://issues.apache.org/jira/browse/HIVE-4879?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Edward Capriolo updated HIVE-4879: -- Attachment: HIVE-4879.2.patch.txt One more try Window functions that imply order can only be registered at compile time Key: HIVE-4879 URL: https://issues.apache.org/jira/browse/HIVE-4879 Project: Hive Issue Type: Improvement Affects Versions: 0.11.0 Reporter: Edward Capriolo Assignee: Edward Capriolo Fix For: 0.12.0 Attachments: HIVE-4879.1.patch.txt, HIVE-4879.2.patch.txt Adding an annotation for impliesOrder -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3561) Build a full SQL-compliant parser for Hive
[ https://issues.apache.org/jira/browse/HIVE-3561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13721883#comment-13721883 ] Edward Capriolo commented on HIVE-3561: --- I think supporting two query languages is nuts. What is the obsession with SQL compliance? I like hive's query language. It has less constructs and that makes it easier to read and write queries. Also, knowing hive means learning and understanding map reduce you can make things too easy for people and they get into anti-patterns. Having a new parser or two parsers activated by a switch sounds not good to me either. I really do not see the big win. For the people who can write the complicated sub queries that hive can not currently do there are usually smart enough to look in the language manual and figure out a way to re-write the query. Build a full SQL-compliant parser for Hive -- Key: HIVE-3561 URL: https://issues.apache.org/jira/browse/HIVE-3561 Project: Hive Issue Type: Sub-task Components: Query Processor Affects Versions: 0.10.0 Reporter: Shane Huang To build a full SQL compliant engine on Hive, we'll need a full SQL complant parser. The current Hive parser missed a lot of grammar units from standard SQL. To support full SQL there're possibly four approaches: 1.Extend the existing Hive parser to support full SQL constructs. We need to modify the current Hive.g and add any missing grammar units and resolve conflicts. 2.Reuse an existing open source SQL compliant parser and extend it to support Hive extensions. We may need to adapt Semantic Analyzers to the new AST structure. 3.Reuse an existing SQL compliant parser and make it co-exist with the existing Hive parser. Both parsers share the same CliDriver interface. Use a query mode configuration to switch the query mode between SQL and HQL (this is the approach we're now using in the 0.9.0 demo project) 4.Reuse an existing SQL compliant parser and make it co-exist with the existing Hive parser. Use a separate xxxCliDriver interface for standard SQL. Let's discuss which is the best approach. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
