[jira] [Commented] (HIVE-12956) run CBO in tests with mapred.mode=strict
[
https://issues.apache.org/jira/browse/HIVE-12956?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15125065#comment-15125065
]
Hive QA commented on HIVE-12956:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12785291/HIVE-12956.2.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 8 failed/errored test(s), 10046 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby13
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_input39
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_input41
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join_view
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_uri_import
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_join_view
org.apache.hive.jdbc.TestSSL.testSSLVersion
org.apache.hive.service.cli.TestEmbeddedThriftBinaryCLIService.testExecuteStatementAsync
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6803/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6803/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-6803/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 8 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12785291 - PreCommit-HIVE-TRUNK-Build
> run CBO in tests with mapred.mode=strict
>
>
> Key: HIVE-12956
> URL: https://issues.apache.org/jira/browse/HIVE-12956
> Project: Hive
> Issue Type: Test
>Reporter: Sergey Shelukhin
>Assignee: Ashutosh Chauhan
> Attachments: HIVE-12956.2.patch, HIVE-12956.patch
>
>
> There's a strange condition in CBO check that prevents CBO from running in
> Hive tests (in tests specifically) when mapred.mode is set to strict. I
> remember seeing it before, and noticed it again recently.
> It is both surprising that we wouldn't test CBO in strict mode, and also
> problematic for some q files because strict mode is going to be deprecated in
> HIVE-12727. This needs to be removed.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-12953) Update description of hive.ppd.remove.duplicatefilters in HiveConf.java
[ https://issues.apache.org/jira/browse/HIVE-12953?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15125087#comment-15125087 ] Shannon Ladymon commented on HIVE-12953: Thanks for reviewing and committing it, Ashutosh! > Update description of hive.ppd.remove.duplicatefilters in HiveConf.java > --- > > Key: HIVE-12953 > URL: https://issues.apache.org/jira/browse/HIVE-12953 > Project: Hive > Issue Type: Improvement > Components: Documentation >Affects Versions: 1.2.1 >Reporter: Shannon Ladymon >Assignee: Shannon Ladymon >Priority: Minor > Fix For: 2.1.0 > > Attachments: HIVE-12953.patch > > > Per [discussion in HIVE-1538| > https://issues.apache.org/jira/browse/HIVE-1538?focusedCommentId=14568180=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14568180], > we should update the description of the the > *hive.ppd.remove.duplicatefilters* parameter in HiveConf.java. > NO PRECOMMIT TESTS -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12958) Make embedded Jetty server more configurable
[
https://issues.apache.org/jira/browse/HIVE-12958?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15125008#comment-15125008
]
Hive QA commented on HIVE-12958:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12785170/HIVE-12958.2.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 3 failed/errored test(s), 10046 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_uri_import
org.apache.hive.jdbc.TestSSL.testSSLVersion
org.apache.hive.service.cli.TestEmbeddedThriftBinaryCLIService.testExecuteStatementAsync
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6802/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6802/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-6802/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 3 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12785170 - PreCommit-HIVE-TRUNK-Build
> Make embedded Jetty server more configurable
>
>
> Key: HIVE-12958
> URL: https://issues.apache.org/jira/browse/HIVE-12958
> Project: Hive
> Issue Type: Improvement
> Components: HCatalog
>Affects Versions: 2.1.0
>Reporter: Aihua Xu
>Assignee: Aihua Xu
> Attachments: HIVE-12958.1.patch, HIVE-12958.2.patch
>
>
> Currently you can't configure embedded jetty within HCatalog. Propose to
> support add an xml configuration which Jetty already supports. A new Web-hcat
> configuration will be added to specify the configure file location. If the
> file doesn't exist, falls back to old behavior. If it exists, load such
> configuration to configure Jetty server.
> Some default parameters may not be sufficient such as request/response buffer
> size. This improvement allows to make such configuration change.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-1608) use sequencefile as the default for storing intermediate results
[ https://issues.apache.org/jira/browse/HIVE-1608?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15124980#comment-15124980 ] Aihua Xu commented on HIVE-1608: I will take a look as well. Didn't know this issue before. > use sequencefile as the default for storing intermediate results > > > Key: HIVE-1608 > URL: https://issues.apache.org/jira/browse/HIVE-1608 > Project: Hive > Issue Type: Bug > Components: Query Processor >Affects Versions: 0.7.0 >Reporter: Namit Jain >Assignee: Brock Noland > Attachments: HIVE-1608.1.patch, HIVE-1608.patch > > > The only argument for having a text file for storing intermediate results > seems to be better debuggability. > But, tailing a sequence file is possible, and it should be more space > efficient -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12947) SMB join in tez has ClassCastException when container reuse is on
[
https://issues.apache.org/jira/browse/HIVE-12947?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15124976#comment-15124976
]
Hive QA commented on HIVE-12947:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12785251/HIVE-12947.3.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 5 failed/errored test(s), 10042 tests
executed
*Failed tests:*
{noformat}
TestHBaseAggrStatsCacheIntegration - did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_smb_cache
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_uri_import
org.apache.hive.jdbc.TestSSL.testSSLVersion
org.apache.hive.service.cli.TestEmbeddedThriftBinaryCLIService.testExecuteStatementAsync
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6799/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6799/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-6799/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 5 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12785251 - PreCommit-HIVE-TRUNK-Build
> SMB join in tez has ClassCastException when container reuse is on
> -
>
> Key: HIVE-12947
> URL: https://issues.apache.org/jira/browse/HIVE-12947
> Project: Hive
> Issue Type: Bug
> Components: Tez
>Affects Versions: 2.0.0
>Reporter: Vikram Dixit K
>Assignee: Vikram Dixit K
>Priority: Critical
> Attachments: HIVE-12947.1.patch, HIVE-12947.2.patch,
> HIVE-12947.3.patch
>
>
> SMB join in tez has multiple work items that are connected based on input tag
> followed by input initialization etc. In case of container re-use, what ends
> up happening is that we try to reconnect the work items and fail. If we try
> to work around that issue by recognizing somehow that the cache was in play,
> we will run into other initialization issues with respect to record readers.
> So the plan is to disable caching of the SMB work items by clearing out
> during the close phase.
> {code}
> java.lang.RuntimeException: Map operator initialization failed
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.init(MapRecordProcessor.java:247)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:147)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:137)
> at
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:344)
> at
> org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:179)
> at
> org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:171)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
> at
> org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.callInternal(TezTaskRunner.java:171)
> at
> org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.callInternal(TezTaskRunner.java:167)
> at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.ClassCastException:
> org.apache.hadoop.hive.ql.exec.FileSinkOperator cannot be cast to
> org.apache.hadoop.hive.ql.exec.DummyStoreOperator
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.getJoinParentOp(MapRecordProcessor.java:300)
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.getJoinParentOp(MapRecordProcessor.java:302)
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.getJoinParentOp(MapRecordProcessor.java:302)
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.getJoinParentOp(MapRecordProcessor.java:302)
> at
>
[jira] [Commented] (HIVE-12855) LLAP: add checks when resolving UDFs to enforce whitelist
[
https://issues.apache.org/jira/browse/HIVE-12855?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15124899#comment-15124899
]
Hive QA commented on HIVE-12855:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12784998/HIVE-12855.04.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 5 failed/errored test(s), 10031 tests
executed
*Failed tests:*
{noformat}
TestSparkCliDriver-timestamp_lazy.q-bucketsortoptimize_insert_4.q-date_udf.q-and-12-more
- did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_uri_import
org.apache.hive.jdbc.TestMultiSessionsHS2WithLocalClusterSpark.testSparkQuery
org.apache.hive.jdbc.TestSSL.testSSLVersion
org.apache.hive.service.cli.TestEmbeddedThriftBinaryCLIService.testExecuteStatementAsync
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6798/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6798/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-6798/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 5 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12784998 - PreCommit-HIVE-TRUNK-Build
> LLAP: add checks when resolving UDFs to enforce whitelist
> -
>
> Key: HIVE-12855
> URL: https://issues.apache.org/jira/browse/HIVE-12855
> Project: Hive
> Issue Type: Sub-task
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: HIVE-12855.01.patch, HIVE-12855.02.patch,
> HIVE-12855.03.patch, HIVE-12855.04.patch, HIVE-12855.part.patch
>
>
> Currently, adding a temporary UDF and calling LLAP with it (bypassing the
> LlapDecider check, I did it by just modifying the source) only fails because
> the class could not be found. If the UDF was accessible to LLAP, it would
> execute. Inside the daemon, UDF instantiation should fail for custom UDFs
> (and only succeed for whitelisted custom UDFs, once that is implemented).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-12944) Support SUM(DISTINCT) for partitioning query.
[
https://issues.apache.org/jira/browse/HIVE-12944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15124842#comment-15124842
]
Hive QA commented on HIVE-12944:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12784994/HIVE-12944.3.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 4 failed/errored test(s), 10031 tests
executed
*Failed tests:*
{noformat}
TestSparkCliDriver-timestamp_lazy.q-bucketsortoptimize_insert_4.q-date_udf.q-and-12-more
- did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_uri_import
org.apache.hive.jdbc.TestSSL.testSSLVersion
org.apache.hive.service.cli.TestEmbeddedThriftBinaryCLIService.testExecuteStatementAsync
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6797/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6797/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-6797/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 4 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12784994 - PreCommit-HIVE-TRUNK-Build
> Support SUM(DISTINCT) for partitioning query.
> -
>
> Key: HIVE-12944
> URL: https://issues.apache.org/jira/browse/HIVE-12944
> Project: Hive
> Issue Type: Sub-task
> Components: PTF-Windowing
>Affects Versions: 2.1.0
>Reporter: Aihua Xu
>Assignee: Aihua Xu
> Attachments: HIVE-12944.2.patch, HIVE-12944.3.patch, HIVE-12944.patch
>
>
> We need to support avg(distinct), count(distinct), sum(distinct) for the
> parent jira HIVE-9534. Separate the work for sum(distinct) in this subtask.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-12963) LIMIT statement with SORT BY creates additional MR job with hardcoded only one reducer
[ https://issues.apache.org/jira/browse/HIVE-12963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15124823#comment-15124823 ] Alina Abramova commented on HIVE-12963: --- But I see that if line with creating of genReduceSinkPlan in method genLimitMapRedPlan is commented then finish set is sorted too. It means that we could refuse the creating of extra job, and do sorting in the same MR job, doesn't it? > LIMIT statement with SORT BY creates additional MR job with hardcoded only > one reducer > -- > > Key: HIVE-12963 > URL: https://issues.apache.org/jira/browse/HIVE-12963 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 1.0.0, 1.2.1, 0.13 >Reporter: Alina Abramova >Assignee: Alina Abramova > Attachments: HIVE-12963.1.patch > > > I execute query: > hive> select age from test1 sort by age.age limit 10; > Total jobs = 2 > Launching Job 1 out of 2 > Number of reduce tasks not specified. Estimated from input data size: 1 > Launching Job 2 out of 2 > Number of reduce tasks determined at compile time: 1 > When I have a large number of rows then the last stage of the job takes a > long time. I think we could allow to user choose number of reducers of last > job or refuse extra MR job. > The same behavior I observed with queries: > hive> create table new_test as select age from test1 group by age.age limit > 10; -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-6147) Support avro data stored in HBase columns
[ https://issues.apache.org/jira/browse/HIVE-6147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15125129#comment-15125129 ] Ilya Kats commented on HIVE-6147: - Thank you for a quick reply, Swarnim. I'll try to implement a custom AvroSchemaRetriever. However, I'm not quite clear on why the writer schema is necessarily loaded if the reader schema is provided. It is pretty common to use schema-less avro objects in HBase (if there are billions of rows with objects of the same type, it is not reasonable to store the same schema in all of them) and it is not convenient to write a customer schema retriever for each such case. So, I wonder if it is better to assume that the writer schema is equal to the reader schema if the former cannot be found neither in data nor via a customer retriever? > Support avro data stored in HBase columns > - > > Key: HIVE-6147 > URL: https://issues.apache.org/jira/browse/HIVE-6147 > Project: Hive > Issue Type: Improvement > Components: HBase Handler >Affects Versions: 0.12.0, 0.13.0 >Reporter: Swarnim Kulkarni >Assignee: Swarnim Kulkarni > Labels: TODOC14 > Fix For: 0.14.0 > > Attachments: HIVE-6147.1.patch.txt, HIVE-6147.2.patch.txt, > HIVE-6147.3.patch.txt, HIVE-6147.3.patch.txt, HIVE-6147.4.patch.txt, > HIVE-6147.5.patch.txt, HIVE-6147.6.patch.txt > > > Presently, the HBase Hive integration supports querying only primitive data > types in columns. It would be nice to be able to store and query Avro objects > in HBase columns by making them visible as structs to Hive. This will allow > Hive to perform ad hoc analysis of HBase data which can be deeply structured. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12934) Refactor llap module structure to allow for a usable client
[ https://issues.apache.org/jira/browse/HIVE-12934?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated HIVE-12934: -- Attachment: HIVE-12934.02.patch Updated patch for jenkins. > Refactor llap module structure to allow for a usable client > --- > > Key: HIVE-12934 > URL: https://issues.apache.org/jira/browse/HIVE-12934 > Project: Hive > Issue Type: Task >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Attachments: HIVE-12934.01.patch, HIVE-12934.01.patch, > HIVE-12934.02.patch, HIVE-12934.02.review.patch, HIVE-12934.1.patch, > HIVE-12934.1.review.txt, HIVE-12934.1.txt > > > The client isn't really usable at the moment, and all of the code resides in > the llap-server module. Restructure this so that the daemon execution code > and cache code remains in server, common components move to a different > module and relevant client pieces sit in the client module. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12934) Refactor llap module structure to allow for a usable client
[ https://issues.apache.org/jira/browse/HIVE-12934?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated HIVE-12934: -- Attachment: HIVE-12934.02.review.patch Updated review patch with additional dependencies for the test failures. > Refactor llap module structure to allow for a usable client > --- > > Key: HIVE-12934 > URL: https://issues.apache.org/jira/browse/HIVE-12934 > Project: Hive > Issue Type: Task >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Attachments: HIVE-12934.01.patch, HIVE-12934.01.patch, > HIVE-12934.02.patch, HIVE-12934.02.review.patch, HIVE-12934.1.patch, > HIVE-12934.1.review.txt, HIVE-12934.1.txt > > > The client isn't really usable at the moment, and all of the code resides in > the llap-server module. Restructure this so that the daemon execution code > and cache code remains in server, common components move to a different > module and relevant client pieces sit in the client module. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11609) Capability to add a filter to hbase scan via composite key doesn't work
[ https://issues.apache.org/jira/browse/HIVE-11609?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15125126#comment-15125126 ] Swarnim Kulkarni commented on HIVE-11609: - [~ychena] Thanks for taking a look. Unfortunately I am not 100% sure at this point why but looking at the sequence of patches it looks like that was in to address a test failure that was caused by the first patch. > Capability to add a filter to hbase scan via composite key doesn't work > --- > > Key: HIVE-11609 > URL: https://issues.apache.org/jira/browse/HIVE-11609 > Project: Hive > Issue Type: Bug > Components: HBase Handler >Reporter: Swarnim Kulkarni >Assignee: Swarnim Kulkarni > Attachments: HIVE-11609.1.patch.txt, HIVE-11609.2.patch.txt, > HIVE-11609.3.patch.txt, HIVE-11609.4.patch.txt, HIVE-11609.5.patch > > > It seems like the capability to add filter to an hbase scan which was added > as part of HIVE-6411 doesn't work. This is primarily because in the > HiveHBaseInputFormat, the filter is added in the getsplits instead of > getrecordreader. This works fine for start and stop keys but not for filter > because a filter is respected only when an actual scan is performed. This is > also related to the initial refactoring that was done as part of HIVE-3420. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11790) HCatLoader documentation refers to deprecated package.
[
https://issues.apache.org/jira/browse/HIVE-11790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15125132#comment-15125132
]
Lefty Leverenz commented on HIVE-11790:
---
Thank you, Andrew. Here are links to the doc changes:
* [Running Pig |
https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=34013511#HCatalogLoadStore-RunningPig]
* [HCatLoader -- Usage |
https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=34013511#HCatalogLoadStore-Usage]
* [HCatLoader -- Load Examples |
https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=34013511#HCatalogLoadStore-LoadExamples]
* [HCatStorer -- Usage |
https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=34013511#HCatalogLoadStore-Usage.1]
* [HCatStorer -- Store Examples |
https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=34013511#HCatalogLoadStore-StoreExamples]
A question: In the Stale Content Warning box, shouldn't "Pig Versions" be
"Hive Versions"? I'm not familiar with Pig versions but in this jira's
description [~mithun] says "The old package was deprecated and removed in Hive
0.13." Hmm ... that would mean it's Hive 0.13+ instead of 0.14+. Could you
please check?
The same question applies to the new subheading "Using Pig 0.14+" in the
HCatLoader Usage section.
> HCatLoader documentation refers to deprecated package.
> --
>
> Key: HIVE-11790
> URL: https://issues.apache.org/jira/browse/HIVE-11790
> Project: Hive
> Issue Type: Bug
>Reporter: Mithun Radhakrishnan
>Assignee: Andrew Sears
>Priority: Trivial
>
> The [HCatLoader documentation
> page|https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=34013511]
> seems to refer to {{org.apache.hcatalog.pig.HCatLoader}} instead of
> {{org.apache.hive.hcatalog.pig.HCatLoader}}. (Similarly, the {{HCatStorer}}
> documentation might need change.) The old package was deprecated and removed
> in Hive 0.13.
> Let's change the documentation to reflect the new package-name.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-12931) Shuffle tokens stay around forever in LLAP
[ https://issues.apache.org/jira/browse/HIVE-12931?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated HIVE-12931: -- Attachment: HIVE-12931.02.patch Re-uploading for another jenkins run. > Shuffle tokens stay around forever in LLAP > -- > > Key: HIVE-12931 > URL: https://issues.apache.org/jira/browse/HIVE-12931 > Project: Hive > Issue Type: Bug >Affects Versions: 2.0.0 >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Attachments: HIVE-12931.02.patch, HIVE-12931.02.patch, > HIVE-12931.02.patch, HIVE-12931.1.txt, HIVE-12931.2.txt > > > Shuffle tokens are never cleaned up, resulting in a slow leak. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-6147) Support avro data stored in HBase columns
[
https://issues.apache.org/jira/browse/HIVE-6147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15125117#comment-15125117
]
Swarnim Kulkarni commented on HIVE-6147:
{noformat}
it tries to retrieve the write schema from data (ws =
retrieveSchemaFromBytes(data)) even if the schema URL (reader schema) had been
provided
{noformat}
Correct. That is the default behavior. The writer schema defaults to the reader
schema if one has not been provided. If it has been(like you are doing in your
case), it would use the reader schema from the given URL but still default to
the writer schema from the data. If you want to provide the writer schema as
well, I would recommend you take a look into the AvroSchemaRetriever[1]. You
can provide a custom implementation of it and provide both reader and write
schema from any custom source that you would like. A test implementation can be
found here for reference[2] and the corresponding test that uses this
implementation here[3]. Once done, simply plug it in with
"avro.schema.retriever" property. One caveat is that this will currently apply
to the whole table and not individual columns. So it makes the assumption that
there is a uniform schema across the table.
Hope this helps. Let me know if there are any additional questions.
[1]
https://github.com/apache/hive/blob/release-1.2.1/serde/src/java/org/apache/hadoop/hive/serde2/avro/AvroSchemaRetriever.java
[2]
https://github.com/apache/hive/blob/release-1.2.1/hbase-handler/src/test/org/apache/hadoop/hive/hbase/HBaseTestAvroSchemaRetriever.java
[3]
https://github.com/apache/hive/blob/release-1.2.1/hbase-handler/src/test/org/apache/hadoop/hive/hbase/TestHBaseSerDe.java#L1293-L1344
> Support avro data stored in HBase columns
> -
>
> Key: HIVE-6147
> URL: https://issues.apache.org/jira/browse/HIVE-6147
> Project: Hive
> Issue Type: Improvement
> Components: HBase Handler
>Affects Versions: 0.12.0, 0.13.0
>Reporter: Swarnim Kulkarni
>Assignee: Swarnim Kulkarni
> Labels: TODOC14
> Fix For: 0.14.0
>
> Attachments: HIVE-6147.1.patch.txt, HIVE-6147.2.patch.txt,
> HIVE-6147.3.patch.txt, HIVE-6147.3.patch.txt, HIVE-6147.4.patch.txt,
> HIVE-6147.5.patch.txt, HIVE-6147.6.patch.txt
>
>
> Presently, the HBase Hive integration supports querying only primitive data
> types in columns. It would be nice to be able to store and query Avro objects
> in HBase columns by making them visible as structs to Hive. This will allow
> Hive to perform ad hoc analysis of HBase data which can be deeply structured.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-9941) sql std authorization on partitioned table: truncate and insert
[
https://issues.apache.org/jira/browse/HIVE-9941?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15125128#comment-15125128
]
Hive QA commented on HIVE-9941:
---
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12785024/HIVE-9941.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 5 failed/errored test(s), 10033 tests
executed
*Failed tests:*
{noformat}
TestMiniTezCliDriver-vector_left_outer_join2.q-vector_outer_join5.q-custom_input_output_format.q-and-12-more
- did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_uri_import
org.apache.hive.jdbc.TestSSL.testSSLVersion
org.apache.hive.service.TestHS2ImpersonationWithRemoteMS.org.apache.hive.service.TestHS2ImpersonationWithRemoteMS
org.apache.hive.service.cli.TestEmbeddedThriftBinaryCLIService.testExecuteStatementAsync
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6804/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6804/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-6804/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 5 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12785024 - PreCommit-HIVE-TRUNK-Build
> sql std authorization on partitioned table: truncate and insert
> ---
>
> Key: HIVE-9941
> URL: https://issues.apache.org/jira/browse/HIVE-9941
> Project: Hive
> Issue Type: Bug
> Components: Authorization
>Affects Versions: 1.0.0, 1.2.0
>Reporter: Olaf Flebbe
>Assignee: Sushanth Sowmyan
> Attachments: HIVE-9941.patch
>
>
> sql std authorization works as expected.
> However if a table is partitioned any user can truncate it
> User foo:
> {code}
> create table bla (a string) partitioned by (b string);
> #.. loading values ...
> {code}
> Admin:
> {code}
> 0: jdbc:hive2://localhost:1/default> set role admin;
> No rows affected (0,074 seconds)
> 0: jdbc:hive2://localhost:1/default> show grant on bla;
> +---+++-+-+-++---++--+--+
> | database | table | partition | column | principal_name |
> principal_type | privilege | grant_option | grant_time | grantor |
> +---+++-+-+-++---++--+--+
> | default | bla|| | foo | USER
> | DELETE | true | 1426158997000 | foo |
> | default | bla|| | foo | USER
> | INSERT | true | 1426158997000 | foo |
> | default | bla|| | foo | USER
> | SELECT | true | 1426158997000 | foo |
> | default | bla|| | foo | USER
> | UPDATE | true | 1426158997000 | foo |
> +---+++-+-+-++---++--+--+
> {code}
> now user olaf
> {code}
> 0: jdbc:hive2://localhost:1/default> select * from bla;
> Error: Error while compiling statement: FAILED: HiveAccessControlException
> Permission denied: Principal [name=olaf, type=USER] does not have following
> privileges for operation QUERY [[SELECT] on Object [type=TABLE_OR_VIEW,
> name=default.bla]] (state=42000,code=4)
> {code}
> works as expected.
> _BUT_
> {code}
> 0: jdbc:hive2://localhost:1/default> truncate table bla;
> No rows affected (0,18 seconds)
> {code}
> _And table is empty afterwards_.
> Similarily: {{insert into table}} works, too.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-11790) HCatLoader documentation refers to deprecated package.
[
https://issues.apache.org/jira/browse/HIVE-11790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15125157#comment-15125157
]
Lefty Leverenz commented on HIVE-11790:
---
I found the JIRA issue that removed the deprecated package: HIVE-6432 in
release 0.14.0 ("Remove deprecated methods in HCatalog"). So the doc should
say Hive version 0.14 instead of Pig version 0.14.
I'll fix it later if you don't get to it first.
> HCatLoader documentation refers to deprecated package.
> --
>
> Key: HIVE-11790
> URL: https://issues.apache.org/jira/browse/HIVE-11790
> Project: Hive
> Issue Type: Bug
>Reporter: Mithun Radhakrishnan
>Assignee: Andrew Sears
>Priority: Trivial
>
> The [HCatLoader documentation
> page|https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=34013511]
> seems to refer to {{org.apache.hcatalog.pig.HCatLoader}} instead of
> {{org.apache.hive.hcatalog.pig.HCatLoader}}. (Similarly, the {{HCatStorer}}
> documentation might need change.) The old package was deprecated and removed
> in Hive 0.13.
> Let's change the documentation to reflect the new package-name.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-12969) ix Javadoc for PredicatePushDown class
[ https://issues.apache.org/jira/browse/HIVE-12969?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Sears updated HIVE-12969: Attachment: hive-12969.patch update Javadoc > ix Javadoc for PredicatePushDown class > -- > > Key: HIVE-12969 > URL: https://issues.apache.org/jira/browse/HIVE-12969 > Project: Hive > Issue Type: Bug > Components: Documentation >Affects Versions: 0.4.0 >Reporter: Andrew Sears >Assignee: Andrew Sears >Priority: Trivial > Attachments: hive-12969.patch > > > Fix Javadocs for hive.optimize.ppd - Default Value: true > Added In: Hive 0.4.0 with HIVE-279, default changed to true in Hive 0.4.0 > with HIVE-626 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12790) Metastore connection leaks in HiveServer2
[
https://issues.apache.org/jira/browse/HIVE-12790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15125197#comment-15125197
]
Hive QA commented on HIVE-12790:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12785008/HIVE-12790.2.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 6 failed/errored test(s), 10047 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.org.apache.hadoop.hive.cli.TestMiniTezCliDriver
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_insert_into1
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_aggregate_without_gby
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_uri_import
org.apache.hive.jdbc.TestSSL.testSSLVersion
org.apache.hive.service.cli.TestEmbeddedThriftBinaryCLIService.testExecuteStatementAsync
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6805/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6805/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-6805/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 6 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12785008 - PreCommit-HIVE-TRUNK-Build
> Metastore connection leaks in HiveServer2
> -
>
> Key: HIVE-12790
> URL: https://issues.apache.org/jira/browse/HIVE-12790
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 1.1.0
>Reporter: Naveen Gangam
>Assignee: Naveen Gangam
> Attachments: HIVE-12790.2.patch, HIVE-12790.patch, snippedLog.txt
>
>
> HiveServer2 keeps opening new connections to HMS each time it launches a
> task. These connections do not appear to be closed when the task completes
> thus causing a HMS connection leak. "lsof" for the HS2 process shows
> connections to port 9083.
> {code}
> 2015-12-03 04:20:56,352 INFO [HiveServer2-Background-Pool: Thread-424756()]:
> ql.Driver (SessionState.java:printInfo(558)) - Launching Job 11 out of 41
> 2015-12-03 04:20:56,354 INFO [Thread-405728()]: hive.metastore
> (HiveMetaStoreClient.java:open(311)) - Trying to connect to metastore with
> URI thrift://:9083
> 2015-12-03 04:20:56,360 INFO [Thread-405728()]: hive.metastore
> (HiveMetaStoreClient.java:open(351)) - Opened a connection to metastore,
> current connections: 14824
> 2015-12-03 04:20:56,360 INFO [Thread-405728()]: hive.metastore
> (HiveMetaStoreClient.java:open(400)) - Connected to metastore.
>
> 2015-12-03 04:21:06,355 INFO [HiveServer2-Background-Pool: Thread-424756()]:
> ql.Driver (SessionState.java:printInfo(558)) - Launching Job 12 out of 41
> 2015-12-03 04:21:06,357 INFO [Thread-405756()]: hive.metastore
> (HiveMetaStoreClient.java:open(311)) - Trying to connect to metastore with
> URI thrift://:9083
> 2015-12-03 04:21:06,362 INFO [Thread-405756()]: hive.metastore
> (HiveMetaStoreClient.java:open(351)) - Opened a connection to metastore,
> current connections: 14825
> 2015-12-03 04:21:06,362 INFO [Thread-405756()]: hive.metastore
> (HiveMetaStoreClient.java:open(400)) - Connected to metastore.
> ...
> 2015-12-03 04:21:08,357 INFO [HiveServer2-Background-Pool: Thread-424756()]:
> ql.Driver (SessionState.java:printInfo(558)) - Launching Job 13 out of 41
> 2015-12-03 04:21:08,360 INFO [Thread-405782()]: hive.metastore
> (HiveMetaStoreClient.java:open(311)) - Trying to connect to metastore with
> URI thrift://:9083
> 2015-12-03 04:21:08,364 INFO [Thread-405782()]: hive.metastore
> (HiveMetaStoreClient.java:open(351)) - Opened a connection to metastore,
> current connections: 14826
> 2015-12-03 04:21:08,365 INFO [Thread-405782()]: hive.metastore
> (HiveMetaStoreClient.java:open(400)) - Connected to metastore.
> ...
> {code}
> The TaskRunner thread starts a new SessionState each time, which creates a
> new connection to the HMS (via Hive.get(conf).getMSC()) that is never closed.
> Even SessionState.close(), currently not being called by the TaskRunner
> thread, does not close this connection.
> Attaching a anonymized log snippet where the number of HMS connections
> reaches north of 25000+ connections.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-12971) Hive Support for Kudu
[ https://issues.apache.org/jira/browse/HIVE-12971?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lenni Kuff updated HIVE-12971: -- Assignee: (was: Lenni Kuff) > Hive Support for Kudu > - > > Key: HIVE-12971 > URL: https://issues.apache.org/jira/browse/HIVE-12971 > Project: Hive > Issue Type: New Feature >Affects Versions: 2.0.0 >Reporter: Lenni Kuff > > JIRA for tracking work related to Hive/Kudu integration. > It would be useful to allow Kudu data to be accessible via Hive. This would > involve creating a Kudu SerDe/StorageHandler and implementing support for > QUERY and DML commands like SELECT, INSERT, UPDATE, and DELETE. Kudu > Input/OutputFormats classes already exist. The work can be staged to support > this functionality incrementally. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11675) make use of file footer PPD API in ETL strategy or separate strategy
[
https://issues.apache.org/jira/browse/HIVE-11675?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15125198#comment-15125198
]
Hive QA commented on HIVE-11675:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12785052/HIVE-11675.03.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6806/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6806/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-6806/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hive-ptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ [[ -n /usr/java/jdk1.7.0_45-cloudera ]]
+ export JAVA_HOME=/usr/java/jdk1.7.0_45-cloudera
+ JAVA_HOME=/usr/java/jdk1.7.0_45-cloudera
+ export
PATH=/usr/java/jdk1.7.0_45-cloudera/bin/:/usr/local/apache-maven-3.0.5/bin:/usr/java/jdk1.7.0_45-cloudera/bin:/usr/local/apache-ant-1.9.1/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/hiveptest/bin
+
PATH=/usr/java/jdk1.7.0_45-cloudera/bin/:/usr/local/apache-maven-3.0.5/bin:/usr/java/jdk1.7.0_45-cloudera/bin:/usr/local/apache-ant-1.9.1/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/hiveptest/bin
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'M2_OPTS=-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost
-Dhttp.proxyPort=3128'
+ M2_OPTS='-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost
-Dhttp.proxyPort=3128'
+ cd /data/hive-ptest/working/
+ tee /data/hive-ptest/logs/PreCommit-HIVE-TRUNK-Build-6806/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ cd apache-github-source-source
+ git fetch origin
+ git reset --hard HEAD
HEAD is now at 208ab35 HIVE-12727 : refactor Hive strict checks to be more
granular, allow order by no limit and no partition filter by default for now
(Sergey Shelukhin, reviewed by Xuefu Zhang) ADDENDUM2
+ git clean -f -d
+ git checkout master
Already on 'master'
+ git reset --hard origin/master
HEAD is now at 208ab35 HIVE-12727 : refactor Hive strict checks to be more
granular, allow order by no limit and no partition filter by default for now
(Sergey Shelukhin, reviewed by Xuefu Zhang) ADDENDUM2
+ git merge --ff-only origin/master
Already up-to-date.
+ git gc
+ patchCommandPath=/data/hive-ptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hive-ptest/working/scratch/build.patch
+ [[ -f /data/hive-ptest/working/scratch/build.patch ]]
+ chmod +x /data/hive-ptest/working/scratch/smart-apply-patch.sh
+ /data/hive-ptest/working/scratch/smart-apply-patch.sh
/data/hive-ptest/working/scratch/build.patch
The patch does not appear to apply with p0, p1, or p2
+ exit 1
'
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12785052 - PreCommit-HIVE-TRUNK-Build
> make use of file footer PPD API in ETL strategy or separate strategy
>
>
> Key: HIVE-11675
> URL: https://issues.apache.org/jira/browse/HIVE-11675
> Project: Hive
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: HIVE-11675.01.patch, HIVE-11675.02.patch,
> HIVE-11675.03.patch, HIVE-11675.patch
>
>
> Need to take a look at the best flow. It won't be much different if we do
> filtering metastore call for each partition. So perhaps we'd need the custom
> sync point/batching after all.
> Or we can make it opportunistic and not fetch any footers unless it can be
> pushed down to metastore or fetched from local cache, that way the only slow
> threaded op is directory listings
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-12934) Refactor llap module structure to allow for a usable client
[ https://issues.apache.org/jira/browse/HIVE-12934?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated HIVE-12934: -- Attachment: HIVE-12934.03.patch > Refactor llap module structure to allow for a usable client > --- > > Key: HIVE-12934 > URL: https://issues.apache.org/jira/browse/HIVE-12934 > Project: Hive > Issue Type: Task >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Attachments: HIVE-12934.01.patch, HIVE-12934.01.patch, > HIVE-12934.02.patch, HIVE-12934.02.review.patch, HIVE-12934.03.patch, > HIVE-12934.03.review.patch, HIVE-12934.1.patch, HIVE-12934.1.review.txt, > HIVE-12934.1.txt > > > The client isn't really usable at the moment, and all of the code resides in > the llap-server module. Restructure this so that the daemon execution code > and cache code remains in server, common components move to a different > module and relevant client pieces sit in the client module. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12934) Refactor llap module structure to allow for a usable client
[ https://issues.apache.org/jira/browse/HIVE-12934?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated HIVE-12934: -- Attachment: HIVE-12934.03.review.patch > Refactor llap module structure to allow for a usable client > --- > > Key: HIVE-12934 > URL: https://issues.apache.org/jira/browse/HIVE-12934 > Project: Hive > Issue Type: Task >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Attachments: HIVE-12934.01.patch, HIVE-12934.01.patch, > HIVE-12934.02.patch, HIVE-12934.02.review.patch, HIVE-12934.03.patch, > HIVE-12934.03.review.patch, HIVE-12934.1.patch, HIVE-12934.1.review.txt, > HIVE-12934.1.txt > > > The client isn't really usable at the moment, and all of the code resides in > the llap-server module. Restructure this so that the daemon execution code > and cache code remains in server, common components move to a different > module and relevant client pieces sit in the client module. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12969) Fix Javadoc for PredicatePushDown class
[ https://issues.apache.org/jira/browse/HIVE-12969?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Sears updated HIVE-12969: Summary: Fix Javadoc for PredicatePushDown class (was: ix Javadoc for PredicatePushDown class) > Fix Javadoc for PredicatePushDown class > --- > > Key: HIVE-12969 > URL: https://issues.apache.org/jira/browse/HIVE-12969 > Project: Hive > Issue Type: Bug > Components: Documentation >Affects Versions: 0.4.0 >Reporter: Andrew Sears >Assignee: Andrew Sears >Priority: Trivial > Attachments: hive-12969.patch > > > Fix Javadocs for hive.optimize.ppd - Default Value: true > Added In: Hive 0.4.0 with HIVE-279, default changed to true in Hive 0.4.0 > with HIVE-626 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12950) get rid of the NullScan emptyFile madness (part 1 - at least for Tez and LLAP)
[
https://issues.apache.org/jira/browse/HIVE-12950?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15125210#comment-15125210
]
Hive QA commented on HIVE-12950:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12785287/HIVE-12950.01.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 11 failed/errored test(s), 10049 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_llap_nullscan
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_metadataonly1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_optimize_nullscan
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vector_decimal_precision
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_metadataonly1
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_optimize_nullscan
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_decimal_precision
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_uri_import
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_optimize_nullscan
org.apache.hive.jdbc.TestSSL.testSSLVersion
org.apache.hive.service.cli.TestEmbeddedThriftBinaryCLIService.testExecuteStatementAsync
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6808/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6808/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-6808/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 11 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12785287 - PreCommit-HIVE-TRUNK-Build
> get rid of the NullScan emptyFile madness (part 1 - at least for Tez and LLAP)
> --
>
> Key: HIVE-12950
> URL: https://issues.apache.org/jira/browse/HIVE-12950
> Project: Hive
> Issue Type: Improvement
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: HIVE-12950.01.patch, HIVE-12950.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-12965) Insert overwrite local directory should perserve the overwritten directory permission
[ https://issues.apache.org/jira/browse/HIVE-12965?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15125202#comment-15125202 ] Chaoyu Tang commented on HIVE-12965: HIVE-12908 does not solve this issue because they are in different code path. > Insert overwrite local directory should perserve the overwritten directory > permission > - > > Key: HIVE-12965 > URL: https://issues.apache.org/jira/browse/HIVE-12965 > Project: Hive > Issue Type: Bug >Reporter: Chaoyu Tang >Assignee: Chaoyu Tang > > In Hive, "insert overwrite local directory" first deletes the overwritten > directory if exists, recreate a new one, then copy the files from src > directory to the new local directory. This process sometimes changes the > permissions of the to-be-overwritten local directory, therefore causing some > applications no more to be able to access its content. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12970) Add total open connections in HiveServer2
[ https://issues.apache.org/jira/browse/HIVE-12970?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shinichi Yamashita updated HIVE-12970: -- Attachment: HIVE-12970.1.patch > Add total open connections in HiveServer2 > - > > Key: HIVE-12970 > URL: https://issues.apache.org/jira/browse/HIVE-12970 > Project: Hive > Issue Type: Improvement > Components: Diagnosability >Reporter: Shinichi Yamashita >Priority: Minor > Attachments: HIVE-12970.1.patch > > > I add the metrics to HiveServer2 in order to confirm the change per unit > time. I will be able to use the information at the time of monitoring. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11716) Reading ACID table from non-acid session should raise an error
[
https://issues.apache.org/jira/browse/HIVE-11716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15125238#comment-15125238
]
Hive QA commented on HIVE-11716:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12785122/HIVE-11716.5.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 5 failed/errored test(s), 10032 tests
executed
*Failed tests:*
{noformat}
TestSparkCliDriver-timestamp_lazy.q-bucketsortoptimize_insert_4.q-date_udf.q-and-12-more
- did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_uri_import
org.apache.hive.jdbc.TestMultiSessionsHS2WithLocalClusterSpark.testSparkQuery
org.apache.hive.jdbc.TestSSL.testSSLVersion
org.apache.hive.service.cli.TestEmbeddedThriftBinaryCLIService.testExecuteStatementAsync
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6809/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6809/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-6809/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 5 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12785122 - PreCommit-HIVE-TRUNK-Build
> Reading ACID table from non-acid session should raise an error
> --
>
> Key: HIVE-11716
> URL: https://issues.apache.org/jira/browse/HIVE-11716
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 1.0.0
>Reporter: Eugene Koifman
>Assignee: Wei Zheng
>Priority: Critical
> Attachments: HIVE-11716.1.patch, HIVE-11716.2.patch,
> HIVE-11716.3.patch, HIVE-11716.4.patch, HIVE-11716.5.patch
>
>
> if someone performs a delete using DbTxnManager and than starts a new session
> with DummyTxnManager, then previously successfully deleted data will show up.
> Once the logic to identify all acid tables being read is there, make sure that
> Driver.recordValidTxns() called iff there ACID tables in the query.
> We should also not allow writing to ACID table from non-acid session
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-12908) Improve dynamic partition loading III
[
https://issues.apache.org/jira/browse/HIVE-12908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15125201#comment-15125201
]
Chaoyu Tang commented on HIVE-12908:
A question about this change
{code}
@@ -2723,7 +2721,10 @@ public static boolean moveFile(HiveConf conf, Path srcf,
Path destf,
try {
if (inheritPerms || replace) {
try{
- destStatus = shims.getFullFileStatus(conf, destFs,
destf.getParent());
+ destStatus = shims.getFullFileStatus(conf, destFs, destf);
+ if(destStatus.getFileStatus().isFile()) {
+destStatus = shims.getFullFileStatus(conf, destFs,
destf.getParent());
+ }
{code}
If the dest is a file, should not we preserve its initial permission instead to
inherit its parent permission after it is replaced (renamed)?
> Improve dynamic partition loading III
> -
>
> Key: HIVE-12908
> URL: https://issues.apache.org/jira/browse/HIVE-12908
> Project: Hive
> Issue Type: Improvement
> Components: Query Processor
>Reporter: Ashutosh Chauhan
>Assignee: Ashutosh Chauhan
> Attachments: HIVE-12908.2.patch, HIVE-12908.3.patch,
> HIVE-12908.4.patch, HIVE-12908.5.patch, HIVE-12908.patch
>
>
> Remove unnecessary Namenode calls.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-12969) Fix Javadoc for PredicatePushDown class
[ https://issues.apache.org/jira/browse/HIVE-12969?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-12969: Description: Fix Javadocs for hive.optimize.ppd - Default Value: true Added In: Hive 0.4.0 with HIVE-279, default changed to true in Hive 0.4.0 with HIVE-626 NO PRECOMMIT TESTS was: Fix Javadocs for hive.optimize.ppd - Default Value: true Added In: Hive 0.4.0 with HIVE-279, default changed to true in Hive 0.4.0 with HIVE-626 > Fix Javadoc for PredicatePushDown class > --- > > Key: HIVE-12969 > URL: https://issues.apache.org/jira/browse/HIVE-12969 > Project: Hive > Issue Type: Bug > Components: Documentation >Affects Versions: 0.4.0 >Reporter: Andrew Sears >Assignee: Andrew Sears >Priority: Trivial > Attachments: hive-12969.patch > > > Fix Javadocs for hive.optimize.ppd - Default Value: true > Added In: Hive 0.4.0 with HIVE-279, default changed to true in Hive 0.4.0 > with HIVE-626 > NO PRECOMMIT TESTS -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12969) Fix Javadoc for PredicatePushDown class
[ https://issues.apache.org/jira/browse/HIVE-12969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15125234#comment-15125234 ] Ashutosh Chauhan commented on HIVE-12969: - +1 > Fix Javadoc for PredicatePushDown class > --- > > Key: HIVE-12969 > URL: https://issues.apache.org/jira/browse/HIVE-12969 > Project: Hive > Issue Type: Bug > Components: Documentation >Affects Versions: 0.4.0 >Reporter: Andrew Sears >Assignee: Andrew Sears >Priority: Trivial > Attachments: hive-12969.patch > > > Fix Javadocs for hive.optimize.ppd - Default Value: true > Added In: Hive 0.4.0 with HIVE-279, default changed to true in Hive 0.4.0 > with HIVE-626 > NO PRECOMMIT TESTS -- This message was sent by Atlassian JIRA (v6.3.4#6332)
