[jira] [Updated] (HIVE-23123) Export/Import of views and materialized views
[
https://issues.apache.org/jira/browse/HIVE-23123?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Miklos Gergely updated HIVE-23123:
--
Description:
According to
[https://cwiki.apache.org/confluence/display/Hive/LanguageManual+ImportExport]
import and export can be done by using the
{code:java}
export table ...
import table ...
{code}
commands. The document doesn't mention views or materialized views at all, and
in fact we don't support commands like
{code:java}
export view ...
import view ...
export materialized view ...
import materialized view ...
{code}
they can not be parsed at all. The word table is often used though in a broader
sense, when it means all table like entities, including views and materialized
views. For example the various Table classes may represent any of these as well.
If I try to export a view with the export table ... command, it goes fine. A
_metadata file will be created, but no data directory, which is what we'd
expect. If I try to import it back, an exception is thrown due to the lack of
the data dir:
{code:java}
java.lang.AssertionError: null==getPath() for exim_view
at org.apache.hadoop.hive.ql.metadata.Hive.loadTable(Hive.java:3088)
at org.apache.hadoop.hive.ql.exec.MoveTask.execute(MoveTask.java:419)
at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:213)
at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:105)
at org.apache.hadoop.hive.ql.Executor.launchTask(Executor.java:364)
at org.apache.hadoop.hive.ql.Executor.launchTasks(Executor.java:335)
at org.apache.hadoop.hive.ql.Executor.runTasks(Executor.java:246)
at org.apache.hadoop.hive.ql.Executor.execute(Executor.java:109)
at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:722)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:491)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:485)
{code}
Still the view gets imported successfully, as data movement wasn't even
necessary.
If we try to export a materialized view which is transactional, then this
exception occurs:
{code:java}
org.apache.hadoop.hive.ql.parse.SemanticException:
org.apache.hadoop.hive.ql.metadata.InvalidTableException: Table not found
exim_materialized_view_da21d41a_9fe4_4446_9c72_d251496abf9d
at
org.apache.hadoop.hive.ql.parse.AcidExportSemanticAnalyzer.analyzeAcidExport(AcidExportSemanticAnalyzer.java:163)
at
org.apache.hadoop.hive.ql.parse.AcidExportSemanticAnalyzer.analyze(AcidExportSemanticAnalyzer.java:71)
at
org.apache.hadoop.hive.ql.parse.RewriteSemanticAnalyzer.analyzeInternal(RewriteSemanticAnalyzer.java:72)
at
org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:289)
at org.apache.hadoop.hive.ql.Compiler.analyze(Compiler.java:220)
at org.apache.hadoop.hive.ql.Compiler.compile(Compiler.java:104)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:183)
at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:601)
at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:547)
at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:541)
{code}
So the export process can not handle it, as the temporary table is not getting
created.
The import command handling have a lot of codes dedicated to importing views
and materialized views, which suggests that we support the importing (and thus
also suggests implicitly that we support the exporting) of views and
materialiezed views.
So the conclusion is that we have to decide if we support exporting/importing
of views and materialized views.
If we decide not to support them then:
- export process should throw an exception if a view or materialized view is
the subject
- the codes specific to view imports should be removed
If we decide to support them, then:
- the commands mentioned above above should be introduced
- exception should be thrown if not the proper command used (e.g. export view
on a table)
- the exceptions mentioned above should be fixed
I prefer not to support them, I don't think we should support the exporting /
importing of views. The point of exporting / importing is the transfer of data,
not DDL, it causes more issues than it solves. Our current documentation also
suggests that it is only supported for tables.
was:
According to
[https://cwiki.apache.org/confluence/display/Hive/LanguageManual+ImportExport]
import and export can be done by using the
{code:java}
export table ...
import table ...
{code}
commands. The document doesn't mention views or materialized views at all, and
in fact we don't support commands like
{code:java}
export view ...
import view ...
export materialized view ...
import materialized view ...
{code}
they can not be parsed at all. The word table is often used though in a broader
sense, when it means all table like entities, including views and materialized
views. For example the various Table classes may represent
[jira] [Commented] (HIVE-23098) Allow Operation assertState to Accept a Collection
[
https://issues.apache.org/jira/browse/HIVE-23098?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17072814#comment-17072814
]
David Mollitor commented on HIVE-23098:
---
[~ngangam] Here is another trivial review if you have any spare cycles (lol).
> Allow Operation assertState to Accept a Collection
> --
>
> Key: HIVE-23098
> URL: https://issues.apache.org/jira/browse/HIVE-23098
> Project: Hive
> Issue Type: Improvement
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Minor
> Attachments: HIVE-23098.1.patch, HIVE-23098.2.patch,
> HIVE-23098.2.patch, HIVE-23098.2.patch, HIVE-23098.3.patch
>
>
> {code:java|title=Operation.java}
> protected final void assertState(List states) throws
> HiveSQLException {
> if (!states.contains(state)) {
> throw new HiveSQLException("Expected states: " + states.toString() + ",
> but found "
> + this.state);
> }
> this.lastAccessTime = System.currentTimeMillis();
> }
> /*/
> public void someMethod() {
> assertState(new
> ArrayList(Arrays.asList(OperationState.FINISHED)));
> }
> {code}
> By allowing {{assertState}} to accept a {{Collection}}, one can save an
> allocation and simplify the code:
> {code:java}
> assertState(Collections.singleton(OperationState.FINISHED));
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (HIVE-22484) Remove Calls to printStackTrace
[
https://issues.apache.org/jira/browse/HIVE-22484?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Mollitor resolved HIVE-22484.
---
Fix Version/s: 4.0.0
Resolution: Fixed
> Remove Calls to printStackTrace
> ---
>
> Key: HIVE-22484
> URL: https://issues.apache.org/jira/browse/HIVE-22484
> Project: Hive
> Issue Type: Improvement
>Affects Versions: 3.2.0
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Major
> Fix For: 4.0.0
>
>
> In many cases, the call to {{printStackTrace}} bypasses the logging
> framework, in other cases, the error stack trace is printed and the exception
> is re-thrown (log-and-throw is a bad pattern), and then there are some other
> edge cases.
> Remove this call and replace with calls to the logging framework or remove
> altogether if exception is wrapped and re-thrown.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23110) Prevent NPE in ReExecDriver if the processing is aborted
[ https://issues.apache.org/jira/browse/HIVE-23110?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17072499#comment-17072499 ] Miklos Gergely commented on HIVE-23110: --- [~prasanth_j] do you have a log for 2) ? Should we run failure hooks on abort? Right now we don't do that (explicitly not), and I thought it was intentional as if a query is aborted, it is not a failure. Should we run them? > Prevent NPE in ReExecDriver if the processing is aborted > > > Key: HIVE-23110 > URL: https://issues.apache.org/jira/browse/HIVE-23110 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Reporter: Miklos Gergely >Assignee: Zoltan Haindrich >Priority: Major > Attachments: HIVE-23110.01.patch > > > In case of abort the context would be null, and thus the planMapper can not > be obtained from it. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23110) Prevent NPE in ReExecDriver if the processing is aborted
[
https://issues.apache.org/jira/browse/HIVE-23110?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17072509#comment-17072509
]
Prasanth Jayachandran commented on HIVE-23110:
--
I have partial logs
{code:java}
hiveserver2 <14>1 2020-03-31T20:52:24.702Z
hiveserver2-0.hiveserver2-service.compute-1585643974-lwrg.svc.cluster.local
hiveserver2 1 6ba03ff1-251f-4878-81ea-1ba72d36c465 [mdc@18060 class="ql.Driver"
level="INFO" operationLogLevel="EXECUTION"
queryId="hive_20200331205007_6397e486-03a9-41ec-a56b-e0c4ff1ff26e"
sessionId="94e0ab1a-e5ca-4237-9713-235b5dd2559a"
thread="HiveServer2-Background-Pool: Thread-74"] Executing
command(queryId=hive_20200331205007_6397e486-03a9-41ec-a56b-e0c4ff1ff26e) has
been interrupted after 133.75 seconds
hiveserver2 <14>1 2020-03-31T20:52:24.702Z
hiveserver2-0.hiveserver2-service.compute-1585643974-lwrg.svc.cluster.local
hiveserver2 1 6ba03ff1-251f-4878-81ea-1ba72d36c465 [mdc@18060 class="ql.Driver"
level="INFO" operationLogLevel="EXECUTION"
queryId="hive_20200331205007_6397e486-03a9-41ec-a56b-e0c4ff1ff26e"
sessionId="94e0ab1a-e5ca-4237-9713-235b5dd2559a"
thread="HiveServer2-Background-Pool: Thread-74"] OK
hiveserver2 <15>1 2020-03-31T20:52:24.702Z
hiveserver2-0.hiveserver2-service.compute-1585643974-lwrg.svc.cluster.local
hiveserver2 1 6ba03ff1-251f-4878-81ea-1ba72d36c465 [mdc@18060
class="log.PerfLogger" level="DEBUG" operationLogLevel="EXECUTION"
queryId="hive_20200331205007_6397e486-03a9-41ec-a56b-e0c4ff1ff26e"
sessionId="94e0ab1a-e5ca-4237-9713-235b5dd2559a"
thread="HiveServer2-Background-Pool: Thread-74"]
hiveserver2 <14>1 2020-03-31T20:52:24.711Z
hiveserver2-0.hiveserver2-service.compute-1585643974-lwrg.svc.cluster.local
hiveserver2 1 6ba03ff1-251f-4878-81ea-1ba72d36c465 [mdc@18060
class="common.LogUtils" level="INFO" thread="HiveServer2-Background-Pool:
Thread-74"] Unregistered logging context.
hiveserver2 <14>1 2020-03-31T20:52:24.702Z
hiveserver2-0.hiveserver2-service.compute-1585643974-lwrg.svc.cluster.local
hiveserver2 1 6ba03ff1-251f-4878-81ea-1ba72d36c465 [mdc@18060
class="lockmgr.DbLockManager" level="INFO" operationLogLevel="EXECUTION"
queryId="hive_20200331205007_6397e486-03a9-41ec-a56b-e0c4ff1ff26e"
sessionId="94e0ab1a-e5ca-4237-9713-235b5dd2559a"
thread="HiveServer2-Background-Pool: Thread-74"] releaseLocks:
hiveserver2 <15>1 2020-03-31T20:52:24.703Z
hiveserver2-0.hiveserver2-service.compute-1585643974-lwrg.svc.cluster.local
hiveserver2 1 6ba03ff1-251f-4878-81ea-1ba72d36c465 [mdc@18060
class="log.PerfLogger" level="DEBUG" operationLogLevel="EXECUTION"
queryId="hive_20200331205007_6397e486-03a9-41ec-a56b-e0c4ff1ff26e"
sessionId="94e0ab1a-e5ca-4237-9713-235b5dd2559a"
thread="HiveServer2-Background-Pool: Thread-74"]
hiveserver2 <11>1 2020-03-31T20:52:24.711Z
hiveserver2-0.hiveserver2-service.compute-1585643974-lwrg.svc.cluster.local
hiveserver2 1 6ba03ff1-251f-4878-81ea-1ba72d36c465 [mdc@18060
class="operation.Operation" level="ERROR" operationLogLevel="EXECUTION"
queryId="hive_20200331205007_6397e486-03a9-41ec-a56b-e0c4ff1ff26e"
sessionId="94e0ab1a-e5ca-4237-9713-235b5dd2559a"
thread="HiveServer2-Background-Pool: Thread-74"] Error running hive query:
org.apache.hive.service.cli.HiveSQLException: Illegal Operation state
transition from CANCELED to FINISHED
at
org.apache.hive.service.cli.OperationState.validateTransition(OperationState.java:97)
at
org.apache.hive.service.cli.OperationState.validateTransition(OperationState.java:103)
at
org.apache.hive.service.cli.operation.Operation.setState(Operation.java:161)
at
org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:248)
at
org.apache.hive.service.cli.operation.SQLOperation.access$700(SQLOperation.java:87)
at
org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork$1.run(SQLOperation.java:322)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1876)
at
org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork.run(SQLOperation.java:340)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
hiveserver2 2020-03-31 20:52:24,710
Log4j2-TF-1-AsyncLogger[AsyncContext@18b4aac2]-1 ERROR
/tmp/hive/operation_logs/94e0ab1a-e5ca-4237-9713-235b5dd2559a/hive_20200331205007_6397e486-03a9-41ec-a56b-e0c4ff1ff26e
was closed
[jira] [Commented] (HIVE-21492) VectorizedParquetRecordReader can't to read parquet file generated using thrift/custom tool
[
https://issues.apache.org/jira/browse/HIVE-21492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17072517#comment-17072517
]
Ganesha Shreedhara commented on HIVE-21492:
---
[~Ferd] Please review the latest patch.
> VectorizedParquetRecordReader can't to read parquet file generated using
> thrift/custom tool
> ---
>
> Key: HIVE-21492
> URL: https://issues.apache.org/jira/browse/HIVE-21492
> Project: Hive
> Issue Type: Bug
>Reporter: Ganesha Shreedhara
>Assignee: Ganesha Shreedhara
>Priority: Major
> Attachments: HIVE-21492.2.patch, HIVE-21492.3.patch, HIVE-21492.patch
>
>
> Taking an example of a parquet table having array of integers as below.
> {code:java}
> CREATE EXTERNAL TABLE ( list_of_ints` array)
> STORED AS PARQUET
> LOCATION '{location}';
> {code}
> Parquet file generated using hive will have schema for Type as below:
> {code:java}
> group list_of_ints (LIST) { repeated group bag { optional int32 array;\n};\n}
> {code}
> Parquet file generated using thrift or any custom tool (using
> org.apache.parquet.io.api.RecordConsumer)
> may have schema for Type as below:
> {code:java}
> required group list_of_ints (LIST) { repeated int32 list_of_tuple} {code}
> VectorizedParquetRecordReader handles only parquet file generated using hive.
> It throws the following exception when parquet file generated using thrift is
> read because of the changes done as part of HIVE-18553 .
> {code:java}
> Caused by: java.lang.ClassCastException: repeated int32 list_of_ints_tuple is
> not a group
> at org.apache.parquet.schema.Type.asGroupType(Type.java:207)
> at
> org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.getElementType(VectorizedParquetRecordReader.java:479)
> at
> org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.buildVectorizedParquetReader(VectorizedParquetRecordReader.java:532)
> at
> org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.checkEndOfRowGroup(VectorizedParquetRecordReader.java:440)
> at
> org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.nextBatch(VectorizedParquetRecordReader.java:401)
> at
> org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.next(VectorizedParquetRecordReader.java:353)
> at
> org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.next(VectorizedParquetRecordReader.java:92)
> at
> org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.doNext(HiveContextAwareRecordReader.java:365){code}
>
> I have done a small change to handle the case where the child type of group
> type can be PrimitiveType.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23067) Use batch DB calls in TxnHandler for commitTxn and abortTxns
[
https://issues.apache.org/jira/browse/HIVE-23067?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17072529#comment-17072529
]
Hive QA commented on HIVE-23067:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12998333/HIVE-23067.2.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 18163 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[topnkey_grouping_sets]
(batchId=1)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/21366/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/21366/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-21366/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12998333 - PreCommit-HIVE-Build
> Use batch DB calls in TxnHandler for commitTxn and abortTxns
>
>
> Key: HIVE-23067
> URL: https://issues.apache.org/jira/browse/HIVE-23067
> Project: Hive
> Issue Type: Improvement
>Reporter: Marton Bod
>Assignee: Marton Bod
>Priority: Major
> Attachments: HIVE-23067.1.patch, HIVE-23067.1.patch,
> HIVE-23067.1.patch, HIVE-23067.1.patch, HIVE-23067.2.patch
>
>
> To reduce the number of database calls and network roundtrips, we could use
> more batching in TxnHandler, where currently in many places we call insert
> commands in loops sequentially.
> Some examples:
> * openTxns (TXNS, REPL_TXN_MAP)
> * commitTxn (COMPLETED_TXN_COMPONENTS)
> * replTableWriteIdState (TXN_TO_WRITE_ID)
> * allocateTableWriteIds (TXN_TO_WRITE_ID)
> *
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23110) Prevent NPE in ReExecDriver if the processing is aborted
[ https://issues.apache.org/jira/browse/HIVE-23110?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17072531#comment-17072531 ] Zoltan Haindrich commented on HIVE-23110: - I think this issue is not connect to the NPE from ReExecDriver; because if that would be there the catch would have done a [return here|https://github.com/apache/hive/blob/d2ad5b061706a1d3cd55e59c769ed4f2af01cdbe/service/src/java/org/apache/hive/service/cli/operation/SQLOperation.java#L240] I don't think we have guarantees that after cancel an exception like that may not happen: * cleanup is synch method (not that it matters) * it sets state as first step * and then it start calling driver with .close and .destroy * as a result of that internally the driver changes to some aborted state * however..if the job is already near completion it an exception may not happen; and when the actual thread (Thread-74 in the logs) starts getting out - it doesn't throw an exception; and that is not in line with the SQLOperation's expectations; which causes the illegal transition [SQLOperation.cleanup|https://github.com/apache/hive/blob/d2ad5b061706a1d3cd55e59c769ed4f2af01cdbe/service/src/java/org/apache/hive/service/cli/operation/SQLOperation.java#L396] > Prevent NPE in ReExecDriver if the processing is aborted > > > Key: HIVE-23110 > URL: https://issues.apache.org/jira/browse/HIVE-23110 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Reporter: Miklos Gergely >Assignee: Zoltan Haindrich >Priority: Major > Attachments: HIVE-23110.01.patch > > > In case of abort the context would be null, and thus the planMapper can not > be obtained from it. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-5312) Let HiveServer2 run simultaneously in HTTP (over thrift) and Binary (normal thrift transport) mode
[ https://issues.apache.org/jira/browse/HIVE-5312?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ádám Szita updated HIVE-5312: - Attachment: HIVE-5312.3.patch > Let HiveServer2 run simultaneously in HTTP (over thrift) and Binary (normal > thrift transport) mode > --- > > Key: HIVE-5312 > URL: https://issues.apache.org/jira/browse/HIVE-5312 > Project: Hive > Issue Type: Sub-task > Components: HiveServer2 >Reporter: Vaibhav Gumashta >Assignee: Ádám Szita >Priority: Major > Attachments: HIVE-5132.branch-2.3.patch, HIVE-5312.1.patch, > HIVE-5312.2.patch, HIVE-5312.3.patch > > > [HIVE-4763|https://issues.apache.org/jira/browse/HIVE-4763] adds support for > HTTP transport over thrift. With that, HS2 can be configured to run either > using using HTTP or using normal thrift binary transport. Ideally HS2 should > be supporting both modes simultaneously and the client should be able to > specify the mode used in serving the request. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23122) LLAP TaskExecutorService should log details about task eviction decision details
[ https://issues.apache.org/jira/browse/HIVE-23122?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] László Bodor updated HIVE-23122: Status: Patch Available (was: Open) > LLAP TaskExecutorService should log details about task eviction decision > details > > > Key: HIVE-23122 > URL: https://issues.apache.org/jira/browse/HIVE-23122 > Project: Hive > Issue Type: Improvement >Reporter: László Bodor >Assignee: László Bodor >Priority: Major > Attachments: HIVE-23122.01.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23107) Remove MIN_HISTORY_LEVEL table
[
https://issues.apache.org/jira/browse/HIVE-23107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17072736#comment-17072736
]
Hive QA commented on HIVE-23107:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
43s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
45s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
27s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
2s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 1m
9s{color} | {color:blue} standalone-metastore/metastore-server in master has
190 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
40s{color} | {color:blue} ql in master has 1529 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
12s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
27s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
26s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
23s{color} | {color:red} standalone-metastore/metastore-server: The patch
generated 4 new + 669 unchanged - 2 fixed = 673 total (was 671) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
22s{color} | {color:green} standalone-metastore/metastore-server generated 0
new + 187 unchanged - 3 fixed = 187 total (was 190) {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
46s{color} | {color:green} ql in the patch passed. {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
20s{color} | {color:red} standalone-metastore_metastore-server generated 1 new
+ 24 unchanged - 0 fixed = 25 total (was 24) {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 30m 10s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-21369/dev-support/hive-personality.sh
|
| git revision | master / 94f90bc |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-21369/yetus/diff-checkstyle-standalone-metastore_metastore-server.txt
|
| javadoc |
http://104.198.109.242/logs//PreCommit-HIVE-Build-21369/yetus/diff-javadoc-javadoc-standalone-metastore_metastore-server.txt
|
| modules | C: standalone-metastore/metastore-server ql U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-21369/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Remove MIN_HISTORY_LEVEL table
> --
>
> Key: HIVE-23107
> URL: https://issues.apache.org/jira/browse/HIVE-23107
> Project: Hive
> Issue Type: Improvement
> Components: Hive
>Reporter: László Pintér
>Assignee: László Pintér
>Priority: Major
> Attachments: HIVE-23107.01.patch, HIVE-23107.02.patch
>
>
> MIN_HISTORY_LEVEL table is used in two places:
> * Cleaner uses it to decide if the files can
[jira] [Updated] (HIVE-21304) Show Bucketing version for ReduceSinkOp in explain extended plan
[ https://issues.apache.org/jira/browse/HIVE-21304?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Haindrich updated HIVE-21304: Attachment: HIVE-21304.20.patch > Show Bucketing version for ReduceSinkOp in explain extended plan > > > Key: HIVE-21304 > URL: https://issues.apache.org/jira/browse/HIVE-21304 > Project: Hive > Issue Type: Bug >Reporter: Deepak Jaiswal >Assignee: Zoltan Haindrich >Priority: Major > Attachments: HIVE-21304.01.patch, HIVE-21304.02.patch, > HIVE-21304.03.patch, HIVE-21304.04.patch, HIVE-21304.05.patch, > HIVE-21304.06.patch, HIVE-21304.07.patch, HIVE-21304.08.patch, > HIVE-21304.09.patch, HIVE-21304.10.patch, HIVE-21304.11.patch, > HIVE-21304.12.patch, HIVE-21304.13.patch, HIVE-21304.14.patch, > HIVE-21304.15.patch, HIVE-21304.16.patch, HIVE-21304.17.patch, > HIVE-21304.18.patch, HIVE-21304.19.patch, HIVE-21304.20.patch > > > Show Bucketing version for ReduceSinkOp in explain extended plan. > This helps identify what hashing algorithm is being used by by ReduceSinkOp. > > cc [~vgarg] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23122) LLAP TaskExecutorService should log details about task eviction decision details
[ https://issues.apache.org/jira/browse/HIVE-23122?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] László Bodor updated HIVE-23122: Attachment: HIVE-23122.01.patch > LLAP TaskExecutorService should log details about task eviction decision > details > > > Key: HIVE-23122 > URL: https://issues.apache.org/jira/browse/HIVE-23122 > Project: Hive > Issue Type: Improvement >Reporter: László Bodor >Assignee: László Bodor >Priority: Major > Attachments: HIVE-23122.01.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23079) Remove Calls to printStackTrace in Module hive-serde
[ https://issues.apache.org/jira/browse/HIVE-23079?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Mollitor updated HIVE-23079: -- Fix Version/s: 4.0.0 Resolution: Fixed Status: Resolved (was: Patch Available) Pushed to master. Thanks [~pvary] for the review! > Remove Calls to printStackTrace in Module hive-serde > > > Key: HIVE-23079 > URL: https://issues.apache.org/jira/browse/HIVE-23079 > Project: Hive > Issue Type: Sub-task >Reporter: David Mollitor >Assignee: David Mollitor >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-23079.1.patch, HIVE-23079.1.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23123) Export/Import of views and materialized views
[
https://issues.apache.org/jira/browse/HIVE-23123?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Miklos Gergely updated HIVE-23123:
--
Description:
According to
[https://cwiki.apache.org/confluence/display/Hive/LanguageManual+ImportExport]
import and export can be done by using the
{code:java}
export table ...
import table ...
{code}
commands. The document doesn't mention views or materialized views at all, and
in fact we don't support commands like
{code:java}
export view ...
import view ...
export materialized view ...
import materialized view ...
{code}
they can not be parsed at all. The word table is often used though in a broader
sense, when it means all table like entities, including views and materialized
views. For example the various Table classes may represent any of these as well.
If I try to export a view with the export table ... command, it goes fine. A
_metadata file will be created, but no data directory, which is what we'd
expect. If I try to import it back, an exception is thrown due to the lack of
the data dir:
{code:java}
java.lang.AssertionError: null==getPath() for exim_view
at org.apache.hadoop.hive.ql.metadata.Hive.loadTable(Hive.java:3088)
at org.apache.hadoop.hive.ql.exec.MoveTask.execute(MoveTask.java:419)
at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:213)
at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:105)
at org.apache.hadoop.hive.ql.Executor.launchTask(Executor.java:364)
at org.apache.hadoop.hive.ql.Executor.launchTasks(Executor.java:335)
at org.apache.hadoop.hive.ql.Executor.runTasks(Executor.java:246)
at org.apache.hadoop.hive.ql.Executor.execute(Executor.java:109)
at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:722)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:491)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:485)
{code}
Still the view gets imported successfully, as data movement wasn't even
necessary.
If we try to export a materialized view which is transactional, then this
exception occurs:
{code:java}
org.apache.hadoop.hive.ql.parse.SemanticException:
org.apache.hadoop.hive.ql.metadata.InvalidTableException: Table not found
exim_materialized_view_da21d41a_9fe4_4446_9c72_d251496abf9d
at
org.apache.hadoop.hive.ql.parse.AcidExportSemanticAnalyzer.analyzeAcidExport(AcidExportSemanticAnalyzer.java:163)
at
org.apache.hadoop.hive.ql.parse.AcidExportSemanticAnalyzer.analyze(AcidExportSemanticAnalyzer.java:71)
at
org.apache.hadoop.hive.ql.parse.RewriteSemanticAnalyzer.analyzeInternal(RewriteSemanticAnalyzer.java:72)
at
org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:289)
at org.apache.hadoop.hive.ql.Compiler.analyze(Compiler.java:220)
at org.apache.hadoop.hive.ql.Compiler.compile(Compiler.java:104)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:183)
at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:601)
at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:547)
at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:541)
{code}
So the export process can not handle it, as the temporary table is not getting
created.
The import command handling have a lot of codes dedicated to importing views
and materialized views, which suggests that we support the importing (and thus
also suggests implicitly that we support the exporting) of views and
materialiezed views.
So the conclusion is that we have to decide if we support exporting/importing
of views and materialized views.
If we decide not to support them then:
- export process should throw an exception if a view or materialized view is
the subject
- the codes specific to view imports should be removed
If we decide to support them, then:
- the commands mentioned above above should be introduced
- exception should be thrown if not the proper command used (e.g. export view
on a table)
- the exceptions mentioned above should be fixed
I prefer not to support them, I don't think we should support the exporting /
importing of views. The point of exporting / importing is the transfer of data,
not DDL, it causes more issues than it solves. Our current documentation also
suggests that it is only supported for tables.
was:
According to
[https://cwiki.apache.org/confluence/display/Hive/LanguageManual+ImportExport]
import and export can be done by using the
{code:java}
export table ...
import table ...
{code}
commands. The document doesn't mention views or materialized views at all, and
in fact we don't support commands like
{code:java}
export view ...
import view ...
export materialized view ...
import materialized view ...
{code}
they can not be parsed at all. The word table is often used though in a broader
sense, when it means all table like entities, including views and materialized
views. For example the various Table classes
[jira] [Commented] (HIVE-23101) Fix topnkey_grouping_sets
[
https://issues.apache.org/jira/browse/HIVE-23101?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17072549#comment-17072549
]
Peter Vary commented on HIVE-23101:
---
Pushed the patch 1 to master to unblock PreCommits.
> Fix topnkey_grouping_sets
> -
>
> Key: HIVE-23101
> URL: https://issues.apache.org/jira/browse/HIVE-23101
> Project: Hive
> Issue Type: Sub-task
>Reporter: Krisztian Kasa
>Assignee: Krisztian Kasa
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-23101.1.patch, HIVE-23101.2.patch
>
>
> Example:
> {code}
> Client Execution succeeded but contained differences (error code = 1) after
> executing topnkey_grouping_sets.q
> 260c260
> < NULL NULL
> ---
> > NULL 1
> {code}
>
> Test *topnkey_grouping_sets* fails intermittently.
> Queries which project 2 columns but order by only one of them can have more
> than one good result set:
> {code:java}
> CREATE TABLE t_test_grouping_sets(
> a int,
> b int,
> c int
> );
> INSERT INTO t_test_grouping_sets VALUES
> (NULL, NULL, NULL),
> (5, 2, 3),
> (10, 11, 12),
> (NULL, NULL, NULL),
> (NULL, NULL, NULL),
> (6, 2, 1),
> (7, 8, 4), (7, 8, 4), (7, 8, 4),
> (5, 1, 2), (5, 1, 2), (5, 1, 2),
> (NULL, NULL, NULL);
> SELECT a, b FROM t_test_grouping_sets GROUP BY GROUPING SETS ((a, b), (a),
> (b), ()) ORDER BY a LIMIT 10;
> {code}
> {code:java}
> 5 NULL
> 5 2
> 5 1
> 6 2
> 6 NULL
> 7 8
> 7 NULL
> 10NULL
> 1011
> NULL 1
> {code}
> {code:java}
> 5 NULL
> 5 2
> 5 1
> 6 2
> 6 NULL
> 7 8
> 7 NULL
> 10NULL
> 1011
> NULL NULL
> {code}
> Since we don't order by *b* both result sets are valid.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23067) Use batch DB calls in TxnHandler for commitTxn and abortTxns
[ https://issues.apache.org/jira/browse/HIVE-23067?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marton Bod updated HIVE-23067: -- Attachment: HIVE-23067.2.patch > Use batch DB calls in TxnHandler for commitTxn and abortTxns > > > Key: HIVE-23067 > URL: https://issues.apache.org/jira/browse/HIVE-23067 > Project: Hive > Issue Type: Improvement >Reporter: Marton Bod >Assignee: Marton Bod >Priority: Major > Attachments: HIVE-23067.1.patch, HIVE-23067.1.patch, > HIVE-23067.1.patch, HIVE-23067.1.patch, HIVE-23067.2.patch, HIVE-23067.2.patch > > > To reduce the number of database calls and network roundtrips, we could use > more batching in TxnHandler, where currently in many places we call insert > commands in loops sequentially. > Some examples: > * openTxns (TXNS, REPL_TXN_MAP) > * commitTxn (COMPLETED_TXN_COMPONENTS) > * replTableWriteIdState (TXN_TO_WRITE_ID) > * allocateTableWriteIds (TXN_TO_WRITE_ID) > * > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23113) Clean Up HiveCallableStatement
[
https://issues.apache.org/jira/browse/HIVE-23113?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17072699#comment-17072699
]
Hive QA commented on HIVE-23113:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12998347/HIVE-23113.1.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 18162 tests
executed
*Failed tests:*
{noformat}
org.apache.hive.beeline.TestBeeLineWithArgs.testRowsAffected (batchId=286)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/21368/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/21368/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-21368/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12998347 - PreCommit-HIVE-Build
> Clean Up HiveCallableStatement
> --
>
> Key: HIVE-23113
> URL: https://issues.apache.org/jira/browse/HIVE-23113
> Project: Hive
> Issue Type: Improvement
> Components: JDBC
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Trivial
> Attachments: HIVE-23113.1.patch
>
>
> * Add a useful class comment
> * Remove all non-javadoc comments
> * Remove 'TODO' tags
> * Add {{@override}} tags
> * Checkstyle formatting
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23039) Checkpointing for repl dump bootstrap phase
[ https://issues.apache.org/jira/browse/HIVE-23039?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aasha Medhi updated HIVE-23039: --- Attachment: HIVE-23039.02.patch Status: Patch Available (was: In Progress) > Checkpointing for repl dump bootstrap phase > --- > > Key: HIVE-23039 > URL: https://issues.apache.org/jira/browse/HIVE-23039 > Project: Hive > Issue Type: Bug >Reporter: Aasha Medhi >Assignee: Aasha Medhi >Priority: Major > Labels: pull-request-available > Attachments: HIVE-23039.01.patch, HIVE-23039.02.patch > > Time Spent: 50m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23122) LLAP TaskExecutorService should log details about task eviction decision details
[
https://issues.apache.org/jira/browse/HIVE-23122?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
László Bodor updated HIVE-23122:
Description:
TaskExecutorService maintains a waitQueue, and can evict a task in favor of
another. Under the hood, the queue uses a configurable
[comparator|https://github.com/apache/hive/tree/master/llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/comparator].
The currently available comparators typically use the following properties of
a task(wrapper):
getWithinDagPriority: related to vertex
currentAttemptStartTime
firstAttemptStartTime
knownPending: remaining upstream tasks
The problem is, when an eviction happens, the INFO level message doesn't
provide any insight about the decision, only attempts ids like below:
{code}
attempt_1585248378306_0010_72_02_96_8 evicted from wait queue in favor of
attempt_1585248378306_0003_175_02_79_175 because of lower priority
{code}
was:
TaskExecutorService maintains a waitQueue, and can evict a task in favor of
another. Under the hood, the queue uses a configurable
[comparator|https://github.com/apache/hive/tree/master/llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/comparator].
The currently available comparators typically use the following properties of
a task:
getWithinDagPriority: related to vertex
> LLAP TaskExecutorService should log details about task eviction decision
> details
>
>
> Key: HIVE-23122
> URL: https://issues.apache.org/jira/browse/HIVE-23122
> Project: Hive
> Issue Type: Improvement
>Reporter: László Bodor
>Assignee: László Bodor
>Priority: Major
> Attachments: HIVE-23122.01.patch
>
>
> TaskExecutorService maintains a waitQueue, and can evict a task in favor of
> another. Under the hood, the queue uses a configurable
> [comparator|https://github.com/apache/hive/tree/master/llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/comparator].
> The currently available comparators typically use the following properties
> of a task(wrapper):
> getWithinDagPriority: related to vertex
> currentAttemptStartTime
> firstAttemptStartTime
> knownPending: remaining upstream tasks
> The problem is, when an eviction happens, the INFO level message doesn't
> provide any insight about the decision, only attempts ids like below:
> {code}
> attempt_1585248378306_0010_72_02_96_8 evicted from wait queue in favor of
> attempt_1585248378306_0003_175_02_79_175 because of lower priority
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23052) Optimize lock enqueueing in TxnHandler
[ https://issues.apache.org/jira/browse/HIVE-23052?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marton Bod updated HIVE-23052: -- Attachment: HIVE-23052.14.patch > Optimize lock enqueueing in TxnHandler > -- > > Key: HIVE-23052 > URL: https://issues.apache.org/jira/browse/HIVE-23052 > Project: Hive > Issue Type: Improvement >Reporter: Marton Bod >Assignee: Marton Bod >Priority: Major > Attachments: HIVE-23052.1.patch, HIVE-23052.10.patch, > HIVE-23052.11.patch, HIVE-23052.12.patch, HIVE-23052.13.patch, > HIVE-23052.14.patch, HIVE-23052.2.patch, HIVE-23052.3.patch, > HIVE-23052.4.patch, HIVE-23052.5.patch, HIVE-23052.6.patch, > HIVE-23052.7.patch, HIVE-23052.8.patch, HIVE-23052.8.patch, > HIVE-23052.8.patch, HIVE-23052.9.patch > > > * Reduce scope of next_lock_id select-for-update by moving the txn_component > inserts before the S4U + inserting the hive_locks entries before the S4U > (first with a temp ID, which will be replaced later in a single update). This > helps decrease the overall time that the next_lock_id table is locked, > thereby increasing concurrency > * Insert txn_components in a batch instead of one-by-one (also in > TxnHandler::addDynamicPartition) > * Increment next_lock_id and update hive_locks table in a single batch > statement > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23039) Checkpointing for repl dump bootstrap phase
[ https://issues.apache.org/jira/browse/HIVE-23039?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aasha Medhi updated HIVE-23039: --- Status: In Progress (was: Patch Available) > Checkpointing for repl dump bootstrap phase > --- > > Key: HIVE-23039 > URL: https://issues.apache.org/jira/browse/HIVE-23039 > Project: Hive > Issue Type: Bug >Reporter: Aasha Medhi >Assignee: Aasha Medhi >Priority: Major > Labels: pull-request-available > Attachments: HIVE-23039.01.patch, HIVE-23039.02.patch > > Time Spent: 50m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23113) Clean Up HiveCallableStatement
[
https://issues.apache.org/jira/browse/HIVE-23113?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Mollitor updated HIVE-23113:
--
Attachment: HIVE-23113.1.patch
> Clean Up HiveCallableStatement
> --
>
> Key: HIVE-23113
> URL: https://issues.apache.org/jira/browse/HIVE-23113
> Project: Hive
> Issue Type: Improvement
> Components: JDBC
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Trivial
> Attachments: HIVE-23113.1.patch, HIVE-23113.1.patch
>
>
> * Add a useful class comment
> * Remove all non-javadoc comments
> * Remove 'TODO' tags
> * Add {{@override}} tags
> * Checkstyle formatting
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23123) Export/Import of views and materialized views
[
https://issues.apache.org/jira/browse/HIVE-23123?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Miklos Gergely updated HIVE-23123:
--
Description:
According to
[https://cwiki.apache.org/confluence/display/Hive/LanguageManual+ImportExport]
import and export can be done by using the
{code:java}
export table ...
import table ...
{code}
commands. The document doesn't mention views or materialized views at all, and
in fact we don't support commands like
{code:java}
export view ...
import view ...
export materialized view ...
import materialized view ...
{code}
they can not be parsed at all. The word table is often used though in a broader
sense, when it means all table like entities, including views and materialized
views. For example the various Table classes may represent any of these as well.
If I try to export a view with the export table ... command, it goes fine. A
_metadata file will be created, but no data directory, which is what we'd
expect. If I try to import it back, an exception is thrown due to the lack of
the data dir:
{code:java}
java.lang.AssertionError: null==getPath() for exim_view
at org.apache.hadoop.hive.ql.metadata.Hive.loadTable(Hive.java:3088)
at org.apache.hadoop.hive.ql.exec.MoveTask.execute(MoveTask.java:419)
at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:213)
at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:105)
at org.apache.hadoop.hive.ql.Executor.launchTask(Executor.java:364)
at org.apache.hadoop.hive.ql.Executor.launchTasks(Executor.java:335)
at org.apache.hadoop.hive.ql.Executor.runTasks(Executor.java:246)
at org.apache.hadoop.hive.ql.Executor.execute(Executor.java:109)
at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:722)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:491)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:485)
{code}
Still the view gets imported successfully, as data movement wasn't even
necessary.
If we try to export a materialized view which is transactional, then this
exception occurs:
{code:java}
org.apache.hadoop.hive.ql.parse.SemanticException:
org.apache.hadoop.hive.ql.metadata.InvalidTableException: Table not found
exim_materialized_view_da21d41a_9fe4_4446_9c72_d251496abf9d
at
org.apache.hadoop.hive.ql.parse.AcidExportSemanticAnalyzer.analyzeAcidExport(AcidExportSemanticAnalyzer.java:163)
at
org.apache.hadoop.hive.ql.parse.AcidExportSemanticAnalyzer.analyze(AcidExportSemanticAnalyzer.java:71)
at
org.apache.hadoop.hive.ql.parse.RewriteSemanticAnalyzer.analyzeInternal(RewriteSemanticAnalyzer.java:72)
at
org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:289)
at org.apache.hadoop.hive.ql.Compiler.analyze(Compiler.java:220)
at org.apache.hadoop.hive.ql.Compiler.compile(Compiler.java:104)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:183)
at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:601)
at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:547)
at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:541)
{code}
So the export process can not handle it, as the temporary table is not getting
created.
The import command handling have a lot of codes dedicated to importing views
and materialized views, which suggests that we support the importing (and thus
also suggests implicitly that we support the exporting) of views and
materialiezed views.
So the conclusion is that we have to decide if we support exporting/importing
of views and materialized views.
If we decide not to support them then:
- export process should throw an exception if a view or materialized view is
the subject
- the codes specific to view imports should be removed
If we decide to support them, then:
- the commands mentioned above above should be introduced
- exception should be thrown if not the proper command used (e.g. export view
on a table)
- the exceptions mentioned above should be fixed
I prefer #1, I don't think we should support the exporting / importing of
views. The point of exporting / importing is the transfer of data, not DDL, it
causes more issues than it solves. Our current documentation also suggests that
it is only supported for tables.
was:
According to
[https://cwiki.apache.org/confluence/display/Hive/LanguageManual+ImportExport]
import and export can be done by using the
{code:java}
export table ...
import table ...
{code}
commands. The document doesn't mention views or materialized views at all, and
in fact we don't support commands like
{code:java}
export view ...
import view ...
export materialized view ...
import materialized view ...
{code}
they can not be parsed at all. The word table is often used though in a broader
sense, when it means all table like entities, including views and materialized
views. For example the various Table classes may represent any
[jira] [Updated] (HIVE-23123) Export/Import of views and materialized views
[
https://issues.apache.org/jira/browse/HIVE-23123?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Miklos Gergely updated HIVE-23123:
--
Description:
According to
[https://cwiki.apache.org/confluence/display/Hive/LanguageManual+ImportExport]
import and export can be done by using the
{code:java}
export table ...
import table ...
{code}
commands. The document doesn't mention views or materialized views at all, and
in fact we don't support commands like
{code:java}
export view ...
import view ...
export materialized view ...
import materialized view ...
{code}
they can not be parsed at all. The word table is often used though in a broader
sense, when it means all table like entities, including views and materialized
views. For example the various Table classes may represent any of these as well.
If I try to export a view with the export table ... command, it goes fine. A
_metadata file will be created, but no data directory, which is what we'd
expect. If I try to import it back, an exception is thrown due to the lack of
the data dir:
{code:java}
java.lang.AssertionError: null==getPath() for exim_view
at org.apache.hadoop.hive.ql.metadata.Hive.loadTable(Hive.java:3088)
at org.apache.hadoop.hive.ql.exec.MoveTask.execute(MoveTask.java:419)
at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:213)
at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:105)
at org.apache.hadoop.hive.ql.Executor.launchTask(Executor.java:364)
at org.apache.hadoop.hive.ql.Executor.launchTasks(Executor.java:335)
at org.apache.hadoop.hive.ql.Executor.runTasks(Executor.java:246)
at org.apache.hadoop.hive.ql.Executor.execute(Executor.java:109)
at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:722)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:491)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:485)
{code}
Still the view gets imported successfully, as data movement wasn't even
necessary.
If we try to export a materialized view which is transactional, then this
exception occurs:
{code:java}
org.apache.hadoop.hive.ql.parse.SemanticException:
org.apache.hadoop.hive.ql.metadata.InvalidTableException: Table not found
exim_materialized_view_da21d41a_9fe4_4446_9c72_d251496abf9d
at
org.apache.hadoop.hive.ql.parse.AcidExportSemanticAnalyzer.analyzeAcidExport(AcidExportSemanticAnalyzer.java:163)
at
org.apache.hadoop.hive.ql.parse.AcidExportSemanticAnalyzer.analyze(AcidExportSemanticAnalyzer.java:71)
at
org.apache.hadoop.hive.ql.parse.RewriteSemanticAnalyzer.analyzeInternal(RewriteSemanticAnalyzer.java:72)
at
org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:289)
at org.apache.hadoop.hive.ql.Compiler.analyze(Compiler.java:220)
at org.apache.hadoop.hive.ql.Compiler.compile(Compiler.java:104)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:183)
at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:601)
at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:547)
at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:541)
{code}
So the export process can not handle it, as the temporary table is not getting
created.
The import command handling have a lot of codes dedicated to importing views
and materialized views, which suggests that we support the importing (and thus
also suggests implicitly that we support the exporting) of views and
materialiezed views.
So the conclusion is that we have to decide if we support exporting/importing
of views and materialized views.
# If we decide not to support them then:
#
- export process should throw an exception if a view or materialized view is
the subject
- the codes specific to view imports should be removed
# If we decide to support them, then:
- the commands mentioned above above should be introduced
- exception should be thrown if not the proper command used (e.g. export view
on a table)
- the exceptions mentioned above should be fixed
I prefer #1, I don't think we should support the exporting / importing of
views. The point of exporting / importing is the transfer of data, not DDL, it
causes more issues than it solves. Our current documentation also suggests that
it is only supported for tables.
was:
According to
https://cwiki.apache.org/confluence/display/Hive/LanguageManual+ImportExport
import and export can be done by using the
{code:java}
export table ...
import table ...
{code}
commands. The document doesn't mention views or materialized views at all, and
in fact we don't support commands like
{code:java}
export view ...
import view ...
export materialized view ...
import materialized view ...
{code}
they can not be parsed at all. The word table is often used though in a broader
sense, when it means all table like entities, including views and materialized
views. For example the various Table classes may
[jira] [Work logged] (HIVE-22821) Add necessary endpoints for proactive cache eviction
[
https://issues.apache.org/jira/browse/HIVE-22821?focusedWorklogId=413925=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-413925
]
ASF GitHub Bot logged work on HIVE-22821:
-
Author: ASF GitHub Bot
Created on: 01/Apr/20 09:30
Start Date: 01/Apr/20 09:30
Worklog Time Spent: 10m
Work Description: szlta commented on pull request #909: HIVE-22821
URL: https://github.com/apache/hive/pull/909#discussion_r401479045
##
File path: ql/src/java/org/apache/hadoop/hive/llap/ProactiveEviction.java
##
@@ -0,0 +1,311 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hive.llap;
+
+import java.io.IOException;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.LinkedHashMap;
+import java.util.LinkedHashSet;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.TimeUnit;
+import javax.net.SocketFactory;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hive.common.io.CacheTag;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.llap.daemon.rpc.LlapDaemonProtocolProtos;
+import org.apache.hadoop.hive.llap.impl.LlapManagementProtocolClientImpl;
+import org.apache.hadoop.hive.llap.registry.LlapServiceInstance;
+import org.apache.hadoop.hive.llap.registry.impl.LlapRegistryService;
+import org.apache.hadoop.io.retry.RetryPolicies;
+import org.apache.hadoop.io.retry.RetryPolicy;
+import org.apache.hadoop.net.NetUtils;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Through this class the caller (typically HS2) can request eviction of
buffers from LLAP cache by specifying a DB,
+ * table or partition name/(value). Request sending is implemented here.
+ */
+public final class ProactiveEviction {
+
+ private ProactiveEviction() {
+// Not to be used;
+ }
+
+ /**
+ * Trigger LLAP cache eviction of buffers related to entities residing in
request parameter.
+ * @param conf
+ * @param request
+ */
+ public static void evict(Configuration conf, Request request) {
+if (!HiveConf.getBoolVar(conf,
HiveConf.ConfVars.LLAP_IO_PROACTIVE_EVICTION_ENABLED)) {
+ return;
+}
+
+try {
+ LlapRegistryService llapRegistryService =
LlapRegistryService.getClient(conf);
+ Collection instances =
llapRegistryService.getInstances().getAll();
+ if (instances.size() == 0) {
+// Not in LLAP mode.
+return;
+ }
+ ExecutorService executorService = Executors.newCachedThreadPool();
Review comment:
Hi @b-slim can you take a look on the recent commit please?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 413925)
Time Spent: 5h (was: 4h 50m)
> Add necessary endpoints for proactive cache eviction
>
>
> Key: HIVE-22821
> URL: https://issues.apache.org/jira/browse/HIVE-22821
> Project: Hive
> Issue Type: Sub-task
> Components: llap
>Reporter: Ádám Szita
>Assignee: Ádám Szita
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-22821.0.patch, HIVE-22821.1.patch,
> HIVE-22821.2.patch
>
> Time Spent: 5h
> Remaining Estimate: 0h
>
> Implement the parts required for iHS2 -> LLAP daemons communication:
> * protobuf message schema and endpoints
> * Hive configuration
> * for use cases:
> ** dropping db
> ** dropping table
> ** dropping partition from a table
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23076) Add batching for openTxn
[
https://issues.apache.org/jira/browse/HIVE-23076?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17072575#comment-17072575
]
Hive QA commented on HIVE-23076:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
59s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
26s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
23s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 1m
14s{color} | {color:blue} standalone-metastore/metastore-server in master has
190 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
20s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
27s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
22s{color} | {color:red} standalone-metastore/metastore-server: The patch
generated 1 new + 546 unchanged - 0 fixed = 547 total (was 546) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
24s{color} | {color:green} standalone-metastore/metastore-server generated 0
new + 189 unchanged - 1 fixed = 189 total (was 190) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
20s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 15m 8s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-21367/dev-support/hive-personality.sh
|
| git revision | master / 94f90bc |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-21367/yetus/diff-checkstyle-standalone-metastore_metastore-server.txt
|
| modules | C: standalone-metastore/metastore-server U:
standalone-metastore/metastore-server |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-21367/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Add batching for openTxn
>
>
> Key: HIVE-23076
> URL: https://issues.apache.org/jira/browse/HIVE-23076
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Reporter: Peter Vary
>Assignee: Peter Vary
>Priority: Major
> Attachments: HIVE-23076.2.patch, HIVE-23076.3.patch,
> HIVE-23076.4.patch, HIVE-23076.5.patch, HIVE-23076.6.patch,
> HIVE-23076.7.patch, HIVE-23076.8.patch, HIVE-23076.9.patch, HIVE-23076.patch
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23076) Add batching for openTxn
[
https://issues.apache.org/jira/browse/HIVE-23076?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17072629#comment-17072629
]
Hive QA commented on HIVE-23076:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12998336/HIVE-23076.9.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 18163 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/21367/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/21367/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-21367/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12998336 - PreCommit-HIVE-Build
> Add batching for openTxn
>
>
> Key: HIVE-23076
> URL: https://issues.apache.org/jira/browse/HIVE-23076
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Reporter: Peter Vary
>Assignee: Peter Vary
>Priority: Major
> Attachments: HIVE-23076.2.patch, HIVE-23076.3.patch,
> HIVE-23076.4.patch, HIVE-23076.5.patch, HIVE-23076.6.patch,
> HIVE-23076.7.patch, HIVE-23076.8.patch, HIVE-23076.9.patch, HIVE-23076.patch
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-21304) Show Bucketing version for ReduceSinkOp in explain extended plan
[ https://issues.apache.org/jira/browse/HIVE-21304?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Haindrich updated HIVE-21304: Attachment: HIVE-21304.19.patch > Show Bucketing version for ReduceSinkOp in explain extended plan > > > Key: HIVE-21304 > URL: https://issues.apache.org/jira/browse/HIVE-21304 > Project: Hive > Issue Type: Bug >Reporter: Deepak Jaiswal >Assignee: Zoltan Haindrich >Priority: Major > Attachments: HIVE-21304.01.patch, HIVE-21304.02.patch, > HIVE-21304.03.patch, HIVE-21304.04.patch, HIVE-21304.05.patch, > HIVE-21304.06.patch, HIVE-21304.07.patch, HIVE-21304.08.patch, > HIVE-21304.09.patch, HIVE-21304.10.patch, HIVE-21304.11.patch, > HIVE-21304.12.patch, HIVE-21304.13.patch, HIVE-21304.14.patch, > HIVE-21304.15.patch, HIVE-21304.16.patch, HIVE-21304.17.patch, > HIVE-21304.18.patch, HIVE-21304.19.patch > > > Show Bucketing version for ReduceSinkOp in explain extended plan. > This helps identify what hashing algorithm is being used by by ReduceSinkOp. > > cc [~vgarg] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-23122) LLAP TaskSchedulerService should log details about task eviction decision details
[ https://issues.apache.org/jira/browse/HIVE-23122?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] László Bodor reassigned HIVE-23122: --- Assignee: László Bodor > LLAP TaskSchedulerService should log details about task eviction decision > details > - > > Key: HIVE-23122 > URL: https://issues.apache.org/jira/browse/HIVE-23122 > Project: Hive > Issue Type: Improvement >Reporter: László Bodor >Assignee: László Bodor >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23122) LLAP TaskExecutorService should log details about task eviction decision details
[ https://issues.apache.org/jira/browse/HIVE-23122?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] László Bodor updated HIVE-23122: Summary: LLAP TaskExecutorService should log details about task eviction decision details (was: LLAP TaskSchedulerService should log details about task eviction decision details) > LLAP TaskExecutorService should log details about task eviction decision > details > > > Key: HIVE-23122 > URL: https://issues.apache.org/jira/browse/HIVE-23122 > Project: Hive > Issue Type: Improvement >Reporter: László Bodor >Assignee: László Bodor >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23122) LLAP TaskExecutorService should log details about task eviction decision details
[ https://issues.apache.org/jira/browse/HIVE-23122?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] László Bodor updated HIVE-23122: Description: TaskExecutorService maintains a waitQueue, and can evict a task in favor of another. Under the hood, the queue uses a configurable [comparator|https://github.com/apache/hive/tree/master/llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/comparator]. The currently available comparators typically use the following properties of a task: getWithinDagPriority: related to vertex > LLAP TaskExecutorService should log details about task eviction decision > details > > > Key: HIVE-23122 > URL: https://issues.apache.org/jira/browse/HIVE-23122 > Project: Hive > Issue Type: Improvement >Reporter: László Bodor >Assignee: László Bodor >Priority: Major > Attachments: HIVE-23122.01.patch > > > TaskExecutorService maintains a waitQueue, and can evict a task in favor of > another. Under the hood, the queue uses a configurable > [comparator|https://github.com/apache/hive/tree/master/llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/comparator]. > The currently available comparators typically use the following properties > of a task: > getWithinDagPriority: related to vertex -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23122) LLAP TaskExecutorService should log details about task eviction decision details
[
https://issues.apache.org/jira/browse/HIVE-23122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17072716#comment-17072716
]
Panagiotis Garefalakis commented on HIVE-23122:
---
Thanks for the patch [~abstractdog] – pretty useful – would it make sense to
add the canFinishForPriority property used by the BaseComparator as well?
> LLAP TaskExecutorService should log details about task eviction decision
> details
>
>
> Key: HIVE-23122
> URL: https://issues.apache.org/jira/browse/HIVE-23122
> Project: Hive
> Issue Type: Improvement
>Reporter: László Bodor
>Assignee: László Bodor
>Priority: Major
> Attachments: HIVE-23122.01.patch
>
>
> TaskExecutorService maintains a waitQueue, and can evict a task in favor of
> another. Under the hood, the queue uses a configurable
> [comparator|https://github.com/apache/hive/tree/master/llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/comparator].
> The currently available comparators typically use the following properties
> of a task(wrapper):
> getWithinDagPriority: related to vertex
> currentAttemptStartTime
> firstAttemptStartTime
> knownPending: remaining upstream tasks
> The problem is, when an eviction happens, the INFO level message doesn't
> provide any insight about the decision, only attempts ids like below:
> {code}
> attempt_1585248378306_0010_72_02_96_8 evicted from wait queue in favor of
> attempt_1585248378306_0003_175_02_79_175 because of lower priority
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23113) Clean Up HiveCallableStatement
[
https://issues.apache.org/jira/browse/HIVE-23113?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17072649#comment-17072649
]
Hive QA commented on HIVE-23113:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
59s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
18s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
12s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
29s{color} | {color:blue} jdbc in master has 16 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
13s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
11s{color} | {color:green} jdbc: The patch generated 0 new + 0 unchanged - 1
fixed = 0 total (was 1) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
13s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 12m 55s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-21368/dev-support/hive-personality.sh
|
| git revision | master / 94f90bc |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| modules | C: jdbc U: jdbc |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-21368/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Clean Up HiveCallableStatement
> --
>
> Key: HIVE-23113
> URL: https://issues.apache.org/jira/browse/HIVE-23113
> Project: Hive
> Issue Type: Improvement
> Components: JDBC
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Trivial
> Attachments: HIVE-23113.1.patch
>
>
> * Add a useful class comment
> * Remove all non-javadoc comments
> * Remove 'TODO' tags
> * Add {{@override}} tags
> * Checkstyle formatting
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (HIVE-23121) Re-examine TestWarehouseExternalDir to see if it uses HMS translation.
[ https://issues.apache.org/jira/browse/HIVE-23121?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Naveen Gangam reassigned HIVE-23121: > Re-examine TestWarehouseExternalDir to see if it uses HMS translation. > -- > > Key: HIVE-23121 > URL: https://issues.apache.org/jira/browse/HIVE-23121 > Project: Hive > Issue Type: Sub-task >Reporter: Naveen Gangam >Assignee: Naveen Gangam >Priority: Major > > TestWarehouseExternalDir currently passes with just one change related to > HIVE-22995. But that change was assuming it was using HMS Translation to > convert non-acid managed table to external. > Ensure that it still does. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-23039) Checkpointing for repl dump bootstrap phase
[
https://issues.apache.org/jira/browse/HIVE-23039?focusedWorklogId=413961=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-413961
]
ASF GitHub Bot logged work on HIVE-23039:
-
Author: ASF GitHub Bot
Created on: 01/Apr/20 10:23
Start Date: 01/Apr/20 10:23
Worklog Time Spent: 10m
Work Description: anishek commented on pull request #965: HIVE-23039
Checkpointing for repl dump bootstrap phase
URL: https://github.com/apache/hive/pull/965#discussion_r401496956
##
File path: ql/src/java/org/apache/hadoop/hive/ql/exec/repl/util/ReplUtils.java
##
@@ -296,4 +300,17 @@ public static boolean includeAcidTableInDump(HiveConf
conf) {
public static boolean tableIncludedInReplScope(ReplScope replScope, String
tableName) {
return ((replScope == null) ||
replScope.tableIncludedInReplScope(tableName));
}
+
+ public static boolean dataCopyCompleted(Path toPath, HiveConf conf) throws
IOException {
+FileSystem dstFs = null;
+dstFs = toPath.getFileSystem(conf);
+if (dstFs.exists(new Path(toPath, ReplUtils.COPY_ACKNOWLEDGEMENT))) {
+ return true;
+}
+return false;
+ }
+
+ public static void setDataCopyComplete(Path toPath, HiveConf conf) throws
SemanticException {
Review comment:
rename method to ackCopy() ?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 413961)
Time Spent: 50m (was: 40m)
> Checkpointing for repl dump bootstrap phase
> ---
>
> Key: HIVE-23039
> URL: https://issues.apache.org/jira/browse/HIVE-23039
> Project: Hive
> Issue Type: Bug
>Reporter: Aasha Medhi
>Assignee: Aasha Medhi
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-23039.01.patch
>
> Time Spent: 50m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-23039) Checkpointing for repl dump bootstrap phase
[
https://issues.apache.org/jira/browse/HIVE-23039?focusedWorklogId=413957=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-413957
]
ASF GitHub Bot logged work on HIVE-23039:
-
Author: ASF GitHub Bot
Created on: 01/Apr/20 10:23
Start Date: 01/Apr/20 10:23
Worklog Time Spent: 10m
Work Description: anishek commented on pull request #965: HIVE-23039
Checkpointing for repl dump bootstrap phase
URL: https://github.com/apache/hive/pull/965#discussion_r401486847
##
File path:
itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/parse/TestReplicationScenariosExternalTables.java
##
@@ -906,6 +908,131 @@ public void replicationWithTableNameContainsKeywords()
throws Throwable {
.verifyReplTargetProperty(replicatedDbName);
}
+ @Test
+ public void testCheckPointing() throws Throwable {
+List withClauseOptions = externalTableBasePathWithClause();
+WarehouseInstance.Tuple bootstrapDump = primary.run("use " + primaryDbName)
+.run("CREATE TABLE t1(a string) STORED AS TEXTFILE")
+.run("CREATE EXTERNAL TABLE t2(a string) STORED AS TEXTFILE")
+.run("insert into t1 values (1)")
+.run("insert into t1 values (2)")
+.run("insert into t2 values (11)")
+.run("insert into t2 values (21)")
+.dump(primaryDbName, withClauseOptions);
+
+// verify that the external table info is written correctly for bootstrap
+assertExternalFileInfo(Arrays.asList("t2"), bootstrapDump.dumpLocation,
primaryDbName);
Review comment:
might be better to use another location here completely, sometimes since we
are writing to same dump location and also the db directory is there you might
have files in locations that are slight off and you wont realize it till you
run actual prod scenario
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 413957)
Time Spent: 20m (was: 10m)
> Checkpointing for repl dump bootstrap phase
> ---
>
> Key: HIVE-23039
> URL: https://issues.apache.org/jira/browse/HIVE-23039
> Project: Hive
> Issue Type: Bug
>Reporter: Aasha Medhi
>Assignee: Aasha Medhi
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-23039.01.patch
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-23039) Checkpointing for repl dump bootstrap phase
[
https://issues.apache.org/jira/browse/HIVE-23039?focusedWorklogId=413959=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-413959
]
ASF GitHub Bot logged work on HIVE-23039:
-
Author: ASF GitHub Bot
Created on: 01/Apr/20 10:23
Start Date: 01/Apr/20 10:23
Worklog Time Spent: 10m
Work Description: anishek commented on pull request #965: HIVE-23039
Checkpointing for repl dump bootstrap phase
URL: https://github.com/apache/hive/pull/965#discussion_r401491670
##
File path: ql/src/java/org/apache/hadoop/hive/ql/plan/ReplCopyWork.java
##
@@ -120,4 +122,12 @@ public boolean isNeedCheckDuplicateCopy() {
public void setCheckDuplicateCopy(boolean flag) {
checkDuplicateCopy = flag;
}
+
+ public boolean isCheckpointEnabled() {
+return checkpointEnabled;
+ }
+
+ public void setCheckpointEnabled(boolean checkpointEnabled) {
Review comment:
do initialization based on constructor rather than setters ?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 413959)
> Checkpointing for repl dump bootstrap phase
> ---
>
> Key: HIVE-23039
> URL: https://issues.apache.org/jira/browse/HIVE-23039
> Project: Hive
> Issue Type: Bug
>Reporter: Aasha Medhi
>Assignee: Aasha Medhi
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-23039.01.patch
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-23039) Checkpointing for repl dump bootstrap phase
[
https://issues.apache.org/jira/browse/HIVE-23039?focusedWorklogId=413958=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-413958
]
ASF GitHub Bot logged work on HIVE-23039:
-
Author: ASF GitHub Bot
Created on: 01/Apr/20 10:23
Start Date: 01/Apr/20 10:23
Worklog Time Spent: 10m
Work Description: anishek commented on pull request #965: HIVE-23039
Checkpointing for repl dump bootstrap phase
URL: https://github.com/apache/hive/pull/965#discussion_r401496558
##
File path: ql/src/java/org/apache/hadoop/hive/ql/exec/repl/util/ReplUtils.java
##
@@ -296,4 +300,17 @@ public static boolean includeAcidTableInDump(HiveConf
conf) {
public static boolean tableIncludedInReplScope(ReplScope replScope, String
tableName) {
return ((replScope == null) ||
replScope.tableIncludedInReplScope(tableName));
}
+
+ public static boolean dataCopyCompleted(Path toPath, HiveConf conf) throws
IOException {
+FileSystem dstFs = null;
+dstFs = toPath.getFileSystem(conf);
+if (dstFs.exists(new Path(toPath, ReplUtils.COPY_ACKNOWLEDGEMENT))) {
Review comment:
do return of condition directly rather than using if clause
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 413958)
Time Spent: 0.5h (was: 20m)
> Checkpointing for repl dump bootstrap phase
> ---
>
> Key: HIVE-23039
> URL: https://issues.apache.org/jira/browse/HIVE-23039
> Project: Hive
> Issue Type: Bug
>Reporter: Aasha Medhi
>Assignee: Aasha Medhi
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-23039.01.patch
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-23039) Checkpointing for repl dump bootstrap phase
[ https://issues.apache.org/jira/browse/HIVE-23039?focusedWorklogId=413960=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-413960 ] ASF GitHub Bot logged work on HIVE-23039: - Author: ASF GitHub Bot Created on: 01/Apr/20 10:23 Start Date: 01/Apr/20 10:23 Worklog Time Spent: 10m Work Description: anishek commented on pull request #965: HIVE-23039 Checkpointing for repl dump bootstrap phase URL: https://github.com/apache/hive/pull/965#discussion_r401496301 ## File path: ql/src/java/org/apache/hadoop/hive/ql/exec/repl/util/ReplUtils.java ## @@ -98,6 +100,8 @@ public static final String DUMP_ACKNOWLEDGEMENT = "_finished_dump"; //Acknowledgement for repl load complete public static final String LOAD_ACKNOWLEDGEMENT = "_finished_load"; + //Acknowledgement for data copy complete. Used for checkpointing + public static final String COPY_ACKNOWLEDGEMENT = "_finished_copy"; Review comment: looks like we need a constants Class or a Enum class, the utils one seem to have a lot of magic variables. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 413960) Time Spent: 40m (was: 0.5h) > Checkpointing for repl dump bootstrap phase > --- > > Key: HIVE-23039 > URL: https://issues.apache.org/jira/browse/HIVE-23039 > Project: Hive > Issue Type: Bug >Reporter: Aasha Medhi >Assignee: Aasha Medhi >Priority: Major > Labels: pull-request-available > Attachments: HIVE-23039.01.patch > > Time Spent: 40m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-23020) Avoid using _files for replication data copy during incremental run
[
https://issues.apache.org/jira/browse/HIVE-23020?focusedWorklogId=413976=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-413976
]
ASF GitHub Bot logged work on HIVE-23020:
-
Author: ASF GitHub Bot
Created on: 01/Apr/20 10:51
Start Date: 01/Apr/20 10:51
Worklog Time Spent: 10m
Work Description: aasha commented on pull request #963: HIVE-23020 :
Avoid using _files for replication data copy during incr…
URL: https://github.com/apache/hive/pull/963#discussion_r401526351
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/parse/repl/dump/events/AddPartitionHandler.java
##
@@ -108,10 +108,8 @@ public void handle(Context withinContext) throws
Exception {
Iterable files = partitionFilesIter.next().getFiles();
if (files != null) {
// encoded filename/checksum of files, write into _files
- try (BufferedWriter fileListWriter = writer(withinContext, qlPtn)) {
-for (String file : files) {
- writeFileEntry(qlMdTable.getDbName(), qlMdTable, file,
fileListWriter, withinContext);
-}
+ for (String file : files) {
Review comment:
tests for this
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 413976)
Time Spent: 0.5h (was: 20m)
> Avoid using _files for replication data copy during incremental run
> ---
>
> Key: HIVE-23020
> URL: https://issues.apache.org/jira/browse/HIVE-23020
> Project: Hive
> Issue Type: Task
>Reporter: PRAVIN KUMAR SINHA
>Assignee: PRAVIN KUMAR SINHA
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-23020.01.patch, HIVE-23020.02.patch
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-23020) Avoid using _files for replication data copy during incremental run
[
https://issues.apache.org/jira/browse/HIVE-23020?focusedWorklogId=413974=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-413974
]
ASF GitHub Bot logged work on HIVE-23020:
-
Author: ASF GitHub Bot
Created on: 01/Apr/20 10:50
Start Date: 01/Apr/20 10:50
Worklog Time Spent: 10m
Work Description: aasha commented on pull request #963: HIVE-23020 :
Avoid using _files for replication data copy during incr…
URL: https://github.com/apache/hive/pull/963#discussion_r401522879
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/parse/ImportSemanticAnalyzer.java
##
@@ -505,9 +506,8 @@ private static ImportTableDesc
getBaseCreateTableDescFromTable(String dbName,
Task copyTask = null;
if (replicationSpec.isInReplicationScope()) {
- boolean isImport =
ReplicationSpec.Type.IMPORT.equals(replicationSpec.getReplSpecType());
copyTask = ReplCopyTask.getLoadCopyTask(replicationSpec, dataPath,
destPath, x.getConf(),
- isAutoPurge, needRecycle, copyToMigratedTxnTable, !isImport);
+ isAutoPurge, needRecycle, copyToMigratedTxnTable, false);
Review comment:
Why is this set to false? Do we even need this boolean if everyone is
setting it to false?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 413974)
Time Spent: 20m (was: 10m)
> Avoid using _files for replication data copy during incremental run
> ---
>
> Key: HIVE-23020
> URL: https://issues.apache.org/jira/browse/HIVE-23020
> Project: Hive
> Issue Type: Task
>Reporter: PRAVIN KUMAR SINHA
>Assignee: PRAVIN KUMAR SINHA
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-23020.01.patch, HIVE-23020.02.patch
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (HIVE-23123) Export/Import of views and materialized views
[
https://issues.apache.org/jira/browse/HIVE-23123?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Miklos Gergely reassigned HIVE-23123:
-
> Export/Import of views and materialized views
> -
>
> Key: HIVE-23123
> URL: https://issues.apache.org/jira/browse/HIVE-23123
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Reporter: Miklos Gergely
>Assignee: Miklos Gergely
>Priority: Major
>
> According to
> https://cwiki.apache.org/confluence/display/Hive/LanguageManual+ImportExport
> import and export can be done by using the
>
> {code:java}
> export table ...
> import table ...
> {code}
> commands. The document doesn't mention views or materialized views at all,
> and in fact we don't support commands like
>
> {code:java}
> export view ...
> import view ...
> export materialized view ...
> import materialized view ...
> {code}
> they can not be parsed at all. The word table is often used though in a
> broader sense, when it means all table like entities, including views and
> materialized views. For example the various Table classes may represent any
> of these as well.
> If I try to export a view with the export table ... command, it goes fine. A
> _metadata file will be created, but no data directory, which is what we'd
> expect. If I try to import it back, an exception is thrown due to the lack of
> the data dir:
>
> {code:java}
> java.lang.AssertionError: null==getPath() for exim_view
> at org.apache.hadoop.hive.ql.metadata.Hive.loadTable(Hive.java:3088)
> at org.apache.hadoop.hive.ql.exec.MoveTask.execute(MoveTask.java:419)
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:213)
> at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:105)
> at org.apache.hadoop.hive.ql.Executor.launchTask(Executor.java:364)
> at org.apache.hadoop.hive.ql.Executor.launchTasks(Executor.java:335)
> at org.apache.hadoop.hive.ql.Executor.runTasks(Executor.java:246)
> at org.apache.hadoop.hive.ql.Executor.execute(Executor.java:109)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:722)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:491)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:485)
> {code}
> Still the view gets imported successfully, as data movement wasn't even
> necessary.
> If we try to export a materialized view which is transactional, then this
> exception occurs:
>
> {code:java}
> org.apache.hadoop.hive.ql.parse.SemanticException:
> org.apache.hadoop.hive.ql.metadata.InvalidTableException: Table not found
> exim_materialized_view_da21d41a_9fe4_4446_9c72_d251496abf9d
> at
> org.apache.hadoop.hive.ql.parse.AcidExportSemanticAnalyzer.analyzeAcidExport(AcidExportSemanticAnalyzer.java:163)
> at
> org.apache.hadoop.hive.ql.parse.AcidExportSemanticAnalyzer.analyze(AcidExportSemanticAnalyzer.java:71)
> at
> org.apache.hadoop.hive.ql.parse.RewriteSemanticAnalyzer.analyzeInternal(RewriteSemanticAnalyzer.java:72)
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:289)
> at org.apache.hadoop.hive.ql.Compiler.analyze(Compiler.java:220)
> at org.apache.hadoop.hive.ql.Compiler.compile(Compiler.java:104)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:183)
> at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:601)
> at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:547)
> at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:541)
> {code}
> So the export process can not handle it, as the temporary table is not
> getting created.
>
> The import command handling have a lot of codes dedicated to importing views
> and materialized views, which suggests that we support the importing (and
> thus also suggests implicitly that we support the exporting) of views and
> materialiezed views.
>
> So the conclusion is that we have to decide if we support exporting/importing
> of views and materialized views.
> # If we decide not to support them then:
> - export process should throw an exception if a view or materialized view is
> the subject
> - the codes specific to view imports should be removed
> # If we decide to support them, then:
> - the commands mentioned above above should be introduced
> - exception should be thrown if not the proper command used (e.g. export
> view on a table)
> - the exceptions mentioned above should be fixed
> I prefer #1, I don't think we should support the exporting / importing of
> views. The point of exporting / importing is the transfer of data, not DDL,
> it causes more issues than it solves. Our current documentation also suggests
> that it is only supported for tables.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23118) Option for exposing compile time counters as tez counters
[ https://issues.apache.org/jira/browse/HIVE-23118?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17072409#comment-17072409 ] Prasanth Jayachandran commented on HIVE-23118: -- [~Sreenath] These are hive side counters merged with dag counters on the client side. These counters will be added to any tez task during hive query compilation. I don't think this will be available at tez side as it does not attach to any tez context. It will accessible to hive hooks though (hive proto hook can dump it). > Option for exposing compile time counters as tez counters > - > > Key: HIVE-23118 > URL: https://issues.apache.org/jira/browse/HIVE-23118 > Project: Hive > Issue Type: Improvement >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Minor > Labels: pull-request-available > Attachments: HIVE-23118.1.patch > > Time Spent: 10m > Remaining Estimate: 0h > > TezCounters currently are runtime only. Some compile time information from > optimizer can be exposed as counters which can then be used by workload > management to make runtime decisions. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23052) Optimize lock enqueueing in TxnHandler
[
https://issues.apache.org/jira/browse/HIVE-23052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17072414#comment-17072414
]
Hive QA commented on HIVE-23052:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
19s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
27s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
22s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 1m
13s{color} | {color:blue} standalone-metastore/metastore-server in master has
190 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
20s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
27s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
22s{color} | {color:red} standalone-metastore/metastore-server: The patch
generated 36 new + 535 unchanged - 36 fixed = 571 total (was 571) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
19s{color} | {color:green} standalone-metastore/metastore-server generated 0
new + 189 unchanged - 1 fixed = 189 total (was 190) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
21s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 15m 25s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-21365/dev-support/hive-personality.sh
|
| git revision | master / aa142d1 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-21365/yetus/diff-checkstyle-standalone-metastore_metastore-server.txt
|
| modules | C: standalone-metastore/metastore-server U:
standalone-metastore/metastore-server |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-21365/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Optimize lock enqueueing in TxnHandler
> --
>
> Key: HIVE-23052
> URL: https://issues.apache.org/jira/browse/HIVE-23052
> Project: Hive
> Issue Type: Improvement
>Reporter: Marton Bod
>Assignee: Marton Bod
>Priority: Major
> Attachments: HIVE-23052.1.patch, HIVE-23052.10.patch,
> HIVE-23052.11.patch, HIVE-23052.12.patch, HIVE-23052.13.patch,
> HIVE-23052.2.patch, HIVE-23052.3.patch, HIVE-23052.4.patch,
> HIVE-23052.5.patch, HIVE-23052.6.patch, HIVE-23052.7.patch,
> HIVE-23052.8.patch, HIVE-23052.8.patch, HIVE-23052.8.patch, HIVE-23052.9.patch
>
>
> * Reduce scope of next_lock_id select-for-update by moving the txn_component
> inserts before the S4U + inserting the hive_locks entries before the S4U
> (first with a temp ID, which will be replaced later in a single update). This
> helps decrease the overall time that the next_lock_id table is locked,
> thereby increasing concurrency
> * Insert txn_components in a batch instead of one-by-one (also in
> TxnHandler::addDynamicPartition)
> * Increment next_lock_id and update hive_locks table in a single batch
> statement
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23082) PK/FK stat rescale doesn't work in some cases
[
https://issues.apache.org/jira/browse/HIVE-23082?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zoltan Haindrich updated HIVE-23082:
Attachment: HIVE-23082.03.patch
> PK/FK stat rescale doesn't work in some cases
> -
>
> Key: HIVE-23082
> URL: https://issues.apache.org/jira/browse/HIVE-23082
> Project: Hive
> Issue Type: Bug
> Components: Statistics
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Attachments: HIVE-23082.01.patch, HIVE-23082.02.patch,
> HIVE-23082.03.patch
>
>
> As a result in Joins may retain the original estimate; see MAPJOIN_33 in this
> plan ; which retained the estimate of SEL_32
> {code}
> ++
> | Explain |
> ++
> | Plan optimized by CBO. |
> ||
> | Vertex dependency in root stage|
> | Map 1 <- Map 2 (BROADCAST_EDGE)|
> ||
> | Stage-0|
> | Fetch Operator |
> | limit:12 |
> | Stage-1|
> | Map 1 vectorized |
> | File Output Operator [FS_36] |
> | Limit [LIM_35] (rows=12 width=4) |
> | Number of rows:12|
> | Select Operator [SEL_34] (rows=5040 width=4) |
> | Output:["_col0"] |
> | Map Join Operator [MAPJOIN_33] (rows=5040 width=8) |
> | Conds:SEL_32._col0=RS_30._col0(Inner) |
> | <-Map 2 [BROADCAST_EDGE] vectorized|
> | BROADCAST [RS_30]|
> | PartitionCols:_col0|
> | Select Operator [SEL_29] (rows=1 width=8) |
> | Output:["_col0"] |
> | Filter Operator [FIL_28] (rows=1 width=108) |
> | predicate:((r_reason_id = 'reason 66') and r_reason_sk
> is not null) |
> | TableScan [TS_3] (rows=2 width=108) |
> |
> default@rx0,reason,Tbl:COMPLETE,Col:COMPLETE,Output:["r_reason_id","r_reason_sk"]
> |
> | <-Select Operator [SEL_32] (rows=5040 width=7) |
> | Output:["_col0"] |
> | Filter Operator [FIL_31] (rows=5040 width=7) |
> | predicate:sr_reason_sk is not null |
> | TableScan [TS_0] (rows=5112 width=7) |
> |
> default@sr0,store_returns,Tbl:COMPLETE,Col:COMPLETE,Output:["sr_reason_sk"] |
> ||
> ++
> {code}
> repro:
> {code}
> set hive.query.results.cache.enabled=false;
> set hive.explain.user=true;
> drop table if exists default.rx0;
> drop table if exists default.sr0;
> create table rx0 (r_reason_id string, r_reason_sk bigint);
> create table sr0 (sr_reason_sk bigint);
> insert into rx0 values ('',1),('GEAA',70);
> insert into sr0 values (NULL),(1),(2),(3),(4),(5),(6),(7),(8),(9),(10),
> (11),(12),(13),(14),(15),(16),(17),(18),(19),(20),(21),(22),(23),(24),(25),
> (26),(27),(28),(29),(30),(31),(32),(33),(34),(35),(36),(37),(38),(39),(40),
> (41),(42),(43),(44),(45),(46),(47),(48),(49),(50),(51),(52),(53),(54),(55),
> (56),(57),(58),(59),(60),(61),(62),(63),(64),(65),(66),(67),(68),(69),(70);
> insert into sr0 select a.* from sr0 a,sr0 b;
> -- |sr0| ~ 5112
> explain select 1
> from default.sr0 store_returns , default.rx0 reason
> where sr_reason_sk = r_reason_sk
> and r_reason_id = 'reason 66'
> limit 12;
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23095) NDV might be overestimated for a table with ~70 value
[ https://issues.apache.org/jira/browse/HIVE-23095?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Haindrich updated HIVE-23095: Attachment: HIVE-23095.04.patch > NDV might be overestimated for a table with ~70 value > - > > Key: HIVE-23095 > URL: https://issues.apache.org/jira/browse/HIVE-23095 > Project: Hive > Issue Type: Bug >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Major > Labels: pull-request-available > Attachments: HIVE-23095.01.patch, HIVE-23095.02.patch, > HIVE-23095.03.patch, HIVE-23095.04.patch > > Time Spent: 10m > Remaining Estimate: 0h > > uncovered during looking into HIVE-23082 > https://issues.apache.org/jira/browse/HIVE-23082?focusedCommentId=17067773=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17067773 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23101) Fix topnkey_grouping_sets
[
https://issues.apache.org/jira/browse/HIVE-23101?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Krisztian Kasa updated HIVE-23101:
--
Status: Patch Available (was: Open)
> Fix topnkey_grouping_sets
> -
>
> Key: HIVE-23101
> URL: https://issues.apache.org/jira/browse/HIVE-23101
> Project: Hive
> Issue Type: Sub-task
>Reporter: Krisztian Kasa
>Assignee: Krisztian Kasa
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-23101.1.patch, HIVE-23101.2.patch
>
>
> Example:
> {code}
> Client Execution succeeded but contained differences (error code = 1) after
> executing topnkey_grouping_sets.q
> 260c260
> < NULL NULL
> ---
> > NULL 1
> {code}
>
> Test *topnkey_grouping_sets* fails intermittently.
> Queries which project 2 columns but order by only one of them can have more
> than one good result set:
> {code:java}
> CREATE TABLE t_test_grouping_sets(
> a int,
> b int,
> c int
> );
> INSERT INTO t_test_grouping_sets VALUES
> (NULL, NULL, NULL),
> (5, 2, 3),
> (10, 11, 12),
> (NULL, NULL, NULL),
> (NULL, NULL, NULL),
> (6, 2, 1),
> (7, 8, 4), (7, 8, 4), (7, 8, 4),
> (5, 1, 2), (5, 1, 2), (5, 1, 2),
> (NULL, NULL, NULL);
> SELECT a, b FROM t_test_grouping_sets GROUP BY GROUPING SETS ((a, b), (a),
> (b), ()) ORDER BY a LIMIT 10;
> {code}
> {code:java}
> 5 NULL
> 5 2
> 5 1
> 6 2
> 6 NULL
> 7 8
> 7 NULL
> 10NULL
> 1011
> NULL 1
> {code}
> {code:java}
> 5 NULL
> 5 2
> 5 1
> 6 2
> 6 NULL
> 7 8
> 7 NULL
> 10NULL
> 1011
> NULL NULL
> {code}
> Since we don't order by *b* both result sets are valid.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23101) Fix topnkey_grouping_sets
[
https://issues.apache.org/jira/browse/HIVE-23101?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Krisztian Kasa updated HIVE-23101:
--
Status: Open (was: Patch Available)
> Fix topnkey_grouping_sets
> -
>
> Key: HIVE-23101
> URL: https://issues.apache.org/jira/browse/HIVE-23101
> Project: Hive
> Issue Type: Sub-task
>Reporter: Krisztian Kasa
>Assignee: Krisztian Kasa
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-23101.1.patch, HIVE-23101.2.patch
>
>
> Example:
> {code}
> Client Execution succeeded but contained differences (error code = 1) after
> executing topnkey_grouping_sets.q
> 260c260
> < NULL NULL
> ---
> > NULL 1
> {code}
>
> Test *topnkey_grouping_sets* fails intermittently.
> Queries which project 2 columns but order by only one of them can have more
> than one good result set:
> {code:java}
> CREATE TABLE t_test_grouping_sets(
> a int,
> b int,
> c int
> );
> INSERT INTO t_test_grouping_sets VALUES
> (NULL, NULL, NULL),
> (5, 2, 3),
> (10, 11, 12),
> (NULL, NULL, NULL),
> (NULL, NULL, NULL),
> (6, 2, 1),
> (7, 8, 4), (7, 8, 4), (7, 8, 4),
> (5, 1, 2), (5, 1, 2), (5, 1, 2),
> (NULL, NULL, NULL);
> SELECT a, b FROM t_test_grouping_sets GROUP BY GROUPING SETS ((a, b), (a),
> (b), ()) ORDER BY a LIMIT 10;
> {code}
> {code:java}
> 5 NULL
> 5 2
> 5 1
> 6 2
> 6 NULL
> 7 8
> 7 NULL
> 10NULL
> 1011
> NULL 1
> {code}
> {code:java}
> 5 NULL
> 5 2
> 5 1
> 6 2
> 6 NULL
> 7 8
> 7 NULL
> 10NULL
> 1011
> NULL NULL
> {code}
> Since we don't order by *b* both result sets are valid.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23101) Fix topnkey_grouping_sets
[
https://issues.apache.org/jira/browse/HIVE-23101?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Krisztian Kasa updated HIVE-23101:
--
Attachment: HIVE-23101.2.patch
> Fix topnkey_grouping_sets
> -
>
> Key: HIVE-23101
> URL: https://issues.apache.org/jira/browse/HIVE-23101
> Project: Hive
> Issue Type: Sub-task
>Reporter: Krisztian Kasa
>Assignee: Krisztian Kasa
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-23101.1.patch, HIVE-23101.2.patch
>
>
> Example:
> {code}
> Client Execution succeeded but contained differences (error code = 1) after
> executing topnkey_grouping_sets.q
> 260c260
> < NULL NULL
> ---
> > NULL 1
> {code}
>
> Test *topnkey_grouping_sets* fails intermittently.
> Queries which project 2 columns but order by only one of them can have more
> than one good result set:
> {code:java}
> CREATE TABLE t_test_grouping_sets(
> a int,
> b int,
> c int
> );
> INSERT INTO t_test_grouping_sets VALUES
> (NULL, NULL, NULL),
> (5, 2, 3),
> (10, 11, 12),
> (NULL, NULL, NULL),
> (NULL, NULL, NULL),
> (6, 2, 1),
> (7, 8, 4), (7, 8, 4), (7, 8, 4),
> (5, 1, 2), (5, 1, 2), (5, 1, 2),
> (NULL, NULL, NULL);
> SELECT a, b FROM t_test_grouping_sets GROUP BY GROUPING SETS ((a, b), (a),
> (b), ()) ORDER BY a LIMIT 10;
> {code}
> {code:java}
> 5 NULL
> 5 2
> 5 1
> 6 2
> 6 NULL
> 7 8
> 7 NULL
> 10NULL
> 1011
> NULL 1
> {code}
> {code:java}
> 5 NULL
> 5 2
> 5 1
> 6 2
> 6 NULL
> 7 8
> 7 NULL
> 10NULL
> 1011
> NULL NULL
> {code}
> Since we don't order by *b* both result sets are valid.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23119) Test sort_acid should be run by TestMiniLlapLocalCliDriver only
[ https://issues.apache.org/jira/browse/HIVE-23119?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Krisztian Kasa updated HIVE-23119: -- Priority: Minor (was: Major) > Test sort_acid should be run by TestMiniLlapLocalCliDriver only > --- > > Key: HIVE-23119 > URL: https://issues.apache.org/jira/browse/HIVE-23119 > Project: Hive > Issue Type: Task > Components: CBO >Reporter: Krisztian Kasa >Priority: Minor > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-21492) VectorizedParquetRecordReader can't to read parquet file generated using thrift/custom tool
[
https://issues.apache.org/jira/browse/HIVE-21492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17072466#comment-17072466
]
Ferdinand Xu commented on HIVE-21492:
-
Could you update the indents below?
{code:java}
+ return childType.asGroupType().getFields().get(0)
+ .asPrimitiveType();
{code}
> VectorizedParquetRecordReader can't to read parquet file generated using
> thrift/custom tool
> ---
>
> Key: HIVE-21492
> URL: https://issues.apache.org/jira/browse/HIVE-21492
> Project: Hive
> Issue Type: Bug
>Reporter: Ganesha Shreedhara
>Assignee: Ganesha Shreedhara
>Priority: Major
> Attachments: HIVE-21492.2.patch, HIVE-21492.patch
>
>
> Taking an example of a parquet table having array of integers as below.
> {code:java}
> CREATE EXTERNAL TABLE ( list_of_ints` array)
> STORED AS PARQUET
> LOCATION '{location}';
> {code}
> Parquet file generated using hive will have schema for Type as below:
> {code:java}
> group list_of_ints (LIST) { repeated group bag { optional int32 array;\n};\n}
> {code}
> Parquet file generated using thrift or any custom tool (using
> org.apache.parquet.io.api.RecordConsumer)
> may have schema for Type as below:
> {code:java}
> required group list_of_ints (LIST) { repeated int32 list_of_tuple} {code}
> VectorizedParquetRecordReader handles only parquet file generated using hive.
> It throws the following exception when parquet file generated using thrift is
> read because of the changes done as part of HIVE-18553 .
> {code:java}
> Caused by: java.lang.ClassCastException: repeated int32 list_of_ints_tuple is
> not a group
> at org.apache.parquet.schema.Type.asGroupType(Type.java:207)
> at
> org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.getElementType(VectorizedParquetRecordReader.java:479)
> at
> org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.buildVectorizedParquetReader(VectorizedParquetRecordReader.java:532)
> at
> org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.checkEndOfRowGroup(VectorizedParquetRecordReader.java:440)
> at
> org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.nextBatch(VectorizedParquetRecordReader.java:401)
> at
> org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.next(VectorizedParquetRecordReader.java:353)
> at
> org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.next(VectorizedParquetRecordReader.java:92)
> at
> org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.doNext(HiveContextAwareRecordReader.java:365){code}
>
> I have done a small change to handle the case where the child type of group
> type can be PrimitiveType.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23119) Test sort_acid should be run by TestMiniLlapLocalCliDriver only
[ https://issues.apache.org/jira/browse/HIVE-23119?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Krisztian Kasa updated HIVE-23119: -- Attachment: HIVE-23119.1.patch > Test sort_acid should be run by TestMiniLlapLocalCliDriver only > --- > > Key: HIVE-23119 > URL: https://issues.apache.org/jira/browse/HIVE-23119 > Project: Hive > Issue Type: Task > Components: CBO >Reporter: Krisztian Kasa >Priority: Minor > Attachments: HIVE-23119.1.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23119) Test sort_acid should be run by TestMiniLlapLocalCliDriver only
[ https://issues.apache.org/jira/browse/HIVE-23119?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Krisztian Kasa updated HIVE-23119: -- Status: Patch Available (was: Open) > Test sort_acid should be run by TestMiniLlapLocalCliDriver only > --- > > Key: HIVE-23119 > URL: https://issues.apache.org/jira/browse/HIVE-23119 > Project: Hive > Issue Type: Task > Components: CBO >Reporter: Krisztian Kasa >Priority: Minor > Attachments: HIVE-23119.1.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-23119) Test sort_acid should be run by TestMiniLlapLocalCliDriver only
[ https://issues.apache.org/jira/browse/HIVE-23119?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Krisztian Kasa reassigned HIVE-23119: - Assignee: Krisztian Kasa > Test sort_acid should be run by TestMiniLlapLocalCliDriver only > --- > > Key: HIVE-23119 > URL: https://issues.apache.org/jira/browse/HIVE-23119 > Project: Hive > Issue Type: Task > Components: CBO >Reporter: Krisztian Kasa >Assignee: Krisztian Kasa >Priority: Minor > Attachments: HIVE-23119.1.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-21492) VectorizedParquetRecordReader can't to read parquet file generated using thrift/custom tool
[
https://issues.apache.org/jira/browse/HIVE-21492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17072471#comment-17072471
]
Ganesha Shreedhara commented on HIVE-21492:
---
Done.
> VectorizedParquetRecordReader can't to read parquet file generated using
> thrift/custom tool
> ---
>
> Key: HIVE-21492
> URL: https://issues.apache.org/jira/browse/HIVE-21492
> Project: Hive
> Issue Type: Bug
>Reporter: Ganesha Shreedhara
>Assignee: Ganesha Shreedhara
>Priority: Major
> Attachments: HIVE-21492.2.patch, HIVE-21492.3.patch, HIVE-21492.patch
>
>
> Taking an example of a parquet table having array of integers as below.
> {code:java}
> CREATE EXTERNAL TABLE ( list_of_ints` array)
> STORED AS PARQUET
> LOCATION '{location}';
> {code}
> Parquet file generated using hive will have schema for Type as below:
> {code:java}
> group list_of_ints (LIST) { repeated group bag { optional int32 array;\n};\n}
> {code}
> Parquet file generated using thrift or any custom tool (using
> org.apache.parquet.io.api.RecordConsumer)
> may have schema for Type as below:
> {code:java}
> required group list_of_ints (LIST) { repeated int32 list_of_tuple} {code}
> VectorizedParquetRecordReader handles only parquet file generated using hive.
> It throws the following exception when parquet file generated using thrift is
> read because of the changes done as part of HIVE-18553 .
> {code:java}
> Caused by: java.lang.ClassCastException: repeated int32 list_of_ints_tuple is
> not a group
> at org.apache.parquet.schema.Type.asGroupType(Type.java:207)
> at
> org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.getElementType(VectorizedParquetRecordReader.java:479)
> at
> org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.buildVectorizedParquetReader(VectorizedParquetRecordReader.java:532)
> at
> org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.checkEndOfRowGroup(VectorizedParquetRecordReader.java:440)
> at
> org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.nextBatch(VectorizedParquetRecordReader.java:401)
> at
> org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.next(VectorizedParquetRecordReader.java:353)
> at
> org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.next(VectorizedParquetRecordReader.java:92)
> at
> org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.doNext(HiveContextAwareRecordReader.java:365){code}
>
> I have done a small change to handle the case where the child type of group
> type can be PrimitiveType.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (HIVE-23120) TopNKey related tests should be run by TestMiniLlapLocalCliDriver only
[ https://issues.apache.org/jira/browse/HIVE-23120?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Krisztian Kasa reassigned HIVE-23120: - > TopNKey related tests should be run by TestMiniLlapLocalCliDriver only > -- > > Key: HIVE-23120 > URL: https://issues.apache.org/jira/browse/HIVE-23120 > Project: Hive > Issue Type: Task > Components: Physical Optimizer >Reporter: Krisztian Kasa >Assignee: Krisztian Kasa >Priority: Minor > > TopNKey optimization is only used when the execution framework is Tez. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-21492) VectorizedParquetRecordReader can't to read parquet file generated using thrift/custom tool
[
https://issues.apache.org/jira/browse/HIVE-21492?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ganesha Shreedhara updated HIVE-21492:
--
Status: Patch Available (was: In Progress)
> VectorizedParquetRecordReader can't to read parquet file generated using
> thrift/custom tool
> ---
>
> Key: HIVE-21492
> URL: https://issues.apache.org/jira/browse/HIVE-21492
> Project: Hive
> Issue Type: Bug
>Reporter: Ganesha Shreedhara
>Assignee: Ganesha Shreedhara
>Priority: Major
> Attachments: HIVE-21492.2.patch, HIVE-21492.3.patch, HIVE-21492.patch
>
>
> Taking an example of a parquet table having array of integers as below.
> {code:java}
> CREATE EXTERNAL TABLE ( list_of_ints` array)
> STORED AS PARQUET
> LOCATION '{location}';
> {code}
> Parquet file generated using hive will have schema for Type as below:
> {code:java}
> group list_of_ints (LIST) { repeated group bag { optional int32 array;\n};\n}
> {code}
> Parquet file generated using thrift or any custom tool (using
> org.apache.parquet.io.api.RecordConsumer)
> may have schema for Type as below:
> {code:java}
> required group list_of_ints (LIST) { repeated int32 list_of_tuple} {code}
> VectorizedParquetRecordReader handles only parquet file generated using hive.
> It throws the following exception when parquet file generated using thrift is
> read because of the changes done as part of HIVE-18553 .
> {code:java}
> Caused by: java.lang.ClassCastException: repeated int32 list_of_ints_tuple is
> not a group
> at org.apache.parquet.schema.Type.asGroupType(Type.java:207)
> at
> org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.getElementType(VectorizedParquetRecordReader.java:479)
> at
> org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.buildVectorizedParquetReader(VectorizedParquetRecordReader.java:532)
> at
> org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.checkEndOfRowGroup(VectorizedParquetRecordReader.java:440)
> at
> org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.nextBatch(VectorizedParquetRecordReader.java:401)
> at
> org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.next(VectorizedParquetRecordReader.java:353)
> at
> org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.next(VectorizedParquetRecordReader.java:92)
> at
> org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.doNext(HiveContextAwareRecordReader.java:365){code}
>
> I have done a small change to handle the case where the child type of group
> type can be PrimitiveType.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-21492) VectorizedParquetRecordReader can't to read parquet file generated using thrift/custom tool
[
https://issues.apache.org/jira/browse/HIVE-21492?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ganesha Shreedhara updated HIVE-21492:
--
Attachment: HIVE-21492.3.patch
> VectorizedParquetRecordReader can't to read parquet file generated using
> thrift/custom tool
> ---
>
> Key: HIVE-21492
> URL: https://issues.apache.org/jira/browse/HIVE-21492
> Project: Hive
> Issue Type: Bug
>Reporter: Ganesha Shreedhara
>Assignee: Ganesha Shreedhara
>Priority: Major
> Attachments: HIVE-21492.2.patch, HIVE-21492.3.patch, HIVE-21492.patch
>
>
> Taking an example of a parquet table having array of integers as below.
> {code:java}
> CREATE EXTERNAL TABLE ( list_of_ints` array)
> STORED AS PARQUET
> LOCATION '{location}';
> {code}
> Parquet file generated using hive will have schema for Type as below:
> {code:java}
> group list_of_ints (LIST) { repeated group bag { optional int32 array;\n};\n}
> {code}
> Parquet file generated using thrift or any custom tool (using
> org.apache.parquet.io.api.RecordConsumer)
> may have schema for Type as below:
> {code:java}
> required group list_of_ints (LIST) { repeated int32 list_of_tuple} {code}
> VectorizedParquetRecordReader handles only parquet file generated using hive.
> It throws the following exception when parquet file generated using thrift is
> read because of the changes done as part of HIVE-18553 .
> {code:java}
> Caused by: java.lang.ClassCastException: repeated int32 list_of_ints_tuple is
> not a group
> at org.apache.parquet.schema.Type.asGroupType(Type.java:207)
> at
> org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.getElementType(VectorizedParquetRecordReader.java:479)
> at
> org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.buildVectorizedParquetReader(VectorizedParquetRecordReader.java:532)
> at
> org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.checkEndOfRowGroup(VectorizedParquetRecordReader.java:440)
> at
> org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.nextBatch(VectorizedParquetRecordReader.java:401)
> at
> org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.next(VectorizedParquetRecordReader.java:353)
> at
> org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.next(VectorizedParquetRecordReader.java:92)
> at
> org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.doNext(HiveContextAwareRecordReader.java:365){code}
>
> I have done a small change to handle the case where the child type of group
> type can be PrimitiveType.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23032) Add batching in Lock generation
[ https://issues.apache.org/jira/browse/HIVE-23032?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Denys Kuzmenko updated HIVE-23032: -- Resolution: Fixed Status: Resolved (was: Patch Available) > Add batching in Lock generation > --- > > Key: HIVE-23032 > URL: https://issues.apache.org/jira/browse/HIVE-23032 > Project: Hive > Issue Type: Improvement > Components: Locking >Reporter: Denys Kuzmenko >Assignee: Denys Kuzmenko >Priority: Major > Attachments: HIVE-23032.1.patch, HIVE-23032.2.patch, > HIVE-23032.3.patch, HIVE-23032.4.patch, HIVE-23032.5.patch, > HIVE-23032.6.patch, HIVE-23032.7.patch, HIVE-23032.8.patch > > > Replace multi-row insert in Oracle with batching. Performance tests showed > significant performance improvement after turning batching on. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23095) NDV might be overestimated for a table with ~70 value
[ https://issues.apache.org/jira/browse/HIVE-23095?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Haindrich updated HIVE-23095: Attachment: HIVE-23095.03.patch > NDV might be overestimated for a table with ~70 value > - > > Key: HIVE-23095 > URL: https://issues.apache.org/jira/browse/HIVE-23095 > Project: Hive > Issue Type: Bug >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Major > Labels: pull-request-available > Attachments: HIVE-23095.01.patch, HIVE-23095.02.patch, > HIVE-23095.03.patch > > Time Spent: 10m > Remaining Estimate: 0h > > uncovered during looking into HIVE-23082 > https://issues.apache.org/jira/browse/HIVE-23082?focusedCommentId=17067773=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17067773 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23052) Optimize lock enqueueing in TxnHandler
[
https://issues.apache.org/jira/browse/HIVE-23052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17072470#comment-17072470
]
Hive QA commented on HIVE-23052:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12998346/HIVE-23052.13.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 18163 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[topnkey_grouping_sets]
(batchId=1)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/21365/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/21365/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-21365/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12998346 - PreCommit-HIVE-Build
> Optimize lock enqueueing in TxnHandler
> --
>
> Key: HIVE-23052
> URL: https://issues.apache.org/jira/browse/HIVE-23052
> Project: Hive
> Issue Type: Improvement
>Reporter: Marton Bod
>Assignee: Marton Bod
>Priority: Major
> Attachments: HIVE-23052.1.patch, HIVE-23052.10.patch,
> HIVE-23052.11.patch, HIVE-23052.12.patch, HIVE-23052.13.patch,
> HIVE-23052.2.patch, HIVE-23052.3.patch, HIVE-23052.4.patch,
> HIVE-23052.5.patch, HIVE-23052.6.patch, HIVE-23052.7.patch,
> HIVE-23052.8.patch, HIVE-23052.8.patch, HIVE-23052.8.patch, HIVE-23052.9.patch
>
>
> * Reduce scope of next_lock_id select-for-update by moving the txn_component
> inserts before the S4U + inserting the hive_locks entries before the S4U
> (first with a temp ID, which will be replaced later in a single update). This
> helps decrease the overall time that the next_lock_id table is locked,
> thereby increasing concurrency
> * Insert txn_components in a batch instead of one-by-one (also in
> TxnHandler::addDynamicPartition)
> * Increment next_lock_id and update hive_locks table in a single batch
> statement
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-21492) VectorizedParquetRecordReader can't to read parquet file generated using thrift/custom tool
[
https://issues.apache.org/jira/browse/HIVE-21492?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ganesha Shreedhara updated HIVE-21492:
--
Status: In Progress (was: Patch Available)
> VectorizedParquetRecordReader can't to read parquet file generated using
> thrift/custom tool
> ---
>
> Key: HIVE-21492
> URL: https://issues.apache.org/jira/browse/HIVE-21492
> Project: Hive
> Issue Type: Bug
>Reporter: Ganesha Shreedhara
>Assignee: Ganesha Shreedhara
>Priority: Major
> Attachments: HIVE-21492.2.patch, HIVE-21492.patch
>
>
> Taking an example of a parquet table having array of integers as below.
> {code:java}
> CREATE EXTERNAL TABLE ( list_of_ints` array)
> STORED AS PARQUET
> LOCATION '{location}';
> {code}
> Parquet file generated using hive will have schema for Type as below:
> {code:java}
> group list_of_ints (LIST) { repeated group bag { optional int32 array;\n};\n}
> {code}
> Parquet file generated using thrift or any custom tool (using
> org.apache.parquet.io.api.RecordConsumer)
> may have schema for Type as below:
> {code:java}
> required group list_of_ints (LIST) { repeated int32 list_of_tuple} {code}
> VectorizedParquetRecordReader handles only parquet file generated using hive.
> It throws the following exception when parquet file generated using thrift is
> read because of the changes done as part of HIVE-18553 .
> {code:java}
> Caused by: java.lang.ClassCastException: repeated int32 list_of_ints_tuple is
> not a group
> at org.apache.parquet.schema.Type.asGroupType(Type.java:207)
> at
> org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.getElementType(VectorizedParquetRecordReader.java:479)
> at
> org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.buildVectorizedParquetReader(VectorizedParquetRecordReader.java:532)
> at
> org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.checkEndOfRowGroup(VectorizedParquetRecordReader.java:440)
> at
> org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.nextBatch(VectorizedParquetRecordReader.java:401)
> at
> org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.next(VectorizedParquetRecordReader.java:353)
> at
> org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.next(VectorizedParquetRecordReader.java:92)
> at
> org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.doNext(HiveContextAwareRecordReader.java:365){code}
>
> I have done a small change to handle the case where the child type of group
> type can be PrimitiveType.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23067) Use batch DB calls in TxnHandler for commitTxn and abortTxns
[
https://issues.apache.org/jira/browse/HIVE-23067?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17072490#comment-17072490
]
Hive QA commented on HIVE-23067:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
8s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
28s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
21s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 1m
13s{color} | {color:blue} standalone-metastore/metastore-server in master has
190 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
21s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
26s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
23s{color} | {color:red} standalone-metastore/metastore-server: The patch
generated 1 new + 545 unchanged - 1 fixed = 546 total (was 546) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
22s{color} | {color:red} standalone-metastore/metastore-server generated 1 new
+ 190 unchanged - 0 fixed = 191 total (was 190) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
19s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 15m 14s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:standalone-metastore/metastore-server |
| | A prepared statement is generated from a nonconstant String in
org.apache.hadoop.hive.metastore.txn.TxnHandler.commitTxn(CommitTxnRequest)
At TxnHandler.java:from a nonconstant String in
org.apache.hadoop.hive.metastore.txn.TxnHandler.commitTxn(CommitTxnRequest)
At TxnHandler.java:[line 1298] |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-21366/dev-support/hive-personality.sh
|
| git revision | master / aa142d1 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-21366/yetus/diff-checkstyle-standalone-metastore_metastore-server.txt
|
| findbugs |
http://104.198.109.242/logs//PreCommit-HIVE-Build-21366/yetus/new-findbugs-standalone-metastore_metastore-server.html
|
| modules | C: standalone-metastore/metastore-server U:
standalone-metastore/metastore-server |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-21366/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Use batch DB calls in TxnHandler for commitTxn and abortTxns
>
>
> Key: HIVE-23067
> URL: https://issues.apache.org/jira/browse/HIVE-23067
> Project: Hive
> Issue Type: Improvement
>Reporter: Marton Bod
>Assignee: Marton Bod
>Priority: Major
> Attachments: HIVE-23067.1.patch, HIVE-23067.1.patch,
> HIVE-23067.1.patch, HIVE-23067.1.patch, HIVE-23067.2.patch
>
>
> To reduce the number of database calls and network roundtrips, we could use
> more batching in TxnHandler, where currently in many places we call insert
> commands in loops sequentially.
> Some examples:
> * openTxns (TXNS, REPL_TXN_MAP)
> * commitTxn
[jira] [Commented] (HIVE-23107) Remove MIN_HISTORY_LEVEL table
[
https://issues.apache.org/jira/browse/HIVE-23107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17072781#comment-17072781
]
Hive QA commented on HIVE-23107:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12998338/HIVE-23107.02.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 160 failed/errored test(s), 18162 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.metastore.txn.TestCompactionTxnHandler.testFindNextToClean
(batchId=318)
org.apache.hadoop.hive.metastore.txn.TestCompactionTxnHandler.testMarkCleaned
(batchId=318)
org.apache.hadoop.hive.metastore.txn.TestCompactionTxnHandler.testMarkCleanedCleansTxnsAndTxnComponents
(batchId=318)
org.apache.hadoop.hive.metastore.txn.TestCompactionTxnHandler.testMarkCompacted
(batchId=318)
org.apache.hadoop.hive.ql.TestAcidOnTez.testInsertWithRemoveUnion (batchId=253)
org.apache.hadoop.hive.ql.TestAcidOnTez.testMapJoinOnMR (batchId=253)
org.apache.hadoop.hive.ql.TestAcidOnTez.testMapJoinOnTez (batchId=253)
org.apache.hadoop.hive.ql.TestAcidOnTez.testMergeJoinOnMR (batchId=253)
org.apache.hadoop.hive.ql.TestAcidOnTez.testMergeJoinOnTez (batchId=253)
org.apache.hadoop.hive.ql.TestTxnCommands.testNonAcidToAcidConversion01
(batchId=363)
org.apache.hadoop.hive.ql.TestTxnCommands.testVersioning (batchId=363)
org.apache.hadoop.hive.ql.TestTxnCommands2.testACIDwithSchemaEvolutionAndCompaction
(batchId=344)
org.apache.hadoop.hive.ql.TestTxnCommands2.testCleanerForTxnToWriteId
(batchId=344)
org.apache.hadoop.hive.ql.TestTxnCommands2.testCompactWithDelete (batchId=344)
org.apache.hadoop.hive.ql.TestTxnCommands2.testEmptyInTblproperties
(batchId=344)
org.apache.hadoop.hive.ql.TestTxnCommands2.testInitiatorWithMultipleFailedCompactions
(batchId=344)
org.apache.hadoop.hive.ql.TestTxnCommands2.testInsertOverwrite1 (batchId=344)
org.apache.hadoop.hive.ql.TestTxnCommands2.testInsertOverwrite2 (batchId=344)
org.apache.hadoop.hive.ql.TestTxnCommands2.testNonAcidToAcidConversion02
(batchId=344)
org.apache.hadoop.hive.ql.TestTxnCommands2.testNonAcidToAcidConversion1
(batchId=344)
org.apache.hadoop.hive.ql.TestTxnCommands2.testNonAcidToAcidConversion2
(batchId=344)
org.apache.hadoop.hive.ql.TestTxnCommands2.testNonAcidToAcidConversion3
(batchId=344)
org.apache.hadoop.hive.ql.TestTxnCommands2.testOriginalFileReaderWhenNonAcidConvertedToAcid
(batchId=344)
org.apache.hadoop.hive.ql.TestTxnCommands2.testSchemaEvolutionCompaction
(batchId=344)
org.apache.hadoop.hive.ql.TestTxnCommands2.writeBetweenWorkerAndCleaner
(batchId=344)
org.apache.hadoop.hive.ql.TestTxnCommands2WithSplitUpdateAndVectorization.testACIDwithSchemaEvolutionAndCompaction
(batchId=358)
org.apache.hadoop.hive.ql.TestTxnCommands2WithSplitUpdateAndVectorization.testCleanerForTxnToWriteId
(batchId=358)
org.apache.hadoop.hive.ql.TestTxnCommands2WithSplitUpdateAndVectorization.testCompactWithDelete
(batchId=358)
org.apache.hadoop.hive.ql.TestTxnCommands2WithSplitUpdateAndVectorization.testEmptyInTblproperties
(batchId=358)
org.apache.hadoop.hive.ql.TestTxnCommands2WithSplitUpdateAndVectorization.testInitiatorWithMultipleFailedCompactions
(batchId=358)
org.apache.hadoop.hive.ql.TestTxnCommands2WithSplitUpdateAndVectorization.testInsertOverwrite1
(batchId=358)
org.apache.hadoop.hive.ql.TestTxnCommands2WithSplitUpdateAndVectorization.testInsertOverwrite2
(batchId=358)
org.apache.hadoop.hive.ql.TestTxnCommands2WithSplitUpdateAndVectorization.testNonAcidToAcidConversion02
(batchId=358)
org.apache.hadoop.hive.ql.TestTxnCommands2WithSplitUpdateAndVectorization.testNonAcidToAcidConversion1
(batchId=358)
org.apache.hadoop.hive.ql.TestTxnCommands2WithSplitUpdateAndVectorization.testNonAcidToAcidConversion2
(batchId=358)
org.apache.hadoop.hive.ql.TestTxnCommands2WithSplitUpdateAndVectorization.testNonAcidToAcidConversion3
(batchId=358)
org.apache.hadoop.hive.ql.TestTxnCommands2WithSplitUpdateAndVectorization.testOriginalFileReaderWhenNonAcidConvertedToAcid
(batchId=358)
org.apache.hadoop.hive.ql.TestTxnCommands2WithSplitUpdateAndVectorization.testSchemaEvolutionCompaction
(batchId=358)
org.apache.hadoop.hive.ql.TestTxnCommands2WithSplitUpdateAndVectorization.writeBetweenWorkerAndCleaner
(batchId=358)
org.apache.hadoop.hive.ql.TestTxnCommands3.testAcidMetaColumsDecode
(batchId=359)
org.apache.hadoop.hive.ql.TestTxnCommands3.testCleaner2 (batchId=359)
org.apache.hadoop.hive.ql.TestTxnCommands3.testCompactionAbort (batchId=359)
org.apache.hadoop.hive.ql.TestTxnCommands3.testDeleteEventPruningOff
(batchId=359)
org.apache.hadoop.hive.ql.TestTxnCommands3.testDeleteEventPruningOn
(batchId=359)
org.apache.hadoop.hive.ql.TestTxnCommands3.testNotEnoughToCompactNeedsCleaning
(batchId=359)
org.apache.hadoop.hive.ql.TestTxnCommandsForMmTable.testInsertOverwriteForPartitionedMmTable
(batchId=318)
[jira] [Commented] (HIVE-23109) Query-based compaction omits database
[
https://issues.apache.org/jira/browse/HIVE-23109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17072850#comment-17072850
]
Hive QA commented on HIVE-23109:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12998353/HIVE-23109.02.patch
{color:green}SUCCESS:{color} +1 due to 3 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 18166 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/21370/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/21370/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-21370/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12998353 - PreCommit-HIVE-Build
> Query-based compaction omits database
> -
>
> Key: HIVE-23109
> URL: https://issues.apache.org/jira/browse/HIVE-23109
> Project: Hive
> Issue Type: Bug
>Reporter: Karen Coppage
>Assignee: Karen Coppage
>Priority: Major
> Attachments: HIVE-23109.01.patch, HIVE-23109.02.patch
>
>
> E.g. MM major compaction query looks like:
> {code:java}
> insert into tmp_table select * from src_table;
> {code}
> it should be
> {code:java}
> insert into tmp_table select * from src_db.src_table;
> {code}
> Therefore compaction fails if db of source table isn't default.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23123) Export/Import of views and materialized views
[
https://issues.apache.org/jira/browse/HIVE-23123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17072779#comment-17072779
]
Miklos Gergely commented on HIVE-23123:
---
[~jcamachorodriguez] please let me know what you think.
> Export/Import of views and materialized views
> -
>
> Key: HIVE-23123
> URL: https://issues.apache.org/jira/browse/HIVE-23123
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Reporter: Miklos Gergely
>Assignee: Miklos Gergely
>Priority: Major
>
> According to
> [https://cwiki.apache.org/confluence/display/Hive/LanguageManual+ImportExport]
> import and export can be done by using the
>
> {code:java}
> export table ...
> import table ...
> {code}
> commands. The document doesn't mention views or materialized views at all,
> and in fact we don't support commands like
>
> {code:java}
> export view ...
> import view ...
> export materialized view ...
> import materialized view ...
> {code}
> they can not be parsed at all. The word table is often used though in a
> broader sense, when it means all table like entities, including views and
> materialized views. For example the various Table classes may represent any
> of these as well.
> If I try to export a view with the export table ... command, it goes fine. A
> _metadata file will be created, but no data directory, which is what we'd
> expect. If I try to import it back, an exception is thrown due to the lack of
> the data dir:
>
> {code:java}
> java.lang.AssertionError: null==getPath() for exim_view
> at org.apache.hadoop.hive.ql.metadata.Hive.loadTable(Hive.java:3088)
> at org.apache.hadoop.hive.ql.exec.MoveTask.execute(MoveTask.java:419)
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:213)
> at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:105)
> at org.apache.hadoop.hive.ql.Executor.launchTask(Executor.java:364)
> at org.apache.hadoop.hive.ql.Executor.launchTasks(Executor.java:335)
> at org.apache.hadoop.hive.ql.Executor.runTasks(Executor.java:246)
> at org.apache.hadoop.hive.ql.Executor.execute(Executor.java:109)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:722)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:491)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:485)
> {code}
> Still the view gets imported successfully, as data movement wasn't even
> necessary.
> If we try to export a materialized view which is transactional, then this
> exception occurs:
>
> {code:java}
> org.apache.hadoop.hive.ql.parse.SemanticException:
> org.apache.hadoop.hive.ql.metadata.InvalidTableException: Table not found
> exim_materialized_view_da21d41a_9fe4_4446_9c72_d251496abf9d
> at
> org.apache.hadoop.hive.ql.parse.AcidExportSemanticAnalyzer.analyzeAcidExport(AcidExportSemanticAnalyzer.java:163)
> at
> org.apache.hadoop.hive.ql.parse.AcidExportSemanticAnalyzer.analyze(AcidExportSemanticAnalyzer.java:71)
> at
> org.apache.hadoop.hive.ql.parse.RewriteSemanticAnalyzer.analyzeInternal(RewriteSemanticAnalyzer.java:72)
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:289)
> at org.apache.hadoop.hive.ql.Compiler.analyze(Compiler.java:220)
> at org.apache.hadoop.hive.ql.Compiler.compile(Compiler.java:104)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:183)
> at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:601)
> at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:547)
> at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:541)
> {code}
> So the export process can not handle it, as the temporary table is not
> getting created.
>
> The import command handling have a lot of codes dedicated to importing views
> and materialized views, which suggests that we support the importing (and
> thus also suggests implicitly that we support the exporting) of views and
> materialiezed views.
>
> So the conclusion is that we have to decide if we support exporting/importing
> of views and materialized views.
> If we decide not to support them then:
> - export process should throw an exception if a view or materialized view is
> the subject
> - the codes specific to view imports should be removed
> If we decide to support them, then:
> - the commands mentioned above above should be introduced
> - exception should be thrown if not the proper command used (e.g. export
> view on a table)
> - the exceptions mentioned above should be fixed
> I prefer not to support them, I don't think we should support the exporting /
> importing of views. The point of exporting / importing is the transfer of
> data, not DDL, it causes more issues than it solves. Our current
> documentation also suggests that it is only supported for tables.
>
[jira] [Updated] (HIVE-23123) Export/Import of views and materialized views
[
https://issues.apache.org/jira/browse/HIVE-23123?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Miklos Gergely updated HIVE-23123:
--
Description:
According to
[https://cwiki.apache.org/confluence/display/Hive/LanguageManual+ImportExport]
import and export can be done by using the
{code:java}
export table ...
import table ...
{code}
commands. The document doesn't mention views or materialized views at all, and
in fact we don't support commands like
{code:java}
export view ...
import view ...
export materialized view ...
import materialized view ...
{code}
they can not be parsed at all. The word table is often used though in a broader
sense, when it means all table like entities, including views and materialized
views. For example the various Table classes may represent any of these as well.
If I try to export a view with the export table ... command, it goes fine. A
_metadata file will be created, but no data directory, which is what we'd
expect. If I try to import it back, an exception is thrown due to the lack of
the data dir:
{code:java}
java.lang.AssertionError: null==getPath() for exim_view
at org.apache.hadoop.hive.ql.metadata.Hive.loadTable(Hive.java:3088)
at org.apache.hadoop.hive.ql.exec.MoveTask.execute(MoveTask.java:419)
at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:213)
at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:105)
at org.apache.hadoop.hive.ql.Executor.launchTask(Executor.java:364)
at org.apache.hadoop.hive.ql.Executor.launchTasks(Executor.java:335)
at org.apache.hadoop.hive.ql.Executor.runTasks(Executor.java:246)
at org.apache.hadoop.hive.ql.Executor.execute(Executor.java:109)
at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:722)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:491)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:485)
{code}
Still the view gets imported successfully, as data movement wasn't even
necessary.
If we try to export a materialized view which is transactional, then this
exception occurs:
{code:java}
org.apache.hadoop.hive.ql.parse.SemanticException:
org.apache.hadoop.hive.ql.metadata.InvalidTableException: Table not found
exim_materialized_view_da21d41a_9fe4_4446_9c72_d251496abf9d
at
org.apache.hadoop.hive.ql.parse.AcidExportSemanticAnalyzer.analyzeAcidExport(AcidExportSemanticAnalyzer.java:163)
at
org.apache.hadoop.hive.ql.parse.AcidExportSemanticAnalyzer.analyze(AcidExportSemanticAnalyzer.java:71)
at
org.apache.hadoop.hive.ql.parse.RewriteSemanticAnalyzer.analyzeInternal(RewriteSemanticAnalyzer.java:72)
at
org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:289)
at org.apache.hadoop.hive.ql.Compiler.analyze(Compiler.java:220)
at org.apache.hadoop.hive.ql.Compiler.compile(Compiler.java:104)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:183)
at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:601)
at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:547)
at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:541)
{code}
So the export process can not handle it, as the temporary table is not getting
created.
The import command handling have a lot of codes dedicated to importing views
and materialized views, which suggests that we support the importing (and thus
also suggests implicitly that we support the exporting) of views and
materialiezed views.
So the conclusion is that we have to decide if we support exporting/importing
of views and materialized views.
If we decide not to support them then:
- export process should throw an exception if a view or materialized view is
the subject
- the codes specific to view imports should be removed
If we decide to support them, then:
- the commands mentioned above above should be introduced
- exception should be thrown if not the proper command used (e.g. export view
on a table)
- the exceptions mentioned above should be fixed
I prefer not to support them, I don't think we should support the exporting /
importing of views. The point of exporting / importing is the transfer of data,
not DDL, it causes more issues than it solves. Our current documentation also
suggests that it is only supported for tables.
was:
According to
[https://cwiki.apache.org/confluence/display/Hive/LanguageManual+ImportExport]
import and export can be done by using the
{code:java}
export table ...
import table ...
{code}
commands. The document doesn't mention views or materialized views at all, and
in fact we don't support commands like
{code:java}
export view ...
import view ...
export materialized view ...
import materialized view ...
{code}
they can not be parsed at all. The word table is often used though in a broader
sense, when it means all table like entities, including views and materialized
views. For example the various Table classes may represent any of
[jira] [Updated] (HIVE-23122) LLAP: TaskExecutorService should log details about task eviction decision details
[
https://issues.apache.org/jira/browse/HIVE-23122?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
László Bodor updated HIVE-23122:
Summary: LLAP: TaskExecutorService should log details about task eviction
decision details (was: LLAP TaskExecutorService should log details about task
eviction decision details)
> LLAP: TaskExecutorService should log details about task eviction decision
> details
> -
>
> Key: HIVE-23122
> URL: https://issues.apache.org/jira/browse/HIVE-23122
> Project: Hive
> Issue Type: Improvement
>Reporter: László Bodor
>Assignee: László Bodor
>Priority: Major
> Attachments: HIVE-23122.02.patch
>
>
> TaskExecutorService maintains a waitQueue, and can evict a task in favor of
> another. Under the hood, the queue uses a configurable
> [comparator|https://github.com/apache/hive/tree/master/llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/comparator].
> The currently available comparators typically use the following properties
> of a task(wrapper):
> getWithinDagPriority: related to vertex
> currentAttemptStartTime
> firstAttemptStartTime
> knownPending: remaining upstream tasks
> The problem is, when an eviction happens, the INFO level message doesn't
> provide any insight about the decision, only attempts ids like below:
> {code}
> attempt_1585248378306_0010_72_02_96_8 evicted from wait queue in favor of
> attempt_1585248378306_0003_175_02_79_175 because of lower priority
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23124) Review of SQLOperation Class
[ https://issues.apache.org/jira/browse/HIVE-23124?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Mollitor updated HIVE-23124: -- Summary: Review of SQLOperation Class (was: Review of SQLOperation ) > Review of SQLOperation Class > > > Key: HIVE-23124 > URL: https://issues.apache.org/jira/browse/HIVE-23124 > Project: Hive > Issue Type: Improvement >Reporter: David Mollitor >Assignee: David Mollitor >Priority: Major > Attachments: HIVE-23124.1.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23089) Add constraint checks to CBO plan
[
https://issues.apache.org/jira/browse/HIVE-23089?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Krisztian Kasa updated HIVE-23089:
--
Attachment: HIVE-23089.1.patch
> Add constraint checks to CBO plan
> -
>
> Key: HIVE-23089
> URL: https://issues.apache.org/jira/browse/HIVE-23089
> Project: Hive
> Issue Type: Improvement
> Components: CBO
>Reporter: Krisztian Kasa
>Assignee: Krisztian Kasa
>Priority: Major
> Attachments: HIVE-23089.1.patch
>
>
> {code}
> create table acid_uami(i int,
> de decimal(5,2) constraint nn1 not null enforced,
> vc varchar(128) constraint nn2 not null enforced) clustered
> by (i) into 2 buckets stored as orc TBLPROPERTIES ('transactional'='true');
> explain
> update acid_uami set de=null where i=1;
> {code}
> Non-CBO path:
> {code:java}
> Map Operator Tree:
> TableScan
> alias: acid_uami
> filterExpr: ((i = 1) and enforce_constraint(vc is not null))
> (type: boolean)
> Statistics: Num rows: 1 Data size: 216 Basic stats: COMPLETE
> Column stats: NONE
> Filter Operator
> predicate: ((i = 1) and enforce_constraint(vc is not null))
> (type: boolean)
> {code}
> CBO path:
> {code:java}
> Map Reduce
> Map Operator Tree:
> TableScan
> alias: acid_uami
> filterExpr: (i = 1) (type: boolean)
> Statistics: Num rows: 1 Data size: 216 Basic stats: COMPLETE
> Column stats: NONE
> Filter Operator
> predicate: (i = 1) (type: boolean)
> ...
> Reduce Operator Tree:
> ...
> Filter Operator
> predicate: enforce_constraint((null is not null and _col3 is not
> null)) (type: boolean)
> {code}
> In CBO path the enforce_constraint function is added to the plan when CBO
> plan is already generated and optimized.
> {code}
> HiveSortExchange(distribution=[any], collation=[[0]])
> HiveProject(row__id=[$5], i=[CAST(1):INTEGER], _o__c2=[null:NULL], vc=[$2])
> HiveFilter(condition=[=($0, 1)])
> HiveTableScan(table=[[default, acid_uami]], table:alias=[acid_uami])
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23109) Query-based compaction omits database
[
https://issues.apache.org/jira/browse/HIVE-23109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17072809#comment-17072809
]
Hive QA commented on HIVE-23109:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
39s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
49s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
46s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
57s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
43s{color} | {color:blue} ql in master has 1528 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
43s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
21s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
27s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
40s{color} | {color:green} ql: The patch generated 0 new + 0 unchanged - 1
fixed = 0 total (was 1) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
17s{color} | {color:green} The patch hive-unit passed checkstyle {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
20s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 30m 9s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-21370/dev-support/hive-personality.sh
|
| git revision | master / 484d182 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| modules | C: ql itests/hive-unit U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-21370/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Query-based compaction omits database
> -
>
> Key: HIVE-23109
> URL: https://issues.apache.org/jira/browse/HIVE-23109
> Project: Hive
> Issue Type: Bug
>Reporter: Karen Coppage
>Assignee: Karen Coppage
>Priority: Major
> Attachments: HIVE-23109.01.patch, HIVE-23109.02.patch
>
>
> E.g. MM major compaction query looks like:
> {code:java}
> insert into tmp_table select * from src_table;
> {code}
> it should be
> {code:java}
> insert into tmp_table select * from src_db.src_table;
> {code}
> Therefore compaction fails if db of source table isn't default.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23114) Insert overwrite with dynamic partitioning is not working correctly with direct insert
[
https://issues.apache.org/jira/browse/HIVE-23114?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17072883#comment-17072883
]
Hive QA commented on HIVE-23114:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
42s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
57s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
0s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
44s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
45s{color} | {color:blue} ql in master has 1528 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
54s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
29s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
2s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
44s{color} | {color:red} ql: The patch generated 1 new + 314 unchanged - 0
fixed = 315 total (was 314) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 3m
54s{color} | {color:red} ql generated 1 new + 1528 unchanged - 0 fixed = 1529
total (was 1528) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
55s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 25m 25s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:ql |
| | The field
org.apache.hadoop.hive.ql.exec.FileSinkOperator.dynamicPartitionSpecs is
transient but isn't set by deserialization In FileSinkOperator.java:but isn't
set by deserialization In FileSinkOperator.java |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-21371/dev-support/hive-personality.sh
|
| git revision | master / 709235c |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-21371/yetus/diff-checkstyle-ql.txt
|
| findbugs |
http://104.198.109.242/logs//PreCommit-HIVE-Build-21371/yetus/new-findbugs-ql.html
|
| modules | C: ql itests U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-21371/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Insert overwrite with dynamic partitioning is not working correctly with
> direct insert
> --
>
> Key: HIVE-23114
> URL: https://issues.apache.org/jira/browse/HIVE-23114
> Project: Hive
> Issue Type: Bug
>Reporter: Marta Kuczora
>Assignee: Marta Kuczora
>Priority: Major
> Attachments: HIVE-23114.1.patch
>
>
> This is a follow-up Jira for the
> [conversation|https://issues.apache.org/jira/browse/HIVE-21164?focusedCommentId=17059280=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-17059280]
> in HIVE-21164
> Doing an insert overwrite from a multi-insert statement with dynamic
> partitioning will give wrong results for ACID tables
[jira] [Updated] (HIVE-23124) Review of SQLOperation Class
[
https://issues.apache.org/jira/browse/HIVE-23124?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Mollitor updated HIVE-23124:
--
Description:
* Use ConcurrentHashMap instead of synchronized methods to improve
multi-threaded access
* Use JDK 8 facilities where applicable
* General cleanup
* Better log messages and Exception messages
* Use {{switch}} statement instead of if/else blocks
* Checkstyle fixes
> Review of SQLOperation Class
>
>
> Key: HIVE-23124
> URL: https://issues.apache.org/jira/browse/HIVE-23124
> Project: Hive
> Issue Type: Improvement
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Major
> Attachments: HIVE-23124.1.patch
>
>
> * Use ConcurrentHashMap instead of synchronized methods to improve
> multi-threaded access
> * Use JDK 8 facilities where applicable
> * General cleanup
> * Better log messages and Exception messages
> * Use {{switch}} statement instead of if/else blocks
> * Checkstyle fixes
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (HIVE-18956) AvroSerDe Race Condition
[
https://issues.apache.org/jira/browse/HIVE-18956?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Mollitor resolved HIVE-18956.
---
Resolution: Won't Fix
> AvroSerDe Race Condition
>
>
> Key: HIVE-18956
> URL: https://issues.apache.org/jira/browse/HIVE-18956
> Project: Hive
> Issue Type: Bug
> Components: Serializers/Deserializers
>Affects Versions: 3.0.0, 2.3.2
>Reporter: David Mollitor
>Priority: Trivial
>
> {code}
> @Override
> public Writable serialize(Object o, ObjectInspector objectInspector) throws
> SerDeException {
> if(badSchema) {
> throw new BadSchemaException();
> }
> return getSerializer().serialize(o, objectInspector, columnNames,
> columnTypes, schema);
> }
> @Override
> public Object deserialize(Writable writable) throws SerDeException {
> if(badSchema) {
> throw new BadSchemaException();
> }
> return getDeserializer().deserialize(columnNames, columnTypes, writable,
> schema);
> }
> ...
> private AvroDeserializer getDeserializer() {
> if(avroDeserializer == null) {
> avroDeserializer = new AvroDeserializer();
> }
> return avroDeserializer;
> }
> private AvroSerializer getSerializer() {
> if(avroSerializer == null) {
> avroSerializer = new AvroSerializer();
> }
> return avroSerializer;
> }
> {code}
> {{getDeserializer}} and {{getSerializer}} methods are not thread safe, so
> neither are {{deserialize}} and {{serialize}} methods. It probably didn't
> matter with MapReduce, but now that we have Spark/Tez, it may be an issue.
> You could visualize a scenario where three threads all enter
> {{getSerializer}} and all see that {{avroSerializer}} is _null_ and create
> three instances, then they would fight to assign the new object to the
> {{avroSerializer}} variable.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23122) LLAP TaskExecutorService should log details about task eviction decision details
[
https://issues.apache.org/jira/browse/HIVE-23122?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
László Bodor updated HIVE-23122:
Attachment: HIVE-23122.02.patch
> LLAP TaskExecutorService should log details about task eviction decision
> details
>
>
> Key: HIVE-23122
> URL: https://issues.apache.org/jira/browse/HIVE-23122
> Project: Hive
> Issue Type: Improvement
>Reporter: László Bodor
>Assignee: László Bodor
>Priority: Major
> Attachments: HIVE-23122.02.patch
>
>
> TaskExecutorService maintains a waitQueue, and can evict a task in favor of
> another. Under the hood, the queue uses a configurable
> [comparator|https://github.com/apache/hive/tree/master/llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/comparator].
> The currently available comparators typically use the following properties
> of a task(wrapper):
> getWithinDagPriority: related to vertex
> currentAttemptStartTime
> firstAttemptStartTime
> knownPending: remaining upstream tasks
> The problem is, when an eviction happens, the INFO level message doesn't
> provide any insight about the decision, only attempts ids like below:
> {code}
> attempt_1585248378306_0010_72_02_96_8 evicted from wait queue in favor of
> attempt_1585248378306_0003_175_02_79_175 because of lower priority
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23122) LLAP TaskExecutorService should log details about task eviction decision details
[
https://issues.apache.org/jira/browse/HIVE-23122?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
László Bodor updated HIVE-23122:
Attachment: (was: HIVE-23122.01.patch)
> LLAP TaskExecutorService should log details about task eviction decision
> details
>
>
> Key: HIVE-23122
> URL: https://issues.apache.org/jira/browse/HIVE-23122
> Project: Hive
> Issue Type: Improvement
>Reporter: László Bodor
>Assignee: László Bodor
>Priority: Major
> Attachments: HIVE-23122.02.patch
>
>
> TaskExecutorService maintains a waitQueue, and can evict a task in favor of
> another. Under the hood, the queue uses a configurable
> [comparator|https://github.com/apache/hive/tree/master/llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/comparator].
> The currently available comparators typically use the following properties
> of a task(wrapper):
> getWithinDagPriority: related to vertex
> currentAttemptStartTime
> firstAttemptStartTime
> knownPending: remaining upstream tasks
> The problem is, when an eviction happens, the INFO level message doesn't
> provide any insight about the decision, only attempts ids like below:
> {code}
> attempt_1585248378306_0010_72_02_96_8 evicted from wait queue in favor of
> attempt_1585248378306_0003_175_02_79_175 because of lower priority
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23109) Query-based compaction omits database
[
https://issues.apache.org/jira/browse/HIVE-23109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17072869#comment-17072869
]
László Pintér commented on HIVE-23109:
--
+1
> Query-based compaction omits database
> -
>
> Key: HIVE-23109
> URL: https://issues.apache.org/jira/browse/HIVE-23109
> Project: Hive
> Issue Type: Bug
>Reporter: Karen Coppage
>Assignee: Karen Coppage
>Priority: Major
> Attachments: HIVE-23109.01.patch, HIVE-23109.02.patch
>
>
> E.g. MM major compaction query looks like:
> {code:java}
> insert into tmp_table select * from src_table;
> {code}
> it should be
> {code:java}
> insert into tmp_table select * from src_db.src_table;
> {code}
> Therefore compaction fails if db of source table isn't default.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23122) LLAP: TaskExecutorService should log details about task eviction decision details
[
https://issues.apache.org/jira/browse/HIVE-23122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17072785#comment-17072785
]
Panagiotis Garefalakis commented on HIVE-23122:
---
Thanks [~abstractdog]
+1 tests pending
> LLAP: TaskExecutorService should log details about task eviction decision
> details
> -
>
> Key: HIVE-23122
> URL: https://issues.apache.org/jira/browse/HIVE-23122
> Project: Hive
> Issue Type: Improvement
>Reporter: László Bodor
>Assignee: László Bodor
>Priority: Major
> Attachments: HIVE-23122.02.patch
>
>
> TaskExecutorService maintains a waitQueue, and can evict a task in favor of
> another. Under the hood, the queue uses a configurable
> [comparator|https://github.com/apache/hive/tree/master/llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/comparator].
> The currently available comparators typically use the following properties
> of a task(wrapper):
> getWithinDagPriority: related to vertex
> currentAttemptStartTime
> firstAttemptStartTime
> knownPending: remaining upstream tasks
> The problem is, when an eviction happens, the INFO level message doesn't
> provide any insight about the decision, only attempts ids like below:
> {code}
> attempt_1585248378306_0010_72_02_96_8 evicted from wait queue in favor of
> attempt_1585248378306_0003_175_02_79_175 because of lower priority
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23089) Add constraint checks to CBO plan
[
https://issues.apache.org/jira/browse/HIVE-23089?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Krisztian Kasa updated HIVE-23089:
--
Status: Patch Available (was: Open)
> Add constraint checks to CBO plan
> -
>
> Key: HIVE-23089
> URL: https://issues.apache.org/jira/browse/HIVE-23089
> Project: Hive
> Issue Type: Improvement
> Components: CBO
>Reporter: Krisztian Kasa
>Assignee: Krisztian Kasa
>Priority: Major
> Attachments: HIVE-23089.1.patch
>
>
> {code}
> create table acid_uami(i int,
> de decimal(5,2) constraint nn1 not null enforced,
> vc varchar(128) constraint nn2 not null enforced) clustered
> by (i) into 2 buckets stored as orc TBLPROPERTIES ('transactional'='true');
> explain
> update acid_uami set de=null where i=1;
> {code}
> Non-CBO path:
> {code:java}
> Map Operator Tree:
> TableScan
> alias: acid_uami
> filterExpr: ((i = 1) and enforce_constraint(vc is not null))
> (type: boolean)
> Statistics: Num rows: 1 Data size: 216 Basic stats: COMPLETE
> Column stats: NONE
> Filter Operator
> predicate: ((i = 1) and enforce_constraint(vc is not null))
> (type: boolean)
> {code}
> CBO path:
> {code:java}
> Map Reduce
> Map Operator Tree:
> TableScan
> alias: acid_uami
> filterExpr: (i = 1) (type: boolean)
> Statistics: Num rows: 1 Data size: 216 Basic stats: COMPLETE
> Column stats: NONE
> Filter Operator
> predicate: (i = 1) (type: boolean)
> ...
> Reduce Operator Tree:
> ...
> Filter Operator
> predicate: enforce_constraint((null is not null and _col3 is not
> null)) (type: boolean)
> {code}
> In CBO path the enforce_constraint function is added to the plan when CBO
> plan is already generated and optimized.
> {code}
> HiveSortExchange(distribution=[any], collation=[[0]])
> HiveProject(row__id=[$5], i=[CAST(1):INTEGER], _o__c2=[null:NULL], vc=[$2])
> HiveFilter(condition=[=($0, 1)])
> HiveTableScan(table=[[default, acid_uami]], table:alias=[acid_uami])
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (HIVE-23124) Review of SQLOperation
[ https://issues.apache.org/jira/browse/HIVE-23124?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Mollitor reassigned HIVE-23124: - Assignee: David Mollitor > Review of SQLOperation > --- > > Key: HIVE-23124 > URL: https://issues.apache.org/jira/browse/HIVE-23124 > Project: Hive > Issue Type: Improvement >Reporter: David Mollitor >Assignee: David Mollitor >Priority: Major > Attachments: HIVE-23124.1.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23124) Review of SQLOperation Class
[ https://issues.apache.org/jira/browse/HIVE-23124?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Mollitor updated HIVE-23124: -- Attachment: HIVE-23124.1.patch > Review of SQLOperation Class > > > Key: HIVE-23124 > URL: https://issues.apache.org/jira/browse/HIVE-23124 > Project: Hive > Issue Type: Improvement >Reporter: David Mollitor >Assignee: David Mollitor >Priority: Major > Attachments: HIVE-23124.1.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23124) Review of SQLOperation Class
[ https://issues.apache.org/jira/browse/HIVE-23124?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Mollitor updated HIVE-23124: -- Status: Patch Available (was: Open) > Review of SQLOperation Class > > > Key: HIVE-23124 > URL: https://issues.apache.org/jira/browse/HIVE-23124 > Project: Hive > Issue Type: Improvement >Reporter: David Mollitor >Assignee: David Mollitor >Priority: Major > Attachments: HIVE-23124.1.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23122) LLAP TaskExecutorService should log details about task eviction decision details
[
https://issues.apache.org/jira/browse/HIVE-23122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17072783#comment-17072783
]
László Bodor commented on HIVE-23122:
-
thanks for the comments [~pgaref], included canFinishForPriority
removing 01.patch for saving ptest time
> LLAP TaskExecutorService should log details about task eviction decision
> details
>
>
> Key: HIVE-23122
> URL: https://issues.apache.org/jira/browse/HIVE-23122
> Project: Hive
> Issue Type: Improvement
>Reporter: László Bodor
>Assignee: László Bodor
>Priority: Major
> Attachments: HIVE-23122.02.patch
>
>
> TaskExecutorService maintains a waitQueue, and can evict a task in favor of
> another. Under the hood, the queue uses a configurable
> [comparator|https://github.com/apache/hive/tree/master/llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/comparator].
> The currently available comparators typically use the following properties
> of a task(wrapper):
> getWithinDagPriority: related to vertex
> currentAttemptStartTime
> firstAttemptStartTime
> knownPending: remaining upstream tasks
> The problem is, when an eviction happens, the INFO level message doesn't
> provide any insight about the decision, only attempts ids like below:
> {code}
> attempt_1585248378306_0010_72_02_96_8 evicted from wait queue in favor of
> attempt_1585248378306_0003_175_02_79_175 because of lower priority
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23122) LLAP: TaskExecutorService should log details about task eviction decision details
[
https://issues.apache.org/jira/browse/HIVE-23122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17072821#comment-17072821
]
László Bodor commented on HIVE-23122:
-
cc: [~rajesh.balamohan]
> LLAP: TaskExecutorService should log details about task eviction decision
> details
> -
>
> Key: HIVE-23122
> URL: https://issues.apache.org/jira/browse/HIVE-23122
> Project: Hive
> Issue Type: Improvement
>Reporter: László Bodor
>Assignee: László Bodor
>Priority: Major
> Attachments: HIVE-23122.02.patch
>
>
> TaskExecutorService maintains a waitQueue, and can evict a task in favor of
> another. Under the hood, the queue uses a configurable
> [comparator|https://github.com/apache/hive/tree/master/llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/comparator].
> The currently available comparators typically use the following properties
> of a task(wrapper):
> getWithinDagPriority: related to vertex
> currentAttemptStartTime
> firstAttemptStartTime
> knownPending: remaining upstream tasks
> The problem is, when an eviction happens, the INFO level message doesn't
> provide any insight about the decision, only attempts ids like below:
> {code}
> attempt_1585248378306_0010_72_02_96_8 evicted from wait queue in favor of
> attempt_1585248378306_0003_175_02_79_175 because of lower priority
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23114) Insert overwrite with dynamic partitioning is not working correctly with direct insert
[
https://issues.apache.org/jira/browse/HIVE-23114?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17072955#comment-17072955
]
Hive QA commented on HIVE-23114:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12998356/HIVE-23114.1.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 18166 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/21371/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/21371/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-21371/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12998356 - PreCommit-HIVE-Build
> Insert overwrite with dynamic partitioning is not working correctly with
> direct insert
> --
>
> Key: HIVE-23114
> URL: https://issues.apache.org/jira/browse/HIVE-23114
> Project: Hive
> Issue Type: Bug
>Reporter: Marta Kuczora
>Assignee: Marta Kuczora
>Priority: Major
> Attachments: HIVE-23114.1.patch
>
>
> This is a follow-up Jira for the
> [conversation|https://issues.apache.org/jira/browse/HIVE-21164?focusedCommentId=17059280=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-17059280]
> in HIVE-21164
> Doing an insert overwrite from a multi-insert statement with dynamic
> partitioning will give wrong results for ACID tables when
> 'hive.acid.direct.insert.enabled' is true or for insert-only tables.
> Reproduction:
> {noformat}
> set hive.acid.direct.insert.enabled=true;
> set hive.support.concurrency=true;
> set hive.txn.manager=org.apache.hadoop.hive.ql.lockmgr.DbTxnManager;
> set hive.vectorized.execution.enabled=false;
> set hive.stats.autogather=false;
> create external table multiinsert_test_text (a int, b int, c int) stored as
> textfile;
> insert into multiinsert_test_text values (, 11, ), (, 22, ),
> (, 33, ), (, 44, NULL), (, 55, NULL);
> create table multiinsert_test_acid (a int, b int) partitioned by (c int)
> stored as orc tblproperties('transactional'='true');
> create table multiinsert_test_mm (a int, b int) partitioned by (c int) stored
> as orc tblproperties('transactional'='true',
> 'transactional_properties'='insert_only');
> from multiinsert_test_text a
> insert overwrite table multiinsert_test_acid partition (c)
> select
> a.a,
> a.b,
> a.c
> where a.c is not null
> insert overwrite table multiinsert_test_acid partition (c)
> select
> a.a,
> a.b,
> a.c
> where a.c is null;
> select * from multiinsert_test_acid;
> from multiinsert_test_text a
> insert overwrite table multiinsert_test_mm partition (c)
> select
> a.a,
> a.b,
> a.c
> where a.c is not null
> insert overwrite table multiinsert_test_mm partition (c)
> select
> a.a,
> a.b,
> a.c
> where a.c is null;
> select * from multiinsert_test_mm;
> {noformat}
> The result of these steps can be different, it depends on the execution order
> of the FileSinkOperators of the insert overwrite statements. It can happen
> that an error occurs due to manifest file collision, it can happen that no
> error occurs but the result will be incorrect.
> Running the same insert query with an external table of with and ACID table
> with 'hive.acid.direct.insert.enabled=false' will give the follwing result:
> {noformat}
> 11
> 22
> 33
> 44 NULL
> 55 NULL
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23039) Checkpointing for repl dump bootstrap phase
[
https://issues.apache.org/jira/browse/HIVE-23039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17073001#comment-17073001
]
Hive QA commented on HIVE-23039:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
46s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
7s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
44s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
8s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
44s{color} | {color:blue} ql in master has 1528 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
40s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
22s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
29s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
47s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
40s{color} | {color:red} ql: The patch generated 9 new + 59 unchanged - 0 fixed
= 68 total (was 59) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
23s{color} | {color:red} itests/hive-unit: The patch generated 2 new + 688
unchanged - 0 fixed = 690 total (was 688) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
24s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 31m 9s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-21372/dev-support/hive-personality.sh
|
| git revision | master / 709235c |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-21372/yetus/diff-checkstyle-ql.txt
|
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-21372/yetus/diff-checkstyle-itests_hive-unit.txt
|
| modules | C: ql itests/hive-unit U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-21372/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Checkpointing for repl dump bootstrap phase
> ---
>
> Key: HIVE-23039
> URL: https://issues.apache.org/jira/browse/HIVE-23039
> Project: Hive
> Issue Type: Bug
>Reporter: Aasha Medhi
>Assignee: Aasha Medhi
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-23039.01.patch, HIVE-23039.02.patch
>
> Time Spent: 50m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23067) Use batch DB calls in TxnHandler for commitTxn and abortTxns
[ https://issues.apache.org/jira/browse/HIVE-23067?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marton Bod updated HIVE-23067: -- Attachment: (was: HIVE-23067.3.patch) > Use batch DB calls in TxnHandler for commitTxn and abortTxns > > > Key: HIVE-23067 > URL: https://issues.apache.org/jira/browse/HIVE-23067 > Project: Hive > Issue Type: Improvement >Reporter: Marton Bod >Assignee: Marton Bod >Priority: Major > Attachments: HIVE-23067.1.patch, HIVE-23067.1.patch, > HIVE-23067.1.patch, HIVE-23067.1.patch, HIVE-23067.2.patch, > HIVE-23067.2.patch, HIVE-23067.4.patch > > > To reduce the number of database calls and network roundtrips, we could use > more batching in TxnHandler, where currently in many places we call insert > commands in loops sequentially. > Some examples: > * openTxns (TXNS, REPL_TXN_MAP) > * commitTxn (COMPLETED_TXN_COMPONENTS) > * replTableWriteIdState (TXN_TO_WRITE_ID) > * allocateTableWriteIds (TXN_TO_WRITE_ID) > * > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23067) Use batch DB calls in TxnHandler for commitTxn and abortTxns
[ https://issues.apache.org/jira/browse/HIVE-23067?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marton Bod updated HIVE-23067: -- Attachment: HIVE-23067.4.patch > Use batch DB calls in TxnHandler for commitTxn and abortTxns > > > Key: HIVE-23067 > URL: https://issues.apache.org/jira/browse/HIVE-23067 > Project: Hive > Issue Type: Improvement >Reporter: Marton Bod >Assignee: Marton Bod >Priority: Major > Attachments: HIVE-23067.1.patch, HIVE-23067.1.patch, > HIVE-23067.1.patch, HIVE-23067.1.patch, HIVE-23067.2.patch, > HIVE-23067.2.patch, HIVE-23067.4.patch > > > To reduce the number of database calls and network roundtrips, we could use > more batching in TxnHandler, where currently in many places we call insert > commands in loops sequentially. > Some examples: > * openTxns (TXNS, REPL_TXN_MAP) > * commitTxn (COMPLETED_TXN_COMPONENTS) > * replTableWriteIdState (TXN_TO_WRITE_ID) > * allocateTableWriteIds (TXN_TO_WRITE_ID) > * > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HIVE-18897) Hive is Double-Logging Invalid UDF Error
[
https://issues.apache.org/jira/browse/HIVE-18897?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Mollitor resolved HIVE-18897.
---
Resolution: Not A Problem
Based on the current master branch, it looks like the logging was fixed to
remove the ERROR log in the {{Driver}} class.
> Hive is Double-Logging Invalid UDF Error
>
>
> Key: HIVE-18897
> URL: https://issues.apache.org/jira/browse/HIVE-18897
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2
>Affects Versions: 3.0.0, 2.4.0
>Reporter: David Mollitor
>Assignee: Ramesh Kumar Thangarajan
>Priority: Minor
> Labels: newbie, noob
>
> It logs the "invalid" function twice. Once at ERROR level and once at WARN
> level. Please change so that the error is logged once, at the WARN level.
> The stack trace seems overkill here as well for such a trivial error...
> usually a user typo or the function needs to be registered.
> {code:java}
> 2018-03-05 07:50:44,473 ERROR org.apache.hadoop.hive.ql.Driver:
> [HiveServer2-Handler-Pool: Thread-43]: FAILED: SemanticException [Error
> 10011]: Line 1:7 Invalid function 'aes_encrypt'
> org.apache.hadoop.hive.ql.parse.SemanticException: Line 1:7 Invalid function
> 'aes_encrypt'
> at
> org.apache.hadoop.hive.ql.parse.TypeCheckProcFactory$DefaultExprProcessor.getXpathOrFuncExprNodeDesc(TypeCheckProcFactory.java:836)
> at
> org.apache.hadoop.hive.ql.parse.TypeCheckProcFactory$DefaultExprProcessor.process(TypeCheckProcFactory.java:1176)
> at
> org.apache.hadoop.hive.ql.lib.DefaultRuleDispatcher.dispatch(DefaultRuleDispatcher.java:90)
> at
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatchAndReturn(DefaultGraphWalker.java:94)
> at
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatch(DefaultGraphWalker.java:78)
> at
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.walk(DefaultGraphWalker.java:132)
> at
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.startWalking(DefaultGraphWalker.java:109)
> at
> org.apache.hadoop.hive.ql.parse.TypeCheckProcFactory.genExprNode(TypeCheckProcFactory.java:193)
> at
> org.apache.hadoop.hive.ql.parse.TypeCheckProcFactory.genExprNode(TypeCheckProcFactory.java:146)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genAllExprNodeDesc(SemanticAnalyzer.java:10422)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genExprNodeDesc(SemanticAnalyzer.java:10378)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genSelectPlan(SemanticAnalyzer.java:3771)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genSelectPlan(SemanticAnalyzer.java:3550)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPostGroupByBodyPlan(SemanticAnalyzer.java:8830)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genBodyPlan(SemanticAnalyzer.java:8785)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPlan(SemanticAnalyzer.java:9652)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPlan(SemanticAnalyzer.java:9545)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genOPTree(SemanticAnalyzer.java:10018)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:10029)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:9909)
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:223)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:488)
> at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1274)
> at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:1261)
> at
> org.apache.hive.service.cli.operation.SQLOperation.prepare(SQLOperation.java:143)
> at
> org.apache.hive.service.cli.operation.SQLOperation.runInternal(SQLOperation.java:215)
> at
> org.apache.hive.service.cli.operation.Operation.run(Operation.java:337)
> at
> org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementInternal(HiveSessionImpl.java:425)
> at
> org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementAsync(HiveSessionImpl.java:402)
> at
> org.apache.hive.service.cli.CLIService.executeStatementAsync(CLIService.java:258)
> at
> org.apache.hive.service.cli.thrift.ThriftCLIService.ExecuteStatement(ThriftCLIService.java:500)
> at
> org.apache.hive.service.cli.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1313)
> at
> org.apache.hive.service.cli.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1298)
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> at
[jira] [Commented] (HIVE-23106) Cleanup CalcitePlanner genOPTree exception handling
[
https://issues.apache.org/jira/browse/HIVE-23106?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17073189#comment-17073189
]
Hive QA commented on HIVE-23106:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12998377/HIVE-23106.2.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 18162 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/21374/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/21374/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-21374/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12998377 - PreCommit-HIVE-Build
> Cleanup CalcitePlanner genOPTree exception handling
> ---
>
> Key: HIVE-23106
> URL: https://issues.apache.org/jira/browse/HIVE-23106
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2
>Reporter: John Sherman
>Assignee: John Sherman
>Priority: Minor
> Attachments: HIVE-23106.1.patch, HIVE-23106.2.patch
>
>
> The logic where genOPTree handles exceptions during CBO planning is a bit
> twisty and could use some cleanup and comments.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23126) Add TXNS table to sysdb
[ https://issues.apache.org/jira/browse/HIVE-23126?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ramesh Kumar Thangarajan updated HIVE-23126: Attachment: HIVE-23126.1.patch Status: Patch Available (was: Open) > Add TXNS table to sysdb > --- > > Key: HIVE-23126 > URL: https://issues.apache.org/jira/browse/HIVE-23126 > Project: Hive > Issue Type: Task >Reporter: Ramesh Kumar Thangarajan >Assignee: Ramesh Kumar Thangarajan >Priority: Major > Attachments: HIVE-23126.1.patch > > > Add TXNS table to sysdb -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23106) Cleanup CalcitePlanner genOPTree exception handling
[
https://issues.apache.org/jira/browse/HIVE-23106?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17073159#comment-17073159
]
Hive QA commented on HIVE-23106:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
26s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
1s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
40s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
43s{color} | {color:blue} ql in master has 1528 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
55s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
42s{color} | {color:green} ql: The patch generated 0 new + 115 unchanged - 7
fixed = 115 total (was 122) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
56s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 24m 39s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-21374/dev-support/hive-personality.sh
|
| git revision | master / 709235c |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-21374/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Cleanup CalcitePlanner genOPTree exception handling
> ---
>
> Key: HIVE-23106
> URL: https://issues.apache.org/jira/browse/HIVE-23106
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2
>Reporter: John Sherman
>Assignee: John Sherman
>Priority: Minor
> Attachments: HIVE-23106.1.patch, HIVE-23106.2.patch
>
>
> The logic where genOPTree handles exceptions during CBO planning is a bit
> twisty and could use some cleanup and comments.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23094) Implement Explain CBO of Update and Delete statements
[
https://issues.apache.org/jira/browse/HIVE-23094?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17073141#comment-17073141
]
Hive QA commented on HIVE-23094:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12998369/HIVE-23094.1.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 18162 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/21373/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/21373/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-21373/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12998369 - PreCommit-HIVE-Build
> Implement Explain CBO of Update and Delete statements
> -
>
> Key: HIVE-23094
> URL: https://issues.apache.org/jira/browse/HIVE-23094
> Project: Hive
> Issue Type: Improvement
> Components: CBO
>Reporter: Krisztian Kasa
>Assignee: Krisztian Kasa
>Priority: Major
> Attachments: HIVE-23094.1.patch
>
>
> {code}
> set hive.support.concurrency=true;
> set hive.txn.manager=org.apache.hadoop.hive.ql.lockmgr.DbTxnManager;
> create table acidtlb(a int, b int) clustered by (a) into 2 buckets stored as
> orc TBLPROPERTIES ('transactional'='true');
> explain cbo
> update acidtlb set b=777;
> {code}
> doesn't print CBO plan.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23039) Checkpointing for repl dump bootstrap phase
[
https://issues.apache.org/jira/browse/HIVE-23039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17073058#comment-17073058
]
Hive QA commented on HIVE-23039:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12998454/HIVE-23039.02.patch
{color:green}SUCCESS:{color} +1 due to 3 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 8 failed/errored test(s), 18168 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.ql.parse.TestReplWithJsonMessageFormat.testCheckPointing
(batchId=260)
org.apache.hadoop.hive.ql.parse.TestReplWithJsonMessageFormat.testCheckPointingInDumpFailure
(batchId=260)
org.apache.hadoop.hive.ql.parse.TestReplicationOnHDFSEncryptedZones.targetAndSourceHaveDifferentEncryptionZoneKeys
(batchId=283)
org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testCheckPointing
(batchId=270)
org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testCheckPointingInDumpFailure
(batchId=270)
org.apache.hadoop.hive.ql.parse.TestReplicationScenariosExternalTables.externalTableIncrementalReplication
(batchId=268)
org.apache.hadoop.hive.ql.parse.TestReplicationScenariosExternalTables.externalTableReplicationWithCustomPaths
(batchId=268)
org.apache.hadoop.hive.ql.parse.TestReplicationScenariosExternalTables.externalTableWithPartitions
(batchId=268)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/21372/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/21372/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-21372/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 8 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12998454 - PreCommit-HIVE-Build
> Checkpointing for repl dump bootstrap phase
> ---
>
> Key: HIVE-23039
> URL: https://issues.apache.org/jira/browse/HIVE-23039
> Project: Hive
> Issue Type: Bug
>Reporter: Aasha Medhi
>Assignee: Aasha Medhi
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-23039.01.patch, HIVE-23039.02.patch,
> HIVE-23039.03.patch
>
> Time Spent: 50m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23039) Checkpointing for repl dump bootstrap phase
[ https://issues.apache.org/jira/browse/HIVE-23039?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aasha Medhi updated HIVE-23039: --- Attachment: HIVE-23039.03.patch Status: Patch Available (was: In Progress) > Checkpointing for repl dump bootstrap phase > --- > > Key: HIVE-23039 > URL: https://issues.apache.org/jira/browse/HIVE-23039 > Project: Hive > Issue Type: Bug >Reporter: Aasha Medhi >Assignee: Aasha Medhi >Priority: Major > Labels: pull-request-available > Attachments: HIVE-23039.01.patch, HIVE-23039.02.patch, > HIVE-23039.03.patch > > Time Spent: 50m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-23126) Add TXNS table to sysdb
[ https://issues.apache.org/jira/browse/HIVE-23126?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ramesh Kumar Thangarajan reassigned HIVE-23126: --- > Add TXNS table to sysdb > --- > > Key: HIVE-23126 > URL: https://issues.apache.org/jira/browse/HIVE-23126 > Project: Hive > Issue Type: Task >Reporter: Ramesh Kumar Thangarajan >Assignee: Ramesh Kumar Thangarajan >Priority: Major > > Add TXNS table to sysdb -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23039) Checkpointing for repl dump bootstrap phase
[ https://issues.apache.org/jira/browse/HIVE-23039?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aasha Medhi updated HIVE-23039: --- Status: In Progress (was: Patch Available) > Checkpointing for repl dump bootstrap phase > --- > > Key: HIVE-23039 > URL: https://issues.apache.org/jira/browse/HIVE-23039 > Project: Hive > Issue Type: Bug >Reporter: Aasha Medhi >Assignee: Aasha Medhi >Priority: Major > Labels: pull-request-available > Attachments: HIVE-23039.01.patch, HIVE-23039.02.patch > > Time Spent: 50m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23094) Implement Explain CBO of Update and Delete statements
[
https://issues.apache.org/jira/browse/HIVE-23094?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17073085#comment-17073085
]
Hive QA commented on HIVE-23094:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
13s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
1s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
39s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
45s{color} | {color:blue} ql in master has 1528 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
53s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
54s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 24m 13s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-21373/dev-support/hive-personality.sh
|
| git revision | master / 709235c |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-21373/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Implement Explain CBO of Update and Delete statements
> -
>
> Key: HIVE-23094
> URL: https://issues.apache.org/jira/browse/HIVE-23094
> Project: Hive
> Issue Type: Improvement
> Components: CBO
>Reporter: Krisztian Kasa
>Assignee: Krisztian Kasa
>Priority: Major
> Attachments: HIVE-23094.1.patch
>
>
> {code}
> set hive.support.concurrency=true;
> set hive.txn.manager=org.apache.hadoop.hive.ql.lockmgr.DbTxnManager;
> create table acidtlb(a int, b int) clustered by (a) into 2 buckets stored as
> orc TBLPROPERTIES ('transactional'='true');
> explain cbo
> update acidtlb set b=777;
> {code}
> doesn't print CBO plan.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-21328) Call To Hadoop Text getBytes() Without Call to getLength()
[
https://issues.apache.org/jira/browse/HIVE-21328?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Mollitor updated HIVE-21328:
--
Attachment: HIVE-21328.1.patch
> Call To Hadoop Text getBytes() Without Call to getLength()
> --
>
> Key: HIVE-21328
> URL: https://issues.apache.org/jira/browse/HIVE-21328
> Project: Hive
> Issue Type: Bug
> Components: Query Planning
>Affects Versions: 4.0.0, 3.2.0
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Major
> Attachments: HIVE-21328.1.patch
>
>
> I'm not sure if there is actually a bug, but this looks highly suspect:
> {code:java}
> public Object set(final Object o, final Text text) {
> return new BytesWritable(text == null ? null : text.getBytes());
> }
> {code}
> https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/io/parquet/serde/primitive/ParquetStringInspector.java#L104-L106
> There are two components to a Text object. There are the internal bytes and
> the length of the bytes. The two are independent. I.e., a quick "reset" on
> the Text object simply sets the internal length counter to zero. This code
> is potentially looking at obsolete data that it shouldn't be seeing because
> it is not considering the length of the Text.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
