Re: Result of the TPC-DS benchmark on Hive master branch
Hi Sungwoo,
There is https://issues.apache.org/jira/browse/HIVE-23975 causing a
regression in runtime. There is a ticket open to fix it (
https://issues.apache.org/jira/browse/HIVE-24139) which is still in
progress. You might want to revert 23975 before trying.
On Wed, Nov 4, 2020 at 2:55 PM Stamatis Zampetakis
wrote:
> Hi Sungwoo,
>
> Personally, I would be also interested to see the results of these
> experiments if they are available somewhere.
>
> I didn't understand if the queries are failing at runtime or compile time.
> Are the above errors the only ones that you're getting?
>
> If you can reproduce the problem with a smaller dataset then I think the
> best would be to create unit tests and JIRAS for each query separately.
>
> It may not be worth going through the commits to find those that caused the
> regression because it will be time-consuming and you may bump into
> something that is not trivial to revert.
>
> Best,
> Stamatis
>
>
> On Wed, Nov 4, 2020 at 7:24 PM Sungwoo Park wrote:
>
> > Hello,
> >
> > I have tested a recent commit of the master branch using the TPC-DS
> > benchmark. I used Hive on Tez (not Hive-LLAP). The way I tested is:
> >
> > 1) create a database consisting of external tables from a 100GB TPC-DS
> text
> > dataset
> > 2) create a database consisting of ORC tables from the previous database
> > 3) compute column statistics
> > 4) run TPC-DS queries and check the results
> >
> > Previously we tested the commit 5f47808c02816edcd4c323dfa25194536f3f20fd
> > (HIVE-23114: Insert overwrite with dynamic partitioning is not working
> > correctly with direct insert, Fri Apr 10), and all queries ran okay.
> >
> > This time I used the following commits. I made a few changes to pom.xml
> of
> > both Hive and Tez, but these changes should not affect the result of
> > running queries.
> >
> > 1) Hive, master, 96aacdc50043fa442c2277b7629812e69241a507 (Tue Nov
> > 3), HIVE-24314: compactor.Cleaner should not set state mark cleaned if it
> > didn't remove any files
> > 2) Tez, 0.10.0, 22fec6c0ecc7ebe6f6f28800935cc6f69794dad5 (Thu Oct
> > 8), CHANGES.txt updated with TEZ-4238
> >
> > The result is that 14 queries (out of 99 queries) fail, and a query fails
> > during compilation for one of the following two reasons.
> >
> > 1)
> > org.apache.hive.service.cli.HiveSQLException: Error while compiling
> > statement: FAILED: Execution Error, return code 1 from
> > org.apache.hadoop.hive.ql.exec.tez.TezTask. Edge [Map 12 :
> > org.apache.hadoop.hive.ql.exec.tez.MapTezProcessor] -> [Map 7 :
> > org.apache.hadoop.hive.ql.exec.tez.MapTezProcessor] ({ BROADCAST :
> > org.apache.tez.runtime.library.input.UnorderedKVInput >> PERSISTED >>
> > org.apache.tez.runtime.library.output.UnorderedKVOutput >>
> NullEdgeManager
> > }) already defined!
> > at
> >
> >
> org.apache.hive.service.cli.operation.Operation.toSQLException(Operation.java:365)
> > at
> >
> >
> org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:241)
> > at
> >
> >
> org.apache.hive.service.cli.operation.SQLOperation.access$500(SQLOperation.java:88)
> > at
> >
> >
> org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork$1.run(SQLOperation.java:325)
> > at java.security.AccessController.doPrivileged(Native Method)
> > at javax.security.auth.Subject.doAs(Subject.java:422)
> > at
> >
> >
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1682)
> > at
> >
> >
> org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork.run(SQLOperation.java:343)
> > at
> > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> > at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> > at
> >
> >
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> > at
> >
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> > at java.lang.Thread.run(Thread.java:745)
> > Caused by: java.lang.IllegalArgumentException: Edge [Map 12 :
> > org.apache.hadoop.hive.ql.exec.tez.MapTezProcessor] -> [Map 7 :
> > org.apache.hadoop.hive.ql.exec.tez.MapTezProcessor] ({ BROADCAST :
> > org.apache.tez.runtime.library.input.UnorderedKVInput >> PERSISTED >>
> > org.apache.tez.runtime.library.output.UnorderedKVOutput >>
> NullEdgeManager
> > }) already defined!
> > at org.apache.tez.dag.api.DAG.addEdge(DAG.java:297)
> > at org.apache.hadoop.hive.ql.exec.tez.TezTask.build(TezTask.java:519)
> > at org.apache.hadoop.hive.ql.exec.tez.TezTask.execute(TezTask.java:213)
> > at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:213)
> > at
> >
> >
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:105)
> > at org.apache.hadoop.hive.ql.Executor.launchTask(Executor.java:361)
> > at org.apache.hadoop.hive.ql.Executor.launchTasks(Executor.java:334)
> > at org.apache.hadoop.hive.ql.Executor.runTasks(Executor.java:245)
> > at org.apache.hadoop.hive.ql.Executor.execute(Exe
[jira] [Created] (HIVE-24350) NullScanTaskDispatcher should use stats
Mustafa Iman created HIVE-24350: --- Summary: NullScanTaskDispatcher should use stats Key: HIVE-24350 URL: https://issues.apache.org/jira/browse/HIVE-24350 Project: Hive Issue Type: Improvement Reporter: Mustafa Iman Assignee: Mustafa Iman NullScanTaskDispatcher manually checks each partition directory to see if they are empty. While this is necessary in external tables, we can just use stats for managed tables. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HIVE-24270) Move scratchdir cleanup to background
Mustafa Iman created HIVE-24270: --- Summary: Move scratchdir cleanup to background Key: HIVE-24270 URL: https://issues.apache.org/jira/browse/HIVE-24270 Project: Hive Issue Type: Improvement Reporter: Mustafa Iman Assignee: Mustafa Iman In cloud environment, scratchdir cleaning at the end of the query may take long time. This causes client to hang up to 1 minute even after the results were streamed back. During this time client just waits for cleanup to finish. Cleanup can take place in the background in HiveServer. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HIVE-24139) VectorGroupByOperator is not flushing hash table entries as needed
Mustafa Iman created HIVE-24139: --- Summary: VectorGroupByOperator is not flushing hash table entries as needed Key: HIVE-24139 URL: https://issues.apache.org/jira/browse/HIVE-24139 Project: Hive Issue Type: Bug Reporter: Mustafa Iman Assignee: Mustafa Iman After https://issues.apache.org/jira/browse/HIVE-23975 introduced a bug where copyKey mutates some key wrappers while copying. This Jira is to fix it. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HIVE-24093) Remove unused hive.debug.localtask
Mustafa Iman created HIVE-24093: --- Summary: Remove unused hive.debug.localtask Key: HIVE-24093 URL: https://issues.apache.org/jira/browse/HIVE-24093 Project: Hive Issue Type: Improvement Reporter: Mustafa Iman Assignee: Mustafa Iman hive.debug.local.task was added in HIVE-1642. Even then, it was never used. It was possibly a leftover from development/debugging. There are no references to either HIVEDEBUGLOCALTASK or hive.debug.localtask in the codebase. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HIVE-23975) Reuse evicted keys from aggregation buffers
Mustafa Iman created HIVE-23975: --- Summary: Reuse evicted keys from aggregation buffers Key: HIVE-23975 URL: https://issues.apache.org/jira/browse/HIVE-23975 Project: Hive Issue Type: Improvement Reporter: Mustafa Iman Assignee: Mustafa Iman -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HIVE-23830) Remove shutdownhook after query is completed
Mustafa Iman created HIVE-23830: --- Summary: Remove shutdownhook after query is completed Key: HIVE-23830 URL: https://issues.apache.org/jira/browse/HIVE-23830 Project: Hive Issue Type: Bug Reporter: Mustafa Iman Assignee: Mustafa Iman Each query registers a shutdownHook to release transactional resources in case JVM shuts down mid query. These hooks are not cleaned up until session is closed. Session life time is unbounded. So these hooks are a memory leak. They should be cleaned as soon as transaction is completed. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HIVE-23780) Fail dropTable if acid cleanup fails
Mustafa Iman created HIVE-23780: --- Summary: Fail dropTable if acid cleanup fails Key: HIVE-23780 URL: https://issues.apache.org/jira/browse/HIVE-23780 Project: Hive Issue Type: Bug Components: Metastore, Standalone Metastore, Transactions Reporter: Mustafa Iman Assignee: Mustafa Iman Acid cleanup happens after dropTable is committed. If cleanup fails for some reason, there are leftover entries in acid tables. This later causes dropped table's name to be unusable by new tables. [~pvary] [~ngangam] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HIVE-23747) Increase the number of parallel tasks sent to daemons from am
Mustafa Iman created HIVE-23747: --- Summary: Increase the number of parallel tasks sent to daemons from am Key: HIVE-23747 URL: https://issues.apache.org/jira/browse/HIVE-23747 Project: Hive Issue Type: Sub-task Reporter: Mustafa Iman Assignee: Mustafa Iman The number of inflight tasks from AM to a single executor is hardcoded to 1 currently([https://github.com/apache/hive/blob/master/llap-client/src/java/org/apache/hadoop/hive/llap/tez/LlapProtocolClientProxy.java#L57] ). It does not make sense to increase this right now as communication between am and daemons happen synchronously anyway. After resolving https://issues.apache.org/jira/browse/HIVE-23746 this must be increased to at least number of execution slots per daemon. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HIVE-23746) Send task attempts async from AM to daemons
Mustafa Iman created HIVE-23746: --- Summary: Send task attempts async from AM to daemons Key: HIVE-23746 URL: https://issues.apache.org/jira/browse/HIVE-23746 Project: Hive Issue Type: Sub-task Components: llap Reporter: Mustafa Iman Assignee: Mustafa Iman LlapTaskCommunicator uses sync client to send task attempts. There are fixed number of communication threads (10 by default). This causes unneccessary delays when there are enough free execution slots in daemons but they do not receive all the tasks because of this bottleneck. LlapTaskCommunicator can use an async client to pass these tasks to daemons. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HIVE-23745) Avoid copying userpayload in task communicator
Mustafa Iman created HIVE-23745: --- Summary: Avoid copying userpayload in task communicator Key: HIVE-23745 URL: https://issues.apache.org/jira/browse/HIVE-23745 Project: Hive Issue Type: Sub-task Reporter: Mustafa Iman Assignee: Mustafa Iman [https://github.com/apache/hive/blob/master/llap-common/src/java/org/apache/hadoop/hive/llap/tez/Converters.java#L182] I see this copy take a few milliseconds sometimes. Delay here adds up for all tasks of a single vertex in LlapTaskCommunicator as it processes tasks one by one. User payload never changes in this codepath. Copy is made because of limitations of Protobuf library. Protobuf adds a UnsafeByteOperations class that avoid copying of ByteBuffers in 3.1 version. This can be resolved when Protobuf is upgraded. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HIVE-23744) Reduce query startup latency
Mustafa Iman created HIVE-23744: --- Summary: Reduce query startup latency Key: HIVE-23744 URL: https://issues.apache.org/jira/browse/HIVE-23744 Project: Hive Issue Type: Task Components: llap Affects Versions: 4.0.0 Reporter: Mustafa Iman Assignee: Mustafa Iman Attachments: am_schedule_and_transmit.png, task_start.png When I run queries with large number of tasks for a single vertex, I see a significant delay before all tasks start execution in llap daemons. Although llap daemons have the free capacity to run the tasks, it takes a significant time to schedule all the tasks in AM and actually transmit them to executors. "am_schedule_and_transmit" shows scheduling of tasks of tpcds query 55. It shows only the tasks scheduled for one of 10 llap daemons. The scheduler works in a single thread, scheduling tasks one by one. A delay in scheduling of one task, delays all the tasks. !am_schedule_and_transmit.png|width=831,height=573! Another issue is that it takes long time to fill all the execution slots in llap daemons even though they are all empty initially. This is caused by LlapTaskCommunicator using a fixed number of threads (10 by default) to send the tasks to daemons. Also this communication is synchronized so these threads block communication staying idle. "task_start.png" shows running tasks on an llap daemon that has 12 execution slots. By the time 12th task starts running, more than 100ms already passes. That slot stays idle all this time. !task_start.png|width=1166,height=635! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HIVE-23687) Fix Spotbugs issues in hive-standalone-metastore
Mustafa Iman created HIVE-23687: --- Summary: Fix Spotbugs issues in hive-standalone-metastore Key: HIVE-23687 URL: https://issues.apache.org/jira/browse/HIVE-23687 Project: Hive Issue Type: Sub-task Reporter: Mustafa Iman Assignee: Mustafa Iman -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HIVE-23686) Fix Spotbugs issues in hive-shims
Mustafa Iman created HIVE-23686: --- Summary: Fix Spotbugs issues in hive-shims Key: HIVE-23686 URL: https://issues.apache.org/jira/browse/HIVE-23686 Project: Hive Issue Type: Sub-task Reporter: Mustafa Iman Assignee: Mustafa Iman -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HIVE-23629) Enforce clean findbugs in PRs
Mustafa Iman created HIVE-23629: --- Summary: Enforce clean findbugs in PRs Key: HIVE-23629 URL: https://issues.apache.org/jira/browse/HIVE-23629 Project: Hive Issue Type: Sub-task Reporter: Mustafa Iman Assignee: Mustafa Iman We should start enforcing clean findbugs reports as soon as we fix a module. Otherwise, it will continue collecting findbugs errors. We can add a stage to Jenkins pipeline to enforce findbugs and possibly other checks. It will selectively run findbugs for specified sub modules. Eventually we can get rid of the list and enable findbugs for the whole project. -- This message was sent by Atlassian Jira (v8.3.4#803005)
Re: [DISCUSS] Replace ptest with hive-test-kube
Thank you Zoltan for all this work. I see many PRs are merged based on the new workflow already. The old workflow generates many reports like ASF license/findbugs/checkstyle etc. I don't see these in the new Github PR workflow. I am concerned the codebase is going to suffer from lack of these reports very quickly. Do these checks still happen but are not visible? On Tue, Jun 2, 2020 at 4:41 AM Zoltan Haindrich wrote: > Hello, > > I would like to note that you may login to the jenkins instance - to > start/kill builds (or create new jobs). > I've configured github oauth - but since team membership can't be queried > from the "apache organization" - it's harder to configure all "hive > committers". > However...I think I've made it available for most of us - if you can't > start builds/etc just let me know your github user and I'll add it. > > cheers, > Zoltan > > > > On 5/29/20 2:32 PM, Zoltan Haindrich wrote: > > Hey all! > > > > The patch is now in master - so every new PR or a push on it will > trigger a new run. > > > > Please decide which one would you like to use - open a PR to see the new > one work...or upload a patch file to the jira - but please don't do both; > because in that case 2 > > execution will happen. > > > > The job execution time(2-4 hours) of a single run is a bit higher than > the usual on the ptest server - this is mostly to increase throughput. > > > > The patch also disabled a set of tests; I will send the full list of > skipped tests shortly. > > > > cheers, > > Zoltan > > > > > > On 5/27/20 1:50 PM, Zoltan Haindrich wrote: > >> Hello all! > >> > >> The new stuff is ready to be switched on-to. It needs to be merged into > master - and after that anyone who opens a PR will get a run by the new > HiveQA infra. > >> I propose to run the 2 systems side-by-side for some time - the regular > master builds will start; and we will see how frequently that is polluted > by flaky tests. > >> > >> Note that the current patch also disables around ~25 more tests to > increase stability - to get a better overview about the disabled tests I > think the "direction of the > >> information flow" should be altered; what I mean by that is: instead of > just throwing in a jira for "disable test x" and opening a new one like > "fix test x"; only open > >> the latter and place the jira reference into the ignore message; > meanwhile also add a regular report about the actually disabled tests - so > people who do know about the > >> importance of a particular test can get involved. > >> > >> Note: the builds.apache.org instance will be shutdown somewhere in the > future as well...but I think the new one is a good-enough alternative to > not have to migrate the > >> Hive-precommit job over to https://ci-hadoop.apache.org/. > >> > >> http://34.66.156.144:8080/job/hive-precommit/job/PR-948/5/ > >> https://issues.apache.org/jira/browse/HIVE-22942 > >> https://github.com/apache/hive/pull/948/files > >> > >> cheers, > >> Zoltan > >> > >> On 5/18/20 1:42 PM, Zoltan Haindrich wrote: > >>> Hey! > >>> > >>> On 5/18/20 11:51 AM, Zoltan Chovan wrote: > Thank you for all of your efforts, this looks really promising. With > moving > to github PRs, would that also mean that we move away from the > reviewboard > for code review? > >>> I didn't thinked about that. I think using github's review interface > will remain optional, because both review systems has there own strong > points - I wouldn't force > >>> anyone to use one over the other. (For some patches reviewboard is > much better; because it's able to track content moves a bit better than > github. - meanwhile github has > >>> a small feature that enables to mark files as reviewed) > >>> As a matter of fact we had sometimes patches on the jira's which never > had neither an RB or a PR to review them - having a PR there at least will > make it easier for > >>> reviewers to comment. > >>> > Also, what happens if a PR is updated? Will the tests run for both or > just > for the latest version? > >>> It will trigger a new build - if there is already a build in progress > that will prevent a new build from starting until it finishes...and there > is also a 5 builds/day > >>> limit; which might induce some wait. > >>> > >>> cheers, > >>> Zoltan > >>> > > Regards, > Zoltan > > On Sun, May 17, 2020 at 10:51 PM Zoltan Haindrich > wrote: > > > Hello all! > > > > The proposed system have become more stable lately - and I think I've > > solved a few sources of flakiness. > > To be really usable I also wanted to add a way to dynamically > > enable/disable a set of tests (for example the replication tests > take ~7 > > hours to execute from the total of 24 > > hours - and they are also a bit unstable, so not running them when > not > > neccesary would be beneficial in multiple ways) - but to do this the > best > > would be to throw in > > junit5; unfortunately the current ptest
[jira] [Created] (HIVE-23596) Encode guaranteed task information in containerId
Mustafa Iman created HIVE-23596: --- Summary: Encode guaranteed task information in containerId Key: HIVE-23596 URL: https://issues.apache.org/jira/browse/HIVE-23596 Project: Hive Issue Type: Improvement Reporter: Mustafa Iman Assignee: Mustafa Iman We should avoid calling LlapTaskScheduler to get initial isguaranteed flag for all the tasks. It causes arbitrary delays in sending tasks out. Since communicator is a single thread, any blocking there delays all the tasks. There are [https://jira.apache.org/jira/browse/TEZ-4192] and [https://jira.apache.org/jira/browse/HIVE-23589] for a proper solution to this. However, that requires a Tez release which seems far right now. We can replace the current hack with another hack that does not require locking. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HIVE-23595) Do not query task guaranteed status when wlm off
Mustafa Iman created HIVE-23595: --- Summary: Do not query task guaranteed status when wlm off Key: HIVE-23595 URL: https://issues.apache.org/jira/browse/HIVE-23595 Project: Hive Issue Type: Improvement Reporter: Mustafa Iman Assignee: Mustafa Iman LlapTaskCommunicator queries scheduler for every task guaranteed status. When workload management is off it is always false. There is no need for the synchronous check. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HIVE-23589) Eliminate blocking call to scheduler for isguaranteed
Mustafa Iman created HIVE-23589: --- Summary: Eliminate blocking call to scheduler for isguaranteed Key: HIVE-23589 URL: https://issues.apache.org/jira/browse/HIVE-23589 Project: Hive Issue Type: Improvement Reporter: Mustafa Iman Assignee: Mustafa Iman LlapTaskCommunicator requires isGuaranteed information from LlapTaskScheduler just before sending the actual request to the executor. There is no particular ordering between the writing the submit request to the network and scheduler decisions. Therefore, we can just put the initial isGuaranteed information in task allocation rather than calling back scheduler. Depends on https://jira.apache.org/jira/browse/TEZ-4192 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HIVE-23584) Dont send default configs(HiveConf) to AM and tasks
Mustafa Iman created HIVE-23584: --- Summary: Dont send default configs(HiveConf) to AM and tasks Key: HIVE-23584 URL: https://issues.apache.org/jira/browse/HIVE-23584 Project: Hive Issue Type: Improvement Reporter: Mustafa Iman Attachments: hiveconf.wip.patch About 80% of the configs left after HIVE-23175 are default settings coming from HiveConf object. We can remove these from payload also. Only problem is that TezTask relies on some of these configs when building dag. We can explicitly add those settings that are needed in dag build phase in HiveServer2. The rest is ok to be removed from payload. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HIVE-23529) CTAS is broken for uniontype when row_deserialize
Mustafa Iman created HIVE-23529: --- Summary: CTAS is broken for uniontype when row_deserialize Key: HIVE-23529 URL: https://issues.apache.org/jira/browse/HIVE-23529 Project: Hive Issue Type: Bug Reporter: Mustafa Iman Assignee: Mustafa Iman CTAS queries fail when there is a uniontype in source table and hive.vectorized.use.vector.serde.deserialize=false. ObjectInspectorUtils.copyToStandardObject in ROW_DESERIALIZE path extracts the value from union type. However, VectorAssignRow expects a StandardUnion object causing ClassCastException for any CTAS query. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HIVE-23455) Improve error message for external orc table
Mustafa Iman created HIVE-23455: --- Summary: Improve error message for external orc table Key: HIVE-23455 URL: https://issues.apache.org/jira/browse/HIVE-23455 Project: Hive Issue Type: Improvement Reporter: Mustafa Iman Assignee: Mustafa Iman Since there is no schema validation for external tables, users may face various errors if their orc data and external table schema does not match. If orc schema has fewer columns than projection OrcEncodedDataConsumer may receive an incomplete TypeDescription array which will manifest itself as NullPointerException later. We can at least verify that OrcEncodedDataConsumer gets enough TypeDescriptions. If assertion fails, user sees there is something wrong with the schema and hopefully resolves the problem quickly. If there are enough columns in the file but the schema of the query does not match, user generally sees a ClassCastException. If there are enough columns and types accidentally match, there is nothing we can do as this is an external table. We have seen this when trying to use a managed table as external table location. Although user facing schemas are the same, managed table has acid related metadata. I am adding a q file demonstrating NullPointerException with TestMiniLlapLocalCliDriver and the output after the fix. I haven't added this to precommit tests as it is hard to assert the exception message from mini driver framework and effectively it is just changing the error. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HIVE-23448) Remove hive-site.xml from input/output/processor payload
Mustafa Iman created HIVE-23448: --- Summary: Remove hive-site.xml from input/output/processor payload Key: HIVE-23448 URL: https://issues.apache.org/jira/browse/HIVE-23448 Project: Hive Issue Type: Improvement Components: Tez Reporter: Mustafa Iman Assignee: Mustafa Iman Depends on https://jira.apache.org/jira/browse/TEZ-4137?filter=-1 We remove most xml configs from payloads in https://jira.apache.org/jira/browse/HIVE-23175 However, hive-site.xml could not be removed from those configs in early stage for reasons outlined in that jira. This Jira removes hive-site.xml configs from configuration just before serializing payloads. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HIVE-23191) Prevent redundant output descriptor config serialization
Mustafa Iman created HIVE-23191:
---
Summary: Prevent redundant output descriptor config serialization
Key: HIVE-23191
URL: https://issues.apache.org/jira/browse/HIVE-23191
Project: Hive
Issue Type: Improvement
Reporter: Mustafa Iman
Assignee: Mustafa Iman
{code:java}
DagUtils#createVertex(JobConf, BaseWork, Path,
TezWork, Map){code}
creates an output descriptor if it is leaf vertex. It uses the same config
object that is used in processor descriptor. It should not create payload from
scratch when processor descriptor has the identical payload.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (HIVE-23180) Remove unused variables from tez build dag
Mustafa Iman created HIVE-23180: --- Summary: Remove unused variables from tez build dag Key: HIVE-23180 URL: https://issues.apache.org/jira/browse/HIVE-23180 Project: Hive Issue Type: Improvement Reporter: Mustafa Iman Assignee: Mustafa Iman Attachments: HIVE-23180.patch This is a simple refactoring around TezTask build dag functionality. Unused options are removed from function calls. Also some variables are given meaningful names. Gets rid of unneccessary filesystem creation. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HIVE-23175) Skip serializing hadoop and tez config on HS side
Mustafa Iman created HIVE-23175: --- Summary: Skip serializing hadoop and tez config on HS side Key: HIVE-23175 URL: https://issues.apache.org/jira/browse/HIVE-23175 Project: Hive Issue Type: Improvement Reporter: Mustafa Iman Assignee: Mustafa Iman HiveServer spends a lot of time serializing configuration objects. We can skip putting hadoop and tez config xml files in payload assuming that the configs are the same on both HS and AM side. This depends on Tez to load local xml configs when creating config objects https://issues.apache.org/jira/browse/TEZ-4141 -- This message was sent by Atlassian Jira (v8.3.4#803005)
Re: Review Request 72113: DML execution on TEZ always outputs the message 'No rows affected'
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/72113/#review219553 --- +1 - Mustafa Iman On Feb. 11, 2020, 8:01 a.m., Attila Magyar wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/72113/ > --- > > (Updated Feb. 11, 2020, 8:01 a.m.) > > > Review request for hive, Laszlo Bodor, Mustafa Iman, Panos Garefalakis, and > Ramesh Kumar Thangarajan. > > > Bugs: HIVE-22870 > https://issues.apache.org/jira/browse/HIVE-22870 > > > Repository: hive-git > > > Description > --- > > Executing an update or insert statement in beeline doesn't show the actual > rows inserted/updated. > > > Diffs > - > > ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezTask.java 25dd970a9b1 > > > Diff: https://reviews.apache.org/r/72113/diff/1/ > > > Testing > --- > > with insert and updates > > > Thanks, > > Attila Magyar > >
[jira] [Created] (HIVE-22877) Wrong decimal boundary for casting to Decimal64
Mustafa Iman created HIVE-22877:
---
Summary: Wrong decimal boundary for casting to Decimal64
Key: HIVE-22877
URL: https://issues.apache.org/jira/browse/HIVE-22877
Project: Hive
Issue Type: Bug
Components: Vectorization
Affects Versions: 4.0.0
Reporter: Mustafa Iman
Assignee: Mustafa Iman
During vectorization, decimal fields that are obtained via generic udfs are
cast to Decimal64 in some circumstances. For decimal to decimal64 cast, hive
compares the source column's `scale + precision` to 18(maximum number of digits
that can be represented by a long). A decimal can fit in a long as long as its
`scale` is smaller than or equal to 18. Precision is irrelevant.
Since vectorized generic udf expression takes precision into account, it
computes wrong output column vector: Decimal instead of Decimal64. This in turn
causes ClassCastException down the operator chain.
Below query fails with class cast exception:
{code:java}
create table mini_store
(
s_store_sk int,
s_store_id string
)
row format delimited fields terminated by '\t'
STORED AS ORC;
create table mini_sales
(
ss_store_sk int,
ss_quantity int,
ss_sales_price decimal(7,2)
)
row format delimited fields terminated by '\t'
STORED AS ORC;
insert into mini_store values (1, 'store');
insert into mini_sales values (1, 2, 1.2);
select s_store_id, coalesce(ss_sales_price*ss_quantity,0) sumsales
from mini_sales, mini_store where ss_store_sk = s_store_sk
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (HIVE-22555) Upgrade ORC version to 1.5.8
Mustafa Iman created HIVE-22555: --- Summary: Upgrade ORC version to 1.5.8 Key: HIVE-22555 URL: https://issues.apache.org/jira/browse/HIVE-22555 Project: Hive Issue Type: Bug Affects Versions: 4.0.0 Reporter: Mustafa Iman Assignee: Mustafa Iman Hive currently depends on ORC 1.5.6. We need 1.5.8 upgrade for https://issues.apache.org/jira/browse/HIVE-22499 ORC-1.5.7 includes https://issues.apache.org/jira/browse/ORC-361 . It causes some tests overriding MemoryManager to fail. These need to be addressed while upgrading. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HIVE-22537) getAcidState() not saving directory snapshot causes multiple calls to S3 api
Mustafa Iman created HIVE-22537: --- Summary: getAcidState() not saving directory snapshot causes multiple calls to S3 api Key: HIVE-22537 URL: https://issues.apache.org/jira/browse/HIVE-22537 Project: Hive Issue Type: Bug Affects Versions: 4.0.0 Reporter: Mustafa Iman Assignee: Mustafa Iman Fix for HIVE-21225 is not enabled in query coordinator codepath. The last argument (generateDirSnapshots) for getAcidState() is set to false when invoked by callInternal(). Also, snapshot is not used for file exists calls. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HIVE-22466) Prevent synchronized calls in LlapDaemon configuration
Mustafa Iman created HIVE-22466: --- Summary: Prevent synchronized calls in LlapDaemon configuration Key: HIVE-22466 URL: https://issues.apache.org/jira/browse/HIVE-22466 Project: Hive Issue Type: Bug Components: llap Affects Versions: 4.0.0 Reporter: Mustafa Iman Assignee: Mustafa Iman LlapDaemonConfig extends from Hadoop's Configuration class. Configuration class makes use of synchronized calls for both reads and writes to configuration. LlapDaemon does not change configuration in runtime. We can remove synchronized access to configuration in LlapConfiguration -- This message was sent by Atlassian Jira (v8.3.4#803005)
