Re: 【Hive Alter Table Add column at specified position】
Yeah, according to our DDL doc, we don't support this use case at the moment. Perhaps you can use REPLACE COLUMNS as a workaround. On Sat, Jun 27, 2020 at 5:32 PM 忝忝向仧 <153488...@qq.com> wrote: > Hi,all: > > > It seems that Hive can not alter table to add column atspecified > position. > For instance,the Table A has c1,c2,c3 columns,and i want to add column c4 > after c1,therefore,the table would be like c1,c4,c2,c3 instead of > c1,c2,c3,c4. > > > Thanks. -- Best regards! Rui Li
Hit HIVE-13023 with 2.0.1 maven artifacts
Hello guys, I hit HIVE-13023 <https://issues.apache.org/jira/browse/HIVE-13023> when I programmatically executed some queries with Hive-2.0.1. I did some investigation and there seemed to be some issues with the 2.0.1 artifacts we published. I compared the hive-exec artifact from maven central <https://repo1.maven.org/maven2/org/apache/hive/hive-exec/2.0.1/hive-exec-2.0.1.jar> with the jar in our binary distribution <https://archive.apache.org/dist/hive/hive-2.0.1/apache-hive-2.0.1-bin.tar.gz>, and the two hive-exec jars are of different sizes. I also decompiled these two jars to check the offending method *StorageFormat::fillStorageFormat*. It turned out the jar from maven got the ordinals of some tokens wrong (the ordinal of TOK_FILEFORMAT_GENERIC should be 715): [image: 屏幕快照 2019-09-04 下午5.08.59.png] And the jar in our distribution has the correct ordinals: [image: 屏幕快照 2019-09-04 下午5.14.00.png] I wonder whether anybody could help verify the issue, and whether it's possible to update the published jars if the issue is valid? -- Best regards! Rui Li
[jira] [Created] (HIVE-22053) Function name is not normalized when creating function
Rui Li created HIVE-22053: - Summary: Function name is not normalized when creating function Key: HIVE-22053 URL: https://issues.apache.org/jira/browse/HIVE-22053 Project: Hive Issue Type: Bug Components: Standalone Metastore Reporter: Rui Li Assignee: Rui Li If a function is created with a name containing upper case characters, we get NoSuchObjectException when trying to get that function. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
Re: Is Github PR mandatory?
Got it. Thanks! On Fri, Mar 15, 2019 at 1:47 PM Mani M wrote: > It's used as substitution for review board. > > > With Regards > M.Mani > +61 432 461 087 > > On Fri, 15 Mar 2019, 13:59 Rui Li, wrote: > > > Hi, > > > > I believe we still need to upload patch to JIRA for precommit testing. So > > just want to make sure whether opening a github PR is mandatory? Or is it > > just a substitution for the review board? > > > > -- > > Best regards! > > Rui Li > > > -- Best regards! Rui Li
Is Github PR mandatory?
Hi, I believe we still need to upload patch to JIRA for precommit testing. So just want to make sure whether opening a github PR is mandatory? Or is it just a substitution for the review board? -- Best regards! Rui Li
[jira] [Created] (HIVE-19895) The unique ID in SparkPartitionPruningSinkOperator is no longer needed
Rui Li created HIVE-19895: - Summary: The unique ID in SparkPartitionPruningSinkOperator is no longer needed Key: HIVE-19895 URL: https://issues.apache.org/jira/browse/HIVE-19895 Project: Hive Issue Type: Bug Components: Spark Reporter: Rui Li -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (HIVE-19671) Distribute by rand() can lead to data inconsistency
Rui Li created HIVE-19671:
-
Summary: Distribute by rand() can lead to data inconsistency
Key: HIVE-19671
URL: https://issues.apache.org/jira/browse/HIVE-19671
Project: Hive
Issue Type: Bug
Reporter: Rui Li
Noticed the following queries can give different results:
{code}
select count(*) from tbl;
select count(*) from (select * from tbl distribute by rand());
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
Re: [VOTE] Stricter commit guidelines
+1 On Tue, May 15, 2018 at 2:24 PM, Prasanth Jayachandran < pjayachand...@hortonworks.com> wrote: > +1 > > > > Thanks > Prasanth > > > > On Mon, May 14, 2018 at 10:44 PM -0700, "Jesus Camacho Rodriguez" < > jcama...@apache.org<mailto:jcama...@apache.org>> wrote: > > > After work has been done to ignore most of the tests that were failing > consistently/intermittently [1], I wanted to start this vote to gather > support from the community to be stricter wrt committing patches to Hive. > The committers guide [2] already specifies that a +1 should be obtained > before committing, but there is another clause that allows committing under > the presence of flaky tests (clause 4). Flaky tests are as good as having > no tests, hence I propose to remove clause 4 and enforce the +1 from > testing infra before committing. > > > > As I see it, by enforcing that we always get a +1 from the testing infra > before committing, 1) we will have a more stable project, and 2) we will > have another incentive as a community to create a more robust testing > infra, e.g., replacing flaky tests for similar unit tests that are not > flaky, trying to decrease running time for tests, etc. > > > > Please, share your thoughts about this. > > > > Here is my +1. > > > > Thanks, > > Jes?s > > > > [1] http://mail-archives.apache.org/mod_mbox/hive-dev/201805. > mbox/%3C63023673-AEE5-41A9-BA52-5A5DFB2078B6%40apache.org%3E > > [2] https://cwiki.apache.org/confluence/display/Hive/ > HowToCommit#HowToCommit-PreCommitruns,andcommittingpatches > > > > > -- Best regards! Rui Li
Re: [DISCUSS] Unsustainable situation with ptests
cally. May be create a profile which runs a subset of > > important tests which are consistent. We can apply some label that > > pre-checkin-local tests are runs successful and only then we submit > > for the > > full-suite. > > > > More thoughts are welcome. Thanks for starting this conversation. > > > > On Fri, May 11, 2018 at 1:27 PM, Jesus Camacho Rodriguez < > > jcama...@apache.org<mailto:jcama...@apache.org>> wrote: > > > > I believe we have reached a state (maybe we did reach it a while ago) > > that > > is not sustainable anymore, as there are so many tests failing / > > timing out > > that it is not possible to verify whether a patch is breaking some > > critical > > parts of the system or not. It also seems to me that due to the > > timeouts > > (maybe due to infra, maybe not), ptest runs are taking even longer > than > > usual, which in turn creates even longer queue of patches. > > > > There is an ongoing effort to improve ptests usability ( > > https://issues.apache.org/jira/browse/HIVE-19425), but apart from > > that, > > we need to make an effort to stabilize existing tests and bring that > > failure count to zero. > > > > Hence, I am suggesting *we stop committing any patch before we get a > > green > > run*. If someone thinks this proposal is too radical, please come up > > with > > an alternative, because I do not think it is OK to have the ptest > runs > > in > > their current state. Other projects of certain size (e.g., Hadoop, > > Spark) > > are always green, we should be able to do the same. > > > > Finally, once we get to zero failures, I suggest we are less tolerant > > with > > committing without getting a clean ptests run. If there is a failure, > > we > > need to fix it or revert the patch that caused it, then we continue > > developing. > > > > Please, let’s all work together as a community to fix this issue, > that > > is > > the only way to get to zero quickly. > > > > Thanks, > > Jesús > > > > PS. I assume the flaky tests will come into the discussion. Let´s see > > first how many of those we have, then we can work to find a fix. > > > > > > > > > > > > > > > > > -- Best regards! Rui Li
[jira] [Created] (HIVE-19439) MapWork shouldn't be reused when Spark task fails during initialization
Rui Li created HIVE-19439: - Summary: MapWork shouldn't be reused when Spark task fails during initialization Key: HIVE-19439 URL: https://issues.apache.org/jira/browse/HIVE-19439 Project: Hive Issue Type: Bug Components: Spark Reporter: Rui Li Issue identified in HIVE-19388. When a Spark task fails during initializing the map operator, the task is retried with the same MapWork retrieved from cache. This can be problematic because the MapWork may be partially initialized, e.g. some operators are already in INIT state. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (HIVE-19316) StatsTask fails due to ClassCastException
Rui Li created HIVE-19316:
-
Summary: StatsTask fails due to ClassCastException
Key: HIVE-19316
URL: https://issues.apache.org/jira/browse/HIVE-19316
Project: Hive
Issue Type: Bug
Components: Statistics
Reporter: Rui Li
The stack trace:
{noformat}
2018-04-26T20:17:37,674 ERROR [pool-7-thread-11] metastore.RetryingHMSHandler:
java.lang.ClassCastException:
org.apache.hadoop.hive.metastore.api.LongColumnStatsData cannot be cast to
org.apache.hadoop.hive.metastore.columnstats.cache.LongColumnStatsDataInspector
at
org.apache.hadoop.hive.metastore.columnstats.merge.LongColumnStatsMerger.merge(LongColumnStatsMerger.java:30)
at
org.apache.hadoop.hive.metastore.utils.MetaStoreUtils.mergeColStats(MetaStoreUtils.java:1052)
at
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.set_aggr_stats_for(HiveMetaStore.java:7202)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
at
org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108)
at com.sun.proxy.$Proxy26.set_aggr_stats_for(Unknown Source)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$set_aggr_stats_for.getResult(ThriftHiveMetastore.java:16795)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$set_aggr_stats_for.getResult(ThriftHiveMetastore.java:16779)
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
at
org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:111)
at
org.apache.hadoop.hive.metastore.TUGIBasedProcessor$1.run(TUGIBasedProcessor.java:107)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1836)
at
org.apache.hadoop.hive.metastore.TUGIBasedProcessor.process(TUGIBasedProcessor.java:119)
at
org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
{noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (HIVE-18955) HoS: Unable to create Channel from class NioServerSocketChannel
Rui Li created HIVE-18955:
-
Summary: HoS: Unable to create Channel from class
NioServerSocketChannel
Key: HIVE-18955
URL: https://issues.apache.org/jira/browse/HIVE-18955
Project: Hive
Issue Type: Bug
Components: Spark
Reporter: Rui Li
Hit the issue when trying launch spark job. Stack trace:
{noformat}
Caused by: java.lang.NoSuchMethodError:
io.netty.channel.DefaultChannelId.newInstance()Lio/netty/channel/DefaultChannelId;
at io.netty.channel.AbstractChannel.newId(AbstractChannel.java:111)
~[netty-all-4.1.17.Final.jar:4.1.17.Final]
at io.netty.channel.AbstractChannel.(AbstractChannel.java:83)
~[netty-all-4.1.17.Final.jar:4.1.17.Final]
at
io.netty.channel.nio.AbstractNioChannel.(AbstractNioChannel.java:84)
~[netty-all-4.1.17.Final.jar:4.1.17.Final]
at
io.netty.channel.nio.AbstractNioMessageChannel.(AbstractNioMessageChannel.java:42)
~[netty-all-4.1.17.Final.jar:4.1.17.Final]
at
io.netty.channel.socket.nio.NioServerSocketChannel.(NioServerSocketChannel.java:86)
~[netty-all-4.1.17.Final.jar:4.1.17.Final]
at
io.netty.channel.socket.nio.NioServerSocketChannel.(NioServerSocketChannel.java:72)
~[netty-all-4.1.17.Final.jar:4.1.17.Final]
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
Method) ~[?:1.8.0_151]
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
~[?:1.8.0_151]
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
~[?:1.8.0_151]
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
~[?:1.8.0_151]
at
io.netty.channel.ReflectiveChannelFactory.newChannel(ReflectiveChannelFactory.java:38)
~[netty-all-4.1.17.Final.jar:4.1.17.Final]
... 32 more
{noformat}
It seems we have conflicts versions of class
{{io.netty.channel.DefaultChannelId}} from async-http-client.jar and
netty-all.jar
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (HIVE-18647) Cannot create table: Unknown column 'CREATION_METADATA_MV_CREATION_METADATA_ID_OID'
Rui Li created HIVE-18647: - Summary: Cannot create table: Unknown column 'CREATION_METADATA_MV_CREATION_METADATA_ID_OID' Key: HIVE-18647 URL: https://issues.apache.org/jira/browse/HIVE-18647 Project: Hive Issue Type: Bug Reporter: Rui Li -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (HIVE-18442) HoS: No FileSystem for scheme: nullscan
Rui Li created HIVE-18442: - Summary: HoS: No FileSystem for scheme: nullscan Key: HIVE-18442 URL: https://issues.apache.org/jira/browse/HIVE-18442 Project: Hive Issue Type: Bug Components: Spark Reporter: Rui Li Assignee: Rui Li -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (HIVE-18282) Spark tar is downloaded every time for itest
Rui Li created HIVE-18282: - Summary: Spark tar is downloaded every time for itest Key: HIVE-18282 URL: https://issues.apache.org/jira/browse/HIVE-18282 Project: Hive Issue Type: Test Reporter: Rui Li Seems we missed the md5 file for spark-2.2.0? cc [~kellyzly], [~stakiar] -- This message was sent by Atlassian JIRA (v6.4.14#64029)
Re: checkstyle changes
I also believe 140 is a little too long. BTW do we use 2 or 4 chars for continuation indent? I personally prefer 4, but I do find both cases in out code. On Fri, Dec 8, 2017 at 6:20 AM, Alexander Kolbasov <ak...@cloudera.com> wrote: > Problem with 140-wide code isn't just laptops - in many cases we need to do > side-by-side diffs (e.g. for code reviews) and this doubles the required > size. > > - Alex. > > On Thu, Dec 7, 2017 at 1:38 PM, Sergey Shelukhin <ser...@hortonworks.com> > wrote: > > > I think the 140-character change will make the code hard to use on a > > laptop without a monitor. > > > > > > On 17/12/7, 02:43, "Peter Vary" <pv...@cloudera.com> wrote: > > > > >Disclaimer: I did not have time to test it out, but according to > > >http://checkstyle.sourceforge.net/config_misc.html#Indentation > > ><http://checkstyle.sourceforge.net/config_misc.html#Indentation> > > >Maybe the indentation could be solved by: > > >lineWrappingIndentation=2 (default 4) > > >forceStrictCondition=false (default false) > > > > > >http://checkstyle.sourceforge.net/config_misc.html#TrailingComment > > ><http://checkstyle.sourceforge.net/config_misc.html#TrailingComment> > > >might help with the comments > > > > > >Sorry for not being more helpful. Maybe sometime later I will have time > > >to check these out. > > > > > >Thanks, > > >Peter > > > > > >> On Dec 7, 2017, at 10:26 AM, Zoltan Haindrich > > >><zhaindr...@hortonworks.com> wrote: > > >> > > >> Hello Eugene! > > >> > > >> I've looked into doing something with these; but I was not able to > > >>relieve the warnings you've mentioned: > > >> > > >> * the ;// is seems to be not configurable > > >> It seems like its handled by the whitespaceafter module; I'm not > sure > > >>how to allow / after ; > > >> * I think that indentation of 4 for many method arguments makes it > more > > >>readable; so I think it would be the best to just drop this check...but > > >>I've not seen any way to do this(w/o disabling the whole indentation > > >>module...) > > >> > > >> maybe someone else should take a look at itI find it pretty hard > to > > >>get docs about specific chechkstyle configurations; since the search > > >>keywords mostly contain keywords like: semicolon, whitespace, > > >>comment...which tends to pull in all kind of garbage results :) > > >> > > >> cheers, > > >> Zoltan > > >> > > >> On 6 Dec 2017 8:53 p.m., Eugene Koifman <ekoif...@hortonworks.com> > > >>wrote: > > >> It currently complains about no space between ; and // as in > “…);//foo” > > >> > > >> And also about indentation when a single method call is split into > > >>multiple lines. > > >> It insists on 4 chars in this case, though we use 2 in (all?) other > > >>cases. > > >> > > >> Could this be dialed down as well? > > >> > > >> > > >> On 12/5/17, 7:26 AM, "Peter Vary" <pv...@cloudera.com> wrote: > > >> > > >>+1 for the changes > > >> > > >>> On Dec 5, 2017, at 1:02 PM, Zoltan Haindrich <k...@rxd.hu> wrote: > > >>> > > >>> Hello, > > >>> > > >>> I've filed a ticket to make the checkstyle warnings less noisy > > >>>(https://issues.apache.org/jira/browse/HIVE-18222) > > >>> > > >>> * set maxlinelength to 140 > > >>> I think everyone is working with big-enough displays to handle this > > >>>:) > > >>> There are many methods which have complicated names / arguments / > > >>>etc ; breaking the lines more frequently hurts readability... > > >>> * disabled some restrictions like: declaration via get/set > > >>>methods for protected/package fields are not mandatory > > >>> > > >>> If you don't feel comfortable with these changes, please share your > > >>>point of view. > > >>> > > >>> cheers, > > >>> Zoltan > > >>> > > >>> > > >> > > >> > > >> > > >> > > > > > > > > -- Best regards! Rui Li
[jira] [Created] (HIVE-18242) VectorizedRowBatch cast exception when analyzing partitioned table
Rui Li created HIVE-18242:
-
Summary: VectorizedRowBatch cast exception when analyzing
partitioned table
Key: HIVE-18242
URL: https://issues.apache.org/jira/browse/HIVE-18242
Project: Hive
Issue Type: Bug
Reporter: Rui Li
Happens when I run the following (vectorization enabled):
{code}
ANALYZE TABLE srcpart PARTITION(ds, hr) COMPUTE STATISTICS;
{code}
The stack trace is:
{noformat}
Caused by: java.lang.ClassCastException:
org.apache.hadoop.hive.ql.exec.vector.VectorizedRowBatch cannot be cast to
org.apache.hadoop.io.Text
at
org.apache.hadoop.hive.serde2.objectinspector.primitive.WritableStringObjectInspector.copyObject(WritableStringObjectInspector.java:36)
at
org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorUtils.copyToStandardObject(ObjectInspectorUtils.java:425)
at
org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorUtils.partialCopyToStandardObject(ObjectInspectorUtils.java:314)
at
org.apache.hadoop.hive.ql.exec.TableScanOperator.gatherStats(TableScanOperator.java:191)
at
org.apache.hadoop.hive.ql.exec.TableScanOperator.process(TableScanOperator.java:138)
at
org.apache.hadoop.hive.ql.exec.vector.VectorMapOperator.setupPartitionContextVars(VectorMapOperator.java:682)
at
org.apache.hadoop.hive.ql.exec.vector.VectorMapOperator.cleanUpInputFileChangedOp(VectorMapOperator.java:607)
at
org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1187)
at
org.apache.hadoop.hive.ql.exec.vector.VectorMapOperator.process(VectorMapOperator.java:784)
{noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Created] (HIVE-18148) NPE in SparkDynamicPartitionPruningResolver
Rui Li created HIVE-18148:
-
Summary: NPE in SparkDynamicPartitionPruningResolver
Key: HIVE-18148
URL: https://issues.apache.org/jira/browse/HIVE-18148
Project: Hive
Issue Type: Bug
Components: Spark
Reporter: Rui Li
Assignee: Rui Li
The stack trace is:
{noformat}
2017-11-27T10:32:38,752 ERROR [e6c8aab5-ddd2-461d-b185-a7597c3e7519 main]
ql.Driver: FAILED: NullPointerException null
java.lang.NullPointerException
at
org.apache.hadoop.hive.ql.optimizer.physical.SparkDynamicPartitionPruningResolver$SparkDynamicPartitionPruningDispatcher.dispatch(SparkDynamicPartitionPruningResolver.java:100)
at
org.apache.hadoop.hive.ql.lib.TaskGraphWalker.dispatch(TaskGraphWalker.java:111)
at
org.apache.hadoop.hive.ql.lib.TaskGraphWalker.walk(TaskGraphWalker.java:180)
at
org.apache.hadoop.hive.ql.lib.TaskGraphWalker.startWalking(TaskGraphWalker.java:125)
at
org.apache.hadoop.hive.ql.optimizer.physical.SparkDynamicPartitionPruningResolver.resolve(SparkDynamicPartitionPruningResolver.java:74)
at
org.apache.hadoop.hive.ql.parse.spark.SparkCompiler.optimizeTaskPlan(SparkCompiler.java:568)
{noformat}
At this stage, there shouldn't be a DPP sink whose target map work is null. The
root cause seems to be a malformed operator tree generated by SplitOpTreeForDPP.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Created] (HIVE-18129) The ConditionalResolverMergeFiles doesn't merge empty files
Rui Li created HIVE-18129: - Summary: The ConditionalResolverMergeFiles doesn't merge empty files Key: HIVE-18129 URL: https://issues.apache.org/jira/browse/HIVE-18129 Project: Hive Issue Type: Bug Reporter: Rui Li Assignee: Rui Li If a query produces lots of empty files, these files won't be merged by the merge-small-file feature. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (HIVE-18111) Fix temp path for Spark DPP sink
Rui Li created HIVE-18111: - Summary: Fix temp path for Spark DPP sink Key: HIVE-18111 URL: https://issues.apache.org/jira/browse/HIVE-18111 Project: Hive Issue Type: Bug Reporter: Rui Li Assignee: Rui Li -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (HIVE-18041) Add SORT_QUERY_RESULTS to subquery_multi
Rui Li created HIVE-18041: - Summary: Add SORT_QUERY_RESULTS to subquery_multi Key: HIVE-18041 URL: https://issues.apache.org/jira/browse/HIVE-18041 Project: Hive Issue Type: Test Reporter: Rui Li Priority: Trivial -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (HIVE-17976) HoS: don't set output collector if there's no data to process
Rui Li created HIVE-17976:
-
Summary: HoS: don't set output collector if there's no data to
process
Key: HIVE-17976
URL: https://issues.apache.org/jira/browse/HIVE-17976
Project: Hive
Issue Type: Bug
Components: Spark
Reporter: Rui Li
Assignee: Rui Li
Priority: Minor
MR doesn't set an output collector if no row is processed, i.e.
{{ExecMapper::map}} is never called. Let's investigate whether Spark should do
the same.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Created] (HIVE-17964) HoS: some spark configs doesn't require re-creating a session
Rui Li created HIVE-17964:
-
Summary: HoS: some spark configs doesn't require re-creating a
session
Key: HIVE-17964
URL: https://issues.apache.org/jira/browse/HIVE-17964
Project: Hive
Issue Type: Improvement
Reporter: Rui Li
Priority: Minor
I guess the {{hive.spark.}} configs were initially intended for the RSC.
Therefore when they're changed, we'll re-create the session for them to take
effect. There're some configs not related to RSC that also start with
{{hive.spark.}}. We'd better rename them so that we don't unnecessarily
re-create sessions, which is usually time consuming.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Created] (HIVE-17877) HoS: combine equivalent DPP sink works
Rui Li created HIVE-17877: - Summary: HoS: combine equivalent DPP sink works Key: HIVE-17877 URL: https://issues.apache.org/jira/browse/HIVE-17877 Project: Hive Issue Type: Improvement Reporter: Rui Li Assignee: Rui Li -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (HIVE-17383) ArrayIndexOutOfBoundsException in VectorGroupByOperator
Rui Li created HIVE-17383:
-
Summary: ArrayIndexOutOfBoundsException in VectorGroupByOperator
Key: HIVE-17383
URL: https://issues.apache.org/jira/browse/HIVE-17383
Project: Hive
Issue Type: Bug
Reporter: Rui Li
Query to reproduce:
{noformat}
set hive.cbo.enable=false;
select count(*) from (select key from src group by key) s where s.key='98';
{noformat}
The stack trace is:
{noformat}
Caused by: java.lang.ArrayIndexOutOfBoundsException: 1
at
org.apache.hadoop.hive.ql.exec.vector.VectorGroupKeyHelper.copyGroupKey(VectorGroupKeyHelper.java:107)
at
org.apache.hadoop.hive.ql.exec.vector.VectorGroupByOperator$ProcessingModeReduceMergePartial.doProcessBatch(VectorGroupByOperator.java:831)
at
org.apache.hadoop.hive.ql.exec.vector.VectorGroupByOperator$ProcessingModeBase.processBatch(VectorGroupByOperator.java:174)
at
org.apache.hadoop.hive.ql.exec.vector.VectorGroupByOperator.process(VectorGroupByOperator.java:1046)
at
org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.processVectorGroup(ReduceRecordSource.java:462)
... 18 more
{noformat}
More details can be found in HIVE-16823
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Created] (HIVE-17321) HoS: analyze ORC table doesn't compute raw data size when noscan/partialscan is not specified
Rui Li created HIVE-17321: - Summary: HoS: analyze ORC table doesn't compute raw data size when noscan/partialscan is not specified Key: HIVE-17321 URL: https://issues.apache.org/jira/browse/HIVE-17321 Project: Hive Issue Type: Bug Components: Spark Reporter: Rui Li Assignee: Rui Li -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (HIVE-17193) HoS: don't combine map works that are targets of different DPPs
Rui Li created HIVE-17193: - Summary: HoS: don't combine map works that are targets of different DPPs Key: HIVE-17193 URL: https://issues.apache.org/jira/browse/HIVE-17193 Project: Hive Issue Type: Bug Reporter: Rui Li Assignee: Rui Li -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (HIVE-17133) NoSuchMethodError in Hadoop FileStatus.compareTo
Rui Li created HIVE-17133:
-
Summary: NoSuchMethodError in Hadoop FileStatus.compareTo
Key: HIVE-17133
URL: https://issues.apache.org/jira/browse/HIVE-17133
Project: Hive
Issue Type: Bug
Reporter: Rui Li
The stack trace is:
{noformat}
Caused by: java.lang.NoSuchMethodError:
org.apache.hadoop.fs.FileStatus.compareTo(Lorg/apache/hadoop/fs/FileStatus;)I
at
org.apache.hadoop.hive.ql.io.AcidUtils.lambda$getAcidState$0(AcidUtils.java:931)
at java.util.TimSort.countRunAndMakeAscending(TimSort.java:355)
at java.util.TimSort.sort(TimSort.java:234)
at java.util.Arrays.sort(Arrays.java:1512)
at java.util.ArrayList.sort(ArrayList.java:1454)
at java.util.Collections.sort(Collections.java:175)
at
org.apache.hadoop.hive.ql.io.AcidUtils.getAcidState(AcidUtils.java:929)
{noformat}
I'm on Hive master and using Hadoop 2.7.2. The method signature in Hadoop 2.7.2
is:
https://github.com/apache/hadoop/blob/release-2.7.2-RC2/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileStatus.java#L336
In Hadoop 2.8.0 it becomes:
https://github.com/apache/hadoop/blob/release-2.8.0-RC3/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileStatus.java#L332
I think that breaks binary compatibility.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Created] (HIVE-17114) HoS: Possible skew in shuffling when data is not really skewed
Rui Li created HIVE-17114:
-
Summary: HoS: Possible skew in shuffling when data is not really
skewed
Key: HIVE-17114
URL: https://issues.apache.org/jira/browse/HIVE-17114
Project: Hive
Issue Type: Bug
Reporter: Rui Li
Assignee: Rui Li
Priority: Minor
Observed in HoS and may apply to other engines as well.
When we join 2 tables on a single int key, we use the key itself as hash code
in {{ObjectInspectorUtils.hashCode}}:
{code}
case INT:
return ((IntObjectInspector) poi).get(o);
{code}
Suppose the keys are different but are all some multiples of 10. And if we
choose 10 as #reducers, the shuffle will be skewed.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Created] (HIVE-17034) The spark tar for itests is downloaded every time if md5sum is not installed
Rui Li created HIVE-17034: - Summary: The spark tar for itests is downloaded every time if md5sum is not installed Key: HIVE-17034 URL: https://issues.apache.org/jira/browse/HIVE-17034 Project: Hive Issue Type: Test Reporter: Rui Li Assignee: Rui Li I think we should either skip verifying md5, or fail the build to let developer know md5sum is required. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
Re: Review Request 60632: HIVE-16659: Query plan should reflect hive.spark.use.groupby.shuffle
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/60632/#review179595 --- Ship it! Ship It! - Rui Li On July 5, 2017, 4:07 a.m., Bing Li wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/60632/ > --- > > (Updated July 5, 2017, 4:07 a.m.) > > > Review request for hive. > > > Repository: hive-git > > > Description > --- > > HIVE-16659: Query plan should reflect hive.spark.use.groupby.shuffle > > > Diffs > - > > itests/src/test/resources/testconfiguration.properties 19ff316 > ql/src/java/org/apache/hadoop/hive/ql/exec/spark/RepartitionShuffler.java > d0c708c > ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkPlanGenerator.java > 5f85f9e > ql/src/java/org/apache/hadoop/hive/ql/parse/spark/GenSparkUtils.java > b9901da > ql/src/java/org/apache/hadoop/hive/ql/parse/spark/GenSparkWork.java afbeccb > ql/src/test/queries/clientpositive/spark_explain_groupbyshuffle.q > PRE-CREATION > ql/src/test/results/clientpositive/spark/spark_explain_groupbyshuffle.q.out > PRE-CREATION > > > Diff: https://reviews.apache.org/r/60632/diff/2/ > > > Testing > --- > > set hive.spark.use.groupby.shuffle=true; > explain select key, count(val) from t1 group by key; > > STAGE DEPENDENCIES: > Stage-1 is a root stage > Stage-0 depends on stages: Stage-1 > > STAGE PLANS: > Stage: Stage-1 > Spark > Edges: > Reducer 2 <- Map 1 (GROUP, 2) > DagName: root_20170625202742_58335619-7107-4026-9911-43d2ec449088:2 > Vertices: > Map 1 > Map Operator Tree: > TableScan > alias: t1 > Statistics: Num rows: 20 Data size: 140 Basic stats: > COMPLETE Column stats: NONE > Select Operator > expressions: key (type: int), val (type: string) > outputColumnNames: key, val > Statistics: Num rows: 20 Data size: 140 Basic stats: > COMPLETE Column stats: NONE > Group By Operator > aggregations: count(val) > keys: key (type: int) > mode: hash > outputColumnNames: _col0, _col1 > Statistics: Num rows: 20 Data size: 140 Basic stats: > COMPLETE Column stats: NONE > Reduce Output Operator > key expressions: _col0 (type: int) > sort order: + > Map-reduce partition columns: _col0 (type: int) > Statistics: Num rows: 20 Data size: 140 Basic stats: > COMPLETE Column stats: NONE > value expressions: _col1 (type: bigint) > Reducer 2 > Reduce Operator Tree: > Group By Operator > aggregations: count(VALUE._col0) > keys: KEY._col0 (type: int) > mode: mergepartial > outputColumnNames: _col0, _col1 > Statistics: Num rows: 10 Data size: 70 Basic stats: COMPLETE > Column stats: NONE > File Output Operator > compressed: false > Statistics: Num rows: 10 Data size: 70 Basic stats: > COMPLETE Column stats: NONE > table: > input format: > org.apache.hadoop.mapred.SequenceFileInputFormat > output format: > org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat > serde: > org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe > > Stage: Stage-0 > Fetch Operator > limit: -1 > Processor Tree: > ListSink > > > set hive.spark.use.groupby.shuffle=false; > explain select key, count(val) from t1 group by key; > > STAGE DEPENDENCIES: > Stage-1 is a root stage > Stage-0 depends on stages: Stage-1 > > STAGE PLANS: > Stage: Stage-1 > Spark > Edges: > Reducer 2 <- Map 1 (GROUP, 2) > DagName: root_20170625203122_3afe01dd-41cc-477e-9098-ddd58b37ad4e:3 > Vertices: > Map 1 > Map Operator Tree: > TableScan > alias: t1 > Statistics: Num rows: 20 Data size: 140 Basic stats: > COMPLETE Colum
[jira] [Created] (HIVE-17020) Aggressive RS dedup can incorrectly remove OP tree branch
Rui Li created HIVE-17020:
-
Summary: Aggressive RS dedup can incorrectly remove OP tree branch
Key: HIVE-17020
URL: https://issues.apache.org/jira/browse/HIVE-17020
Project: Hive
Issue Type: Bug
Reporter: Rui Li
Assignee: Rui Li
Suppose we have an OP tree like this:
{noformat}
...
|
RS[1]
|
SEL[2]
/\
SEL[3] SEL[4]
| |
RS[5] FS[6]
|
...
{noformat}
When doing aggressive RS dedup, we'll remove all the operators between RS5 and
RS1, and thus the branch containing FS6 is lost.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
Re: Review Request 60632: HIVE-16659: Query plan should reflect hive.spark.use.groupby.shuffle
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/60632/#review179554 --- ql/src/java/org/apache/hadoop/hive/ql/parse/spark/GenSparkUtils.java Lines 68 (patched) <https://reviews.apache.org/r/60632/#comment254315> Please avoid * import ql/src/java/org/apache/hadoop/hive/ql/parse/spark/GenSparkUtils.java Lines 432 (patched) <https://reviews.apache.org/r/60632/#comment254316> it's preferable to use HiveConf::getBoolVar ql/src/java/org/apache/hadoop/hive/ql/parse/spark/GenSparkUtils.java Line 438 (original), 441 (patched) <https://reviews.apache.org/r/60632/#comment254317> nit: extra space before !useSparkGroupBy ql/src/java/org/apache/hadoop/hive/ql/parse/spark/GenSparkUtils.java Line 471 (original), 477 (patched) <https://reviews.apache.org/r/60632/#comment254319> let's delete this comment - Rui Li On July 4, 2017, 8:48 a.m., Bing Li wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/60632/ > --- > > (Updated July 4, 2017, 8:48 a.m.) > > > Review request for hive. > > > Repository: hive-git > > > Description > --- > > HIVE-16659: Query plan should reflect hive.spark.use.groupby.shuffle > > > Diffs > - > > itests/src/test/resources/testconfiguration.properties 19ff316 > ql/src/java/org/apache/hadoop/hive/ql/exec/spark/RepartitionShuffler.java > d0c708c > ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkPlanGenerator.java > 5f85f9e > ql/src/java/org/apache/hadoop/hive/ql/parse/spark/GenSparkUtils.java > b9901da > ql/src/java/org/apache/hadoop/hive/ql/parse/spark/GenSparkWork.java afbeccb > ql/src/test/queries/clientpositive/spark_explain_groupbyshuffle.q > PRE-CREATION > ql/src/test/results/clientpositive/spark/spark_explain_groupbyshuffle.q.out > PRE-CREATION > > > Diff: https://reviews.apache.org/r/60632/diff/1/ > > > Testing > --- > > set hive.spark.use.groupby.shuffle=true; > explain select key, count(val) from t1 group by key; > > STAGE DEPENDENCIES: > Stage-1 is a root stage > Stage-0 depends on stages: Stage-1 > > STAGE PLANS: > Stage: Stage-1 > Spark > Edges: > Reducer 2 <- Map 1 (GROUP, 2) > DagName: root_20170625202742_58335619-7107-4026-9911-43d2ec449088:2 > Vertices: > Map 1 > Map Operator Tree: > TableScan > alias: t1 > Statistics: Num rows: 20 Data size: 140 Basic stats: > COMPLETE Column stats: NONE > Select Operator > expressions: key (type: int), val (type: string) > outputColumnNames: key, val > Statistics: Num rows: 20 Data size: 140 Basic stats: > COMPLETE Column stats: NONE > Group By Operator > aggregations: count(val) > keys: key (type: int) > mode: hash > outputColumnNames: _col0, _col1 > Statistics: Num rows: 20 Data size: 140 Basic stats: > COMPLETE Column stats: NONE > Reduce Output Operator > key expressions: _col0 (type: int) > sort order: + > Map-reduce partition columns: _col0 (type: int) > Statistics: Num rows: 20 Data size: 140 Basic stats: > COMPLETE Column stats: NONE > value expressions: _col1 (type: bigint) > Reducer 2 > Reduce Operator Tree: > Group By Operator > aggregations: count(VALUE._col0) > keys: KEY._col0 (type: int) > mode: mergepartial > outputColumnNames: _col0, _col1 > Statistics: Num rows: 10 Data size: 70 Basic stats: COMPLETE > Column stats: NONE > File Output Operator > compressed: false > Statistics: Num rows: 10 Data size: 70 Basic stats: > COMPLETE Column stats: NONE > table: > input format: > org.apache.hadoop.mapred.SequenceFileInputFormat > output format: > org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat > serde: > org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe > >
[jira] [Created] (HIVE-16876) RpcServer should be re-created when Rpc configs change
Rui Li created HIVE-16876: - Summary: RpcServer should be re-created when Rpc configs change Key: HIVE-16876 URL: https://issues.apache.org/jira/browse/HIVE-16876 Project: Hive Issue Type: Bug Reporter: Rui Li Assignee: Rui Li -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (HIVE-16767) Update people website with recent changes
Rui Li created HIVE-16767: - Summary: Update people website with recent changes Key: HIVE-16767 URL: https://issues.apache.org/jira/browse/HIVE-16767 Project: Hive Issue Type: Task Components: Documentation Reporter: Rui Li Assignee: Rui Li -- This message was sent by Atlassian JIRA (v6.3.15#6346)
Re: Welcome Rui Li to Hive PMC
Thank you guys :) On Thu, May 25, 2017 at 3:29 PM, Peter Vary <pv...@cloudera.com> wrote: > Congratulations Rui! > > > On May 25, 2017, at 6:19 AM, Xuefu Zhang <xu...@apache.org> wrote: > > > > Hi all, > > > > It's an honer to announce that Apache Hive PMC has recently voted to > invite > > Rui Li as a new Hive PMC member. Rui is a long time Hive contributor and > > committer, and has made significant contribution in Hive especially in > Hive > > on Spark. Please join me in congratulating him and looking forward to a > > bigger role that he will play in Apache Hive project. > > > > Thanks, > > Xuefu > > -- Best regards! Rui Li Cell: (+86) 13564950210
[jira] [Created] (HIVE-16739) HoS DPP generates malformed plan when hive.tez.dynamic.semijoin.reduction is on
Rui Li created HIVE-16739:
-
Summary: HoS DPP generates malformed plan when
hive.tez.dynamic.semijoin.reduction is on
Key: HIVE-16739
URL: https://issues.apache.org/jira/browse/HIVE-16739
Project: Hive
Issue Type: Bug
Components: Spark
Reporter: Rui Li
Assignee: Rui Li
HoS DPP currently can't handle dynamic semi join and will result in
{{ClassCastException org.apache.hadoop.hive.ql.plan.ReduceWork cannot be cast
to org.apache.hadoop.hive.ql.plan.MapWork}}.
We should either disable or implement it for HoS.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Created] (HIVE-16659) Query plan should reflect hive.spark.use.groupby.shuffle
Rui Li created HIVE-16659: - Summary: Query plan should reflect hive.spark.use.groupby.shuffle Key: HIVE-16659 URL: https://issues.apache.org/jira/browse/HIVE-16659 Project: Hive Issue Type: Bug Reporter: Rui Li Assignee: Rui Li It's useful to show the shuffle type used in the query plan. Currently it shows "GROUP" no matter what we set for hive.spark.use.groupby.shuffle. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
Re: [Announce] New committer: Vineet Garg
Congrats :)
On Wed, May 10, 2017 at 2:06 PM, Zoltan Haindrich <
zhaindr...@hortonworks.com> wrote:
> Congratulations!
>
>
> On 10 May 2017 7:57 a.m., Prasanth Jayachandran <
> pjayachand...@hortonworks.com> wrote:
> Congratulations Vineeth!!
>
> Thanks
> Prasanth
>
>
>
> On Tue, May 9, 2017 at 10:52 PM -0700, "Jesus Camacho Rodriguez" <
> jcama...@apache.org<mailto:jcama...@apache.org>> wrote:
>
>
> Congrats Vineet! Well deserved!
>
> --
> Jesús
>
>
>
>
>
> On 5/10/17, 6:45 AM, "Peter Vary" wrote:
>
> >Congratulations Vineet! :)
> >
> >2017. máj. 9. 22:25 ezt írta ("Ashutosh Chauhan" ):
> >
> >> The Project Management Committee (PMC) for Apache Hive has invited
> Vineet
> >> Garg to become a committer and we are pleased to announce that he has
> >> accepted.
> >>
> >> Welcome, Vineet!
> >>
> >> Thanks,
> >> Ashutosh
> >>
>
>
>
>
>
--
Best regards!
Rui Li
Cell: (+86) 13564950210
Re: Review Request 50787: Add a timezone-aware timestamp
/BinarySortableSerDe.java 89e15c3 serde/src/java/org/apache/hadoop/hive/serde2/io/TimestampTZWritable.java PRE-CREATION serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazyFactory.java 23dbe6a serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazyTimestampTZ.java PRE-CREATION serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazyUtils.java 73c72e1 serde/src/java/org/apache/hadoop/hive/serde2/lazy/objectinspector/primitive/LazyPrimitiveObjectInspectorFactory.java 5601734 serde/src/java/org/apache/hadoop/hive/serde2/lazy/objectinspector/primitive/LazyTimestampTZObjectInspector.java PRE-CREATION serde/src/java/org/apache/hadoop/hive/serde2/lazybinary/LazyBinaryFactory.java 52f3527 serde/src/java/org/apache/hadoop/hive/serde2/lazybinary/LazyBinarySerDe.java 56b4ca3 serde/src/java/org/apache/hadoop/hive/serde2/lazybinary/LazyBinaryTimestampTZ.java PRE-CREATION serde/src/java/org/apache/hadoop/hive/serde2/lazybinary/LazyBinaryUtils.java 8237b64 serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/ObjectInspectorConverters.java 24b3d4e serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/ObjectInspectorUtils.java ba44bae serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/PrimitiveObjectInspector.java 70633f3 serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/JavaTimestampTZObjectInspector.java PRE-CREATION serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/PrimitiveObjectInspectorConverter.java e08ad43 serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/PrimitiveObjectInspectorFactory.java 2ed0843 serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/PrimitiveObjectInspectorUtils.java 9642a7e serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/SettableTimestampTZObjectInspector.java PRE-CREATION serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/TimestampTZObjectInspector.java PRE-CREATION serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/WritableConstantTimestampTZObjectInspector.java PRE-CREATION serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/WritableTimestampTZObjectInspector.java PRE-CREATION serde/src/java/org/apache/hadoop/hive/serde2/thrift/Type.java 0ad8c02 serde/src/java/org/apache/hadoop/hive/serde2/typeinfo/TypeInfoFactory.java 43c4819 serde/src/test/org/apache/hadoop/hive/serde2/io/TestTimestampTZWritable.java PRE-CREATION service-rpc/if/TCLIService.thrift 824b049 service-rpc/src/gen/thrift/gen-cpp/TCLIService_constants.cpp 991cb2e service-rpc/src/gen/thrift/gen-cpp/TCLIService_types.h 8accf66 service-rpc/src/gen/thrift/gen-cpp/TCLIService_types.cpp b6995c4 service-rpc/src/gen/thrift/gen-javabean/org/apache/hive/service/rpc/thrift/TCLIServiceConstants.java 930bed7 service-rpc/src/gen/thrift/gen-javabean/org/apache/hive/service/rpc/thrift/TProtocolVersion.java 18a7825 service-rpc/src/gen/thrift/gen-javabean/org/apache/hive/service/rpc/thrift/TTypeId.java a3735eb service-rpc/src/gen/thrift/gen-php/Types.php ee5acd2 service-rpc/src/gen/thrift/gen-py/TCLIService/constants.py c8d4f8f service-rpc/src/gen/thrift/gen-py/TCLIService/ttypes.py e9faa2a service-rpc/src/gen/thrift/gen-rb/t_c_l_i_service_constants.rb 25adbb4 service-rpc/src/gen/thrift/gen-rb/t_c_l_i_service_types.rb 714309c service/src/java/org/apache/hive/service/cli/ColumnValue.java 76e8c03 service/src/java/org/apache/hive/service/cli/TypeDescriptor.java d634bef Diff: https://reviews.apache.org/r/50787/diff/9/ Changes: https://reviews.apache.org/r/50787/diff/8-9/ Testing --- Thanks, Rui Li
Re: Review Request 50787: Add a timezone-aware timestamp
> On May 9, 2017, 11:05 p.m., Ashutosh Chauhan wrote: > > ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/translator/TypeConverter.java > > Lines 204 (patched) > > <https://reviews.apache.org/r/50787/diff/8/?file=1710538#file1710538line204> > > > > Can you file a bug in Calcite that it should have sql type to represent > > TS w TZ? Filed CALCITE-1784 for it > On May 9, 2017, 11:05 p.m., Ashutosh Chauhan wrote: > > ql/src/java/org/apache/hadoop/hive/ql/udf/UDFToString.java > > Lines 160 (patched) > > <https://reviews.apache.org/r/50787/diff/8/?file=1710546#file1710546line160> > > > > Add a comment that string reperesentation will return TS in UTC zone > > and not in original TZ. Here we convert timestamptz to string, which means TZ is already in UTC. I will add the comment when we convert string to timestamptz. - Rui --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/50787/#review174383 ------- On May 8, 2017, 3:17 p.m., Rui Li wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/50787/ > --- > > (Updated May 8, 2017, 3:17 p.m.) > > > Review request for hive, pengcheng xiong and Xuefu Zhang. > > > Bugs: HIVE-14412 > https://issues.apache.org/jira/browse/HIVE-14412 > > > Repository: hive-git > > > Description > --- > > The 1st patch to add timezone-aware timestamp. > > > Diffs > - > > common/src/java/org/apache/hadoop/hive/common/type/TimestampTZ.java > PRE-CREATION > common/src/test/org/apache/hadoop/hive/common/type/TestTimestampTZ.java > PRE-CREATION > contrib/src/test/queries/clientnegative/serde_regex.q a676338 > contrib/src/test/queries/clientpositive/serde_regex.q d75d607 > contrib/src/test/results/clientnegative/serde_regex.q.out 58b1c02 > contrib/src/test/results/clientpositive/serde_regex.q.out 2984293 > hbase-handler/src/test/queries/positive/hbase_timestamp.q 0350afe > hbase-handler/src/test/results/positive/hbase_timestamp.q.out 3918121 > itests/hive-blobstore/src/test/queries/clientpositive/orc_format_part.q > 358eccd > > itests/hive-blobstore/src/test/queries/clientpositive/orc_nonstd_partitions_loc.q > c462538 > itests/hive-blobstore/src/test/queries/clientpositive/rcfile_format_part.q > c563d3a > > itests/hive-blobstore/src/test/queries/clientpositive/rcfile_nonstd_partitions_loc.q > d17c281 > itests/hive-blobstore/src/test/results/clientpositive/orc_format_part.q.out > 5d1319f > > itests/hive-blobstore/src/test/results/clientpositive/orc_nonstd_partitions_loc.q.out > 70e72f7 > > itests/hive-blobstore/src/test/results/clientpositive/rcfile_format_part.q.out > bed10ab > > itests/hive-blobstore/src/test/results/clientpositive/rcfile_nonstd_partitions_loc.q.out > c6442f9 > jdbc/src/java/org/apache/hive/jdbc/HiveBaseResultSet.java ade1900 > jdbc/src/java/org/apache/hive/jdbc/JdbcColumn.java 38918f0 > ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java 1b556ac > ql/src/java/org/apache/hadoop/hive/ql/exec/GroupByOperator.java f8b55da > ql/src/java/org/apache/hadoop/hive/ql/exec/SerializationUtilities.java > 01a652d > > ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/translator/TypeConverter.java > 38308c9 > ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java > 0cf9205 > ql/src/java/org/apache/hadoop/hive/ql/parse/HiveLexer.g 190b66b > ql/src/java/org/apache/hadoop/hive/ql/parse/HiveParser.g ca639d3 > ql/src/java/org/apache/hadoop/hive/ql/parse/IdentifiersParser.g 645ced9 > ql/src/java/org/apache/hadoop/hive/ql/parse/TypeCheckProcFactory.java > c3227c9 > ql/src/java/org/apache/hadoop/hive/ql/stats/StatsUtils.java bda2050 > ql/src/java/org/apache/hadoop/hive/ql/udf/UDFToBoolean.java 7cdf2c3 > ql/src/java/org/apache/hadoop/hive/ql/udf/UDFToString.java 5cacd59 > ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDF.java 68d98f5 > ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFDate.java > 5a31e61 > > ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFToTimestampTZ.java > PRE-CREATION > > ql/src/test/org/apache/hadoop/hive/ql/parse/TestSQL11ReservedKeyWordsNegative.java > 0dc6b19 > ql/src/test/queries/clientnegative/serde_regex.q c9cfc7d > ql/src/test/queries/clientnegative/serde_regex2.q a
[jira] [Created] (HIVE-16613) SaslClientHandler.sendHello is eating exceptions
Rui Li created HIVE-16613: - Summary: SaslClientHandler.sendHello is eating exceptions Key: HIVE-16613 URL: https://issues.apache.org/jira/browse/HIVE-16613 Project: Hive Issue Type: Bug Reporter: Rui Li Assignee: Rui Li -- This message was sent by Atlassian JIRA (v6.3.15#6346)
Re: Review Request 50787: Add a timezone-aware timestamp
/BinarySortableSerDe.java 89e15c3 serde/src/java/org/apache/hadoop/hive/serde2/io/TimestampTZWritable.java PRE-CREATION serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazyFactory.java 23dbe6a serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazyTimestampTZ.java PRE-CREATION serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazyUtils.java 73c72e1 serde/src/java/org/apache/hadoop/hive/serde2/lazy/objectinspector/primitive/LazyPrimitiveObjectInspectorFactory.java 5601734 serde/src/java/org/apache/hadoop/hive/serde2/lazy/objectinspector/primitive/LazyTimestampTZObjectInspector.java PRE-CREATION serde/src/java/org/apache/hadoop/hive/serde2/lazybinary/LazyBinaryFactory.java 52f3527 serde/src/java/org/apache/hadoop/hive/serde2/lazybinary/LazyBinarySerDe.java 56b4ca3 serde/src/java/org/apache/hadoop/hive/serde2/lazybinary/LazyBinaryTimestampTZ.java PRE-CREATION serde/src/java/org/apache/hadoop/hive/serde2/lazybinary/LazyBinaryUtils.java 8237b64 serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/ObjectInspectorConverters.java 24b3d4e serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/ObjectInspectorUtils.java ba44bae serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/PrimitiveObjectInspector.java 70633f3 serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/JavaTimestampTZObjectInspector.java PRE-CREATION serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/PrimitiveObjectInspectorConverter.java e08ad43 serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/PrimitiveObjectInspectorFactory.java 2ed0843 serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/PrimitiveObjectInspectorUtils.java 9642a7e serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/SettableTimestampTZObjectInspector.java PRE-CREATION serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/TimestampTZObjectInspector.java PRE-CREATION serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/WritableConstantTimestampTZObjectInspector.java PRE-CREATION serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/WritableTimestampTZObjectInspector.java PRE-CREATION serde/src/java/org/apache/hadoop/hive/serde2/thrift/Type.java 0ad8c02 serde/src/java/org/apache/hadoop/hive/serde2/typeinfo/TypeInfoFactory.java 43c4819 serde/src/test/org/apache/hadoop/hive/serde2/io/TestTimestampTZWritable.java PRE-CREATION service-rpc/if/TCLIService.thrift 824b049 service-rpc/src/gen/thrift/gen-cpp/TCLIService_constants.cpp 991cb2e service-rpc/src/gen/thrift/gen-cpp/TCLIService_types.h 8accf66 service-rpc/src/gen/thrift/gen-cpp/TCLIService_types.cpp b6995c4 service-rpc/src/gen/thrift/gen-javabean/org/apache/hive/service/rpc/thrift/TCLIServiceConstants.java 930bed7 service-rpc/src/gen/thrift/gen-javabean/org/apache/hive/service/rpc/thrift/TProtocolVersion.java 18a7825 service-rpc/src/gen/thrift/gen-javabean/org/apache/hive/service/rpc/thrift/TTypeId.java a3735eb service-rpc/src/gen/thrift/gen-php/Types.php ee5acd2 service-rpc/src/gen/thrift/gen-py/TCLIService/constants.py c8d4f8f service-rpc/src/gen/thrift/gen-py/TCLIService/ttypes.py e9faa2a service-rpc/src/gen/thrift/gen-rb/t_c_l_i_service_constants.rb 25adbb4 service-rpc/src/gen/thrift/gen-rb/t_c_l_i_service_types.rb 714309c service/src/java/org/apache/hive/service/cli/ColumnValue.java 76e8c03 service/src/java/org/apache/hive/service/cli/TypeDescriptor.java d634bef Diff: https://reviews.apache.org/r/50787/diff/8/ Changes: https://reviews.apache.org/r/50787/diff/7-8/ Testing --- Thanks, Rui Li
Re: Review Request 50787: Add a timezone-aware timestamp
> On May 7, 2017, 11:22 p.m., Xuefu Zhang wrote: > > common/src/java/org/apache/hadoop/hive/common/type/TimestampTZ.java > > Lines 138 (patched) > > <https://reviews.apache.org/r/50787/diff/7/?file=1706764#file1706764line138> > > > > Not sure if I understand this, but why cannot we get seconds/nanos from > > date/timestamp and then convert to TimestapTZ? I assume this is a faster > > way. Hi Xuefu, the reason why I did this: 1. As Ashutosh suggested, we will use LocalDate and LocalDateTime for Date and Timestamp in the future. When that happens, date/timestamp won't have seconds/nanos part, instead they're only descriptions of time. So the conversion should be done based on text format. 2. At the moment, the seconds/nanos of date/timestamp is computed using system timezone. So the conversion can have different results in different systems. I noted Carter also suggested that SQL standard requires session zone should be taken into consideration in the conversion. Consolidating your suggestions with Carter's, I think we can: make the conversion text-wise, and append the system zone (Hive currently doesn't have session zone). For example, a date of '2017-01-01' in LA will be converted to timestamptz as '2017-01-01 00:00:00 America/Los_Angeles', which in turn converted to '2017-01-01 08:00:00.0 Z'. Does this make sense? > On May 7, 2017, 11:22 p.m., Xuefu Zhang wrote: > > ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/translator/TypeConverter.java > > Lines 204 (patched) > > <https://reviews.apache.org/r/50787/diff/7/?file=1706785#file1706785line204> > > > > What does this imply? The method converts our primitive type to a SqlTypeName in calcite. But SqlTypeName currently doesn't have timestamp with time zone. This will have some impact when calcite does the optimization, e.g. computing average value sizes. But I think we have to live with it untile SqlTypeName supports timestamp with time zone. > On May 7, 2017, 11:22 p.m., Xuefu Zhang wrote: > > serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazyTimestampTZ.java > > Lines 32 (patched) > > <https://reviews.apache.org/r/50787/diff/7/?file=1706827#file1706827line32> > > > > Can you also make a note about the source of the code, like > > TimeStampTZWritable? sure, will do - Rui --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/50787/#review174136 --- On May 8, 2017, 6:51 a.m., Rui Li wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/50787/ > --- > > (Updated May 8, 2017, 6:51 a.m.) > > > Review request for hive, pengcheng xiong and Xuefu Zhang. > > > Bugs: HIVE-14412 > https://issues.apache.org/jira/browse/HIVE-14412 > > > Repository: hive-git > > > Description > --- > > The 1st patch to add timezone-aware timestamp. > > > Diffs > - > > common/src/java/org/apache/hadoop/hive/common/type/TimestampTZ.java > PRE-CREATION > common/src/test/org/apache/hadoop/hive/common/type/TestTimestampTZ.java > PRE-CREATION > contrib/src/test/queries/clientnegative/serde_regex.q a676338 > contrib/src/test/queries/clientpositive/serde_regex.q d75d607 > contrib/src/test/results/clientnegative/serde_regex.q.out 58b1c02 > contrib/src/test/results/clientpositive/serde_regex.q.out 2984293 > hbase-handler/src/test/queries/positive/hbase_timestamp.q 0350afe > hbase-handler/src/test/results/positive/hbase_timestamp.q.out 3918121 > itests/hive-blobstore/src/test/queries/clientpositive/orc_format_part.q > 358eccd > > itests/hive-blobstore/src/test/queries/clientpositive/orc_nonstd_partitions_loc.q > c462538 > itests/hive-blobstore/src/test/queries/clientpositive/rcfile_format_part.q > c563d3a > > itests/hive-blobstore/src/test/queries/clientpositive/rcfile_nonstd_partitions_loc.q > d17c281 > itests/hive-blobstore/src/test/results/clientpositive/orc_format_part.q.out > 5d1319f > > itests/hive-blobstore/src/test/results/clientpositive/orc_nonstd_partitions_loc.q.out > 70e72f7 > > itests/hive-blobstore/src/test/results/clientpositive/rcfile_format_part.q.out > bed10ab > > itests/hive-blobstore/src/test/results/clientpositive/rcfile_nonstd_partitions_loc.q.out > c6442f9 > jdbc/src/java/org/apache/hive/jdbc/HiveBaseResultSet.java ade1900 > jdbc/src/java/org/apache/hive/jdbc/JdbcColumn.java 38918f0 > ql/src/ja
Re: Welcome new Hive committer, Zhihai Xu
Congrats Zhihai :) On Sun, May 7, 2017 at 3:41 PM, Lefty Leverenz <leftylever...@gmail.com> wrote: > Congratulations Zhihai! > > -- Lefty > > > On Sat, May 6, 2017 at 2:06 AM, Zoltan Haindrich < > zhaindr...@hortonworks.com > > wrote: > > > Congratulations Zhihai! > > > > On 6 May 2017 9:52 a.m., Mohammad Islam <misla...@yahoo.com.INVALID> > > wrote: > > Congrats Zhihai!! > > > > On Friday, May 5, 2017 9:52 AM, Xuefu Zhang <xu...@apache.org> > wrote: > > > > > > Hi all, > > > > I'm very please to announce that Hive PMC has recently voted to offer > > Zhihai a committership which he accepted. Please join me in > congratulating > > on this recognition and thanking him for his contributions to Hive. > > > > Regards, > > Xuefu > > > > > > > > > > > -- Best regards! Rui Li Cell: (+86) 13564950210
[jira] [Created] (HIVE-16593) SparkClientFactory.stop may prevent JVM from exiting
Rui Li created HIVE-16593: - Summary: SparkClientFactory.stop may prevent JVM from exiting Key: HIVE-16593 URL: https://issues.apache.org/jira/browse/HIVE-16593 Project: Hive Issue Type: Bug Components: Spark Reporter: Rui Li Assignee: Rui Li -- This message was sent by Atlassian JIRA (v6.3.15#6346)
Re: Review Request 50787: Add a timezone-aware timestamp
> On May 3, 2017, 9:57 p.m., pengcheng xiong wrote: > > ql/src/java/org/apache/hadoop/hive/ql/parse/HiveLexer.g > > Lines 132 (patched) > > <https://reviews.apache.org/r/50787/diff/7/?file=1706787#file1706787line132> > > > > I think Identifier["timestamptz"] and Identifier["zone"] may be > > sufficient. It is not necessary to make them as key words and then add them > > back as identifiers. You can have a try and see if it works. Thanks.. Hi Pengcheng, sorry I'm quite ignorant about antlr. Could you please be more specific how to add the Identifiers? Let me explain what I intend to do. The new data type is named "timestamp with time zone", and "timestamptz" is added as a type alias. I thought it's required to add key words for type names. And according to the PostgreSQL doc we referenced (https://www.postgresql.org/docs/9.5/static/sql-keywords-appendix.html), "zone" is a non-reserved SQL key word and "timestamptz" is not a key word. So I added them in IdentifierParser.g as nonReserved. - Rui --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/50787/#review173833 --- On May 3, 2017, 6:34 a.m., Rui Li wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/50787/ > --- > > (Updated May 3, 2017, 6:34 a.m.) > > > Review request for hive. > > > Bugs: HIVE-14412 > https://issues.apache.org/jira/browse/HIVE-14412 > > > Repository: hive-git > > > Description > --- > > The 1st patch to add timezone-aware timestamp. > > > Diffs > - > > common/src/java/org/apache/hadoop/hive/common/type/TimestampTZ.java > PRE-CREATION > common/src/test/org/apache/hadoop/hive/common/type/TestTimestampTZ.java > PRE-CREATION > contrib/src/test/queries/clientnegative/serde_regex.q a676338 > contrib/src/test/queries/clientpositive/serde_regex.q d75d607 > contrib/src/test/results/clientnegative/serde_regex.q.out 58b1c02 > contrib/src/test/results/clientpositive/serde_regex.q.out 2984293 > hbase-handler/src/test/queries/positive/hbase_timestamp.q 0350afe > hbase-handler/src/test/results/positive/hbase_timestamp.q.out 3918121 > itests/hive-blobstore/src/test/queries/clientpositive/orc_format_part.q > 358eccd > > itests/hive-blobstore/src/test/queries/clientpositive/orc_nonstd_partitions_loc.q > c462538 > itests/hive-blobstore/src/test/queries/clientpositive/rcfile_format_part.q > c563d3a > > itests/hive-blobstore/src/test/queries/clientpositive/rcfile_nonstd_partitions_loc.q > d17c281 > itests/hive-blobstore/src/test/results/clientpositive/orc_format_part.q.out > 5d1319f > > itests/hive-blobstore/src/test/results/clientpositive/orc_nonstd_partitions_loc.q.out > 70e72f7 > > itests/hive-blobstore/src/test/results/clientpositive/rcfile_format_part.q.out > bed10ab > > itests/hive-blobstore/src/test/results/clientpositive/rcfile_nonstd_partitions_loc.q.out > c6442f9 > jdbc/src/java/org/apache/hive/jdbc/HiveBaseResultSet.java ade1900 > jdbc/src/java/org/apache/hive/jdbc/JdbcColumn.java 38918f0 > ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java 8dc5f2e > ql/src/java/org/apache/hadoop/hive/ql/exec/GroupByOperator.java f8b55da > ql/src/java/org/apache/hadoop/hive/ql/exec/SerializationUtilities.java > 01a652d > > ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/translator/TypeConverter.java > 38308c9 > ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java > 0cf9205 > ql/src/java/org/apache/hadoop/hive/ql/parse/HiveLexer.g 0721b92 > ql/src/java/org/apache/hadoop/hive/ql/parse/HiveParser.g d98a663 > ql/src/java/org/apache/hadoop/hive/ql/parse/IdentifiersParser.g 8598fae > ql/src/java/org/apache/hadoop/hive/ql/parse/TypeCheckProcFactory.java > 8f8eab0 > ql/src/java/org/apache/hadoop/hive/ql/stats/StatsUtils.java bda2050 > ql/src/java/org/apache/hadoop/hive/ql/udf/UDFToBoolean.java 7cdf2c3 > ql/src/java/org/apache/hadoop/hive/ql/udf/UDFToString.java 5cacd59 > ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDF.java 68d98f5 > ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFDate.java > 5a31e61 > > ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFToTimestampTZ.java > PRE-CREATION > > ql/src/test/org/apache/hadoop/hive/ql/parse/TestSQL11Res
[jira] [Created] (HIVE-16573) In-place update for HoS can't be disabled
Rui Li created HIVE-16573:
-
Summary: In-place update for HoS can't be disabled
Key: HIVE-16573
URL: https://issues.apache.org/jira/browse/HIVE-16573
Project: Hive
Issue Type: Bug
Components: Spark
Reporter: Rui Li
Assignee: Rui Li
Priority: Minor
{{hive.spark.exec.inplace.progress}} has no effect
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
Re: Review Request 58865: HIVE-16552: Limit the number of tasks a Spark job may contain
> On May 3, 2017, 3:35 a.m., Rui Li wrote:
> >
Xuefu, the patch looks good to me overall. Thanks for the work. Do you think we
should add some negative test case for it?
> On May 3, 2017, 3:35 a.m., Rui Li wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkTask.java
> > Lines 132 (patched)
> > <https://reviews.apache.org/r/58865/diff/3/?file=1705971#file1705971line132>
> >
> > I think the log is unnecessary because the failure should already be
> > logged in the monitor
>
> Xuefu Zhang wrote:
> This is not new code.
Do you mean "LOG.info("Failed to submit Spark job " + sparkJobID);" is not new
code? I don't find it in the current SparkTask.java.
> On May 3, 2017, 3:35 a.m., Rui Li wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkTask.java
> > Lines 135 (patched)
> > <https://reviews.apache.org/r/58865/diff/3/?file=1705971#file1705971line135>
> >
> > Same as above. Can we consolidate the logs a bit?
>
> Xuefu Zhang wrote:
> Jobmonitor prints it on console, while the log here is written to
> hive.log.
The console.printInfo method does both printing and logging:
public void printInfo(String info, String detail, boolean isSilent) {
if (!isSilent) {
getInfoStream().println(info);
}
LOG.info(info + StringUtils.defaultString(detail));
}
> On May 3, 2017, 3:35 a.m., Rui Li wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/exec/spark/status/RemoteSparkJobMonitor.java
> > Lines 104 (patched)
> > <https://reviews.apache.org/r/58865/diff/3/?file=1705972#file1705972line104>
> >
> > Maybe I was being misleading. I mean we can compute the total task only
> > once when the job first reaches RUNNING state, i.e. in the "if (!running)".
> > At this point, the total count is determined and won't change.
>
> Xuefu Zhang wrote:
> Yeah. However, I'd like to keep the state transition to running first
> before breaking up and returning rc=4. In fact, if we lose the transition,
> Hive actually goes into an instable state. What you said was what I tried in
> first place.
I see. Thanks for the explanation.
- Rui
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/58865/#review173689
---
On May 2, 2017, 6:49 p.m., Xuefu Zhang wrote:
>
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/58865/
> ---
>
> (Updated May 2, 2017, 6:49 p.m.)
>
>
> Review request for hive.
>
>
> Bugs: HIVE-16552
> https://issues.apache.org/jira/browse/HIVE-16552
>
>
> Repository: hive-git
>
>
> Description
> ---
>
> See JIRA description
>
>
> Diffs
> -
>
> common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 84398c6
> ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkTask.java 32a7730
>
> ql/src/java/org/apache/hadoop/hive/ql/exec/spark/status/RemoteSparkJobMonitor.java
> dd73f3e
>
> ql/src/java/org/apache/hadoop/hive/ql/exec/spark/status/SparkJobMonitor.java
> 0b224f2
>
>
> Diff: https://reviews.apache.org/r/58865/diff/3/
>
>
> Testing
> ---
>
> Test locally
>
>
> Thanks,
>
> Xuefu Zhang
>
>
Re: Review Request 50787: Add a timezone-aware timestamp
/apache/hadoop/hive/serde2/io/TimestampTZWritable.java PRE-CREATION serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazyFactory.java 23dbe6a serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazyTimestampTZ.java PRE-CREATION serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazyUtils.java 73c72e1 serde/src/java/org/apache/hadoop/hive/serde2/lazy/objectinspector/primitive/LazyPrimitiveObjectInspectorFactory.java 5601734 serde/src/java/org/apache/hadoop/hive/serde2/lazy/objectinspector/primitive/LazyTimestampTZObjectInspector.java PRE-CREATION serde/src/java/org/apache/hadoop/hive/serde2/lazybinary/LazyBinaryFactory.java 52f3527 serde/src/java/org/apache/hadoop/hive/serde2/lazybinary/LazyBinarySerDe.java 56b4ca3 serde/src/java/org/apache/hadoop/hive/serde2/lazybinary/LazyBinaryTimestampTZ.java PRE-CREATION serde/src/java/org/apache/hadoop/hive/serde2/lazybinary/LazyBinaryUtils.java 8237b64 serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/ObjectInspectorConverters.java 24b3d4e serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/ObjectInspectorUtils.java ba44bae serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/PrimitiveObjectInspector.java 70633f3 serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/JavaTimestampTZObjectInspector.java PRE-CREATION serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/PrimitiveObjectInspectorConverter.java e08ad43 serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/PrimitiveObjectInspectorFactory.java 2ed0843 serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/PrimitiveObjectInspectorUtils.java 9642a7e serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/SettableTimestampTZObjectInspector.java PRE-CREATION serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/TimestampTZObjectInspector.java PRE-CREATION serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/WritableConstantTimestampTZObjectInspector.java PRE-CREATION serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/WritableTimestampTZObjectInspector.java PRE-CREATION serde/src/java/org/apache/hadoop/hive/serde2/thrift/Type.java 0ad8c02 serde/src/java/org/apache/hadoop/hive/serde2/typeinfo/TypeInfoFactory.java 43c4819 serde/src/test/org/apache/hadoop/hive/serde2/io/TestTimestampTZWritable.java PRE-CREATION service-rpc/if/TCLIService.thrift 824b049 service-rpc/src/gen/thrift/gen-cpp/TCLIService_constants.cpp 991cb2e service-rpc/src/gen/thrift/gen-cpp/TCLIService_types.h 8accf66 service-rpc/src/gen/thrift/gen-cpp/TCLIService_types.cpp b6995c4 service-rpc/src/gen/thrift/gen-javabean/org/apache/hive/service/rpc/thrift/TCLIServiceConstants.java 930bed7 service-rpc/src/gen/thrift/gen-javabean/org/apache/hive/service/rpc/thrift/TProtocolVersion.java 18a7825 service-rpc/src/gen/thrift/gen-javabean/org/apache/hive/service/rpc/thrift/TTypeId.java a3735eb service-rpc/src/gen/thrift/gen-php/Types.php ee5acd2 service-rpc/src/gen/thrift/gen-py/TCLIService/constants.py c8d4f8f service-rpc/src/gen/thrift/gen-py/TCLIService/ttypes.py e9faa2a service-rpc/src/gen/thrift/gen-rb/t_c_l_i_service_constants.rb 25adbb4 service-rpc/src/gen/thrift/gen-rb/t_c_l_i_service_types.rb 714309c service/src/java/org/apache/hive/service/cli/ColumnValue.java 76e8c03 service/src/java/org/apache/hive/service/cli/TypeDescriptor.java d634bef Diff: https://reviews.apache.org/r/50787/diff/7/ Changes: https://reviews.apache.org/r/50787/diff/6-7/ Testing --- Thanks, Rui Li
Re: [VOTE] Apache Hive 2.3.0 Release Candidate 0
The patch has been reverted in master and branch-2.3 On Wed, May 3, 2017 at 3:01 AM, Sergio Pena <sergio.p...@cloudera.com> wrote: > Hi Pengcheng, > > There is a request from the HDFS team to revert the patch committed on > HIVE-16047 from > our code because it might cause problems when future Hadoop versions are > released due to being a > private API on Hadoop. This API method signature has been changed between > releases, and > we don't want to have additional shims to support future Hadoop versions > just for this method. > > I'd like to revert it from 2.3.0 release before doing the release. It is > marked as being fixed on 2.2 but it is not cherry-picked on branch-2.2 but > branch-2.3. > > Do you agree? > > - Sergio > > On Fri, Apr 28, 2017 at 1:40 PM, Pengcheng Xiong <pxi...@apache.org> > wrote: > > > Withdraw the VOTE on candidate 0. Will propose candidate 1 soon. Thanks. > > > > On Thu, Apr 27, 2017 at 8:10 PM, Owen O'Malley <owen.omal...@gmail.com> > > wrote: > > > > > -1 you need a release of storage-API first. > > > > > > .. Owen > > > > > > > On Apr 27, 2017, at 17:43, Pengcheng Xiong <pxi...@apache.org> > wrote: > > > > > > > > Apache Hive 2.3.0 Release Candidate 0 is available here: > > > > http://home.apache.org/~pxiong/apache-hive-2.3.0-rc0/ > > > > > > > > > > > > Maven artifacts are available here: > > > > https://repository.apache.org/content/repositories/ > orgapachehive-1073/ > > > > > > > > > > > > Source tag for RC0 is at: > > > > > > > > https://github.com/apache/hive/releases/tag/release-2.3.0-rc0 > > > > > > > > Voting will conclude in 72 hours. > > > > > > > > Hive PMC Members: Please test and vote. > > > > > > > > Thanks. > > > > > > -- Best regards! Rui Li Cell: (+86) 13564950210
Re: Review Request 58865: HIVE-16552: Limit the number of tasks a Spark job may contain
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/58865/#review173689 --- ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkTask.java Lines 132 (patched) <https://reviews.apache.org/r/58865/#comment246728> I think the log is unnecessary because the failure should already be logged in the monitor ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkTask.java Lines 135 (patched) <https://reviews.apache.org/r/58865/#comment246729> Same as above. Can we consolidate the logs a bit? ql/src/java/org/apache/hadoop/hive/ql/exec/spark/status/RemoteSparkJobMonitor.java Lines 104 (patched) <https://reviews.apache.org/r/58865/#comment246731> Maybe I was being misleading. I mean we can compute the total task only once when the job first reaches RUNNING state, i.e. in the "if (!running)". At this point, the total count is determined and won't change. - Rui Li On May 2, 2017, 6:49 p.m., Xuefu Zhang wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/58865/ > --- > > (Updated May 2, 2017, 6:49 p.m.) > > > Review request for hive. > > > Bugs: HIVE-16552 > https://issues.apache.org/jira/browse/HIVE-16552 > > > Repository: hive-git > > > Description > --- > > See JIRA description > > > Diffs > - > > common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 84398c6 > ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkTask.java 32a7730 > > ql/src/java/org/apache/hadoop/hive/ql/exec/spark/status/RemoteSparkJobMonitor.java > dd73f3e > > ql/src/java/org/apache/hadoop/hive/ql/exec/spark/status/SparkJobMonitor.java > 0b224f2 > > > Diff: https://reviews.apache.org/r/58865/diff/3/ > > > Testing > --- > > Test locally > > > Thanks, > > Xuefu Zhang > >
Re: Review Request 58865: HIVE-16552: Limit the number of tasks a Spark job may contain
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/58865/#review173556 --- ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkTask.java Lines 135 (patched) <https://reviews.apache.org/r/58865/#comment246543> The log is incorrect because cancelling the job doesn't mean killing the application. ql/src/java/org/apache/hadoop/hive/ql/exec/spark/status/RemoteSparkJobMonitor.java Lines 106 (patched) <https://reviews.apache.org/r/58865/#comment246544> I think the total task count needs only be computed once. It shouldn't change during the execution of the job, assuming we don't count failed/retried tasks. - Rui Li On May 1, 2017, 5:13 p.m., Xuefu Zhang wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/58865/ > --- > > (Updated May 1, 2017, 5:13 p.m.) > > > Review request for hive. > > > Bugs: HIVE-16552 > https://issues.apache.org/jira/browse/HIVE-16552 > > > Repository: hive-git > > > Description > --- > > See JIRA description > > > Diffs > - > > common/src/java/org/apache/hadoop/hive/conf/HiveConf.java d3ea824 > ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkTask.java 32a7730 > > ql/src/java/org/apache/hadoop/hive/ql/exec/spark/status/RemoteSparkJobMonitor.java > dd73f3e > > ql/src/java/org/apache/hadoop/hive/ql/exec/spark/status/SparkJobMonitor.java > 0b224f2 > > > Diff: https://reviews.apache.org/r/58865/diff/2/ > > > Testing > --- > > Test locally > > > Thanks, > > Xuefu Zhang > >
Re: Review Request 50787: Add a timezone-aware timestamp
/org/apache/hadoop/hive/serde2/io/TimestampTZWritable.java PRE-CREATION serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazyFactory.java 23dbe6a serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazyTimestampTZ.java PRE-CREATION serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazyUtils.java 73c72e1 serde/src/java/org/apache/hadoop/hive/serde2/lazy/objectinspector/primitive/LazyPrimitiveObjectInspectorFactory.java 5601734 serde/src/java/org/apache/hadoop/hive/serde2/lazy/objectinspector/primitive/LazyTimestampTZObjectInspector.java PRE-CREATION serde/src/java/org/apache/hadoop/hive/serde2/lazybinary/LazyBinaryFactory.java 52f3527 serde/src/java/org/apache/hadoop/hive/serde2/lazybinary/LazyBinarySerDe.java 56b4ca3 serde/src/java/org/apache/hadoop/hive/serde2/lazybinary/LazyBinaryTimestampTZ.java PRE-CREATION serde/src/java/org/apache/hadoop/hive/serde2/lazybinary/LazyBinaryUtils.java 8237b64 serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/ObjectInspectorConverters.java 24b3d4e serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/ObjectInspectorUtils.java ba44bae serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/PrimitiveObjectInspector.java 70633f3 serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/JavaTimestampTZObjectInspector.java PRE-CREATION serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/PrimitiveObjectInspectorConverter.java e08ad43 serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/PrimitiveObjectInspectorFactory.java 2ed0843 serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/PrimitiveObjectInspectorUtils.java 9642a7e serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/SettableTimestampTZObjectInspector.java PRE-CREATION serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/TimestampTZObjectInspector.java PRE-CREATION serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/WritableConstantTimestampTZObjectInspector.java PRE-CREATION serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/WritableTimestampTZObjectInspector.java PRE-CREATION serde/src/java/org/apache/hadoop/hive/serde2/thrift/Type.java 0ad8c02 serde/src/java/org/apache/hadoop/hive/serde2/typeinfo/TypeInfoFactory.java 43c4819 serde/src/test/org/apache/hadoop/hive/serde2/io/TestTimestampTZWritable.java PRE-CREATION service-rpc/if/TCLIService.thrift 824b049 service-rpc/src/gen/thrift/gen-cpp/TCLIService_constants.cpp 991cb2e service-rpc/src/gen/thrift/gen-cpp/TCLIService_types.h 8accf66 service-rpc/src/gen/thrift/gen-cpp/TCLIService_types.cpp b6995c4 service-rpc/src/gen/thrift/gen-javabean/org/apache/hive/service/rpc/thrift/TCLIServiceConstants.java 930bed7 service-rpc/src/gen/thrift/gen-javabean/org/apache/hive/service/rpc/thrift/TProtocolVersion.java 18a7825 service-rpc/src/gen/thrift/gen-javabean/org/apache/hive/service/rpc/thrift/TTypeId.java a3735eb service-rpc/src/gen/thrift/gen-php/Types.php ee5acd2 service-rpc/src/gen/thrift/gen-py/TCLIService/constants.py c8d4f8f service-rpc/src/gen/thrift/gen-py/TCLIService/ttypes.py e9faa2a service-rpc/src/gen/thrift/gen-rb/t_c_l_i_service_constants.rb 25adbb4 service-rpc/src/gen/thrift/gen-rb/t_c_l_i_service_types.rb 714309c service/src/java/org/apache/hive/service/cli/ColumnValue.java 76e8c03 service/src/java/org/apache/hive/service/cli/TypeDescriptor.java d634bef Diff: https://reviews.apache.org/r/50787/diff/6/ Changes: https://reviews.apache.org/r/50787/diff/5-6/ Testing --- Thanks, Rui Li
[jira] [Created] (HIVE-16459) Cancel outstanding RPCs when channel closes
Rui Li created HIVE-16459: - Summary: Cancel outstanding RPCs when channel closes Key: HIVE-16459 URL: https://issues.apache.org/jira/browse/HIVE-16459 Project: Hive Issue Type: Bug Reporter: Rui Li Assignee: Rui Li -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (HIVE-16418) Allow HiveKey to skip some bytes for comparison
Rui Li created HIVE-16418: - Summary: Allow HiveKey to skip some bytes for comparison Key: HIVE-16418 URL: https://issues.apache.org/jira/browse/HIVE-16418 Project: Hive Issue Type: New Feature Reporter: Rui Li Assignee: Rui Li The feature is required when we have to serialize some fields and prevent them from being used in comparison, e.g. HIVE-14412. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (HIVE-16315) Describe table doesn't show num of partitions
Rui Li created HIVE-16315: - Summary: Describe table doesn't show num of partitions Key: HIVE-16315 URL: https://issues.apache.org/jira/browse/HIVE-16315 Project: Hive Issue Type: Bug Reporter: Rui Li Assignee: Rui Li This doesn't comply with our wiki: https://cwiki.apache.org/confluence/display/Hive/StatsDev#StatsDev-Examples -- This message was sent by Atlassian JIRA (v6.3.15#6346)
Re: [ANNOUNCE] New PMC Member : Eugene Koifman
Congratulations :)
On Thu, Mar 16, 2017 at 4:26 PM, Lefty Leverenz <leftylever...@gmail.com>
wrote:
> More congratulations!
>
> -- Lefty
>
> On Wed, Mar 15, 2017 at 1:27 PM, Eugene Koifman <ekoif...@hortonworks.com>
> wrote:
>
> > Thank you everyone!
> >
> > On 3/15/17, 12:21 PM, "Gunther Hagleitner" <ghagleit...@hortonworks.com>
> > wrote:
> >
> > Congratulations!
> >
> > From: Sergey Shelukhin <ser...@hortonworks.com>
> > Sent: Wednesday, March 15, 2017 11:18 AM
> > To: dev@hive.apache.org
> > Subject: Re: [ANNOUNCE] New PMC Member : Eugene Koifman
> >
> > Congrats!
> >
> > On 17/3/15, 01:02, "Zoltan Haindrich" <zhaindr...@hortonworks.com>
> > wrote:
> >
> > >Congrats Eugene!!
> > >
> > >On 15 Mar 2017 07:50, Peter Vary <pv...@cloudera.com> wrote:
> > >Congratulations! :)
> > >
> > >2017. márc. 15. 7:05 ezt írta ("Vaibhav Gumashta"
> > ><vgumas...@hortonworks.com
> > >>):
> > >
> > >> Congrats Eugene!
> > >>
> > >>
> > >> On 3/14/17, 11:03 PM, "Rajesh Balamohan" <rbalamo...@apache.org>
> > wrote:
> > >>
> > >> >Congrats Eugene!! :)
> > >> >
> > >> >~Rajesh.B
> > >> >
> > >> >On Wed, Mar 15, 2017 at 11:21 AM, Pengcheng Xiong <
> > pxi...@apache.org>
> > >> >wrote:
> > >> >
> > >> >> Congrats! Well deserved!
> > >> >>
> > >> >> Thanks.
> > >> >> Pengcheng
> > >> >>
> > >> >> On Tue, Mar 14, 2017 at 10:39 PM, Ashutosh Chauhan
> > >> >><hashut...@apache.org>
> > >> >> wrote:
> > >> >>
> > >> >> > On behalf of the Hive PMC I am delighted to announce Eugene
> > >>Koifman is
> > >> >> > joining Hive PMC.
> > >> >> > Eugene is a long time contributor in Hive and is focusing on
> > ACID
> > >> >>support
> > >> >> > areas these days.
> > >> >> >
> > >> >> > Welcome, Eugene!
> > >> >> >
> > >> >> > Thanks,
> > >> >> > Ashutosh
> > >> >> >
> > >> >>
> > >>
> > >>
> > >
> >
> >
> >
> >
> >
>
--
Best regards!
Rui Li
Cell: (+86) 13564950210
Re: Review Request 57586: HIVE-16183: Fix potential thread safety issues with static variables
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/57586/#review169094 --- Ship it! Ship It! - Rui Li On March 15, 2017, 10:03 p.m., Xuefu Zhang wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/57586/ > --- > > (Updated March 15, 2017, 10:03 p.m.) > > > Review request for hive. > > > Bugs: HIVE-16183 > https://issues.apache.org/jira/browse/HIVE-16183 > > > Repository: hive-git > > > Description > --- > > Please see JIRA description > > > Diffs > - > > beeline/src/java/org/apache/hive/beeline/BeeLineOpts.java 7e6846d > beeline/src/java/org/apache/hive/beeline/HiveSchemaHelper.java 181f0d2 > beeline/src/java/org/apache/hive/beeline/HiveSchemaTool.java 2c088c9 > cli/src/java/org/apache/hadoop/hive/cli/RCFileCat.java f1806a0 > cli/src/test/org/apache/hadoop/hive/cli/TestRCFileCat.java 11ceb31 > common/src/java/org/apache/hadoop/hive/common/LogUtils.java c2a0d9a > common/src/java/org/apache/hadoop/hive/common/StatsSetupConst.java 926b4a6 > > metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreSchemaInfo.java > 9c30ee7 > ql/src/java/org/apache/hadoop/hive/ql/exec/ArchiveUtils.java 6381a21 > ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java 4ac25c2 > ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java 6693134 > ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorizationContext.java > 5b0c2bf > > ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/CuckooSetBytes.java > 6383e8a > > ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/fast/VectorMapJoinFastHashTable.java > 9030e5f > ql/src/java/org/apache/hadoop/hive/ql/history/HiveHistoryImpl.java 6582cdd > ql/src/java/org/apache/hadoop/hive/ql/index/HiveIndex.java a1408e9 > ql/src/java/org/apache/hadoop/hive/ql/io/HiveFileFormatUtils.java 7727114 > ql/src/java/org/apache/hadoop/hive/ql/io/HiveInputFormat.java 4995bdf > ql/src/java/org/apache/hadoop/hive/ql/io/RCFile.java d391164 > ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcInputFormat.java 59682db > ql/src/java/org/apache/hadoop/hive/ql/io/rcfile/stats/PartialScanTask.java > 90b1dff > ql/src/java/org/apache/hadoop/hive/ql/metadata/VirtualColumn.java 044d64c > ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java 0e67ea6 > > ql/src/java/org/apache/hadoop/hive/ql/optimizer/listbucketingpruner/ListBucketingPrunerUtils.java > 4d3e74e > > ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/GenMRSkewJoinProcessor.java > 93202c3 > ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/Vectorizer.java > 50eda15 > > ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/VectorizerReason.java > e0a6198 > ql/src/java/org/apache/hadoop/hive/ql/parse/BaseSemanticAnalyzer.java > f762fee > ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java > f175663 > ql/src/java/org/apache/hadoop/hive/ql/parse/WindowingSpec.java 01b5559 > ql/src/java/org/apache/hadoop/hive/ql/plan/AbstractVectorDesc.java e85a418 > ql/src/java/org/apache/hadoop/hive/ql/plan/GroupByDesc.java 0b49294 > ql/src/java/org/apache/hadoop/hive/ql/plan/MapJoinDesc.java ca69697 > ql/src/java/org/apache/hadoop/hive/ql/plan/ReduceSinkDesc.java 9ae30ab > ql/src/java/org/apache/hadoop/hive/ql/plan/VectorAppMasterEventDesc.java > 2e11321 > ql/src/java/org/apache/hadoop/hive/ql/plan/VectorFileSinkDesc.java 325ac91 > ql/src/java/org/apache/hadoop/hive/ql/plan/VectorFilterDesc.java 6feed84 > ql/src/java/org/apache/hadoop/hive/ql/plan/VectorGroupByDesc.java f8554e2 > ql/src/java/org/apache/hadoop/hive/ql/plan/VectorLimitDesc.java c9bc45a > ql/src/java/org/apache/hadoop/hive/ql/plan/VectorMapJoinDesc.java 3aa65d3 > ql/src/java/org/apache/hadoop/hive/ql/plan/VectorMapJoinInfo.java 9429785 > ql/src/java/org/apache/hadoop/hive/ql/plan/VectorPartitionDesc.java 4078c7d > ql/src/java/org/apache/hadoop/hive/ql/plan/VectorReduceSinkDesc.java > 2eb44b8 > ql/src/java/org/apache/hadoop/hive/ql/plan/VectorReduceSinkInfo.java > 8c35415 > ql/src/java/org/apache/hadoop/hive/ql/plan/VectorSMBJoinDesc.java 031f11e > ql/src/java/org/apache/hadoop/hive/ql/plan/VectorSelectDesc.java c2c9450 > > ql/src/java/org/apache/hadoop/hive/ql/plan/VectorSparkHashTableSinkDesc.java > 7fb59db > > ql/src/java/org
Re: Review Request 57586: HIVE-16183: Fix potential thread safety issues with static variables
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/57586/#review168965 --- metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreSchemaInfo.java Line 58 (original), 56 (patched) <https://reviews.apache.org/r/57586/#comment241305> shall we remove the hiveConf parameter as it's not needed? ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/Vectorizer.java Line 502 (original), 480 (patched) <https://reviews.apache.org/r/57586/#comment241306> do we still need this method? ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/Vectorizer.java Line 545 (original), 520 (patched) <https://reviews.apache.org/r/57586/#comment241307> same as above - Rui Li On March 14, 2017, 4:32 a.m., Xuefu Zhang wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/57586/ > --- > > (Updated March 14, 2017, 4:32 a.m.) > > > Review request for hive. > > > Bugs: HIVE-16183 > https://issues.apache.org/jira/browse/HIVE-16183 > > > Repository: hive-git > > > Description > --- > > Please see JIRA description > > > Diffs > - > > beeline/src/java/org/apache/hive/beeline/BeeLineOpts.java 7e6846d > beeline/src/java/org/apache/hive/beeline/HiveSchemaHelper.java 181f0d2 > cli/src/java/org/apache/hadoop/hive/cli/RCFileCat.java f1806a0 > cli/src/test/org/apache/hadoop/hive/cli/TestRCFileCat.java 11ceb31 > common/src/java/org/apache/hadoop/hive/common/LogUtils.java c2a0d9a > common/src/java/org/apache/hadoop/hive/common/StatsSetupConst.java 926b4a6 > > metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreSchemaInfo.java > 9c30ee7 > ql/src/java/org/apache/hadoop/hive/ql/exec/ArchiveUtils.java 6381a21 > ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java 4ac25c2 > ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java 6693134 > ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorizationContext.java > 5b0c2bf > > ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/CuckooSetBytes.java > 6383e8a > > ql/src/java/org/apache/hadoop/hive/ql/exec/vector/mapjoin/fast/VectorMapJoinFastHashTable.java > 9030e5f > ql/src/java/org/apache/hadoop/hive/ql/history/HiveHistoryImpl.java 6582cdd > ql/src/java/org/apache/hadoop/hive/ql/index/HiveIndex.java a1408e9 > ql/src/java/org/apache/hadoop/hive/ql/io/HiveFileFormatUtils.java 7727114 > ql/src/java/org/apache/hadoop/hive/ql/io/HiveInputFormat.java 4995bdf > ql/src/java/org/apache/hadoop/hive/ql/io/RCFile.java d391164 > ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcInputFormat.java 369584b > ql/src/java/org/apache/hadoop/hive/ql/io/rcfile/stats/PartialScanTask.java > 90b1dff > ql/src/java/org/apache/hadoop/hive/ql/metadata/VirtualColumn.java 044d64c > ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java 0e67ea6 > > ql/src/java/org/apache/hadoop/hive/ql/optimizer/listbucketingpruner/ListBucketingPrunerUtils.java > 4d3e74e > > ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/GenMRSkewJoinProcessor.java > 93202c3 > ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/Vectorizer.java > 50eda15 > > ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/VectorizerReason.java > e0a6198 > ql/src/java/org/apache/hadoop/hive/ql/parse/BaseSemanticAnalyzer.java > 36009bf > ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java > f175663 > ql/src/java/org/apache/hadoop/hive/ql/parse/WindowingSpec.java 01b5559 > ql/src/java/org/apache/hadoop/hive/ql/plan/AbstractVectorDesc.java e85a418 > ql/src/java/org/apache/hadoop/hive/ql/plan/GroupByDesc.java 0b49294 > ql/src/java/org/apache/hadoop/hive/ql/plan/MapJoinDesc.java ca69697 > ql/src/java/org/apache/hadoop/hive/ql/plan/ReduceSinkDesc.java 9ae30ab > ql/src/java/org/apache/hadoop/hive/ql/plan/VectorAppMasterEventDesc.java > 2e11321 > ql/src/java/org/apache/hadoop/hive/ql/plan/VectorFileSinkDesc.java 325ac91 > ql/src/java/org/apache/hadoop/hive/ql/plan/VectorFilterDesc.java 6feed84 > ql/src/java/org/apache/hadoop/hive/ql/plan/VectorGroupByDesc.java f8554e2 > ql/src/java/org/apache/hadoop/hive/ql/plan/VectorLimitDesc.java c9bc45a > ql/src/java/org/apache/hadoop/hive/ql/plan/VectorMapJoinDesc.java 3aa65d3 > ql/src/java/org/apache/hadoop/hive/ql/plan/VectorMapJoinInfo.java 9429785 > ql/src/java/or
[jira] [Created] (HIVE-16155) No need for ConditionalTask if no conditional map join is created
Rui Li created HIVE-16155: - Summary: No need for ConditionalTask if no conditional map join is created Key: HIVE-16155 URL: https://issues.apache.org/jira/browse/HIVE-16155 Project: Hive Issue Type: Bug Reporter: Rui Li Assignee: Rui Li Priority: Minor -- This message was sent by Atlassian JIRA (v6.3.15#6346)
Re: Review Request 56687: Intern strings in various critical places to reduce memory consumption.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/56687/#review166991 --- ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java (line 3178) <https://reviews.apache.org/r/56687/#comment239095> do we still need this? I think createEmptyFile will intern the strings for us? ql/src/java/org/apache/hadoop/hive/ql/plan/MapWork.java (line 173) <https://reviews.apache.org/r/56687/#comment239097> instead of creating a new map, can we use the pathToAliases map and intern the paths in-place? - Rui Li On Feb. 27, 2017, 7:42 p.m., Misha Dmitriev wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/56687/ > --- > > (Updated Feb. 27, 2017, 7:42 p.m.) > > > Review request for hive, Chaoyu Tang, Mohit Sabharwal, and Sergio Pena. > > > Bugs: https://issues.apache.org/jira/browse/HIVE-15882 > > https://issues.apache.org/jira/browse/https://issues.apache.org/jira/browse/HIVE-15882 > > > Repository: hive-git > > > Description > --- > > See the description of the problem in > https://issues.apache.org/jira/browse/HIVE-15882 Interning strings per this > review removes most of the overhead due to duplicate strings. > > Also, where maps in several places are created from other maps, use the > original map's size for the new map. This is to avoid the situation when a > map with default capacity (typically 16) is created to hold just 2-3 entries, > and the rest of the internal 16-entry array is wasted. > > > Diffs > - > > common/src/java/org/apache/hadoop/hive/common/StringInternUtils.java > PRE-CREATION > ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java > e81cbce3e333d44a4088c10491f399e92a505293 > ql/src/java/org/apache/hadoop/hive/ql/hooks/Entity.java > 08420664d59f28f75872c25c9f8ee42577b23451 > ql/src/java/org/apache/hadoop/hive/ql/io/CombineHiveInputFormat.java > e91064b9c75e8adb2b36f21ff19ec0c1539b03b9 > ql/src/java/org/apache/hadoop/hive/ql/io/HiveInputFormat.java > 51530ac16c92cc75d501bfcb573557754ba0c964 > ql/src/java/org/apache/hadoop/hive/ql/io/SymbolicInputFormat.java > 55b3b551a1dac92583b6e03b10beb8172ca93d45 > ql/src/java/org/apache/hadoop/hive/ql/lockmgr/HiveLockObject.java > 82dc89803be9cf9e0018720eeceb90ff450bfdc8 > ql/src/java/org/apache/hadoop/hive/ql/metadata/Partition.java > c0edde9e92314d86482b5c46178987e79fae57fe > ql/src/java/org/apache/hadoop/hive/ql/metadata/Table.java > c6ae6f290857cfd10f1023058ede99bf4a10f057 > ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java > 24d16812515bdfa90b4be7a295c0388fcdfe95ef > > ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/GenMRSkewJoinProcessor.java > ede4fcbe342052ad86dadebcc49da2c0f515ea98 > > ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/NullScanTaskDispatcher.java > 0882ae2c6205b1636cbc92e76ef66bb70faadc76 > > ql/src/java/org/apache/hadoop/hive/ql/plan/ConditionalResolverMergeFiles.java > 68b0ad9ea63f051f16fec3652d8525f7ab07eb3f > ql/src/java/org/apache/hadoop/hive/ql/plan/MapWork.java > d4bdd96eaf8d179bed43b8a8c3be0d338940154a > ql/src/java/org/apache/hadoop/hive/ql/plan/MsckDesc.java > b7a7e4b7a5f8941b080c7805d224d3885885f444 > ql/src/java/org/apache/hadoop/hive/ql/plan/PartitionDesc.java > 73981e826870139a42ad881103fdb0a2ef8433a2 > > Diff: https://reviews.apache.org/r/56687/diff/ > > > Testing > --- > > I've measured how much memory this change plus another one (interning > Properties in PartitionDesc) save in my HS2 benchmark - the result is 37%. > See the details in HIVE-15882. > > > Thanks, > > Misha Dmitriev > >
Re: Review Request 56687: Intern strings in various critical places to reduce memory consumption.
> On Feb. 24, 2017, 7:38 a.m., Rui Li wrote: > > ql/src/java/org/apache/hadoop/hive/ql/plan/ConditionalResolverMergeFiles.java, > > line 322 > > <https://reviews.apache.org/r/56687/diff/2/?file=1643011#file1643011line322> > > > > will this cause the hash map to resize since the default load factor is > > 0.75? and several similar concerns below > > Misha Dmitriev wrote: > You are probably right, in that this constructor's parameter is the > initial capacity of this table (more or less the size of the internal array) > - not how many elements the table is expected to hold. However, if you check > the code of HashMap, the things are more interesting. The actual capacity of > the table is always a power of two, so unless this parameter is also a power > of two, the capacity will be chosen as the nearest higher power of two, i.e. > it will be higher than the parameter and closer to what we actually need. > Also, if we create a table with the default size (16) here and then will put > many more elements into it, it will be resized several times, whereas with > the current code it will be resized at most once. Trying to "factor in" the > load factor will likely add more confusion/complexity. All in all, given that > choosing capacity in HashMap internally is non-trivial, I think it's > easier/safer to just call 'new HashMap(oldMap.size())' as we do now. Then could you explain why we need to change the current code? The JavaDoc of LinkedHashMap(Map m) indicates it will create an instance "with a default load factor (0.75) and an initial capacity sufficient to hold the mappings in the specified map". Looking at the code, it computes the initial cap like "m.size()/loadFactor + 1", rounds it to next power of two, and it avoids re-hashing. Won't that be good enough for us? - Rui --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/56687/#review166649 --- On Feb. 24, 2017, 9:27 p.m., Misha Dmitriev wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/56687/ > --- > > (Updated Feb. 24, 2017, 9:27 p.m.) > > > Review request for hive, Chaoyu Tang, Mohit Sabharwal, and Sergio Pena. > > > Bugs: https://issues.apache.org/jira/browse/HIVE-15882 > > https://issues.apache.org/jira/browse/https://issues.apache.org/jira/browse/HIVE-15882 > > > Repository: hive-git > > > Description > --- > > See the description of the problem in > https://issues.apache.org/jira/browse/HIVE-15882 Interning strings per this > review removes most of the overhead due to duplicate strings. > > Also, where maps in several places are created from other maps, use the > original map's size for the new map. This is to avoid the situation when a > map with default capacity (typically 16) is created to hold just 2-3 entries, > and the rest of the internal 16-entry array is wasted. > > > Diffs > - > > common/src/java/org/apache/hadoop/hive/common/StringInternUtils.java > PRE-CREATION > ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java > e81cbce3e333d44a4088c10491f399e92a505293 > ql/src/java/org/apache/hadoop/hive/ql/hooks/Entity.java > 08420664d59f28f75872c25c9f8ee42577b23451 > ql/src/java/org/apache/hadoop/hive/ql/io/CombineHiveInputFormat.java > e91064b9c75e8adb2b36f21ff19ec0c1539b03b9 > ql/src/java/org/apache/hadoop/hive/ql/io/HiveInputFormat.java > 51530ac16c92cc75d501bfcb573557754ba0c964 > ql/src/java/org/apache/hadoop/hive/ql/io/SymbolicInputFormat.java > 55b3b551a1dac92583b6e03b10beb8172ca93d45 > ql/src/java/org/apache/hadoop/hive/ql/lockmgr/HiveLockObject.java > 82dc89803be9cf9e0018720eeceb90ff450bfdc8 > ql/src/java/org/apache/hadoop/hive/ql/metadata/Partition.java > c0edde9e92314d86482b5c46178987e79fae57fe > ql/src/java/org/apache/hadoop/hive/ql/metadata/Table.java > c6ae6f290857cfd10f1023058ede99bf4a10f057 > ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java > 24d16812515bdfa90b4be7a295c0388fcdfe95ef > > ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/GenMRSkewJoinProcessor.java > ede4fcbe342052ad86dadebcc49da2c0f515ea98 > > ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/NullScanTaskDispatcher.java > 0882ae2c6205b1636cbc92e76ef66bb70faadc76 > > ql/src/java/org/apache/hadoop/hive/ql/plan/ConditionalResolverMergeFiles.java > 68b
[jira] [Created] (HIVE-16047) Shouldn't try to get KeyProvider unless encryption is enabled
Rui Li created HIVE-16047:
-
Summary: Shouldn't try to get KeyProvider unless encryption is
enabled
Key: HIVE-16047
URL: https://issues.apache.org/jira/browse/HIVE-16047
Project: Hive
Issue Type: Bug
Reporter: Rui Li
Assignee: Rui Li
Priority: Minor
Found lots of following errors in HS2 log:
{noformat}
hdfs.KeyProviderCache: Could not find uri with key
[dfs.encryption.key.provider.uri] to create a keyProvider !!
{noformat}
Similar to HDFS-7931
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
Re: Review Request 56687: Intern strings in various critical places to reduce memory consumption.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/56687/#review166649 --- common/src/java/org/apache/hadoop/hive/common/StringInternUtils.java (line 65) <https://reviews.apache.org/r/56687/#comment238671> do we need to check whether uri is null? common/src/java/org/apache/hadoop/hive/common/StringInternUtils.java (line 67) <https://reviews.apache.org/r/56687/#comment238672> why the stringField doesn't need the null check like other fields? ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java (line 3147) <https://reviews.apache.org/r/56687/#comment238673> How about intern the path in the createEmptyFile method? ql/src/java/org/apache/hadoop/hive/ql/lockmgr/HiveLockObject.java (line 183) <https://reviews.apache.org/r/56687/#comment238676> can we call the util method? ql/src/java/org/apache/hadoop/hive/ql/lockmgr/HiveLockObject.java (line 188) <https://reviews.apache.org/r/56687/#comment238677> guess we can also add a util method for this ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java (line 253) <https://reviews.apache.org/r/56687/#comment238681> since we'll intern strings in the new path, do we have to intern taskTmpDir here? ql/src/java/org/apache/hadoop/hive/ql/plan/ConditionalResolverMergeFiles.java (line 322) <https://reviews.apache.org/r/56687/#comment238682> will this cause the hash map to resize since the default load factor is 0.75? and several similar concerns below - Rui Li On Feb. 23, 2017, 9:01 p.m., Misha Dmitriev wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/56687/ > --- > > (Updated Feb. 23, 2017, 9:01 p.m.) > > > Review request for hive, Chaoyu Tang, Mohit Sabharwal, and Sergio Pena. > > > Bugs: https://issues.apache.org/jira/browse/HIVE-15882 > > https://issues.apache.org/jira/browse/https://issues.apache.org/jira/browse/HIVE-15882 > > > Repository: hive-git > > > Description > --- > > See the description of the problem in > https://issues.apache.org/jira/browse/HIVE-15882 Interning strings per this > review removes most of the overhead due to duplicate strings. > > Also, where maps in several places are created from other maps, use the > original map's size for the new map. This is to avoid the situation when a > map with default capacity (typically 16) is created to hold just 2-3 entries, > and the rest of the internal 16-entry array is wasted. > > > Diffs > - > > common/src/java/org/apache/hadoop/hive/common/StringInternUtils.java > PRE-CREATION > ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java > e81cbce3e333d44a4088c10491f399e92a505293 > ql/src/java/org/apache/hadoop/hive/ql/hooks/Entity.java > 08420664d59f28f75872c25c9f8ee42577b23451 > ql/src/java/org/apache/hadoop/hive/ql/io/CombineHiveInputFormat.java > e91064b9c75e8adb2b36f21ff19ec0c1539b03b9 > ql/src/java/org/apache/hadoop/hive/ql/io/HiveInputFormat.java > 51530ac16c92cc75d501bfcb573557754ba0c964 > ql/src/java/org/apache/hadoop/hive/ql/io/SymbolicInputFormat.java > 55b3b551a1dac92583b6e03b10beb8172ca93d45 > ql/src/java/org/apache/hadoop/hive/ql/lockmgr/HiveLockObject.java > 82dc89803be9cf9e0018720eeceb90ff450bfdc8 > ql/src/java/org/apache/hadoop/hive/ql/metadata/Partition.java > c0edde9e92314d86482b5c46178987e79fae57fe > ql/src/java/org/apache/hadoop/hive/ql/metadata/Table.java > c6ae6f290857cfd10f1023058ede99bf4a10f057 > ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java > 24d16812515bdfa90b4be7a295c0388fcdfe95ef > > ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/GenMRSkewJoinProcessor.java > ede4fcbe342052ad86dadebcc49da2c0f515ea98 > > ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/NullScanTaskDispatcher.java > 0882ae2c6205b1636cbc92e76ef66bb70faadc76 > > ql/src/java/org/apache/hadoop/hive/ql/plan/ConditionalResolverMergeFiles.java > 68b0ad9ea63f051f16fec3652d8525f7ab07eb3f > ql/src/java/org/apache/hadoop/hive/ql/plan/MapWork.java > d4bdd96eaf8d179bed43b8a8c3be0d338940154a > ql/src/java/org/apache/hadoop/hive/ql/plan/MsckDesc.java > b7a7e4b7a5f8941b080c7805d224d3885885f444 > ql/src/java/org/apache/hadoop/hive/ql/plan/PartitionDesc.java > 73981e826870139a42ad881103fdb0a2ef8433a2 > > Diff: https://reviews.apache.org/r/56687/diff/ > > > Testing > --- > > I've measured how much memory this change plus another one (interning > Properties in PartitionDesc) save in my HS2 benchmark - the result is 37%. > See the details in HIVE-15882. > > > Thanks, > > Misha Dmitriev > >
Re: [ANNOUNCE] New committer: Zoltan Haindrich
Congratulations! On Wed, Feb 22, 2017 at 8:47 AM, Sergey Shelukhin <ser...@hortonworks.com> wrote: > Congratulations! > > On 17/2/21, 16:43, "Prasanth Jayachandran" <pjayachand...@hortonworks.com> > wrote: > > >Congratulations Zoltan!! > > > >Thanks > >Prasanth > > > > > > > > > >On Tue, Feb 21, 2017 at 4:35 PM -0800, "Eugene Koifman" > ><ekoif...@hortonworks.com<mailto:ekoif...@hortonworks.com>> wrote: > > > > > >Congratulations! > > > >On 2/21/17, 4:17 PM, "Vihang Karajgaonkar" wrote: > > > >Congrats Zoltan! > > > >On Tue, Feb 21, 2017 at 4:16 PM, Vaibhav Gumashta wrote: > > > >> Congrats Zoltan! > >> > >> On 2/21/17, 4:16 PM, "Jimmy Xiang" wrote: > >> > >> >Congrats, Zoltan!! > >> > > >> >On Tue, Feb 21, 2017 at 4:15 PM, Sushanth Sowmyan > >> >wrote: > >> >> Congrats, Zoltan! > >> >> > >> >> Welcome aboard. :) > >> >> > >> >> On Feb 21, 2017 15:42, "Rajesh Balamohan" > >> wrote: > >> >> > >> >>> Congrats Zoltan. :) > >> >>> > >> >>> ~Rajesh.B > >> >>> > >> >>> On Wed, Feb 22, 2017 at 4:43 AM, Wei Zheng > >> >>>wrote: > >> >>> > >> >>> > Congrats Zoltan! > >> >>> > > >> >>> > Thanks, > >> >>> > Wei > >> >>> > > >> >>> > On 2/21/17, 13:09, "Alan Gates" wrote: > >> >>> > > >> >>> > On behalf of the Hive PMC I am happy to announce Zoltan > >> >>>Haindrich is > >> >>> > our newest committer. He has been contributing to Hive for > >several > >> >>> months > >> >>> > across a number of areas, including the parser, HiveServer2, > >and > >> >>>cleaning > >> >>> > up unit tests and documentation. Please join me in welcoming > >Zoltan > >> >>>to > >> >>> > Hive. > >> >>> > > >> >>> > Zoltan, feel free to say a few words introducing yourself > >if you > >> >>> would > >> >>> > like to. > >> >>> > > >> >>> > Alan. > >> >>> > > >> >>> > > >> >>> > > >> >>> > >> > > >> > >> > > > > > > > > -- Best regards! Rui Li Cell: (+86) 13564950210
[jira] [Created] (HIVE-15860) RemoteSparkJobMonitor may hang when RemoteDriver exists abnormally
Rui Li created HIVE-15860: - Summary: RemoteSparkJobMonitor may hang when RemoteDriver exists abnormally Key: HIVE-15860 URL: https://issues.apache.org/jira/browse/HIVE-15860 Project: Hive Issue Type: Bug Reporter: Rui Li Assignee: Rui Li -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (HIVE-15526) Some tests need SORT_QUERY_RESULTS
Rui Li created HIVE-15526: - Summary: Some tests need SORT_QUERY_RESULTS Key: HIVE-15526 URL: https://issues.apache.org/jira/browse/HIVE-15526 Project: Hive Issue Type: Test Reporter: Rui Li Assignee: Rui Li Priority: Minor -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: Invitation for Hive committers to become ORC committers
I'm interested. Thanks! On Fri, Dec 16, 2016 at 1:18 PM, Chinna Rao Lalam < lalamchinnara...@gmail.com> wrote: > I would be interested. Thanks. > > Chinna Rao Lalam > > On Fri, Dec 16, 2016 at 6:43 AM, Owen O'Malley <omal...@apache.org> wrote: > > > Ok, I've added the people who have responded so far and updated the ORC > > website. > > > > http://orc.apache.org/news/2016/12/15/new-committers/ > > http://orc.apache.org/develop/ > > > > Please make sure that I didn't typo your names. > > > > .. Owen > > > > On Thu, Dec 15, 2016 at 4:44 PM, Chaoyu Tang <ctang...@gmail.com> wrote: > > > > > I am interested in. Thanks > > > > > > Chaoyu > > > > > > On Thu, Dec 15, 2016 at 5:13 PM, Rajesh Balamohan < > rbalamo...@apache.org > > > > > > wrote: > > > > > > > I would be interested. Thanks. > > > > > > > > ~Rajesh.B > > > > > > > > On Fri, Dec 16, 2016 at 3:31 AM, Mithun Radhakrishnan < > > > > mithun.radhakrish...@yahoo.com.invalid> wrote: > > > > > > > > > I'd be keen. > > > > > Thanks,Mithun > > > > > On Thursday, December 15, 2016, 1:37:36 PM PST, Wei Zheng < > > > > > wzh...@hortonworks.com> wrote:I’m interested. Thanks. > > > > > > > > > > Thanks, > > > > > Wei > > > > > > > > > > On 12/15/16, 13:21, "Vaibhav Gumashta" <vgumas...@hortonworks.com> > > > > wrote: > > > > > > > > > > I¹d be interested. > > > > > > > > > > Thanks, > > > > > ‹Vaibhav > > > > > > > > > > On 12/15/16, 1:12 PM, "Owen O'Malley" <omal...@apache.org> > > wrote: > > > > > > > > > > >All, > > > > > > As you are aware, we are in the last stages of removing the > > > forked > > > > > ORC > > > > > >code out of Hive. The goal of moving ORC out of Hive was to > > > increase > > > > > its > > > > > >community and we want to be very deliberately inclusive of the > > > Hive > > > > > >development community. Towards that end, the ORC PMC wants to > > > > welcome > > > > > >anyone who is already a Hive committer to become a committer > on > > > ORC. > > > > > > > > > > > > Please respond on this thread to let us know if you are > > > > interested. > > > > > > > > > > > >Thanks, > > > > > > Owen on behalf of the ORC PMC > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > Hope It Helps, > Chinna > -- Best regards! Rui Li Cell: (+86) 13564950210
Re: [ANNOUNCE] New Hive Committer - Rajesh Balamohan
Congratulations :) On Thu, Dec 15, 2016 at 6:50 AM, Gunther Hagleitner < ghagleit...@hortonworks.com> wrote: > Congrats Rajesh! > > From: Jimmy Xiang <jxc...@gmail.com> > Sent: Wednesday, December 14, 2016 11:38 AM > To: u...@hive.apache.org > Cc: dev@hive.apache.org; rbalamo...@apache.org > Subject: Re: [ANNOUNCE] New Hive Committer - Rajesh Balamohan > > Congrats, Rajesh!! > > On Wed, Dec 14, 2016 at 11:32 AM, Sergey Shelukhin > <ser...@hortonworks.com> wrote: > > Congratulations! > > > > From: Chao Sun <sunc...@apache.org> > > Reply-To: "u...@hive.apache.org" <u...@hive.apache.org> > > Date: Wednesday, December 14, 2016 at 10:52 > > To: "dev@hive.apache.org" <dev@hive.apache.org> > > Cc: "u...@hive.apache.org" <u...@hive.apache.org>, " > rbalamo...@apache.org" > > <rbalamo...@apache.org> > > Subject: Re: [ANNOUNCE] New Hive Committer - Rajesh Balamohan > > > > Congrats Rajesh! > > > > On Wed, Dec 14, 2016 at 9:26 AM, Vihang Karajgaonkar < > vih...@cloudera.com> > > wrote: > >> > >> Congrats Rajesh! > >> > >> On Wed, Dec 14, 2016 at 1:54 AM, Jesus Camacho Rodriguez < > >> jcamachorodrig...@hortonworks.com> wrote: > >> > >> > Congrats Rajesh, well deserved! :) > >> > > >> > -- > >> > Jesús > >> > > >> > > >> > > >> > > >> > On 12/14/16, 8:41 AM, "Lefty Leverenz" <leftylever...@gmail.com> > wrote: > >> > > >> > >Congratulations Rajesh! > >> > > > >> > >-- Lefty > >> > > > >> > > > >> > >On Tue, Dec 13, 2016 at 11:58 PM, Rajesh Balamohan > >> > > <rbalamo...@apache.org > >> > > > >> > >wrote: > >> > > > >> > >> Thanks a lot for providing this opportunity and to all for their > >> > messages. > >> > >> :) > >> > >> > >> > >> ~Rajesh.B > >> > >> > >> > >> On Wed, Dec 14, 2016 at 11:33 AM, Dharmesh Kakadia > >> > >> <dhkaka...@gmail.com > >> > > > >> > >> wrote: > >> > >> > >> > >> > Congrats Rajesh ! > >> > >> > > >> > >> > Thanks, > >> > >> > Dharmesh > >> > >> > > >> > >> > On Tue, Dec 13, 2016 at 7:37 PM, Vikram Dixit K < > >> > vikram.di...@gmail.com> > >> > >> > wrote: > >> > >> > > >> > >> >> Congrats Rajesh! :) > >> > >> >> > >> > >> >> On Tue, Dec 13, 2016 at 9:36 PM, Pengcheng Xiong > >> > >> >> <pxi...@apache.org> > >> > >> >> wrote: > >> > >> >> > >> > >> >>> Congrats Rajesh! :) > >> > >> >>> > >> > >> >>> On Tue, Dec 13, 2016 at 6:51 PM, Prasanth Jayachandran < > >> > >> >>> prasan...@apache.org > >> > >> >>> > wrote: > >> > >> >>> > >> > >> >>> > The Apache Hive PMC has voted to make Rajesh Balamohan a > >> > committer on > >> > >> >>> the > >> > >> >>> > Apache Hive Project. Please join me in congratulating Rajesh. > >> > >> >>> > > >> > >> >>> > Congratulations Rajesh! > >> > >> >>> > > >> > >> >>> > Thanks > >> > >> >>> > Prasanth > >> > >> >>> > >> > >> >> > >> > >> >> > >> > >> >> > >> > >> >> -- > >> > >> >> Nothing better than when appreciated for hard work. > >> > >> >> -Mark > >> > >> >> > >> > >> > > >> > >> > > >> > >> > >> > > > > > > > -- Best regards! Rui Li Cell: (+86) 13564950210
[jira] [Created] (HIVE-15428) HoS DPP doesn't remove cyclic dependency
Rui Li created HIVE-15428: - Summary: HoS DPP doesn't remove cyclic dependency Key: HIVE-15428 URL: https://issues.apache.org/jira/browse/HIVE-15428 Project: Hive Issue Type: Bug Reporter: Rui Li Assignee: Rui Li -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-15357) Fix and re-enable the spark-only tests
Rui Li created HIVE-15357: - Summary: Fix and re-enable the spark-only tests Key: HIVE-15357 URL: https://issues.apache.org/jira/browse/HIVE-15357 Project: Hive Issue Type: Test Reporter: Rui Li Assignee: Rui Li -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-15302) Relax the requirement that HoS needs Spark built w/o Hive
Rui Li created HIVE-15302: - Summary: Relax the requirement that HoS needs Spark built w/o Hive Key: HIVE-15302 URL: https://issues.apache.org/jira/browse/HIVE-15302 Project: Hive Issue Type: Improvement Reporter: Rui Li Assignee: Rui Li This requirement becomes more and more unacceptable as SparkSQL becomes widely adopted. Let's use this JIRA to find out how we can relax the limitation. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-15299) Yarn-cluster and yarn-client deprecated in Spark 2.0
Rui Li created HIVE-15299: - Summary: Yarn-cluster and yarn-client deprecated in Spark 2.0 Key: HIVE-15299 URL: https://issues.apache.org/jira/browse/HIVE-15299 Project: Hive Issue Type: Bug Reporter: Rui Li Assignee: Rui Li Priority: Minor Need to use master "yarn" with specified deploy mode instead. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-15202) Concurrent compactions for the same partition may generate malformed folder structure
Rui Li created HIVE-15202:
-
Summary: Concurrent compactions for the same partition may
generate malformed folder structure
Key: HIVE-15202
URL: https://issues.apache.org/jira/browse/HIVE-15202
Project: Hive
Issue Type: Bug
Reporter: Rui Li
If two compactions run concurrently on a single partition, it may generate
folder structure like this: (nested base dir)
{noformat}
drwxr-xr-x - root supergroup 0 2016-11-14 22:23
/user/hive/warehouse/test/z=1/base_007/base_007
-rw-r--r-- 3 root supergroup201 2016-11-14 21:46
/user/hive/warehouse/test/z=1/base_007/bucket_0
-rw-r--r-- 3 root supergroup611 2016-11-14 21:46
/user/hive/warehouse/test/z=1/base_007/bucket_1
-rw-r--r-- 3 root supergroup614 2016-11-14 21:46
/user/hive/warehouse/test/z=1/base_007/bucket_2
-rw-r--r-- 3 root supergroup621 2016-11-14 21:46
/user/hive/warehouse/test/z=1/base_007/bucket_3
-rw-r--r-- 3 root supergroup621 2016-11-14 21:46
/user/hive/warehouse/test/z=1/base_007/bucket_4
-rw-r--r-- 3 root supergroup201 2016-11-14 21:46
/user/hive/warehouse/test/z=1/base_007/bucket_5
-rw-r--r-- 3 root supergroup201 2016-11-14 21:46
/user/hive/warehouse/test/z=1/base_007/bucket_6
-rw-r--r-- 3 root supergroup201 2016-11-14 21:46
/user/hive/warehouse/test/z=1/base_007/bucket_7
-rw-r--r-- 3 root supergroup201 2016-11-14 21:46
/user/hive/warehouse/test/z=1/base_007/bucket_8
-rw-r--r-- 3 root supergroup201 2016-11-14 21:46
/user/hive/warehouse/test/z=1/base_007/bucket_9
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HIVE-15139) HoS local mode fails with NumberFormatException
Rui Li created HIVE-15139:
-
Summary: HoS local mode fails with NumberFormatException
Key: HIVE-15139
URL: https://issues.apache.org/jira/browse/HIVE-15139
Project: Hive
Issue Type: Bug
Reporter: Rui Li
Assignee: Rui Li
It's because we store {{stageId_attemptNum}} in JobMetricsListener but expect
only {{stageId}} in LocalSparkJobStatus.
{noformat}
java.lang.NumberFormatException: For input string: "0_0"
at
java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
at java.lang.Integer.parseInt(Integer.java:580)
at java.lang.Integer.parseInt(Integer.java:615)
at
org.apache.hadoop.hive.ql.exec.spark.status.impl.LocalSparkJobStatus.getSparkStatistics(LocalSparkJobStatus.java:146)
at
org.apache.hadoop.hive.ql.exec.spark.SparkTask.execute(SparkTask.java:104)
at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:199)
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HIVE-15081) RetryingMetaStoreClient.getProxy(HiveConf, Boolean) doesn't match constructor of HiveMetaStoreClient
Rui Li created HIVE-15081:
-
Summary: RetryingMetaStoreClient.getProxy(HiveConf, Boolean)
doesn't match constructor of HiveMetaStoreClient
Key: HIVE-15081
URL: https://issues.apache.org/jira/browse/HIVE-15081
Project: Hive
Issue Type: Bug
Reporter: Rui Li
Assignee: Rui Li
Calling RetryingMetaStoreClient.getProxy(HiveConf, Boolean) will result in error
{noformat}
Exception in thread "main" java.lang.RuntimeException: Unable to instantiate
org.apache.hadoop.hive.metastore.HiveMetaStoreClient
at
org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1661)
at
org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.(RetryingMetaStoreClient.java:81)
at
org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:131)
at
org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:87)
Caused by: java.lang.NoSuchMethodException:
org.apache.hadoop.hive.metastore.HiveMetaStoreClient.(org.apache.hadoop.hive.conf.HiveConf,
java.lang.Boolean)
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HIVE-15039) A better job monitor console output for HoS
Rui Li created HIVE-15039: - Summary: A better job monitor console output for HoS Key: HIVE-15039 URL: https://issues.apache.org/jira/browse/HIVE-15039 Project: Hive Issue Type: Improvement Reporter: Rui Li Assignee: Rui Li When there're many stages, it's very difficult to read the console output of job progress of HoS. Attached screenshot is an example. We may learn from HoT as it does a much better than HoS. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: Review Request 50787: Add a timezone-aware timestamp
> On Sept. 22, 2016, 11:20 a.m., Jason Dere wrote: > > - How about compatbility with the various date functions > > (year()/month()/day()/etc)? For most of the functions, TIMESTAMPTZ is implicitly converted to text. Therefore I think we can get correct results. I added some special handle in HOUR because some hour may be unavailable due to DST. So far I've verified the following funcsions work: to_date year quarter month day dayofmonth hour minute second weekofyear Is it OK we leave others in follow-on tasks? I'd like to keep the patch small. > On Sept. 22, 2016, 11:20 a.m., Jason Dere wrote: > > ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFToTimestampTZ.java, > > line 58 > > <https://reviews.apache.org/r/50787/diff/4/?file=1507773#file1507773line58> > > > > No conversions to/from DATE/TIMESTAMP? Added conversion from date/timestamp to timestamptz. Default timezone is used for the converted timestamptz. We can add convertion from numeric types in follow-on task. > On Sept. 22, 2016, 11:20 a.m., Jason Dere wrote: > > serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/PrimitiveObjectInspectorUtils.java, > > line 1109 > > <https://reviews.apache.org/r/50787/diff/4/?file=1507829#file1507829line1109> > > > > If the local timezone is different from the timezone in the > > TimestampTZ, is it possible that the year/month/day of the DATE might be > > different from the year/month/day of the TimestampTZ? Good catch! It makes more sense to convert from the text representation than the time/nanos. So I convert the timestamptz to string first, and use that string to create the date. Same applies when converting to timestamp. - Rui --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/50787/#review149983 --- On Sept. 22, 2016, 4:05 a.m., Rui Li wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/50787/ > --- > > (Updated Sept. 22, 2016, 4:05 a.m.) > > > Review request for hive. > > > Bugs: HIVE-14412 > https://issues.apache.org/jira/browse/HIVE-14412 > > > Repository: hive-git > > > Description > --- > > The 1st patch to add timezone-aware timestamp. > > > Diffs > - > > common/src/test/org/apache/hadoop/hive/common/type/TestTimestampTZ.java > PRE-CREATION > contrib/src/test/queries/clientnegative/serde_regex.q a676338 > contrib/src/test/queries/clientpositive/serde_regex.q d75d607 > contrib/src/test/results/clientnegative/serde_regex.q.out 0f9b036 > contrib/src/test/results/clientpositive/serde_regex.q.out 2984293 > hbase-handler/src/test/queries/positive/hbase_timestamp.q 0350afe > hbase-handler/src/test/results/positive/hbase_timestamp.q.out 3918121 > jdbc/src/java/org/apache/hive/jdbc/HiveBaseResultSet.java 93f093f > jdbc/src/java/org/apache/hive/jdbc/JdbcColumn.java 38918f0 > ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java de74c3e > ql/src/java/org/apache/hadoop/hive/ql/exec/GroupByOperator.java f28d33e > ql/src/java/org/apache/hadoop/hive/ql/exec/SerializationUtilities.java > 7be628e > > ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/translator/TypeConverter.java > ba41518 > ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java > 8b0db4a > ql/src/java/org/apache/hadoop/hive/ql/parse/HiveLexer.g 7ceb005 > ql/src/java/org/apache/hadoop/hive/ql/parse/HiveParser.g 62bbcc6 > ql/src/java/org/apache/hadoop/hive/ql/parse/IdentifiersParser.g 9ba1865 > ql/src/java/org/apache/hadoop/hive/ql/parse/TypeCheckProcFactory.java > 82080eb > ql/src/java/org/apache/hadoop/hive/ql/stats/StatsUtils.java a718264 > ql/src/java/org/apache/hadoop/hive/ql/udf/UDFToBoolean.java 17b892c > ql/src/java/org/apache/hadoop/hive/ql/udf/UDFToByte.java efae82d > ql/src/java/org/apache/hadoop/hive/ql/udf/UDFToDouble.java 9cbc114 > ql/src/java/org/apache/hadoop/hive/ql/udf/UDFToFloat.java 5808c90 > ql/src/java/org/apache/hadoop/hive/ql/udf/UDFToInteger.java a7551cb > ql/src/java/org/apache/hadoop/hive/ql/udf/UDFToLong.java c961d14 > ql/src/java/org/apache/hadoop/hive/ql/udf/UDFToShort.java 570408a > ql/src/java/org/apache/hadoop/hive/ql/udf/UDFToString.java 5cacd59 > ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDF.java 259fde8 > > ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFToTimestampT
Re: Review Request 50787: Add a timezone-aware timestamp
/src/gen/thrift/gen-rb/serde_constants.rb 0ce9f27 serde/src/java/org/apache/hadoop/hive/serde2/SerDeUtils.java 7ffc964 serde/src/java/org/apache/hadoop/hive/serde2/binarysortable/BinarySortableSerDe.java 5e119d7 serde/src/java/org/apache/hadoop/hive/serde2/binarysortable/fast/BinarySortableDeserializeRead.java a7785b2 serde/src/java/org/apache/hadoop/hive/serde2/io/TimestampTZWritable.java PRE-CREATION serde/src/java/org/apache/hadoop/hive/serde2/io/TimestampWritable.java bbccc7f serde/src/java/org/apache/hadoop/hive/serde2/io/TimestampWritableBase.java PRE-CREATION serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazyFactory.java 23dbe6a serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazyTimestamp.java 56945d1 serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazyTimestampTZ.java PRE-CREATION serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazyUtils.java 73c72e1 serde/src/java/org/apache/hadoop/hive/serde2/lazy/objectinspector/primitive/LazyPrimitiveObjectInspectorFactory.java 5601734 serde/src/java/org/apache/hadoop/hive/serde2/lazy/objectinspector/primitive/LazyTimestampTZObjectInspector.java PRE-CREATION serde/src/java/org/apache/hadoop/hive/serde2/lazybinary/LazyBinaryFactory.java 52f3527 serde/src/java/org/apache/hadoop/hive/serde2/lazybinary/LazyBinarySerDe.java 54bfd2d serde/src/java/org/apache/hadoop/hive/serde2/lazybinary/LazyBinaryTimestampTZ.java PRE-CREATION serde/src/java/org/apache/hadoop/hive/serde2/lazybinary/LazyBinaryUtils.java f8a110d serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/ObjectInspectorConverters.java 24b3d4e serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/ObjectInspectorUtils.java 1ac72c6 serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/PrimitiveObjectInspector.java 70633f3 serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/JavaTimestampObjectInspector.java 509189e serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/JavaTimestampTZObjectInspector.java PRE-CREATION serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/PrimitiveObjectInspectorConverter.java e08ad43 serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/PrimitiveObjectInspectorFactory.java 2ed0843 serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/PrimitiveObjectInspectorUtils.java 51b529e serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/SettableTimestampTZObjectInspector.java PRE-CREATION serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/TimestampTZObjectorInspector.java PRE-CREATION serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/WritableConstantTimestampTZObjectInspector.java PRE-CREATION serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/WritableTimestampTZObjectInspector.java PRE-CREATION serde/src/java/org/apache/hadoop/hive/serde2/thrift/Type.java 0ad8c02 serde/src/java/org/apache/hadoop/hive/serde2/typeinfo/TypeInfoFactory.java 43c4819 serde/src/test/org/apache/hadoop/hive/serde2/io/TestTimestampWritable.java 3c483cc service-rpc/if/TCLIService.thrift a4fa7b0 service-rpc/src/gen/thrift/gen-cpp/TCLIService_constants.cpp 991cb2e service-rpc/src/gen/thrift/gen-cpp/TCLIService_types.h b249544 service-rpc/src/gen/thrift/gen-cpp/TCLIService_types.cpp 2f460e8 service-rpc/src/gen/thrift/gen-javabean/org/apache/hive/service/rpc/thrift/TCLIServiceConstants.java 930bed7 service-rpc/src/gen/thrift/gen-javabean/org/apache/hive/service/rpc/thrift/TProtocolVersion.java bce2a0c service-rpc/src/gen/thrift/gen-javabean/org/apache/hive/service/rpc/thrift/TTypeId.java a3735eb service-rpc/src/gen/thrift/gen-php/Types.php 786c773 service-rpc/src/gen/thrift/gen-py/TCLIService/constants.py c8d4f8f service-rpc/src/gen/thrift/gen-py/TCLIService/ttypes.py fdf6b1f service-rpc/src/gen/thrift/gen-rb/t_c_l_i_service_constants.rb 25adbb4 service-rpc/src/gen/thrift/gen-rb/t_c_l_i_service_types.rb 4b1854c service/src/java/org/apache/hive/service/cli/ColumnValue.java 76e8c03 service/src/java/org/apache/hive/service/cli/TypeDescriptor.java d634bef storage-api/src/java/org/apache/hadoop/hive/common/type/TimestampTZ.java PRE-CREATION storage-api/src/java/org/apache/hadoop/hive/ql/util/JavaDataModel.java 4a745e4 storage-api/src/java/org/apache/hadoop/hive/ql/util/TimestampUtils.java 41db9ca Diff: https://reviews.apache.org/r/50787/diff/ Testing --- Thanks, Rui Li
Re: Review Request 50787: Add a timezone-aware timestamp
/fast/BinarySortableDeserializeRead.java a7785b2 serde/src/java/org/apache/hadoop/hive/serde2/io/TimestampTZWritable.java PRE-CREATION serde/src/java/org/apache/hadoop/hive/serde2/io/TimestampWritable.java bbccc7f serde/src/java/org/apache/hadoop/hive/serde2/io/TimestampWritableBase.java PRE-CREATION serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazyFactory.java 23dbe6a serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazyTimestamp.java 56945d1 serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazyTimestampTZ.java PRE-CREATION serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazyUtils.java 73c72e1 serde/src/java/org/apache/hadoop/hive/serde2/lazy/objectinspector/primitive/LazyPrimitiveObjectInspectorFactory.java 5601734 serde/src/java/org/apache/hadoop/hive/serde2/lazy/objectinspector/primitive/LazyTimestampTZObjectInspector.java PRE-CREATION serde/src/java/org/apache/hadoop/hive/serde2/lazybinary/LazyBinaryFactory.java 52f3527 serde/src/java/org/apache/hadoop/hive/serde2/lazybinary/LazyBinarySerDe.java 54bfd2d serde/src/java/org/apache/hadoop/hive/serde2/lazybinary/LazyBinaryTimestampTZ.java PRE-CREATION serde/src/java/org/apache/hadoop/hive/serde2/lazybinary/LazyBinaryUtils.java f8a110d serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/ObjectInspectorConverters.java 24b3d4e serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/ObjectInspectorUtils.java 1ac72c6 serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/PrimitiveObjectInspector.java 70633f3 serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/JavaTimestampObjectInspector.java 509189e serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/JavaTimestampTZObjectInspector.java PRE-CREATION serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/PrimitiveObjectInspectorConverter.java e08ad43 serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/PrimitiveObjectInspectorFactory.java 2ed0843 serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/PrimitiveObjectInspectorUtils.java 51b529e serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/SettableTimestampTZObjectInspector.java PRE-CREATION serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/TimestampTZObjectorInspector.java PRE-CREATION serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/WritableConstantTimestampTZObjectInspector.java PRE-CREATION serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/WritableTimestampTZObjectInspector.java PRE-CREATION serde/src/java/org/apache/hadoop/hive/serde2/thrift/Type.java 0ad8c02 serde/src/java/org/apache/hadoop/hive/serde2/typeinfo/TypeInfoFactory.java 43c4819 serde/src/test/org/apache/hadoop/hive/serde2/io/TestTimestampWritable.java 3c483cc service-rpc/if/TCLIService.thrift a4fa7b0 service-rpc/src/gen/thrift/gen-cpp/TCLIService_constants.cpp 991cb2e service-rpc/src/gen/thrift/gen-cpp/TCLIService_types.h b249544 service-rpc/src/gen/thrift/gen-cpp/TCLIService_types.cpp 2f460e8 service-rpc/src/gen/thrift/gen-javabean/org/apache/hive/service/rpc/thrift/TCLIServiceConstants.java 930bed7 service-rpc/src/gen/thrift/gen-javabean/org/apache/hive/service/rpc/thrift/TProtocolVersion.java bce2a0c service-rpc/src/gen/thrift/gen-javabean/org/apache/hive/service/rpc/thrift/TTypeId.java a3735eb service-rpc/src/gen/thrift/gen-php/Types.php 786c773 service-rpc/src/gen/thrift/gen-py/TCLIService/constants.py c8d4f8f service-rpc/src/gen/thrift/gen-py/TCLIService/ttypes.py fdf6b1f service-rpc/src/gen/thrift/gen-rb/t_c_l_i_service_constants.rb 25adbb4 service-rpc/src/gen/thrift/gen-rb/t_c_l_i_service_types.rb 4b1854c service/src/java/org/apache/hive/service/cli/ColumnValue.java 76e8c03 service/src/java/org/apache/hive/service/cli/TypeDescriptor.java d634bef storage-api/src/java/org/apache/hadoop/hive/common/type/TimestampTZ.java PRE-CREATION storage-api/src/java/org/apache/hadoop/hive/ql/util/JavaDataModel.java 4a745e4 storage-api/src/java/org/apache/hadoop/hive/ql/util/TimestampUtils.java 41db9ca Diff: https://reviews.apache.org/r/50787/diff/ Testing --- Thanks, Rui Li
Re: Review Request 50787: Add a timezone-aware timestamp
/fast/BinarySortableDeserializeRead.java a7785b2 serde/src/java/org/apache/hadoop/hive/serde2/io/TimestampTZWritable.java PRE-CREATION serde/src/java/org/apache/hadoop/hive/serde2/io/TimestampWritable.java bbccc7f serde/src/java/org/apache/hadoop/hive/serde2/io/TimestampWritableBase.java PRE-CREATION serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazyFactory.java 23dbe6a serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazyTimestamp.java 56945d1 serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazyTimestampTZ.java PRE-CREATION serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazyUtils.java 73c72e1 serde/src/java/org/apache/hadoop/hive/serde2/lazy/objectinspector/primitive/LazyPrimitiveObjectInspectorFactory.java 5601734 serde/src/java/org/apache/hadoop/hive/serde2/lazy/objectinspector/primitive/LazyTimestampTZObjectInspector.java PRE-CREATION serde/src/java/org/apache/hadoop/hive/serde2/lazybinary/LazyBinaryFactory.java 52f3527 serde/src/java/org/apache/hadoop/hive/serde2/lazybinary/LazyBinarySerDe.java 54bfd2d serde/src/java/org/apache/hadoop/hive/serde2/lazybinary/LazyBinaryTimestampTZ.java PRE-CREATION serde/src/java/org/apache/hadoop/hive/serde2/lazybinary/LazyBinaryUtils.java f8a110d serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/ObjectInspectorConverters.java 24b3d4e serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/ObjectInspectorUtils.java 1ac72c6 serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/PrimitiveObjectInspector.java 70633f3 serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/JavaTimestampObjectInspector.java 509189e serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/JavaTimestampTZObjectInspector.java PRE-CREATION serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/PrimitiveObjectInspectorConverter.java e08ad43 serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/PrimitiveObjectInspectorFactory.java 2ed0843 serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/PrimitiveObjectInspectorUtils.java 51b529e serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/SettableTimestampTZObjectInspector.java PRE-CREATION serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/TimestampTZObjectorInspector.java PRE-CREATION serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/WritableConstantTimestampTZObjectInspector.java PRE-CREATION serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/WritableTimestampTZObjectInspector.java PRE-CREATION serde/src/java/org/apache/hadoop/hive/serde2/thrift/Type.java 0ad8c02 serde/src/java/org/apache/hadoop/hive/serde2/typeinfo/TypeInfoFactory.java 43c4819 serde/src/test/org/apache/hadoop/hive/serde2/io/TestTimestampWritable.java 3c483cc service-rpc/if/TCLIService.thrift a4fa7b0 service-rpc/src/gen/thrift/gen-cpp/TCLIService_constants.cpp 991cb2e service-rpc/src/gen/thrift/gen-cpp/TCLIService_types.h b249544 service-rpc/src/gen/thrift/gen-cpp/TCLIService_types.cpp 2f460e8 service-rpc/src/gen/thrift/gen-javabean/org/apache/hive/service/rpc/thrift/TCLIServiceConstants.java 930bed7 service-rpc/src/gen/thrift/gen-javabean/org/apache/hive/service/rpc/thrift/TProtocolVersion.java bce2a0c service-rpc/src/gen/thrift/gen-javabean/org/apache/hive/service/rpc/thrift/TTypeId.java a3735eb service-rpc/src/gen/thrift/gen-php/Types.php 786c773 service-rpc/src/gen/thrift/gen-py/TCLIService/constants.py c8d4f8f service-rpc/src/gen/thrift/gen-py/TCLIService/ttypes.py fdf6b1f service-rpc/src/gen/thrift/gen-rb/t_c_l_i_service_constants.rb 25adbb4 service-rpc/src/gen/thrift/gen-rb/t_c_l_i_service_types.rb 4b1854c service/src/java/org/apache/hive/service/cli/ColumnValue.java 76e8c03 service/src/java/org/apache/hive/service/cli/TypeDescriptor.java d634bef storage-api/src/java/org/apache/hadoop/hive/common/type/TimestampTZ.java PRE-CREATION storage-api/src/java/org/apache/hadoop/hive/ql/util/JavaDataModel.java 4a745e4 storage-api/src/java/org/apache/hadoop/hive/ql/util/TimestampUtils.java 41db9ca Diff: https://reviews.apache.org/r/50787/diff/ Testing --- Thanks, Rui Li
Re: Review Request 50787: Add a timezone-aware timestamp
/hive/serde2/lazybinary/LazyBinaryUtils.java f8a110d serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/ObjectInspectorConverters.java 24b3d4e serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/ObjectInspectorUtils.java 1ac72c6 serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/PrimitiveObjectInspector.java 70633f3 serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/HiveTimestampObjectorInspector.java PRE-CREATION serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/JavaHiveTimestampObjectInspector.java PRE-CREATION serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/JavaTimestampObjectInspector.java 509189e serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/PrimitiveObjectInspectorConverter.java e08ad43 serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/PrimitiveObjectInspectorFactory.java 2ed0843 serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/PrimitiveObjectInspectorUtils.java 51b529e serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/SettableHiveTimestampObjectInspector.java PRE-CREATION serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/WritableConstantHiveTimestampObjectInspector.java PRE-CREATION serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/WritableHiveTimestampObjectInspector.java PRE-CREATION serde/src/java/org/apache/hadoop/hive/serde2/thrift/Type.java 0ad8c02 serde/src/java/org/apache/hadoop/hive/serde2/typeinfo/TypeInfoFactory.java 43c4819 serde/src/test/org/apache/hadoop/hive/serde2/io/TestTimestampWritable.java 3c483cc service-rpc/if/TCLIService.thrift a4fa7b0 service-rpc/src/gen/thrift/gen-cpp/TCLIService_constants.cpp 991cb2e service-rpc/src/gen/thrift/gen-cpp/TCLIService_types.h b249544 service-rpc/src/gen/thrift/gen-cpp/TCLIService_types.cpp 2f460e8 service-rpc/src/gen/thrift/gen-javabean/org/apache/hive/service/rpc/thrift/TCLIServiceConstants.java 930bed7 service-rpc/src/gen/thrift/gen-javabean/org/apache/hive/service/rpc/thrift/TProtocolVersion.java bce2a0c service-rpc/src/gen/thrift/gen-javabean/org/apache/hive/service/rpc/thrift/TTypeId.java a3735eb service-rpc/src/gen/thrift/gen-php/Types.php 786c773 service-rpc/src/gen/thrift/gen-py/TCLIService/constants.py c8d4f8f service-rpc/src/gen/thrift/gen-py/TCLIService/ttypes.py fdf6b1f service-rpc/src/gen/thrift/gen-rb/t_c_l_i_service_constants.rb 25adbb4 service-rpc/src/gen/thrift/gen-rb/t_c_l_i_service_types.rb 4b1854c service/src/java/org/apache/hive/service/cli/ColumnValue.java 76e8c03 service/src/java/org/apache/hive/service/cli/TypeDescriptor.java d634bef storage-api/src/java/org/apache/hadoop/hive/common/type/HiveTimestamp.java PRE-CREATION storage-api/src/java/org/apache/hadoop/hive/ql/util/JavaDataModel.java 4a745e4 storage-api/src/java/org/apache/hadoop/hive/ql/util/TimestampUtils.java 41db9ca Diff: https://reviews.apache.org/r/50787/diff/ Testing --- Thanks, Rui Li
[jira] [Created] (HIVE-14728) Redundant orig files
Rui Li created HIVE-14728: - Summary: Redundant orig files Key: HIVE-14728 URL: https://issues.apache.org/jira/browse/HIVE-14728 Project: Hive Issue Type: Bug Reporter: Rui Li Priority: Minor I find some orig files in master, e.g. SemanticAnalyzer.java.orig. Wondering if they are added by mistake? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-14719) ASTNode rootNode is not maintained properly when changing child/parent relation
Rui Li created HIVE-14719:
-
Summary: ASTNode rootNode is not maintained properly when changing
child/parent relation
Key: HIVE-14719
URL: https://issues.apache.org/jira/browse/HIVE-14719
Project: Hive
Issue Type: Bug
Reporter: Rui Li
Assignee: Rui Li
When I run some query like:
{code}
set hive.cbo.enable=false;
select * from A where exists (select * from B where B.k1=A.k1 and B.k2=A.k2);
{code}
It gets error like:
{noformat}
FAILED: SemanticException Line 0:-1 Invalid table alias or column reference
'sq_1': (possible column names are: _table_or_col b) k2) sq_corr_1)) (tok, (.
(tok_table_or_col sq_1) sq_corr_1))
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HIVE-14595) TimestampWritable::setTimestamp gives wrong result when 2nd VInt exists
Rui Li created HIVE-14595: - Summary: TimestampWritable::setTimestamp gives wrong result when 2nd VInt exists Key: HIVE-14595 URL: https://issues.apache.org/jira/browse/HIVE-14595 Project: Hive Issue Type: Bug Reporter: Rui Li Assignee: Rui Li -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: YourKit open source license
Our wiki doesn't mention it's only for committers. Anyway I suggest you contact YourKit sales to figure out. On Tue, Aug 16, 2016 at 8:38 PM, calvin hung <calvinh...@wasaitech.com> wrote: > > > Thanks for your response, Rui. > > I don't have an apache email account. > > It looks like only committer can get an email account according to this > page http://www.apache.org/dev/committers.html > > Does it mean that only Hive committers can get YourKit free licenses for > Hive performance profiling? > > > > > > On Tue, 16 Aug 2016 13:33:34 +0800 Rui Li lirui.fu...@gmail.com > wrote > > > > > If I remember correctly, I just contacted the sales of Yourkit and they > > sent me the license by email. You'd better send your email using your > > apache email account, in order to convince them you're a developer of Hive. > > > > On Tue, Aug 16, 2016 at 2:51 AM, calvin hung calvinh...@wasaitech.com& > gt; > > wrote: > > > > Hi Rui and Alan, > > > > Could you or any nice guy share more detail steps of getting a Yourkit > > license for Hive? > > I've searched the full Hive dev mail archive but got no exact steps > to get > > one. > > Thanks! > > > > Calvin > > From: "Li, Rui"rui...@intel.com > > Date: Tue, 31 Mar 2015 01:22:51 + > > To: "dev@hive.apache.org"dev@hive.apache.org > > > > - Contents - > > > > Thanks Alan! But I don’t see Hive in the sponsored open source project > > list. I’ll contact them anyway. > > > > > > > > Cheers, > > > > Rui Li > > > > > > > > From: Alan Gates [mailto:alanfga...@gmail.com] > > Sent: Tuesday, March 31, 2015 1:02 AM > > To: dev@hive.apache.org > > Subject: Re: YourKit open source license > > > > > > > > Seehttps://www.yourkit.com/customers/. > > > > Alan. > > > > > > > > > > > > Li, Rui > > > > March 30, 2015 at 0:54 > > > > Hi guys, > > > > I want to use YourKit to profile hive performance. According to the > wiki > > https://cwiki.apache.org/confluence/display/Hive/Performance; > hive has > > been granted open source license. Could anybody tell me how I can get > the > > license? Thanks! > > > > Cheers, > > Rui Li > > > > > > > > > > -- > > Best regards! > > Rui Li > > Cell: (+86) 13564950210 > > > > > > > -- Best regards! Rui Li Cell: (+86) 13564950210
Re: YourKit open source license
If I remember correctly, I just contacted the sales of Yourkit and they sent me the license by email. You'd better send your email using your apache email account, in order to convince them you're a developer of Hive. On Tue, Aug 16, 2016 at 2:51 AM, calvin hung <calvinh...@wasaitech.com> wrote: > Hi Rui and Alan, > > Could you or any nice guy share more detail steps of getting a Yourkit > license for Hive? > I've searched the full Hive dev mail archive but got no exact steps to get > one. > Thanks! > > Calvin > From: "Li, Rui"<rui...@intel.com> > Date: Tue, 31 Mar 2015 01:22:51 + > To: "dev@hive.apache.org"<dev@hive.apache.org> > > - Contents - > > Thanks Alan! But I don’t see Hive in the sponsored open source project > list. I’ll contact them anyway. > > > > Cheers, > > Rui Li > > > > From: Alan Gates [mailto:alanfga...@gmail.com] > Sent: Tuesday, March 31, 2015 1:02 AM > To: dev@hive.apache.org > Subject: Re: YourKit open source license > > > > Seehttps://www.yourkit.com/customers/. > > Alan. > > > > > > Li, Rui > > March 30, 2015 at 0:54 > > Hi guys, > > I want to use YourKit to profile hive performance. According to the wiki< > https://cwiki.apache.org/confluence/display/Hive/Performance> hive has > been granted open source license. Could anybody tell me how I can get the > license? Thanks! > > Cheers, > Rui Li -- Best regards! Rui Li Cell: (+86) 13564950210
Review Request 50787: Add a timezone-aware timestamp
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/50787/ --- Review request for hive. Bugs: HIVE-14412 https://issues.apache.org/jira/browse/HIVE-14412 Repository: hive-git Description --- The 1st patch to add timezone-aware timestamp. Diffs - common/src/test/org/apache/hadoop/hive/common/type/TestHiveTimestamp.java PRE-CREATION serde/src/java/org/apache/hadoop/hive/serde2/io/TimestampWritable.java b90e576 serde/src/test/org/apache/hadoop/hive/serde2/io/TestTimestampWritable.java 7619efa storage-api/src/java/org/apache/hadoop/hive/common/type/HiveTimestamp.java PRE-CREATION Diff: https://reviews.apache.org/r/50787/diff/ Testing --- Thanks, Rui Li
[jira] [Created] (HIVE-14412) Add a timezone-aware timestamp
Rui Li created HIVE-14412: - Summary: Add a timezone-aware timestamp Key: HIVE-14412 URL: https://issues.apache.org/jira/browse/HIVE-14412 Project: Hive Issue Type: Sub-task Reporter: Rui Li Assignee: Rui Li Java's Timestamp stores the time elapsed since the epoch. While it's by itself unambiguous, ambiguity comes when we parse a string into timestamp, or convert a timestamp to string, causing problems like HIVE-14305. To solve the issue, I think we should make timestamp aware of timezone. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-14305) To/From UTC timestamp may return incorrect result because of DST
Rui Li created HIVE-14305: - Summary: To/From UTC timestamp may return incorrect result because of DST Key: HIVE-14305 URL: https://issues.apache.org/jira/browse/HIVE-14305 Project: Hive Issue Type: Bug Reporter: Rui Li Assignee: Rui Li -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-14238) Ownership shouldn't be checked if external table location doesn't exist
Rui Li created HIVE-14238: - Summary: Ownership shouldn't be checked if external table location doesn't exist Key: HIVE-14238 URL: https://issues.apache.org/jira/browse/HIVE-14238 Project: Hive Issue Type: Bug Reporter: Rui Li Assignee: Rui Li When creating external table with SQL authorization, we require RWX permission + ownership of the table location. If the location doesn't exist, we check on parent dir (recursively), which means we require the user owns everything under parent dir. I think this is not necessary - we don't have to check ownership of parent dir, or we just check non-recursively. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-14139) NPE dropping permanent function
Rui Li created HIVE-14139:
-
Summary: NPE dropping permanent function
Key: HIVE-14139
URL: https://issues.apache.org/jira/browse/HIVE-14139
Project: Hive
Issue Type: Bug
Reporter: Rui Li
Assignee: Rui Li
To reproduce:
1. Start a CLI session and create a permanent function.
2. Exit current CLI session.
3. Start a new CLI session and drop the function.
Stack trace:
{noformat}
FAILED: error during drop function: java.lang.NullPointerException
at
org.apache.hadoop.hive.ql.exec.Registry.removePersistentFunctionUnderLock(Registry.java:513)
at
org.apache.hadoop.hive.ql.exec.Registry.unregisterFunction(Registry.java:501)
at
org.apache.hadoop.hive.ql.exec.FunctionRegistry.unregisterPermanentFunction(FunctionRegistry.java:1532)
at
org.apache.hadoop.hive.ql.exec.FunctionTask.dropPermanentFunction(FunctionTask.java:228)
at
org.apache.hadoop.hive.ql.exec.FunctionTask.execute(FunctionTask.java:95)
at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:197)
at
org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:100)
at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1860)
at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1564)
at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1316)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1085)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1073)
at
org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:232)
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HIVE-13997) Insert overwrite directory doesn't overwrite existing files
Rui Li created HIVE-13997:
-
Summary: Insert overwrite directory doesn't overwrite existing
files
Key: HIVE-13997
URL: https://issues.apache.org/jira/browse/HIVE-13997
Project: Hive
Issue Type: Bug
Reporter: Rui Li
Assignee: Rui Li
Can be easily reproduced by running {{INSERT OVERWRITE DIRECTORY}} to the same
dir twice.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HIVE-13921) Fix spark on yarn tests for HoS
Rui Li created HIVE-13921:
-
Summary: Fix spark on yarn tests for HoS
Key: HIVE-13921
URL: https://issues.apache.org/jira/browse/HIVE-13921
Project: Hive
Issue Type: Test
Reporter: Rui Li
Assignee: Rui Li
{{index_bitmap3}} and {{constprog_partitioner}} have been failing. Let's fix
them here.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HIVE-13895) HoS start-up overhead in yarn-client mode
Rui Li created HIVE-13895:
-
Summary: HoS start-up overhead in yarn-client mode
Key: HIVE-13895
URL: https://issues.apache.org/jira/browse/HIVE-13895
Project: Hive
Issue Type: Bug
Reporter: Rui Li
Assignee: Rui Li
To avoid the too verbose app state report, HIVE-13376 increases the state check
interval to a default 60s. However, bigger interval brings considerable
start-up wait time for yarn-client mode.
Since the state report only exists in yarn-cluster mode, we can disable it
using {{spark.yarn.submit.waitAppCompletion}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HIVE-13843) Re-enable the HoS tests disabled in HIVE-13402
Rui Li created HIVE-13843: - Summary: Re-enable the HoS tests disabled in HIVE-13402 Key: HIVE-13843 URL: https://issues.apache.org/jira/browse/HIVE-13843 Project: Hive Issue Type: Test Reporter: Rui Li Assignee: Rui Li With HIVE-13525, we can now fix and re-enable the tests for Spark. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-13789) Repeatedly checking configuration in TextRecordWriter/Reader hurts performance
Rui Li created HIVE-13789: - Summary: Repeatedly checking configuration in TextRecordWriter/Reader hurts performance Key: HIVE-13789 URL: https://issues.apache.org/jira/browse/HIVE-13789 Project: Hive Issue Type: Improvement Reporter: Rui Li Assignee: Rui Li Priority: Minor We check configuration to decide whether to escape certain characters each time write/read a record for custom scripts. In our benchmark this becomes a hot spot method. And fixing it improves the execution of the custom script by 7% (3TB TPCx-BB dataset). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: Review Request 47242: HIVE-13726
> On May 12, 2016, 5:48 a.m., Ashutosh Chauhan wrote: > > ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java, line 3074 > > <https://reviews.apache.org/r/47242/diff/1/?file=1379752#file1379752line3074> > > > > It could have been. Feels cleaner to me as non-static. Just a matter of > > style. Is there any advantage of keeping it static? Just thought the method is protected and may be accessed in sub-classes. If that's not a concern, I'm OK with non-static. - Rui --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/47242/#review132822 --- On May 11, 2016, 4:36 p.m., Ashutosh Chauhan wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/47242/ > --- > > (Updated May 11, 2016, 4:36 p.m.) > > > Review request for hive and Rui Li. > > > Bugs: HIVE-13726 > https://issues.apache.org/jira/browse/HIVE-13726 > > > Repository: hive-git > > > Description > --- > > For insert overwrite significant amount might be spent in deleting existing > files. This patch parallelizes this task. > > > Diffs > - > > common/src/java/org/apache/hadoop/hive/common/FileUtils.java 71c9188 > ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java dd14124 > > Diff: https://reviews.apache.org/r/47242/diff/ > > > Testing > --- > > existing regression tests. No change in functionality. > > > Thanks, > > Ashutosh Chauhan > >
Re: Review Request 47242: HIVE-13726
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/47242/#review132821 --- ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java (line 3074) <https://reviews.apache.org/r/47242/#comment197074> Any reason why remove the static? I think the newly added method can be static too. ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java (line 3168) <https://reviews.apache.org/r/47242/#comment197073> I think FileNotFoundException is not needed here. - Rui Li On May 11, 2016, 4:36 p.m., Ashutosh Chauhan wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/47242/ > --- > > (Updated May 11, 2016, 4:36 p.m.) > > > Review request for hive and Rui Li. > > > Bugs: HIVE-13726 > https://issues.apache.org/jira/browse/HIVE-13726 > > > Repository: hive-git > > > Description > --- > > For insert overwrite significant amount might be spent in deleting existing > files. This patch parallelizes this task. > > > Diffs > - > > common/src/java/org/apache/hadoop/hive/common/FileUtils.java 71c9188 > ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java dd14124 > > Diff: https://reviews.apache.org/r/47242/diff/ > > > Testing > --- > > existing regression tests. No change in functionality. > > > Thanks, > > Ashutosh Chauhan > >
