Re: HIVE building on ARM
Hi Ashutosh, I want to migrate https://builds.apache.org/view/H-L/view/Hive/job/Hive-linux-ARM-trunk/ this job to the new system https://ci-hadoop.apache.org. Please grant me rights to create jobs on the new Jenkins https://ci-builds.apache.org. My LDAP userid : chinnaraol. I have requested the same in apache infra. Thank you. Thank you Stamatis,Zoltan for the pointers. Thanks, Chinna On Thu, Jun 18, 2020 at 5:36 PM Stamatis Zampetakis wrote: > Hello Chinna, > > The hudson-jobadmin privilege can be granted by PMC chairs. > I don't know if there is any particular policy in Hive on who should have > this privilege so I guess you should request it from Ashutosh. > > Best, > Stamatis > > On Thu, Jun 18, 2020 at 12:05 PM Zoltan Haindrich wrote: > >> Hey Chinna! >> >> On 6/18/20 11:43 AM, Chinna Rao Lalam wrote: >> > As you said, migrating this job to the new ci-hadoop instance looks >> good as >> > Hadoop also shares the same armN slaves. >> >> Sounds great! >> >> > I am able to login the new ci-hadoop instance with Apache LDAP >> credentials, >> > but i am not able to see the job creation option. Should I request >> access >> > or the process for creation of a job is different than jenkin?. >> > Please guide me to create the new job in the ci-hadoop instance. I will >> > migrate this job after connecting the armN slaves to the new system. >> >> >> I've also logged in - and apparently I've create job rights; I'm happy to >> help, but the best would be to self-service yourselft :) >> I think you may miss the "hudson-jobadmin" privilege. >> Probably Gavin (or someone on the infra team) could help you with that.. >> to talk to them quickly - you can reach them on the #asfinfra channel (on >> the asf-slack). >> >> The migration effort is coordinated thru the hadoop-migrations mailing >> list (I've cc-ed that list) >> you may want to subscribe to it by sending a mail to: >> hadoop-migrations-subscr...@infra.apache.org >> >> cheers, >> Zoltan >> >> >> >> > >> > Thanks >> > Chinna >> > >> > On Wed, Jun 17, 2020 at 11:57 AM Zhenyu Zheng < >> zhengzhenyul...@gmail.com> >> > wrote: >> > >> >> Hi Zoltan, >> >> >> >> Thanks alot for the information, so looks like one possible solution >> is as >> >> you suggest, move the current ARM2 and ARM3 (those two were donate to >> >> builds.apache.org by us) to the new ci-hadoop cluster and set up the >> jobs >> >> just as what has been done in current jenkins. >> >> >> >> I will also ask our team member works on other projects to find out >> what >> >> the status of other projects is. >> >> >> >> BR, >> >> >> >> On Tue, Jun 16, 2020 at 6:41 PM Zoltan Haindrich wrote: >> >> >> >>> Hey, >> >>> >> >>> There is an effort by the Apache Infra to change the way Jenkins >> stuff is >> >>> organized; a couple months ago Gavin wrote an email about it: >> >>> >> >>> >> http://mail-archives.apache.org/mod_mbox/tez-dev/202004.mbox/%3ccan0gg1dodepzatjz9bofe-2ver7qg7h0hmvyjmsldgjr8_r...@mail.gmail.com%3E >> >>> The resources for running these jobs are coming from the H0~H21 slaves >> >>> which will be migrated to the new jenkins master eventually. >> >>> >> >>> >> So please >> >>> >> suggest a way which direction we can move and can you share some >> >>> details >> >>> >> about the new ci-hadoop instance. >> >>> >> >>> Since Hadoop testing is also happening on ARM - I think the best >> would be >> >>> to also migrate the armN slaves and the Hive arm nightly over to the >> new >> >>> ci-hadoop instance. >> >>> >> >>> On 6/16/20 8:40 AM, Zhenyu Zheng wrote: >> >>>> Thanks for the info, I wonder if where does the resource of ci-hadoop >> >>> and >> >>>> hive-test-kube come from? Do they include ARM resources? >> >>> >> >>> Interesting question; the resources for Hive testing are donated by >> >>> Cloudera. >> >>> About the ARM workers I think Chinna could provide more details. >> >>> ...I've no idea don't know who sponsors the Hxx slaves >> >>> >> >>>> Can you provide some m
Re: HIVE building on ARM
Hi Zoltan, As you said, migrating this job to the new ci-hadoop instance looks good as Hadoop also shares the same armN slaves. I am able to login the new ci-hadoop instance with Apache LDAP credentials, but i am not able to see the job creation option. Should I request access or the process for creation of a job is different than jenkin?. Please guide me to create the new job in the ci-hadoop instance. I will migrate this job after connecting the armN slaves to the new system. Thanks Chinna On Wed, Jun 17, 2020 at 11:57 AM Zhenyu Zheng wrote: > Hi Zoltan, > > Thanks alot for the information, so looks like one possible solution is as > you suggest, move the current ARM2 and ARM3 (those two were donate to > builds.apache.org by us) to the new ci-hadoop cluster and set up the jobs > just as what has been done in current jenkins. > > I will also ask our team member works on other projects to find out what > the status of other projects is. > > BR, > > On Tue, Jun 16, 2020 at 6:41 PM Zoltan Haindrich wrote: > >> Hey, >> >> There is an effort by the Apache Infra to change the way Jenkins stuff is >> organized; a couple months ago Gavin wrote an email about it: >> >> http://mail-archives.apache.org/mod_mbox/tez-dev/202004.mbox/%3ccan0gg1dodepzatjz9bofe-2ver7qg7h0hmvyjmsldgjr8_r...@mail.gmail.com%3E >> The resources for running these jobs are coming from the H0~H21 slaves >> which will be migrated to the new jenkins master eventually. >> >> >> So please >> >> suggest a way which direction we can move and can you share some >> details >> >> about the new ci-hadoop instance. >> >> Since Hadoop testing is also happening on ARM - I think the best would be >> to also migrate the armN slaves and the Hive arm nightly over to the new >> ci-hadoop instance. >> >> On 6/16/20 8:40 AM, Zhenyu Zheng wrote: >> > Thanks for the info, I wonder if where does the resource of ci-hadoop >> and >> > hive-test-kube come from? Do they include ARM resources? >> >> Interesting question; the resources for Hive testing are donated by >> Cloudera. >> About the ARM workers I think Chinna could provide more details. >> ...I've no idea don't know who sponsors the Hxx slaves >> >> > Can you provide some more information about how the new hive-test-kube >> is >> > running? >> It's basically a Jenkins instance which is using kubernetes pods to run >> things. >> The whole thing is running on a GKE cluster. >> While I was working on it I collected stuff needed for it in this repo: >> https://github.com/kgyrtkirk/hive-test-kube/ >> it should be possible to start a new deployment using that stuff >> >> cheers, >> Zoltan >> >> > >> > BR, >> > Kevin Zheng >> > >> > On Tue, Jun 16, 2020 at 12:41 PM Chinna Rao Lalam < >> > lalamchinnara...@gmail.com> wrote: >> > >> >> Hi Zoltan, >> >> >> >> Thanks for the update. >> >> >> >> Current https://builds.apache.org/job/Hive-linux-ARM-trunk/ job is >> >> targeting to run hive tests daily on "arm" slaves, it is using 2 arm >> >> slaves. >> >> To find any potential issues with "arm" and fix the issues. So please >> >> suggest a way which direction we can move and can you share some >> details >> >> about the new ci-hadoop instance. >> >> >> >> Thanks, >> >> Chinna >> >> >> >> On Mon, Jun 15, 2020 at 3:56 PM Zoltan Haindrich wrote: >> >> >> >>> Hey all, >> >>> >> >>> In an ticket (INFRA-20416) Gavin asked me if we are completely off >> >>> builds.apache.org - when I went over the jobs I've saw that >> >>> https://builds.apache.org/job/Hive-linux-ARM-trunk/ is running there >> >>> once a day. >> >>> >> >>> Since builds.apache.org will be shut down in sometime in the future >> - we >> >>> should move this job to the new ci-hadoop instance or to >> hive-test-kube. >> >>> The key feature of the job is that it runs the test on the "armX" >> slaves; >> >>> which are statically configured on b.a.o. >> >>> Not sure which way to go - but we will have to move in some direction. >> >>> >> >>> cheers, >> >>> Zoltan >> >>> >> >>> >> >>> On 3/13/20 7:22 AM, Zhenyu Zheng wrote: >> >>>> Hi Chinna, >> >>&
Re: HIVE building on ARM
Hi Zoltan, Thanks for the update. Current https://builds.apache.org/job/Hive-linux-ARM-trunk/ job is targeting to run hive tests daily on "arm" slaves, it is using 2 arm slaves. To find any potential issues with "arm" and fix the issues. So please suggest a way which direction we can move and can you share some details about the new ci-hadoop instance. Thanks, Chinna On Mon, Jun 15, 2020 at 3:56 PM Zoltan Haindrich wrote: > Hey all, > > In an ticket (INFRA-20416) Gavin asked me if we are completely off > builds.apache.org - when I went over the jobs I've saw that > https://builds.apache.org/job/Hive-linux-ARM-trunk/ is running there once > a day. > > Since builds.apache.org will be shut down in sometime in the future - we > should move this job to the new ci-hadoop instance or to hive-test-kube. > The key feature of the job is that it runs the test on the "armX" slaves; > which are statically configured on b.a.o. > Not sure which way to go - but we will have to move in some direction. > > cheers, > Zoltan > > > On 3/13/20 7:22 AM, Zhenyu Zheng wrote: > > Hi Chinna, > > > > Thanks alot for the reply, I uploaded a patch and also a github PR for > > https://issues.apache.org/jira/browse/HIVE-21939 . > > In the patch, I bumped the protobuf used in standalone-metadata to 2.6.1 > > and added a new profile, this profile will identify > > the hardware architecture and if it is Aarch64, it will override the > > protobuf group.id and package to com.github.os72 which > > includes ARM support. For X86 platform, Hive will still download the > > protobuf packages from org.google repo. I think with > > this method, we can keep the influence to existing x86 users to the > > minimum. I hope this could be a acceptable short-term > > solution. > > > > I've manually tested on my machine and the github PR travis CI test has > > already passed, so the build process is OK, so let's > > wait for the full test result from builds.apache.org. > > > > BR, > > > > Zhenyu > > > > On Thu, Mar 12, 2020 at 9:23 PM Chinna Rao Lalam < > lalamchinnara...@gmail.com> > > wrote: > > > >> Hi Zhenyu, > >> > >> Until HBase dependency resolved, without effecting the existing code on > X86 > >> i suggest create a separate profile with "os72" repo. > >> > >> Down the line we should have common version for both X86 and ARM. > >> > >> Hope It Helps, > >> Chinna > >> > >> On Wed, Mar 11, 2020 at 8:39 AM Zhenyu Zheng > > >> wrote: > >> > >>> Hi Chinna, David and others might interested, > >>> > >>> Thanks for bring this up, we are currently working on improving > enabling > >>> big-data software on the ARM platform, > >>> we have already done fixes and providing CIs to some of the well-know > >>> projects like: > >>> 1. Hadoop: > >>> > >>> > >> > https://builds.apache.org/view/H-L/view/Hadoop/job/Hadoop-qbt-linux-ARM-trunk/ > >>> 2. Spark: https://amplab.cs.berkeley.edu/jenkins/label/spark-arm/ > >>> 3. HBase: > >>> https://builds.apache.org/view/H-L/view/HBase/job/HBase-Nightly-ARM/ > >>> > >>> And we are now working on projects including Hive, Kudu, etc. > >>> > >>> Regarding to the protobuf upgrades in Hive, except upgrading to 3.x and > >>> break dependency for HBase, there can > >>> be some possible short-term plan(or walk-arounds), doing thes can make > >> Hive > >>> work on ARM without break any > >>> dependencies, and then we can interact with Hbase project to see how > can > >> we > >>> both upgrade to 3.x(since this > >>> make take some time). > >>> > >>> Those possible solutions can be: > >>> 1. Using pre-patched protobuf 2.5.0 with ARM support > >>> from org.openlabtesting repo, some projects(HBase did > >>> this: https://github.com/apache/hbase/pull/959, and we will add a > >> profile > >>> for this, it will detact the host arch and > >>> will only use this pre-patched protobuf when the host arch is ARM so > that > >>> nothing will be affected for existing x86 > >>> users; Spark and Hadoop used some other package from this repo as they > >> have > >>> already upgraded protobuf). > >>> > >>> 2. Using pre-patched protobuf 2.6.1-build3 from os72's repo: > >>> https://mvnrepository.com/artifact/com.gi
Re: HIVE building on ARM
Hi Zhenyu, Until HBase dependency resolved, without effecting the existing code on X86 i suggest create a separate profile with "os72" repo. Down the line we should have common version for both X86 and ARM. Hope It Helps, Chinna On Wed, Mar 11, 2020 at 8:39 AM Zhenyu Zheng wrote: > Hi Chinna, David and others might interested, > > Thanks for bring this up, we are currently working on improving enabling > big-data software on the ARM platform, > we have already done fixes and providing CIs to some of the well-know > projects like: > 1. Hadoop: > > https://builds.apache.org/view/H-L/view/Hadoop/job/Hadoop-qbt-linux-ARM-trunk/ > 2. Spark: https://amplab.cs.berkeley.edu/jenkins/label/spark-arm/ > 3. HBase: > https://builds.apache.org/view/H-L/view/HBase/job/HBase-Nightly-ARM/ > > And we are now working on projects including Hive, Kudu, etc. > > Regarding to the protobuf upgrades in Hive, except upgrading to 3.x and > break dependency for HBase, there can > be some possible short-term plan(or walk-arounds), doing thes can make Hive > work on ARM without break any > dependencies, and then we can interact with Hbase project to see how can we > both upgrade to 3.x(since this > make take some time). > > Those possible solutions can be: > 1. Using pre-patched protobuf 2.5.0 with ARM support > from org.openlabtesting repo, some projects(HBase did > this: https://github.com/apache/hbase/pull/959, and we will add a profile > for this, it will detact the host arch and > will only use this pre-patched protobuf when the host arch is ARM so that > nothing will be affected for existing x86 > users; Spark and Hadoop used some other package from this repo as they have > already upgraded protobuf). > > 2. Using pre-patched protobuf 2.6.1-build3 from os72's repo: > https://mvnrepository.com/artifact/com.github.os72/protoc/2.6.1-build3 > Hive already used ``protoc-jar-maven-plugin`` from this repo: > > https://github.com/apache/hive/blob/master/standalone-metastore/metastore-common/pom.xml#L484 > > > https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/pom.xml#L472 > so it is a minor update. > > I've tested both ways, worked OK on the ARM machine. > > So my ideas will be that we make this a two-step-work, first step we use > one of the short-term plan to enable > Hive on ARM, and 2nd Step, we look for the long run to discuss with HBase > to find out a fesiable solution. > > BR, > > Zhenyu Zheng > > On Tue, Mar 10, 2020 at 9:29 PM Chinna Rao Lalam < > lalamchinnara...@gmail.com> > wrote: > > > Thanks David for the quick reply. > > Yes currently HBase protobuf version mismatch is the issue. > > > > We are investigating any common version of protobuf is there to support > for > > HBase an HIVE. > > > > > > > > > > On Tue, Mar 10, 2020 at 6:08 PM David Lavati > > > > > wrote: > > > > > Hi, > > > > > > I recently investigated this in > > > https://issues.apache.org/jira/browse/HIVE-20359 and it didn't look > > > feasible, as protobuf has to be in sync across the whole stack, but the > > > external protobuf version for hbase is on 2.5.0 for hbase-2.x, see my > > > comment on the jira and > > > https://github.com/apache/hbase/blob/branch-2.2/pom.xml#L1414. > > > > > > Kind Regards, > > > David > > > > > > On Tue, Mar 10, 2020 at 1:11 PM Chinna Rao Lalam < > > > lalamchinnara...@gmail.com> > > > wrote: > > > > > > > Hi All, > > > > > > > > We are working on HIVE building and running on ARM architecture. As > > part > > > of > > > > this i want to add one periodic HIVE JENKINS JOB on ARM machines. > Which > > > > helps to identity gaps for ARM, If any issues identified we will keep > > on > > > > working to fix those issues. > > > > > > > > Currently to run HIVE on aarch64, we need to upgrade to > protobuf-3.7.1. > > > > HIVE-21939 <https://issues.apache.org/jira/browse/HIVE-21939> > upgraded > > > > protobuf-2.5.0 to protobuf-3.7.1, It have mainly 2 failures. We are > > > working > > > > on this. > > > > > > > > Thanks, > > > > Chinna Rao Lalam > > > > > > > > > > > > > -- > > Hope It Helps, > > Chinna > > > -- Hope It Helps, Chinna
Re: HIVE building on ARM
Thanks David for the quick reply. Yes currently HBase protobuf version mismatch is the issue. We are investigating any common version of protobuf is there to support for HBase an HIVE. On Tue, Mar 10, 2020 at 6:08 PM David Lavati wrote: > Hi, > > I recently investigated this in > https://issues.apache.org/jira/browse/HIVE-20359 and it didn't look > feasible, as protobuf has to be in sync across the whole stack, but the > external protobuf version for hbase is on 2.5.0 for hbase-2.x, see my > comment on the jira and > https://github.com/apache/hbase/blob/branch-2.2/pom.xml#L1414. > > Kind Regards, > David > > On Tue, Mar 10, 2020 at 1:11 PM Chinna Rao Lalam < > lalamchinnara...@gmail.com> > wrote: > > > Hi All, > > > > We are working on HIVE building and running on ARM architecture. As part > of > > this i want to add one periodic HIVE JENKINS JOB on ARM machines. Which > > helps to identity gaps for ARM, If any issues identified we will keep on > > working to fix those issues. > > > > Currently to run HIVE on aarch64, we need to upgrade to protobuf-3.7.1. > > HIVE-21939 <https://issues.apache.org/jira/browse/HIVE-21939> upgraded > > protobuf-2.5.0 to protobuf-3.7.1, It have mainly 2 failures. We are > working > > on this. > > > > Thanks, > > Chinna Rao Lalam > > > -- Hope It Helps, Chinna
HIVE building on ARM
Hi All, We are working on HIVE building and running on ARM architecture. As part of this i want to add one periodic HIVE JENKINS JOB on ARM machines. Which helps to identity gaps for ARM, If any issues identified we will keep on working to fix those issues. Currently to run HIVE on aarch64, we need to upgrade to protobuf-3.7.1. HIVE-21939 <https://issues.apache.org/jira/browse/HIVE-21939> upgraded protobuf-2.5.0 to protobuf-3.7.1, It have mainly 2 failures. We are working on this. Thanks, Chinna Rao Lalam
Re: [Announce] New committer : Laszlo Pinter
Congratulations Laszlo ! On Tue, Feb 11, 2020 at 11:12 AM Anishek Agarwal wrote: > Congratulations Lazlo > > On Tue, Feb 11, 2020 at 9:44 AM Ashutosh Chauhan > wrote: > > > Apache Hive's Project Management Committee (PMC) has invited Laszlo > Pinter > > to become a committer, and we are pleased to announce that he has > accepted. > > > > Laszlo welcome, thank you for your contributions, and we look forward > your > > further interactions with the community! > > > > Thanks, > > Ashutosh > > > -- Hope It Helps, Chinna
Re: Welcome Anishek To Apache Hive PMC
Congratulations Anishek Agarwal ! On Tue, Feb 11, 2020 at 9:40 AM Ashutosh Chauhan wrote: > I'm happy to announce Anishek Agarwal as the latest addition to the Apache > Hive Project Management Committee (PMC). > > He has been an important committer to the project and active member of the > community helping advance Apache Hive. > > Congratulations, and thank you for your hard work > > Thanks, > Ashutosh > -- Hope It Helps, Chinna
Re: Welcome Mahesh to Hive PMC
Congratulations Mahesh Kumar Behera ! On Tue, Feb 11, 2020 at 9:36 AM Ashutosh Chauhan wrote: > Hi all, > > It's an honor to announce that Apache Hive PMC has recently voted to invite > Mahesh Kumar Behera as a new Hive PMC member. Mahesh is a long time Hive > contributor and committer, and has made significant contribution in Hive. > Please join me in congratulating him and looking forward to a bigger role > that he will play in Apache Hive project. > > Thanks, > Ashutosh > -- Hope It Helps, Chinna
Re: Is there any way to find Hive query to Datanucleus queries mapping
Thanks Zoltan for the prompt reply,
I have checked the code with your insights, Yes with this call we can get
the information like below. Using this data we can add a log for each
HIVESql overall how much time spent in metadata operations.
metadata.Hive: Time spent in each metastore function (ms):
{getTableColumnStatistics_(String, String, List, String, )=18,
getNotNullConstraints_(NotNullConstraintsRequest, )=3, getTable_(String,
String, boolean, String, )=41, commitTxn_(long, )=53, getValidTxns_(long,
)=5, isCompatibleWith_(Configuration, )=1, openTxn_(String, TxnType, )=17,
flushCache_()=0, getUniqueConstraints_(UniqueConstraintsRequest, )=2,
getPrimaryKeys_(PrimaryKeysRequest, )=2,
getForeignKeys_(ForeignKeysRequest, )=3}
Thanks,
Chinna
On Mon, Feb 10, 2020 at 5:49 PM Zoltan Haindrich wrote:
> Hey Chinna!
>
> I don't think a mapping like that is easy to get...I would rather try to
> narrow down to a single call which consumes most of the time.
> There is a log message which can help you get to the most relevant
> metastore call:
>
> https://github.com/apache/hive/blob/0d9deba3c15038df4c64ea9b8494d554eb8eea2f/ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java#L5405
>
> cheers,
> Zoltan
>
> On 2/10/20 1:07 PM, Chinna Rao Lalam wrote:
> > Hi All,
> >
> > Is there any way to find Hive query to Datanucleus queries mapping.
> >
> > "select * from table" this hive query will generate multiple Datanucleus
> > queries and execute on configured DB.
> > In our DB some of the queries are running slow, So we want to see
> > hivequery->datanucleus query mapping to find out which hive query of
> > datanucleus query is running slow.
> >
> > If we enable Datanucleus debug log we can see generated queries but not
> > mapping.
> >
> > Thanks
> > Chinna
> >
>
--
Hope It Helps,
Chinna
Is there any way to find Hive query to Datanucleus queries mapping
Hi All, Is there any way to find Hive query to Datanucleus queries mapping. "select * from table" this hive query will generate multiple Datanucleus queries and execute on configured DB. In our DB some of the queries are running slow, So we want to see hivequery->datanucleus query mapping to find out which hive query of datanucleus query is running slow. If we enable Datanucleus debug log we can see generated queries but not mapping. Thanks Chinna
What is the release plan for Hive 4.0.0
Hi all. Do we have any timelines for Hive 4.0.0 release. Thanks, Chinna
Re: [Announce] New committer : David Mollitor
Congratulations David On Fri, 13 Sep 2019 at 12:10 AM, Andrew Sherman wrote: > Congratulations David, you deserve this! > > On Thu, Sep 12, 2019 at 2:19 AM Peter Vary > wrote: > > > Congratulations David! > > > > Ashutosh Chauhan ezt írta (időpont: 2019. szept. > > 12., Csü 1:24): > > > > > Hi, > > > > > > Apache Hive's Project Management Committee (PMC) has invited David > > Mollitor > > > to become a committer, and we are pleased to announce that he has > > accepted. > > > > > > David welcome, thank you for your contributions, and we look forward > your > > > further interactions with the community! > > > > > > Ashutosh Chauhan (on behalf of the Apache Hive PMC) > > > > > > -- Hope It Helps, Chinna
Re: [ANNOUNCE] New committer: Rajkumar Singh
Congratulations Rajkumar Singh. Regards, Chinna Rao Lalam On Fri, Jul 26, 2019 at 9:23 AM Ashutosh Chauhan wrote: > Apache Hive's Project Management Committee (PMC) has invited Rajkumar Singh > to become a committer, and we are pleased to announce that he has accepted. > > Raj welcome, thank you for your contributions, and we look forward your > further interactions with the community! > > Ashutosh Chauhan (on behalf of the Apache Hive PMC) > -- Hope It Helps, Chinna
Re: Welcome new Hive committer, Zhihai Xu
Congratulations Zhihai... On Fri, May 5, 2017 at 10:22 PM, Xuefu Zhang <xu...@apache.org> wrote: > Hi all, > > I'm very please to announce that Hive PMC has recently voted to offer > Zhihai a committership which he accepted. Please join me in congratulating > on this recognition and thanking him for his contributions to Hive. > > Regards, > Xuefu > -- Hope It Helps, Chinna
Review Request 55045: HIVE-15324 : Enable round() function to accept scale argument as non-constants
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/55045/ --- Review request for hive, Ashutosh Chauhan and Xuefu Zhang. Repository: hive-git Description --- Enable round() function to accept scale argument as non-constants Diffs - data/files/round.txt PRE-CREATION data/files/round1.txt PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFRound.java e8b0d15 ql/src/test/queries/clientpositive/udf_round.q 88b2274 ql/src/test/results/clientpositive/udf_round.q.out 456e6ea Diff: https://reviews.apache.org/r/55045/diff/ Testing --- All tests are passed. Thanks, Chinna Rao Lalam
Review Request 55036: Scale is greater than decimal values trunc(d, s) returns wrong results
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/55036/ --- Review request for hive and Ashutosh Chauhan. Repository: hive-git Description --- When scale arguments is greater than the decimal values count, it is returning wrong results. Now returned the actuval value when scale is greater than decimal values count, when scale is positive. Added few more negative tests also. Diffs - ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFTrunc.java a95248f ql/src/test/queries/clientpositive/udf_trunc_number.q b3fd9e5 ql/src/test/results/clientpositive/udf_trunc_number.q.out dfc9d70 Diff: https://reviews.apache.org/r/55036/diff/ Testing --- All tests are passed. Thanks, Chinna Rao Lalam
Re: Invitation for Hive committers to become ORC committers
I would be interested. Thanks. Chinna Rao Lalam On Fri, Dec 16, 2016 at 6:43 AM, Owen O'Malley <omal...@apache.org> wrote: > Ok, I've added the people who have responded so far and updated the ORC > website. > > http://orc.apache.org/news/2016/12/15/new-committers/ > http://orc.apache.org/develop/ > > Please make sure that I didn't typo your names. > > .. Owen > > On Thu, Dec 15, 2016 at 4:44 PM, Chaoyu Tang <ctang...@gmail.com> wrote: > > > I am interested in. Thanks > > > > Chaoyu > > > > On Thu, Dec 15, 2016 at 5:13 PM, Rajesh Balamohan <rbalamo...@apache.org > > > > wrote: > > > > > I would be interested. Thanks. > > > > > > ~Rajesh.B > > > > > > On Fri, Dec 16, 2016 at 3:31 AM, Mithun Radhakrishnan < > > > mithun.radhakrish...@yahoo.com.invalid> wrote: > > > > > > > I'd be keen. > > > > Thanks,Mithun > > > > On Thursday, December 15, 2016, 1:37:36 PM PST, Wei Zheng < > > > > wzh...@hortonworks.com> wrote:I’m interested. Thanks. > > > > > > > > Thanks, > > > > Wei > > > > > > > > On 12/15/16, 13:21, "Vaibhav Gumashta" <vgumas...@hortonworks.com> > > > wrote: > > > > > > > > I¹d be interested. > > > > > > > > Thanks, > > > > ‹Vaibhav > > > > > > > > On 12/15/16, 1:12 PM, "Owen O'Malley" <omal...@apache.org> > wrote: > > > > > > > > >All, > > > > > As you are aware, we are in the last stages of removing the > > forked > > > > ORC > > > > >code out of Hive. The goal of moving ORC out of Hive was to > > increase > > > > its > > > > >community and we want to be very deliberately inclusive of the > > Hive > > > > >development community. Towards that end, the ORC PMC wants to > > > welcome > > > > >anyone who is already a Hive committer to become a committer on > > ORC. > > > > > > > > > > Please respond on this thread to let us know if you are > > > interested. > > > > > > > > > >Thanks, > > > > > Owen on behalf of the ORC PMC > > > > > > > > > > > > > > > > > > > > > > > > > > -- Hope It Helps, Chinna
[jira] [Created] (HIVE-15431) Round(1234567891.1234567891,50) returns null, result is not consistent with Mysql.
Chinna Rao Lalam created HIVE-15431:
---
Summary: Round(1234567891.1234567891,50) returns null, result is
not consistent with Mysql.
Key: HIVE-15431
URL: https://issues.apache.org/jira/browse/HIVE-15431
Project: Hive
Issue Type: Bug
Components: UDF
Reporter: Chinna Rao Lalam
Assignee: Chinna Rao Lalam
round(1234567891.1234567891,50) returns null. Result is not consistent with
Mysql.
Mysql output:
{quote}
select round(1234567891.1234567891,50);
'1234567891.1234567891'
{quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
Re: Review Request 53983: HIVE-14582 : Add trunc(numeric) udf
> On Dec. 1, 2016, 12:05 a.m., Vineet Garg wrote: > > ql/src/test/queries/clientnegative/udf_trunc_error3.q, line 1 > > <https://reviews.apache.org/r/53983/diff/2/?file=1573142#file1573142line1> > > > > I think it'll be good to add tests with negative numbers as well as > > no-op (e.g. select trunc (12.34, 100). Thanks for the review. I will add tests as part of this JIRA HIVE-15325 - Chinna Rao --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/53983/#review157471 --- On Nov. 30, 2016, 7:04 p.m., Chinna Rao Lalam wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/53983/ > --- > > (Updated Nov. 30, 2016, 7:04 p.m.) > > > Review request for hive and Ashutosh Chauhan. > > > Repository: hive-git > > > Description > --- > > Overload trunc() function to accept numbers. > > Now trunc() will accept date or number type arguments and it will behave as > below > > trunc(date, fmt) / trunc(N,D) - Returns > > If input is date returns date with the time portion of the day truncated to > the unit specified by the format model fmt. > If you omit fmt, then date is truncated to "the nearest day. It now only > supports 'MONTH'/'MON'/'MM' and 'YEAR'/''/'YY' as format. > > If input is a number group returns N truncated to D decimal places. If D is > omitted, then N is truncated to 0 places. > D can be negative to truncate (make zero) D digits left of the decimal point. > > > Diffs > - > > data/files/trunc_number.txt PRE-CREATION > data/files/trunc_number1.txt PRE-CREATION > ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFTrunc.java > e20ad65 > ql/src/test/queries/clientnegative/udf_trunc_error3.q PRE-CREATION > ql/src/test/queries/clientpositive/udf_trunc_number.q PRE-CREATION > ql/src/test/results/clientnegative/udf_trunc_error1.q.out 5d65b11 > ql/src/test/results/clientnegative/udf_trunc_error2.q.out 55a2185 > ql/src/test/results/clientnegative/udf_trunc_error3.q.out PRE-CREATION > ql/src/test/results/clientpositive/udf_trunc.q.out 4c9f76d > ql/src/test/results/clientpositive/udf_trunc_number.q.out PRE-CREATION > > Diff: https://reviews.apache.org/r/53983/diff/ > > > Testing > --- > > All tests are pass. > > > Thanks, > > Chinna Rao Lalam > >
[jira] [Created] (HIVE-15325) Add tests with negative numbers as well as no-op tests
Chinna Rao Lalam created HIVE-15325: --- Summary: Add tests with negative numbers as well as no-op tests Key: HIVE-15325 URL: https://issues.apache.org/jira/browse/HIVE-15325 Project: Hive Issue Type: Bug Components: UDF Reporter: Chinna Rao Lalam Assignee: Chinna Rao Lalam Priority: Minor Add tests with negative numbers as well as no-op (e.g. select trunc (12.34, 100)) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-15324) Enable round() function to accept scale argument as non-constants
Chinna Rao Lalam created HIVE-15324:
---
Summary: Enable round() function to accept scale argument as
non-constants
Key: HIVE-15324
URL: https://issues.apache.org/jira/browse/HIVE-15324
Project: Hive
Issue Type: Bug
Components: UDF
Reporter: Chinna Rao Lalam
Assignee: Chinna Rao Lalam
round() function should accept scale argument as non-constants, it will enable
queries like:
{quote}
create table sampletable(c double, d int);
select round(c,d) from sampletable;
{quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HIVE-14571) Document configuration hive.msck.repair.batch.size
Chinna Rao Lalam created HIVE-14571:
---
Summary: Document configuration hive.msck.repair.batch.size
Key: HIVE-14571
URL: https://issues.apache.org/jira/browse/HIVE-14571
Project: Hive
Issue Type: Improvement
Components: Documentation
Reporter: Chinna Rao Lalam
Assignee: Chinna Rao Lalam
Priority: Minor
Fix For: 2.2.0
Update here
[https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DDL#LanguageManualDDL-RecoverPartitions(MSCKREPAIRTABLE)]
{quote}
When there is a large number of untracked partitions for the MSCK REPAIR TABLE
command, there is a provision to run the msck repair table batch wise to avoid
OOME. By giving the configured batch size for the property
*hive.msck.repair.batch.size* it can run in the batches internally. The default
value of the property is zero, it means it will execute all the partitions at
one short.
{quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HIVE-14032) INSERT OVERWRITE command failed with sensitive names.
Chinna Rao Lalam created HIVE-14032: --- Summary: INSERT OVERWRITE command failed with sensitive names. Key: HIVE-14032 URL: https://issues.apache.org/jira/browse/HIVE-14032 Project: Hive Issue Type: Bug Components: Metastore Affects Versions: 2.0.1 Reporter: Chinna Rao Lalam Assignee: Chinna Rao Lalam -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: [ANNOUNCE] New Hive Committer - Yongzhi Chen
Congrats Yongzhi! On Tue, Nov 3, 2015 at 12:37 AM, Lefty Leverenz <leftylever...@gmail.com> wrote: > Congratulations Yongzhi! > > -- Lefty > > On Mon, Nov 2, 2015 at 1:19 PM, Vaibhav Gumashta < > vgumas...@hortonworks.com> > wrote: > > > Congrats Yongzhi! > > > > ‹Vaibhav > > > > On 11/2/15, 9:56 AM, "Zhuoluo (Clark) Yang" <yangzhuo...@gmail.com> > wrote: > > > > >Congrats > > > > > >Thanks, > > >Zhuoluo (Clark) Yang > > > > > >On Mon, Nov 2, 2015 at 9:54 AM, Jimmy Xiang <jxi...@cloudera.com> > wrote: > > > > > >> Congrats! > > >> > > >> On Mon, Nov 2, 2015 at 9:43 AM, Xuefu Zhang <xzh...@cloudera.com> > > wrote: > > >> > > >> > Hi all, > > >> > > > >> > Apache Hive PMC has just voted to make Yongzhi Chen a new committer > on > > >> > Apache Hive Project. > > >> > > > >> > Please join me in congratulating Yongzhi! > > >> > > > >> > Thanks, > > >> > > > >> > Xuefu > > >> > > > >> > > > > > -- Hope It Helps, Chinna
Re: [ANNOUNCE] New Hive Committer - Siddharth Seth
Congratulations! On Thu, Oct 22, 2015 at 12:18 PM, Santlal J Gupta < santlal.gu...@bitwiseglobal.com> wrote: > Congratulations !! > > -Original Message- > From: Chetna C [mailto:chetna@gmail.com] > Sent: Thursday, October 22, 2015 8:58 AM > To: dev@hive.apache.org > Cc: Siddharth Seth > Subject: Re: [ANNOUNCE] New Hive Committer - Siddharth Seth > > Congratulations !! > On Oct 22, 2015 5:13 AM, "Pengcheng Xiong" <pxi...@apache.org> wrote: > > > Congrats Sid! > > > > On Wed, Oct 21, 2015 at 2:14 PM, Sergey Shelukhin > > <ser...@hortonworks.com> > > wrote: > > > > > The Apache Hive PMC has voted to make Siddharth Seth a committer on > > > the Apache Hive Project. > > > > > > Please join me in congratulating Sid! > > > > > > Thanks, > > > Sergey. > > > > > > > > > -- Hope It Helps, Chinna
Re: [ANNOUNCE] New Hive Committer- Aihua Xu
Congrats Aihua! On Thu, Oct 22, 2015 at 10:45 AM, Vaibhav Gumashta < vgumas...@hortonworks.com> wrote: > Congrats Aihua! > > ‹Vaibhav > > On 10/21/15, 4:42 PM, "Pengcheng Xiong" <pxi...@apache.org> wrote: > > >Congrats Aihua! > > > >On Wed, Oct 21, 2015 at 2:09 PM, Szehon Ho <sze...@cloudera.com> wrote: > > > >> The Apache Hive PMC has voted to make Aihua Xu a committer on the Apache > >> Hive Project. > >> > >> Please join me in congratulating Aihua! > >> > >> Thanks, > >> Szehon > >> > > -- Hope It Helps, Chinna
Re: [ANNOUNCE] New Hive Committers - Jesus Camacho Rodriguez and Chinna Rao Lalam
Thank you everyone. I'm excited to continue contributing to the Hive community. Congrats to Jesus. Regards, Chinna On Sat, Jun 27, 2015 at 11:18 AM, Lefty Leverenz leftylever...@gmail.com wrote: Congratulations China and Jesus, and thanks for all your contributions! -- Lefty On Fri, Jun 26, 2015 at 7:01 PM, Sergio Pena sergio.p...@cloudera.com wrote: Congratulations China and Jesus !!!. - Sergio On Fri, Jun 26, 2015 at 1:57 PM, Carl Steinbach c...@apache.org wrote: On behalf of the Apache Hive PMC I am pleased to announce that Jesus Camacho Rodriguez and Chinna Rao Lalam have been voted in as committers. Please join me in congratulating Jesus and Chinna! Thanks. - Carl -- Hope It Helps, Chinna
[jira] [Created] (HIVE-10905) QuitExit fails ending with ';' [beeline-cli Branch]
Chinna Rao Lalam created HIVE-10905: --- Summary: QuitExit fails ending with ';' [beeline-cli Branch] Key: HIVE-10905 URL: https://issues.apache.org/jira/browse/HIVE-10905 Project: Hive Issue Type: Bug Affects Versions: beeline-cli-branch Reporter: Chinna Rao Lalam Assignee: Chinna Rao Lalam In CLI quit and exit will expect ending ';' In Updated CLI quit and exit without ending ; is working. quit and exit ending with ';' throwing exception. Support quit and exit with ending ';' for the compatibility; -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-10904) Use beeline-log4j.properties for migrated CLI [beeline-cli Branch]
Chinna Rao Lalam created HIVE-10904: --- Summary: Use beeline-log4j.properties for migrated CLI [beeline-cli Branch] Key: HIVE-10904 URL: https://issues.apache.org/jira/browse/HIVE-10904 Project: Hive Issue Type: Bug Affects Versions: beeline-cli-branch Reporter: Chinna Rao Lalam Assignee: Chinna Rao Lalam Updated CLI printing logs on the console. Use beeline-log4j.properties for redirecting to file. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-10847) Support CLI specific configurations with beeline functionality
Chinna Rao Lalam created HIVE-10847:
---
Summary: Support CLI specific configurations with beeline
functionality
Key: HIVE-10847
URL: https://issues.apache.org/jira/browse/HIVE-10847
Project: Hive
Issue Type: Sub-task
Affects Versions: beeline-cli-branch
Reporter: Chinna Rao Lalam
Assignee: Chinna Rao Lalam
CLI have some specific configurations these need to supported in beeline
functionality.
Some of those configurations..
{quote}
hive.cli.errors.ignore
hive.cli.print.current.db
hive.cli.prompt
hive.cli.pretty.output.num.cols
{quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HIVE-10822) CLI start script throwing error message on console
Chinna Rao Lalam created HIVE-10822:
---
Summary: CLI start script throwing error message on console
Key: HIVE-10822
URL: https://issues.apache.org/jira/browse/HIVE-10822
Project: Hive
Issue Type: Sub-task
Components: CLI
Affects Versions: beeline-cli-branch
Reporter: Chinna Rao Lalam
Assignee: Chinna Rao Lalam
Starting cli throwing following message on console
{noformat}
[chinna@stobdtserver1 bin]$ ./hive
./ext/cli.sh: line 20: [: ==: unary operator expected
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HIVE-10823) CLI start script throwing error message on console
Chinna Rao Lalam created HIVE-10823:
---
Summary: CLI start script throwing error message on console
Key: HIVE-10823
URL: https://issues.apache.org/jira/browse/HIVE-10823
Project: Hive
Issue Type: Sub-task
Components: CLI
Affects Versions: beeline-cli-branch
Reporter: Chinna Rao Lalam
Assignee: Chinna Rao Lalam
Starting cli throwing following message on console
{noformat}
[chinna@stobdtserver1 bin]$ ./hive
./ext/cli.sh: line 20: [: ==: unary operator expected
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HIVE-10824) Need to update start script changes in .cmd files
Chinna Rao Lalam created HIVE-10824: --- Summary: Need to update start script changes in .cmd files Key: HIVE-10824 URL: https://issues.apache.org/jira/browse/HIVE-10824 Project: Hive Issue Type: Sub-task Affects Versions: beeline-cli-branch Reporter: Chinna Rao Lalam Assignee: Chinna Rao Lalam Need to updated start script changes in .cmd files -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: [ANNOUNCE] New Hive Committer - Chaoyu Tang
Congratulations Chaoyu. On Thu, May 21, 2015 at 9:42 AM, Naveen Gangam ngan...@cloudera.com wrote: Congrats Chaoyu, Well deserved indeed!!! On Wed, May 20, 2015 at 10:04 PM, Yongzhi Chen yc...@cloudera.com wrote: Congratulations Chaoyu! On Wed, May 20, 2015 at 8:10 PM, Lefty Leverenz leftylever...@gmail.com wrote: Congratulations Chaoyu! Thanks for all your contributions. -- Lefty On Wed, May 20, 2015 at 4:11 PM, Hari Subramaniyan hsubramani...@hortonworks.com wrote: Congrats Chaoyu! From: Lenni Kuff lsk...@cloudera.com Sent: Wednesday, May 20, 2015 4:08 PM To: dev@hive.apache.org Subject: Re: [ANNOUNCE] New Hive Committer - Chaoyu Tang Congrats Chaoyu! Well deserved. On Wed, May 20, 2015 at 4:07 PM, Sushanth Sowmyan khorg...@gmail.com wrote: Congrats Chaoyu, welcome aboard! :) On May 20, 2015 3:45 PM, Vaibhav Gumashta vgumas...@hortonworks.com wrote: Congratulations! ‹Vaibhav On 5/20/15, 3:40 PM, Jimmy Xiang jxi...@cloudera.com wrote: Congrats!! On Wed, May 20, 2015 at 3:29 PM, Carl Steinbach c...@apache.org wrote: The Apache Hive PMC has voted to make Chaoyu Tang a committer on the Apache Hive Project. Please join me in congratulating Chaoyu! Thanks. - Carl -- Hope It Helps, Chinna
[jira] [Created] (HIVE-10626) Spark paln need to be updated [Spark Branch]
Chinna Rao Lalam created HIVE-10626: --- Summary: Spark paln need to be updated [Spark Branch] Key: HIVE-10626 URL: https://issues.apache.org/jira/browse/HIVE-10626 Project: Hive Issue Type: Bug Components: Spark Affects Versions: spark-branch Reporter: Chinna Rao Lalam Assignee: Chinna Rao Lalam [HIVE-8858] basic patch was committed, latest patch need to be committed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: Review Request 33211: [HIVE-10288] : permanent UDFs are not working after restarting the server
On April 27, 2015, 9:31 a.m., Amareshwari Sriramadasu wrote: trunk/ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java, line 179 https://reviews.apache.org/r/33211/diff/1/?file=929630#file929630line179 I did not understand how the changes are related to restart? We have not faced in our setup. This issue was faced in trunk version.. After restarting the service while loading functions from DB, type was not determined and null was returned. - Chinna --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/33211/#review81667 --- On April 15, 2015, 6:04 a.m., Chinna Lalam wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/33211/ --- (Updated April 15, 2015, 6:04 a.m.) Review request for hive, Jason Dere and Navis Ryu. Repository: hive Description --- While reloading the permanent functions we need to add udf type (genericUDF/genericUDTF/genericUDAFResolver). Currently this information is not there, so it is throwing NullpointerException. We have download the related jars when using this functions. Diffs - trunk/ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java 1673646 Diff: https://reviews.apache.org/r/33211/diff/ Testing --- Related tests are pass. Thanks, Chinna Lalam
[jira] [Created] (HIVE-10415) hive.start.cleanup.scratchdir configuration is not taking effect
Chinna Rao Lalam created HIVE-10415: --- Summary: hive.start.cleanup.scratchdir configuration is not taking effect Key: HIVE-10415 URL: https://issues.apache.org/jira/browse/HIVE-10415 Project: Hive Issue Type: Bug Reporter: Chinna Rao Lalam Assignee: Chinna Rao Lalam Fix For: 1.2.0 This configuration hive.start.cleanup.scratchdir is not taking effect -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Review Request 33259: [HIVE-10277] : Unable to process Comment line '--'
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/33259/ --- Review request for hive and Szehon Ho. Repository: hive Description --- commented line are skipped before processing the command. Diffs - trunk/cli/src/java/org/apache/hadoop/hive/cli/CliDriver.java 1673646 Diff: https://reviews.apache.org/r/33259/diff/ Testing --- Related tests are passed. Thanks, Chinna Lalam
Review Request 33211: [HIVE-10288] : permanent UDFs are not working after restarting the server
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/33211/ --- Review request for hive, Jason Dere and Navis Ryu. Repository: hive Description --- While reloading the permanent functions we need to add udf type (genericUDF/genericUDTF/genericUDAFResolver). Currently this information is not there, so it is throwing NullpointerException. We have download the related jars when using this functions. Diffs - trunk/ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java 1673646 Diff: https://reviews.apache.org/r/33211/diff/ Testing --- Related tests are pass. Thanks, Chinna Lalam
Re: cannot call permanent udfs with the latest trunk
Hi, I have uploaded a patch for HIVE-10288.. Please try this.. Hope It Helps, Chinna On Tue, Apr 14, 2015 at 5:37 AM, Nezih Yigitbasi nyigitb...@netflix.com.invalid wrote: Hey everyone, With the latest trunk I cannot call permanent UDFs, please see HIVE-10288 https://issues.apache.org/jira/browse/HIVE-10288 for details. Will be happy to fix it, but still trying to understand what the root cause is. Any ideas? Thanks, Nezih -- Hope It Helps, Chinna
[jira] [Created] (HIVE-10058) Log the information of cached RDD [Spark Branch]
Chinna Rao Lalam created HIVE-10058: --- Summary: Log the information of cached RDD [Spark Branch] Key: HIVE-10058 URL: https://issues.apache.org/jira/browse/HIVE-10058 Project: Hive Issue Type: Sub-task Components: Spark Reporter: Chinna Rao Lalam Assignee: Chinna Rao Lalam Fix For: spark-branch Log the cached RDD Id's at info level. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
RE: ORC separate project
Hi Owen, I'd like to get involved. Please add me as well. Thanks, Chinna Rao Lalam -- Forwarded message -- From: Owen O'Malley omal...@apache.orgmailto:omal...@apache.org Date: Fri, Mar 20, 2015 at 3:14 AM Subject: ORC separate project To: dev@hive.apache.orgmailto:dev@hive.apache.org dev@hive.apache.orgmailto:dev@hive.apache.org, Lefty Leverenz leftylever...@gmail.commailto:leftylever...@gmail.com All, Over the last year, there has been a fair number of projects that want to integrate with ORC, but don't want a dependence on Hive's exec jar. Additionally, we've been working on a C++ reader (and soon writer) and it would be great to host them both in the same project. Toward that end, I'd like to create a separate ORC project at Apache. There will be lots of technical details to work out, but I wanted to give the Hive community a chance to discuss it. Do any of the Hive committers want to be included on the proposal? Of the current Hive committers, my list looks like: * Alan * Gunther * Prasanth * Lefty * Owen * Sergey * Gopal * Kevin Did I miss anyone? Thanks! Owen
Re: [ANNOUNCE] New Hive Committers - Jimmy Xiang, Matt McCline, and Sergio Pena
Congratulations to all... On Mon, Mar 23, 2015 at 11:38 PM, Carl Steinbach c...@apache.org wrote: The Apache Hive PMC has voted to make Jimmy Xiang, Matt McCline, and Sergio Pena committers on the Apache Hive Project. Please join me in congratulating Jimmy, Matt, and Sergio. Thanks. - Carl -- Hope It Helps, Chinna
[jira] [Created] (HIVE-10017) SparkTask log improvement [Spark Branch]
Chinna Rao Lalam created HIVE-10017: --- Summary: SparkTask log improvement [Spark Branch] Key: HIVE-10017 URL: https://issues.apache.org/jira/browse/HIVE-10017 Project: Hive Issue Type: Bug Components: Spark Reporter: Chinna Rao Lalam Priority: Minor Fix For: spark-branch Initialize log object in the own class for better log message. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-9939) Code cleanup for redundant if check in ExplainTask
Chinna Rao Lalam created HIVE-9939: -- Summary: Code cleanup for redundant if check in ExplainTask Key: HIVE-9939 URL: https://issues.apache.org/jira/browse/HIVE-9939 Project: Hive Issue Type: Bug Reporter: Chinna Rao Lalam Assignee: Chinna Rao Lalam Fix For: spark-branch ExplainTask.execute() method have redundant if check. Same applicable for trunk also.. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-9871) Print spark job id in history file [spark branch]
Chinna Rao Lalam created HIVE-9871: -- Summary: Print spark job id in history file [spark branch] Key: HIVE-9871 URL: https://issues.apache.org/jira/browse/HIVE-9871 Project: Hive Issue Type: Sub-task Reporter: Chinna Rao Lalam Assignee: Chinna Rao Lalam Maintain the spark job id in history file for the corresponding queries. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9638) Drop Index does not check Index or Table exisit or not
[ https://issues.apache.org/jira/browse/HIVE-9638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14319806#comment-14319806 ] Chinna Rao Lalam commented on HIVE-9638: Hi, In Hive 0.7.0 or later, DROP returns an error if the index doesn't exist, unless IF EXISTS is specified or the configuration variable hive.exec.drop.ignorenonexistent is set to true. Drop Index does not check Index or Table exisit or not -- Key: HIVE-9638 URL: https://issues.apache.org/jira/browse/HIVE-9638 Project: Hive Issue Type: Bug Components: Parser Affects Versions: 0.11.0, 0.13.0, 0.14.0, 1.0.0 Reporter: Will Du DROP INDEX index_name ON table_name; statement will be always successful no matter the index_name or table_name exsit -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-9516) Enable CBO related tests [Spark Branch]
[ https://issues.apache.org/jira/browse/HIVE-9516?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chinna Rao Lalam updated HIVE-9516: --- Status: Open (was: Patch Available) Failed tests results order is not consistent. Need to add orderby in these queries. Enable CBO related tests [Spark Branch] --- Key: HIVE-9516 URL: https://issues.apache.org/jira/browse/HIVE-9516 Project: Hive Issue Type: Sub-task Components: spark-branch Affects Versions: spark-branch Reporter: Chao Assignee: Chinna Rao Lalam Attachments: HIVE-9516.1-spark.patch In Spark branch we enabled CBO, but hasn't turned on CBO related unit tests. We should do this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (HIVE-9516) Enable CBO related tests [Spark Branch]
[ https://issues.apache.org/jira/browse/HIVE-9516?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chinna Rao Lalam reassigned HIVE-9516: -- Assignee: Chinna Rao Lalam Enable CBO related tests [Spark Branch] --- Key: HIVE-9516 URL: https://issues.apache.org/jira/browse/HIVE-9516 Project: Hive Issue Type: Sub-task Components: spark-branch Affects Versions: spark-branch Reporter: Chao Assignee: Chinna Rao Lalam In Spark branch we enabled CBO, but hasn't turned on CBO related unit tests. We should do this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-9516) Enable CBO related tests [Spark Branch]
[ https://issues.apache.org/jira/browse/HIVE-9516?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chinna Rao Lalam updated HIVE-9516: --- Attachment: HIVE-9516.1-spark.patch Enable CBO related tests [Spark Branch] --- Key: HIVE-9516 URL: https://issues.apache.org/jira/browse/HIVE-9516 Project: Hive Issue Type: Sub-task Components: spark-branch Affects Versions: spark-branch Reporter: Chao Assignee: Chinna Rao Lalam Attachments: HIVE-9516.1-spark.patch In Spark branch we enabled CBO, but hasn't turned on CBO related unit tests. We should do this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-9516) Enable CBO related tests [Spark Branch]
[ https://issues.apache.org/jira/browse/HIVE-9516?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chinna Rao Lalam updated HIVE-9516: --- Status: Patch Available (was: Open) Enabled some of the CBO related tests covering all features. Enable CBO related tests [Spark Branch] --- Key: HIVE-9516 URL: https://issues.apache.org/jira/browse/HIVE-9516 Project: Hive Issue Type: Sub-task Components: spark-branch Affects Versions: spark-branch Reporter: Chao Assignee: Chinna Rao Lalam Attachments: HIVE-9516.1-spark.patch In Spark branch we enabled CBO, but hasn't turned on CBO related unit tests. We should do this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-9170) UT: udf_in_file fails with filenotfoundexception
[ https://issues.apache.org/jira/browse/HIVE-9170?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chinna Rao Lalam updated HIVE-9170: --- Attachment: HIVE-9170.1-spark.patch UT: udf_in_file fails with filenotfoundexception Key: HIVE-9170 URL: https://issues.apache.org/jira/browse/HIVE-9170 Project: Hive Issue Type: Sub-task Components: Tests Affects Versions: spark-branch Reporter: Thomas Friedrich Assignee: Chinna Rao Lalam Priority: Minor Attachments: HIVE-9170.1-spark.patch The test case references ../../data/files/test2.dat, but for some reasons it can't find the file: 2014-12-18 17:59:46,360 ERROR [main]: CliDriver (SessionState.java:printError(834)) - Failed with exception java.io.IOException:org.apache.hadoop.hive.ql.metadata.HiveException: java.io.FileNotFoundException: test2.dat (No such file or directory) java.io.IOException: org.apache.hadoop.hive.ql.metadata.HiveException: java.io.FileNotFoundException: test2.dat (No such file or directory) at org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:152) at org.apache.hadoop.hive.ql.Driver.getResults(Driver.java:1648) at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:226) at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:158) at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:369) at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:304) at org.apache.hadoop.hive.ql.QTestUtil.executeClient(QTestUtil.java:837) at org.apache.hadoop.hive.cli.TestSparkCliDriver.runTest(TestSparkCliDriver.java:136) at org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_udf_in_file(TestSparkCliDriver.java:120) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-9170) UT: udf_in_file fails with filenotfoundexception
[ https://issues.apache.org/jira/browse/HIVE-9170?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chinna Rao Lalam updated HIVE-9170: --- Status: Patch Available (was: Open) This testcase is working. Enable this test. UT: udf_in_file fails with filenotfoundexception Key: HIVE-9170 URL: https://issues.apache.org/jira/browse/HIVE-9170 Project: Hive Issue Type: Sub-task Components: Tests Affects Versions: spark-branch Reporter: Thomas Friedrich Assignee: Chinna Rao Lalam Priority: Minor Attachments: HIVE-9170.1-spark.patch The test case references ../../data/files/test2.dat, but for some reasons it can't find the file: 2014-12-18 17:59:46,360 ERROR [main]: CliDriver (SessionState.java:printError(834)) - Failed with exception java.io.IOException:org.apache.hadoop.hive.ql.metadata.HiveException: java.io.FileNotFoundException: test2.dat (No such file or directory) java.io.IOException: org.apache.hadoop.hive.ql.metadata.HiveException: java.io.FileNotFoundException: test2.dat (No such file or directory) at org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:152) at org.apache.hadoop.hive.ql.Driver.getResults(Driver.java:1648) at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:226) at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:158) at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:369) at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:304) at org.apache.hadoop.hive.ql.QTestUtil.executeClient(QTestUtil.java:837) at org.apache.hadoop.hive.cli.TestSparkCliDriver.runTest(TestSparkCliDriver.java:136) at org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_udf_in_file(TestSparkCliDriver.java:120) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (HIVE-9170) UT: udf_in_file fails with filenotfoundexception
[ https://issues.apache.org/jira/browse/HIVE-9170?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chinna Rao Lalam reassigned HIVE-9170: -- Assignee: Chinna Rao Lalam UT: udf_in_file fails with filenotfoundexception Key: HIVE-9170 URL: https://issues.apache.org/jira/browse/HIVE-9170 Project: Hive Issue Type: Sub-task Components: Tests Affects Versions: spark-branch Reporter: Thomas Friedrich Assignee: Chinna Rao Lalam Priority: Minor The test case references ../../data/files/test2.dat, but for some reasons it can't find the file: 2014-12-18 17:59:46,360 ERROR [main]: CliDriver (SessionState.java:printError(834)) - Failed with exception java.io.IOException:org.apache.hadoop.hive.ql.metadata.HiveException: java.io.FileNotFoundException: test2.dat (No such file or directory) java.io.IOException: org.apache.hadoop.hive.ql.metadata.HiveException: java.io.FileNotFoundException: test2.dat (No such file or directory) at org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:152) at org.apache.hadoop.hive.ql.Driver.getResults(Driver.java:1648) at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:226) at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:158) at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:369) at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:304) at org.apache.hadoop.hive.ql.QTestUtil.executeClient(QTestUtil.java:837) at org.apache.hadoop.hive.cli.TestSparkCliDriver.runTest(TestSparkCliDriver.java:136) at org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_udf_in_file(TestSparkCliDriver.java:120) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-9289) TODO : Store user name in session [Spark Branch]
[ https://issues.apache.org/jira/browse/HIVE-9289?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chinna Rao Lalam updated HIVE-9289: --- Attachment: HIVE-9289.2-spark.patch TODO : Store user name in session [Spark Branch] Key: HIVE-9289 URL: https://issues.apache.org/jira/browse/HIVE-9289 Project: Hive Issue Type: Bug Components: Spark Reporter: Chinna Rao Lalam Assignee: Chinna Rao Lalam Attachments: HIVE-9289.1-spark.patch, HIVE-9289.2-spark.patch TODO : this we need to store the session username somewhere else as getUGIForConf never used the conf SparkSessionManagerImpl.java /hive-exec/src/java/org/apache/hadoop/hive/ql/exec/spark/session line 145 Java Task -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-9289) TODO : Store user name in session [Spark Branch]
[ https://issues.apache.org/jira/browse/HIVE-9289?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chinna Rao Lalam updated HIVE-9289: --- Status: Open (was: Patch Available) TODO : Store user name in session [Spark Branch] Key: HIVE-9289 URL: https://issues.apache.org/jira/browse/HIVE-9289 Project: Hive Issue Type: Bug Components: Spark Reporter: Chinna Rao Lalam Assignee: Chinna Rao Lalam Attachments: HIVE-9289.1-spark.patch TODO : this we need to store the session username somewhere else as getUGIForConf never used the conf SparkSessionManagerImpl.java /hive-exec/src/java/org/apache/hadoop/hive/ql/exec/spark/session line 145 Java Task -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-9289) TODO : Store user name in session [Spark Branch]
[ https://issues.apache.org/jira/browse/HIVE-9289?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chinna Rao Lalam updated HIVE-9289: --- Status: Patch Available (was: Open) TODO : Store user name in session [Spark Branch] Key: HIVE-9289 URL: https://issues.apache.org/jira/browse/HIVE-9289 Project: Hive Issue Type: Bug Components: Spark Reporter: Chinna Rao Lalam Assignee: Chinna Rao Lalam Attachments: HIVE-9289.1-spark.patch, HIVE-9289.2-spark.patch TODO : this we need to store the session username somewhere else as getUGIForConf never used the conf SparkSessionManagerImpl.java /hive-exec/src/java/org/apache/hadoop/hive/ql/exec/spark/session line 145 Java Task -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9289) TODO : Store user name in session [Spark Branch]
[ https://issues.apache.org/jira/browse/HIVE-9289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14286161#comment-14286161 ] Chinna Rao Lalam commented on HIVE-9289: I have verified this code, reusing the session is not happening because as [~chengxiang li] explained Hive Client-SessionHandler(session id inside)- HiveSessionImpl-SessionState-SparkSession this linear mapping is maintained. Updated the patch by removing that code. TODO : Store user name in session [Spark Branch] Key: HIVE-9289 URL: https://issues.apache.org/jira/browse/HIVE-9289 Project: Hive Issue Type: Bug Components: Spark Reporter: Chinna Rao Lalam Assignee: Chinna Rao Lalam Attachments: HIVE-9289.1-spark.patch, HIVE-9289.2-spark.patch TODO : this we need to store the session username somewhere else as getUGIForConf never used the conf SparkSessionManagerImpl.java /hive-exec/src/java/org/apache/hadoop/hive/ql/exec/spark/session line 145 Java Task -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-9289) TODO : Store user name in session [Spark Branch]
Chinna Rao Lalam created HIVE-9289: -- Summary: TODO : Store user name in session [Spark Branch] Key: HIVE-9289 URL: https://issues.apache.org/jira/browse/HIVE-9289 Project: Hive Issue Type: Bug Components: Spark Reporter: Chinna Rao Lalam Assignee: Chinna Rao Lalam TODO : this we need to store the session username somewhere else as getUGIForConf never used the conf SparkSessionManagerImpl.java /hive-exec/src/java/org/apache/hadoop/hive/ql/exec/spark/session line 145 Java Task -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-9289) TODO : Store user name in session [Spark Branch]
[ https://issues.apache.org/jira/browse/HIVE-9289?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chinna Rao Lalam updated HIVE-9289: --- Attachment: HIVE-9289.1-spark.patch TODO : Store user name in session [Spark Branch] Key: HIVE-9289 URL: https://issues.apache.org/jira/browse/HIVE-9289 Project: Hive Issue Type: Bug Components: Spark Reporter: Chinna Rao Lalam Assignee: Chinna Rao Lalam Attachments: HIVE-9289.1-spark.patch TODO : this we need to store the session username somewhere else as getUGIForConf never used the conf SparkSessionManagerImpl.java /hive-exec/src/java/org/apache/hadoop/hive/ql/exec/spark/session line 145 Java Task -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-9289) TODO : Store user name in session [Spark Branch]
[ https://issues.apache.org/jira/browse/HIVE-9289?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chinna Rao Lalam updated HIVE-9289: --- Status: Patch Available (was: Open) Now maintaining user name in the session. RB: https://reviews.apache.org/r/29658/ TODO : Store user name in session [Spark Branch] Key: HIVE-9289 URL: https://issues.apache.org/jira/browse/HIVE-9289 Project: Hive Issue Type: Bug Components: Spark Reporter: Chinna Rao Lalam Assignee: Chinna Rao Lalam Attachments: HIVE-9289.1-spark.patch TODO : this we need to store the session username somewhere else as getUGIForConf never used the conf SparkSessionManagerImpl.java /hive-exec/src/java/org/apache/hadoop/hive/ql/exec/spark/session line 145 Java Task -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9282) hive could not able to integrate with spark
[ https://issues.apache.org/jira/browse/HIVE-9282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14267316#comment-14267316 ] Chinna Rao Lalam commented on HIVE-9282: Hive on spark is not available in hive-0.12.0. Currently work is happening on spark branch of hive. Soon it will be merged to trunk. hive could not able to integrate with spark --- Key: HIVE-9282 URL: https://issues.apache.org/jira/browse/HIVE-9282 Project: Hive Issue Type: Bug Components: Spark Affects Versions: 0.12.0 Environment: centOS 6.4 and hadoop-1.0.4 and hive-0.12.0 and spark-0.8.0 Reporter: suraj Fix For: spark-branch Original Estimate: 12h Remaining Estimate: 12h i have installed hadoop-1.0.4 and on top this i have installed everything by just following this site : https://cwiki.apache.org/confluence/display/Hive/Hive+on+Spark%3A+Getting+Started hive-0.12.0 and spark-0.8.0 in that site they have mentioned that i have to install spark-1.2.x assembly but i have installed spark as spark-0.8.0 even i have compiled hive library using maven still i am getting issue that wrong FS. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-8508) UT: fix bucketsort_insert tests - related to SMBMapJoinOperator
[
https://issues.apache.org/jira/browse/HIVE-8508?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chinna Rao Lalam updated HIVE-8508:
---
Attachment: HIVE-8508.1-spark.patch
UT: fix bucketsort_insert tests - related to SMBMapJoinOperator
---
Key: HIVE-8508
URL: https://issues.apache.org/jira/browse/HIVE-8508
Project: Hive
Issue Type: Sub-task
Components: Spark
Reporter: Thomas Friedrich
Assignee: Chinna Rao Lalam
Attachments: HIVE-8508.1-spark.patch
The 4 tests
bucketsortoptimize_insert_2
bucketsortoptimize_insert_4
bucketsortoptimize_insert_6
bucketsortoptimize_insert_7
bucketsortoptimize_insert_8
all fail with the same NPE related in SMBMapJoinOperator:
order object is null in SMBMapJoinOperator:
// fetch the first group for all small table aliases
for (byte pos = 0; pos order.length; pos++) {
if (pos != posBigTable)
{ fetchNextGroup(pos); }
Daemon Thread [Executor task launch worker-3] (Suspended (exception
NullPointerException))
SMBMapJoinOperator.processOp(Object, int) line: 258
FilterOperator(OperatorT).forward(Object, ObjectInspector) line: 799
FilterOperator.processOp(Object, int) line: 137
TableScanOperator(OperatorT).forward(Object, ObjectInspector) line: 799
TableScanOperator.processOp(Object, int) line: 95
MapOperator(OperatorT).forward(Object, ObjectInspector) line: 799
MapOperator.process(Writable) line: 536
SparkMapRecordHandler.processRow(Object, Object) line: 139
HiveMapFunctionResultList.processNextRecord(Tuple2BytesWritable,BytesWritable)
line: 47
HiveMapFunctionResultList.processNextRecord(Object) line: 28
HiveBaseFunctionResultList$ResultIterator.hasNext() line: 108
Wrappers$JIteratorWrapperA.hasNext() line: 41
Iterator$class.foreach(Iterator, Function1) line: 727
Wrappers$JIteratorWrapperA(AbstractIteratorA).foreach(Function1A,U)
line: 1157
RDD$$anonfun$foreach$1.apply(IteratorT) line: 760
RDD$$anonfun$foreach$1.apply(Object) line: 760
SparkContext$$anonfun$runJob$3.apply(TaskContext, IteratorT) line: 1118
SparkContext$$anonfun$runJob$3.apply(Object, Object) line: 1118
ResultTaskT,U.runTask(TaskContext) line: 61
ResultTaskT,U(TaskT).run(long) line: 56
Executor$TaskRunner.run() line: 182
ThreadPoolExecutor.runWorker(ThreadPoolExecutor$Worker) line: 1145
ThreadPoolExecutor$Worker.run() line: 615
Thread.run() line: 745
There is also a NPE in the FileSinkOperator: the FileSystem object fs is null:
// in recent hadoop versions, use deleteOnExit to clean tmp files.
if (isNativeTable) {
autoDelete = fs.deleteOnExit(fsp.outPaths[0]);
Daemon Thread [Executor task launch worker-1] (Suspended (exception
NullPointerException))
FileSinkOperator.createBucketFiles(FileSinkOperator$FSPaths) line: 495
FileSinkOperator.closeOp(boolean) line: 925
FileSinkOperator(OperatorT).close(boolean) line: 582
SelectOperator(OperatorT).close(boolean) line: 594
SMBMapJoinOperator(OperatorT).close(boolean) line: 594
DummyStoreOperator(OperatorT).close(boolean) line: 594
FilterOperator(OperatorT).close(boolean) line: 594
TableScanOperator(OperatorT).close(boolean) line: 594
MapOperator(OperatorT).close(boolean) line: 594
SparkMapRecordHandler.close() line: 175
HiveMapFunctionResultList.closeRecordProcessor() line: 57
HiveBaseFunctionResultList$ResultIterator.hasNext() line: 122
Wrappers$JIteratorWrapperA.hasNext() line: 41
Iterator$class.foreach(Iterator, Function1) line: 727
Wrappers$JIteratorWrapperA(AbstractIteratorA).foreach(Function1A,U)
line: 1157
RDD$$anonfun$foreach$1.apply(IteratorT) line: 760
RDD$$anonfun$foreach$1.apply(Object) line: 760
SparkContext$$anonfun$runJob$3.apply(TaskContext, IteratorT) line: 1118
SparkContext$$anonfun$runJob$3.apply(Object, Object) line: 1118
ResultTaskT,U.runTask(TaskContext) line: 61
ResultTaskT,U(TaskT).run(long) line: 56
Executor$TaskRunner.run() line: 182
ThreadPoolExecutor.runWorker(ThreadPoolExecutor$Worker) line: 1145
ThreadPoolExecutor$Worker.run() line: 615
Thread.run() line: 745
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-8508) UT: fix bucketsort_insert tests - related to SMBMapJoinOperator
[
https://issues.apache.org/jira/browse/HIVE-8508?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chinna Rao Lalam updated HIVE-8508:
---
Status: Patch Available (was: Open)
Verified these tests and these are working on current code base.
UT: fix bucketsort_insert tests - related to SMBMapJoinOperator
---
Key: HIVE-8508
URL: https://issues.apache.org/jira/browse/HIVE-8508
Project: Hive
Issue Type: Sub-task
Components: Spark
Reporter: Thomas Friedrich
Assignee: Chinna Rao Lalam
Attachments: HIVE-8508.1-spark.patch
The 4 tests
bucketsortoptimize_insert_2
bucketsortoptimize_insert_4
bucketsortoptimize_insert_6
bucketsortoptimize_insert_7
bucketsortoptimize_insert_8
all fail with the same NPE related in SMBMapJoinOperator:
order object is null in SMBMapJoinOperator:
// fetch the first group for all small table aliases
for (byte pos = 0; pos order.length; pos++) {
if (pos != posBigTable)
{ fetchNextGroup(pos); }
Daemon Thread [Executor task launch worker-3] (Suspended (exception
NullPointerException))
SMBMapJoinOperator.processOp(Object, int) line: 258
FilterOperator(OperatorT).forward(Object, ObjectInspector) line: 799
FilterOperator.processOp(Object, int) line: 137
TableScanOperator(OperatorT).forward(Object, ObjectInspector) line: 799
TableScanOperator.processOp(Object, int) line: 95
MapOperator(OperatorT).forward(Object, ObjectInspector) line: 799
MapOperator.process(Writable) line: 536
SparkMapRecordHandler.processRow(Object, Object) line: 139
HiveMapFunctionResultList.processNextRecord(Tuple2BytesWritable,BytesWritable)
line: 47
HiveMapFunctionResultList.processNextRecord(Object) line: 28
HiveBaseFunctionResultList$ResultIterator.hasNext() line: 108
Wrappers$JIteratorWrapperA.hasNext() line: 41
Iterator$class.foreach(Iterator, Function1) line: 727
Wrappers$JIteratorWrapperA(AbstractIteratorA).foreach(Function1A,U)
line: 1157
RDD$$anonfun$foreach$1.apply(IteratorT) line: 760
RDD$$anonfun$foreach$1.apply(Object) line: 760
SparkContext$$anonfun$runJob$3.apply(TaskContext, IteratorT) line: 1118
SparkContext$$anonfun$runJob$3.apply(Object, Object) line: 1118
ResultTaskT,U.runTask(TaskContext) line: 61
ResultTaskT,U(TaskT).run(long) line: 56
Executor$TaskRunner.run() line: 182
ThreadPoolExecutor.runWorker(ThreadPoolExecutor$Worker) line: 1145
ThreadPoolExecutor$Worker.run() line: 615
Thread.run() line: 745
There is also a NPE in the FileSinkOperator: the FileSystem object fs is null:
// in recent hadoop versions, use deleteOnExit to clean tmp files.
if (isNativeTable) {
autoDelete = fs.deleteOnExit(fsp.outPaths[0]);
Daemon Thread [Executor task launch worker-1] (Suspended (exception
NullPointerException))
FileSinkOperator.createBucketFiles(FileSinkOperator$FSPaths) line: 495
FileSinkOperator.closeOp(boolean) line: 925
FileSinkOperator(OperatorT).close(boolean) line: 582
SelectOperator(OperatorT).close(boolean) line: 594
SMBMapJoinOperator(OperatorT).close(boolean) line: 594
DummyStoreOperator(OperatorT).close(boolean) line: 594
FilterOperator(OperatorT).close(boolean) line: 594
TableScanOperator(OperatorT).close(boolean) line: 594
MapOperator(OperatorT).close(boolean) line: 594
SparkMapRecordHandler.close() line: 175
HiveMapFunctionResultList.closeRecordProcessor() line: 57
HiveBaseFunctionResultList$ResultIterator.hasNext() line: 122
Wrappers$JIteratorWrapperA.hasNext() line: 41
Iterator$class.foreach(Iterator, Function1) line: 727
Wrappers$JIteratorWrapperA(AbstractIteratorA).foreach(Function1A,U)
line: 1157
RDD$$anonfun$foreach$1.apply(IteratorT) line: 760
RDD$$anonfun$foreach$1.apply(Object) line: 760
SparkContext$$anonfun$runJob$3.apply(TaskContext, IteratorT) line: 1118
SparkContext$$anonfun$runJob$3.apply(Object, Object) line: 1118
ResultTaskT,U.runTask(TaskContext) line: 61
ResultTaskT,U(TaskT).run(long) line: 56
Executor$TaskRunner.run() line: 182
ThreadPoolExecutor.runWorker(ThreadPoolExecutor$Worker) line: 1145
ThreadPoolExecutor$Worker.run() line: 615
Thread.run() line: 745
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Assigned] (HIVE-8507) UT: fix rcfile_bigdata test
[ https://issues.apache.org/jira/browse/HIVE-8507?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chinna Rao Lalam reassigned HIVE-8507: -- Assignee: Chinna Rao Lalam UT: fix rcfile_bigdata test --- Key: HIVE-8507 URL: https://issues.apache.org/jira/browse/HIVE-8507 Project: Hive Issue Type: Sub-task Components: Spark Reporter: Thomas Friedrich Assignee: Chinna Rao Lalam Priority: Minor Attachments: HIVE-8507.1-spark.patch The tests groupby_bigdata rcfile_bigdata fail because it can't find the dumpdata_script.py file that is referenced in the script: rcfile_bigdata.q /usr/bin/python: can't open file 'dumpdata_script.py': [Errno 2] No such file or directory There are two references: add file ../../dumpdata_script.py; FROM (FROM src MAP src.key,src.value USING 'python dumpdata_script.py' Since it's using relative path it seems to be related to spark tests being one level deeper than regular tests. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-8507) UT: fix rcfile_bigdata test
[ https://issues.apache.org/jira/browse/HIVE-8507?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chinna Rao Lalam updated HIVE-8507: --- Attachment: HIVE-8507.1-spark.patch UT: fix rcfile_bigdata test --- Key: HIVE-8507 URL: https://issues.apache.org/jira/browse/HIVE-8507 Project: Hive Issue Type: Sub-task Components: Spark Reporter: Thomas Friedrich Priority: Minor Attachments: HIVE-8507.1-spark.patch The tests groupby_bigdata rcfile_bigdata fail because it can't find the dumpdata_script.py file that is referenced in the script: rcfile_bigdata.q /usr/bin/python: can't open file 'dumpdata_script.py': [Errno 2] No such file or directory There are two references: add file ../../dumpdata_script.py; FROM (FROM src MAP src.key,src.value USING 'python dumpdata_script.py' Since it's using relative path it seems to be related to spark tests being one level deeper than regular tests. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-8507) UT: fix rcfile_bigdata test
[ https://issues.apache.org/jira/browse/HIVE-8507?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chinna Rao Lalam updated HIVE-8507: --- Status: Patch Available (was: Open) These test are working on the current code base.. UT: fix rcfile_bigdata test --- Key: HIVE-8507 URL: https://issues.apache.org/jira/browse/HIVE-8507 Project: Hive Issue Type: Sub-task Components: Spark Reporter: Thomas Friedrich Assignee: Chinna Rao Lalam Priority: Minor Attachments: HIVE-8507.1-spark.patch The tests groupby_bigdata rcfile_bigdata fail because it can't find the dumpdata_script.py file that is referenced in the script: rcfile_bigdata.q /usr/bin/python: can't open file 'dumpdata_script.py': [Errno 2] No such file or directory There are two references: add file ../../dumpdata_script.py; FROM (FROM src MAP src.key,src.value USING 'python dumpdata_script.py' Since it's using relative path it seems to be related to spark tests being one level deeper than regular tests. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-8507) UT: fix rcfile_bigdata test
[ https://issues.apache.org/jira/browse/HIVE-8507?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chinna Rao Lalam updated HIVE-8507: --- Status: Open (was: Patch Available) UT: fix rcfile_bigdata test --- Key: HIVE-8507 URL: https://issues.apache.org/jira/browse/HIVE-8507 Project: Hive Issue Type: Sub-task Components: Spark Reporter: Thomas Friedrich Assignee: Chinna Rao Lalam Priority: Minor Attachments: HIVE-8507.1-spark.patch The tests groupby_bigdata rcfile_bigdata fail because it can't find the dumpdata_script.py file that is referenced in the script: rcfile_bigdata.q /usr/bin/python: can't open file 'dumpdata_script.py': [Errno 2] No such file or directory There are two references: add file ../../dumpdata_script.py; FROM (FROM src MAP src.key,src.value USING 'python dumpdata_script.py' Since it's using relative path it seems to be related to spark tests being one level deeper than regular tests. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-8507) UT: fix rcfile_bigdata test
[ https://issues.apache.org/jira/browse/HIVE-8507?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chinna Rao Lalam updated HIVE-8507: --- Attachment: HIVE-8507.2-spark.patch Added tests in testconfiguration.properties UT: fix rcfile_bigdata test --- Key: HIVE-8507 URL: https://issues.apache.org/jira/browse/HIVE-8507 Project: Hive Issue Type: Sub-task Components: Spark Reporter: Thomas Friedrich Assignee: Chinna Rao Lalam Priority: Minor Attachments: HIVE-8507.1-spark.patch, HIVE-8507.2-spark.patch The tests groupby_bigdata rcfile_bigdata fail because it can't find the dumpdata_script.py file that is referenced in the script: rcfile_bigdata.q /usr/bin/python: can't open file 'dumpdata_script.py': [Errno 2] No such file or directory There are two references: add file ../../dumpdata_script.py; FROM (FROM src MAP src.key,src.value USING 'python dumpdata_script.py' Since it's using relative path it seems to be related to spark tests being one level deeper than regular tests. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-8507) UT: fix rcfile_bigdata test
[ https://issues.apache.org/jira/browse/HIVE-8507?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chinna Rao Lalam updated HIVE-8507: --- Status: Patch Available (was: Open) UT: fix rcfile_bigdata test --- Key: HIVE-8507 URL: https://issues.apache.org/jira/browse/HIVE-8507 Project: Hive Issue Type: Sub-task Components: Spark Reporter: Thomas Friedrich Assignee: Chinna Rao Lalam Priority: Minor Attachments: HIVE-8507.1-spark.patch, HIVE-8507.2-spark.patch The tests groupby_bigdata rcfile_bigdata fail because it can't find the dumpdata_script.py file that is referenced in the script: rcfile_bigdata.q /usr/bin/python: can't open file 'dumpdata_script.py': [Errno 2] No such file or directory There are two references: add file ../../dumpdata_script.py; FROM (FROM src MAP src.key,src.value USING 'python dumpdata_script.py' Since it's using relative path it seems to be related to spark tests being one level deeper than regular tests. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (HIVE-8508) UT: fix bucketsort_insert tests - related to SMBMapJoinOperator
[
https://issues.apache.org/jira/browse/HIVE-8508?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chinna Rao Lalam reassigned HIVE-8508:
--
Assignee: Chinna Rao Lalam
UT: fix bucketsort_insert tests - related to SMBMapJoinOperator
---
Key: HIVE-8508
URL: https://issues.apache.org/jira/browse/HIVE-8508
Project: Hive
Issue Type: Sub-task
Components: Spark
Reporter: Thomas Friedrich
Assignee: Chinna Rao Lalam
The 4 tests
bucketsortoptimize_insert_2
bucketsortoptimize_insert_4
bucketsortoptimize_insert_6
bucketsortoptimize_insert_7
bucketsortoptimize_insert_8
all fail with the same NPE related in SMBMapJoinOperator:
order object is null in SMBMapJoinOperator:
// fetch the first group for all small table aliases
for (byte pos = 0; pos order.length; pos++) {
if (pos != posBigTable)
{ fetchNextGroup(pos); }
Daemon Thread [Executor task launch worker-3] (Suspended (exception
NullPointerException))
SMBMapJoinOperator.processOp(Object, int) line: 258
FilterOperator(OperatorT).forward(Object, ObjectInspector) line: 799
FilterOperator.processOp(Object, int) line: 137
TableScanOperator(OperatorT).forward(Object, ObjectInspector) line: 799
TableScanOperator.processOp(Object, int) line: 95
MapOperator(OperatorT).forward(Object, ObjectInspector) line: 799
MapOperator.process(Writable) line: 536
SparkMapRecordHandler.processRow(Object, Object) line: 139
HiveMapFunctionResultList.processNextRecord(Tuple2BytesWritable,BytesWritable)
line: 47
HiveMapFunctionResultList.processNextRecord(Object) line: 28
HiveBaseFunctionResultList$ResultIterator.hasNext() line: 108
Wrappers$JIteratorWrapperA.hasNext() line: 41
Iterator$class.foreach(Iterator, Function1) line: 727
Wrappers$JIteratorWrapperA(AbstractIteratorA).foreach(Function1A,U)
line: 1157
RDD$$anonfun$foreach$1.apply(IteratorT) line: 760
RDD$$anonfun$foreach$1.apply(Object) line: 760
SparkContext$$anonfun$runJob$3.apply(TaskContext, IteratorT) line: 1118
SparkContext$$anonfun$runJob$3.apply(Object, Object) line: 1118
ResultTaskT,U.runTask(TaskContext) line: 61
ResultTaskT,U(TaskT).run(long) line: 56
Executor$TaskRunner.run() line: 182
ThreadPoolExecutor.runWorker(ThreadPoolExecutor$Worker) line: 1145
ThreadPoolExecutor$Worker.run() line: 615
Thread.run() line: 745
There is also a NPE in the FileSinkOperator: the FileSystem object fs is null:
// in recent hadoop versions, use deleteOnExit to clean tmp files.
if (isNativeTable) {
autoDelete = fs.deleteOnExit(fsp.outPaths[0]);
Daemon Thread [Executor task launch worker-1] (Suspended (exception
NullPointerException))
FileSinkOperator.createBucketFiles(FileSinkOperator$FSPaths) line: 495
FileSinkOperator.closeOp(boolean) line: 925
FileSinkOperator(OperatorT).close(boolean) line: 582
SelectOperator(OperatorT).close(boolean) line: 594
SMBMapJoinOperator(OperatorT).close(boolean) line: 594
DummyStoreOperator(OperatorT).close(boolean) line: 594
FilterOperator(OperatorT).close(boolean) line: 594
TableScanOperator(OperatorT).close(boolean) line: 594
MapOperator(OperatorT).close(boolean) line: 594
SparkMapRecordHandler.close() line: 175
HiveMapFunctionResultList.closeRecordProcessor() line: 57
HiveBaseFunctionResultList$ResultIterator.hasNext() line: 122
Wrappers$JIteratorWrapperA.hasNext() line: 41
Iterator$class.foreach(Iterator, Function1) line: 727
Wrappers$JIteratorWrapperA(AbstractIteratorA).foreach(Function1A,U)
line: 1157
RDD$$anonfun$foreach$1.apply(IteratorT) line: 760
RDD$$anonfun$foreach$1.apply(Object) line: 760
SparkContext$$anonfun$runJob$3.apply(TaskContext, IteratorT) line: 1118
SparkContext$$anonfun$runJob$3.apply(Object, Object) line: 1118
ResultTaskT,U.runTask(TaskContext) line: 61
ResultTaskT,U(TaskT).run(long) line: 56
Executor$TaskRunner.run() line: 182
ThreadPoolExecutor.runWorker(ThreadPoolExecutor$Worker) line: 1145
ThreadPoolExecutor$Worker.run() line: 615
Thread.run() line: 745
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HIVE-8986) Add limit_partition_metadataonly.q in testconfiguration.properties [Spark Branch]
Chinna Rao Lalam created HIVE-8986: -- Summary: Add limit_partition_metadataonly.q in testconfiguration.properties [Spark Branch] Key: HIVE-8986 URL: https://issues.apache.org/jira/browse/HIVE-8986 Project: Hive Issue Type: Bug Reporter: Chinna Rao Lalam Assignee: Chinna Rao Lalam Priority: Trivial Adding limit_partition_metadataonly.q in testconfiguration.properties is missed in HIVE-8788 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-8986) Add limit_partition_metadataonly.q in testconfiguration.properties [Spark Branch]
[ https://issues.apache.org/jira/browse/HIVE-8986?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chinna Rao Lalam updated HIVE-8986: --- Status: Patch Available (was: Open) Add limit_partition_metadataonly.q in testconfiguration.properties [Spark Branch] - Key: HIVE-8986 URL: https://issues.apache.org/jira/browse/HIVE-8986 Project: Hive Issue Type: Bug Reporter: Chinna Rao Lalam Assignee: Chinna Rao Lalam Priority: Trivial Attachments: HIVE-8986-spark.patch Adding limit_partition_metadataonly.q in testconfiguration.properties is missed in HIVE-8788 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-8986) Add limit_partition_metadataonly.q in testconfiguration.properties [Spark Branch]
[ https://issues.apache.org/jira/browse/HIVE-8986?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chinna Rao Lalam updated HIVE-8986: --- Attachment: HIVE-8986-spark.patch Add limit_partition_metadataonly.q in testconfiguration.properties [Spark Branch] - Key: HIVE-8986 URL: https://issues.apache.org/jira/browse/HIVE-8986 Project: Hive Issue Type: Bug Reporter: Chinna Rao Lalam Assignee: Chinna Rao Lalam Priority: Trivial Attachments: HIVE-8986-spark.patch Adding limit_partition_metadataonly.q in testconfiguration.properties is missed in HIVE-8788 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-8788) UT: fix partition test case
[ https://issues.apache.org/jira/browse/HIVE-8788?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chinna Rao Lalam updated HIVE-8788: --- Attachment: HIVE-8788-spark.patch UT: fix partition test case --- Key: HIVE-8788 URL: https://issues.apache.org/jira/browse/HIVE-8788 Project: Hive Issue Type: Sub-task Components: Tests Affects Versions: spark-branch Reporter: Thomas Friedrich Assignee: Chinna Rao Lalam Priority: Minor Attachments: HIVE-8788-spark.patch The test limit_partition_metadataonly fails with 2014-11-06 18:40:12,891 ERROR ql.Driver (SessionState.java:printError(829)) - FAILED: SemanticException Number of partitions scanned (=4) on table srcpart exceeds limit (=1). This is controlled by hive.limit.query.max.table.partition. org.apache.hadoop.hive.ql.parse.SemanticException: Number of partitions scanned (=4) on table srcpart exceeds limit (=1). This is controlled by hive.limit.query.max.table.partition. at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.enforceScanLimits(SemanticAnalyzer.java:10358) at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:10190) at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:221) at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:419) In the test, SemanticAnalyzer.enforceScanLimits expects only 1 partition ds=2008-04-08/hr=11 but gets 4 partitions: [srcpart(ds=2008-04-08/hr=11), srcpart(ds=2008-04-08/hr=12), srcpart(ds=2008-04-09/hr=11), srcpart(ds=2008-04-09/hr=12)] In the log it shows that the ParitionPruner ran, and it should have only retained one partition: 2014-11-07 14:18:09,147 DEBUG ppr.PartitionPruner (PartitionPruner.java:prune(206)) - Filter w/ compacting: ((hr = 11) and (ds = '2008-04-08')); filter w/o compacting: ((hr = 11) and (ds = '2008-04-08')) 2014-11-07 14:18:09,147 INFO metastore.HiveMetaStore (HiveMetaStore.java:logInfo(719)) - 0: get_partitions_by_expr : db=default tbl=srcpart 2014-11-07 14:18:09,165 DEBUG ppr.PartitionPruner (PartitionPruner.java:prunePartitionNames(491)) - retained partition: ds=2008-04-08/hr=11 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-8788) UT: fix partition test case
[ https://issues.apache.org/jira/browse/HIVE-8788?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chinna Rao Lalam updated HIVE-8788: --- Attachment: HIVE-8788.1-spark.patch UT: fix partition test case --- Key: HIVE-8788 URL: https://issues.apache.org/jira/browse/HIVE-8788 Project: Hive Issue Type: Sub-task Components: Tests Affects Versions: spark-branch Reporter: Thomas Friedrich Assignee: Chinna Rao Lalam Priority: Minor Attachments: HIVE-8788-spark.patch, HIVE-8788.1-spark.patch The test limit_partition_metadataonly fails with 2014-11-06 18:40:12,891 ERROR ql.Driver (SessionState.java:printError(829)) - FAILED: SemanticException Number of partitions scanned (=4) on table srcpart exceeds limit (=1). This is controlled by hive.limit.query.max.table.partition. org.apache.hadoop.hive.ql.parse.SemanticException: Number of partitions scanned (=4) on table srcpart exceeds limit (=1). This is controlled by hive.limit.query.max.table.partition. at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.enforceScanLimits(SemanticAnalyzer.java:10358) at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:10190) at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:221) at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:419) In the test, SemanticAnalyzer.enforceScanLimits expects only 1 partition ds=2008-04-08/hr=11 but gets 4 partitions: [srcpart(ds=2008-04-08/hr=11), srcpart(ds=2008-04-08/hr=12), srcpart(ds=2008-04-09/hr=11), srcpart(ds=2008-04-09/hr=12)] In the log it shows that the ParitionPruner ran, and it should have only retained one partition: 2014-11-07 14:18:09,147 DEBUG ppr.PartitionPruner (PartitionPruner.java:prune(206)) - Filter w/ compacting: ((hr = 11) and (ds = '2008-04-08')); filter w/o compacting: ((hr = 11) and (ds = '2008-04-08')) 2014-11-07 14:18:09,147 INFO metastore.HiveMetaStore (HiveMetaStore.java:logInfo(719)) - 0: get_partitions_by_expr : db=default tbl=srcpart 2014-11-07 14:18:09,165 DEBUG ppr.PartitionPruner (PartitionPruner.java:prunePartitionNames(491)) - retained partition: ds=2008-04-08/hr=11 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-8788) UT: fix partition test case
[ https://issues.apache.org/jira/browse/HIVE-8788?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chinna Rao Lalam updated HIVE-8788: --- Priority: Major (was: Minor) UT: fix partition test case --- Key: HIVE-8788 URL: https://issues.apache.org/jira/browse/HIVE-8788 Project: Hive Issue Type: Sub-task Components: Tests Affects Versions: spark-branch Reporter: Thomas Friedrich Assignee: Chinna Rao Lalam Attachments: HIVE-8788-spark.patch, HIVE-8788.1-spark.patch The test limit_partition_metadataonly fails with 2014-11-06 18:40:12,891 ERROR ql.Driver (SessionState.java:printError(829)) - FAILED: SemanticException Number of partitions scanned (=4) on table srcpart exceeds limit (=1). This is controlled by hive.limit.query.max.table.partition. org.apache.hadoop.hive.ql.parse.SemanticException: Number of partitions scanned (=4) on table srcpart exceeds limit (=1). This is controlled by hive.limit.query.max.table.partition. at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.enforceScanLimits(SemanticAnalyzer.java:10358) at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:10190) at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:221) at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:419) In the test, SemanticAnalyzer.enforceScanLimits expects only 1 partition ds=2008-04-08/hr=11 but gets 4 partitions: [srcpart(ds=2008-04-08/hr=11), srcpart(ds=2008-04-08/hr=12), srcpart(ds=2008-04-09/hr=11), srcpart(ds=2008-04-09/hr=12)] In the log it shows that the ParitionPruner ran, and it should have only retained one partition: 2014-11-07 14:18:09,147 DEBUG ppr.PartitionPruner (PartitionPruner.java:prune(206)) - Filter w/ compacting: ((hr = 11) and (ds = '2008-04-08')); filter w/o compacting: ((hr = 11) and (ds = '2008-04-08')) 2014-11-07 14:18:09,147 INFO metastore.HiveMetaStore (HiveMetaStore.java:logInfo(719)) - 0: get_partitions_by_expr : db=default tbl=srcpart 2014-11-07 14:18:09,165 DEBUG ppr.PartitionPruner (PartitionPruner.java:prunePartitionNames(491)) - retained partition: ds=2008-04-08/hr=11 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-8788) UT: fix partition test case
[ https://issues.apache.org/jira/browse/HIVE-8788?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chinna Rao Lalam updated HIVE-8788: --- Status: Patch Available (was: Open) Enabled MetadataOnlyOptimizer. RB request: https://reviews.apache.org/r/28436/ UT: fix partition test case --- Key: HIVE-8788 URL: https://issues.apache.org/jira/browse/HIVE-8788 Project: Hive Issue Type: Sub-task Components: Tests Affects Versions: spark-branch Reporter: Thomas Friedrich Assignee: Chinna Rao Lalam Attachments: HIVE-8788-spark.patch, HIVE-8788.1-spark.patch The test limit_partition_metadataonly fails with 2014-11-06 18:40:12,891 ERROR ql.Driver (SessionState.java:printError(829)) - FAILED: SemanticException Number of partitions scanned (=4) on table srcpart exceeds limit (=1). This is controlled by hive.limit.query.max.table.partition. org.apache.hadoop.hive.ql.parse.SemanticException: Number of partitions scanned (=4) on table srcpart exceeds limit (=1). This is controlled by hive.limit.query.max.table.partition. at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.enforceScanLimits(SemanticAnalyzer.java:10358) at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:10190) at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:221) at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:419) In the test, SemanticAnalyzer.enforceScanLimits expects only 1 partition ds=2008-04-08/hr=11 but gets 4 partitions: [srcpart(ds=2008-04-08/hr=11), srcpart(ds=2008-04-08/hr=12), srcpart(ds=2008-04-09/hr=11), srcpart(ds=2008-04-09/hr=12)] In the log it shows that the ParitionPruner ran, and it should have only retained one partition: 2014-11-07 14:18:09,147 DEBUG ppr.PartitionPruner (PartitionPruner.java:prune(206)) - Filter w/ compacting: ((hr = 11) and (ds = '2008-04-08')); filter w/o compacting: ((hr = 11) and (ds = '2008-04-08')) 2014-11-07 14:18:09,147 INFO metastore.HiveMetaStore (HiveMetaStore.java:logInfo(719)) - 0: get_partitions_by_expr : db=default tbl=srcpart 2014-11-07 14:18:09,165 DEBUG ppr.PartitionPruner (PartitionPruner.java:prunePartitionNames(491)) - retained partition: ds=2008-04-08/hr=11 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (HIVE-8962) Add SORT_QUERY_RESULTS for join tests that do not guarantee order #2
[
https://issues.apache.org/jira/browse/HIVE-8962?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chinna Rao Lalam reassigned HIVE-8962:
--
Assignee: Chinna Rao Lalam
Add SORT_QUERY_RESULTS for join tests that do not guarantee order #2
Key: HIVE-8962
URL: https://issues.apache.org/jira/browse/HIVE-8962
Project: Hive
Issue Type: Sub-task
Components: Spark
Reporter: Chao
Assignee: Chinna Rao Lalam
Priority: Minor
Similar to HIVE-8936, we need to add {{SORT_QUERY_RESULTS}} to the following
q-files:
{noformat}
ppd_multi_insert.q
ptf_streaming.q
subquery_exists.q
subquery_multiinsert.q
vectorized_ptf.q
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Assigned] (HIVE-8962) Add SORT_QUERY_RESULTS for join tests that do not guarantee order #2
[
https://issues.apache.org/jira/browse/HIVE-8962?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chinna Rao Lalam reassigned HIVE-8962:
--
Assignee: Chao (was: Chinna Rao Lalam)
Add SORT_QUERY_RESULTS for join tests that do not guarantee order #2
Key: HIVE-8962
URL: https://issues.apache.org/jira/browse/HIVE-8962
Project: Hive
Issue Type: Sub-task
Components: Spark
Reporter: Chao
Assignee: Chao
Priority: Minor
Attachments: HIVE-8962.patch
Similar to HIVE-8936, we need to add {{SORT_QUERY_RESULTS}} to the following
q-files:
{noformat}
ppd_multi_insert.q
ptf_streaming.q
subquery_exists.q
subquery_multiinsert.q
vectorized_ptf.q
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-8639) Convert SMBJoin to MapJoin [Spark Branch]
[ https://issues.apache.org/jira/browse/HIVE-8639?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14218168#comment-14218168 ] Chinna Rao Lalam commented on HIVE-8639: Hi [~brocknoland], I am investigating test failures. I need some time for this issue, If folks freeing up they can take it over. Convert SMBJoin to MapJoin [Spark Branch] - Key: HIVE-8639 URL: https://issues.apache.org/jira/browse/HIVE-8639 Project: Hive Issue Type: Sub-task Components: Spark Reporter: Szehon Ho Assignee: Chinna Rao Lalam HIVE-8202 supports auto-conversion of SMB Join. However, if the tables are partitioned, there could be a slow down as each mapper would need to get a very small chunk of a partition which has a single key. Thus, in some scenarios it's beneficial to convert SMB join to map join. The task is to research and support the conversion from SMB join to map join for Spark execution engine. See the equivalent of MapReduce in SortMergeJoinResolver. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (HIVE-8504) UT: fix bucket_num_reducers test
[ https://issues.apache.org/jira/browse/HIVE-8504?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chinna Rao Lalam reassigned HIVE-8504: -- Assignee: Chinna Rao Lalam UT: fix bucket_num_reducers test Key: HIVE-8504 URL: https://issues.apache.org/jira/browse/HIVE-8504 Project: Hive Issue Type: Sub-task Components: Spark Reporter: Thomas Friedrich Assignee: Chinna Rao Lalam Priority: Minor The test bucket_num_reducers fails with a error: Exception: Number of MapReduce jobs is incorrect expected:1 but was:0 junit.framework.AssertionFailedError: Number of MapReduce jobs is incorrect expected:1 but was:0 at org.apache.hadoop.hive.ql.hooks.VerifyNumReducersHook.run(VerifyNumReducersHook.java:46) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (HIVE-8509) UT: fix list_bucket_dml_2 test
[
https://issues.apache.org/jira/browse/HIVE-8509?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chinna Rao Lalam reassigned HIVE-8509:
--
Assignee: Chinna Rao Lalam
UT: fix list_bucket_dml_2 test
--
Key: HIVE-8509
URL: https://issues.apache.org/jira/browse/HIVE-8509
Project: Hive
Issue Type: Sub-task
Components: Spark
Reporter: Thomas Friedrich
Assignee: Chinna Rao Lalam
Priority: Minor
The test list_bucket_dml_2 fails in FileSinkOperator.publishStats:
org.apache.hadoop.hive.ql.metadata.HiveException: [Error 30002]:
StatsPublisher cannot be connected to.There was a error while connecting to
the StatsPublisher, and retrying might help. If you dont want the query to
fail because accurate statistics could not be collected, set
hive.stats.reliable=false
at
org.apache.hadoop.hive.ql.exec.FileSinkOperator.publishStats(FileSinkOperator.java:1079)
at
org.apache.hadoop.hive.ql.exec.FileSinkOperator.closeOp(FileSinkOperator.java:971)
at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:582)
at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:594)
at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:594)
at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:594)
at
org.apache.hadoop.hive.ql.exec.spark.SparkMapRecordHandler.close(SparkMapRecordHandler.java:175)
at
org.apache.hadoop.hive.ql.exec.spark.HiveMapFunctionResultList.closeRecordProcessor(HiveMapFunctionResultList.java:57)
at
org.apache.hadoop.hive.ql.exec.spark.HiveBaseFunctionResultList$ResultIterator.hasNext(HiveBaseFunctionResultList.java:121)
I debugged and found that FileSinkOperator.publishStats throws the exception
when calling statsPublisher.connect here:
if (!statsPublisher.connect(hconf)) {
// just return, stats gathering should not block the main query
LOG.error(StatsPublishing error: cannot connect to database);
if (isStatsReliable)
{ throw new
HiveException(ErrorMsg.STATSPUBLISHER_CONNECTION_ERROR.getErrorCodedMsg()); }
return;
}
With the hive.stats.dbclass set to counter in data/conf/spark/hive-site.xml,
the statsPuvlisher is of type CounterStatsPublisher.
In CounterStatsPublisher, the exception is thrown because getReporter()
returns null for the MapredContext:
MapredContext context = MapredContext.get();
if (context == null || context.getReporter() == null)
{ return false; }
When changing hive.stats.dbclass to jdbc:derby in
data/conf/spark/hive-site.xml, similar to TestCliDriver it works:
property
namehive.stats.dbclass/name
!-- valuecounter/value --
valuejdbc:derby/value
descriptionThe default storatge that stores temporary hive statistics.
Currently, jdbc, hbase and counter type is supported/description
/property
In addition, I had to generate the out file for the test case for spark.
When running this test with TestCliDriver and hive.stats.dbclass set to
counter, the test case still works. The reporter is set to
org.apache.hadoop.mapred.Task$TaskReporter.
Might need some additional investigation why the CounterStatsPublisher has no
reporter in case of spark.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-8509) UT: fix list_bucket_dml_2 test
[
https://issues.apache.org/jira/browse/HIVE-8509?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chinna Rao Lalam updated HIVE-8509:
---
Attachment: HIVE-8509-spark.patch
UT: fix list_bucket_dml_2 test
--
Key: HIVE-8509
URL: https://issues.apache.org/jira/browse/HIVE-8509
Project: Hive
Issue Type: Sub-task
Components: Spark
Reporter: Thomas Friedrich
Assignee: Chinna Rao Lalam
Priority: Minor
Attachments: HIVE-8509-spark.patch
The test list_bucket_dml_2 fails in FileSinkOperator.publishStats:
org.apache.hadoop.hive.ql.metadata.HiveException: [Error 30002]:
StatsPublisher cannot be connected to.There was a error while connecting to
the StatsPublisher, and retrying might help. If you dont want the query to
fail because accurate statistics could not be collected, set
hive.stats.reliable=false
at
org.apache.hadoop.hive.ql.exec.FileSinkOperator.publishStats(FileSinkOperator.java:1079)
at
org.apache.hadoop.hive.ql.exec.FileSinkOperator.closeOp(FileSinkOperator.java:971)
at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:582)
at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:594)
at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:594)
at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:594)
at
org.apache.hadoop.hive.ql.exec.spark.SparkMapRecordHandler.close(SparkMapRecordHandler.java:175)
at
org.apache.hadoop.hive.ql.exec.spark.HiveMapFunctionResultList.closeRecordProcessor(HiveMapFunctionResultList.java:57)
at
org.apache.hadoop.hive.ql.exec.spark.HiveBaseFunctionResultList$ResultIterator.hasNext(HiveBaseFunctionResultList.java:121)
I debugged and found that FileSinkOperator.publishStats throws the exception
when calling statsPublisher.connect here:
if (!statsPublisher.connect(hconf)) {
// just return, stats gathering should not block the main query
LOG.error(StatsPublishing error: cannot connect to database);
if (isStatsReliable)
{ throw new
HiveException(ErrorMsg.STATSPUBLISHER_CONNECTION_ERROR.getErrorCodedMsg()); }
return;
}
With the hive.stats.dbclass set to counter in data/conf/spark/hive-site.xml,
the statsPuvlisher is of type CounterStatsPublisher.
In CounterStatsPublisher, the exception is thrown because getReporter()
returns null for the MapredContext:
MapredContext context = MapredContext.get();
if (context == null || context.getReporter() == null)
{ return false; }
When changing hive.stats.dbclass to jdbc:derby in
data/conf/spark/hive-site.xml, similar to TestCliDriver it works:
property
namehive.stats.dbclass/name
!-- valuecounter/value --
valuejdbc:derby/value
descriptionThe default storatge that stores temporary hive statistics.
Currently, jdbc, hbase and counter type is supported/description
/property
In addition, I had to generate the out file for the test case for spark.
When running this test with TestCliDriver and hive.stats.dbclass set to
counter, the test case still works. The reporter is set to
org.apache.hadoop.mapred.Task$TaskReporter.
Might need some additional investigation why the CounterStatsPublisher has no
reporter in case of spark.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-8509) UT: fix list_bucket_dml_2 test
[
https://issues.apache.org/jira/browse/HIVE-8509?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chinna Rao Lalam updated HIVE-8509:
---
Status: Patch Available (was: Open)
UT: fix list_bucket_dml_2 test
--
Key: HIVE-8509
URL: https://issues.apache.org/jira/browse/HIVE-8509
Project: Hive
Issue Type: Sub-task
Components: Spark
Reporter: Thomas Friedrich
Assignee: Chinna Rao Lalam
Priority: Minor
Attachments: HIVE-8509-spark.patch
The test list_bucket_dml_2 fails in FileSinkOperator.publishStats:
org.apache.hadoop.hive.ql.metadata.HiveException: [Error 30002]:
StatsPublisher cannot be connected to.There was a error while connecting to
the StatsPublisher, and retrying might help. If you dont want the query to
fail because accurate statistics could not be collected, set
hive.stats.reliable=false
at
org.apache.hadoop.hive.ql.exec.FileSinkOperator.publishStats(FileSinkOperator.java:1079)
at
org.apache.hadoop.hive.ql.exec.FileSinkOperator.closeOp(FileSinkOperator.java:971)
at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:582)
at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:594)
at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:594)
at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:594)
at
org.apache.hadoop.hive.ql.exec.spark.SparkMapRecordHandler.close(SparkMapRecordHandler.java:175)
at
org.apache.hadoop.hive.ql.exec.spark.HiveMapFunctionResultList.closeRecordProcessor(HiveMapFunctionResultList.java:57)
at
org.apache.hadoop.hive.ql.exec.spark.HiveBaseFunctionResultList$ResultIterator.hasNext(HiveBaseFunctionResultList.java:121)
I debugged and found that FileSinkOperator.publishStats throws the exception
when calling statsPublisher.connect here:
if (!statsPublisher.connect(hconf)) {
// just return, stats gathering should not block the main query
LOG.error(StatsPublishing error: cannot connect to database);
if (isStatsReliable)
{ throw new
HiveException(ErrorMsg.STATSPUBLISHER_CONNECTION_ERROR.getErrorCodedMsg()); }
return;
}
With the hive.stats.dbclass set to counter in data/conf/spark/hive-site.xml,
the statsPuvlisher is of type CounterStatsPublisher.
In CounterStatsPublisher, the exception is thrown because getReporter()
returns null for the MapredContext:
MapredContext context = MapredContext.get();
if (context == null || context.getReporter() == null)
{ return false; }
When changing hive.stats.dbclass to jdbc:derby in
data/conf/spark/hive-site.xml, similar to TestCliDriver it works:
property
namehive.stats.dbclass/name
!-- valuecounter/value --
valuejdbc:derby/value
descriptionThe default storatge that stores temporary hive statistics.
Currently, jdbc, hbase and counter type is supported/description
/property
In addition, I had to generate the out file for the test case for spark.
When running this test with TestCliDriver and hive.stats.dbclass set to
counter, the test case still works. The reporter is set to
org.apache.hadoop.mapred.Task$TaskReporter.
Might need some additional investigation why the CounterStatsPublisher has no
reporter in case of spark.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-7930) enable vectorization_short_regress.q, vector_string_concat.q [Spark Branch]
[
https://issues.apache.org/jira/browse/HIVE-7930?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chinna Rao Lalam updated HIVE-7930:
---
Attachment: HIVE-7930-spark.patch
vector_string_concat.q is already enabled. Patch updated with the
vectorization_short_regress.q
enable vectorization_short_regress.q, vector_string_concat.q [Spark Branch]
Key: HIVE-7930
URL: https://issues.apache.org/jira/browse/HIVE-7930
Project: Hive
Issue Type: Bug
Affects Versions: spark-branch
Reporter: Chinna Rao Lalam
Assignee: Chinna Rao Lalam
Fix For: spark-branch
Attachments: HIVE-7930-spark.patch
{quote}
vector_string_concat.q
vectorization_short_regress.q
{quote}
queries executed as normal queries.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-7930) enable vectorization_short_regress.q, vector_string_concat.q [Spark Branch]
[
https://issues.apache.org/jira/browse/HIVE-7930?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chinna Rao Lalam updated HIVE-7930:
---
Status: Patch Available (was: Open)
enable vectorization_short_regress.q, vector_string_concat.q [Spark Branch]
Key: HIVE-7930
URL: https://issues.apache.org/jira/browse/HIVE-7930
Project: Hive
Issue Type: Bug
Affects Versions: spark-branch
Reporter: Chinna Rao Lalam
Assignee: Chinna Rao Lalam
Fix For: spark-branch
Attachments: HIVE-7930-spark.patch
{quote}
vector_string_concat.q
vectorization_short_regress.q
{quote}
queries executed as normal queries.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Assigned] (HIVE-8639) Convert SMBJoin to MapJoin [Spark Branch]
[ https://issues.apache.org/jira/browse/HIVE-8639?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chinna Rao Lalam reassigned HIVE-8639: -- Assignee: Chinna Rao Lalam Convert SMBJoin to MapJoin [Spark Branch] - Key: HIVE-8639 URL: https://issues.apache.org/jira/browse/HIVE-8639 Project: Hive Issue Type: Sub-task Components: Spark Reporter: Szehon Ho Assignee: Chinna Rao Lalam HIVE-8202 supports auto-conversion of SMB Join. However, if the tables are partitioned, there could be a slow down as each mapper would need to get a very small chunk of a partition which has a single key. Thus, in some scenarios it's beneficial to convert SMB join to map join. The task is to research and support the conversion from SMB join to map join for Spark execution engine. See the equivalent of MapReduce in SortMergeJoinResolver. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-7930) enable vectorization_short_regress.q, vector_string_concat.q
[ https://issues.apache.org/jira/browse/HIVE-7930?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chinna Rao Lalam updated HIVE-7930: --- Summary: enable vectorization_short_regress.q, vector_string_concat.q (was: enable vectorization_short_regress.q, vectorized_timestamp_funcs.q, vectorized_nested_mapjoin.q) enable vectorization_short_regress.q, vector_string_concat.q - Key: HIVE-7930 URL: https://issues.apache.org/jira/browse/HIVE-7930 Project: Hive Issue Type: Bug Affects Versions: spark-branch Reporter: Chinna Rao Lalam Assignee: Chinna Rao Lalam Fix For: spark-branch enable vectorization_short_regress.q, vectorized_timestamp_funcs.q, vectorized_nested_mapjoin.q. Initial thought is because of UDF's and nested map join these are failing. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-7930) enable vectorization_short_regress.q, vector_string_concat.q
[
https://issues.apache.org/jira/browse/HIVE-7930?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chinna Rao Lalam updated HIVE-7930:
---
Description:
{quote}
vector_string_concat.q
vectorization_short_regress.q
{quote}
queries executed as normal queries.
was:
enable vectorization_short_regress.q, vectorized_timestamp_funcs.q,
vectorized_nested_mapjoin.q.
Initial thought is because of UDF's and nested map join these are failing.
enable vectorization_short_regress.q, vector_string_concat.q
-
Key: HIVE-7930
URL: https://issues.apache.org/jira/browse/HIVE-7930
Project: Hive
Issue Type: Bug
Affects Versions: spark-branch
Reporter: Chinna Rao Lalam
Assignee: Chinna Rao Lalam
Fix For: spark-branch
{quote}
vector_string_concat.q
vectorization_short_regress.q
{quote}
queries executed as normal queries.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HIVE-8425) enable vectorized_mapjoin.q,vectorized_nested_mapjoin.q [Sparch Branch]
Chinna Rao Lalam created HIVE-8425: -- Summary: enable vectorized_mapjoin.q,vectorized_nested_mapjoin.q [Sparch Branch] Key: HIVE-8425 URL: https://issues.apache.org/jira/browse/HIVE-8425 Project: Hive Issue Type: Bug Affects Versions: spark-branch Reporter: Chinna Rao Lalam Assignee: Chinna Rao Lalam Fix For: spark-branch enable vectorized_mapjoin.q,vectorized_nested_mapjoin.q. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-7794) Enable tests on Spark branch (4) [Sparch Branch]
[
https://issues.apache.org/jira/browse/HIVE-7794?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chinna Rao Lalam updated HIVE-7794:
---
Attachment: HIVE-7794.1-spark.patch
Enable tests on Spark branch (4) [Sparch Branch]
Key: HIVE-7794
URL: https://issues.apache.org/jira/browse/HIVE-7794
Project: Hive
Issue Type: Sub-task
Reporter: Brock Noland
Assignee: Chinna Rao Lalam
Attachments: HIVE-7794-spark.patch, HIVE-7794.1-spark.patch
This jira is to enable *most* of the tests below. If tests don't pass because
of some unsupported feature, ensure that a JIRA exists and move on.
{noformat}
vector_cast_constant.q,\
vector_data_types.q,\
vector_decimal_aggregate.q,\
vector_left_outer_join.q,\
vector_string_concat.q,\
vectorization_12.q,\
vectorization_13.q,\
vectorization_14.q,\
vectorization_15.q,\
vectorization_9.q,\
vectorization_part_project.q,\
vectorization_short_regress.q,\
vectorized_mapjoin.q,\
vectorized_nested_mapjoin.q,\
vectorized_ptf.q,\
vectorized_shufflejoin.q,\
vectorized_timestamp_funcs.q
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-7794) Enable tests on Spark branch (4) [Sparch Branch]
[
https://issues.apache.org/jira/browse/HIVE-7794?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chinna Rao Lalam updated HIVE-7794:
---
Status: Patch Available (was: Open)
Updated the patch.
Patch doesnot contain below q files
{quote}
vector_string_concat.q
vectorization_short_regress.q
{quote}
These queries executed as normal queries, will handle this in HIVE-7930
{quote}
vectorized_mapjoin.q
vectorized_nested_mapjoin.q
{quote}
These queries will enable after enabling the map join, will handle this in
HIVE-8425
Enable tests on Spark branch (4) [Sparch Branch]
Key: HIVE-7794
URL: https://issues.apache.org/jira/browse/HIVE-7794
Project: Hive
Issue Type: Sub-task
Reporter: Brock Noland
Assignee: Chinna Rao Lalam
Attachments: HIVE-7794-spark.patch, HIVE-7794.1-spark.patch
This jira is to enable *most* of the tests below. If tests don't pass because
of some unsupported feature, ensure that a JIRA exists and move on.
{noformat}
vector_cast_constant.q,\
vector_data_types.q,\
vector_decimal_aggregate.q,\
vector_left_outer_join.q,\
vector_string_concat.q,\
vectorization_12.q,\
vectorization_13.q,\
vectorization_14.q,\
vectorization_15.q,\
vectorization_9.q,\
vectorization_part_project.q,\
vectorization_short_regress.q,\
vectorized_mapjoin.q,\
vectorized_nested_mapjoin.q,\
vectorized_ptf.q,\
vectorized_shufflejoin.q,\
vectorized_timestamp_funcs.q
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-8425) enable vectorized_mapjoin.q,vectorized_nested_mapjoin.q [Sparch Branch]
[ https://issues.apache.org/jira/browse/HIVE-8425?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chinna Rao Lalam updated HIVE-8425: --- Attachment: HIVE-8425-spark.patch enable vectorized_mapjoin.q,vectorized_nested_mapjoin.q [Sparch Branch] --- Key: HIVE-8425 URL: https://issues.apache.org/jira/browse/HIVE-8425 Project: Hive Issue Type: Bug Affects Versions: spark-branch Reporter: Chinna Rao Lalam Assignee: Chinna Rao Lalam Fix For: spark-branch Attachments: HIVE-8425-spark.patch enable vectorized_mapjoin.q,vectorized_nested_mapjoin.q. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-8425) enable vectorized_mapjoin.q,vectorized_nested_mapjoin.q [Sparch Branch]
[ https://issues.apache.org/jira/browse/HIVE-8425?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chinna Rao Lalam updated HIVE-8425: --- Status: Patch Available (was: Open) Thanks [~xuefuz], I have missed this HIVE-8412 JIRA. Updated the patch. Please take a look. enable vectorized_mapjoin.q,vectorized_nested_mapjoin.q [Sparch Branch] --- Key: HIVE-8425 URL: https://issues.apache.org/jira/browse/HIVE-8425 Project: Hive Issue Type: Bug Affects Versions: spark-branch Reporter: Chinna Rao Lalam Assignee: Chinna Rao Lalam Fix For: spark-branch Attachments: HIVE-8425-spark.patch enable vectorized_mapjoin.q,vectorized_nested_mapjoin.q. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-6954) After ALTER FILEFORMAT, DESCRIBE throwing exception
[
https://issues.apache.org/jira/browse/HIVE-6954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14154586#comment-14154586
]
Chinna Rao Lalam commented on HIVE-6954:
Hi [~vikram.dixit],
I have vefified this issue on the current trunk base. This issue is not
reproduced.
I will close this issue as a not problem, if no one faces this issue.
After ALTER FILEFORMAT, DESCRIBE throwing exception
---
Key: HIVE-6954
URL: https://issues.apache.org/jira/browse/HIVE-6954
Project: Hive
Issue Type: Bug
Components: Query Processor
Affects Versions: 0.14.0
Reporter: Chinna Rao Lalam
Assignee: Chinna Rao Lalam
Priority: Critical
Fix For: 0.14.0
After ALTER TABLE FILEFORMAT, DESCRIBE EXTENDS TABLE throwing exception.
{quote}
CREATE TABLE alter_test ( id int, str string,mp MAPSTRING,STRING,lst
ARRAYSTRING,strct STRUCTA:STRING,B:STRING) PARTITIONED BY (part string)
STORED AS TEXTFILE;
DESCRIBE EXTENDED alter_test;
ALTER TABLE alter_test SET FILEFORMAT PARQUET;
DESCRIBE EXTENDED alter_test;
SELECT * FROM alter_test;
{quote}
hive DESCRIBE EXTENDED alter_test;
FAILED: SemanticException : Table not found alter_test
hive SELECT * FROM alter_test;
FAILED: SemanticException must specify an InputFormat class
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-8180) Update SparkReduceRecordHandler for processing the vectors [spark branch]
[ https://issues.apache.org/jira/browse/HIVE-8180?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chinna Rao Lalam updated HIVE-8180: --- Attachment: HIVE-8180.3-spark.patch Removed trailing spaces. Update SparkReduceRecordHandler for processing the vectors [spark branch] - Key: HIVE-8180 URL: https://issues.apache.org/jira/browse/HIVE-8180 Project: Hive Issue Type: Bug Components: Spark Reporter: Chinna Rao Lalam Assignee: Chinna Rao Lalam Labels: Spark-M1 Attachments: HIVE-8180-spark.patch, HIVE-8180.1-spark.patch, HIVE-8180.2-spark.patch, HIVE-8180.3-spark.patch Update SparkReduceRecordHandler for processing the vectors. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-8180) Update SparkReduceRecordHandler for processing the vectors [spark branch]
[ https://issues.apache.org/jira/browse/HIVE-8180?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14151412#comment-14151412 ] Chinna Rao Lalam commented on HIVE-8180: RB link : https://reviews.apache.org/r/26130/ Update SparkReduceRecordHandler for processing the vectors [spark branch] - Key: HIVE-8180 URL: https://issues.apache.org/jira/browse/HIVE-8180 Project: Hive Issue Type: Bug Components: Spark Reporter: Chinna Rao Lalam Assignee: Chinna Rao Lalam Labels: Spark-M1 Attachments: HIVE-8180-spark.patch, HIVE-8180.1-spark.patch, HIVE-8180.2-spark.patch Update SparkReduceRecordHandler for processing the vectors. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-8180) Update SparkReduceRecordHandler for processing the vectors [spark branch]
[ https://issues.apache.org/jira/browse/HIVE-8180?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chinna Rao Lalam updated HIVE-8180: --- Attachment: HIVE-8180-spark.patch Update SparkReduceRecordHandler for processing the vectors [spark branch] - Key: HIVE-8180 URL: https://issues.apache.org/jira/browse/HIVE-8180 Project: Hive Issue Type: Bug Reporter: Chinna Rao Lalam Assignee: Chinna Rao Lalam Attachments: HIVE-8180-spark.patch Update SparkReduceRecordHandler for processing the vectors. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
