Re: Proposal: Revamp Apache Hive website.

[Owen O'Malley]

I had a PR to replace the website with markdown. Apache Infra was supposed
to make it autopublish. *sigh*

.. Owen

On Thu, Sep 15, 2022 at 4:23 PM Pau Tallada  wrote:

> Hi,
>
> Great work!
> +1 on updating it as well
>
> Missatge de Ayush Saxena  del dia dj., 15 de set.
> 2022 a les 17:40:
>
>> Hi Simhadri,
>> Thanx for the initiative, +1 on updating our current website.
>> The new website looks way better than the existing one.
>> Can create a Jira and link this to that after a couple of days if there
>> aren’t any objections to the move, so as people can drop further
>> suggestions over there.
>>
>> -Ayush
>>
>> > On 15-Sep-2022, at 8:33 PM, SG  wrote:
>> >
>> > Hi Everyone,
>> >
>> > The existing apache hive website https://hive.apache.org/ hasn't been
>> > updated for a very long time. Additionally, I was not able to build the
>> > docker image associated with the site to test out new changes as well.
>> > https://github.com/apache/hive-site
>> >
>> > Since the website is the front page of the project, I believe it would
>> be
>> > good to revamp the apache hive website with the latest features and
>> > releases.
>> >
>> > As a result, I have spent some time setting up an initial draft of the
>> > website. There are still quite a few things that still need to be
>> > added/updated in the draft.  Here is the prototype site that I put
>> together
>> > so everyone could check it out:
>> https://simhadri-g.github.io/hive-site-hugo/
>> >
>> > The new website is based on Hugo. Hugo, is one of the most popular
>> > open-source static site generators. I was able to get a hugo version of
>> the
>> > site up very quickly with a landing page and docs and hugo also makes
>> > maintaining the site very simple.
>> >
>> > Please have a look. Any feedback is much appreciated.
>> >
>> > Existing website: https://hive.apache.org/
>> > New website: https://simhadri-g.github.io/hive-site-hugo/
>> >
>> > Thanks!
>> > Simhadri G
>>
>
>
> --
> --
> Pau Tallada Crespí
> Departament de Serveis
> Port d'Informació Científica (PIC)
> Tel: +34 93 170 2729
> --
>
>

FYI: MetaStore running out of threads

[Owen O'Malley]

We're using HMS with Storage-Based Authorization and have been having
trouble with the HMS running out of threads. Looking at the jstack & code,
it appears to that the problem is that RPC's ConnectionId is using UGI's
equal/hash, which uses the Subject's Object equals/hash. Proxy user UGI's
always create a new Subject and thus are always unique.

This leads to the HMS creating too many threads. I've created a jira in
Hadoop. https://issues.apache.org/jira/browse/HADOOP-18434

Thanks,
   Owen

Re: Should we consider Spark3 support for Hive on Spark

[Owen O'Malley]

Hive on Spark is not recommended. The recommended path is to use either Tez
or LLAP. If you already are using Spark 3, it would be far easier to use
Spark SQL.

.. Owen

On Wed, Aug 24, 2022 at 3:46 AM Fred Bai  wrote:

> Hi everyone:
>
> Do we have any support for Hive on Spark? I need Hive on Spark, but my
> Spark version is 3.X.
>
> I found Hive incompatible with Spark3, I modify a lot of code to be
> compatible.
>
> Hive on Spark has deprecated?
>
> And. Hive on Spark is very slow when the job executes.
>

[RESULT][VOTE] Should we release Hive Storage API 2.8.1 rc2?

[Owen O'Malley]

With four +1's (3 binding) and no -1's, the vote passes. I'll finalize the
release.

Thanks for voting,
   Owen

On Tue, Aug 3, 2021 at 4:01 AM Chao Sun  wrote:

> +1 (binding)
>
> - build the release and ran all the tests successfully
> - verified checksum and signatures
>
> Chao
>
> On Mon, Aug 2, 2021 at 4:49 PM Pavan Lanka 
> wrote:
>
> > +1 (non-binding)
> >
> > * Built the release
> > * Built ORC using 2.8.1
> > * Ran the benchmarks compactExpression on the SArg
> >
> > Regards,
> > Pavan
> >
> > > On Aug 2, 2021, at 4:09 PM, Alan Gates  wrote:
> > >
> > > +1.  Built it, ran rat, checked the signature and the hash.
> > >
> > > Alan.
> > >
> > > On Thu, Jul 29, 2021 at 3:13 PM Owen O'Malley 
> > > wrote:
> > >
> > >> Hello all,
> > >>
> > >> I would like to propose a new storage-api release including
> HIVE-25400.
> > >>
> > >> Shall we release the following artifacts as Hive Storage API 2.8.1?
> > >>
> > >> tar: http://home.apache.org/~omalley/hive-storage-2.8.1/
> > >> tag:
> > https://github.com/apache/hive/releases/tag/storage-release-2.8.1-rc2
> > >> jiras: https://issues.apache.org/jira/projects/HIVE/versions/12350456
> > >>
> > >> .. Owen
> > >>
> >
> >
>

[RESULT][VOTE] Should we release Hive Storage API 2.7.3-rc2?

[Owen O'Malley]

With four +1's (3 binding) and no -1's the vote passes. I'll finalize the
release and then backport HIVE-23617 on to the release branch.

Thanks all for voting,
   Owen

On Tue, Aug 3, 2021 at 4:31 PM Owen O'Malley  wrote:

> Alright, I tracked this down to the default java character encoding. This
> was fixed in storage api 2.8 by HIVE-23617. With the environment variable
> setting the decoding, the test case passes.
>
> export JAVA_TOOL_OPTIONS=-Dfile.encoding=UTF-8
>
> .. Owen
>
> On Tue, Aug 3, 2021 at 12:29 AM Alan Gates  wrote:
>
>> Centos, java-1.8.0-openjdk-devel
>>
>> Alan.
>>
>> On Mon, Aug 2, 2021 at 5:20 PM Owen O'Malley 
>> wrote:
>>
>> > Ok, I was able to reproduce it in ORC's Ubuntu 18 docker image. I'll
>> track
>> > it down.
>> >
>> > On Tue, Aug 3, 2021 at 12:06 AM Owen O'Malley 
>> > wrote:
>> >
>> > > Hmm, I just reran the build both from git and the tar ball and
>> > > TestBytesColumnVector passed both times. Which jvm were you using?
>> > >
>> > > On Tue, Aug 3, 2021 at 12:02 AM Pavan Lanka > >
>> > > wrote:
>> > >
>> > >> +1 (non-binding)
>> > >>
>> > >> * Built Hive Storage API (did not get the failure that Alan called
>> out)
>> > >> using the tag
>> > >> * Ran the individual test TestBytesColumnVector successfully
>> > >> * Built ORC using 2.7.3 successfully
>> > >>
>> > >> Regards,
>> > >> Pavan
>> > >>
>> > >> > On Aug 2, 2021, at 4:03 PM, Alan Gates 
>> wrote:
>> > >> >
>> > >> > +1.  Checked the keys and the sha checksum.  Did a build and ran
>> the
>> > >> > tests.  Ran rat.  I did get one failing test, but it doesn't look
>> > >> crucial:
>> > >> >
>> > >> > TestBytesColumnVector.testConcatAndPadding:109 row 0 expected:<[?]>
>> > but
>> > >> > was:<[???]>
>> > >> >
>> > >> > Alan.
>> > >> >
>> > >> > On Thu, Jul 29, 2021 at 3:14 PM Owen O'Malley <
>> owen.omal...@gmail.com
>> > >
>> > >> > wrote:
>> > >> >
>> > >> >> Hello all,
>> > >> >>
>> > >> >> Following on previous discussions, I would like to propose a new
>> > >> >> storage-api release including HIVE-25400.
>> > >> >>
>> > >> >> Shall we release the following artifacts as Hive Storage API
>> 2.7.3?
>> > >> >>
>> > >> >> tar: http://home.apache.org/~omalley/hive-storage-2.7.3/
>> > >> >> tag:
>> > >>
>> https://github.com/apache/hive/releases/tag/storage-release-2.7.3-rc2
>> > >> >> jiras:
>> > https://issues.apache.org/jira/projects/HIVE/versions/12350287
>> > >> >>
>> > >> >> .. Owen
>> > >> >>
>> > >>
>> > >>
>> >
>>
>

Re: [VOTE] Should we release Hive Storage API 2.7.3-rc2?

[Owen O'Malley]

Alright, I tracked this down to the default java character encoding. This
was fixed in storage api 2.8 by HIVE-23617. With the environment variable
setting the decoding, the test case passes.

export JAVA_TOOL_OPTIONS=-Dfile.encoding=UTF-8

.. Owen

On Tue, Aug 3, 2021 at 12:29 AM Alan Gates  wrote:

> Centos, java-1.8.0-openjdk-devel
>
> Alan.
>
> On Mon, Aug 2, 2021 at 5:20 PM Owen O'Malley 
> wrote:
>
> > Ok, I was able to reproduce it in ORC's Ubuntu 18 docker image. I'll
> track
> > it down.
> >
> > On Tue, Aug 3, 2021 at 12:06 AM Owen O'Malley 
> > wrote:
> >
> > > Hmm, I just reran the build both from git and the tar ball and
> > > TestBytesColumnVector passed both times. Which jvm were you using?
> > >
> > > On Tue, Aug 3, 2021 at 12:02 AM Pavan Lanka 
> > > wrote:
> > >
> > >> +1 (non-binding)
> > >>
> > >> * Built Hive Storage API (did not get the failure that Alan called
> out)
> > >> using the tag
> > >> * Ran the individual test TestBytesColumnVector successfully
> > >> * Built ORC using 2.7.3 successfully
> > >>
> > >> Regards,
> > >> Pavan
> > >>
> > >> > On Aug 2, 2021, at 4:03 PM, Alan Gates 
> wrote:
> > >> >
> > >> > +1.  Checked the keys and the sha checksum.  Did a build and ran the
> > >> > tests.  Ran rat.  I did get one failing test, but it doesn't look
> > >> crucial:
> > >> >
> > >> > TestBytesColumnVector.testConcatAndPadding:109 row 0 expected:<[?]>
> > but
> > >> > was:<[???]>
> > >> >
> > >> > Alan.
> > >> >
> > >> > On Thu, Jul 29, 2021 at 3:14 PM Owen O'Malley <
> owen.omal...@gmail.com
> > >
> > >> > wrote:
> > >> >
> > >> >> Hello all,
> > >> >>
> > >> >> Following on previous discussions, I would like to propose a new
> > >> >> storage-api release including HIVE-25400.
> > >> >>
> > >> >> Shall we release the following artifacts as Hive Storage API 2.7.3?
> > >> >>
> > >> >> tar: http://home.apache.org/~omalley/hive-storage-2.7.3/
> > >> >> tag:
> > >> https://github.com/apache/hive/releases/tag/storage-release-2.7.3-rc2
> > >> >> jiras:
> > https://issues.apache.org/jira/projects/HIVE/versions/12350287
> > >> >>
> > >> >> .. Owen
> > >> >>
> > >>
> > >>
> >
>

Re: [VOTE] Should we release Hive Storage API 2.7.3-rc2?

[Owen O'Malley]

Ok, I was able to reproduce it in ORC's Ubuntu 18 docker image. I'll track
it down.

On Tue, Aug 3, 2021 at 12:06 AM Owen O'Malley 
wrote:

> Hmm, I just reran the build both from git and the tar ball and
> TestBytesColumnVector passed both times. Which jvm were you using?
>
> On Tue, Aug 3, 2021 at 12:02 AM Pavan Lanka 
> wrote:
>
>> +1 (non-binding)
>>
>> * Built Hive Storage API (did not get the failure that Alan called out)
>> using the tag
>> * Ran the individual test TestBytesColumnVector successfully
>> * Built ORC using 2.7.3 successfully
>>
>> Regards,
>> Pavan
>>
>> > On Aug 2, 2021, at 4:03 PM, Alan Gates  wrote:
>> >
>> > +1.  Checked the keys and the sha checksum.  Did a build and ran the
>> > tests.  Ran rat.  I did get one failing test, but it doesn't look
>> crucial:
>> >
>> > TestBytesColumnVector.testConcatAndPadding:109 row 0 expected:<[?]> but
>> > was:<[???]>
>> >
>> > Alan.
>> >
>> > On Thu, Jul 29, 2021 at 3:14 PM Owen O'Malley 
>> > wrote:
>> >
>> >> Hello all,
>> >>
>> >> Following on previous discussions, I would like to propose a new
>> >> storage-api release including HIVE-25400.
>> >>
>> >> Shall we release the following artifacts as Hive Storage API 2.7.3?
>> >>
>> >> tar: http://home.apache.org/~omalley/hive-storage-2.7.3/
>> >> tag:
>> https://github.com/apache/hive/releases/tag/storage-release-2.7.3-rc2
>> >> jiras: https://issues.apache.org/jira/projects/HIVE/versions/12350287
>> >>
>> >> .. Owen
>> >>
>>
>>

Re: [VOTE] Should we release Hive Storage API 2.7.3-rc2?

[Owen O'Malley]

Hmm, I just reran the build both from git and the tar ball and
TestBytesColumnVector passed both times. Which jvm were you using?

On Tue, Aug 3, 2021 at 12:02 AM Pavan Lanka 
wrote:

> +1 (non-binding)
>
> * Built Hive Storage API (did not get the failure that Alan called out)
> using the tag
> * Ran the individual test TestBytesColumnVector successfully
> * Built ORC using 2.7.3 successfully
>
> Regards,
> Pavan
>
> > On Aug 2, 2021, at 4:03 PM, Alan Gates  wrote:
> >
> > +1.  Checked the keys and the sha checksum.  Did a build and ran the
> > tests.  Ran rat.  I did get one failing test, but it doesn't look
> crucial:
> >
> > TestBytesColumnVector.testConcatAndPadding:109 row 0 expected:<[?]> but
> > was:<[???]>
> >
> > Alan.
> >
> > On Thu, Jul 29, 2021 at 3:14 PM Owen O'Malley 
> > wrote:
> >
> >> Hello all,
> >>
> >> Following on previous discussions, I would like to propose a new
> >> storage-api release including HIVE-25400.
> >>
> >> Shall we release the following artifacts as Hive Storage API 2.7.3?
> >>
> >> tar: http://home.apache.org/~omalley/hive-storage-2.7.3/
> >> tag:
> https://github.com/apache/hive/releases/tag/storage-release-2.7.3-rc2
> >> jiras: https://issues.apache.org/jira/projects/HIVE/versions/12350287
> >>
> >> .. Owen
> >>
>
>

[VOTE] Should we release Hive Storage API 2.7.3-rc2?

[Owen O'Malley]

Hello all,

Following on previous discussions, I would like to propose a new
storage-api release including HIVE-25400.

Shall we release the following artifacts as Hive Storage API 2.7.3?

tar: http://home.apache.org/~omalley/hive-storage-2.7.3/
tag: https://github.com/apache/hive/releases/tag/storage-release-2.7.3-rc2
jiras: https://issues.apache.org/jira/projects/HIVE/versions/12350287

.. Owen

[VOTE] Should we release Hive Storage API 2.8.1 rc2?

[Owen O'Malley]

Hello all,

I would like to propose a new storage-api release including HIVE-25400.

Shall we release the following artifacts as Hive Storage API 2.8.1?

tar: http://home.apache.org/~omalley/hive-storage-2.8.1/
tag: https://github.com/apache/hive/releases/tag/storage-release-2.8.1-rc2
jiras: https://issues.apache.org/jira/projects/HIVE/versions/12350456

.. Owen

Re: [VOTE] Should we release Hive Storage API 2.8.1 rc1?

[Owen O'Malley]

https://github.com/apache/hive/pull/2543

.. Owen

> On Jul 28, 2021, at 09:33, Dongjoon Hyun  wrote:
> 
> Thank you for sharing. What is the JIRA ID for redact issue?
> 
> Dongjoon
> 
>> On 2021/07/28 16:22:03, Owen O'Malley  wrote: 
>> Ok, pavan found that hive 25190 broke the redact mask code in orc. Let me 
>> fix that. 
>> 
>> .. Owen
>> 
>>>> On Jul 26, 2021, at 14:18, Owen O'Malley  wrote:
>>> 
>>> 
>>> Hello all,
>>> 
>>> I would like to propose a new storage-api release including HIVE-25386 and 
>>> HIVE-25190.
>>> 
>>> Shall we release the following artifacts as Hive Storage API 2.8.1?
>>> 
>>> tar: http://home.apache.org/~omalley/hive-storage-2.8.1/
>>> tag: https://github.com/apache/hive/releases/tag/storage-release-2.8.1-rc1
>>> jiras: https://issues.apache.org/jira/projects/HIVE/versions/12350456
>>> 
>>> .. Owen
>> 

[jira] [Created] (HIVE-25400) Move the offset updating in BytesColumnVector to setValPreallocated.

[Owen O'Malley (Jira)]

Owen O'Malley created HIVE-25400:


 Summary: Move the offset updating in BytesColumnVector to 
setValPreallocated.
 Key: HIVE-25400
 URL: https://issues.apache.org/jira/browse/HIVE-25400
 Project: Hive
  Issue Type: Bug
Reporter: Owen O'Malley
Assignee: Owen O'Malley
 Fix For: storage-2.7.3, storage-2.8.1, storage-2.9.0


HIVE-25190 changed the semantics of BytesColumnVector so that 
ensureValPreallocated reserved the room, which interacted badly with ORC's 
redact mask code. The redact mask code needs to be able to increase the 
allocation as it goes so it can call the ensureValPreallocated multiple times.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [VOTE] Should we release Hive Storage API 2.8.1 rc1?

[Owen O'Malley]

Ok, pavan found that hive 25190 broke the redact mask code in orc. Let me fix 
that. 

.. Owen

> On Jul 26, 2021, at 14:18, Owen O'Malley  wrote:
> 
> 
> Hello all,
> 
> I would like to propose a new storage-api release including HIVE-25386 and 
> HIVE-25190.
> 
> Shall we release the following artifacts as Hive Storage API 2.8.1?
> 
> tar: http://home.apache.org/~omalley/hive-storage-2.8.1/
> tag: https://github.com/apache/hive/releases/tag/storage-release-2.8.1-rc1
> jiras: https://issues.apache.org/jira/projects/HIVE/versions/12350456
> 
> .. Owen

[VOTE] Should we release Hive Storage API 2.7.3-rc1?

[Owen O'Malley]

Hello all,

Following on previous discussions, I would like to propose a new
storage-api release including HIVE-25190.

Shall we release the following artifacts as Hive Storage API 2.7.3?

tar: http://home.apache.org/~omalley/hive-storage-2.7.3/
tag: https://github.com/apache/hive/releases/tag/storage-release-2.7.3-rc1
jiras: https://issues.apache.org/jira/projects/HIVE/versions/12350287

.. Owen

[VOTE] Should we release Hive Storage API 2.8.1 rc1?

[Owen O'Malley]

Hello all,

I would like to propose a new storage-api release including HIVE-25386 and
HIVE-25190.

Shall we release the following artifacts as Hive Storage API 2.8.1?

tar: http://home.apache.org/~omalley/hive-storage-2.8.1/
tag: https://github.com/apache/hive/releases/tag/storage-release-2.8.1-rc1
jiras: https://issues.apache.org/jira/projects/HIVE/versions/12350456

.. Owen

Re: [VOTE] Should we release Hive Storage API 2.8.0-rc0 ?

[Owen O'Malley]

Thanks, Dongjoon.

I've committed the fix to master and storage-branch-2.8. Given that I
wanted to get HIVE-25190 released,
I'll roll RCs for 2.8.1 and 2.7.3 today.

.. Owen

On Mon, Jul 26, 2021 at 8:36 AM Panos Garefalakis 
wrote:

> Thanks Dongjoon for the details!
> Looks like we are going to need a follow-up release sooner than later!
>
> On the positive side, HIVE-25386 <https://github.com/apache/hive/pull/2531
> >
> passed tests and we can now move forward.
>
> Cheers,
> Panagiotis
>
> On Mon, Jul 26, 2021 at 9:56 AM Dongjoon Hyun  wrote:
>
> > Hi, All.
> >
> > It turns out that 2.8.0 has a regression in terms of the dependency
> > although it passed all tests.
> > At Apache Hive 2.2.0, HIVE-13906 removed the Guava compile dependency.
> > However, recently, HIVE-24542 converts the test dependency to the compile
> > dependency.
> >
> >
> >
> https://github.com/apache/hive/pull/1784/files#diff-e5e45b8481f3d87f4fa561d45b22fa20bb5084e5240d825717e6bde511b733baL122
> >
> > I noticed this breaking dependency change during Apache ORC integration
> > stage.
> >
> > https://github.com/apache/orc/pull/773
> >
> > For now, I filed a JIRA issue and made a PR. Please comment your opinion
> > on the Hive and ORC PRs.
> > - HIVE-25386: hive-storage-api should not have guava compile dependency
> > - https://github.com/apache/hive/pull/2531
> >
> > Bests,
> > Dongjoon.
> >
> > On 2021/07/23 16:40:39, "Owen O'Malley"  wrote:
> > > I can do the publishing. Thank you all!
> > >
> > > .. Owen
> > >
> > > On Fri, Jul 23, 2021 at 10:16 AM Panos Garefalakis  >
> > > wrote:
> > >
> > > > Thank you all for your help! With 3 binding votes this release
> passed!
> > > > I am going to send a vote result follow up but will need some PMC
> help
> > > > creating the new Storage API version (on Jira) and releasing (upload
> > > > artifacts)!
> > > >
> > > > Cheers,
> > > > Panagiotis
> > > >
> > > > On Fri, Jul 23, 2021 at 2:21 AM Chao Sun  wrote:
> > > >
> > > > > Thanks Owen! I just verified the checksum and gpg signature and
> they
> > both
> > > > > look good, so +1 too.
> > > > >
> > > > > Panos: please fix the "Fix version" of the JIRA when you get a
> > chance.
> > > > > Thanks.
> > > > >
> > > > > Best,
> > > > > Chao
> > > > >
> > > > >
> > > > > On Thu, Jul 22, 2021 at 2:29 PM Owen O'Malley <
> > owen.omal...@gmail.com>
> > > > > wrote:
> > > > >
> > > > >> Chao,
> > > > >>Panos key doesn't seem to have propagated to the Apache
> servers.
> > It
> > > > >> referenced here:
> > > > >>
> > > > >> https://people.apache.org/keys/committer/ as "pgaref
> > > > >> 7DFAB216AB7D96B3B2072184DC11DE4D00F8FA1D"
> > > > >>
> > > > >> The key itself can be found here:
> > > > >>
> > > > >>
> > > > >>
> > > >
> >
> https://keyserver.ubuntu.com/pks/lookup?search=pgaref=on=index
> > > > >>
> > > > >>
> > > > >> On Wed, Jul 21, 2021 at 8:47 PM Chao Sun 
> > wrote:
> > > > >>
> > > > >> > I built the source from the branch and ran the tests, which all
> > > > passed.
> > > > >> > However I was not able to find the public GPG key. Panos: could
> > you
> > > > >> point
> > > > >> > me to the location?
> > > > >> >
> > > > >> > Also seems we should create a new version 2.8.0 in the JIRA
> page:
> > > > >> >
> > > > >> >
> > > > >>
> > > >
> >
> https://issues.apache.org/jira/projects/HIVE?selectedItem=com.atlassian.jira.jira-projects-plugin:release-page
> > > > >> > and update "Fix version" of
> > > > >> > https://issues.apache.org/jira/browse/HIVE-24458
> > > > >> > .
> > > > >> >
> > > > >> > Chao
> > > > >> >
> > > > >> > On Wed, Jul 21, 2021 at 9:10 AM Szehon Ho <
> > szehon.apa...@gmail.com>
>

Re: [VOTE] Should we release Hive Storage API 2.8.0-rc0 ?

[Owen O'Malley]

I can do the publishing. Thank you all!

.. Owen

On Fri, Jul 23, 2021 at 10:16 AM Panos Garefalakis 
wrote:

> Thank you all for your help! With 3 binding votes this release passed!
> I am going to send a vote result follow up but will need some PMC help
> creating the new Storage API version (on Jira) and releasing (upload
> artifacts)!
>
> Cheers,
> Panagiotis
>
> On Fri, Jul 23, 2021 at 2:21 AM Chao Sun  wrote:
>
> > Thanks Owen! I just verified the checksum and gpg signature and they both
> > look good, so +1 too.
> >
> > Panos: please fix the "Fix version" of the JIRA when you get a chance.
> > Thanks.
> >
> > Best,
> > Chao
> >
> >
> > On Thu, Jul 22, 2021 at 2:29 PM Owen O'Malley 
> > wrote:
> >
> >> Chao,
> >>Panos key doesn't seem to have propagated to the Apache servers. It
> >> referenced here:
> >>
> >> https://people.apache.org/keys/committer/ as "pgaref
> >> 7DFAB216AB7D96B3B2072184DC11DE4D00F8FA1D"
> >>
> >> The key itself can be found here:
> >>
> >>
> >>
> https://keyserver.ubuntu.com/pks/lookup?search=pgaref=on=index
> >>
> >>
> >> On Wed, Jul 21, 2021 at 8:47 PM Chao Sun  wrote:
> >>
> >> > I built the source from the branch and ran the tests, which all
> passed.
> >> > However I was not able to find the public GPG key. Panos: could you
> >> point
> >> > me to the location?
> >> >
> >> > Also seems we should create a new version 2.8.0 in the JIRA page:
> >> >
> >> >
> >>
> https://issues.apache.org/jira/projects/HIVE?selectedItem=com.atlassian.jira.jira-projects-plugin:release-page
> >> > and update "Fix version" of
> >> > https://issues.apache.org/jira/browse/HIVE-24458
> >> > .
> >> >
> >> > Chao
> >> >
> >> > On Wed, Jul 21, 2021 at 9:10 AM Szehon Ho 
> >> wrote:
> >> >
> >> > > +1 (binding)
> >> > >
> >> > > * Built module
> >> > > * Ran tests
> >> > > * Checked artifact checksum and signature
> >> > >
> >> > > Thanks
> >> > > Szehon
> >> > >
> >> > > On Tue, Jul 20, 2021 at 2:11 PM Owen O'Malley <
> owen.omal...@gmail.com
> >> >
> >> > > wrote:
> >> > >
> >> > > > I think we should go ahead and release storage-api 2.8.0 and catch
> >> it
> >> > on
> >> > > > the next cycle. HIVE-25190 is a long standing bug that rarely
> >> affects
> >> > > > users. (We have had a user at LinkedIn hit it, which is why I
> fixed
> >> > it.)
> >> > > > I'll sign up to make the 2.8.1 (and 2.7.3) bug fix releases
> >> afterwards.
> >> > > >
> >> > > > .. Owen
> >> > > >
> >> > > > On Tue, Jul 20, 2021 at 8:53 PM Chao Sun 
> >> wrote:
> >> > > >
> >> > > > > Going to check the release and vote here too. Since HIVE-25190
> is
> >> > > already
> >> > > > > merged, instead of waiting for another release, should we start
> >> > another
> >> > > > RC1
> >> > > > > with that included?
> >> > > > >
> >> > > > > Chao
> >> > > > >
> >> > > > > On Tue, Jul 20, 2021 at 1:30 PM Dongjoon Hyun <
> >> dongj...@apache.org>
> >> > > > wrote:
> >> > > > >
> >> > > > > > +1
> >> > > > > >
> >> > > > > > * Build and tested locally.
> >> > > > > >
> >> > > > > > Thanks,
> >> > > > > > Dongjoon.
> >> > > > > >
> >> > > > > > On 2021/07/19 23:15:46, "Owen O'Malley" <
> owen.omal...@gmail.com
> >> >
> >> > > > wrote:
> >> > > > > > > +1 (binding):
> >> > > > > > > * Built and tested
> >> > > > > > > * Built hive main branch using it
> >> > > > > > > * Verified signatures and checksums
> >> > > > > > >
> >> > > > > > > It is too bad that we didn't get HIVE-25190 into it, but
> that
>

Re: [VOTE] Should we release Hive Storage API 2.8.0-rc0 ?

[Owen O'Malley]

Chao,
   Panos key doesn't seem to have propagated to the Apache servers. It
referenced here:

https://people.apache.org/keys/committer/ as "pgaref
7DFAB216AB7D96B3B2072184DC11DE4D00F8FA1D"

The key itself can be found here:

https://keyserver.ubuntu.com/pks/lookup?search=pgaref=on=index


On Wed, Jul 21, 2021 at 8:47 PM Chao Sun  wrote:

> I built the source from the branch and ran the tests, which all passed.
> However I was not able to find the public GPG key. Panos: could you point
> me to the location?
>
> Also seems we should create a new version 2.8.0 in the JIRA page:
>
> https://issues.apache.org/jira/projects/HIVE?selectedItem=com.atlassian.jira.jira-projects-plugin:release-page
> and update "Fix version" of
> https://issues.apache.org/jira/browse/HIVE-24458
> .
>
> Chao
>
> On Wed, Jul 21, 2021 at 9:10 AM Szehon Ho  wrote:
>
> > +1 (binding)
> >
> > * Built module
> > * Ran tests
> > * Checked artifact checksum and signature
> >
> > Thanks
> > Szehon
> >
> > On Tue, Jul 20, 2021 at 2:11 PM Owen O'Malley 
> > wrote:
> >
> > > I think we should go ahead and release storage-api 2.8.0 and catch it
> on
> > > the next cycle. HIVE-25190 is a long standing bug that rarely affects
> > > users. (We have had a user at LinkedIn hit it, which is why I fixed
> it.)
> > > I'll sign up to make the 2.8.1 (and 2.7.3) bug fix releases afterwards.
> > >
> > > .. Owen
> > >
> > > On Tue, Jul 20, 2021 at 8:53 PM Chao Sun  wrote:
> > >
> > > > Going to check the release and vote here too. Since HIVE-25190 is
> > already
> > > > merged, instead of waiting for another release, should we start
> another
> > > RC1
> > > > with that included?
> > > >
> > > > Chao
> > > >
> > > > On Tue, Jul 20, 2021 at 1:30 PM Dongjoon Hyun 
> > > wrote:
> > > >
> > > > > +1
> > > > >
> > > > > * Build and tested locally.
> > > > >
> > > > > Thanks,
> > > > > Dongjoon.
> > > > >
> > > > > On 2021/07/19 23:15:46, "Owen O'Malley" 
> > > wrote:
> > > > > > +1 (binding):
> > > > > > * Built and tested
> > > > > > * Built hive main branch using it
> > > > > > * Verified signatures and checksums
> > > > > >
> > > > > > It is too bad that we didn't get HIVE-25190 into it, but that can
> > > wait
> > > > > for
> > > > > > 2.8.1.
> > > > > >
> > > > > > .. Owen
> > > > > >
> > > > > > On Mon, Jun 28, 2021 at 9:44 PM Pavan Lanka
> >  > > >
> > > > > > wrote:
> > > > > >
> > > > > > > +1 (non-binding)
> > > > > > >
> > > > > > > I have done the following:
> > > > > > > * Built and Tested storage-release-2.8.0-rc0 using OpenJDK8
> > > > > > > * Built and Tested ORC with updated storage api version
> > > > > > >   - Had to fix a test class that implements PredicateLeaf which
> > > has a
> > > > > new
> > > > > > > method. This is a breaking change but I think this should be ok
> > > > > > > * Verified the performance gains of HIVE-24458
> > > > > > >
> > > > > > > Regards,
> > > > > > > Pavan
> > > > > > >
> > > > > > >
> > > > > > > > On Jun 21, 2021, at 8:07 AM, Panos Garefalakis <
> > > panga...@gmail.com
> > > > >
> > > > > > > wrote:
> > > > > > > >
> > > > > > > > Hello all,
> > > > > > > >
> > > > > > > > Following on previous discussions, I would like to propose a
> > new
> > > > > > > > storage-api release including HIVE-24458
> > > > > > > > <https://issues.apache.org/jira/browse/HIVE-24458>.
> > > > > > > >
> > > > > > > > Shall we release the following artifacts as Hive Storage API
> > > 2.8.0?
> > > > > > > >
> > > > > > > > tar: http://home.apache.org/~pgaref/hive-storage-2.8.0/
> > > > > > > > tag:
> > > > > > >
> > > >
> https://github.com/apache/hive/releases/tag/storage-release-2.8.0-rc0
> > > > > > > > jiras:
> > > > > https://issues.apache.org/jira/projects/HIVE/versions/12350287
> > > > > > > >
> > > > > > > > Cheers,
> > > > > > > > Panagiotis
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: [VOTE] Should we release Hive Storage API 2.8.0-rc0 ?

[Owen O'Malley]

I think we should go ahead and release storage-api 2.8.0 and catch it on
the next cycle. HIVE-25190 is a long standing bug that rarely affects
users. (We have had a user at LinkedIn hit it, which is why I fixed it.)
I'll sign up to make the 2.8.1 (and 2.7.3) bug fix releases afterwards.

.. Owen

On Tue, Jul 20, 2021 at 8:53 PM Chao Sun  wrote:

> Going to check the release and vote here too. Since HIVE-25190 is already
> merged, instead of waiting for another release, should we start another RC1
> with that included?
>
> Chao
>
> On Tue, Jul 20, 2021 at 1:30 PM Dongjoon Hyun  wrote:
>
> > +1
> >
> > * Build and tested locally.
> >
> > Thanks,
> > Dongjoon.
> >
> > On 2021/07/19 23:15:46, "Owen O'Malley"  wrote:
> > > +1 (binding):
> > > * Built and tested
> > > * Built hive main branch using it
> > > * Verified signatures and checksums
> > >
> > > It is too bad that we didn't get HIVE-25190 into it, but that can wait
> > for
> > > 2.8.1.
> > >
> > > .. Owen
> > >
> > > On Mon, Jun 28, 2021 at 9:44 PM Pavan Lanka 
> > > wrote:
> > >
> > > > +1 (non-binding)
> > > >
> > > > I have done the following:
> > > > * Built and Tested storage-release-2.8.0-rc0 using OpenJDK8
> > > > * Built and Tested ORC with updated storage api version
> > > >   - Had to fix a test class that implements PredicateLeaf which has a
> > new
> > > > method. This is a breaking change but I think this should be ok
> > > > * Verified the performance gains of HIVE-24458
> > > >
> > > > Regards,
> > > > Pavan
> > > >
> > > >
> > > > > On Jun 21, 2021, at 8:07 AM, Panos Garefalakis  >
> > > > wrote:
> > > > >
> > > > > Hello all,
> > > > >
> > > > > Following on previous discussions, I would like to propose a new
> > > > > storage-api release including HIVE-24458
> > > > > <https://issues.apache.org/jira/browse/HIVE-24458>.
> > > > >
> > > > > Shall we release the following artifacts as Hive Storage API 2.8.0?
> > > > >
> > > > > tar: http://home.apache.org/~pgaref/hive-storage-2.8.0/
> > > > > tag:
> > > >
> https://github.com/apache/hive/releases/tag/storage-release-2.8.0-rc0
> > > > > jiras:
> > https://issues.apache.org/jira/projects/HIVE/versions/12350287
> > > > >
> > > > > Cheers,
> > > > > Panagiotis
> > > >
> > > >
> > >
> >
>

Re: [VOTE] Should we release Hive Storage API 2.8.0-rc0 ?

[Owen O'Malley]

+1 (binding):
* Built and tested
* Built hive main branch using it
* Verified signatures and checksums

It is too bad that we didn't get HIVE-25190 into it, but that can wait for
2.8.1.

.. Owen

On Mon, Jun 28, 2021 at 9:44 PM Pavan Lanka 
wrote:

> +1 (non-binding)
>
> I have done the following:
> * Built and Tested storage-release-2.8.0-rc0 using OpenJDK8
> * Built and Tested ORC with updated storage api version
>   - Had to fix a test class that implements PredicateLeaf which has a new
> method. This is a breaking change but I think this should be ok
> * Verified the performance gains of HIVE-24458
>
> Regards,
> Pavan
>
>
> > On Jun 21, 2021, at 8:07 AM, Panos Garefalakis 
> wrote:
> >
> > Hello all,
> >
> > Following on previous discussions, I would like to propose a new
> > storage-api release including HIVE-24458
> > .
> >
> > Shall we release the following artifacts as Hive Storage API 2.8.0?
> >
> > tar: http://home.apache.org/~pgaref/hive-storage-2.8.0/
> > tag:
> https://github.com/apache/hive/releases/tag/storage-release-2.8.0-rc0
> > jiras: https://issues.apache.org/jira/projects/HIVE/versions/12350287
> >
> > Cheers,
> > Panagiotis
>
>

Re: [VOTE] Should we release Hive Storage API 2.7.3-rc0 ?

[Owen O'Malley]

-1 (binding)

Looking at the git history, you've cut a branch from hive's master to make
a new release branch, but used the previous release's number (2.7).

Since HIVE-24458 is a performance/new functionality it should probably be a
new release. I'll rename the branch from storage-branch-2.7.3 to
storage-branch-2.8.

.. Owen

On Fri, Jun 11, 2021 at 9:46 PM Pavan Lanka 
wrote:

> +1 (non binding)
>
> Built and verified unit tests
> Built and tested ORC
> Verified HIVE-24458 functionality and performance gains
>
> Regards,
> Pavan
>
>
> > On Jun 8, 2021, at 2:14 AM, Panos Garefalakis 
> wrote:
> >
> > Hello all,
> >
> > I would like to propose a new storage-api release with HIVE-24458
> >  in it (as discussed
> in a
> > previous thead).
> >
> > Shall we release the following artifacts as Hive Storage API 2.7.3?
> >
> > tar: http://home.apache.org/~pgaref/hive-storage-2.7.3/
> > tag:
> https://github.com/apache/hive/releases/tag/storage-release-2.7.3-rc0
> > jiras: https://issues.apache.org/jira/projects/HIVE/versions/12350287
> >
> > Cheers,
> > Panagiotis
>
>

[jira] [Created] (HIVE-25190) BytesColumnVector fails when the aggregate size is > 1gb

[Owen O'Malley (Jira)]

Owen O'Malley created HIVE-25190:


 Summary: BytesColumnVector fails when the aggregate size is > 1gb
 Key: HIVE-25190
 URL: https://issues.apache.org/jira/browse/HIVE-25190
 Project: Hive
  Issue Type: Bug
Reporter: Owen O'Malley
Assignee: Owen O'Malley


Currently, BytesColumnVector will allocate a buffer for small values (< 1mb), 
but fail with:

{code:java}
new RuntimeException("Overflow of newLength. smallBuffer.length="
+ smallBuffer.length + ", nextElemLength=" + nextElemLength);
{code:java}

if the aggregate size of the buffer crosses over 1gb. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Review for SArg patch

[Owen O'Malley]

All,
   I've had a patch out for review https://github.com/apache/hive/pull/1726

that is needed for an improvement for ORC predicate pushdown. It would be
great to get it reviewed soon.

  It allows the client code to get the unnormalized expression of the SArg,
which is typically much smaller.

Thanks,
   Owen

Re: Upgrade ORC to 1.6

[Owen O'Malley]

This is great news! Thank you for your hard work to make it happen. Getting
ORC 1.6.7 released shouldn't be hard.

.. Owen

On Fri, Jan 15, 2021 at 5:00 PM Dongjoon Hyun  wrote:

> Thank you, Panagiotis and Laszio.
>
> This is a big move forward in both Hive and Hive project.
>
> ORC community will start to release 1.6.7 next week.
>
> Bests,
> Dongjoon.
>
> On 2021/01/15 14:03:13, Panos Garefalakis  wrote:
> > Hey Laszlo,
> >
> > Thanks for taking a look!
> > Yes, releasing ORC-1.6.7 is definitely the way to go (snapshot was only
> > used for testing) and I am planning to start a discussion on the ORC list
> > soon.
> >
> > However, I wanted to get feedback from the Hive community first and
> address
> > any potential concerns for this (significant) change -- so please share
> > your thoughts on the PR :)
> >
> > Cheers,
> > Panagiotis
> >
> > On Fri, Jan 15, 2021 at 2:52 PM László Bodor 
> > wrote:
> >
> > > Hi!
> > >
> > > Thanks, Panos, that's very cool!
> > > I guess we won't depend on a snapshot, is there a chance that ORC
> > > community can fix issues in the near future and release 1.6.7
> officially to
> > > depend upon?
> > >
> > > Regards,
> > > Laszlo Bodor
> > >
> > >
> > > Panos Garefalakis  ezt írta (időpont: 2021. jan.
> 14.,
> > > Cs, 18:33):
> > >
> > >> Hello Hive team,
> > >>
> > >> I am happy to announce that as of today Apache HIve precommit tests
> with
> > >> ORC-1.6.7 (snapshot) are passing on master branch!
> > >> There were more than a few compatibility and bug fixes required to
> make
> > >> the
> > >> jump from 1.5 but the version upgrade actually enables (among other
> > >> things):
> > >>
> > >>- Column-level encryption (ORC-14)
> > >>- Timestamps with local timezone (ORC-189)
> > >>- Zstd support (ORC-636)
> > >>- String stats min/max value trimming (ORC-203)
> > >>- Timestamp predicates with nanosecond precision (ORC-611)
> > >>
> > >> Please feel free to share your feedback as part of HIVE-23553
> > >> 
> > >> Of course, we still maintain a lot of duplicate logic as part of the
> LLAP
> > >> engine that can be optimized down the line -- planning to add a few
> more
> > >> subtasks there.
> > >>
> > >> Cheers,
> > >> Panagiotis
> > >>
> > >
> >
>

[jira] [Created] (HIVE-24458) Allow access to SArgs without converting to disjunctive normal form

[Owen O'Malley (Jira)]

Owen O'Malley created HIVE-24458:


 Summary: Allow access to SArgs without converting to disjunctive 
normal form
 Key: HIVE-24458
 URL: https://issues.apache.org/jira/browse/HIVE-24458
 Project: Hive
  Issue Type: Improvement
Reporter: Owen O'Malley
Assignee: Owen O'Malley


For some use cases, it is useful to have access to the SArg expression in a 
non-normalized form. Currently, the SArg only provides the fully normalized 
expression.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HIVE-24455) Fix broken junit framework in storage-api

[Owen O'Malley (Jira)]

Owen O'Malley created HIVE-24455:


 Summary: Fix broken junit framework in storage-api
 Key: HIVE-24455
 URL: https://issues.apache.org/jira/browse/HIVE-24455
 Project: Hive
  Issue Type: Bug
Reporter: Owen O'Malley
Assignee: Owen O'Malley


The use of junit is broken in storage-api. It results in no tests being found.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: Need privilege to create a new release version in Hive JIRA

[Owen O'Malley]

I created the 2.3.8 release for you. It isn't clear which authorization is
required to create a new release.

.. Owen

On Tue, Nov 24, 2020 at 7:20 PM Chao Sun  wrote:

> Hi all,
>
> As mentioned in a separate email, I'm preparing for the new 2.3.8 release.
> However, currently in JIRA there is no release version for 2.3.8 yet, and I
> don't seem to have the privilege to create it. Can someone grant me the
> permission? Thanks!
>
> Best,
> Chao
>

Re: Your project's website

[Owen O'Malley]

I committed the change to master, thanks Zoltan and John for looking at the
patch.

I filed https://issues.apache.org/jira/browse/INFRA-20776 to move the site.

.. Owen

On Thu, Aug 27, 2020 at 4:06 PM Zoltan Haindrich  wrote:

> Hey Owen!
>
> It looks great!
> Thank you for modernizing the site...we will be able to change the site
> more easily after this change
> +1 merging it
>
> cheers,
> Zoltan
>
>
> On 8/26/20 5:52 PM, Owen O'Malley wrote:
> > Is there any more feedback or concerns on the proposed site?
> >
> > If not, I'll go ahead and check it in and file the jira for Apache Infra
> to
> > move the hosting for our site.
> >
> > .. Owen
> >
> > On Tue, Aug 18, 2020 at 2:05 PM Owen O'Malley 
> > wrote:
> >
> >> All,
> >> I realized that I could mark my fork of the Hive repository as
> having
> >> github pages to effectively publish them. It took some changes, but now
> it
> >> is live on http://omalley.github.io/hive .
> >>
> >> On Tue, Aug 18, 2020 at 6:14 AM Andrew Wetmore 
> wrote:
> >>
> >>> Good progress! We will need an official PMC statement that Infra can
> >>> close down the project presence on the CMS.
> >>>
> >>> Thanks!
> >>>
> >>> Andrew
> >>>
> >>> On Tue, Aug 18, 2020 at 4:09 AM Owen O'Malley 
> >>> wrote:
> >>>
> >>>> All,
> >>>>
> >>>>   I ported the current site to github pages and pushed it to
> >>>> https://github.com/apache/hive/pull/1410 .
> >>>>
> >>>> Take a look. It will certainly be much easier to maintain.
> >>>>
> >>>> .. Owen
> >>>>
> >>>> On Mon, Aug 17, 2020 at 6:26 AM Andrew Wetmore 
> >>>> wrote:
> >>>>
> >>>>> Hi:
> >>>>>
> >>>>> I wrote to you two weeks ago about your project website, but have not
> >>>>> seen
> >>>>> a response. Whom should I contact directly? I am trying to see
> whether
> >>>>> your
> >>>>> project still uses the Apache CMS and, if so, what your plans are to
> >>>>> migrate off it.
> >>>>>
> >>>>> Thanks in advance for your help.
> >>>>>
> >>>>> Andrew
> >>>>>
> >>>>> On Wed, Aug 5, 2020 at 9:17 AM Andrew Wetmore 
> >>>>> wrote:
> >>>>>
> >>>>>> Hi:
> >>>>>>
> >>>>>> I am part of the Infrastructure team, and am writing to ask whether
> >>>>> your
> >>>>>> project is still using the Apache CMS for your project website. As
> you
> >>>>>> know, the CMS is reaching end-of-life, and we need projects to move
> >>>>> their
> >>>>>> websites onto a different option within the next few weeks.
> >>>>>>
> >>>>>> There are several alternatives available, including those listed on
> >>>>> this
> >>>>>> page [1] on managing project websites. Infra is assembling a Wiki
> >>>>> page [2]
> >>>>>> on migrating a website from the CMS, and is looking forward to
> helping
> >>>>>> projects with this transition.
> >>>>>>
> >>>>>> Please let me know whether your site is still on the Apache CMS and,
> >>>>> if
> >>>>>> so, who will be the project point-of-contact with Infra for the
> >>>>> migration.
> >>>>>>
> >>>>>> Thank you!
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> [1] https://infra.apache.org/project-site.html
> >>>>>>
> >>>>>> [2]
> >>>>>>
> >>>>>
> https://cwiki.apache.org/confluence/display/INFRA/Migrate+your+project+website+from+the+Apache+CMS
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> --
> >>>>>> Andrew Wetmore
> >>>>>> Technical Writer-Editor
> >>>>>> Infra
> >>>>>> *Apache Software Foundation*
> >>>>>> andr...@apache.org
> >>>>>>
> >>>>>>
> >>>>>> <
> >>>>>
> https://www.avast.com/sig-email?utm_medium=email_source=link_campaign=sig-email_content=webmail
> >
> >>>>> Virus-free.
> >>>>>> www.avast.com
> >>>>>> <
> >>>>>
> https://www.avast.com/sig-email?utm_medium=email_source=link_campaign=sig-email_content=webmail
> >>>>>>
> >>>>>> <#m_2078696594329065464_DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>
> >>>>>>
> >>>>>
> >>>>>
> >>>>> --
> >>>>> Andrew Wetmore
> >>>>> Technical Writer-Editor
> >>>>> Infra
> >>>>> *Apache Software Foundation*
> >>>>> andr...@apache.org
> >>>>>
> >>>>
> >>>
> >>> --
> >>> Andrew Wetmore
> >>> Technical Writer-Editor
> >>> Infra
> >>> *Apache Software Foundation*
> >>> andr...@apache.org
> >>>
> >>
> >
>

Re: Your project's website

[Owen O'Malley]

Is there any more feedback or concerns on the proposed site?

If not, I'll go ahead and check it in and file the jira for Apache Infra to
move the hosting for our site.

.. Owen

On Tue, Aug 18, 2020 at 2:05 PM Owen O'Malley 
wrote:

> All,
>I realized that I could mark my fork of the Hive repository as having
> github pages to effectively publish them. It took some changes, but now it
> is live on http://omalley.github.io/hive .
>
> On Tue, Aug 18, 2020 at 6:14 AM Andrew Wetmore  wrote:
>
>> Good progress! We will need an official PMC statement that Infra can
>> close down the project presence on the CMS.
>>
>> Thanks!
>>
>> Andrew
>>
>> On Tue, Aug 18, 2020 at 4:09 AM Owen O'Malley 
>> wrote:
>>
>>> All,
>>>
>>>  I ported the current site to github pages and pushed it to
>>> https://github.com/apache/hive/pull/1410 .
>>>
>>> Take a look. It will certainly be much easier to maintain.
>>>
>>> .. Owen
>>>
>>> On Mon, Aug 17, 2020 at 6:26 AM Andrew Wetmore 
>>> wrote:
>>>
>>>> Hi:
>>>>
>>>> I wrote to you two weeks ago about your project website, but have not
>>>> seen
>>>> a response. Whom should I contact directly? I am trying to see whether
>>>> your
>>>> project still uses the Apache CMS and, if so, what your plans are to
>>>> migrate off it.
>>>>
>>>> Thanks in advance for your help.
>>>>
>>>> Andrew
>>>>
>>>> On Wed, Aug 5, 2020 at 9:17 AM Andrew Wetmore 
>>>> wrote:
>>>>
>>>> > Hi:
>>>> >
>>>> > I am part of the Infrastructure team, and am writing to ask whether
>>>> your
>>>> > project is still using the Apache CMS for your project website. As you
>>>> > know, the CMS is reaching end-of-life, and we need projects to move
>>>> their
>>>> > websites onto a different option within the next few weeks.
>>>> >
>>>> > There are several alternatives available, including those listed on
>>>> this
>>>> > page [1] on managing project websites. Infra is assembling a Wiki
>>>> page [2]
>>>> > on migrating a website from the CMS, and is looking forward to helping
>>>> > projects with this transition.
>>>> >
>>>> > Please let me know whether your site is still on the Apache CMS and,
>>>> if
>>>> > so, who will be the project point-of-contact with Infra for the
>>>> migration.
>>>> >
>>>> > Thank you!
>>>> >
>>>> >
>>>> >
>>>> >
>>>> > [1] https://infra.apache.org/project-site.html
>>>> >
>>>> > [2]
>>>> >
>>>> https://cwiki.apache.org/confluence/display/INFRA/Migrate+your+project+website+from+the+Apache+CMS
>>>> >
>>>> >
>>>> >
>>>> > --
>>>> > Andrew Wetmore
>>>> > Technical Writer-Editor
>>>> > Infra
>>>> > *Apache Software Foundation*
>>>> > andr...@apache.org
>>>> >
>>>> >
>>>> > <
>>>> https://www.avast.com/sig-email?utm_medium=email_source=link_campaign=sig-email_content=webmail>
>>>> Virus-free.
>>>> > www.avast.com
>>>> > <
>>>> https://www.avast.com/sig-email?utm_medium=email_source=link_campaign=sig-email_content=webmail
>>>> >
>>>> > <#m_2078696594329065464_DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>
>>>> >
>>>>
>>>>
>>>> --
>>>> Andrew Wetmore
>>>> Technical Writer-Editor
>>>> Infra
>>>> *Apache Software Foundation*
>>>> andr...@apache.org
>>>>
>>>
>>
>> --
>> Andrew Wetmore
>> Technical Writer-Editor
>> Infra
>> *Apache Software Foundation*
>> andr...@apache.org
>>
>

Re: Your project's website

[Owen O'Malley]

All,
   I realized that I could mark my fork of the Hive repository as having
github pages to effectively publish them. It took some changes, but now it
is live on http://omalley.github.io/hive .

On Tue, Aug 18, 2020 at 6:14 AM Andrew Wetmore  wrote:

> Good progress! We will need an official PMC statement that Infra can close
> down the project presence on the CMS.
>
> Thanks!
>
> Andrew
>
> On Tue, Aug 18, 2020 at 4:09 AM Owen O'Malley 
> wrote:
>
>> All,
>>
>>  I ported the current site to github pages and pushed it to
>> https://github.com/apache/hive/pull/1410 .
>>
>> Take a look. It will certainly be much easier to maintain.
>>
>> .. Owen
>>
>> On Mon, Aug 17, 2020 at 6:26 AM Andrew Wetmore 
>> wrote:
>>
>>> Hi:
>>>
>>> I wrote to you two weeks ago about your project website, but have not
>>> seen
>>> a response. Whom should I contact directly? I am trying to see whether
>>> your
>>> project still uses the Apache CMS and, if so, what your plans are to
>>> migrate off it.
>>>
>>> Thanks in advance for your help.
>>>
>>> Andrew
>>>
>>> On Wed, Aug 5, 2020 at 9:17 AM Andrew Wetmore 
>>> wrote:
>>>
>>> > Hi:
>>> >
>>> > I am part of the Infrastructure team, and am writing to ask whether
>>> your
>>> > project is still using the Apache CMS for your project website. As you
>>> > know, the CMS is reaching end-of-life, and we need projects to move
>>> their
>>> > websites onto a different option within the next few weeks.
>>> >
>>> > There are several alternatives available, including those listed on
>>> this
>>> > page [1] on managing project websites. Infra is assembling a Wiki page
>>> [2]
>>> > on migrating a website from the CMS, and is looking forward to helping
>>> > projects with this transition.
>>> >
>>> > Please let me know whether your site is still on the Apache CMS and, if
>>> > so, who will be the project point-of-contact with Infra for the
>>> migration.
>>> >
>>> > Thank you!
>>> >
>>> >
>>> >
>>> >
>>> > [1] https://infra.apache.org/project-site.html
>>> >
>>> > [2]
>>> >
>>> https://cwiki.apache.org/confluence/display/INFRA/Migrate+your+project+website+from+the+Apache+CMS
>>> >
>>> >
>>> >
>>> > --
>>> > Andrew Wetmore
>>> > Technical Writer-Editor
>>> > Infra
>>> > *Apache Software Foundation*
>>> > andr...@apache.org
>>> >
>>> >
>>> > <
>>> https://www.avast.com/sig-email?utm_medium=email_source=link_campaign=sig-email_content=webmail>
>>> Virus-free.
>>> > www.avast.com
>>> > <
>>> https://www.avast.com/sig-email?utm_medium=email_source=link_campaign=sig-email_content=webmail
>>> >
>>> > <#m_2078696594329065464_DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>
>>> >
>>>
>>>
>>> --
>>> Andrew Wetmore
>>> Technical Writer-Editor
>>> Infra
>>> *Apache Software Foundation*
>>> andr...@apache.org
>>>
>>
>
> --
> Andrew Wetmore
> Technical Writer-Editor
> Infra
> *Apache Software Foundation*
> andr...@apache.org
>

Re: Your project's website

[Owen O'Malley]

All,

 I ported the current site to github pages and pushed it to
https://github.com/apache/hive/pull/1410 .

Take a look. It will certainly be much easier to maintain.

.. Owen

On Mon, Aug 17, 2020 at 6:26 AM Andrew Wetmore  wrote:

> Hi:
>
> I wrote to you two weeks ago about your project website, but have not seen
> a response. Whom should I contact directly? I am trying to see whether your
> project still uses the Apache CMS and, if so, what your plans are to
> migrate off it.
>
> Thanks in advance for your help.
>
> Andrew
>
> On Wed, Aug 5, 2020 at 9:17 AM Andrew Wetmore  wrote:
>
> > Hi:
> >
> > I am part of the Infrastructure team, and am writing to ask whether your
> > project is still using the Apache CMS for your project website. As you
> > know, the CMS is reaching end-of-life, and we need projects to move their
> > websites onto a different option within the next few weeks.
> >
> > There are several alternatives available, including those listed on this
> > page [1] on managing project websites. Infra is assembling a Wiki page
> [2]
> > on migrating a website from the CMS, and is looking forward to helping
> > projects with this transition.
> >
> > Please let me know whether your site is still on the Apache CMS and, if
> > so, who will be the project point-of-contact with Infra for the
> migration.
> >
> > Thank you!
> >
> >
> >
> >
> > [1] https://infra.apache.org/project-site.html
> >
> > [2]
> >
> https://cwiki.apache.org/confluence/display/INFRA/Migrate+your+project+website+from+the+Apache+CMS
> >
> >
> >
> > --
> > Andrew Wetmore
> > Technical Writer-Editor
> > Infra
> > *Apache Software Foundation*
> > andr...@apache.org
> >
> >
> > <
> https://www.avast.com/sig-email?utm_medium=email_source=link_campaign=sig-email_content=webmail>
> Virus-free.
> > www.avast.com
> > <
> https://www.avast.com/sig-email?utm_medium=email_source=link_campaign=sig-email_content=webmail
> >
> > <#m_2078696594329065464_DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>
> >
>
>
> --
> Andrew Wetmore
> Technical Writer-Editor
> Infra
> *Apache Software Foundation*
> andr...@apache.org
>

Re: [VOTE] Should we release Hive Storage API 2.7.2-rc1?

[Owen O'Malley]

Thanks, Jesus, Gopal, and Panos for driving this.

.. Owen

On Mon, May 11, 2020 at 7:41 AM Jesus Camacho Rodriguez 
wrote:

> With three +1s the release vote passes.
>
> Thanks Gopal and Ashutosh for voting.
>
> -Jesús
>
> On Sat, May 9, 2020 at 6:23 PM Ashutosh Chauhan 
> wrote:
>
> > +1
> > Thanks Gopal for the script.
> >
> > Ashutosh
> >
> >
> >
> > On Fri, May 8, 2020 at 5:40 PM Gopal V  wrote:
> >
> > > Hi,
> > >
> > > Validated checksums, signatures, built and verified against latest orc.
> > >
> > > Since I was lazy enough to automate this, here's a script for others
> who
> > > might not have voted (or want to add things to this).
> > >
> > > https://github.com/t3rmin4t0r/verify-asf-releases + make -f
> > > Makefile.storage-api
> > >
> > > should do all the checks I did, unattended.
> > >
> > > Cheers,
> > > Gopal
> > >
> > > On 5/5/20 7:49 AM, Jesus Camacho Rodriguez wrote:
> > > > All,
> > > >
> > > > I'd like to make a storage-api release with HIVE-22959
> > > >  and HIVE-23215 <
> > > > https://issues.apache.org/jira/browse/HIVE-23215> in it.
> > > >
> > > > Should we release the following artifacts as Hive Storage API 2.7.2?
> > > >
> > > > tar: http://home.apache.org/~jcamacho/hive-storage-2.7.2/
> > > > tag:
> > > https://github.com/apache/hive/releases/tag/storage-release-2.7.2-rc1
> > > > jiras:
> https://issues.apache.org/jira/projects/HIVE/versions/12347828
> > > >
> > > > Thanks!
> > > >
> > > > -Jesús
> > > >
> > >
> >
>

Re: [VOTE] Apache Hive 2.3.7 Release Candidate 0

[Owen O'Malley]

I'm +1 for the release.

   - I checked the signature & sha.
   - I built the project.
   - I ran a handful of unit tests.

.. Owen

On Tue, Apr 7, 2020 at 8:34 PM Hyukjin Kwon  wrote:

> Thank you so much Alan for doing this.
>
> 2020년 4월 8일 (수) 오전 9:26, Alan Gates 님이 작성:
>
> > Apache Hive 2.3.7 Release Candidate 0 is available here:
> > https://people.apache.org/~gates/apache-hive-2.3.7-rc0/
> >
> > Maven artifacts are available here:
> > https://repository.apache.org/content/repositories/orgapachehive-1100/
> >
> > The tag release-2.3.7-rc0 has been applied to the source for this
> > release in github, you can see it
> > athttps://github.com/apache/hive/tree/release-2.3.7-rc0
> >
> > Voting will conclude in 72 hours (or whenever I scrounge together enough
> > votes).
> >
> > Hive PMC Members: Please test and vote.
> >
> > Thanks.
> >
> >
> > Alan.
> >
>

[jira] [Created] (HIVE-23215) Make FilterContext and MutableFilterContext interfaces

[Owen O'Malley (Jira)]

Owen O'Malley created HIVE-23215:


 Summary: Make FilterContext and MutableFilterContext interfaces
 Key: HIVE-23215
 URL: https://issues.apache.org/jira/browse/HIVE-23215
 Project: Hive
  Issue Type: Bug
  Components: storage-api
Reporter: Owen O'Malley
Assignee: Owen O'Malley


HIVE-22959 introduced FilterContext to support ORC-577. The duplication of 
fields between the FilterContext and VectorizedRowBatch seems likely to cause 
user confusion. This patch makes them interfaces that VectorizedRowBatch 
implements.

Thus, there is a single copy of the data and no need to copy them back and 
forth. LLAP can make its own implementation of the interfaces if it doesn't 
want to use VectorizedRowBatch.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [VOTE] Should we release Hive Storage API 2.7.2-rc0?

[Owen O'Malley]

Sorry Jesus for not getting back to you sooner. I started diving into
ORC-577, but didn't finish.

The release -1 votes aren't vetos. You just need more +1's than -1's.
https://cwiki.apache.org/confluence/display/Hive/Bylaws#Bylaws-Voting

That said, I'd ask for a different approach. Maybe we could pull the common
fields out to a new class and make VectorizedRowBatch and the LLAP class
inherit from it?

This patch adds a bunch of stuff into storage-api that feels like it should
be in LLAP instead of in storage-api.

.. Owen

On Tue, Apr 7, 2020 at 1:05 AM Jesus Camacho Rodriguez 
wrote:

> @Owen, will you be able to provide feedback on HIVE-22959 / ORC-577
> shortly?
> The -1 constitutes a veto to the release, but it has been a week since we
> received that vote and we are still awaiting feedback.
> Currently the storage-api release and rest of Panos contribution are on
> hold because of this.
>
> -Jesús
>
>
> On Mon, Apr 6, 2020 at 10:31 PM Ashutosh Chauhan 
> wrote:
>
> > +1
> > built and ran few tests locally.
> >
> > On Mon, Mar 30, 2020 at 2:21 PM Owen O'Malley 
> > wrote:
> >
> > > In evaluating this RC, I discovered HIVE-22959, which is the only patch
> > in
> > > this RC.
> > >
> > > I'm uncomfortable with the API added by HIVE-22959, because it is
> > > duplicating a lot of the functionality from VectorizedRowBatch. I'll
> look
> > > at the motivating ORC-577 tomorrow, but for now I'm -1 on releasing it.
> > >
> > > .. Owen
> > >
> > > On Mon, Mar 30, 2020 at 1:18 PM Vineet G 
> wrote:
> > >
> > > > +1. Verified the signature, checksum and build.
> > > >
> > > > Vineet
> > > >
> > > > > On Mar 30, 2020, at 1:20 AM, Zoltan Haindrich  wrote:
> > > > >
> > > > > +1
> > > > >
> > > > > * verified checksum/etc
> > > > > * built and run tests locally
> > > > > * built orc/master against it
> > > > > * there doesn't seem to be a staged nexus repo for this - but it
> > seems
> > > > like earlier releases also doesn't had that; meanwhile
> > > >
> > >
> >
> https://repo.maven.apache.org/maven2/org/apache/hive/hive-storage-api/2.7.1/
> > > > seems to have them ; I assume it will be also uploaded there along
> with
> > > > sources/etc
> > > > >
> > > > >
> > > > > On 3/24/20 9:33 PM, Jesus Camacho Rodriguez wrote:
> > > > >> All,
> > > > >> I'd like to make a storage-api release with HIVE-22959
> > > > >> <https://issues.apache.org/jira/browse/HIVE-22959> in it.
> > > > >> Should we release the following artifacts as Hive Storage API
> 2.7.2?
> > > > >> tar: http://home.apache.org/~jcamacho/hive-storage-2.7.2/
> > > > >> tag:
> > > >
> https://github.com/apache/hive/releases/tag/storage-release-2.7.2-rc0
> > > > >> jiras:
> > https://issues.apache.org/jira/projects/HIVE/versions/12347828
> > > > >> Thanks!
> > > > >> -Jesús
> > > >
> > > >
> > >
> >
>

Re: [VOTE] Should we release Hive Storage API 2.7.2-rc0?

[Owen O'Malley]

In evaluating this RC, I discovered HIVE-22959, which is the only patch in
this RC.

I'm uncomfortable with the API added by HIVE-22959, because it is
duplicating a lot of the functionality from VectorizedRowBatch. I'll look
at the motivating ORC-577 tomorrow, but for now I'm -1 on releasing it.

.. Owen

On Mon, Mar 30, 2020 at 1:18 PM Vineet G  wrote:

> +1. Verified the signature, checksum and build.
>
> Vineet
>
> > On Mar 30, 2020, at 1:20 AM, Zoltan Haindrich  wrote:
> >
> > +1
> >
> > * verified checksum/etc
> > * built and run tests locally
> > * built orc/master against it
> > * there doesn't seem to be a staged nexus repo for this - but it seems
> like earlier releases also doesn't had that; meanwhile
> https://repo.maven.apache.org/maven2/org/apache/hive/hive-storage-api/2.7.1/
> seems to have them ; I assume it will be also uploaded there along with
> sources/etc
> >
> >
> > On 3/24/20 9:33 PM, Jesus Camacho Rodriguez wrote:
> >> All,
> >> I'd like to make a storage-api release with HIVE-22959
> >>  in it.
> >> Should we release the following artifacts as Hive Storage API 2.7.2?
> >> tar: http://home.apache.org/~jcamacho/hive-storage-2.7.2/
> >> tag:
> https://github.com/apache/hive/releases/tag/storage-release-2.7.2-rc0
> >> jiras: https://issues.apache.org/jira/projects/HIVE/versions/12347828
> >> Thanks!
> >> -Jesús
>
>

[RESULT][VOTE] Should we release Hive Storage API 2.7.1rc0?

[Owen O'Malley]

With three +1’s and no -1’s the release vote passes.

Thanks, Alan & Jesus for voting.

.. Owen

> On Nov 27, 2019, at 12:36 PM, Alan Gates  wrote:
> 
> +1.  Did a build, check the signature and hash, ran rat.  At some point we
> should update the copyright in the NOTICE file, it still says 2017.
> 
> Alan.
> 
> On Tue, Nov 26, 2019 at 11:57 PM Jesus Camacho Rodriguez <
> jcama...@apache.org> wrote:
> 
>> +1
>> 
>> Downloaded tar, built storage-api and ran test suite, ran rat check, and
>> verified checksum and signature.
>> 
>> Thanks Owen!
>> 
>> On Tue, Nov 26, 2019 at 4:39 PM Owen O'Malley 
>> wrote:
>> 
>>> All,
>>> I'd like to make a storage-api release with HIVE-22405
>>> <https://issues.apache.org/jira/browse/HIVE-22405> in it.
>>> 
>>> Should we release the following artifacts as Hive Storage API 2.7.1?
>>> 
>>> tar: http://home.apache.org/~omalley/storage-api-2.7.1/
>>> tag:
>> https://github.com/apache/hive/releases/tag/storage-release-2.7.1rc0
>>> jiras: https://issues.apache.org/jira/projects/HIVE/versions/12346553
>>> 
>>> Thanks!
>>> 
>> 

[VOTE] Should we release Hive Storage API 2.7.1rc0?

[Owen O'Malley]

All,
I'd like to make a storage-api release with HIVE-22405
 in it.

Should we release the following artifacts as Hive Storage API 2.7.1?

tar: http://home.apache.org/~omalley/storage-api-2.7.1/
tag: https://github.com/apache/hive/releases/tag/storage-release-2.7.1rc0
jiras: https://issues.apache.org/jira/projects/HIVE/versions/12346553

Thanks!

Making a storage-api 2.7.1 release

[Owen O'Malley]

All,
   I'd like to make a storage-api 2.7.1 release with HIVE-22405. Does
anyone have any other patches that they'd like to include?

Thanks,
   Owen

[jira] [Created] (HIVE-22405) Add ColumnVector support for ProlepticCalendar

[Owen O'Malley (Jira)]

Owen O'Malley created HIVE-22405:


 Summary: Add ColumnVector support for ProlepticCalendar
 Key: HIVE-22405
 URL: https://issues.apache.org/jira/browse/HIVE-22405
 Project: Hive
  Issue Type: Bug
Reporter: Owen O'Malley


Hive recently moved its processing to the proleptic calendar, which has created 
some issues for users who have dates before 1580 AD.

I'd propose extending the column vectors for times & dates to encode which 
calendar they are using.

* create DateColumnVector that extends LongColumnVector
* add a method to change calendars to both DateColumnVector and 
TimestampColumnVector.

{code}
  /**
   * Change the calendar to or from proleptic. If the new and old values of the 
flag are the
   * same, nothing is done.
   * useProleptic - set the flag for the proleptic calendar
   * updateData - change the data to match the new value of the flag.
   */
  void changeCalendar(useProleptic: boolean, updateData: boolean);

  /**
   * Detect whether this data is using the proleptic calendar.
   */
  boolean usingProlepticCalendar();
{code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [VOTE] Apache Hive 3.1.2 Release Candidate 1

[Owen O'Malley]

+1

* checked source checksum & signature
* built the code
* ran some orc tests
* ran rat

.. Owen

On Mon, Aug 26, 2019 at 4:06 AM Zoltan Haindrich  wrote:

> * deployed the binary in a single node environment (tez-0.9.1;
> hadoop-3.1.1)
> * run some simple queries
> * checked signatures
> * compiled sources without using any cache servers
>
> I haven't encountered any issues
>
> +1
>
>
> On 8/23/19 1:21 AM, Alan Gates wrote:
> > NOTE:  This is a separate release from the recently voted on Hive 2.3.6.
> > Apache Hive 3.1.2 Release Candidate 1 is available here:
> >
> > https://people.apache.org/~gates/apache-hive-3.1.2-rc-1/
> >
> > Maven artifacts are available here:
> >
> > https://repository.apache.org/content/repositories/orgapachehive-1097/
> >
> > The tag release-3.1.2-rc1 has been applied to the source for this release
> > in github, you can see it at
> > https://github.com/apache/hive/tree/release-3.1.2-rc1
> >
> > Voting will conclude in 72 hours.
> >
> > Hive PMC Members: Please test and vote.
> >
> > Thanks.
> >
>

Re: [VOTE] Apache Hive 2.3.6 Release Candidate 0

[Owen O'Malley]

Sorry, I forgot to include my +1. *grin*

> On Aug 21, 2019, at 9:54 AM, Owen O'Malley  wrote:
> 
> I checked the source artifacts:
> The checksums are correct.
> The build succeeds.
> I ran some of the ORC tests.
> RAT passes.
> .. Owen
> 
> On Wed, Aug 21, 2019 at 9:24 AM Zoltan Haindrich  <mailto:k...@rxd.hu>> wrote:
> +1
> * checked signatures
> * downloaded the binary release; run some queries in a small installation
> 
> I'm undecided about the following: for branch-3 there was an ask to include 
> HIVE-18624; which was already present on branch-2 which is good;
> but I know about another similar issue HIVE-21980; I've just landed it on all 
> relevant branches - including branch-2.3
> I'm not sure if we want to roll another RC to include it - since it needs a 
> specifically constructed large query.
> 
> Thank you Alan for taking up releasing 2.3.6!
> 
> 
> On 8/19/19 6:49 PM, Alan Gates wrote:
> > Apache Hive 2.3.6 Release Candidate 0 is available here:
> > 
> > https://people.apache.org/~gates/apache-hive-2.3.6-rc-0/ 
> > <https://people.apache.org/~gates/apache-hive-2.3.6-rc-0/>
> > 
> > Maven artifacts are available here:
> > https://repository.apache.org/content/repositories/orgapachehive-1096/ 
> > <https://repository.apache.org/content/repositories/orgapachehive-1096/>
> > 
> > The tag release-2.3.6-rc0 has been applied to the source for this release
> > in github, you can see it at
> > https://github.com/apache/hive/tree/release-2.3.6-rc0 
> > <https://github.com/apache/hive/tree/release-2.3.6-rc0>
> > 
> > Voting will stay open for at least 72 hours.
> > 
> > Hive PMC Members: Please test and vote.
> > 
> > Thanks.
> > 

Re: [VOTE] Apache Hive 2.3.6 Release Candidate 0

[Owen O'Malley]

I checked the source artifacts:

   1. The checksums are correct.
   2. The build succeeds.
   3. I ran some of the ORC tests.
   4. RAT passes.

.. Owen

On Wed, Aug 21, 2019 at 9:24 AM Zoltan Haindrich  wrote:

> +1
> * checked signatures
> * downloaded the binary release; run some queries in a small installation
>
> I'm undecided about the following: for branch-3 there was an ask to
> include HIVE-18624; which was already present on branch-2 which is good;
> but I know about another similar issue HIVE-21980; I've just landed it on
> all relevant branches - including branch-2.3
> I'm not sure if we want to roll another RC to include it - since it needs
> a specifically constructed large query.
>
> Thank you Alan for taking up releasing 2.3.6!
>
>
> On 8/19/19 6:49 PM, Alan Gates wrote:
> > Apache Hive 2.3.6 Release Candidate 0 is available here:
> >
> > https://people.apache.org/~gates/apache-hive-2.3.6-rc-0/
> >
> > Maven artifacts are available here:
> > https://repository.apache.org/content/repositories/orgapachehive-1096/
> >
> > The tag release-2.3.6-rc0 has been applied to the source for this release
> > in github, you can see it at
> > https://github.com/apache/hive/tree/release-2.3.6-rc0
> >
> > Voting will stay open for at least 72 hours.
> >
> > Hive PMC Members: Please test and vote.
> >
> > Thanks.
> >
>

[jira] [Created] (HIVE-22105) Update ORC to 1.5.6.

[Owen O'Malley (JIRA)]

Owen O'Malley created HIVE-22105:


 Summary: Update ORC to 1.5.6.
 Key: HIVE-22105
 URL: https://issues.apache.org/jira/browse/HIVE-22105
 Project: Hive
  Issue Type: Bug
Reporter: Owen O'Malley
Assignee: Owen O'Malley


ORC has had some important fixes in the 1.5 branch and they should be picked up 
by Hive.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

Section 742.15 NOTIFICATION - Encryption

[Owen O'Malley]

SUBMISSION TYPE:  Section 742.15

SUBMITTED BY: Owen O'Malley

SUBMITTED FOR:The Apache Software Foundation

POINT OF CONTACT: Secretary, The Apache Software Foundation

FAX:  +1-919-573-9199

MANUFACTURER(S):  The Apache Software Foundation

PRODUCT NAME/MODEL #: Apache Hive

ECCN: 5D002

NOTIFICATION: http://www.apache.org/licenses/exports/

Export control

[Owen O'Malley]

In registering ORC as using encryption, I realised that hive isn't listed, 
although it has used encryption since 1.3. Since I just ran through the 
process, unless someone objects, I'll do it for hive also. 

.. Owen

Re: [VOTE] Apache Hive 2.3.5 Release Candidate 0

[Owen O'Malley]

+1

For the source release:
* check signatures of tarball
* asked Alan to update the KEYS file with his current key
* built code
* ran a set of the unit tests

.. Owen

On Tue, May 7, 2019 at 4:57 PM Alan Gates  wrote:

> Apache Hive 2.3.5 Release Candidate 0 is available
> here:http://people.apache.org/~gates/apache-hive-2.3.5-rc-0/
>
> Maven artifacts are available here:
> https://repository.apache.org/content/repositories/orgapachehive-1094/
>
> Source tag for RC0 is release-2.3.5-rc0
>
> Voting will conclude in 72 hours.
>
> Hive PMC Members: Please test and vote.
>
> Thanks.
>
>
> Alan.
>

[jira] [Created] (HIVE-21585) Upgrade branch-2.3 to ORC 1.3.4

[Owen O'Malley (JIRA)]

Owen O'Malley created HIVE-21585:


 Summary: Upgrade branch-2.3 to ORC 1.3.4
 Key: HIVE-21585
 URL: https://issues.apache.org/jira/browse/HIVE-21585
 Project: Hive
  Issue Type: Bug
Reporter: Owen O'Malley
Assignee: Owen O'Malley


Hive's branch-2.3 currently uses ORC 1.3.3.

I'd like to upgrade it use the bug fix release [ORC 
1.3.4|https://issues.apache.org/jira/sr/jira.issueviews:searchrequest-printable/temp/SearchRequest.html?jqlQuery=project+%3D+ORC+AND+status+%3D+Closed+AND+fixVersion+%3D+%221.3.4%22=500].
 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Force push of branch-2.3

[Owen O'Malley]

Ok, I've done the force push to branch-2.3. I also added a commit
(432960e9) that updated the versions in the poms from 2.3.4-SNAPSHOT to the
now correct 2.3.5-SNAPSHOT.

Thanks,
   Owen

On Fri, Apr 5, 2019 at 9:38 AM Thejas Nair  wrote:

> +1
>
>
> On Wed, Mar 27, 2019 at 9:45 AM Owen O'Malley 
> wrote:
>
> > All,
> >The branch-2.3 currently does not have either the rel/release-2.3.3 or
> > rel/release-2.3.4 tag on it. There is only one commit after the 2.3.4 tag
> > that back ports HIVE-20126. If no one objects, I’d like to force push
> > branch-2.3 to include the 2.3.4 tag (with HIVE-20126 rebased on top). We
> > can’t fix the branch to include the 2.3.3 tag without disconnecting the
> > 2.3.4 tag.
> >
> > Let me know if anyone has any concerns.
> >
> > Thanks!
> >Owen
>

Force push of branch-2.3

[Owen O'Malley]

All,
   The branch-2.3 currently does not have either the rel/release-2.3.3 or 
rel/release-2.3.4 tag on it. There is only one commit after the 2.3.4 tag that 
back ports HIVE-20126. If no one objects, I’d like to force push branch-2.3 to 
include the 2.3.4 tag (with HIVE-20126 rebased on top). We can’t fix the branch 
to include the 2.3.3 tag without disconnecting the 2.3.4 tag.

Let me know if anyone has any concerns.

Thanks!
   Owen

Re: Avoiding merge commits

[Owen O'Malley]

I'm +1 for avoiding merge commits on master and release branches.

.. Owen


On Mon, Mar 11, 2019 at 7:47 AM Ashutosh Chauhan 
wrote:

> Hi,
> With advent of gitbox/github integration, folks have started using merge
> commits (inadvertently I believe by merging via Github). This causes issues
> downstream where in my branch, I can't cherry-pick and rebase, but rather
> get merge history. So, I propose we ban merge commits.
> Too radical?
>
> Thanks,
> Ashutosh
>

Re: [DISCUSS] Consistent Timestamps across Hadoop

[Owen O'Malley]

Toward that end, I've updated the Hive tutorial wiki
https://cwiki.apache.org/confluence/display/Hive/Tutorial with the current
timestamp semantics.

In the DDL section, we should call out the historical semantics and the
former and current bugs.

.. Owen

Fwd: [DISCUSS] Consistent Timestamps across Hadoop

[Owen O'Malley]

-- Forwarded message -
From: Owen O'Malley 
Date: Thu, Jan 10, 2019 at 3:09 PM
Subject: Re: [DISCUSS] Consistent Timestamps across Hadoop
To: Zoltan Ivanfi 


No, that isn't right.

The discussion for Apache projects needs to happen in the open and not the
private google doc that isn't archived at Apache.

Three is a severe underestimate of the projects that care about
timestamps. The Apache projects that care about parts of that document are:

   - Avro
   - Hive
   - Iceberg
   - Impala
   - ORC
   - Parquet
   - Spark

That said, Hive needs to make its decisions about what the semantics of
Hive should be. Impala, Iceberg, and Spark may make separate choices. Avro,
ORC, and Parquet need their bindings for each engine need to agree with the
semantics for that engine.

My point is that Hive should have a page that describes its current
semantics with respect to timestamps, but those discussions need to happen
on the Hive list and result in documents in the Hive wiki. Hive can't tell
other projects what to do, but by clarifying their semantics it makes
inter-operation better. In my opinion, Spark SQL should move to local date
time semantics for timestamp. But they should want to do that to make
themselves more compatible with the SQL standard. Clearly Hive can't force
them to change their semantics.

.. Owen


On Thu, Jan 10, 2019 at 8:04 AM Zoltan Ivanfi  wrote:

> Hi,
>
> Once we are through the discussion phase and hopefully have reached
> agreement, I support moving the document to a more permanent place.
> I'm unsure about what the best place would be though. Since it needs
> the agreement of three communities, it does not strictly belong to any
> single one of them (although the Hive Metastore is certainly a central
> component in this ecosystem, so we could put it in the Hive
> documentation based on that). I am also uncertain about whether we
> should use a wiki page, because it is too easily editable and after
> reaching an agreement it should not be modified without asking or
> notifying the same communities again.
>
> Is there a documentation space where modifications are subject to
> review? Or can we protect a wiki page to achieve that? I'm open to
> your suggestions.
>
> Thanks,
>
> Zoltan
>
>
> On Wed, Jan 9, 2019 at 9:32 PM Owen O'Malley 
> wrote:
> >
> > From an Apache point of view, we really need to move this document and
> the discussion to the Apache wiki and mailing lists.
> >
> > Did you want to take a first pass at moving it to Hive's wiki?
> >
> > .. Owen
> >
> > On Tue, Dec 11, 2018 at 10:40 AM Zoltan Ivanfi  wrote:
> >>
> >> Hi Owen,
> >>
> >> Thanks, I think your email contains a great summary of the problems
> tackled in the proposal. I would like highlight two particular topics from
> the discussion that we are having in the comments (details can be read in
> the document):
> >>
> >> It seems that we have agreement on the desired semantics of the more
> explicit SQL types. In particular, I was glad to hear that the TIMESTAMP
> WITH LOCAL TIME ZONE type that is already implemented in Hive is supposed
> to have Instant semantics. (In fact, it already does have Instant
> semantics, but it also has additional time zone information that is unused
> at this moment, and I wasn't sure whether that will be utilized, changing
> the semantics, or whether the semantics will remain and the superflous time
> zone data will be removed.)
> >> We are still discussing what is the best course of action to take with
> the plain TIMESTAMP type, which behaved differently in different file
> formats in Hive 2 and was made to behave the same way in a
> compatibility-breaking manner in Hive 3. My take on this type is that it
> has already been used to write huge amounts of data and for this reason we
> should restore its Avro- and Parquet-specific incosistent behaviour
> (possibly controlled by a feature flag), so that legacy data remains
> readable and legacy workarounds remain functional. The new, more explicit
> SQL types will provide a clear migration path away from the messy TIMESTAMP
> type.
> >>
> >> All in all, I feel that we are converging towards a common goal and I
> have high hopes that the more explicit timestamp types will have much
> better interoperability and consistency across different Hadoop SQL engines.
> >>
> >> Thanks,
> >>
> >> Zoltan
> >>
> >>
> >> On Mon, Dec 10, 2018 at 7:54 PM Owen O'Malley 
> wrote:
> >>>
> >>> Thank you for starting this discussion. Clearly the Hive semantics on
> timestamp are very messed up, but has been moving in the right direction of
> becoming more SQL standard

Re: [DISCUSS] Consistent Timestamps across Hadoop

[Owen O'Malley]

>From an Apache point of view, we really need to move this document and the
discussion to the Apache wiki and mailing lists.

Did you want to take a first pass at moving it to Hive's wiki?

.. Owen

On Tue, Dec 11, 2018 at 10:40 AM Zoltan Ivanfi  wrote:

> Hi Owen,
>
> Thanks, I think your email contains a great summary of the problems
> tackled in the proposal. I would like highlight two particular topics from
> the discussion that we are having in the comments (details can be read in
> the document
> <https://docs.google.com/document/d/1gNRww9mZJcHvUDCXklzjFEQGpefsuR_akCDfWsdE35Q/edit>
> ):
>
>- It seems that we have agreement on the desired semantics of the more
>explicit SQL types. In particular, I was glad to hear that the TIMESTAMP
>WITH LOCAL TIME ZONE type that is already implemented in Hive is supposed
>to have Instant semantics. (In fact, it already does have Instant
>semantics, but it also has additional time zone information that is unused
>at this moment, and I wasn't sure whether that will be utilized, changing
>the semantics, or whether the semantics will remain and the superflous time
>zone data will be removed.)
>- We are still discussing what is the best course of action to take
>with the plain TIMESTAMP type, which behaved differently in different file
>formats in Hive 2 and was made to behave the same way in a
>compatibility-breaking manner in Hive 3. My take on this type is that it
>has already been used to write huge amounts of data and for this reason we
>should restore its Avro- and Parquet-specific incosistent behaviour
>(possibly controlled by a feature flag), so that legacy data remains
>readable and legacy workarounds remain functional. The new, more explicit
>SQL types will provide a clear migration path away from the messy TIMESTAMP
>type.
>
> All in all, I feel that we are converging towards a common goal and I have
> high hopes that the more explicit timestamp types will have much better
> interoperability and consistency across different Hadoop SQL engines.
>
> Thanks,
>
> Zoltan
>
>
> On Mon, Dec 10, 2018 at 7:54 PM Owen O'Malley 
> wrote:
>
>> Thank you for starting this discussion. Clearly the Hive semantics on
>> timestamp are very messed up, but has been moving in the right direction of
>> becoming more SQL standard compliant. I'm pulling this discussion back to
>> the list rather than the personal GoogleDoc, which isn't very
>> collaborative.
>>
>> I like your breakdown of the semantics:
>>
>>- Instant - point in time that will appear different depending on the
>>reader time zone
>>- LocalDateTime - consistent hour and minute regardless of the reader
>>time zone.
>>- OffsetDateTime - consistent hour and minute with the offset of the
>>writer time zone
>>
>> The SQL standard has:
>>
>>- Timestamp & Timestamp without time zone = LocalDateTime
>>- Timestamp with time zone = OffsetDateTime
>>
>> Hive 2 had very confused semantics for timestamp:
>>
>>- When storage was ORC, text, or RCFile with a text serde it was
>>LocalDateTime
>>- When storage was Avro, Parquet, or RCFile with a binary serde it
>>was Instant
>>
>> Hive 3.1 has moved toward the SQL standard extended with Oracles'
>> timestamp with local time zone:
>>
>>- Timestamp = LocalDateTime
>>- Timestamp with local time zone = Instant
>>
>> This leaves us with a few problems:
>>
>>- The Hive bindings to Parquet and Avro don't handle timestamps
>>correctly.
>>- ORC doesn't support timestamps with local time zone. I start
>>working on it in ORC-189.
>>- We don't have timestamp with time zone support.
>>
>> .. Owen
>>
>> On Thu, Dec 6, 2018 at 7:55 AM Marta Kuczora
>>  wrote:
>>
>>> Hi Hive Community,
>>>
>>> I would like to share the following document on our "Consistent Timestamp
>>> types in Hadoop" plans for review.
>>>
>>> https://docs.google.com/document/d/1gNRww9mZJcHvUDCXklzjFEQGpefsuR_akCDfWsdE35Q/edit
>>>
>>> With this plan we would like to get an agreement on consistent timestamp
>>> behavior on Hive, Spark and Impala and in order to achieve this, we are
>>> sharing this document with all three communities.
>>>
>>> Please review and comment, any feedback is much appreciated!
>>>
>>> Regards,
>>> Marta
>>>
>>

Re: [DISCUSS] Consistent Timestamps across Hadoop

[Owen O'Malley]

Thank you for starting this discussion. Clearly the Hive semantics on
timestamp are very messed up, but has been moving in the right direction of
becoming more SQL standard compliant. I'm pulling this discussion back to
the list rather than the personal GoogleDoc, which isn't very
collaborative.

I like your breakdown of the semantics:

   - Instant - point in time that will appear different depending on the
   reader time zone
   - LocalDateTime - consistent hour and minute regardless of the reader
   time zone.
   - OffsetDateTime - consistent hour and minute with the offset of the
   writer time zone

The SQL standard has:

   - Timestamp & Timestamp without time zone = LocalDateTime
   - Timestamp with time zone = OffsetDateTime

Hive 2 had very confused semantics for timestamp:

   - When storage was ORC, text, or RCFile with a text serde it was
   LocalDateTime
   - When storage was Avro, Parquet, or RCFile with a binary serde it was
   Instant

Hive 3.1 has moved toward the SQL standard extended with Oracles' timestamp
with local time zone:

   - Timestamp = LocalDateTime
   - Timestamp with local time zone = Instant

This leaves us with a few problems:

   - The Hive bindings to Parquet and Avro don't handle timestamps
   correctly.
   - ORC doesn't support timestamps with local time zone. I start working
   on it in ORC-189.
   - We don't have timestamp with time zone support.

.. Owen

On Thu, Dec 6, 2018 at 7:55 AM Marta Kuczora 
wrote:

> Hi Hive Community,
>
> I would like to share the following document on our "Consistent Timestamp
> types in Hadoop" plans for review.
>
> https://docs.google.com/document/d/1gNRww9mZJcHvUDCXklzjFEQGpefsuR_akCDfWsdE35Q/edit
>
> With this plan we would like to get an agreement on consistent timestamp
> behavior on Hive, Spark and Impala and in order to achieve this, we are
> sharing this document with all three communities.
>
> Please review and comment, any feedback is much appreciated!
>
> Regards,
> Marta
>

Re: [VOTE] Should we release storage-api 2.7.0 rc1?

[Owen O'Malley]

+1

built & ran tests
checked checksums & signature
tested with ORC

On Thu, Jul 12, 2018 at 4:37 PM, Deepak Jaiswal 
wrote:

> Hi,
>
> I have prepared the rc1 off of branch-3.1.
> Artifacts:
> Tag : https://github.com/apache/hive/releases/tag/storage-
> release-2.7.0-rc1
> Tar Ball : http://home.apache.org/~djaiswal/hive-storage-2.7.0/
>
> Regards,
> Deepak
>
> On 7/10/18, 10:16 AM, "Deepak Jaiswal"  wrote:
>
> Thanks Owen for finding this out. I will work on the next RC once this
> blocker is resolved.
>
> Regards,
> Deepak
>
> On 7/10/18, 9:40 AM, "Owen O'Malley"  wrote:
>
> Ok, Jesus and I tracked it down and I've filed
> https://issues.apache.org/jira/browse/HIVE-20135 that is a
> blocker on
> storage-api 2.7.0.
>
> The impact was that orc 1.5 and master failed with the RC. orc 1.4
> and
> older were fine.
>
> .. Owen
>
> On Tue, Jul 10, 2018 at 8:17 AM, Owen O'Malley <
> owen.omal...@gmail.com>
> wrote:
>
> > I wanted to give an update on this. For now, I'm -1 because the
> ORC
> > (branch-1.5) tests fail with this RC. I'll dig into what is
> wrong, but it
> > looks like something in the timezone changes broke backwards
> compatibility.
> >
> > .. Owen
> >
> > On Mon, Jul 9, 2018 at 11:12 AM, Deepak Jaiswal <
> djais...@hortonworks.com>
> > wrote:
> >
> >> Thanks Alan.
> >>
> >> On 7/9/18, 10:17 AM, "Alan Gates"  wrote:
> >>
> >> +1.  Did a build with a clean maven repo, checked the
> signature and
> >> sha
> >> hash, ran RAT.
> >>
> >> Alan.
> >>
> >> On Fri, Jul 6, 2018 at 2:21 PM Deepak Jaiswal <
> >> djais...@hortonworks.com>
> >> wrote:
> >>
> >> > Hi,
> >> >
> >> > I would like to make a new release of the storage-api. It
> contains
> >> changes
> >> > required for Hive 3.1 release.
> >> >
> >> > Artifcats:
> >> > Tag :
> >> > https://github.com/apache/hive/releases/tag/storage-
> release-
> >> 2.7.0-rc0
> >> > Tar Ball : http://home.apache.org/~
> djaiswal/hive-storage-2.7.0/
> >> >
> >> > Regards,
> >> > Deepak
> >> >
> >>
> >>
> >>
> >
>
>
>
>
>

Re: [VOTE] Should we release storage-api 2.7.0 rc0?

[Owen O'Malley]

Ok, Jesus and I tracked it down and I've filed
https://issues.apache.org/jira/browse/HIVE-20135 that is a blocker on
storage-api 2.7.0.

The impact was that orc 1.5 and master failed with the RC. orc 1.4 and
older were fine.

.. Owen

On Tue, Jul 10, 2018 at 8:17 AM, Owen O'Malley 
wrote:

> I wanted to give an update on this. For now, I'm -1 because the ORC
> (branch-1.5) tests fail with this RC. I'll dig into what is wrong, but it
> looks like something in the timezone changes broke backwards compatibility.
>
> .. Owen
>
> On Mon, Jul 9, 2018 at 11:12 AM, Deepak Jaiswal 
> wrote:
>
>> Thanks Alan.
>>
>> On 7/9/18, 10:17 AM, "Alan Gates"  wrote:
>>
>> +1.  Did a build with a clean maven repo, checked the signature and
>> sha
>> hash, ran RAT.
>>
>> Alan.
>>
>> On Fri, Jul 6, 2018 at 2:21 PM Deepak Jaiswal <
>> djais...@hortonworks.com>
>> wrote:
>>
>> > Hi,
>> >
>> > I would like to make a new release of the storage-api. It contains
>> changes
>> > required for Hive 3.1 release.
>> >
>> > Artifcats:
>> > Tag :
>> > https://github.com/apache/hive/releases/tag/storage-release-
>> 2.7.0-rc0
>> > Tar Ball : http://home.apache.org/~djaiswal/hive-storage-2.7.0/
>> >
>> > Regards,
>> > Deepak
>> >
>>
>>
>>
>

[jira] [Created] (HIVE-20135) Fix incompatible change in TimestampColumnVector to default to UTC

[Owen O'Malley (JIRA)]

Owen O'Malley created HIVE-20135:


 Summary: Fix incompatible change in TimestampColumnVector to 
default to UTC
 Key: HIVE-20135
 URL: https://issues.apache.org/jira/browse/HIVE-20135
 Project: Hive
  Issue Type: Improvement
Reporter: Owen O'Malley
Assignee: Jesus Camacho Rodriguez


HIVE-20007 changed the default for TimestampColumnVector to be to use UTC, 
which breaks the API compatibility with storage-api 2.6.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [VOTE] Should we release storage-api 2.7.0 rc0?

[Owen O'Malley]

I wanted to give an update on this. For now, I'm -1 because the ORC
(branch-1.5) tests fail with this RC. I'll dig into what is wrong, but it
looks like something in the timezone changes broke backwards compatibility.

.. Owen

On Mon, Jul 9, 2018 at 11:12 AM, Deepak Jaiswal 
wrote:

> Thanks Alan.
>
> On 7/9/18, 10:17 AM, "Alan Gates"  wrote:
>
> +1.  Did a build with a clean maven repo, checked the signature and sha
> hash, ran RAT.
>
> Alan.
>
> On Fri, Jul 6, 2018 at 2:21 PM Deepak Jaiswal <
> djais...@hortonworks.com>
> wrote:
>
> > Hi,
> >
> > I would like to make a new release of the storage-api. It contains
> changes
> > required for Hive 3.1 release.
> >
> > Artifcats:
> > Tag :
> > https://github.com/apache/hive/releases/tag/storage-
> release-2.7.0-rc0
> > Tar Ball : http://home.apache.org/~djaiswal/hive-storage-2.7.0/
> >
> > Regards,
> > Deepak
> >
>
>
>

Re: Integrating HMS microbenchmarks with standalone metastore

[Owen O'Malley]

I think that having benchmarks is a really good idea. 

You will need to make them a separate submodule of the standalone Metastore 
that isn't built by default, because we can't have a dependency on JMH, which 
is GPL licensed. 

.. Owen

> On Jun 14, 2018, at 02:55, Alexander Kolbasov  wrote:
> 
> I have a collection of Hive Metastore tools that talk to metastore over
> Thrift. The most useful part of these is a collection of metastore
> micro-benchmarks.
> 
> Currently tools live in my private repo (https://github.com/akolb1/hclient)
> and I think that now when we have standalone metastore it makes sense to
> integrate them with the standalone metastore.
> 
> What do people thing about it?
> 
> - Alex

Re: Cleaning up old version in dist

[Owen O'Malley]

The more accurate guidance is to keep the latest release from each branch
that is being actively maintained. At this point, that means 2.3 and 3.0
for Hive. I'd propose that we keep those versions of storage-api that match
those versions of Hive. That means we should keep 2.4 and 2.6 for storage
api.

.. Owen


On Fri, Jun 8, 2018 at 1:27 AM, Thejas Nair  wrote:

> +1
>
> On Thu, Jun 7, 2018 at 11:13 AM, Alan Gates  wrote:
> > Apache asks that we keep at most 2 current versions in dist, to minimize
> > the space we take up on distribution mirrors.  Since we are running
> > multiple lines and a have a couple of separately releasable modules we'll
> > have more than 2 versions there.  But we have old versions of Hive 2
> (2.1,
> > 2.2) and of the storage-api (2.4, 2.5).  I think we should remove these.
> > That will leave us with the most up to date versions of Hive 1, 2, 3, the
> > storage api, and the standalone metastore.  Note that this does not
> affect
> > their availability in maven central or the apache archive.
> >
> > Alan.
>

Re: Publishing Hive RC artifacts

[Owen O'Malley]

You need to be a PMC member to commit to the dist directory, so you'll need
to find someone to do it for you.

.. Owen

On Sun, May 20, 2018 at 8:51 PM, Vineet Garg  wrote:

> No I still haven’t figured it out. Wiki edit was for different reason.
>
> > On May 19, 2018, at 3:47 PM, Lefty Leverenz 
> wrote:
> >
> > Oh, I see you've already updated the wiki.  Presumably that takes care of
> > it, thanks.
> >
> > -- Lefty
> >
> >
> > On Sat, May 19, 2018 at 3:35 PM Lefty Leverenz 
> > wrote:
> >
> >> Vineet, did you get an answer?  Does the wiki need to be revised?
> >>
> >> -- Lefty
> >>
> >>
> >> On Tue, May 15, 2018 at 6:06 PM Vineet Garg 
> wrote:
> >>
> >>> Hello,
> >>>
> >>> I am unable to publish Hive 3.0 RC artifacts to maven. Following wiki
> >>> page (
> >>> https://cwiki.apache.org/confluence/display/Hive/
> HowToRelease#HowToRelease-Voting)
> >>> I am running following command:
> >>>
> >>>
> >>> mvn deploy -DskipTests -Papache-release -Dmaven.javadoc.skip=true
> >>>
> >>> But doing so result in error:
> >>>
> >>> ERROR] Failed to execute goal
> >>> org.apache.maven.plugins:maven-deploy-plugin:2.8.2:deploy
> (default-deploy)
> >>> on project hive: Failed to deploy artifacts/metadata: Cannot access
> >>> ${repourl} with type default using the available connector factories:
> >>> BasicRepositoryConnectorFactory: Cannot access ${repourl} using the
> >>> registered transporter factories: WagonTransporterFactory: Unsupported
> >>> transport protocol -> [Help 1]
> >>> [ERROR]
> >>>
> >>> Anybody know how do I fix this and publish RC artifacts to maven?
> >>>
> >>> Vineet
> >>>
> >>
>
>

Re: [VOTE] Stricter commit guidelines

[Owen O'Malley]

+1

On Tue, May 15, 2018 at 8:55 AM, Peter Vary  wrote:

> +1 - Hoping for something like this for a long while! Thanks for taking
> this up all!
>
> > On May 15, 2018, at 5:44 PM, Jesus Camacho Rodriguez <
> jcama...@apache.org> wrote:
> >
> > Forgot to mention the length of the vote in original message.
> >
> > Let's leave the vote open for a shorter period than usual, for instance
> 48 hours, i.e., till Wednesday 10pm PST. Situation can only get worse than
> it is now if we do not take action for a longer period.
> >
> > As Alan suggested, vote passes if there is a lazy majority (at least 3
> votes, more +1s than -1s).
> >
> > Thanks,
> > Jesús
> >
> >
> > On 5/15/18, 8:37 AM, "Andrew Sherman"  wrote:
> >
> >+1
> >
> >On Tue, May 15, 2018 at 2:34 AM Rui Li  wrote:
> >
> >> +1
> >>
> >> On Tue, May 15, 2018 at 2:24 PM, Prasanth Jayachandran <
> >> pjayachand...@hortonworks.com> wrote:
> >>
> >>> +1
> >>>
> >>>
> >>>
> >>> Thanks
> >>> Prasanth
> >>>
> >>>
> >>>
> >>> On Mon, May 14, 2018 at 10:44 PM -0700, "Jesus Camacho Rodriguez" <
> >>> jcama...@apache.org> wrote:
> >>>
> >>>
> >>> After work has been done to ignore most of the tests that were failing
> >>> consistently/intermittently [1], I wanted to start this vote to gather
> >>> support from the community to be stricter wrt committing patches to
> Hive.
> >>> The committers guide [2] already specifies that a +1 should be obtained
> >>> before committing, but there is another clause that allows committing
> >> under
> >>> the presence of flaky tests (clause 4). Flaky tests are as good as
> having
> >>> no tests, hence I propose to remove clause 4 and enforce the +1 from
> >>> testing infra before committing.
> >>>
> >>>
> >>>
> >>> As I see it, by enforcing that we always get a +1 from the testing
> infra
> >>> before committing, 1) we will have a more stable project, and 2) we
> will
> >>> have another incentive as a community to create a more robust testing
> >>> infra, e.g., replacing flaky tests for similar unit tests that are not
> >>> flaky, trying to decrease running time for tests, etc.
> >>>
> >>>
> >>>
> >>> Please, share your thoughts about this.
> >>>
> >>>
> >>>
> >>> Here is my +1.
> >>>
> >>>
> >>>
> >>> Thanks,
> >>>
> >>> Jes?s
> >>>
> >>>
> >>>
> >>> [1] http://mail-archives.apache.org/mod_mbox/hive-dev/201805.
> >>> mbox/%3C63023673-AEE5-41A9-BA52-5A5DFB2078B6%40apache.org%3E
> >>>
> >>> [2] https://cwiki.apache.org/confluence/display/Hive/
> >>> HowToCommit#HowToCommit-PreCommitruns,andcommittingpatches
> >>>
> >>>
> >>>
> >>>
> >>>
> >>
> >>
> >> --
> >> Best regards!
> >> Rui Li
> >>
> >
> >
> >
>
>

Re: [VOTE] Should we release storage-api 2.6.1 rc0?

[Owen O'Malley]

Deepak, you need to let 72 hours go by.

.. Owen

On Wed, May 9, 2018 at 11:40 PM, Deepak Jaiswal <djais...@hortonworks.com>
wrote:

> With 3 +1s and no -1s the vote passes. Thanks Owen and Jesus.
>
> Regards,
> Deepak
>
> On 5/9/18, 9:49 PM, "Owen O'Malley" <owen.omal...@gmail.com> wrote:
>
> +1
> checked signatures
> compiled and ran tests
>
> On Wed, May 9, 2018 at 8:58 PM, Jesus Camacho Rodriguez <
> jcamachorodrig...@hortonworks.com> wrote:
>
> > +1
> > - compiled from src
> > - ran unit tests
> >
> > -Jesús
> >
> >
> > On 5/9/18, 11:48 AM, "Deepak Jaiswal" <djais...@hortonworks.com>
> wrote:
> >
> > Ping!
> >
> > On 5/8/18, 12:59 PM, "Deepak Jaiswal" <djais...@hortonworks.com>
> > wrote:
> >
> > All,
> >
> > I would like to make a new release of the storage-api. It
> contains
> > key changes related to Murmur3 hash.
> >
> > Artifacts:
> >
> > Tag: https://github.com/apache/hive/releases/tag/storage-
> > release-2.6.1-rc0
> > Tar ball : http://home.apache.org/~
> djaiswal/hive-storage-2.6.1/
> >
> > Thanks,
> > Deepak
> >
> >
> >
> >
> >
>
>
>

Re: [VOTE] Should we release storage-api 2.6.1 rc0?

[Owen O'Malley]

+1
checked signatures
compiled and ran tests

On Wed, May 9, 2018 at 8:58 PM, Jesus Camacho Rodriguez <
jcamachorodrig...@hortonworks.com> wrote:

> +1
> - compiled from src
> - ran unit tests
>
> -Jesús
>
>
> On 5/9/18, 11:48 AM, "Deepak Jaiswal"  wrote:
>
> Ping!
>
> On 5/8/18, 12:59 PM, "Deepak Jaiswal" 
> wrote:
>
> All,
>
> I would like to make a new release of the storage-api. It contains
> key changes related to Murmur3 hash.
>
> Artifacts:
>
> Tag: https://github.com/apache/hive/releases/tag/storage-
> release-2.6.1-rc0
> Tar ball : http://home.apache.org/~djaiswal/hive-storage-2.6.1/
>
> Thanks,
> Deepak
>
>
>
>
>

Re: [DISCUSS] Storage-API 2.6.1 release

[Owen O'Malley]

Implied in this is that we are putting features into a dot release. I would
argue that since storage-api 2.6.0 was just released last week and has only
been adopted by ORC, we can go ahead and sneak features into the bug fix
release. Does anyone have concerns with that plan?

.. Owen

On Mon, May 7, 2018 at 4:49 PM, Owen O'Malley <owen.omal...@gmail.com>
wrote:

> Thanks, Deepak.
>
> The storage-api releases are global, not specific to a single branch.
> Please delete the storage-branch-2.6.1 branch, the changes that you need
> should go on the storage-branch-2.6.
>
> Please don't copy all of the patches from branch-3 into
> storage-branch-2.6. Just cherry-pick the ones that touched storage-api.
>
> % git log --format=oneline rel/storage-release-2.6.0..apache/branch-3
> storage-api/
>
> shows that we 6 potential patches:
>
> 54651c783bef724713739e59b0d79672a60c6a2c HIVE-18910 : Migrate to Murmur
> hash for shuffle and bucketing
> cdd31fab6eb51e319df03bff880274dc66dd1c39 HIVE-18988: Support bootstrap
> replication of ACID tables
> ea18769f026429ea6ebbbd66858920ebf869a9d6 HIVE-19124 : implement a basic
> major compactor for MM tables
> a39b24660127ace7459ea1598b12d8add1f7b783 HIVE-19226: Extend storage-api
> to print timestamp values in UTC
> bbb8e27bc8b7c7592f45c1e5ffc2ca00b6db6820 HIVE-19226: Extend storage-api
> to print timestamp values in UTC
> 624e464a2cc4fe4dd9395edf8b377fd7323a299e HIVE-19126: CachedStore: Use
> memory estimation to limit cache size during prewarm
>
> so those are the ones to consider.
>
> .. Owen
>
> On Mon, May 7, 2018 at 4:19 PM, Deepak Jaiswal <djais...@hortonworks.com>
> wrote:
>
>> All,
>>
>> Branch 3 contains changes to migrate from JAVA hash to Murmur hash
>> (HIVE-18910) which required addition of couple of APIs keeping the backward
>> compatibility. Since 2.6.0 is already released, I propose a new branch-3
>> only release of storage-api.
>> Please let me know your thoughts on this. I am working on the release
>> candidate.
>>
>> Regards,
>> Deepak
>>
>
>

Re: [DISCUSS] Storage-API 2.6.1 release

[Owen O'Malley]

Thanks, Deepak.

The storage-api releases are global, not specific to a single branch.
Please delete the storage-branch-2.6.1 branch, the changes that you need
should go on the storage-branch-2.6.

Please don't copy all of the patches from branch-3 into storage-branch-2.6.
Just cherry-pick the ones that touched storage-api.

% git log --format=oneline rel/storage-release-2.6.0..apache/branch-3
storage-api/

shows that we 6 potential patches:

54651c783bef724713739e59b0d79672a60c6a2c HIVE-18910 : Migrate to Murmur
hash for shuffle and bucketing
cdd31fab6eb51e319df03bff880274dc66dd1c39 HIVE-18988: Support bootstrap
replication of ACID tables
ea18769f026429ea6ebbbd66858920ebf869a9d6 HIVE-19124 : implement a basic
major compactor for MM tables
a39b24660127ace7459ea1598b12d8add1f7b783 HIVE-19226: Extend storage-api to
print timestamp values in UTC
bbb8e27bc8b7c7592f45c1e5ffc2ca00b6db6820 HIVE-19226: Extend storage-api to
print timestamp values in UTC
624e464a2cc4fe4dd9395edf8b377fd7323a299e HIVE-19126: CachedStore: Use
memory estimation to limit cache size during prewarm

so those are the ones to consider.

.. Owen

On Mon, May 7, 2018 at 4:19 PM, Deepak Jaiswal 
wrote:

> All,
>
> Branch 3 contains changes to migrate from JAVA hash to Murmur hash
> (HIVE-18910) which required addition of couple of APIs keeping the backward
> compatibility. Since 2.6.0 is already released, I propose a new branch-3
> only release of storage-api.
> Please let me know your thoughts on this. I am working on the release
> candidate.
>
> Regards,
> Deepak
>

Re: Apache Hive 3.0.0 release preparation

[Owen O'Malley]

*Sigh* It looks like branch-3 requires features in storage-api that were
added after storage-api 2.6.0.

You'll need to make another storage-api release or revert those changes.

.. Owen

On Mon, May 7, 2018 at 11:58 AM, Vihang Karajgaonkar 
wrote:

> How do we handle the cases where a patch needs to go in branch-2? I think
> the reason of the commits are going in branch-3 is because you don't want
> have a intermediate version where a patch is missing. Eg: patch is fixed in
> Hive 2.4.0 and Hive 3.1.0 but not Hive 3.0.0. Is that normal?
>
> On Mon, May 7, 2018 at 11:34 AM, Vineet Garg 
> wrote:
>
> > Hello all,
> >
> > It’s been more than a month since we have cutoff branch-3. There are
> tests
> > which are still failing consistently(https://issues.
> > apache.org/jira/browse/HIVE-19142).  At this point we should work on
> > fixing tests and stabilize the branch. So please refrain from committing
> > anything but test fixes to branch-3.
> > If you have a patch beside test fix which you would like to get into
> > branch-3 please talk to me before committing.
> >
> > Also if you are assigned a JIRA for fixing test in branch-3 please fix it
> > as soon as you can.
> >
> > Thanks,
> > Vineet Garg
> >
> > On Apr 13, 2018, at 12:36 PM, Vihang Karajgaonkar  > > wrote:
> >
> > Hi Vineet,
> >
> > I created a profile on ptest-server so that tests can be run on branch-3.
> > It is the same as branch-2 patches. You will need to include branch-3 in
> > the patch name. Eg. HIVE-1234.01-branch-3.patch
> >
> > -Vihang
> >
> >
> >
> > On Mon, Apr 9, 2018 at 4:35 PM, Vineet Garg   > vg...@hortonworks.com>> wrote:
> >
> > I have created an umbrella jira to investigate and fix test failures for
> > hive 3.0.0. LINK : https://issues.apache.org/jira/browse/HIVE-19142.
> > Please link any other existing jira related to test failure with this
> > umbrella jira.
> >
> > Also, how do we run tests on branch-3? Is there some setup to be done?
> >
> > -Vineet
> >
> > On Apr 9, 2018, at 4:26 AM, Zoltan Haindrich  <
> > mailto:zhaindr...@hortonworks.com><
> > mailto:zhaindr...@hortonworks.com>> wrote:
> >
> > Hello
> >
> > A few weeks earlier I've tried to hunt down this problem...
> > so...to my best knowledge the cause of this seems to be the following:
> >
> > * in some cases the "cleanup" after a failed query may somehow leave some
> > threads behind...
> > * these threads have reference to the "customized" session classloader -
> > this makes the threads more memory hungry
> > * after a while these threads/classloaders eat up the heap...
> >
> > I've opened HIVE-18522 for this thread issue
> >
> > I think this problem is not new ...and it might have been present earlier
> > as well...the only thing what changed is that there were a few more new
> > features which have added new udfs/etc which made the memory cost of a
> > session more heavier..
> > ...and as a sidenote: I'm not convinced that this issue will arise in a
> > proper hs2 setup - as it might be easily connected to the fact that these
> > tests are using the cli driver to execute the tests.
> >
> >
> > cheers,
> > Zoltan
> >
> > On 7 Apr 2018 7:15 p.m., Ashutosh Chauhan  ashut...@apache.org> > ashut...@apache.org>> wrote:
> > We need to investigate and find out root cause of these failures. If its
> > determined that its a corner case and fix is non-trivial then we may
> > release note it under known issues. But ideally we should fix these
> > failures.
> > Cutting a branch should make it easier since branch is expected to
> receive
> > lot less commits as compared to master so it should be faster to
> stabilize
> > branch.
> >
> > On Fri, Apr 6, 2018 at 10:49 AM, Eugene Koifman <
> ekoif...@hortonworks.com<
> > mailto:ekoif...@hortonworks.com><
> > mailto:ekoif...@hortonworks.com>>
> > wrote:
> >
> > Cutting the branch before the tests are stabilized would mean we have to
> > fix them in 2 places.
> >
> > On 4/6/18, 10:05 AM, "Thejas Nair"  thejas.n...@gmail.com> > thejas.n...@gmail.com>> wrote:
> >
> >   That needs to be cleaned up. There are far too many right now, its
> >   just not handful of flaky tests.
> >
> >
> >   On Fri, Apr 6, 2018 at 2:48 AM, Peter Vary  pv...@cloudera.com> > pv...@cloudera.com>> wrote:
> > Hi Team,
> >
> > I am new to the Hive release process and it is not clear to me how
> > the failing tests are handled. Do we plan to fix the failing tests before
> > release? Or it is accepted to cut a new major release with known test
> > issues.
> >
> > Thanks,
> > Peter
> >
> > On Apr 5, 2018, at 8:25 PM, Vineet Garg  vg...@hortonworks.com> > vg...@hortonworks.com>>
> > wrote:
> >
> > Hello,
> >
> > I 

Re: [VOTE] Should we release storage-api 2.6.0 rc0?

[Owen O'Malley]

With 3 +1 votes and no -1's the vote passes. Thanks Jesus and Alan!

.. Owen

On Fri, Apr 27, 2018 at 9:31 AM, Alan Gates <alanfga...@gmail.com> wrote:

> +1 Did a build in a clean mvn repo, ran rat, looked over NOTICE and LICENSE
> files.
>
> On Fri, Apr 27, 2018 at 8:53 AM, Jesus Camacho Rodriguez <
> jcama...@apache.org> wrote:
>
> > +1
> > - compiled from src
> > - ran unit tests
> > - ran rat
> >
> > -Jesús
> >
> >
> >
> > On 4/26/18, 8:30 AM, "Owen O'Malley" <owen.omal...@gmail.com> wrote:
> >
> > All,
> >I'd like to make a new release of the storage-api.
> >
> > Artifacts:
> > tag: https://github.com/apache/hive/releases/tag/storage-
> > release-2.6.0-rc0
> > tar ball: http://home.apache.org/~omalley/storage-2.6.0/
> >
> > Thanks,
> >Owen
> >
> >
> >
> >
>

[VOTE] Should we release storage-api 2.6.0 rc0?

[Owen O'Malley]

All,
   I'd like to make a new release of the storage-api.

Artifacts:
tag: https://github.com/apache/hive/releases/tag/storage-release-2.6.0-rc0
tar ball: http://home.apache.org/~omalley/storage-2.6.0/

Thanks,
   Owen

[DISCUSS] Storage-API 2.6 release

[Owen O'Malley]

All,
  For the 3.0 release we need a new version of Storage-API. I had hoped
that there would only be bug fixes on top of the 2.5 release, but some new
additions have gone in. I therefore propose that we cut a new
storage-branch-2.6 from master and make a release.

  Are there any outstanding patches for storage-api that need to go in for
Hive 3.0? If not, I'll roll an RC this afternoon.

  Thoughts?

.. Owen

hive dist area questions

[Owen O'Malley]

Thejas,

While uploading the new storage api 2.5.0 release into the dist area, I
realized that we had some non-releases in there. Thejas aded "ldap-fix" in
May 2015 and Sushanth added "hive-parent-auth-hook" in Jan 2016. Are they
releases? Are they out of date?

I also went through the dist area and replaced our old md5 to sha256, so
that we meet the new requirements for checksums.

Thanks,
   Owen

[RESULT][VOTE] Should we release Hive Storage API 2.5.0 rc0?

[Owen O'Malley]

With three +1’s and no -1’s the vote passes.

Thanks Alan & Prasanth!

[VOTE] Should we release Hive Storage API 2.5.0 rc0?

[Owen O'Malley]

All,
   I'd like to make a new release of the storage-api with some of the new
features.

Artifacts:
tag: https://github.com/apache/hive/releases/tag/storage-release-2.5.0-rc0
tar ball: http://home.apache.org/~omalley/storage-2.5.0/

Thanks,
   Owen

[jira] [Created] (HIVE-19013) Fix some minor build issues in storage-api

[Owen O'Malley (JIRA)]

Owen O'Malley created HIVE-19013:


 Summary: Fix some minor build issues in storage-api
 Key: HIVE-19013
 URL: https://issues.apache.org/jira/browse/HIVE-19013
 Project: Hive
  Issue Type: Bug
  Components: storage-api
Reporter: Owen O'Malley
Assignee: Owen O'Malley


Currently, the storage-api tests complain that there isn't a log4j2.xml and the 
javadoc fails.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

storage-api 2.5 release

[Owen O'Malley]

All,
   A couple of features like Decimal64 have gone in to storage-api and I'd
like to make a release.

Any thoughts?

.. Owen

Re: Long running time for recently added tests in standalone-metastore

[Owen O'Malley]

+1

On Mon, Jan 29, 2018 at 4:38 PM, Alexander Kolbasov 
wrote:

> Quite reasonable.
> +1
>
> On Mon, Jan 29, 2018 at 4:19 PM, Alan Gates  wrote:
>
> > With all of the added tests in standalone-metastore/…/client directory,
> the
> > runtime of ‘mvn test’ in standalone-metastore went from 6 minutes to 26
> on
> > my humble laptop.  We do not want to get ourselves back where the rest of
> > Hive is; currently Hive developers don’t run the unit tests themselves
> > because the tests take too long.  I believe we should be working to push
> > the unit test runtime down to about 2 minutes, so people are willing to
> run
> > it frequently as part of their development.
> >
> > I don’t mean that the new tests aren't valuable.  But we need a balance
> > between test coverage in the unit tests and usability.  So I propose that
> > we carve off many of the current unit tests (including some not in the
> > client module, like TestSetUGI…, TestRemote...) in a new profile,
> > ‘nightly’, or ‘checkin’, or something.  Then ‘mvn install’ will finish
> > quickly while hopefully covering 90% of the areas we need to cover.  We
> can
> > ask developers to run the extended set before checkin and configure the
> > automated tests to do the same.  This way we still cover everything
> before
> > committing.
> >
> > Seem reasonable?
> >
> > Alan.
> >
>

Re: Proposal to create a branch for the standalone-metastore work

[Owen O'Malley]

Alan,
   That sounds reasonable. Having development branches rebase relatively
often leads to a much cleaner final git history that using multiple merges.
For the first couple rebases of your development branch, you might want to
tell people just before you do it by sending email out to dev@hive.

Thanks,
   Owen

On Mon, Oct 30, 2017 at 3:44 PM, Alan Gates  wrote:

> ​I propose that we create a branch to continue the standalone metastore
> work in.  So far we’ve been doing it in master one patch at a time.
> However, the speed that we can produce patches outstrips the speed at which
> we can get them reviewed, fully tested, and into master.  I would like to
> put those patches in a branch so that others can review and contribute,
> while not inflicting the not yet reviewed patches on master.​
>
> I am not proposing to do all the development in the branch with a big-bang
> commit/merge at the end.  That will painful for everyone.  We will continue
> to move patches into master one at a time.  This means we’ll be rebasing
> the branch from time to time.
>
> Thoughts?  If there aren’t any objections I’ll do this in a few days.
>
> Alan.
>

[jira] [Created] (HIVE-17925) Fix TestHooks so that it avoids ClassNotFound on teardown

[Owen O'Malley (JIRA)]

Owen O'Malley created HIVE-17925:


 Summary: Fix TestHooks so that it avoids ClassNotFound on teardown
 Key: HIVE-17925
 URL: https://issues.apache.org/jira/browse/HIVE-17925
 Project: Hive
  Issue Type: Bug
Reporter: Owen O'Malley
Assignee: Owen O'Malley


TestHooks gets a ClassNotFound exception during teardown, which messes up some 
following tests.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (HIVE-17924) Restore SerDe by reverting HIVE-15167 to unbreak API compatibility

[Owen O'Malley (JIRA)]

Owen O'Malley created HIVE-17924:


 Summary: Restore SerDe by reverting HIVE-15167 to unbreak API 
compatibility
 Key: HIVE-17924
 URL: https://issues.apache.org/jira/browse/HIVE-17924
 Project: Hive
  Issue Type: Bug
Affects Versions: 2.3.0, 2.3.1
Reporter: Owen O'Malley
Assignee: Owen O'Malley


HIVE-15167 broke compatibility badly for very little gain and caused a lot of 
pain for our users. We should revert it and restore the SerDe interface.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: [VOTE] Apache Hive 2.3.1 Release Candidate 0

[Owen O'Malley]

Ok, in terms of storage-api, you are missing:

commit dfcdf864db1351a35a7f8618eb67523feb9664e9
Author: sergey 
Date:   Mon Jun 19 13:11:51 2017 -0700

HIVE-16864 : add validation to stream position search in LLAP IO
(Sergey Shelukhin, reviewed by Prasanth Jayachandran)


It has been committed on branch-2.3, but was not included in
rel/storage-release-2.4.0. You should probably make a release of
storage-api 2.4.1 and update the 2.3.1 release candidate to use it.

Ok, looking at the patch, it would be nice, but isn’t required. HIVE-16864
made changes to storage-api, but they only added safety checks.



On Oct 20, 2017, at 1:10 PM, Jesus Camacho Rodriguez 
wrote:

Can we wait till the vote closes? I am not sure I can overwrite the branch
history.

Thanks,
Jesús




On 10/20/17, 12:18 PM, "Aihua Xu"  wrote:

Thanks Jesus. At mean time, can I commit the change to branch-2.3 or should
I wait for the vote?

Thanks,
Aihua

On Fri, Oct 20, 2017 at 11:56 AM, Jesus Camacho Rodriguez <
jcama...@apache.org> wrote:

Hi Sergio,

Aihua pinged me about the issue this morning. Since vote is already going
on, I would suggest to continue the vote.
If vote passes, HIVE-17831 would be part of 2.3.2 even if it is committed
to the branch right now (2.3.2 could be released in a few weeks if needed).
If the vote does not pass, we can check it in before creating another RC
and then it would be part of 2.3.1.

Thanks,
Jesús



On 10/20/17, 11:44 AM, "Sergio Pena"  wrote:

Great, thanks Jesus for preparing this. Just one request, can we include
HIVE-17831 in the 2.3.1 release? Sentry has been waiting to bump the Hive
version to 2.x for a while, and HIVE-17831 fixes an issue that is required
to make it work.

It's a small fix and is already part of 3.0 and 2.4.

- Sergio

On Thu, Oct 19, 2017 at 10:06 PM, Jesus Camacho Rodriguez <
jcama...@apache.org> wrote:

Apache Hive 2.3.1 Release Candidate 0 is available here:
http://people.apache.org/~jcamacho/hive-2.3.1-rc0/

Maven artifacts are available here:
https://repository.apache.org/content/repositories/orgapachehive-1081/

Source tag for RC0 is at:
https://github.com/apache/hive/releases/tag/release-2.3.1-rc0/

Voting will conclude in 72 hours.

Hive PMC Members: Please test and vote.

Thanks.








-- 
Thanks
Aihua

Re: [VOTE] Apache Hive 2.3.1 Release Candidate 0

[Owen O'Malley]

Ok, in terms of storage-api, you are missing:

commit dfcdf864db1351a35a7f8618eb67523feb9664e9
Author: sergey 
Date:   Mon Jun 19 13:11:51 2017 -0700

HIVE-16864 : add validation to stream position search in LLAP IO (Sergey 
Shelukhin, reviewed by Prasanth Jayachandran)

It has been committed on branch-2.3, but was not included in 
rel/storage-release-2.4.0. You should probably make a release of storage-api 
2.4.1 and update the 2.3.1 release candidate to use it.

Ok, looking at the patch, it would be nice, but isn’t required. HIVE-16864 made 
changes to storage-api, but it only added safety checks.

.. Owen



> On Oct 20, 2017, at 1:10 PM, Jesus Camacho Rodriguez  
> wrote:
> 
> Can we wait till the vote closes? I am not sure I can overwrite the branch
> history.
> 
> Thanks,
> Jesús
> 
> 
> 
> 
> On 10/20/17, 12:18 PM, "Aihua Xu"  wrote:
> 
>> Thanks Jesus. At mean time, can I commit the change to branch-2.3 or should
>> I wait for the vote?
>> 
>> Thanks,
>> Aihua
>> 
>> On Fri, Oct 20, 2017 at 11:56 AM, Jesus Camacho Rodriguez <
>> jcama...@apache.org> wrote:
>> 
>>> Hi Sergio,
>>> 
>>> Aihua pinged me about the issue this morning. Since vote is already going
>>> on, I would suggest to continue the vote.
>>> If vote passes, HIVE-17831 would be part of 2.3.2 even if it is committed
>>> to the branch right now (2.3.2 could be released in a few weeks if needed).
>>> If the vote does not pass, we can check it in before creating another RC
>>> and then it would be part of 2.3.1.
>>> 
>>> Thanks,
>>> Jesús
>>> 
>>> 
>>> 
>>> On 10/20/17, 11:44 AM, "Sergio Pena"  wrote:
>>> 
 Great, thanks Jesus for preparing this. Just one request, can we include
 HIVE-17831 in the 2.3.1 release? Sentry has been waiting to bump the Hive
 version to 2.x for a while, and HIVE-17831 fixes an issue that is required
 to make it work.
 
 It's a small fix and is already part of 3.0 and 2.4.
 
 - Sergio
 
 On Thu, Oct 19, 2017 at 10:06 PM, Jesus Camacho Rodriguez <
 jcama...@apache.org> wrote:
 
> Apache Hive 2.3.1 Release Candidate 0 is available here:
> http://people.apache.org/~jcamacho/hive-2.3.1-rc0/
> 
> Maven artifacts are available here:
> https://repository.apache.org/content/repositories/orgapachehive-1081/
> 
> Source tag for RC0 is at:
> https://github.com/apache/hive/releases/tag/release-2.3.1-rc0/
> 
> Voting will conclude in 72 hours.
> 
> Hive PMC Members: Please test and vote.
> 
> Thanks.
> 
> 
> 
> 
>>> 
>>> 
>> 
>> 
>> -- 
>> Thanks
>> Aihua
> 

[jira] [Created] (HIVE-17173) Add some connivence redirects to the Hive site

[Owen O'Malley (JIRA)]

Owen O'Malley created HIVE-17173:


 Summary: Add some connivence redirects to the Hive site
 Key: HIVE-17173
 URL: https://issues.apache.org/jira/browse/HIVE-17173
 Project: Hive
  Issue Type: Improvement
Reporter: Owen O'Malley
Assignee: Owen O'Malley


I'd propose that we add the following redirects to our site's .htaccess:

* http://hive.apache.org/bugs -> https://issues.apache.org/jira/browse/hive
* http://hive.apache.org/downloads -> 
https://www.apache.org/dyn/closer.cgi/hive/
* http://hive.apache.org/releases -> https://hive.apache.org/docs/downloads.html
* http://hive.apache.org/src -> https://github.com/apache/hive
* http://hive.apache.org/web-src -> 
https://svn.apache.org/repos/asf/hive/cms/trunk

Thoughts?



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (HIVE-17171) Remove old javadoc versions

[Owen O'Malley (JIRA)]

Owen O'Malley created HIVE-17171:


 Summary: Remove old javadoc versions
 Key: HIVE-17171
 URL: https://issues.apache.org/jira/browse/HIVE-17171
 Project: Hive
  Issue Type: Improvement
Reporter: Owen O'Malley


We currently have a lot of old javadoc versions. I'd propose that we keep the 
following versions:

* r1.2.2
* r2.1.1
* r2.2.0

(Note that 2.3.0 was not checked in to the site.) In particular, I'd suggest we 
remove:

* hcat-r0.5.0
* r0.10.0
* r0.11.0
* r0.12.0
* r0.13.1
* r1.0.1
* r1.1.1
* r2.0.1

Any concerns?




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: [VOTE] Shall we release Hive 2.2.0 rc1?

[Owen O'Malley]

With three +1's the vote passes. I'll finish up the release.

.. Owen

On Mon, Jul 24, 2017 at 5:51 PM, Ashutosh Chauhan <hashut...@apache.org>
wrote:

> +1 built from sources. Ran few unit tests and queries. Looks good.
>
> Thanks,
> Ashutosh
>
> On Mon, Jul 24, 2017 at 3:02 PM, Alan Gates <alanfga...@gmail.com> wrote:
>
> > +1, checked the signatures, did a build with a clean repo, ran RAT.
> >
> > Alan.
> >
> > On Fri, Jul 21, 2017 at 5:20 PM, Owen O'Malley <owen.omal...@gmail.com>
> > wrote:
> >
> > > Ok, I rolled a new RC with the Apache header problem fixed.
> > >
> > > Artifacts: https://home.apache.org/~omalley/hive-2.2.0/
> > > Tag: https://github.com/apache/hive/releases/tag/release-2.2.0rc1
> > da840b0
> > >
> > > .. Owen
> > >
> >
>

[VOTE] Shall we release Hive 2.2.0 rc1?

[Owen O'Malley]

Ok, I rolled a new RC with the Apache header problem fixed.

Artifacts: https://home.apache.org/~omalley/hive-2.2.0/
Tag: https://github.com/apache/hive/releases/tag/release-2.2.0rc1 da840b0

.. Owen

Re: [VOTE] Shall we release Hive 2.2.0 rc0?

[Owen O'Malley]

Ok, Alan discovered that there are RAT failures, so I'll roll a new RC.

Thanks,
   Owen

On Thu, Jul 20, 2017 at 1:40 PM, Owen O'Malley <owen.omal...@gmail.com>
wrote:

> All,
>The Hive branch-2.2 is passing its tests and I'd like to release it.
>
> Artifacts: https://home.apache.org/~omalley/hive-2.2.0/
> Tag: https://github.com/apache/hive/releases/tag/release-2.2.0rc0 0068fcc
>
> I'm going through the list of issues on Jira today to make sure they match.
>
> Thanks,
>Owen
>

[jira] [Created] (HIVE-17154) fix rat problems in branch-2.2

[Owen O'Malley (JIRA)]

Owen O'Malley created HIVE-17154:


 Summary: fix rat problems in branch-2.2
 Key: HIVE-17154
 URL: https://issues.apache.org/jira/browse/HIVE-17154
 Project: Hive
  Issue Type: Bug
Reporter: Owen O'Malley
Assignee: Owen O'Malley


Fix rat problems in the branch-2.2.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[VOTE] Shall we release Hive 2.2.0 rc0?

[Owen O'Malley]

All,
   The Hive branch-2.2 is passing its tests and I'd like to release it.

Artifacts: https://home.apache.org/~omalley/hive-2.2.0/
Tag: https://github.com/apache/hive/releases/tag/release-2.2.0rc0 0068fcc

I'm going through the list of issues on Jira today to make sure they match.

Thanks,
   Owen

[jira] [Created] (HIVE-17118) Clean up of HIVE-14309 to move the orc source code to org.apache.hive.orc

[Owen O'Malley (JIRA)]

Owen O'Malley created HIVE-17118:


 Summary: Clean up of HIVE-14309 to move the orc source code to 
org.apache.hive.orc
 Key: HIVE-17118
 URL: https://issues.apache.org/jira/browse/HIVE-17118
 Project: Hive
  Issue Type: Bug
  Components: ORC
Reporter: Owen O'Malley
Assignee: Owen O'Malley
 Fix For: 2.2.0


Just for branch-2.2.

HIVE-14309 shaded the hive-orc jar to use a unique package org.apache.hive.orc 
package. This patch moves the source files over to the right directory and 
removes the shading.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: [DISCUSS] Separating out the metastore as its own TLP

[Owen O'Malley]

On Fri, Jun 30, 2017 at 3:26 PM, Chao Sun  wrote:

> and maybe a different project name?
>

Yes, it certainly needs a new name. I'd like to suggest Riven.

.. Owen

Re: [DISCUSS] Separating out the metastore as its own TLP

[Owen O'Malley]

I'm +1 on separating out the metastore. It recognizes the reality that a
lot of different projects use the Hive Metastore and opening up the
community is a great move.

..Owen

On Fri, Jun 30, 2017 at 1:30 PM, Xuefu Zhang  wrote:

> +1, sounds like a good idea!
>
> On Fri, Jun 30, 2017 at 1:24 PM, Harsha  wrote:
>
> > Thanks for the proposal Alan. I am +1 on separating the Hive Metastore.
> > This is a great opportunity for building a Metastore to not only address
> > schemas for the data at rest but also for the data in motion. We have a
> > SchemaRegistry (http://github.com/hortonworks/registry)  project that
> > allows users to register schemas for data in motion and integrates with
> > Kafka, Kinesis, Evenhubs and other messaging queues. This will provide
> > us with opportunity to integrate our apis with Hive Metastore and
> > provide with one project that is truly a single metastore that can hold
> > all schemas.
> >
> > Thanks,
> > Harsha
> >
> > On Fri, Jun 30, 2017, at 01:18 PM, Sergio Pena wrote:
> > > Great, thanks Alan for putting all this in the email.
> > > +1
> > >
> > > Allowing other components to continue to use the Metastore without the
> > > need
> > > to use Hive dependencies is a big plus for them. I agree with
> everything
> > > you mention on the email.
> > >
> > > - Sergio
> > >
> > > On Fri, Jun 30, 2017 at 1:49 PM, Julian Hyde  wrote:
> > >
> > > > +1
> > > >
> > > > As a Calcite PMC member, I am very pleased to see this change.
> Calcite
> > > > reads metadata from a variety of sources (including JDBC databases,
> > NoSQL
> > > > databases such as Cassandra and Druid, and streaming systems), and if
> > more
> > > > of those sources choose to store their metadata in the metastore it
> > will
> > > > make our lives easier.
> > > >
> > > > Hive’s metastore has established a position as the place to go for
> > > > metadata in the Hadoop ecosystem. Not all metadata is relational, or
> > > > processed by Hive, so there are other parties using the metastore who
> > > > justifiably would like to influence its direction. Opening up the
> > metastore
> > > > will help retain and extend this position.
> > > >
> > > > Julian
> > > >
> > > >
> > > > On 2017-06-30 10:00 (-0700), "Dimitris ts...@apache.org> wrote:
> > > > >
> > > > >
> > > > > On 2017-06-30 07:56 (-0700), Alan Gates  wrote: >
> > > > > > A few of us have been talking and come to the conclussion that it
> > > > would be>
> > > > > > a good thing to split out the Hive metastore into its own Apache
> > > > project.>
> > > > > > Below and in the linked wiki page we explain what we see as the
> > > > advantages>
> > > > > > to this and how we would go about it.>
> > > > > > >
> > > > > > Hive’s metastore has long been used by other projects in the
> > Hadoop>
> > > > > > ecosystem to store and access metadata.  Apache Impala, Apache
> > Spark,>
> > > > > > Apache Drill, Presto, and other systems all use Hive’s metastore.
> > > > Some,>
> > > > > > like Impala and Presto can use it as their own metadata system
> with
> > > > the>
> > > > > > rest of Hive not present.>
> > > > > > >
> > > > > > This sharing is excellent for the ecosystem.  Together with HDFS
> it
> > > > allows>
> > > > > > users to use the tool of their choice while still accessing the
> > same
> > > > shared>
> > > > > > data.  But having this shared metadata inside the Hive project
> > limits
> > > > the>
> > > > > > ability of other projects to contribute to the metastore.  It
> also
> > > > makes it>
> > > > > > harder for new systems that have similar but not identical
> > metadata>
> > > > > > requirements (for example, stream processing systems on top of
> > Apache>
> > > > > > Kafka) to use Hive’s metastore.  This difficulty for other
> systems
> > > > comes>
> > > > > > out in two ways.  One, it is hard for non-Hive community members
> > to>
> > > > > > participate in the project.  Second, it adds operational cost
> since
> > > > users>
> > > > > > are forced to deploy all of the Hive jars just to get the
> > metastore to
> > > > work.>
> > > > > > >
> > > > > > Therefore we propose to split Hive’s metastore out into a
> separate
> > > > Apache>
> > > > > > project.  This new project will continue to support the same
> Thrift
> > > > API as>
> > > > > > the current metastore.  It will continue to focus on being a
> high>
> > > > > > performance, fault tolerant, large scale, operational metastore
> for
> > > > SQL>
> > > > > > engines and other systems that want to store schema information
> > about
> > > > their>
> > > > > > data.>
> > > > > > >
> > > > > > By making it a separate project we will enable other projects to
> > join
> > > > us in>
> > > > > > innovating on the metastore.  It will simplify operations for
> > non-Hive>
> > > > > > users that want to use the metastore as they will no longer need
> to
> > > > install>
> > > > > > Hive just to get the metastore.  And it will 

[jira] [Created] (HIVE-16787) Fix itests in branch-2.2

[Owen O'Malley (JIRA)]

Owen O'Malley created HIVE-16787:


 Summary: Fix itests in branch-2.2
 Key: HIVE-16787
 URL: https://issues.apache.org/jira/browse/HIVE-16787
 Project: Hive
  Issue Type: Bug
Reporter: Owen O'Malley
Assignee: Owen O'Malley
 Fix For: 2.2.0


The itests are broken in branch 2.2 and need to be fixed before release.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Re: [VOTE] Should we release hive-storage-2.3.1rc0?

[Owen O'Malley]

With three +1's the vote passes. Thanks Eugene and Prasanth!

.. Owen

On Fri, May 26, 2017 at 4:36 PM, Prasanth Jayachandran <
pjayachand...@hortonworks.com> wrote:

> +1
> - built from src
> - ran unit tests
> - rat check
> - verified checksum, signature
>
> Thanks
> Prasanth
>
> > On May 26, 2017, at 11:26 AM, Eugene Koifman <ekoif...@hortonworks.com>
> wrote:
> >
> > Downloaded tarball, built and ran UTs
> > +1
> >
> > On 5/26/17, 9:43 AM, "Owen O'Malley" <owen.omal...@gmail.com> wrote:
> >
> >All,
> >   There are a couple of fixes for hive-storage that we need in the
> 2.3
> >branch.
> >
> >Artifacts:
> >tag: https://github.com/apache/hive/releases/tag/storage-
> release-2.3.1rc0
> >tar ball: http://home.apache.org/~omalley/hive-storage-2.3.1/
> >
> >Thanks,
> >   Owen
> >
> >
>
>

[VOTE] Should we release hive-storage-2.3.1rc0?

[Owen O'Malley]

All,
   There are a couple of fixes for hive-storage that we need in the 2.3
branch.

Artifacts:
tag: https://github.com/apache/hive/releases/tag/storage-release-2.3.1rc0
tar ball: http://home.apache.org/~omalley/hive-storage-2.3.1/

Thanks,
   Owen

[jira] [Created] (HIVE-16683) ORC WriterVersion gets ArrayIndexOutOfBoundsException on newer ORC files

[Owen O'Malley (JIRA)]

Owen O'Malley created HIVE-16683:


 Summary: ORC WriterVersion gets ArrayIndexOutOfBoundsException on 
newer ORC files
 Key: HIVE-16683
 URL: https://issues.apache.org/jira/browse/HIVE-16683
 Project: Hive
  Issue Type: Bug
  Components: ORC
Affects Versions: 2.1.1, 2.2.0
Reporter: Owen O'Malley
Assignee: Owen O'Malley


This only impacts branch-2.1 and branch-2.2, because it has been fixed in the 
ORC project's code base via ORC-125.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Re: How to create a patch that contains a binary file

[Owen O'Malley]

Try:

% git format-patch --stdout HEAD^ > HIVE-1234.1.patch

That will generate a git format patch that should preserve the binary file.

.. Owen

On Fri, May 5, 2017 at 2:43 PM, Juan Rodríguez Hortalá <
juan.rodriguez.hort...@gmail.com> wrote:

> Hi,
>
> For HIVE-16539 I created a patch that adds a new ORC file, using `git diff
> --no-prefix` as specified in
> https://cwiki.apache.org/confluence/display/Hive/HowToContribute#
> HowToContribute-CreatingaPatch.
> The corresponding jenkins build
>  failed/238_UTBatch_itests__hive-blobstore_2_tests/logs/hive.log>
> is failing with
>
> 2017-05-05T10:00:30,151 ERROR [4dda13e3-e900-4d86-a654-bca8c14720cd
> main] ql.Driver: FAILED: SemanticException Line 3:23 Invalid path
> ''../../data/files/part.orc'': No files matching path
> file:/home/hiveptest/35.188.114.194-hiveptest-1/apache-
> github-source-source/data/files/part.orc
> org.apache.hadoop.hive.ql.parse.SemanticException: Line 3:23 Invalid
> path ''../../data/files/part.orc'': No files matching path
> file:/home/hiveptest/35.188.114.194-hiveptest-1/apache-
> github-source-source/data/files/part.orc
>
>
> I think this is because the patch is not creating the ORC file
> correctly when it is applied. When I apply the patch locally on an
> updated clone of https://github.com/apache/hive.git in master, the
> patches applies ok but the resulting file data/files/part.orc is
> different from the original file I used to build the patch, and when I
> try to load it into a table in a local hive instance I get "FAILED:
> SemanticException Unable to load data to destination table. Error: The
> file that you are trying to load does not match the file format of the
> destination table". Similarly, `hive --service orcfiledump
> data/files/part.orc` fails with "Exception in thread "main"
> java.lang.IndexOutOfBoundsException".
>
> So it looks like the patch is malformed for the ORC file because it is
> binary. Should I use bsdiff to build the patch instead? What is the
> expected way for building patches involving binary files?
>
>
> Thanks,
>
>
> Juan
>

Re: [VOTE] Should we release hive-storage-2.3.0rc0?

[Owen O'Malley]

With three +1 votes and no -1's the vote passes.

Thanks,
   Owen

On Tue, May 2, 2017 at 2:55 PM, Alan Gates <alanfga...@gmail.com> wrote:

> +1.  Did a build with a fresh maven repo, checked the signatures, ran rat.
>
> Alan.
>
> On Apr 27, 2017, 16:08 -0700, Owen O'Malley <omal...@apache.org>, wrote:
> > All,
> > I'd like to make a new release of the storage-api for the upcoming 2.2
> > release. Since storage-api rolls releases faster than Hive, this will be
> > the storage-api 2.3.0.
> >
> > Artifacts:
> > tag: https://github.com/apache/hive/releases/tag/storage-
> release-2.3.0rc0
> > tar ball: http://home.apache.org/~omalley/hive-storage-2.3.0/
> >
> > Thanks,
> > Owen
>

Re: [VOTE] Should we release hive-storage 2.4.0rc0?

[Owen O'Malley]

-1 I forgot to make sure the license and notice files were there. I'll roll a 
new release candidate. 

.. Owen

> On May 1, 2017, at 09:52, Ashutosh Chauhan <hashut...@apache.org> wrote:
> 
> Downloaded tar ball. Shasum matches. Build from src. All looks good.
> +1
> 
>> On Fri, Apr 28, 2017 at 11:39 AM, Pengcheng Xiong <pxi...@apache.org> wrote:
>> 
>> I downloaded the tarball, extracted, installed it. It works well with Hive
>> 2.3 (candidate). Thus, +1 on this. Thanks a lot for your work!
>> 
>> On Fri, Apr 28, 2017 at 10:31 AM, Owen O'Malley <omal...@apache.org>
>> wrote:
>> 
>>> All,
>>>   Yes, I know this is confusing, but I wanted to help get the 2.4.0
>>> storage-api released so that hive 2.3.0rc1 can depend on it. So, please
>>> check this release.
>>> 
>>> Artifacts:
>>> tag: https://github.com/apache/hive/releases/tag/storage-
>> release-2.4.0rc0
>>> branch: storage-branch-2.4
>>> tar ball: http://home.apache.org/~omalley/hive-storage-2.4.0/
>>> 
>>> Thanks,
>>>   Owen
>>> 
>> 

[VOTE] Should we release hive-storage 2.4.0rc0?

[Owen O'Malley]

All,
   Yes, I know this is confusing, but I wanted to help get the 2.4.0
storage-api released so that hive 2.3.0rc1 can depend on it. So, please
check this release.

Artifacts:
tag: https://github.com/apache/hive/releases/tag/storage-release-2.4.0rc0
branch: storage-branch-2.4
tar ball: http://home.apache.org/~omalley/hive-storage-2.4.0/

Thanks,
   Owen