Re: [VOTE] Mark Hive 2.x EOL

[Naveen Gangam]

+1 Totally

On Tue, May 14, 2024 at 3:30 AM Zoltán Rátkai 
wrote:

> +1 (non-binding)
>
> Regards,
>
> Zoltan Ratkai
>
> On Tue, May 14, 2024 at 5:42 AM Sourabh Badhya
>  wrote:
>
>> +1 (non-binding)
>>
>> Regards,
>> Sourabh Badhya
>>
>> On Mon, May 13, 2024 at 10:31 PM Krisztian Kasa
>>  wrote:
>>
>>> +1 (binding)
>>>
>>> On Mon, May 13, 2024 at 4:55 PM Okumin  wrote:
>>>
 +1 (non-binding)

 I appreciate the community's efforts in maintaining 2.x for so long.

 Thanks,
 Okumin

 On Sat, May 11, 2024 at 1:57 AM Abhishek Gupta 
 wrote:
 >
 > Unsubscribe
 >
 > On Fri, 10 May 2024 at 10:26 PM, Aman Sinha 
 wrote:
 >>
 >> +1 (non-binding)
 >>
 >> On Fri, May 10, 2024 at 7:57 AM Mahesh Raju Somalaraju <
 maheshra...@cloudera.com.invalid> wrote:
 >>>
 >>> +1(non-binding)
 >>>
 >>> Thanks
 >>> Mahesh Raju S
 >>>
 >>> On Fri, 10 May 2024, 06:15 Ayush Saxena, 
 wrote:
 
  Hi All,
  Following the discussion at [1]. Starting the official vote thread
 to
  mark Hive 2.x release line as EOL.
 
  Marking a release lines as EOL means there won't be any further
  release made for that release line
 
  I will start with my +1
 
  -Ayush
 
 
  [1]
 https://lists.apache.org/thread/91wk3oy1qo953md7941ojg2q97ofsl2d

>>>

Re: Re: [ANNOUNCE] New Committer: Simhadri Govindappa

[Naveen Gangam]

Congrats Simhadri. Looking forward to many more contributions in the future.

On Thu, Apr 18, 2024 at 12:25 PM Sai Hemanth Gantasala
 wrote:

> Congratulations Simhadri 🎉 well deserved
>
> On Thu, Apr 18, 2024 at 8:41 AM Pau Tallada  wrote:
>
>> Congratulations
>>
>> Missatge de Alessandro Solimando  del
>> dia dj., 18 d’abr. 2024 a les 17:40:
>>
>>> Great news, Simhadri, very well deserved!
>>>
>>> On Thu, 18 Apr 2024 at 15:07, Simhadri G  wrote:
>>>
 Thanks everyone!
 I really appreciate it, it means a lot to me :)
 The Apache Hive project and its community have truly inspired me . I'm
 grateful for the chance to contribute to such a remarkable project.

 Thanks!
 Simhadri Govindappa

 On Thu, Apr 18, 2024 at 6:18 PM Sankar Hariappan
  wrote:

> Congrats Simhadri!
>
>
>
> -Sankar
>
>
>
> *From:* Butao Zhang 
> *Sent:* Thursday, April 18, 2024 5:39 PM
> *To:* u...@hive.apache.org; dev 
> *Subject:* [EXTERNAL] Re: [ANNOUNCE] New Committer: Simhadri
> Govindappa
>
>
>
> You don't often get email from butaozha...@163.com. Learn why this is
> important 
>
> Congratulations Simhadri !!!
>
>
>
> Thanks.
>
>
> --
>
> *发件人**:* user-return-28075-butaozhang1=163@hive.apache.org <
> user-return-28075-butaozhang1=163@hive.apache.org> 代表 Ayush
> Saxena 
> *发送时间**:* 星期四, 四月 18, 2024 7:50 下午
> *收件人**:* dev ; u...@hive.apache.org <
> u...@hive.apache.org>
> *主题**:* [ANNOUNCE] New Committer: Simhadri Govindappa
>
>
>
> Hi All,
>
> Apache Hive's Project Management Committee (PMC) has invited Simhadri
> Govindappa to become a committer, and we are pleased to announce that he
> has accepted.
>
>
>
> Please join me in congratulating him, Congratulations Simhadri,
> Welcome aboard!!!
>
>
>
> -Ayush Saxena
>
> (On behalf of Apache Hive PMC)
>

>>
>> --
>> --
>> Pau Tallada Crespí
>> Departament de Serveis
>> Port d'Informació Científica (PIC)
>> Tel: +34 93 170 2729
>> --
>>
>>

Re: Hive jdbc connector

[Naveen Gangam]

Not sure if you got a response. But should be safe to run with JRE8.

On Thu, Feb 1, 2024 at 2:45 AM stephen vijay  wrote:

> Hi sir,
>
> Which Java version does hive jdbc connector supports?
>
> Thanks,
> Vijay S.
>

Re: [ANNOUNCE] Apache Hive 4.0.0 Released

[Naveen Gangam]

Thank you for the tremendous amount of work put in by many many folks to
make this release happen, including projects hive is dependent upon like
tez.

Thank you to all the PMC members, committers and contributors for all the
work over the past 5+ years in shaping this release.

THANK YOU!!!

On Sun, Mar 31, 2024 at 8:54 AM Battula, Brahma Reddy 
wrote:

> Thank you for your hard work and dedication in releasing Apache Hive
> version 4.0.0.
>
>
>
> Congratulations to the entire team on this achievement. Keep up the great
> work!
>
>
>
> Does this consider as GA.?
>
>
>
> And Looks we need to update in the following location also.?
>
> https://hive.apache.org/general/downloads/
>
>
>
>
>
> *From: *Denys Kuzmenko 
> *Date: *Saturday, March 30, 2024 at 00:07
> *To: *u...@hive.apache.org , dev@hive.apache.org <
> dev@hive.apache.org>
> *Subject: *[ANNOUNCE] Apache Hive 4.0.0 Released
>
> The Apache Hive team is proud to announce the release of Apache Hive
>
> version 4.0.0.
>
>
>
> The Apache Hive (TM) data warehouse software facilitates querying and
>
> managing large datasets residing in distributed storage. Built on top
>
> of Apache Hadoop (TM), it provides, among others:
>
>
>
> * Tools to enable easy data extract/transform/load (ETL)
>
>
>
> * A mechanism to impose structure on a variety of data formats
>
>
>
> * Access to files stored either directly in Apache HDFS (TM) or in other
>
>   data storage systems such as Apache HBase (TM)
>
>
>
> * Query execution via Apache Hadoop MapReduce, Apache Tez and Apache Spark 
> frameworks. (MapReduce is deprecated, and Spark has been removed so the text 
> needs to be modified depending on the release version)
>
>
>
> For Hive release details and downloads, please visit:
>
> https://hive.apache.org/downloads.html
>
>
>
> Hive 4.0.0 Release Notes are available here:
>
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?version=12343343&styleName=Text&projectId=12310843
>
>
>
> We would like to thank the many contributors who made this release
>
> possible.
>
>
>
> Regards,
>
>
>
> The Apache Hive Team
>
>

Re: Release managers

[Naveen Gangam]

Thank you guys. How do we plan to track the items for these releases? JIRA
boards?

On Tue, Sep 12, 2023 at 1:49 AM Ayush Saxena  wrote:

> I can volunteer for one as well, I plan to create a wiki page around
> the release management stuff most specifically where we can update the
> Release Managers & tentative release dates & planned features,
> blockers for those releases.
>
> Will add a page around basic release validation as well, so that we
> can have more volunteers to validate the RC during the release time
> :-)
>
> -Ayush
>
> On Thu, 23 Mar 2023 at 21:09, Sai Hemanth Gantasala
>  wrote:
> >
> > Hi all,
> >
> > I would like to volunteer for the 4.2.0 release.
> >
> > Thanks,
> > Sai.
> >
> > On Thu, Mar 23, 2023 at 2:47 PM Denys Kuzmenko 
> wrote:
> >
> > > Hi, I can take the following one: 4.1.0
> > >
>

[jira] [Created] (HIVE-27173) Add method for Spark to be able to trigger DML events

[Naveen Gangam (Jira)]

Naveen Gangam created HIVE-27173:


 Summary: Add method for Spark to be able to trigger DML events
 Key: HIVE-27173
 URL: https://issues.apache.org/jira/browse/HIVE-27173
 Project: Hive
  Issue Type: Improvement
Reporter: Naveen Gangam


Spark currently uses Hive.java from Hive as a convenient way to hide from the 
having to deal with HMS Client and the thrift objects. Currently, Hive has 
support for DML events (being able to generate events on DML operations but 
does not expose a public method to do so). It has a private method that takes 
in Hive objects like Table etc. Would be nice if we can have something with 
more primitive datatypes.




--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (HIVE-27063) LDAP+JWT auth forms not supported

[Naveen Gangam (Jira)]

Naveen Gangam created HIVE-27063:


 Summary: LDAP+JWT auth forms not supported
 Key: HIVE-27063
 URL: https://issues.apache.org/jira/browse/HIVE-27063
 Project: Hive
  Issue Type: Bug
  Components: HiveServer2
Affects Versions: 4.0.0
Reporter: Naveen Gangam


In HIVE-25875, support for multiple authentication forms was added for Hive 
Server. In HIVE-25575, support for JWT authentication was added. However, 
setting hive.server2.authentication="JWT,LDAP" will fail with the following 
validation error.


{noformat}
<12>1 2023-02-03T09:32:11.018Z hiveserver2-0 hiveserver2 1 
0393cf91-48f7-49e3-b2b1-b983000d4cd6 [mdc@18060 class="server.HiveServer2" 
level="WARN" thread="main"] Error starting HiveServer2 on attempt 2, will retry 
in 6ms\rorg.apache.hive.service.ServiceException: Failed to Start 
HiveServer2\r at 
org.apache.hive.service.CompositeService.start(CompositeService.java:80)\r at 
org.apache.hive.service.server.HiveServer2.start(HiveServer2.java:692)\r at 
org.apache.hive.service.server.HiveServer2.startHiveServer2(HiveServer2.java:1154)\r
 at 
org.apache.hive.service.server.HiveServer2.access$1400(HiveServer2.java:145)\r 
at 
org.apache.hive.service.server.HiveServer2$StartOptionExecutor.execute(HiveServer2.java:1503)\r
 at org.apache.hive.service.server.HiveServer2.main(HiveServer2.java:1316)\r at 
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native 
Method)\r at 
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)\r
 at 
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\r
 at java.base/java.lang.reflect.Method.invoke(Method.java:566)\r at 
org.apache.hadoop.util.RunJar.run(RunJar.java:318)\r at 
org.apache.hadoop.util.RunJar.main(RunJar.java:232)\rCaused by: 
java.lang.RuntimeException: Failed to init HttpServer\r at 
org.apache.hive.service.cli.thrift.ThriftHttpCLIService.initServer(ThriftHttpCLIService.java:239)\r
 at 
org.apache.hive.service.cli.thrift.ThriftCLIService.start(ThriftCLIService.java:235)\r
 at org.apache.hive.service.CompositeService.start(CompositeService.java:70)\r 
... 11 more\rCaused by: java.lang.Exception: The authentication types have 
conflicts: LDAP,JWT\r at 
org.apache.hive.service.auth.AuthType.verifyTypes(AuthType.java:69)\r at 
org.apache.hive.service.auth.AuthType.(AuthType.java:43)\r at 
org.apache.hive.service.cli.thrift.ThriftHttpServlet.(ThriftHttpServlet.java:124)\r
 at 
org.apache.hive.service.cli.thrift.ThriftHttpCLIService.initServer(ThriftHttpCLIService.java:197)\r
 ... 13 more\r
{noformat}

We never fixed the AuthType.validateTypes() to support this.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[ANNOUNCE] New committer for Apache Hive: Alessandro Solimando

[Naveen Gangam]

The Project Management Committee (PMC) for Apache Hive has invited
Alessandro Solimando (asolimando) to become a committer and is pleased
to announce that he has accepted.

Contributions from Alessandro:
He has authored 30 patches for Hive, 18 for Apache Calcite and has
done many code reviews for other contributors. Vast experience and
knowledge in SQL Compiler and Optimization. His most recent work was
added support for histogram-based column stats in Hive.

https://issues.apache.org/jira/issues/?filter=12352498

Being a committer enables easier contribution to the project since
there is no need to go via the patch submission process. This should
enable better productivity.A PMC member helps manage and guide the
direction of the project.

Congratulations
Hive PMC

[ANNOUNCE] New committer for Apache Hive: Laszlo Vegh

[Naveen Gangam]

The Project Management Committee (PMC) for Apache Hive has invited Laszlo
Vegh (veghlaci05) to become a committer and we are pleased
to announce that he has accepted.

Contributions from Laszlo:

He has authored 25 patches. Significant contributions to stabilization of
ACID compaction. Helped review other patches as well.

https://github.com/apache/hive/pulls?q=is%3Amerged+is%3Apr+author%3Aveghlaci05

Being a committer enables easier contribution to the project since there is
no need to go via the patch submission process. This should enable better
productivity.A PMC member helps manage and guide the direction of the
project.

Congratulations
Hive PMC

[ANNOUNCE] New PMC Member: Krisztian Kasa

[Naveen Gangam]

Hello Hive Community,
Apache Hive PMC is pleased to announce that Krisztian Kasa (username:
krisztiankasa) has accepted the Apache Hive PMC's invitation to become PMC
Member, and is now our newest PMC member. Please join me in congratulating
Krisztian !!!

He has been an active member in the hive community across many aspects of
the project. Many thanks to Krisztian for all the contributions he has made
and looking forward to many more future contributions in the expanded role.

https://github.com/apache/hive/commits?author=kasakrisz

* 162 commits in master
* 124 reviews in master
* Reported 159 JIRAS

Cheers,
Naveen (on behalf of Hive PMC)

[ANNOUNCE] New PMC Member: Laszlo Bodor

[Naveen Gangam]

Hello Hive Community,
Apache Hive PMC is pleased to announce that Laszlo Bodor
(username:abstractdog) has accepted the Apache Hive PMC's invitation to
become PMC Member, and is now our newest PMC member. Please join me in
congratulating Laszlo !!!

He has been an active member in the hive community across many aspects of
the project. Many thanks to Laszlo for all the contributions he has made
and looking forward to many more future contributions in the expanded role.

https://github.com/apache/hive/commits?author=abstractdog

* 96 commits in master [2]
* 66 reviews in master [3]
* Reported 163 JIRAS [6]

Cheers,
Naveen (on behalf of Hive PMC)

[ANNOUNCE] New PMC Member: Stamatis Zampetakis

[Naveen Gangam]

Hello Hive Community,
Apache Hive PMC is pleased to announce that Stamatis Zampetakis has
accepted the Apache Hive PMC's invitation to become PMC Member, and is now
our newest PMC member. Please join me in congratulating Stamatis !!!

He has been an active member in the hive community across many aspects of
the project. Many thanks to Stamatis for all the contributions he has made
and looking forward to many more future contributions in the expanded role.

Cheers,
Naveen (on behalf of Hive PMC)

[ANNOUNCE] New PMC Member: Ayush Saxena

[Naveen Gangam]

Hello Hive Community,
Apache Hive PMC is pleased to announce that Ayush Saxena has accepted the
Apache Hive PMC's invitation to become PMC Member, and is now our newest
PMC member. Many thanks to Ayush for all the contributions he has made and
looking forward to many more future contributions in the expanded role.

Please join me in congratulating Ayush !!!

Cheers,
Naveen (on behalf of Hive PMC)

Re: method code too large

[Naveen Gangam]

HiveParser.java is a generated source from the hive
grammer definitions, HiveParser.g. It is possible that some of the methods
are XL with several case statements. It may be hard to change this source.

Wondering if there is an option in this shading plugin to configure larger
size for method frames?



On Mon, Nov 28, 2022 at 2:56 PM Chris Nauroth  wrote:

> Hello Jim,
>
> Can you please share the full mvn command that you are running? I don't
> recall anyone else reporting this when we were testing the 4.0.0-alpha-2
> release candidate.
>
> Chris Nauroth
>
>
> On Mon, Nov 21, 2022 at 1:08 PM Jim Hopper  wrote:
>
> > Hi,
> >
> > what is the best strategy to fix the following error during hive-exec
> > shading?
> >
> > [INFO] --- maven-shade-plugin:3.1.1:shade (build-exec-bundle) @ hive-exec
> > ---
> > ...
> >
> > [ERROR] Failed to execute goal
> > org.apache.maven.plugins:maven-shade-plugin:3.1.1:shade
> (build-exec-bundle)
> > on project hive-exec: Error creating shaded jar: Problem shading JAR
> >
> >
> /home/jh/.m2/repository/org/apache/hive/hive-parser/4.0.0-alpha-2/hive-parser-4.0.0-alpha-2.jar
> > entry org/apache/hadoop/hive/ql/parse/HiveParser.class:
> > java.lang.RuntimeException: Method code too large! -> [Help 1]
> >
> > thanks!
> >
>

Re: [DISCUSS] Jira Public Signup Disabled

[Naveen Gangam]

Stamatis, Thank you for bringing this up. +1

On Tue, Nov 15, 2022 at 8:30 AM Ayush Saxena  wrote:

> +1
>
> -Ayush
>
> On 15-Nov-2022, at 5:57 PM, Stamatis Zampetakis  wrote:
>
> 
> Hi everyone,
>
> Due to the large amount of spam account creation the ASF INFRA team has
> disabled the JIRA account creation [1].
>
> From the 11th of November, contributors who wish to have a JIRA account
> (to create, assign, watch, etc issues) will need to request an account
> through an ASF PMC.
>
> Other projects, such as Calcite, have already taken the necessary actions
> to streamline the process for new contributors [2].
>
> I would suggest drawing inspiration from Calcite and take similar actions
> in Hive.
>
> If you all agree we can start by creating a dedicated (private) mailing
> lists for such requests:
> jira-reque...@hive.apache.org
>
> and then proceed with a brief documentation of the process in the wiki or
> website.
>
> What do you think?
>
> Best,
> Stamatis
>
> [1] https://blogs.apache.org/infra/entry/jira-public-signup-disabled
> [2] https://lists.apache.org/thread/5odg6wyvwfkryk96ls2w3vxnrkftw50s
>
>

Re: [VOTE] Apache Hive 4.0.0-alpha-2 Release Candidate 1

[Naveen Gangam]

Hi Denys,
Thank you for publishing the release bits.

*SIGNATURE VERIFICATION*

gpg --verify apache-hive-4.0.0-alpha-2-bin.tar.gz.asc
apache-hive-4.0.0-alpha-2-bin.tar.gz

gpg: Signature made Mon Nov  7 13:04:05 2022 EST

gpg:using RSA key 50606DE1BDBD5CF862A595A907C5682DAFC73125

gpg:issuer "dkuzme...@apache.org"

gpg: Good signature from "Denys Kuzmenko (CODE SIGNING KEY) <
dkuzme...@apache.org>" [unknown]

gpg: WARNING: The key's User ID is not certified with a trusted signature!

gpg:  There is no indication that the signature belongs to the
owner.

Primary key fingerprint: 5060 6DE1 BDBD 5CF8 62A5  95A9 07C5 682D AFC7 3125


$ gpg --verify apache-hive-4.0.0-alpha-2-src.tar.gz.asc
apache-hive-4.0.0-alpha-2-src.tar.gz

gpg: Signature made Mon Nov  7 13:04:25 2022 EST

gpg:using RSA key 50606DE1BDBD5CF862A595A907C5682DAFC73125

gpg:issuer "dkuzme...@apache.org"

gpg: Good signature from "Denys Kuzmenko (CODE SIGNING KEY) <
dkuzme...@apache.org>" [unknown]

gpg: WARNING: The key's User ID is not certified with a trusted signature!

gpg:  There is no indication that the signature belongs to the
owner.

Primary key fingerprint: 5060 6DE1 BDBD 5CF8 62A5  95A9 07C5 682D AFC7 3125


shasum -a 256 -c apache-hive-4.0.0-alpha-2-src.tar.gz.sha256

apache-hive-4.0.0-alpha-2-src.tar.gz: OK


$ shasum -a 256 -c apache-hive-4.0.0-alpha-2-bin.tar.gz.sha256

apache-hive-4.0.0-alpha-2-bin.tar.gz: OK



*BUILD VERIFICATION:*


   - From the source attachment, I was able to build using "mvn clean
   install -DskipTests -Pitests"
   - I also build from the git tag created for the release.


*CHECKIN TESTS --> I think these tests are flaky*

[*INFO*] Running org.apache.hadoop.hive.metastore.client.*TestCatalogs*

[*WARNING*] *Tests **run: 18*, Failures: 0, Errors: 0, *Skipped: 2*, Time
elapsed: 4.342 s - in org.apache.hadoop.hive.metastore.client.*TestCatalogs*

[*INFO*] Running org.apache.hadoop.hive.metastore.*TestMarkPartition*

[*INFO*] *Tests run: 1*, Failures: 0, Errors: 0, Skipped: 0, Time elapsed:
14.895 s - in org.apache.hadoop.hive.metastore.*TestMarkPartition*

[*INFO*]

[*INFO*] Results:

[*INFO*]

[*ERROR*] *Errors: *

[*ERROR*] *  TestMysql.install » Runtime Unable to start docker container*

[*ERROR*] *  TestMysql.upgrade » Runtime Unable to start docker container*

[*ERROR*] *  TestOracle.install » Runtime Unable to start docker container*

[*ERROR*] *  TestOracle.upgrade » Runtime Failed to get docker logs*

[*INFO*]

[*ERROR*] *Tests run: 2259, Failures: 0, Errors: 4, Skipped: 5*

[*INFO*]

[*INFO*]
**

[*INFO*] *BUILD FAILURE*

[*INFO*]
**

[*INFO*] Total time:  25:20 min

[*INFO*] Finished at: 2022-11-10T15:40:36-05:00

[*INFO*]
**


*RUNTIME*

Started services from the binaries published (using local hadoop 3.1.0)


   - Installed schema for derby
   - started HS2 + HMS
   - Ran queries from beeline (DDL and DML)
   - Explain queries
   - CTAS queries.


+1 for me.


On Mon, Nov 7, 2022 at 2:00 PM Denys Kuzmenko
 wrote:

> UPD: Voting will conclude in 1 week (Monday 14th).
>
>
> On Mon, Nov 7, 2022 at 7:57 PM Denys Kuzmenko 
> wrote:
>
> > Hi team,
> >
> > Let's give it 1 more chance.
> >
> > Apache Hive 4.0.0-alpha-2 Release Candidate 1 is available here:
> https://people.apache.org/~dkuzmenko/release-4.0.0-alpha-2-rc1/
> >
> >
> > Change log:
> >
> > - Updated copyright notices (year of distribution was taken from
> `project.build.outputTimestamp` defined in apache root pom)
> >
> > - javadoc artifacts included in maven staging repo
> >
> >
> > The checksums are these:
> > - f94c773e7ae3328d780138d9259c584e94d8f2e34715776cdfaaa78bcf3f25e1
> apache-hive-4.0.0-alpha-2-bin.tar.gz
> > - 02dbc19f007cb87e725713c67f69836b3efb8ffeb1562aac1f9c869e4429cc36
> apache-hive-4.0.0-alpha-2-src.tar.gz
> >
> >
> > Maven artifacts are available here:
> https://repository.apache.org/content/repositories/orgapachehive-1118/
> >
> > The tag release-4.0.0-alpha-2-rc1 has been created for this RC, you can
> find it at
> > https://github.com/apache/hive/tree/release-4.0.0-alpha-2-rc1
> >
> > The git commit hash is:
> https://github.com/apache/hive/commit/36f5d91acb0fac00a5d46049bd45b744fe9aaab6
> >
> > Voting will conclude in 1 week (Monday 21th).
> >
> > Hive PMC Members: Please test and vote.
> >
> > Thanks,
> >
> > Denys
> >
> >
>

Re: [EXTERNAL] Re: Proposal : New Release 3.2.0 | Fixing CVE's and Bugs on apache hive branch-3

[Naveen Gangam]

Thank you Aman for volunteering to drive this. +1 for a release off
branch-3. We can fix all the CVEs we have fixed on master.

IMHO, the hadoop upgrade might be too big a task for this release. Last I
checked, there were some pending items from this upgrade even on master.
They may not be hard dependencies but if we are committing to this, might
take a bit longer to finish the release.

I started to build this Jira Board for the releases. The goal was to use
this to track release items (for all releases) via the use of jira
labels/target versions.
https://issues.apache.org/jira/secure/RapidBoard.jspa?rapidView=564
At the top of this board, there are some quick filters for release blockers
(jiras with labels "hive-4.0.0-must"). There are currently only 2 jiras
tagged as blockers for 4.0.

If you could tag the jiras for 3.2 release the same way, and add a quick
filter, that would be great.

Thank you again
Naveen

On Fri, Nov 4, 2022 at 7:01 AM Stamatis Zampetakis 
wrote:

> Hey everyone,
>
> It would be nice to have a new release from branch 3 although it might not
> be that trivial to get out.
>
> It will definitely require a bit of investment from multiple people
> including the PMC and the committers of the project. Note that the last
> vote for alpha2 was unsuccessful due to lack of votes, which shows that
> people are pretty busy.
>
> Personally, I support this effort and would like to see it happen but this
> period I don't have sufficient time to invest to help with reviews and
> commits for 3.X line.
>
> Best,
> Stamatis
>
> On Fri, Nov 4, 2022, 5:28 AM Aman Raj 
> wrote:
>
> > Hi Chris,
> >
> > I plan on going through this diff and making a comprehensive list of all
> > the major bug fixes that went into branch-3 and not in hive-313. This
> will
> > be included in the umbrella JIRA that I am creating.
> >
> > In this email thread I have only mentioned CVEs and upgrades that will go
> > on top of these changes in branch-3.
> > Thanks,
> > Aman.
> >
> > 
> > From: Chris Nauroth 
> > Sent: Friday, November 4, 2022 3:44 AM
> > To: dev@hive.apache.org 
> > Subject: Re: [EXTERNAL] Re: Proposal : New Release 3.2.0 | Fixing CVE's
> > and Bugs on apache hive branch-3
> >
> > I noticed that there is a pretty large delta (256 commits) between
> release
> > 3.1.3 and the current branch-3:
> >
> > > git log --oneline rel/release-3.1.3..upstream-branch-3 | wc
> > 2564208   33558
> >
> > I just wanted to mention that a release from branch-3 would include far
> > more than what we are cataloging on this mail thread.
> >
> > Chris Nauroth
> >
> >
> > On Thu, Nov 3, 2022 at 12:16 PM Pravin Sinha 
> > wrote:
> >
> > > +1,
> > >
> > > Thanks for driving this, Aman. Apart from CVE fixes, do you have a list
> > of
> > > JIRAs to be targeted?
> > >
> > > -Pravin
> > >
> > > On Thu, Nov 3, 2022 at 11:12 PM Chris Nauroth 
> > wrote:
> > >
> > > > Thank you for driving this!
> > > >
> > > > To kick things off, I have filed HIVE-26702 for a backport of
> > HIVE-17315
> > > (a
> > > > total of 5 sub-tasks/patches) to 3.2.0. This adds support for more
> > > flexible
> > > > configuration of the metastore's database connection pooling.
> > Dataproc's
> > > > distribution has been running this in production backported onto
> > release
> > > > 3.1.3, so I can provide the patches.
> > > >
> > > > May I assume that our intent is to keep 3.2.x backward-compatible
> with
> > > > 3.1.x?
> > > >
> > > > Chris Nauroth
> > > >
> > > >
> > > > On Thu, Nov 3, 2022 at 3:53 AM Sankar Hariappan
> > > >  wrote:
> > > >
> > > > > +1, I'm excited to see the scope includes important upgrades and
> CVE
> > > > fixes.
> > > > > We should carefully port the relevant patches from master as code
> has
> > > > been
> > > > > heavily refactored. But, it make perfect sense to give another 3.x
> > > > release
> > > > > from Hive to keep the users delighted.
> > > > > Thanks Aman for the initiative!
> > > > >
> > > > > Thanks,
> > > > > Sankar
> > > > >
> > > > > -Original Message-
> > > > > From: 张铎(Duo Zhang) 
> > > > > Sent: Thursday, November 3, 2022 2:53 PM
> > > > > To: dev@hive.apache.org
> > > > > Subject: [EXTERNAL] Re: Proposal : New Release 3.2.0 | Fixing CVE's
> > and
> > > > > Bugs on apache hive branch-3
> > > > >
> > > > > [You don't often get email from palomino...@gmail.com. Learn why
> > this
> > > is
> > > > > important at https://aka.ms/LearnAboutSenderIdentification ]
> > > > >
> > > > > +1, and please include HIVE-24694...
> > > > >
> > > > > Thanks.
> > > > >
> > > > > Aman Raj  于2022年11月3日周四 17:03写道：
> > > > > >
> > > > > > Hi team,
> > > > > >
> > > > > >
> > > > > > We know that Hive 4.0.0 release is ongoing but considering the
> > number
> > > > of
> > > > > changes going into the release, it will take some iterations to
> come
> > up
> > > > > with the stable version for the same. Meanwhile there are a lot of
> > > issues
> > > > > in Hive 3.1.3 which our customers have reported. 

Re: [Draft] Board report for October for Apache Hive

[Naveen Gangam]

Thanks Ayush for the review and the pointers. I wasn't aware of these
statistics (still finding my way around).

According to the Health report, Hive has a health score of 6.33. Compared
to last quarter activity is down but it is activity in the preceding 4
weeks is higher compared to 4 weeks prior to that.
Community Health Score (Chi): 6.33 (Healthy)
<https://reporter.apache.org/chi.py#hive>

Here is a revised report. (sorry about the font color)
## Description:

The Apache Hive ™ data warehouse software facilitates reading, writing, and
managing large datasets residing in distributed storage (Apache Hadoop)
using SQL.

## Issues:

No issues requiring board attention this time.

## Membership Data:

Apache Hive was founded 2010-09-21 (~12 years ago)

There are currently 104 committers and 52 PMC members in this project.

The Committer-to-PMC ratio is roughly 2:1.

Community changes, past month:

No changes

## Project activity

Release criteria for 4.0.0GA Planning underway. The master branch is now
versioned 4.0.0 from alpha2.

Jira activity:

In the trailing 31 days, 109 jiras
<https://issues.apache.org/jira/issues/?filter=12352360> have been opened,
30 <https://issues.apache.org/jira/issues/?filter=12352363> of which have
been FIXED. A total of 68 jiras
<https://issues.apache.org/jira/issues/?filter=12352361> have been
closed/resolved and a total of 59 jiras
<https://issues.apache.org/jira/issues/?filter=12352362> have been FIXED.

## Community Health:

Community Health Score (Chi): 6.33 (Healthy)
<https://reporter.apache.org/chi.py#hive>

Community activity is relatively healthy based on engagement. But compared
to last quarter, overall activity (jira/github/dev lists) is down this
quarter but the activity in the preceding 4 weeks is higher compared to 4
weeks prior to that.




On Wed, Oct 19, 2022 at 11:11 PM Ayush Saxena  wrote:

> +1, Thanx Naveen for driving this. Looks good!!!
>
> I guess the community health is there only as a heading but nothing below
> that. Better to write a line, 'that everything is good' like previous
> reports [1], or maybe you can drive some pointers from reporter.a.o [2] for
> it.
>
> [1] https://whimsy.apache.org/board/minutes/Hive.html
> [2] https://reporter.apache.org/wizard/statistics?hive
>
> -Ayush
>
> On Thu, 20 Oct 2022 at 08:28, Naveen Gangam 
> wrote:
>
> > Please review and provide any feedback.October 2022
> >
> > ## Description:
> >
> > The Apache Hive ™ data warehouse software facilitates reading, writing,
> and
> > managing large datasets residing in distributed storage (Apache Hadoop)
> > using SQL.
> >
> > ## Issues:
> >
> > No issues requiring board attention this time.
> >
> > ## Membership Data:
> >
> > Apache Hive was founded 2010-09-21 (~12 years ago)
> >
> > There are currently 104 committers and 52 PMC members in this project.
> >
> > The Committer-to-PMC ratio is roughly 3:1.
> >
> > Community changes, past month:
> >
> > No changes
> >
> > ## Project activity
> >
> > Release criteria for 4.0.0GA Planning underway. The master branch is now
> > versioned 4.0.0 from alpha2.
> >
> > Jira activity:
> >
> > In the trailing 31 days, 109 jiras
> > <https://issues.apache.org/jira/issues/?filter=12352360> have been
> opened,
> > 30 <https://issues.apache.org/jira/issues/?filter=12352363> of which
> have
> > been FIXED. A total of 68 jiras
> > <https://issues.apache.org/jira/issues/?filter=12352361> have been
> > closed/resolved and a total of 59 jiras
> > <https://issues.apache.org/jira/issues/?filter=12352362> have been
> FIXED.
> >
> > ## Community Health:
> >
>

[Draft] Board report for October for Apache Hive

[Naveen Gangam]

Please review and provide any feedback.October 2022

## Description:

The Apache Hive ™ data warehouse software facilitates reading, writing, and
managing large datasets residing in distributed storage (Apache Hadoop)
using SQL.

## Issues:

No issues requiring board attention this time.

## Membership Data:

Apache Hive was founded 2010-09-21 (~12 years ago)

There are currently 104 committers and 52 PMC members in this project.

The Committer-to-PMC ratio is roughly 3:1.

Community changes, past month:

No changes

## Project activity

Release criteria for 4.0.0GA Planning underway. The master branch is now
versioned 4.0.0 from alpha2.

Jira activity:

In the trailing 31 days, 109 jiras
 have been opened,
30  of which have
been FIXED. A total of 68 jiras
 have been
closed/resolved and a total of 59 jiras
 have been FIXED.

## Community Health:

Re: Consider using bi-directional links in JIRA

[Naveen Gangam]

+1. I find this very useful to know the dependencies/relationships. Thank
you for bringing this up.

On Fri, Oct 14, 2022 at 5:06 AM Stamatis Zampetakis 
wrote:

> Hi all,
>
> This is a small tip/reminder for everyone using JIRA.
>
> It is very common and convenient to refer to other tickets by adding the
> HIVE-X pattern in summary, description, and comments.
>
> The pattern allows someone to navigate quickly to an older JIRA from the
> current one but not the other way around.
>
> Ideally, along with the mention (HIVE-X) pattern, it helps to add an
> explicit link (relates to, causes, depends upon, etc.) so that the
> relationship between tickets is visible from both ends.
>
> This is extremely useful when we are reporting a regression/breaking
> change from a past commit but in other cases as well.
>
> Best,
> Stamatis
>

[jira] [Created] (HIVE-26568) Upgrade Log4j2 to 2.18.0 due to CVEs

[Naveen Gangam (Jira)]

Naveen Gangam created HIVE-26568:


 Summary: Upgrade Log4j2 to 2.18.0 due to CVEs
 Key: HIVE-26568
 URL: https://issues.apache.org/jira/browse/HIVE-26568
 Project: Hive
  Issue Type: Bug
Affects Versions: 3.1.2
Reporter: weidong
Assignee: Hankó Gergely
 Fix For: 4.0.0, 4.0.0-alpha-1


High security vulnerability in Log4J - CVE-2021-44832 bundled with Hive



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (HIVE-26566) Upgrade H2 database version to 2.1.214

[Naveen Gangam (Jira)]

Naveen Gangam created HIVE-26566:


 Summary: Upgrade H2 database version to 2.1.214
 Key: HIVE-26566
 URL: https://issues.apache.org/jira/browse/HIVE-26566
 Project: Hive
  Issue Type: Task
  Components: Testing Infrastructure
Reporter: Stamatis Zampetakis
Assignee: Stamatis Zampetakis
 Fix For: 4.0.0, 4.0.0-alpha-1


The 1.3.166 version, which is in use in Hive, suffers from the following 
security vulnerabilities:
https://nvd.nist.gov/vuln/detail/CVE-2021-42392
https://nvd.nist.gov/vuln/detail/CVE-2022-23221

In the project, we use H2 only for testing purposes (inside the jdbc-handler 
module) thus the H2 binaries are not present in the runtime classpath thus 
these CVEs do not pose a problem for Hive or its users. Nevertheless, it would 
be good to upgrade to a more recent version to avoid Hive coming up in 
vulnerability scans due to this.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (HIVE-26502) Improve LDAP auth to support include generic user filters

[Naveen Gangam (Jira)]

Naveen Gangam created HIVE-26502:


 Summary: Improve LDAP auth to support include generic user filters
 Key: HIVE-26502
 URL: https://issues.apache.org/jira/browse/HIVE-26502
 Project: Hive
  Issue Type: Improvement
  Components: HiveServer2
Affects Versions: 4.0.0-alpha-1
Reporter: Naveen Gangam
Assignee: Naveen Gangam


Currently, Hive's ldap userfiltering is based on configuring a set of patterns 
in which wild cards are replaced by usernames and searched for. While this 
model supports advanced filtering options where a corporate ldap can have users 
in different orgs and trees, it does not quite support generic ldap searches 
like this.
(&(uid={0})(objectClass=person))

To be able to support this without making changes to the semantics of existing 
configuration params, and to be backward compatible, we can enhance the 
existing custom query functionality to support this.

For with a configuration like this, we should be able to perform a search for 
user who uid matches the username being authenticated.
  
hive.server2.authentication.ldap.baseDN
dc=apache,dc=org
  
  
hive.server2.authentication.ldap.customLDAPQuery
(&(uid={0})(objectClass=person))
  




--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[Security] CVE-2021-34538: Security vulnerability in Hive with UDFs

[Naveen Gangam]

*CVE-2021-34538*: Unauthorized access to Hive UDFs

*Severity*: Very Important

*Vendor*: The Apache Software Foundation

*Versions Affected*: This vulnerability affects all versions of Hive prior
to Hive 3.1.3. (3.1.2 or earlier).

*Version Fixed:* Hive 3.1.3, Hive 4.0 (in Beta)

*Description*: Hive's "CREATE" and "DROP" function operations does not
check for necessary authorization of involved entities in the query. It was
found that an unauthorized user can manipulate an existing UDF without
having the privileges to do so. This allowed unauthorized or
underprivileged users to drop and recreate UDFs pointing them to new jars
that could be potentially malicious.

Mitigation: This vulnerability has been addressed vio HIVE-25468 in Hive
3.1.3 release and will be included in Hive 4.0GA (Already in the beta
releases). Please upgrade to one of the releases containing the fix or
apply this patch to an existing release.

Credit: This vulnerability was discovered and reported by Hideyuki Furue.
THANK YOU !!!

[jira] [Created] (HIVE-26321) Upgrade commons-io to 2.11.0

[Naveen Gangam (Jira)]

Naveen Gangam created HIVE-26321:


 Summary: Upgrade commons-io to 2.11.0
 Key: HIVE-26321
 URL: https://issues.apache.org/jira/browse/HIVE-26321
 Project: Hive
  Issue Type: Improvement
Affects Versions: 4.0.0
Reporter: Naveen Gangam
Assignee: Naveen Gangam


Upgrade commons-io to 2.11.0



--
This message was sent by Atlassian Jira
(v8.20.7#820007)

Re: 3.1.4 release?

[Naveen Gangam]

Hi Matt,
There are some other pending CVEs to be addressed (like thrift upgrade) by
upgrading the binaries we build on. So the plan is to do more releases.
Please commit the change to the branch-3 as you see fit.

Thank you
Naveen

On Fri, May 6, 2022 at 11:33 AM Matt Burgess  wrote:

> Hi all,
>
> Are there plans for a 3.1.4 release? I noticed HIVE-22393 [1] didn't
> get into Hive 3.1.3, but that bug makes Hive Streaming unusable, at
> least in Apache NiFi. Is there a workaround, some property or
> something I can set to make the stats work (or at least not throw an
> NPE)?
>
> Thanks,
> Matt
>
> [1] https://issues.apache.org/jira/browse/HIVE-22393
>

Re: [VOTE] Apache Hive 3.1.3 Release Candidate 3

[Naveen Gangam]

Apologies for the missed commits from the release notes. Let me see if I
can re-visit this step.

Thank you


On Mon, May 16, 2022 at 6:37 AM Stamatis Zampetakis 
wrote:

> Hi all,
>
> In case you missed it the release notes for Hive 3.1.3 are broken [1].
>
> To avoid similar problems in the future please remember to:
> * associate commits with JIRA tickets;
> * fill in the appropriate version in the "Fix version" field when
> committing;
> * mark the JIRA ticket as resolved;
>
> Best,
> Stamatis
>
> [1] https://issues.apache.org/jira/browse/HIVE-26214
>
> On Sat, Apr 9, 2022 at 9:09 AM Naveen Gangam  >
> wrote:
>
> > Thank you all the voting. Appreciate it.
> >
> > I have 4 binding +1 votes and no negative votes. Have just completed the
> > remaining release work. Will announce shortly.
> >
> >
> > Naveen
> >
> > On Fri, Apr 8, 2022 at 3:32 AM Stamatis Zampetakis 
> > wrote:
> >
> > > Ubuntu 20.04.4 LTS, jdk1.8.0_261, Apache Maven 3.6.3
> > >
> > >  * Checked signatures and checksums OK
> > >  * Checked for checkstyle modified LGPL files OK
> > >  * Checked for illegal licenses in release binaries (jars) using [1] OK
> > >  * Checked diff between repo and release sources (diff -qr hive
> > > apache-hive-3.1.3-src) OK
> > >  * Built from git tag (mvn clean install -DskipTests -Pitests
> -Pjavadoc)
> > OK
> > >  * Built from release sources (mvn clean install -DskipTests -Pitests
> > > -Pjavadoc) OK
> > >  * Run smoke tests in hive-dev-box using hadoop 3.1.0 and tez 0.9.1 OK
> > >
> > > - Initialized derby metastore
> > > - Simple CREATE, INSERT, ANALYZE queries
> > > - Simple SPJA queries
> > > - EXPLAIN variations
> > >
> > > +1 (non-binding)
> > >
> > > Best,
> > > Stamatis
> > >
> > > [1] for f in `find . -name "*.jar"`; do echo $f; jar xf $f
> > > META-INF/LICENSE; head -5 META-INF/*; done >> ALL_LICENSES
> > >
> > > On Thu, Apr 7, 2022 at 6:43 PM Chao Sun  wrote:
> > >
> > > > +1 (binding)
> > > >
> > > > - verified the signatures and checksums
> > > > - tried the binary and tested a few queries.
> > > > - built from source
> > > >
> > > > Thanks Naveen!
> > > >
> > > > Best,
> > > > Chao
> > > >
> > > >
> > > > On Thu, Apr 7, 2022 at 1:28 AM Peter Vary  >
> > > > wrote:
> > > > >
> > > > > Downloaded the 3.1.3 artifacts, and checked the signatures. They
> are
> > > OK.
> > > > > Used the binary to run some basic tests, and it seems OK.
> > > > >
> > > > > +1 (binding)
> > > > >
> > > > > > On 2022. Apr 6., at 20:32, Szehon Ho 
> > > wrote:
> > > > > >
> > > > > > +1 (binding)
> > > > > >
> > > > > > Downloaded and ran create , insert, simple query on postgres.
> > > > > > Verified checksums.
> > > > > > Built from source.
> > > > > >
> > > > > > Thanks,
> > > > > > Szehon
> > > > > >
> > > > > > On Mon, Apr 4, 2022 at 7:56 AM Naveen Gangam
> > > > 
> > > > > > wrote:
> > > > > >
> > > > > >> *[No new commits from RC2]. Just cleaned up
> > > > **apache-hive-3.1.3-src.tar.gz*
> > > > > >> *archive*
> > > > > >>
> > > > > >>
> > > > > >> Apache Hive 3.1.3 Release Candidate 3 is available here:
> > > > > >> https://people.apache.org/~ngangam/apache-hive-3.1.3-rc-3
> > > > > >>
> > > > > >> The checksums are these:
> > > > > >>
> > > > > >>
> > > > > >> -
> 0c9b6a6359a7341b6029cc9347435ee7b379f93846f779d710b13f795b54bb16
> > > > > >> apache-hive-3.1.3-bin.tar.gz
> > > > > >>
> > > > > >>
> > > > > >> -
> b5e17f664afbb5ac702f0de0a31363caf58e067b19229df63da01c38430f6fcc
> > > > > >> apache-hive-3.1.3-src.tar.gz
> > > > > >>
> > > > > >>
> > > > > >> Maven artifacts are available here:
> > > > > >>
> > > https://repository.apache.org/content/repositories/orgapachehive-1116
> > > > > >>
> > > > > >>
> > > > > >> The tag release-3.1.3-rc3 has been applied to the source for
> this
> > > > > >> release in github, you can see it at
> > > > > >>
> > > > > >> https://github.com/apache/hive/tree/release-3.1.3-rc2
> > > > > >>
> > > > > >> The git commit hash is: 4df4d75bf1e16fe0af75aad0b4179c34c07fc975
> > > > > >> <
> > > > > >>
> > > >
> > >
> >
> https://github.com/apache/hive/commit/4df4d75bf1e16fe0af75aad0b4179c34c07fc975
> > > > > >>>
> > > > > >> Voting will conclude in 72 hours.
> > > > > >>
> > > > > >> Hive PMC Members: Please test and vote.
> > > > > >>
> > > > > >> Thanks.
> > > > > >>
> > > > >
> > > >
> > >
> >
>

[ANNOUNCE] Apache Hive 3.1.3 Released

[Naveen Gangam]

The Apache Hive team is proud to announce the release of Apache Hive
version 3.1.3. There has been a long gap since the previous release on this

branch.

The Apache Hive (TM) data warehouse software facilitates querying and
managing large datasets residing in distributed storage. Built on top
of Apache Hadoop (TM), it provides, among others:

* Tools to enable easy data extract/transform/load (ETL)

* A mechanism to impose structure on a variety of data formats

* Access to files stored either directly in Apache HDFS (TM) or in other
  data storage systems such as Apache HBase (TM)

* Query execution via Apache Hadoop MapReduce, Apache Tez and Apache
Spark frameworks.

For Hive release details and downloads, please
visit:https://hive.apache.org/downloads.html

Hive 3.1.3 Release Notes are available here:

https://issues.apache.org/jira/secure/ReleaseNote.jspa?version=12346277&styleName=Html&projectId=12310843

We would like to thank the many contributors who made this release
possible.

Regards,

The Apache Hive Team

Re: [VOTE] Apache Hive 3.1.3 Release Candidate 3

[Naveen Gangam]

Thank you all the voting. Appreciate it.

I have 4 binding +1 votes and no negative votes. Have just completed the
remaining release work. Will announce shortly.


Naveen

On Fri, Apr 8, 2022 at 3:32 AM Stamatis Zampetakis 
wrote:

> Ubuntu 20.04.4 LTS, jdk1.8.0_261, Apache Maven 3.6.3
>
>  * Checked signatures and checksums OK
>  * Checked for checkstyle modified LGPL files OK
>  * Checked for illegal licenses in release binaries (jars) using [1] OK
>  * Checked diff between repo and release sources (diff -qr hive
> apache-hive-3.1.3-src) OK
>  * Built from git tag (mvn clean install -DskipTests -Pitests -Pjavadoc) OK
>  * Built from release sources (mvn clean install -DskipTests -Pitests
> -Pjavadoc) OK
>  * Run smoke tests in hive-dev-box using hadoop 3.1.0 and tez 0.9.1 OK
>
> - Initialized derby metastore
> - Simple CREATE, INSERT, ANALYZE queries
> - Simple SPJA queries
> - EXPLAIN variations
>
> +1 (non-binding)
>
> Best,
> Stamatis
>
> [1] for f in `find . -name "*.jar"`; do echo $f; jar xf $f
> META-INF/LICENSE; head -5 META-INF/*; done >> ALL_LICENSES
>
> On Thu, Apr 7, 2022 at 6:43 PM Chao Sun  wrote:
>
> > +1 (binding)
> >
> > - verified the signatures and checksums
> > - tried the binary and tested a few queries.
> > - built from source
> >
> > Thanks Naveen!
> >
> > Best,
> > Chao
> >
> >
> > On Thu, Apr 7, 2022 at 1:28 AM Peter Vary 
> > wrote:
> > >
> > > Downloaded the 3.1.3 artifacts, and checked the signatures. They are
> OK.
> > > Used the binary to run some basic tests, and it seems OK.
> > >
> > > +1 (binding)
> > >
> > > > On 2022. Apr 6., at 20:32, Szehon Ho 
> wrote:
> > > >
> > > > +1 (binding)
> > > >
> > > > Downloaded and ran create , insert, simple query on postgres.
> > > > Verified checksums.
> > > > Built from source.
> > > >
> > > > Thanks,
> > > > Szehon
> > > >
> > > > On Mon, Apr 4, 2022 at 7:56 AM Naveen Gangam
> > 
> > > > wrote:
> > > >
> > > >> *[No new commits from RC2]. Just cleaned up
> > **apache-hive-3.1.3-src.tar.gz*
> > > >> *archive*
> > > >>
> > > >>
> > > >> Apache Hive 3.1.3 Release Candidate 3 is available here:
> > > >> https://people.apache.org/~ngangam/apache-hive-3.1.3-rc-3
> > > >>
> > > >> The checksums are these:
> > > >>
> > > >>
> > > >> - 0c9b6a6359a7341b6029cc9347435ee7b379f93846f779d710b13f795b54bb16
> > > >> apache-hive-3.1.3-bin.tar.gz
> > > >>
> > > >>
> > > >> - b5e17f664afbb5ac702f0de0a31363caf58e067b19229df63da01c38430f6fcc
> > > >> apache-hive-3.1.3-src.tar.gz
> > > >>
> > > >>
> > > >> Maven artifacts are available here:
> > > >>
> https://repository.apache.org/content/repositories/orgapachehive-1116
> > > >>
> > > >>
> > > >> The tag release-3.1.3-rc3 has been applied to the source for this
> > > >> release in github, you can see it at
> > > >>
> > > >> https://github.com/apache/hive/tree/release-3.1.3-rc2
> > > >>
> > > >> The git commit hash is: 4df4d75bf1e16fe0af75aad0b4179c34c07fc975
> > > >> <
> > > >>
> >
> https://github.com/apache/hive/commit/4df4d75bf1e16fe0af75aad0b4179c34c07fc975
> > > >>>
> > > >> Voting will conclude in 72 hours.
> > > >>
> > > >> Hive PMC Members: Please test and vote.
> > > >>
> > > >> Thanks.
> > > >>
> > >
> >
>

[jira] [Created] (HIVE-26118) [Standalone Beeline] Jar name mismatch between build and assembly

[Naveen Gangam (Jira)]

Naveen Gangam created HIVE-26118:


 Summary: [Standalone Beeline] Jar name mismatch between build and 
assembly
 Key: HIVE-26118
 URL: https://issues.apache.org/jira/browse/HIVE-26118
 Project: Hive
  Issue Type: Sub-task
  Components: Beeline
Affects Versions: 3.0.0
Reporter: Naveen Gangam
Assignee: Naveen Gangam


Fix from HIVE-25750 has an issue where the beeline builds a jar named 
"jar-with-dependencies.jar" but the assembly looks for a jar name 
"original-jar-with-dependencies.jar". Thus this uber jar never gets included in 
the distribution.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[VOTE] Apache Hive 3.1.3 Release Candidate 3

[Naveen Gangam]

*[No new commits from RC2]. Just cleaned up **apache-hive-3.1.3-src.tar.gz*
*archive*


Apache Hive 3.1.3 Release Candidate 3 is available here:
https://people.apache.org/~ngangam/apache-hive-3.1.3-rc-3

The checksums are these:


- 0c9b6a6359a7341b6029cc9347435ee7b379f93846f779d710b13f795b54bb16
apache-hive-3.1.3-bin.tar.gz


- b5e17f664afbb5ac702f0de0a31363caf58e067b19229df63da01c38430f6fcc
apache-hive-3.1.3-src.tar.gz


Maven artifacts are available here:
https://repository.apache.org/content/repositories/orgapachehive-1116


The tag release-3.1.3-rc3 has been applied to the source for this
release in github, you can see it at

https://github.com/apache/hive/tree/release-3.1.3-rc2

The git commit hash is: 4df4d75bf1e16fe0af75aad0b4179c34c07fc975

Voting will conclude in 72 hours.

Hive PMC Members: Please test and vote.

Thanks.

Re: [VOTE] Apache Hive 3.1.3 Release Candidate 2

[Naveen Gangam]

Hi Stamatis,
Thanks for catching some issues with local files above.

I have cleaned up most of them
Only in apache-hive-3.1.3-src/hcatalog/core: mapred
Only in apache-hive-3.1.3-src/itests: ${project.basedir}
Only in apache-hive-3.1.3-src/itests/hive-unit: metastore_db2
Only in apache-hive-3.1.3-src/itests/qtest: ${project.basedir}
Only in apache-hive-3.1.3-src/itests: qtest-kudu
Only in apache-hive-3.1.3-src/ql: dependency-reduced-pom.xml
Only in apache-hive-3.1.3-src/standalone-metastore: metastore-common
Only in apache-hive-3.1.3-src/standalone-metastore: metastore-server
Only in apache-hive-3.1.3-src/standalone-metastore: metastore-tools
Only in apache-hive-3.1.3-src/upgrade-acid: pre-upgrade

Things I could not clean up in the final archive are (they keep getting
regenerated after a build)
Only in apache-hive-3.1.3-src/standalone-metastore/src/gen: version
Only in apache-hive-3.1.3-src/common/src: gen
Only in apache-hive-3.1.3-src/conf: hive-default.xml.template
Only in apache-hive-3.1.3-src/ql: dependency-reduced-pom.xml

But these files above also exist in the 3.1.2 release as well. So I am not
introducing anything new.

I will be pushing another RC candidate soon with the above changes.

Thank you
Naveen

On Sat, Apr 2, 2022 at 5:20 PM Sylwester Lachiewicz 
wrote:

> +1 (nb)
>
> pon., 28 mar 2022, 23:19 użytkownik Naveen Gangam
>  napisał:
>
> > Apache Hive 3.1.3 Release Candidate 2 is available here:
> > https://people.apache.org/~ngangam/apache-hive-3.1.3-rc-2
> >
> > The checksums are these:
> >
> >
> > - 55c58e0111bd32de3d02f5f25d9eb054ba65ab02aaf669637760eaf56ef1fbb1
> > apache-hive-3.1.3-bin.tar.gz
> >
> >
> > - 22862e6bf76a4783a3d8d298634728cc9d6561563af2413a687fe63e35bcc527
> > apache-hive-3.1.3-src.tar.gz
> >
> >
> > Maven artifacts are available here:
> > https://repository.apache.org/content/repositories/orgapachehive-1114
> >
> > The tag release-3.1.3-rc2 has been applied to the source for this
> > release in github, you can see it
> > athttps://github.com/apache/hive/tree/release-3.1.3-rc2
> >
> > The git commit hash is: 4df4d75bf1e16fe0af75aad0b4179c34c07fc975
> > <
> >
> https://github.com/apache/hive/commit/4df4d75bf1e16fe0af75aad0b4179c34c07fc975
> > >
> > Voting will conclude in 72 hours.
> >
> > Hive PMC Members: Please test and vote.
> >
> > Thanks.
> >
> > Naveen
> >
>

Re: [VOTE] Apache Hive 3.1.3 Release Candidate 2

[Naveen Gangam]

Still seeking votes. Voting ends tomorrow. Any help would be appreciated.

Thank you
Naveen

On Tue, Mar 29, 2022 at 5:51 AM Peter Vary 
wrote:

> Downloaded the 3.1.3 artifacts, and checked the signatures. They are OK.
> Used the binary to run some basic tests, and it seems OK.
>
> +1 (binding)
>
> > On 2022. Mar 28., at 23:19, Naveen Gangam 
> wrote:
> >
> > Apache Hive 3.1.3 Release Candidate 2 is available here:
> > https://people.apache.org/~ngangam/apache-hive-3.1.3-rc-2
> >
> > The checksums are these:
> >
> >
> > - 55c58e0111bd32de3d02f5f25d9eb054ba65ab02aaf669637760eaf56ef1fbb1
> > apache-hive-3.1.3-bin.tar.gz
> >
> >
> > - 22862e6bf76a4783a3d8d298634728cc9d6561563af2413a687fe63e35bcc527
> > apache-hive-3.1.3-src.tar.gz
> >
> >
> > Maven artifacts are available here:
> > https://repository.apache.org/content/repositories/orgapachehive-1114
> >
> > The tag release-3.1.3-rc2 has been applied to the source for this
> > release in github, you can see it
> > athttps://github.com/apache/hive/tree/release-3.1.3-rc2
> >
> > The git commit hash is: 4df4d75bf1e16fe0af75aad0b4179c34c07fc975
> > <
> https://github.com/apache/hive/commit/4df4d75bf1e16fe0af75aad0b4179c34c07fc975
> >
> > Voting will conclude in 72 hours.
> >
> > Hive PMC Members: Please test and vote.
> >
> > Thanks.
> >
> > Naveen
>
>

Re: [VOTE] Apache Hive 4.0.0-alpha-1 Release Candidate 2

[Naveen Gangam]

I have run some additional testing with ACID tables from the install. No
issues found.
+1 (binding)

On Mon, Mar 28, 2022 at 5:27 PM Naveen Gangam  wrote:

>
>- Downloaded the candidate bits and confirmed checksums.
>- Built locally and ran Metastore checkin tests and few other qtests.
>- Installed from binaries and ran some DDL/DMLs locally from beeline
>
> +1 (non-binding)
>
> On Mon, Mar 28, 2022 at 9:32 AM Denys Kuzmenko
>  wrote:
>
>> +1
>>
>> * Downloaded the release candidate artifacts and verified the signatures
>> and checksums;
>> * Built from the release sources;
>> * Initialized schema using schematool and launched hiveserver2 locally;
>> * Used beeline to connect:
>> - Created a few tables and loaded data into them.
>> - Executed some simple queries.
>>
>> Best regards,
>> Denys
>>
>> On Sat, Mar 26, 2022 at 11:24 PM Stamatis Zampetakis 
>> wrote:
>>
>> > Ubuntu 20.04.4 LTS, jdk1.8.0_261, Apache Maven 3.6.3
>> >
>> >  * Checked signatures and checksums OK
>> >  * Checked diff between repo and release sources (diff -qr hive
>> > apache-hive-4.0.0-alpha-1-src) OK
>> >  * Built from git tag (mvn clean install -DskipTests -Pitests) OK
>> >  * Built from release sources (mvn clean install -DskipTests -Pitests)
>> OK
>> >  * Run smoke tests on pseudo cluster using hive-dev-box OK
>> >
>> > All of the issues that were found in the previous RC are either
>> resolved or
>> > tracked under respective JIRAs to be solved for the next release.
>> >
>> > Smoke tests included:
>> > * Derby metastore initialization
>> > * simple CREATE TABLE statements;
>> > * basic INSERT INTO VALUES statements;
>> > * basic SELECT * FROM WHERE variations;
>> > * EXPLAIN statement variations;
>> > * ANALYZE TABLE variations;
>> >
>> > +1 (non-binding)
>> >
>> > Best,
>> > Stamatis
>> >
>> > On Thu, Mar 24, 2022 at 12:01 PM Peter Vary > >
>> > wrote:
>> >
>> > > Hi Team,
>> > >
>> > > Apache Hive 4.0.0-alpha-1 Release Candidate 2 is available here:
>> > > https://people.apache.org/~pvary/apache-hive-4.0.0-alpha-1-rc2/ <
>> > > https://people.apache.org/~pvary/apache-hive-4.0.0-alpha-1-rc2/>
>> > >
>> > > The checksums are these:
>> > > - 1e450197dbf847696b05042eb68b78b968064f1f1b369a7fb0b77a6329a27809
>> > > apache-hive-4.0.0-alpha-1-bin.tar.gz
>> > > - a21a609ec2e30f8cc656242c545bb3a04de21c2a1eee90808648e3aa4bf3d04e
>> > > apache-hive-4.0.0-alpha-1-src.tar.gz
>> > >
>> > > Maven artifacts are available here:
>> > >
>> https://repository.apache.org/content/repositories/orgapachehive-1113/ <
>> > >
>> https://repository.apache.org/content/repositories/orgapachehive-1113/>
>> > >
>> > > The tag 4.0.0-alpha-1-rc1 has been applied to the source for this
>> release
>> > > in github, you can see it at
>> > > https://github.com/apache/hive/tree/release-4.0.0-alpha-1-rc1 <
>> > > https://github.com/apache/hive/tree/release-4.0.0-alpha-1-rc1>
>> > >
>> > > The git commit hash is:
>> > >
>> > >
>> >
>> https://github.com/apache/hive/commit/357d4906f5c806d585fd84db57cf296e12e6049b
>> > > <
>> > >
>> >
>> https://github.com/apache/hive/commit/357d4906f5c806d585fd84db57cf296e12e6049b
>> > > >
>> > >
>> > > Voting will conclude in 72 hours.
>> > >
>> > > All interested parties: Please test.
>> > > Hive PMC Members: Please test and vote.
>> > >
>> > > Thanks.
>> >
>>
>

Re: [VOTE] Apache Hive 4.0.0-alpha-1 Release Candidate 2

[Naveen Gangam]

   - Downloaded the candidate bits and confirmed checksums.
   - Built locally and ran Metastore checkin tests and few other qtests.
   - Installed from binaries and ran some DDL/DMLs locally from beeline

+1 (non-binding)

On Mon, Mar 28, 2022 at 9:32 AM Denys Kuzmenko
 wrote:

> +1
>
> * Downloaded the release candidate artifacts and verified the signatures
> and checksums;
> * Built from the release sources;
> * Initialized schema using schematool and launched hiveserver2 locally;
> * Used beeline to connect:
> - Created a few tables and loaded data into them.
> - Executed some simple queries.
>
> Best regards,
> Denys
>
> On Sat, Mar 26, 2022 at 11:24 PM Stamatis Zampetakis 
> wrote:
>
> > Ubuntu 20.04.4 LTS, jdk1.8.0_261, Apache Maven 3.6.3
> >
> >  * Checked signatures and checksums OK
> >  * Checked diff between repo and release sources (diff -qr hive
> > apache-hive-4.0.0-alpha-1-src) OK
> >  * Built from git tag (mvn clean install -DskipTests -Pitests) OK
> >  * Built from release sources (mvn clean install -DskipTests -Pitests) OK
> >  * Run smoke tests on pseudo cluster using hive-dev-box OK
> >
> > All of the issues that were found in the previous RC are either resolved
> or
> > tracked under respective JIRAs to be solved for the next release.
> >
> > Smoke tests included:
> > * Derby metastore initialization
> > * simple CREATE TABLE statements;
> > * basic INSERT INTO VALUES statements;
> > * basic SELECT * FROM WHERE variations;
> > * EXPLAIN statement variations;
> > * ANALYZE TABLE variations;
> >
> > +1 (non-binding)
> >
> > Best,
> > Stamatis
> >
> > On Thu, Mar 24, 2022 at 12:01 PM Peter Vary 
> > wrote:
> >
> > > Hi Team,
> > >
> > > Apache Hive 4.0.0-alpha-1 Release Candidate 2 is available here:
> > > https://people.apache.org/~pvary/apache-hive-4.0.0-alpha-1-rc2/ <
> > > https://people.apache.org/~pvary/apache-hive-4.0.0-alpha-1-rc2/>
> > >
> > > The checksums are these:
> > > - 1e450197dbf847696b05042eb68b78b968064f1f1b369a7fb0b77a6329a27809
> > > apache-hive-4.0.0-alpha-1-bin.tar.gz
> > > - a21a609ec2e30f8cc656242c545bb3a04de21c2a1eee90808648e3aa4bf3d04e
> > > apache-hive-4.0.0-alpha-1-src.tar.gz
> > >
> > > Maven artifacts are available here:
> > > https://repository.apache.org/content/repositories/orgapachehive-1113/
> <
> > > https://repository.apache.org/content/repositories/orgapachehive-1113/
> >
> > >
> > > The tag 4.0.0-alpha-1-rc1 has been applied to the source for this
> release
> > > in github, you can see it at
> > > https://github.com/apache/hive/tree/release-4.0.0-alpha-1-rc1 <
> > > https://github.com/apache/hive/tree/release-4.0.0-alpha-1-rc1>
> > >
> > > The git commit hash is:
> > >
> > >
> >
> https://github.com/apache/hive/commit/357d4906f5c806d585fd84db57cf296e12e6049b
> > > <
> > >
> >
> https://github.com/apache/hive/commit/357d4906f5c806d585fd84db57cf296e12e6049b
> > > >
> > >
> > > Voting will conclude in 72 hours.
> > >
> > > All interested parties: Please test.
> > > Hive PMC Members: Please test and vote.
> > >
> > > Thanks.
> >
>

[VOTE] Apache Hive 3.1.3 Release Candidate 2

[Naveen Gangam]

Apache Hive 3.1.3 Release Candidate 2 is available here:
https://people.apache.org/~ngangam/apache-hive-3.1.3-rc-2

The checksums are these:


- 55c58e0111bd32de3d02f5f25d9eb054ba65ab02aaf669637760eaf56ef1fbb1
apache-hive-3.1.3-bin.tar.gz


- 22862e6bf76a4783a3d8d298634728cc9d6561563af2413a687fe63e35bcc527
apache-hive-3.1.3-src.tar.gz


Maven artifacts are available here:
https://repository.apache.org/content/repositories/orgapachehive-1114

The tag release-3.1.3-rc2 has been applied to the source for this
release in github, you can see it
athttps://github.com/apache/hive/tree/release-3.1.3-rc2

The git commit hash is: 4df4d75bf1e16fe0af75aad0b4179c34c07fc975

Voting will conclude in 72 hours.

Hive PMC Members: Please test and vote.

Thanks.

Naveen

Re: separated authN configuration for binary and http transports

[Naveen Gangam]

Hi Janos,
LDAP auth works in http mode as well.

We have made some enhancements recently:
HIveServer2 is now capable of supporting multiple authentication mode. For
example: in http mode, you can set it to "LDAP,SAML"
We have just added another auth mode (JWT) for http transport via
HIVE-25575. So now, we can add "JWT" to this list as well.

While we have checks to set it to something like "KERBEROS,SAML"  (KERBEROS
in binary mode and SAML is http mode only), I understand your general point
about having the ability to use LDAP with binary mode and SAML in http mode.

I am not certain this is a huge usecase for us, but if there is general
consensus that we need this, we could create a jira around this. My biggest
concern with the separation of the properties is backward compatibility.

Thank you
Naveen

On Mon, Mar 28, 2022 at 4:56 AM Stamatis Zampetakis 
wrote:

> Hey Janos,
>
> You brought up an interesting subject.
>
> I haven't worked on the code around the authentication process so cannot
> foresee the impact on the codebase but high level your idea seems
> reasonable to me.
>
> I would be favorable in such a change but I would definitely like to see
> some tests and documentation come along from the one who pushes this
> forward.
>
> Best,
> Stamatis
>
> On Fri, Mar 18, 2022, 6:40 PM Janos Kovacs  wrote:
>
> > Hi,
> >
> > I just found that while HS2 can do authentication with mixed methods -
> like
> > Kerberos+LDAP - it only works with the binary protocol. With the
> transport
> > set to http, the authentication basically works only against what is set
> by
> > hive.server2.authentication. If e.g. it's set to LDAP, it doesn't try
> other
> > methods, even if the client is sending the Negotiate headers in the
> > request.
> >
> > While this is something that probably could be fixed, I was thinking
> about
> > a quick(er) fix that might sounds just a workaround first, but adding the
> > fact that HS2 now can do both binary and http transports together
> > (HIVE-5312) and that there are other authentication methods which support
> > only one type of transports - like SAML works only with http transport -,
> > this might be a good enhancement by itself: split the
> > hive.server2.authentication between binary and http with introducing
> > hive.server2.http.authentication.
> >
> > If the http transport could be configured independently from the binary
> > transport, then HS2 could run in dual-transport mode, e.g. binary
> offering
> > Kerberos+LDAP while http offering SAML (or any other independent method).
> >
> > Could you please share your thoughts on splitting the authN method
> between
> > the two transport modes?
> >
> > Thanks, Janos
> >
>

Re: [VOTE] Apache Hive 3.1.3 Release Candidate 1

[Naveen Gangam]

Thanks Stamatis. Let me look into this.

On Thu, Mar 24, 2022 at 5:42 AM Stamatis Zampetakis 
wrote:

> Thanks for pushing this forward Naveen.
>
> I checked the released sources in apache-hive-3.1.3-src and they contain
> modified LGPL files violating the ASF release policy.
> The problem is the same reported under HIVE-25665. I think the fix
> should be backported to branch-3 before moving forward with the release.
>
> -1 (non-binding)
>
> Best,
> Stamatis
>
> On Wed, Mar 23, 2022 at 9:47 PM Naveen Gangam  >
> wrote:
>
> > Apache Hive 3.1.3 Release Candidate 1 is available here:
> > https://people.apache.org/~ngangam/apache-hive-3.1.3-rc-1
> >
> > The checksums are these:
> > - *e0551a6fe328be5ff0fa16d275b65f43f56c35da66ac4e391e47d3e74d466b91*
> > apache-hive-3.1.3-bin.tar.gz
> >
> > - *ce35a179304055004023bec016518fcb40b2ce2b14238ab77aebec99815fde02*
> > apache-hive-3.1.3-src.tar.gz
> >
> >
> > Maven artifacts are available
> > here:
> https://repository.apache.org/content/repositories/orgapachehive-1112
> >
> > The tag release-3.1.3-rc1 has been applied to the source for this
> > release in github, you can see it
> > athttps://github.com/apache/hive/tree/release-3.1.3-rc1
> >
> > The git commit hash is: cc050e40eb55f6c9f1aa08c00c1689f657747afb
> > <
> >
> https://github.com/apache/hive/commit/cc050e40eb55f6c9f1aa08c00c1689f657747afb
> > >
> > Voting will conclude in 72 hours.
> >
> > Hive PMC Members: Please test and vote.
> >
> > Thanks.
> >
>

[VOTE] Apache Hive 3.1.3 Release Candidate 1

[Naveen Gangam]

Apache Hive 3.1.3 Release Candidate 1 is available here:
https://people.apache.org/~ngangam/apache-hive-3.1.3-rc-1

The checksums are these:
- *e0551a6fe328be5ff0fa16d275b65f43f56c35da66ac4e391e47d3e74d466b91*
apache-hive-3.1.3-bin.tar.gz

- *ce35a179304055004023bec016518fcb40b2ce2b14238ab77aebec99815fde02*
apache-hive-3.1.3-src.tar.gz


Maven artifacts are available
here:https://repository.apache.org/content/repositories/orgapachehive-1112

The tag release-3.1.3-rc1 has been applied to the source for this
release in github, you can see it
athttps://github.com/apache/hive/tree/release-3.1.3-rc1

The git commit hash is: cc050e40eb55f6c9f1aa08c00c1689f657747afb

Voting will conclude in 72 hours.

Hive PMC Members: Please test and vote.

Thanks.

Re: [VOTE] Apache Hive 4.0.0-alpha-1 Release Candidate 1

[Naveen Gangam]

I have been able to build and run a quick test. I have NOT verified the
signature. I was trying to run the HMS Checkin tests and got this. I
suspect these are not specific to the alpha-1 branch. But it is not a test
failure (although it appears like it should be)
*"mvn test
-Dtest.groups=org.apache.hadoop.hive.metastore.annotation.MetastoreCheckinTest"*

[*INFO*] Running org.apache.hadoop.hive.common.metrics.*TestLegacyMetrics*

[main] WARN org.apache.hadoop.hive.common.metrics.LegacyMetrics - Could not
find counter value for foo.n, returning null instead.

javax.management.AttributeNotFoundException: Key [foo.n] not found/tracked

at
org.apache.hadoop.hive.common.metrics.MetricsMBeanImpl.getAttribute(MetricsMBeanImpl.java:56)


[*WARNING*] *Tests **run: 18*, Failures: 0, Errors: 0, *Skipped: 2*, Time
elapsed: 4.158 s - in org.apache.hadoop.hive.metastore.client.*TestCatalogs*

[*INFO*] Running org.apache.hadoop.hive.metastore.*TestMarkPartition*

[*INFO*] *Tests run: 1*, Failures: 0, Errors: 0, Skipped: 0, Time elapsed:
15.021 s - in org.apache.hadoop.hive.metastore.*TestMarkPartition*

[*INFO*]

[*INFO*] Results:

[*INFO*]

[*WARNING*] *Tests run: 2182, Failures: 0, Errors: 0, Skipped: 5*


So over no test failures.

+1 pending other votes (& non-binding)

Thank you
Naveen

On Tue, Mar 22, 2022 at 9:32 AM Marton Bod 
wrote:

> +1 (non-binding)
> Tested the checksums, signatures and built it successfully
>
> On Tue, Mar 22, 2022 at 2:26 PM Peter Vary 
> wrote:
>
> > Hi Team,
> >
> > Apache Hive 4.0.0-alpha-1 Release Candidate 1 is available here:
> >
> > https://people.apache.org/~pvary/apache-hive-4.0.0-alpha-1-rc1/
> >
> > Maven artifacts are available here:
> >
> > https://repository.apache.org/content/repositories/orgapachehive-/
> >
> > The tag 4.0.0-alpha-1-rc1 has been applied to the source for this release
> > in github, you can see it at
> > https://github.com/apache/hive/tree/release-4.0.0-alpha-1-rc1
> >
> > Voting will conclude in 72 hours.
> >
> > All interested parties: Please test.
> > Hive PMC Members: Please test and vote.
> >
> > Thanks.
>

[jira] [Created] (HIVE-26046) MySQL's bit datatype is default to void datatype in hive

[Naveen Gangam (Jira)]

Naveen Gangam created HIVE-26046:


 Summary: MySQL's bit datatype is default to void datatype in hive
 Key: HIVE-26046
 URL: https://issues.apache.org/jira/browse/HIVE-26046
 Project: Hive
  Issue Type: Sub-task
  Components: Standalone Metastore
Affects Versions: 4.0.0
Reporter: Naveen Gangam


describe on a table that contains a "bit" datatype gets mapped to void. We need 
a explicit conversion logic in the MySQL ConnectorProvider to map it to a 
suitable datatype in hive.

{noformat}
+---+---++
|   col_name| data_type 
|  comment   |
+---+---++
| tbl_id| bigint
| from deserializer  |
| create_time   | int   
| from deserializer  |
| db_id | bigint
| from deserializer  |
| last_access_time  | int   
| from deserializer  |
| owner | varchar(767)  
| from deserializer  |
| owner_type| varchar(10)   
| from deserializer  |
| retention | int   
| from deserializer  |
| sd_id | bigint
| from deserializer  |
| tbl_name  | varchar(256)  
| from deserializer  |
| tbl_type  | varchar(128)  
| from deserializer  |
| view_expanded_text| string
| from deserializer  |
| view_original_text| string
| from deserializer  |
| is_rewrite_enabled| void  
| from deserializer  |
| write_id  | bigint
| from deserializer  
{noformat}




--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Created] (HIVE-26045) Detect timed out connections for providers and auto-reconnect

[Naveen Gangam (Jira)]

Naveen Gangam created HIVE-26045:


 Summary: Detect timed out connections for providers and 
auto-reconnect
 Key: HIVE-26045
 URL: https://issues.apache.org/jira/browse/HIVE-26045
 Project: Hive
  Issue Type: Sub-task
  Components: HiveServer2
Affects Versions: 4.0.0
Reporter: Naveen Gangam


For the connectors, we use single connection, no pooling. But when the 
connection is idle for an extended period, the JDBC connection times out. We 
need to check for closed connections (Connection.isClosed()?) and re-establish 
the connection. Otherwise it renders the connector fairly useless.

{noformat}
2022-03-17T13:02:16,635  WARN [HiveServer2-Handler-Pool: Thread-116] 
thrift.ThriftCLIService: Error executing statement: 
org.apache.hive.service.cli.HiveSQLException: Error while compiling statement: 
FAILED: SemanticException Unable to fetch table temp_dbs. Error retrieving 
remote 
table:com.mysql.jdbc.exceptions.jdbc4.MySQLNonTransientConnectionException: No 
operations allowed after connection closed.
at 
org.apache.hive.service.cli.operation.Operation.toSQLException(Operation.java:373)
 ~[hive-service-3.1.3000.7.2.15.0-SNAPSHOT.jar:3.1.3000.7.2.15.0-SNAPSHOT]
at 
org.apache.hive.service.cli.operation.SQLOperation.prepare(SQLOperation.java:211)
 ~[hive-service-3.1.3000.7.2.15.0-SNAPSHOT.jar:3.1.3000.7.2.15.0-SNAPSHOT]
at 
org.apache.hive.service.cli.operation.SQLOperation.runInternal(SQLOperation.java:265)
 ~[hive-service-3.1.3000.7.2.15.0-SNAPSHOT.jar:3.1.3000.7.2.15.0-SNAPSHOT]
at 
org.apache.hive.service.cli.operation.Operation.run(Operation.java:285) 
~[hive-service-3.1.3000.7.2.15.0-SNAPSHOT.jar:3.1.3000.7.2.15.0-SNAPSHOT]
at 
org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementInternal(HiveSessionImpl.java:576)
 ~[hive-service-3.1.3000.7.2.15.0-SNAPSHOT.jar:3.1.3000.7.2.15.0-SNAPSHOT]
at 
org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementAsync(HiveSessionImpl.java:562)
 ~[hive-service-3.1.3000.7.2.15.0-SNAPSHOT.jar:3.1.3000.7.2.15.0-SNAPSHOT]
at sun.reflect.GeneratedMethodAccessor52.invoke(Unknown Source) ~[?:?]
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 ~[?:1.8.0_231]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_231]
at 
org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:78)
 ~[hive-service-3.1.3000.7.2.15.0-SNAPSHOT.jar:3.1.3000.7.2.15.0-SNAPSHOT]
at 
org.apache.hive.service.cli.session.HiveSessionProxy.access$000(HiveSessionProxy.java:36)
 ~[hive-service-3.1.3000.7.2.15.0-SNAPSHOT.jar:3.1.3000.7.2.15.0-SNAPSHOT]
at 
org.apache.hive.service.cli.session.HiveSessionProxy$1.run(HiveSessionProxy.java:63)
 ~[hive-service-3.1.3000.7.2.15.0-SNAPSHOT.jar:3.1.3000.7.2.15.0-SNAPSHOT]
at java.security.AccessController.doPrivileged(Native Method) 
~[?:1.8.0_231]
at javax.security.auth.Subject.doAs(Subject.java:422) ~[?:1.8.0_231]
at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1682)
 ~[hadoop-common-3.1.0.jar:?]
at 
org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:59)
 ~[hive-service-3.1.3000.7.2.15.0-SNAPSHOT.jar:3.1.3000.7.2.15.0-SNAPSHOT]
at com.sun.proxy.$Proxy44.executeStatementAsync(Unknown Source) ~[?:?]
at 
org.apache.hive.service.cli.CLIService.executeStatementAsync(CLIService.java:315)
 ~[hive-service-3.1.3000.7.2.15.0-SNAPSHOT.jar:3.1.3000.7.2.15.0-SNAPSHOT]
at 
org.apache.hive.service.cli.thrift.ThriftCLIService.ExecuteStatement(ThriftCLIService.java:567)
 ~[hive-service-3.1.3000.7.2.15.0-SNAPSHOT.jar:3.1.3000.7.2.15.0-SNAPSHOT]
at 
org.apache.hive.service.rpc.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1550)
 ~[hive-exec-3.1.3000.7.2.15.0-SNAPSHOT.jar:3.1.3000.7.2.15.0-SNAPSHOT]
at 
org.apache.hive.service.rpc.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1530)
 ~[hive-exec-3.1.3000.7.2.15.0-SNAPSHOT.jar:3.1.3000.7.2.15.0-SNAPSHOT]
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:38) 
~[hive-exec-3.1.3000.7.2.15.0-SNAPSHOT.jar:3.1.3000.7.2.15.0-SNAPSHOT]
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:38) 
~[hive-exec-3.1.3000.7.2.15.0-SNAPSHOT.jar:3.1.3000.7.2.15.0-SNAPSHOT]
at 
org.apache.hive.service.auth.TSetIpAddressProcessor.process(TSetIpAddressProcessor.java:56)
 ~[hive-service-3.1.3000.7.2.15.0-SNAPSHOT.jar:3.1.3000.7.2.15.0-SNAPSHOT]
at 
org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:313)
 ~[hive-exec-3.1.3000.7.2.15.0-SNAPSHOT.jar:3.1.3000.7.2.15.0-SNAPSHOT]
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149

[jira] [Created] (HIVE-26012) HMS APIs to be enhanced for metadata replication

[Naveen Gangam (Jira)]

Naveen Gangam created HIVE-26012:


 Summary: HMS APIs to be enhanced for metadata replication
 Key: HIVE-26012
 URL: https://issues.apache.org/jira/browse/HIVE-26012
 Project: Hive
  Issue Type: Improvement
  Components: Metastore
Affects Versions: 3.1.0
Reporter: Naveen Gangam


HMS currently has APIs like these that automatically create/delete the 
directories on the associated DFS. 
[create/drop]_database
[create/drop]_table*
[add/append/drop]_partition*

This is expected and should be this way when query processors use this APIs. 
However, when tools that replicate hive metadata use this APIs on the target 
cluster, creating these dirs on target side which cause the replication of 
DFS-snapshots to fail.

So we if provide an option to bypass this creation of dirs, dfs replications 
will be smoother. In the future we will need to restrict users that can use 
these APIs. So we will have some sort of an authorization policy.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

Re: Start releasing the master branch

[Naveen Gangam]

Release for 3.1.3 is underway. Initial set of targetted backports are
complete. I will follow the remaining steps to have a build soon.

Would be nice to release at a regular cadence.

On Tue, Feb 22, 2022 at 6:18 AM Peter Vary 
wrote:

> I would vote for 4.0.0-alpha-1 or similar for all of the components.
>
> When we have more stable releases I would keep the 4.x.x schema, since
> everyone is familiar with it, and I do not see a really good reason to
> change it.
>
> Thanks,
> Peter
>
>
> > On 2022. Feb 10., at 3:34, Szehon Ho  wrote:
> >
> > +1 that would be awesome to see Hive master released after so long.
> >
> > Either 4.0 or 4.0.0-alpha-1 makes sense to me, not sure how we would pick
> > any 3.x or calendar date (which could tend to slip and be more
> confusing?).
> >
> > Thanks in any case to get the ball rolling.
> > Szehon
> >
> > On Wed, Feb 9, 2022 at 4:55 AM Zoltan Haindrich  wrote:
> >
> >> Hey,
> >>
> >> Thank you guys for chiming in; versioning is for sure something we
> should
> >> get to some common ground.
> >> Its a triple problem right now; I think we have the following things:
> >> * storage-api
> >> ** we have "2.7.3-SNAPSHOT" in the repo
> >> ***
> >>
> https://github.com/apache/hive/blob/0d1cc7c5005fe47759298fb35a1c67edc93f/storage-api/pom.xml#L27
> >> ** meanwhile we already have 2.8.1 released to maven central
> >> *** https://mvnrepository.com/artifact/org.apache.hive/hive-storage-api
> >> * standalone-metastore
> >> ** 4.0.0-SNAPSHOT in the repo
> >> ** last release is 3.1.2
> >> * hive
> >> ** 4.0.0-SNAPSHOT in the repo
> >> ** last release is 3.1.2
> >>
> >> Regarding the actual version number I'm not entirely sure where we
> should
> >> start the numbering - that's why I was referring to it as Hive-X in my
> >> first letter.
> >>
> >> I think the key point here would be to start shipping releases
> regularily
> >> and not the actual version number we will use - I'll kinda open to any
> >> versioning scheme which
> >> reflects that this is a newer release than 3.1.2.
> >>
> >> I could imagine the following ones:
> >> (A) start with something less expected; but keep 3 in the prefix to
> >> reflect that this is not yet 4.0
> >> I can imagine the following numbers:
> >> 3.900.0, 3.901.0, ...
> >> 3.9.0, 3.9.1, ...
> >> (B) start 4.0.0
> >> 4.0.0, 4.1.0, ...
> >> (C) jump to some calendar based version number like 2022.2.9
> >> trunk based development has pros and cons...making a move like this
> >> irreversibly pledges trunk based development; and makes release branches
> >> hard to introduce
> >> (X) somewhat orthogonal is to (also) use some suffixes
> >> 4.0.0-alpha1, 4.0.0-alpha2, 4.0.0-beta1
> >> this is probably the most tempting to use - but this versioning
> >> schema with a non-changing MINOR and PATCH number will
> >> also suggest that the actual software is fully compatible - and only
> >> bugs are being fixed - which will not be true...
> >>
> >> I really like the idea to suffix these releases with alpha or beta -
> which
> >> will communicate our level commitment that these are not 100% production
> >> ready artifacts.
> >>
> >> I think we could fix HIVE-25665; and probably experiment with
> 4.0.0-alpha1
> >> for start...
> >>
> >>> This also means there should *not* be a branch-4 after releasing Hive
> >> 4.0
> >>> and let that diverge (and becomes the next, super-ignored branch-3),
> >> correct; no need to keep a branch we don't maintain...but in any case I
> >> think we can postpone this decision until there will be something to
> >> release... :)
> >>
> >> cheers,
> >> Zoltan
> >>
> >>
> >>
> >> On 2/9/22 10:23 AM, László Bodor wrote:
> >>> Hi All!
> >>>
> >>> A purely technical question: what will the SNAPSHOT version become
> after
> >>> releasing Hive 4.0.0? I think this is important, as it defines and
> >> reflects
> >>> the future release plans.
> >>>
> >>> Currently, it's 4.0.0-SNAPSHOT, I guess it's since Hive 3.0 + branch-3.
> >>> Hive is an evolving and super-active project: if we want to make
> regular
> >>> releases, we should simply release Hive 4.0 and bump pom to
> >> 4.1.0-SNAPSHOT,
> >>> which clearly says that we can release Hive 4.1 anytime we want,
> without
> >>> being frustrated about "whether we included enough cool stuff to
> release
> >>> 5.0".
> >>>
> >>> This also means there should *not* be a branch-4 after releasing Hive
> 4.0
> >>> and let that diverge (and becomes the next, super-ignored branch-3),
> only
> >>> when we end up bringing a minor backward-incompatible thing that needs
> a
> >>> 4.0.x, and when it happens, we'll create *branch-4.0 *on demand. For
> me,
> >> a
> >>> branch called *branch-4.0* doesn't imply either I can expect cool
> >> releases
> >>> in the future from there or the branch is maintained and tries to be in
> >>> sync with the *master*.
> >>>
> >>> Regards,
> >>> Laszlo Bodor
> >>>
> >>> Alessandro Solimando  ezt írta
> (időpont:
> >>> 2022. febr. 8., K, 16:42):
> >>>
> >>>

[jira] [Created] (HIVE-25875) Support multiple authentication mechanisms simultaneously

[Naveen Gangam (Jira)]

Naveen Gangam created HIVE-25875:


 Summary: Support multiple authentication mechanisms simultaneously 
 Key: HIVE-25875
 URL: https://issues.apache.org/jira/browse/HIVE-25875
 Project: Hive
  Issue Type: Improvement
  Components: HiveServer2
Affects Versions: 3.1.0
Reporter: Naveen Gangam
Assignee: Naveen Gangam


Currently, HS2 supports a single form of auth on any given instance of 
HiveServer2. Hive should be able to support multiple auth mechanisms on a 
single instance especially with http transport. for example, LDAP and SAML.  In 
both cases, HS2 ends up with receiving an Authorization Header in the request. 
Similarly we could be able to support JWT support or other forms of boundary 
authentication that is done outside of Hive.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Created] (HIVE-25855) Make a branch-3 release

[Naveen Gangam (Jira)]

Naveen Gangam created HIVE-25855:


 Summary: Make a branch-3 release 
 Key: HIVE-25855
 URL: https://issues.apache.org/jira/browse/HIVE-25855
 Project: Hive
  Issue Type: Bug
Reporter: Naveen Gangam
Assignee: Naveen Gangam


This jira is to track commits for a hive release off branch-3



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Created] (HIVE-25798) Update pom.xml

[Naveen Gangam (Jira)]

Naveen Gangam created HIVE-25798:


 Summary: Update pom.xml
 Key: HIVE-25798
 URL: https://issues.apache.org/jira/browse/HIVE-25798
 Project: Hive
  Issue Type: Bug
  Components: HiveServer2
Reporter: Naveen Gangam
Assignee: Naveen Gangam






--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Created] (HIVE-25214) Add hive authorization support for Data connectors.

[Naveen Gangam (Jira)]

Naveen Gangam created HIVE-25214:


 Summary: Add hive authorization support for Data connectors.
 Key: HIVE-25214
 URL: https://issues.apache.org/jira/browse/HIVE-25214
 Project: Hive
  Issue Type: Sub-task
Reporter: Naveen Gangam
Assignee: Naveen Gangam


We need to add authorization support for data connectors in hive. The default 
behavior should be
1) Connectors can be create/dropped by users in admin role.
2) Connectors have READ and WRITE permissions.
*   READ permissions are required to fetch a connector object or fetch all 
connector names. So to create a REMOTE database using a connector, users will 
need READ permission on the connector. DDL queries like "show connectors" and 
"describe " will check for read access on the connector as well.
*   WRITE permissions are required to alter/drop a connector. DDL queries like 
"alter connector" and "drop connector" will need WRITE access on the connector.

Adding this support, Ranger can integrate with this.
   



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HIVE-25213) Implement List getTables() for existing connectors.

[Naveen Gangam (Jira)]

Naveen Gangam created HIVE-25213:


 Summary: Implement List getTables() for existing connectors.
 Key: HIVE-25213
 URL: https://issues.apache.org/jira/browse/HIVE-25213
 Project: Hive
  Issue Type: Sub-task
Reporter: Naveen Gangam
Assignee: Naveen Gangam


In the initial implementation, connector providers do not implement the 
getTables(string pattern) spi. We had deferred it for later. Only 
getTableNames() and getTable() were implemented. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HIVE-24970) Reject location and managed locations in DDL for REMOTE databases.

[Naveen Gangam (Jira)]

Naveen Gangam created HIVE-24970:


 Summary: Reject location and managed locations in DDL for REMOTE 
databases.
 Key: HIVE-24970
 URL: https://issues.apache.org/jira/browse/HIVE-24970
 Project: Hive
  Issue Type: Sub-task
  Components: HiveServer2
Affects Versions: 4.0.0
Reporter: Naveen Gangam
Assignee: Naveen Gangam


This was part of the review feedback from Yongzhi. Creating a followup jira to 
track this discussion.
So, using DB connector for DB, will not create managed tables?

 
@nrg4878 nrg4878 1 hour ago Author Member
we don't support create/drop/alter in REMOTE databases at this point. the 
concepts of managed vs external is not in the picture at this point. When we do 
support it, it will be application to the hive connectors only (or other hive 
based connectors like AWS Glue)

 
@nrg4878 nrg4878 2 minutes ago Author Member
will file a separate jira for this. Basically, instead of ignoring the location 
and managedlocation that may be specified for remote database, the grammer 
needs to not accept any locations in the DDL at all.
The argument is fair, why accept something we do not honor or entirely 
irrelevant for such databases. However, this requires some thought when we have 
additional connectors for remote hive instances. It might have some relevance 
in terms of security with Ranger etc.
So will create new jira for followup discussion.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HIVE-24942) Consider use of lambda expressions in formatters.

[Naveen Gangam (Jira)]

Naveen Gangam created HIVE-24942:


 Summary: Consider use of lambda expressions in formatters.
 Key: HIVE-24942
 URL: https://issues.apache.org/jira/browse/HIVE-24942
 Project: Hive
  Issue Type: Sub-task
Affects Versions: 4.0.0
Reporter: Naveen Gangam
Assignee: Narayanan Venkateswaran


ArrayList dcDescription = new ArrayList();

dcDescription.add(connector);
dcDescription.add(type);
dcDescription.add(ownerName);
dcDescription.add(ownerType);
dcDescription.add(HiveStringUtils.escapeJava(comment));
dcDecription.add(params.toString());

Consumer description_handler = (param) -> { 
out.write(param.getBytes(StandardCharsets.UTF_8));};

dcDescription.forEach(param);





--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HIVE-24941) [Evaluate] if ReplicationSpec is needed for DataConnectors.

[Naveen Gangam (Jira)]

Naveen Gangam created HIVE-24941:


 Summary: [Evaluate] if ReplicationSpec is needed for 
DataConnectors.
 Key: HIVE-24941
 URL: https://issues.apache.org/jira/browse/HIVE-24941
 Project: Hive
  Issue Type: Sub-task
  Components: Hive
Affects Versions: 4.0.0
Reporter: Naveen Gangam
Assignee: Naveen Gangam


We have ReplicationSpec on Connector. Not sure if this is needed, if we do not 
want to replicate connectors.

  public ReplicationSpec getReplicationSpec() {
return replicationSpec;
  }



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HIVE-24938) [Evaluate] Dataconnector URL validation on create

[Naveen Gangam (Jira)]

Naveen Gangam created HIVE-24938:


 Summary: [Evaluate] Dataconnector URL validation on create
 Key: HIVE-24938
 URL: https://issues.apache.org/jira/browse/HIVE-24938
 Project: Hive
  Issue Type: Sub-task
Reporter: Naveen Gangam
Assignee: Naveen Gangam


>From the review feedback, there was a comment about validating URL specified 
>in the connector URL when it is created. Currently, there is no validation 
>except for checking for empty/null value. This is by-design and the desired 
>behavior, IMHO. But filing this to be discussed with wider audience.

{noformat}
I tried creating a connector without the mysql JDBC URL specified properly and 
it went through,

please see below,

CREATE CONNECTOR mysql_test_2
TYPE 'mysql'
URL 'jdbc://'
COMMENT 'test connector'
WITH DCPROPERTIES (
"hive.sql.dbcp.username"="hive1",
"hive.sql.dbcp.password"="hive1");

CREATE CONNECTOR mysql_test_3
TYPE 'mysql'
URL 'jdbc:derby://nightly1.apache.org:3306/hive1'
COMMENT 'test connector'
WITH DCPROPERTIES (
"hive.sql.dbcp.username"="hive1",
"hive.sql.dbcp.password"="hive1");

I am not saying they are wrong, but we should probably call this out in the 
documentation. Document that URLs are not verified.

Another thing I noticed is that the password is displayed in plain
text on the command line. This used be considered a security problem
in a product I worked in a past life. But I notice that an external
table can be created with this semantics. I guess it is acceptable
here.

It is also stored in plain text in the metastore, please see below,

CREATE TABLE DATACONNECTOR_PARAMS (
NAME VARCHAR(128) NOT NULL,
PARAM_KEY VARCHAR(180) NOT NULL,
PARAM_VALUE VARCHAR(4000),
PRIMARY KEY (NAME, PARAM_KEY),
CONSTRAINT DATACONNECTOR_NAME_FK1 FOREIGN KEY (NAME) REFERENCES DATACONNECTORS 
(NAME) ON DELETE CASCADE
) ENGINE=InnoDB DEFAULT CHARSET=latin1;

Again I am not saying this is a problem, but I thought I can call this out to 
you.

 
@nrg4878 nrg4878 24 minutes ago Author Member
We check for null/empty values for URL. We error out in those cases. Other than 
that, any non-empty value is accepted. I dont think we should check for 
correctness of the URL or even can for that matter.
a) The URL is meant to be a freeform value against dozens of datasource types 
(mysql, postgres, hive, AWS Glue, Redshift etc). For each such source type, 
there could be dozens of variations of the url (includes properties and other 
params specific to the source). So I dont think we can meaningfully detect 
incorrect URLs.
For example, MySQL though the URL might look fine syntactically, we cannot 
confirm dbName1 or dbName2 exist without actually attempting to connect to the 
DB.
jdbc:mysql://:3306/
jdbc:mysql://:3306/
b) The format for the URLs could be changing overtime as well. It is 
unnecessary burden for maintaining new formats in hive. We want to be able to 
plugin a new datasource type by simply adding a provider.

c) To be able to validate the URL, we have to establish the connection to the 
datasource at the time of creation. We are trying to delay making that 
connection as long as possible. When actual show tables is called. We avoid 
using up extra resources and leak connections.

d) Users can do "create connector" .. followed by "alter connector set url". So 
any incorrect URLS can be modified using alter. Also in this case, we would be 
checking the URL twice. Better to have the onus of configuring it correctly on 
the end user.

Passwords can be secured using jceks files as described in the "Securing 
Password" section of the doc below.
https://cwiki.apache.org/confluence/display/Hive/JDBC+Storage+Handler
So users have an option of using non-CTVs
{noformat}




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HIVE-24887) getDatabase() to call translation code even if client has no capabilities

[Naveen Gangam (Jira)]

Naveen Gangam created HIVE-24887:


 Summary: getDatabase() to call translation code even if client has 
no capabilities
 Key: HIVE-24887
 URL: https://issues.apache.org/jira/browse/HIVE-24887
 Project: Hive
  Issue Type: Sub-task
Reporter: Naveen Gangam
Assignee: Naveen Gangam


We do this for other calls that go thru translation layer. For some reason, the 
current code only calls it when the client sets the capabilities.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HIVE-24844) Add implementation for a 'hive' connector provider

[Naveen Gangam (Jira)]

Naveen Gangam created HIVE-24844:


 Summary: Add implementation for a 'hive' connector provider
 Key: HIVE-24844
 URL: https://issues.apache.org/jira/browse/HIVE-24844
 Project: Hive
  Issue Type: Sub-task
Reporter: Naveen Gangam


This connector implementation will allow HMS to communicate with remote HMS 
instances for metadata.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HIVE-24770) Upgrade should update changed FQN in HMS DB.

[Naveen Gangam (Jira)]

Naveen Gangam created HIVE-24770:


 Summary: Upgrade should update changed FQN in HMS DB.
 Key: HIVE-24770
 URL: https://issues.apache.org/jira/browse/HIVE-24770
 Project: Hive
  Issue Type: Sub-task
  Components: Hive
Affects Versions: 4.0.0
Reporter: Naveen Gangam
Assignee: Naveen Gangam


While the parent change has does not cause this directly, but post upgrade the 
existing tables that use MultiDelimiterSerDe will be broken as the hive-contrib 
jar would no longer exist. Instead if the Hive schema upgrade script can update 
the SERDES table to alter the classname to the new classname, the old tables 
would work automatically. Much better user experience.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HIVE-24464) Evaluate the need to have directSQL implementation for data connectors

[Naveen Gangam (Jira)]

Naveen Gangam created HIVE-24464:


 Summary: Evaluate the need to have directSQL implementation for 
data connectors
 Key: HIVE-24464
 URL: https://issues.apache.org/jira/browse/HIVE-24464
 Project: Hive
  Issue Type: Sub-task
Reporter: Naveen Gangam


I expect that there will be just a handful of connectors not 100's of them like 
databases. But creating a placeholder item to evaluate at a future time. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HIVE-24461) Provide CachedStore implementation for dataconnectors

[Naveen Gangam (Jira)]

Naveen Gangam created HIVE-24461:


 Summary: Provide CachedStore implementation for dataconnectors
 Key: HIVE-24461
 URL: https://issues.apache.org/jira/browse/HIVE-24461
 Project: Hive
  Issue Type: Sub-task
Reporter: Naveen Gangam


Currently, none of the connectors are cached. They are all delegated to the 
ObjectStore for every call.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HIVE-24452) Add a generic JDBC implementation that can be used to other JDBC DBs

[Naveen Gangam (Jira)]

Naveen Gangam created HIVE-24452:


 Summary: Add a generic JDBC implementation that can be used to 
other JDBC DBs
 Key: HIVE-24452
 URL: https://issues.apache.org/jira/browse/HIVE-24452
 Project: Hive
  Issue Type: Sub-task
Reporter: Naveen Gangam


Currently, we added a custom provider for each of the JDBC DBs supported by 
hive (MySQL, Postgres, MSSQL(pending), Oracle(pending) and Derby (pending)).  
But if there are other JDBC providers we want to add support for, adding a 
generic JDBC provider would be useful that hive can default to.
This means
1) We have to support means to indicate that a connector is for a JDBC 
datasource. So maybe add a property in DCPROPERTIES on connector to indicate 
that the datasource supports JDBC.
2) If there is no custom connector for a data source, use the 
GenericJDBCDatasource connector that is to be added as part of this jira.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HIVE-24451) Add schema changes for MSSQL

[Naveen Gangam (Jira)]

Naveen Gangam created HIVE-24451:


 Summary: Add schema changes for MSSQL
 Key: HIVE-24451
 URL: https://issues.apache.org/jira/browse/HIVE-24451
 Project: Hive
  Issue Type: Sub-task
Reporter: Naveen Gangam


The current patch does not include schema changes for MSSQL backend. This 
should be right after the initial commit.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HIVE-24449) Implement connector provider for Derby DB

[Naveen Gangam (Jira)]

Naveen Gangam created HIVE-24449:


 Summary: Implement connector provider for Derby DB
 Key: HIVE-24449
 URL: https://issues.apache.org/jira/browse/HIVE-24449
 Project: Hive
  Issue Type: Sub-task
  Components: Hive
Affects Versions: 4.0.0
Reporter: Naveen Gangam
Assignee: Naveen Gangam


Provide an implementation of Connector provider for Derby DB.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HIVE-24448) Support case-sensitivity for tables in REMOTE database.

[Naveen Gangam (Jira)]

Naveen Gangam created HIVE-24448:


 Summary: Support case-sensitivity for tables in REMOTE database.
 Key: HIVE-24448
 URL: https://issues.apache.org/jira/browse/HIVE-24448
 Project: Hive
  Issue Type: Sub-task
  Components: HiveServer2
Affects Versions: 4.0.0
Reporter: Naveen Gangam


Hive tables are case-insensitive. So any case specified in user queries are 
converted to lower case for query planning and all of the HMS metadata is also 
persisted as lower case names.
However, with REMOTE data sources, certain data source will support 
case-sensitivity for tables. 
So HiveServer2 query planner needs to preserve user-provided case to be used 
with HMS APIs, for HMS to be able to fetch the metadata from a remote data 
source.
We now see something like this

{noformat}
2020-11-25T16:45:36,402  WARN [HiveServer2-Handler-Pool: Thread-76] 
thrift.ThriftCLIService: Error executing statement: 
org.apache.hive.service.cli.HiveSQLException: Error while compiling statement: 
FAILED: RuntimeException 
MetaException(message:org.apache.hadoop.hive.serde2.SerDeException 
org.apache.hive.storage.jdbc.exception.HiveJdbcDatabaseAccessException: Error 
while trying to get column names: Table 'hive1.txns' doesn't exist)
at 
org.apache.hive.service.cli.operation.Operation.toSQLException(Operation.java:365)
 ~[hive-service-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at 
org.apache.hive.service.cli.operation.SQLOperation.prepare(SQLOperation.java:206)
 ~[hive-service-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at 
org.apache.hive.service.cli.operation.SQLOperation.runInternal(SQLOperation.java:262)
 ~[hive-service-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at 
org.apache.hive.service.cli.operation.Operation.run(Operation.java:277) 
~[hive-service-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at 
org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementInternal(HiveSessionImpl.java:560)
 ~[hive-service-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at 
org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementAsync(HiveSessionImpl.java:545)
 ~[hive-service-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at sun.reflect.GeneratedMethodAccessor68.invoke(Unknown Source) ~[?:?]
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 ~[?:1.8.0_231]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_231]
at 
org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:78)
 ~[hive-service-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at 
org.apache.hive.service.cli.session.HiveSessionProxy.access$000(HiveSessionProxy.java:36)
 ~[hive-service-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at 
org.apache.hive.service.cli.session.HiveSessionProxy$1.run(HiveSessionProxy.java:63)
 ~[hive-service-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at java.security.AccessController.doPrivileged(Native Method) 
~[?:1.8.0_231]
at javax.security.auth.Subject.doAs(Subject.java:422) ~[?:1.8.0_231]
at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1682)
 ~[hadoop-common-3.1.0.jar:?]
at 
org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:59)
 ~[hive-service-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at com.sun.proxy.$Proxy43.executeStatementAsync(Unknown Source) ~[?:?]
at 
org.apache.hive.service.cli.CLIService.executeStatementAsync(CLIService.java:315)
 ~[hive-service-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at 
org.apache.hive.service.cli.thrift.ThriftCLIService.ExecuteStatement(ThriftCLIService.java:571)
 ~[hive-service-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at 
org.apache.hive.service.rpc.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1550)
 ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at 
org.apache.hive.service.rpc.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1530)
 ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:38) 
~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:38) 
~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at 
org.apache.hive.service.auth.TSetIpAddressProcessor.process(TSetIpAddressProcessor.java:56)
 ~[hive-service-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at 
org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:313)
 ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) 
~[?:1.8.0_231]
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) 
~[?:1.8.0_231]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_231]
Caused by: java.lang.R

[jira] [Created] (HIVE-24447) Move create/drop/alter table to the provider interface

[Naveen Gangam (Jira)]

Naveen Gangam created HIVE-24447:


 Summary: Move create/drop/alter table to the provider interface
 Key: HIVE-24447
 URL: https://issues.apache.org/jira/browse/HIVE-24447
 Project: Hive
  Issue Type: Sub-task
  Components: Hive
Affects Versions: 4.0.0
Reporter: Naveen Gangam
Assignee: Naveen Gangam


The support for such operations on a table in a REMOTE database will be left to 
the discretion of the providers to support/implement.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HIVE-24425) Create table in REMOTE db should fail

[Naveen Gangam (Jira)]

Naveen Gangam created HIVE-24425:


 Summary: Create table in REMOTE db should fail
 Key: HIVE-24425
 URL: https://issues.apache.org/jira/browse/HIVE-24425
 Project: Hive
  Issue Type: Sub-task
Reporter: Naveen Gangam
Assignee: Naveen Gangam


Currently it creates the table in that DB but show tables does not show 
anything. Preventing the creation of table will resolve this inconsistency too.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HIVE-24405) Missing datatype for table column in oracle

[Naveen Gangam (Jira)]

Naveen Gangam created HIVE-24405:


 Summary: Missing datatype for table column in oracle
 Key: HIVE-24405
 URL: https://issues.apache.org/jira/browse/HIVE-24405
 Project: Hive
  Issue Type: Sub-task
  Components: Hive
Reporter: Naveen Gangam
Assignee: Naveen Gangam


The parent change introduces an issue in the oracle schema script.  No datatype 
is specified.
{noformat}
1 row created.

  CQ_COMMIT_TIME(19)
*
ERROR at line 19:
ORA-00902: invalid datatype
{noformat}






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HIVE-24396) [New Feature] Add data connector support for remote datasources

[Naveen Gangam (Jira)]

Naveen Gangam created HIVE-24396:


 Summary: [New Feature] Add data connector support for remote 
datasources
 Key: HIVE-24396
 URL: https://issues.apache.org/jira/browse/HIVE-24396
 Project: Hive
  Issue Type: Improvement
  Components: Hive
Reporter: Naveen Gangam
Assignee: Naveen Gangam


This feature work is to be able to support in Hive Metastore to be able to 
configure data connectors for remote datasources and map databases. We 
currently have support for remote tables via StorageHandlers like 
JDBCStorageHandler and HBaseStorageHandler.

Data connectors are a natural extension to this where we can map an entire 
database or catalogs instead of individual tables. The tables within are 
automagically mapped at runtime. The metadata for these tables are not 
persisted in Hive. They are always mapped and built at runtime. 

With this feature, we introduce a concept of type for Databases in Hive. NATIVE 
vs REMOTE. All current databases are NATIVE. To create a REMOTE database, the 
following syntax is to be used
CREATE REMOTE DATABASE remote_db USING  WITH DCPROPERTIES ();

Will attach a design doc to this jira. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HIVE-24348) Beeline: Isolating dependencies and execution with java

[Naveen Gangam (Jira)]

Naveen Gangam created HIVE-24348:


 Summary: Beeline: Isolating dependencies and execution with java
 Key: HIVE-24348
 URL: https://issues.apache.org/jira/browse/HIVE-24348
 Project: Hive
  Issue Type: Improvement
  Components: Beeline
Affects Versions: 3.1.0
Reporter: Naveen Gangam
Assignee: Naveen Gangam


Currently, beeline code, binaries and executables are somewhat tightly coupled 
with the hive product. To be able to execute beeline from a node with just JRE 
installed and some jars in classpath is impossible.
* beeline.sh/hive scripts rely on HADOOP_HOME to be set which are designed to 
use "hadoop" executable to run beeline.
* Ideally, just the hive-beeline.jar and hive-jdbc-standalone jars should be 
enough but sadly they arent. The latter jar adds more problems than it solves 
because all the classfiles are shaded some dependencies cannot be resolved.
* Beeline has many other dependencies like hive-exec, hive-common. 
hadoop-common, supercsv, jline, commons-cli, commons-io, commons-logging etc. 
While it may not be possible to eliminate some of these, we should atleast have 
a self-contains jar that contains all these to be able to make it work.
* the underlying script used to run beeline should use JAVA as an alternate 
means to execute if HADOOP_HOME is not set



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HIVE-24288) Files created by CompileProcessor have incorrect permissions

[Naveen Gangam (Jira)]

Naveen Gangam created HIVE-24288:


 Summary: Files created by CompileProcessor have incorrect 
permissions
 Key: HIVE-24288
 URL: https://issues.apache.org/jira/browse/HIVE-24288
 Project: Hive
  Issue Type: Bug
  Components: Hive
Affects Versions: 3.1.0
Reporter: Naveen Gangam
Assignee: Naveen Gangam


Compile processor generates some temporary files as part of processing.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HIVE-24271) Create managed table relies on hive.create.as.acid settings.

[Naveen Gangam (Jira)]

Naveen Gangam created HIVE-24271:


 Summary: Create managed table relies on hive.create.as.acid 
settings.
 Key: HIVE-24271
 URL: https://issues.apache.org/jira/browse/HIVE-24271
 Project: Hive
  Issue Type: Sub-task
  Components: HiveServer2
Affects Versions: 4.0.0
Reporter: Naveen Gangam
Assignee: Naveen Gangam


0: jdbc:hive2://ngangam-3.ngangam.root.hwx.si> set hive.create.as.acid;
++
|set |
++
| hive.create.as.acid=false  |
++
1 row selected (0.018 seconds)
0: jdbc:hive2://ngangam-3.ngangam.root.hwx.si> set hive.create.as.insert.only;
+---+
|set|
+---+
| hive.create.as.insert.only=false  |
+---+
1 row selected (0.013 seconds)
0: jdbc:hive2://ngangam-3.ngangam.root.hwx.si> create managed table mgd_table(a 
int);
INFO  : Compiling 
command(queryId=hive_20201014053526_9ba1ffa3-3aa2-47c3-8514-1fe58fe4f140): 
create managed table mgd_table(a int)
INFO  : Semantic Analysis Completed (retrial = false)
INFO  : Created Hive schema: Schema(fieldSchemas:null, properties:null)
INFO  : Completed compiling 
command(queryId=hive_20201014053526_9ba1ffa3-3aa2-47c3-8514-1fe58fe4f140); Time 
taken: 0.021 seconds
INFO  : Executing 
command(queryId=hive_20201014053526_9ba1ffa3-3aa2-47c3-8514-1fe58fe4f140): 
create managed table mgd_table(a int)
INFO  : Starting task [Stage-0:DDL] in serial mode
INFO  : Completed executing 
command(queryId=hive_20201014053526_9ba1ffa3-3aa2-47c3-8514-1fe58fe4f140); Time 
taken: 0.048 seconds
INFO  : OK
No rows affected (0.107 seconds)
0: jdbc:hive2://ngangam-3.ngangam.root.hwx.si> describe formatted mgd_table;
INFO  : Compiling 
command(queryId=hive_20201014053533_8919be7d-41b0-41e5-b9eb-847801a9d8c5): 
describe formatted mgd_table
INFO  : Semantic Analysis Completed (retrial = false)
INFO  : Created Hive schema: Schema(fieldSchemas:[FieldSchema(name:col_name, 
type:string, comment:from deserializer), FieldSchema(name:data_type, 
type:string, comment:from deserializer), FieldSchema(name:comment, type:string, 
comment:from deserializer)], properties:null)
INFO  : Completed compiling 
command(queryId=hive_20201014053533_8919be7d-41b0-41e5-b9eb-847801a9d8c5); Time 
taken: 0.037 seconds
INFO  : Executing 
command(queryId=hive_20201014053533_8919be7d-41b0-41e5-b9eb-847801a9d8c5): 
describe formatted mgd_table
INFO  : Starting task [Stage-0:DDL] in serial mode
INFO  : Completed executing 
command(queryId=hive_20201014053533_8919be7d-41b0-41e5-b9eb-847801a9d8c5); Time 
taken: 0.03 seconds
INFO  : OK
+---+++
|   col_name| data_type 
 |  comment   |
+---+++
| a | int   
 ||
|   | NULL  
 | NULL   |
| # Detailed Table Information  | NULL  
 | NULL   |
| Database: | bothfalseonhs2
 | NULL   |
| OwnerType:| USER  
 | NULL   |
| Owner:| hive  
 | NULL   |
| CreateTime:   | Wed Oct 14 05:35:26 UTC 2020  
 | NULL   |
| LastAccessTime:   | UNKNOWN   
 | NULL   |
| Retention:| 0 
 | NULL   |
| Location: | 
hdfs://ngangam-3.ngangam.root.hwx.site:8020/warehouse/tablespace/external/hive/bothfalseonhs2.db/mgd_table
 | NULL   |
| Table Type:   | EXTERNAL_TABLE
 | NULL   |
| Table Parameters: | NULL  

[jira] [Created] (HIVE-24175) Ease database managed location restrictions in HMS translation

[Naveen Gangam (Jira)]

Naveen Gangam created HIVE-24175:


 Summary: Ease database managed location restrictions in HMS 
translation
 Key: HIVE-24175
 URL: https://issues.apache.org/jira/browse/HIVE-24175
 Project: Hive
  Issue Type: Sub-task
Affects Versions: 4.0.0
Reporter: Naveen Gangam
Assignee: Naveen Gangam


Currently, HMS translation layer restricts the path of database's managed 
location to be within hive warehouse. so a getDatabase call will return a 
managedlocation path that adheres to this restriction regardless of what has 
been set in the HMS DB. This leads to issues like having inconsistent paths if 
hive-site.xml is not in sync across HMS and HS2 instances or even different HMS 
instances as each instance has a different version of warehouse root.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HIVE-24152) Comment out test until it is investigated.

[Naveen Gangam (Jira)]

Naveen Gangam created HIVE-24152:


 Summary: Comment out test until it is investigated.
 Key: HIVE-24152
 URL: https://issues.apache.org/jira/browse/HIVE-24152
 Project: Hive
  Issue Type: Sub-task
Affects Versions: 4.0.0
Reporter: Naveen Gangam
Assignee: Naveen Gangam


Looks like this test was re-enabled between the time the precommits were run 
and it was committed (a few hours later). This is blocking all other commits. 
Commenting it out for now



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HIVE-24086) CTAS with HMS translation enabled returns empty results.

[Naveen Gangam (Jira)]

Naveen Gangam created HIVE-24086:


 Summary: CTAS with HMS translation enabled returns empty results.
 Key: HIVE-24086
 URL: https://issues.apache.org/jira/browse/HIVE-24086
 Project: Hive
  Issue Type: Bug
  Components: Hive
Reporter: Naveen Gangam
Assignee: Naveen Gangam


when you execute something like 
create table ctas_table as select * from mgd_table;

if mgd_table is a managed table, the hive query planner creates a plan with 
ctas_table as a managed table, so the location is set to something in the 
managed warehouse directory.

However with HMS translation enabled, non-acid MANAGED tables are converted to 
EXTERNAL with purge set to true. So the table location for this table is 
altered to be in the external warehouse directory.
But after the table creation, the rest of the query executes but the data is 
copied to the location set in the query plan. As a result when you execute a 
select from ctas_table, it will not return any results because that location is 
empty.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HIVE-24076) MetastoreDirectSql.getDatabase() needs a space in the query

[Naveen Gangam (Jira)]

Naveen Gangam created HIVE-24076:


 Summary: MetastoreDirectSql.getDatabase() needs a space in the 
query
 Key: HIVE-24076
 URL: https://issues.apache.org/jira/browse/HIVE-24076
 Project: Hive
  Issue Type: Sub-task
  Components: Hive
Affects Versions: 4.0.0
Reporter: Naveen Gangam
Assignee: Naveen Gangam


String queryTextDbSelector= "select "
  + "\"DB_ID\", \"NAME\", \"DB_LOCATION_URI\", \"DESC\", "
  + "\"OWNER_NAME\", \"OWNER_TYPE\", \"CTLG_NAME\" , \"CREATE_TIME\", 
\"DB_MANAGED_LOCATION_URI\""
  + "FROM "+ DBS

There needs to be a space before FROM so the query is right. Currently it falls 
back to JDO, so not lapse in functionality.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HIVE-23970) Reject database creation if managedlocation is incorrect

[Naveen Gangam (Jira)]

Naveen Gangam created HIVE-23970:


 Summary: Reject database creation if managedlocation is incorrect
 Key: HIVE-23970
 URL: https://issues.apache.org/jira/browse/HIVE-23970
 Project: Hive
  Issue Type: Sub-task
  Components: Hive
Affects Versions: 4.0.0
Reporter: Naveen Gangam
Assignee: Naveen Gangam


With some changes in HIVE-23387, managed location check gets bypassed. Need to 
be fixed.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HIVE-23603) transformDatabase() should work with changes from HIVE-22995

[Naveen Gangam (Jira)]

Naveen Gangam created HIVE-23603:


 Summary: transformDatabase() should work with changes from 
HIVE-22995
 Key: HIVE-23603
 URL: https://issues.apache.org/jira/browse/HIVE-23603
 Project: Hive
  Issue Type: Sub-task
  Components: Hive
Reporter: Naveen Gangam
Assignee: Naveen Gangam
 Fix For: 4.0.0


The translation layer alters the locationUri on Database based on the 
capabilities of the client. Now that we have separate locations for managed and 
external for database, the implementation should be adjusted to work with both 
locations. locationUri could already be external location.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HIVE-23588) create table like tabletype should match source tabletype and proper location

[Naveen Gangam (Jira)]

Naveen Gangam created HIVE-23588:


 Summary: create table like tabletype should match source tabletype 
and proper location
 Key: HIVE-23588
 URL: https://issues.apache.org/jira/browse/HIVE-23588
 Project: Hive
  Issue Type: Sub-task
  Components: Hive
Affects Versions: 4.0.0
Reporter: Naveen Gangam
Assignee: Naveen Gangam






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HIVE-23562) Upgrade thrift version in hive

[Naveen Gangam (Jira)]

Naveen Gangam created HIVE-23562:


 Summary: Upgrade thrift version in hive
 Key: HIVE-23562
 URL: https://issues.apache.org/jira/browse/HIVE-23562
 Project: Hive
  Issue Type: Improvement
  Components: Hive
Affects Versions: 4.0.0
Reporter: Naveen Gangam


Hive has been using thrift 0.9.3 for a long time. We might be able to avail new 
features like deprecation support etc in the newer releases of thrift. But this 
impacts interoperability between older clients and newer servers. We need to 
assess what can break atleast for the purposes of documenting before we make 
this change.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HIVE-23435) Full outer join result is missing rows

[Naveen Gangam (Jira)]

Naveen Gangam created HIVE-23435:


 Summary: Full outer join result is missing rows 
 Key: HIVE-23435
 URL: https://issues.apache.org/jira/browse/HIVE-23435
 Project: Hive
  Issue Type: Bug
  Components: HiveServer2
Affects Versions: 3.1.0
Reporter: Naveen Gangam
Assignee: Jesus Camacho Rodriguez


Full Outer join result has missing rows. Appears to be a bug with the full 
outer join logic. Expected output is receiving when we do a left and right 
outer join.

Reproducible steps are mentioned below.

~~

SUPPORT ANALYSIS

Steps to Reproduce:

1. Create a table and insert data:

create table x (z char(5), x int, y int);

insert into x values ('one', 1, 50),
('two', 2, 30),
('three', 3, 30),
('four', 4, 60),
('five', 5, 70),
('six', 6, 80);

2. Try full outer with the below command. The result is incomplete, it is 
missing the row:

NULLNULLNULLthree   3   30.0
Full Outer Join:

select x1.`z`, x1.`x`, x1.`y`, x2.`z`,
x2.`x`, x2.`y`
from `x` x1 full outer join
`x` x2 on (x1.`x` > 3) and (x2.`x` < 4) and (x1.`x` =
x2.`x`);

Result:

--+

x1.zx1.xx1.yx2.zx2.xx2.y
--+

one 1   50  NULLNULLNULL
NULLNULLNULLone 1   50
two 2   30  NULLNULLNULL
NULLNULLNULLtwo 2   30
three   3   30  NULLNULLNULL
four4   60  NULLNULLNULL
NULLNULLNULLfour4   60
five5   70  NULLNULLNULL
NULLNULLNULLfive5   70
six 6   80  NULLNULLNULL
NULLNULLNULLsix 6   80
--+

3. Expected output is coming when we use left/right join + union:

select x1.`z`, x1.`x`, x1.`y`, x2.`z`,
x2.`x`, x2.`y`
from `x` x1 left outer join
`x` x2 on (x1.`x` > 3) and (x2.`x` < 4) and (x1.`x` =
x2.`x`)
union
select x1.`z`, x1.`x`, x1.`y`, x2.`z`,
x2.`x`, x2.`y`
from `x` x1 right outer join
`x` x2 on (x1.`x` > 3) and (x2.`x` < 4) and (x1.`x` =
x2.`x`);

Result:

+

z   x   y   _col3   _col4   _col5
+

NULLNULLNULLfive5   70
NULLNULLNULLfour4   60
NULLNULLNULLone 1   50
four4   60  NULLNULLNULL
one 1   50  NULLNULLNULL
six 6   80  NULLNULLNULL
three   3   30  NULLNULLNULL
two 2   30  NULLNULLNULL
NULLNULLNULLsix 6   80
NULLNULLNULLthree   3   30
NULLNULLNULLtwo 2   30
five5   70  NULLNULLNULL
+

~~

EXPECTED ENGINEERING ACTION

Confirm this is a bug. If so, any work around or just use left+right outer join.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HIVE-23388) CTAS queries should use target's location for staging.

[Naveen Gangam (Jira)]

Naveen Gangam created HIVE-23388:


 Summary: CTAS queries should use target's location for staging.
 Key: HIVE-23388
 URL: https://issues.apache.org/jira/browse/HIVE-23388
 Project: Hive
  Issue Type: Sub-task
  Components: HiveServer2
Affects Versions: 4.0.0
Reporter: Naveen Gangam
Assignee: Naveen Gangam


In cloud based storage systems, renaming files across different root level 
buckets seem to be disallowed. The S3AFileSystem throws the following 
exception. This appears to be bug in S3FS impl.

Failed with exception Wrong FS 
s3a://hive-managed/clusters/env-x/warehouse--/warehouse/tablespace/managed/hive/tpch.db/customer/delta_001_001_
 -expected s3a://hive-external
2020-04-27T19:34:27,573 INFO  [Thread-6] jdbc.TestDriver: 
java.lang.IllegalArgumentException: Wrong FS 
s3a://hive-managed//clusters/env-/warehouse--/warehouse/tablespace/managed/hive/tpch.db/customer/delta_001_001_
 -expected s3a://hive-external

But we should fix our query plans to use the target table's directory for 
staging as well. That should resolve this issue and it is the right thing to do 
as well (in case there are different encryption zones/keys for these buckets).

Fix in HIVE-22995 probably changed this behavior.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HIVE-23387) Flip the Warehouse.getDefaultTablePath() to return path from ext warehouse

[Naveen Gangam (Jira)]

Naveen Gangam created HIVE-23387:


 Summary: Flip the Warehouse.getDefaultTablePath() to return path 
from ext warehouse
 Key: HIVE-23387
 URL: https://issues.apache.org/jira/browse/HIVE-23387
 Project: Hive
  Issue Type: Sub-task
  Components: Hive
Affects Versions: 4.0.0
Reporter: Naveen Gangam
Assignee: Naveen Gangam


For backward compatibility, initial fix returned path that was set on db. It 
could have been either from managed warehouse or external depending on what was 
set. There were tests relying on certain paths to be returned. This fix is to 
address the tests.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HIVE-23260) Add support for unmodified_metadata capability

[Naveen Gangam (Jira)]

Naveen Gangam created HIVE-23260:


 Summary: Add support for unmodified_metadata capability
 Key: HIVE-23260
 URL: https://issues.apache.org/jira/browse/HIVE-23260
 Project: Hive
  Issue Type: Sub-task
  Components: Hive
Affects Versions: 4.0.0
Reporter: Naveen Gangam
Assignee: Naveen Gangam


Currently, the translator removes bucketing info for tables for clients that do 
not possess the HIVEBUCKET2 capability. While this is desirable, some clients 
that have write access to these tables can turn around overwrite the metadata 
thus corrupting original bucketing info.

So adding support for a capability for client that are capable of interpreting 
the original metadata would prevent such corruption.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HIVE-23192) "default" database locationUri should be external warehouse root.

[Naveen Gangam (Jira)]

Naveen Gangam created HIVE-23192:


 Summary: "default" database locationUri should be external 
warehouse root.
 Key: HIVE-23192
 URL: https://issues.apache.org/jira/browse/HIVE-23192
 Project: Hive
  Issue Type: Sub-task
  Components: Hive
Affects Versions: 4.0.0
Reporter: Naveen Gangam
Assignee: Naveen Gangam


When creating the default database, the database locationUri should be set to 
external warehouse.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HIVE-23121) Re-examine TestWarehouseExternalDir to see if it uses HMS translation.

[Naveen Gangam (Jira)]

Naveen Gangam created HIVE-23121:


 Summary: Re-examine TestWarehouseExternalDir to see if it uses HMS 
translation.
 Key: HIVE-23121
 URL: https://issues.apache.org/jira/browse/HIVE-23121
 Project: Hive
  Issue Type: Sub-task
Reporter: Naveen Gangam
Assignee: Naveen Gangam


TestWarehouseExternalDir currently passes with just one change related to 
HIVE-22995. But that change was assuming it was using HMS Translation to 
convert non-acid managed table to external. 
Ensure that it still does.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HIVE-22995) Add support for location for managed tables on database

[Naveen Gangam (Jira)]

Naveen Gangam created HIVE-22995:


 Summary: Add support for location for managed tables on database
 Key: HIVE-22995
 URL: https://issues.apache.org/jira/browse/HIVE-22995
 Project: Hive
  Issue Type: Improvement
  Components: Hive
Affects Versions: 3.1.0
Reporter: Naveen Gangam
Assignee: Naveen Gangam
 Attachments: Hive Metastore Support for Tenant-based storage 
heirarchy.pdf

I have attached the initial spec to this jira.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HIVE-22874) Beeline unable to use credentials from URL.

[Naveen Gangam (Jira)]

Naveen Gangam created HIVE-22874:


 Summary: Beeline unable to use credentials from URL.
 Key: HIVE-22874
 URL: https://issues.apache.org/jira/browse/HIVE-22874
 Project: Hive
  Issue Type: Bug
  Components: Beeline
Reporter: Naveen Gangam
Assignee: Naveen Gangam
 Fix For: 4.0.0


Beeline is not using password value from the URL. 
Using LDAP Auth in this case, so the failure is on connect.
bin/beeline -u "jdbc:hive2://localhost:1/default;user=test1;password=test1" 

On the server side in LdapAuthenticator, the principals come out to (via a 
special debug logging)

2020-02-11T11:10:31,613  INFO [HiveServer2-Handler-Pool: Thread-67] 
auth.LdapAuthenticationProviderImpl: Connecting to ldap as 
user/password:test1:anonymous


This bug may have been introduced via
https://github.com/apache/hive/commit/749e831060381a8ae4775630efb72d5cd040652f

pass = "" ( an empty string on this line) 
https://github.com/apache/hive/blob/master/beeline/src/java/org/apache/hive/beeline/BeeLine.java#L848

but on this line of code, it checks to see it is null which will not be true 
and hence it never picks up from the jdbc url
https://github.com/apache/hive/blob/master/beeline/src/java/org/apache/hive/beeline/BeeLine.java#L900
It has another chance here but pass != null will always be true and never goes 
into the else condition.
https://github.com/apache/hive/blob/master/beeline/src/java/org/apache/hive/beeline/BeeLine.java#L909



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HIVE-22853) Beeline should use HS2 server defaults for fetchSize

[Naveen Gangam (Jira)]

Naveen Gangam created HIVE-22853:


 Summary: Beeline should use HS2 server defaults for fetchSize
 Key: HIVE-22853
 URL: https://issues.apache.org/jira/browse/HIVE-22853
 Project: Hive
  Issue Type: Bug
  Components: Beeline
Affects Versions: 4.0.0
Reporter: Naveen Gangam
Assignee: Naveen Gangam


Currently beeline uses a hard coded default of 1000 rows for fetchSize. This 
default value is different from what the server has set. While the beeline user 
can reset the value via set command, its cumbersome to change the workloads.
Rather it should default to the server-side value and set should be used to 
override within the session.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HIVE-22794) Disallow ACID table location outside hive warehouse

[Naveen Gangam (Jira)]

Naveen Gangam created HIVE-22794:


 Summary: Disallow ACID table location outside hive warehouse
 Key: HIVE-22794
 URL: https://issues.apache.org/jira/browse/HIVE-22794
 Project: Hive
  Issue Type: Sub-task
  Components: Hive
Affects Versions: 3.1.0
Reporter: Naveen Gangam
Assignee: Naveen Gangam


The co-location of managed tables enables hive to govern them effectively, 
using common policies for security, S3Guard, support quotas etc.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[VOTE] Apache Hive 3.1.3 Release Candidate 0

[Naveen Gangam]

Apache Hive 3.1.3 Release Candidate 0 is available here:
https://people.apache.org/~ngangam/hive-3.1.3-rc-0


Maven artifacts are available
here:https://repository.apache.org/content/repositories/orgapachehive-1099/


The tag release-3.1.3-rc0 has been applied to the source for this
release in github, you can see it
athttps://github.com/apache/hive/tree/release-3.1.3-rc0

Voting will conclude in 72 hours.

Hive PMC Members: Please test and vote.

Thanks.

[jira] [Created] (HIVE-22708) To be updated later

[Naveen Gangam (Jira)]

Naveen Gangam created HIVE-22708:


 Summary: To be updated later
 Key: HIVE-22708
 URL: https://issues.apache.org/jira/browse/HIVE-22708
 Project: Hive
  Issue Type: Bug
  Components: HiveServer2
Affects Versions: 4.0.0
Reporter: Naveen Gangam
Assignee: Naveen Gangam






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HIVE-22498) Schema tool enhancements to merge catalogs

[Naveen Gangam (Jira)]

Naveen Gangam created HIVE-22498:


 Summary: Schema tool enhancements to merge catalogs
 Key: HIVE-22498
 URL: https://issues.apache.org/jira/browse/HIVE-22498
 Project: Hive
  Issue Type: Sub-task
Reporter: Naveen Gangam
Assignee: Naveen Gangam


Schema tool currently supports relocation of database from one catalog to 
another, one at a time. While having to do this one at a time is painful, it 
also lacks support for converting them to external tables during migration, in 
lieu of the changes to the translation layer where a MANAGED table is strictly 
ACID-only table.
Hence we also need to convert them to external tables during relocation.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HIVE-22497) Remove default value for Capabilities from HiveConf

[Naveen Gangam (Jira)]

Naveen Gangam created HIVE-22497:


 Summary: Remove default value for Capabilities from HiveConf
 Key: HIVE-22497
 URL: https://issues.apache.org/jira/browse/HIVE-22497
 Project: Hive
  Issue Type: Sub-task
Reporter: Naveen Gangam


This class is used and bundled in other jars that 3rd party connectors like 
teradata etc. So it would be good to remove this default value from HiveConf 
but rely on it being set in hive-site.xml instead. The HiveServer2 should still 
set this as part of HS2 initialization or via hiveserver2-site.xml



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HIVE-22406) TRUNCATE TABLE fails due MySQL limitations on limit value

[Naveen Gangam (Jira)]

Naveen Gangam created HIVE-22406:


 Summary: TRUNCATE TABLE fails due MySQL limitations on limit value
 Key: HIVE-22406
 URL: https://issues.apache.org/jira/browse/HIVE-22406
 Project: Hive
  Issue Type: Bug
Reporter: Naveen Gangam


HMS currently has some APIs that accepts an integer limit value. Prior to the 
change in HIVE-21734, HMS was silently converting this int to short and thus we 
havent seen this issue. But semantically, its incorrect to do so quietly.

{noformat}
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_191]
Caused by: java.sql.SQLException: setMaxRows() out of range. 2147483647 > 
5000.
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:996) ~[mysql-
connector-java.jar:5.1.33]
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:935) ~[mysql-
connector-java.jar:5.1.33]
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:924) ~[mysql-
connector-java.jar:5.1.33]
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:870) ~[mysql-
connector-java.jar:5.1.33]
at com.mysql.jdbc.StatementImpl.setMaxRows(StatementImpl.java:2525) ~[mysql-
connector-java.jar:5.1.33]
at 
com.zaxxer.hikari.pool.HikariProxyPreparedStatement.setMaxRows(HikariProxyPreparedS
tatement.java) ~[HikariCP-2.6.1.jar:?]
{noformat}

We cannot change the RawStore api to accept shorts instead of ints. 
So we have to fix the caller to use a lower limit instead of Integer.MAX_VALUE.


{noformat}
Caused by: org.apache.hadoop.hive.metastore.api.MetaException: Exception thrown 
when executing query : SELECT DISTINCT 
'org.apache.hadoop.hive.metastore.model.MPartition' AS 
`NUCLEUS_TYPE`,`A0`.`CREATE_TIME`,`A0`.`LAST_ACCESS_TIME`,`A0`.`PART_NAME`,`A0`.`WRITE_ID`,`A0`.`PART_ID`,`A0`.`PART_NAME`
 AS `NUCORDER0` FROM `PARTITIONS` `A0` LEFT OUTER JOIN `TBLS` `B0` ON 
`A0`.`TBL_ID` = `B0`.`TBL_ID` LEFT OUTER JOIN `DBS` `C0` ON `B0`.`DB_ID` = 
`C0`.`DB_ID` WHERE `B0`.`TBL_NAME` = ? AND `C0`.`NAME` = ? AND `C0`.`CTLG_NAME` 
= ? ORDER BY `NUCORDER0` LIMIT 0,2147483647
at 
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$truncate_table_req_result$truncate_table_req_resultStandardScheme.read(ThriftHiveMetastore.java)
 ~[hive-exec-3.1.0.3.1.5.0-17.jar:3.1.0.3.1.5.0-17]
at 
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$truncate_table_req_result$truncate_table_req_resultStandardScheme.read(ThriftHiveMetastore.java)
 ~[hive-exec-3.1.0.3.1.5.0-17.jar:3.1.0.3.1.5.0-17]
at 
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$truncate_table_req_result.read(ThriftHiveMetastore.java)
 ~[hive-exec-3.1.0.3.1.5.0-17.jar:3.1.0.3.1.5.0-17]
at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:86) 
~[hive-exec-3.1.0.3.1.5.0-17.jar:3.1.0.3.1.5.0-17]
at 
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_truncate_table_req(ThriftHiveMetastore.java:1999)
 ~[hive-exec-3.1.0.3.1.5.0-17.jar:3.1.0.3.1.5.0-17]
at 
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.truncate_table_req(ThriftHiveMetastore.java:1986)
 ~[hive-exec-3.1.0.3.1.5.0-17.jar:3.1.0.3.1.5.0-17]
at 
org.apache.hadoop.hive.metastore.HiveMetaStoreClient.truncateTableInternal(HiveMetaStoreClient.java:1450)
 ~[hive-exec-3.1.0.3.1.5.0-17.jar:3.1.0.3.1.5.0-17]
at 
org.apache.hadoop.hive.metastore.HiveMetaStoreClient.truncateTable(HiveMetaStoreClient.java:1427)
 ~[hive-exec-3.1.0.3.1.5.0-17.jar:3.1.0.3.1.5.0-17]
at 
org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.truncateTable(SessionHiveMetaStoreClient.java:171)
 ~[hive-exec-3.1.0.3.1.5.0-17.jar:3.1.0.3.1.5.0-17]
at sun.reflect.GeneratedMethodAccessor112.invoke(Unknown Source) ~[?:?]
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 ~[?:1.8.0_191]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_191]
at 
org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:212)
 ~[hive-exec-3.1.0.3.1.5.0-17.jar:3.1.0.3.1.5.0-17]
at com.sun.proxy.$Proxy59.truncateTable(Unknown Source) ~[?:?]
at sun.reflect.GeneratedMethodAccessor112.invoke(Unknown Source) ~[?:?]
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 ~[?:1.8.0_191]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_191]
at 
org.apache.hadoop.hive.metastore.HiveMetaStoreClient$SynchronizedHandler.invoke(HiveMetaStoreClient.java:3122)
 ~[hive-exec-3.1.0.3.1.5.0-17.jar:3.1.0.3.1.5.0-17]
at com.sun.proxy.$Proxy59.truncateTable(Unknown Source) ~[?:?]
at 
org.apache.hadoop.hive.ql.metadata.Hive.truncateTable(Hive.java:1277) 
~[hive-exec-3.1.0.3.1.5.0-17.jar:3.1.0.3.1.5.0-17]
at 
org.apache.hadoop.hive.ql.exec.DDLTask.truncateTable(DDLTask.java:5111) 
~[hive-exec-3.1.0.3

[jira] [Created] (HIVE-22342) HMS Translation: HIVE-22189 too strict with location for EXTERNAL tables

[Naveen Gangam (Jira)]

Naveen Gangam created HIVE-22342:


 Summary: HMS Translation: HIVE-22189 too strict with location for 
EXTERNAL tables
 Key: HIVE-22342
 URL: https://issues.apache.org/jira/browse/HIVE-22342
 Project: Hive
  Issue Type: Sub-task
  Components: Hive
Affects Versions: 4.0.0
Reporter: Naveen Gangam
Assignee: Naveen Gangam


HIVE-22189 restricts EXTERNAL tables being created to be restricted to the 
EXTERNAL_WAREHOUSE_DIR. This might be too strict as any other location should 
be allowed as long as the location is outside the MANAGED warehouse directory.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HIVE-22291) HMS Translation: Limit translation to hive default catalog only

[Naveen Gangam (Jira)]

Naveen Gangam created HIVE-22291:


 Summary: HMS Translation: Limit translation to hive default 
catalog only
 Key: HIVE-22291
 URL: https://issues.apache.org/jira/browse/HIVE-22291
 Project: Hive
  Issue Type: Sub-task
  Components: Hive
Affects Versions: 4.0.0
Reporter: Naveen Gangam
Assignee: Naveen Gangam


HMS Translation should only be limited to a single catalog.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HIVE-22266) Addendum fix to have HS2 pom add explicit curator dependency

[Naveen Gangam (Jira)]

Naveen Gangam created HIVE-22266:


 Summary: Addendum fix to have HS2 pom add explicit curator 
dependency
 Key: HIVE-22266
 URL: https://issues.apache.org/jira/browse/HIVE-22266
 Project: Hive
  Issue Type: Sub-task
  Components: HiveServer2
Affects Versions: 4.0.0
Reporter: Naveen Gangam
Assignee: Naveen Gangam


It might be better to add an explicit dependency on apache-curator in the 
service/pom.xml.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HIVE-22205) Upgrade zookeeper and curator versions

[Naveen Gangam (Jira)]

Naveen Gangam created HIVE-22205:


 Summary: Upgrade zookeeper and curator versions
 Key: HIVE-22205
 URL: https://issues.apache.org/jira/browse/HIVE-22205
 Project: Hive
  Issue Type: Improvement
  Components: HiveServer2
Affects Versions: 4.0.0
Reporter: Naveen Gangam
Assignee: Naveen Gangam


Other components like hadoop have switched to using new ZK versions. So these 
jars end up in classpath for hive services and could cause issues due to 
in-compatible curator versions that hive uses.

So it makes sense for hive to upgrade the ZK and curator versions to try to 
keep up.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Created] (HIVE-22189) HMS Translation: Enforce strict locations for managed vs external tables.

[Naveen Gangam (Jira)]

Naveen Gangam created HIVE-22189:


 Summary: HMS Translation: Enforce strict locations for managed vs 
external tables.
 Key: HIVE-22189
 URL: https://issues.apache.org/jira/browse/HIVE-22189
 Project: Hive
  Issue Type: Sub-task
  Components: Hive
Affects Versions: 4.0.0
Reporter: Naveen Gangam
Assignee: Naveen Gangam


Currently, HMS allows flexibility with location of a table. External tables can 
be located within Hive managed warehouse space and managed tables can be 
located within the external warehouse directory if the user chooses to do so.

There are certain advantages to restrict such flexibility. We could have 
different encryption policies for different warehouses, different replication 
policies etc.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Created] (HIVE-22158) HMS Translation layer - Disallow non-ACID MANAGED tables.

[Naveen Gangam (Jira)]

Naveen Gangam created HIVE-22158:


 Summary: HMS Translation layer - Disallow non-ACID MANAGED tables.
 Key: HIVE-22158
 URL: https://issues.apache.org/jira/browse/HIVE-22158
 Project: Hive
  Issue Type: Sub-task
  Components: Hive
Affects Versions: 4.0.0
Reporter: Naveen Gangam
Assignee: Naveen Gangam


In the recent commits, we have allowed non-ACID MANAGED tables to be created by 
clients that have some form of ACID WRITE capabilities. 
I think it would make sense to disallow this entirely. MANAGED tables should be 
ACID tables only.




--
This message was sent by Atlassian Jira
(v8.3.2#803003)

[jira] [Created] (HIVE-22159) HMS Translation layer - Turn off HMS Translation by default.

[Naveen Gangam (Jira)]

Naveen Gangam created HIVE-22159:


 Summary: HMS Translation layer - Turn off HMS Translation by 
default.
 Key: HIVE-22159
 URL: https://issues.apache.org/jira/browse/HIVE-22159
 Project: Hive
  Issue Type: Sub-task
  Components: Hive
Affects Versions: 4.0.0
Reporter: Naveen Gangam
Assignee: Naveen Gangam


Because of certain backward incompatibilities in terms of behavior, I think it 
makes sense to turn off this translation in the Apache Hive codebase.
Consumers can selectively enable it and even plugin their own set of 
translation rules as well.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

Re: Review Request 71267: HIVE-22087: Transform Database object on getDatabase() to return location based on client capabilities.

[Naveen Gangam via Review Board]

> On Aug. 14, 2019, 10:40 p.m., Thejas Nair wrote:
> > standalone-metastore/metastore-common/src/main/thrift/hive_metastore.thrift
> > Lines 1938 (patched)
> > <https://reviews.apache.org/r/71267/diff/1-2/?file=2160426#file2160426line1938>
> >
> > To future proof that, it would have better better to have a 
> > GetDatabaseResponse as well
> > similar to get_catalog.
> > That can be a smaller follo wup patch
> 
> Naveen Gangam wrote:
> will create a separate jira for this.

Created HIVE-22123 for the followup.


- Naveen


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71267/#review217212
-------


On Aug. 14, 2019, 6:26 a.m., Naveen Gangam wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/71267/
> ---
> 
> (Updated Aug. 14, 2019, 6:26 a.m.)
> 
> 
> Review request for hive, Daniel Dai, Jason Dere, and Thejas Nair.
> 
> 
> Bugs: HIVE-22087
> https://issues.apache.org/jira/browse/HIVE-22087
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> 1) getDatabase() calls should be transformed to return a Database object that 
> can vary in location depending on the client capabilities. If client has 
> ACID*WRITE* capabilities, location is unaltered. If the client does not have 
> such capabilities, the database will return an location from the external 
> warehouse directory.
> 2) When a non-ACID MANAGED table is translated to EXTERNAL table, its 
> location should be altered to point to an external warehouse directory and 
> not to the managed warehouse.
> 3) Some new test cases.
> 
> 
> Diffs
> -
> 
>   
> itests/hive-unit/src/test/java/org/apache/hadoop/hive/metastore/TestHiveMetastoreTransformer.java
>  e50b577ff7 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/AlterPartitionsRequest.java
>  6453c93d79 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/CreateTableRequest.java
>  5d42a80373 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/FindSchemasByColsResp.java
>  4024751ed3 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/GetDatabaseRequest.java
>  PRE-CREATION 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/GetPartitionsFilterSpec.java
>  fcba6ebb4d 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/GetPartitionsProjectionSpec.java
>  d94cbb1bcc 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/GetPartitionsRequest.java
>  dd4bf8339a 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/GetPartitionsResponse.java
>  ddfa59fb1c 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/RenamePartitionRequest.java
>  de467c298f 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/SchemaVersion.java
>  09fcd476e9 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/ThriftHiveMetastore.java
>  6b117291a6 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/WMFullResourcePlan.java
>  080111d85b 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/WMGetAllResourcePlanResponse.java
>  d0174005ca 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/WMGetTriggersForResourePlanResponse.java
>  e5425909d4 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/WMValidateResourcePlanResponse.java
>  b12c2284a2 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-php/metastore/ThriftHiveMetastore.php
>  4623e9ab5f 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-php/metastore/Types.php
>  0d45371b88 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-py/hive_metastore/ThriftHiveMetastore-remote
>  647c762acd 
>