[jira] [Created] (HIVE-23675) WebHcat: java level deadlock in hcat in presence of InMemoryJAAS

[Rajkumar Singh (Jira)]

Rajkumar Singh created HIVE-23675:
-

 Summary: WebHcat: java level deadlock in hcat in presence of 
InMemoryJAAS
 Key: HIVE-23675
 URL: https://issues.apache.org/jira/browse/HIVE-23675
 Project: Hive
  Issue Type: Improvement
Reporter: Rajkumar Singh


ENV: Keberos/SPNEGO enabled

set hive.exec.post.hook;
org.apache.hadoop.hive.ql.hooks.ATSHook,org.apache.atlas.hive.hook.HiveHook

ATLAS Hook use InMemoryJAASConfiguration

This is a sequence of the event while issue reproduces:

WebHcat -> hcat -> Hive Driver -> post hook execution create ATSHook  -> hook 
start the spnego auth and stuck while finding InMemoryJAASConfiguration used by 
the AtlasHook (this happens in separate thread ATS Logger)

Hcat jstack
{code:java}
Found one Java-level deadlock:
 =
 "ATS Logger 0":
   waiting to lock monitor 0x7efdc8003a38 (object 0xf3fcfe28, a 
org.apache.atlas.plugin.classloader.AtlasPluginClassLoader),
   which is held by "main"
 "main":
   waiting to lock monitor 0x7efdc8003da8 (object 0xc0050d40, a 
org.apache.hadoop.hive.ql.exec.UDFClassLoader),
   which is held by "ATS Logger 0"

Java stack information for the threads listed above:
 ===
 "ATS Logger 0":
 at 
org.apache.atlas.security.InMemoryJAASConfiguration.getAppConfigurationEntry(InMemoryJAASConfiguration.java:238)
 at 
sun.security.jgss.LoginConfigImpl.getAppConfigurationEntry(LoginConfigImpl.java:145)
 at javax.security.auth.login.LoginContext.init(LoginContext.java:251)
 at javax.security.auth.login.LoginContext.(LoginContext.java:512)
 at sun.security.jgss.GSSUtil.login(GSSUtil.java:256)
 at sun.security.jgss.krb5.Krb5Util.getTicket(Krb5Util.java:158)
 at 
sun.security.jgss.krb5.Krb5InitCredential$1.run(Krb5InitCredential.java:335)
 at 
sun.security.jgss.krb5.Krb5InitCredential$1.run(Krb5InitCredential.java:331)
 at java.security.AccessController.doPrivileged(Native Method)
 at 
sun.security.jgss.krb5.Krb5InitCredential.getTgt(Krb5InitCredential.java:330)
 at 
sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredential.java:145)
 at 
sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:122)
 at 
sun.security.jgss.krb5.Krb5MechFactory.getMechanismContext(Krb5MechFactory.java:187)
 at 
sun.security.jgss.GSSManagerImpl.getMechanismContext(GSSManagerImpl.java:224)
 at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:212)
 at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179)
 at 
sun.security.jgss.spnego.SpNegoContext.GSS_initSecContext(SpNegoContext.java:882)
 at 
sun.security.jgss.spnego.SpNegoContext.initSecContext(SpNegoContext.java:317)
 at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:248)
 at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179)
 at 
sun.net.www.protocol.http.spnego.NegotiatorImpl.init(NegotiatorImpl.java:108)
 at 
sun.net.www.protocol.http.spnego.NegotiatorImpl.(NegotiatorImpl.java:117)
 at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
 at 
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
 at 
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
 at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
 at sun.net.www.protocol.http.Negotiator.getNegotiator(Negotiator.java:63)
 at 
sun.net.www.protocol.http.NegotiateAuthentication.isSupportedImpl(NegotiateAuthentication.java:130)
 - locked <0xf48c4d90> (a java.lang.Class for 
sun.net.www.protocol.http.NegotiateAuthentication)
 at 
sun.net.www.protocol.http.NegotiateAuthentication.isSupported(NegotiateAuthentication.java:102)
 - locked <0xc0050d40> (a 
org.apache.hadoop.hive.ql.exec.UDFClassLoader)
 at 
sun.net.www.protocol.http.AuthenticationHeader.parse(AuthenticationHeader.java:180)
 at 
sun.net.www.protocol.http.AuthenticationHeader.(AuthenticationHeader.java:126)
 at 
sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1660)
 - locked <0xf47b7298> (a 
sun.net.www.protocol.https.DelegateHttpsURLConnection)
 at 
sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1441)
 - locked <0xf47b7298> (a 
sun.net.www.protocol.https.DelegateHttpsURLConnection)
 at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:480)
 at 
sun.net.www.protocol.https.HttpsURLConnectionImpl.getResponseCode(HttpsURLConnectionImpl.java:338)
 at 
org.apache.hadoop.security.authentication.client.KerberosAuthenticator.authenticate(KerberosAuthenticator.java:191)
 at 
org.apache.hadoop.security.toke

Re: Fix backslash escaping

[Panos Garefalakis]

Hey Max,

Is there a ticket related to this PR?
Please reopen the PR to trigger the tests so that people can take a look.

Cheers,
Panagiotis

On Wed, Jun 10, 2020 at 8:47 PM Max Khon  wrote:

> Hello,
>
> Hive JDBC has serious issues with the backslash escaping, however this PR
> has been automatically closed due to no activity, Can someone take a look
> there:
> https://github.com/apache/hive/pull/872
>
> Max
>

Fix backslash escaping

[Max Khon]

Hello,

Hive JDBC has serious issues with the backslash escaping, however this PR
has been automatically closed due to no activity, Can someone take a look
there:
https://github.com/apache/hive/pull/872

Max

Re: Re-Running CI Tests

[David Mollitor]

Great, thanks!

What triggers these builds?  Do we need to add something to branch-2,
branch-3 projects to get this to trigger there?

On Wed, Jun 10, 2020 at 3:03 PM Zoltán Haindrich  wrote:

> its on cooldown...5 builds/day/branch
> http://34.66.156.144:8080/job/hive-precommit/job/PR-1082/
>
> On June 10, 2020 6:55:45 PM GMT+02:00, David Mollitor 
> wrote:
> >Zoltán,
> >
> >Even a PR against master is only running Travis.
> >
> >https://github.com/apache/hive/pull/1082
> >
> >On Wed, Jun 10, 2020 at 12:52 PM David Mollitor 
> >wrote:
> >
> >> Hey Zoltán,
> >>
> >> Also in regard to that PR, it only ran the travis build  "mvn clean
> >> install -DskipTests -q -Pitests"
> >>
> >> What does this branch require (and 3.x) to enable running of tests?
> >>
> >> Thanks.
> >>
> >>
> >>
> >> On Wed, Jun 10, 2020 at 11:36 AM David Mollitor 
> >wrote:
> >>
> >>> Hey Zoltán,
> >>>
> >>> Yes.  That is correct.  Community recently put out a 2.x release.
> >This
> >>> is in case someone wishes to release a new one.
> >>>
> >>> Does that have any bearing on re-running tests?
> >>>
> >>> Thanks.
> >>>
> >>> On Wed, Jun 10, 2020 at 11:32 AM Zoltán Haindrich 
> >wrote:
> >>>
>  That pr seems to be for branch-2 and not master
> 
>  On June 10, 2020 5:17:19 PM GMT+02:00, David Mollitor
> >
>  wrote:
> >
> > Zoltan,
> >
> > I just tried to close/re-open a PR and I don't believe it
> >triggered a
> > new CI run:
> >
> > https://github.com/apache/hive/pull/1076
> >
> > Thanks.
> >
> > On Wed, Jun 10, 2020 at 10:59 AM David Mollitor
> >
> > wrote:
> >
> >> Hey Zoltan,
> >>
> >> Can you please research a way to initiate it from the GitHub
> >> interface?  I have a strong feeling we're going to need such a
> >> capability regularly.
> >>
> >> Thanks.
> >>
> >> On Wed, Jun 10, 2020 at 9:29 AM Zoltan Haindrich 
> >wrote:
> >>
> >>> Hey
> >>>
> >>> you could:
> >>> * push new commits to the branch
> >>>- this will create a new merge with the current master
> >>> * login in to the jenkins instance: and launch a new build of
> >that PR
> >>> * close&re-open the pr: will re-emit the github event triggering
> >the
> >>> testrun
> >>> * login in to the jenkins instance: and press retry button
> >>>
> >>> I don't know if the last method (retrigger button) will create a
> >new
> >>> merge with the current master's HEAD or not - I suspect that it
> >doesn't.
> >>>
> >>> cheers,
> >>> Zoltan
> >>>
> >>> On 6/10/20 3:18 PM, David Mollitor wrote:
> >>> > Hey Zoltan,
> >>> >
> >>> > What is the process to trigger a new CI build on GitHub if a
> >>> previous one
> >>> > failed on a flaky test. timeout, or something of that nature?
> >>> >
> >>> > Thanks.
> >>> >
> >>>
> >>
>  --
>  Zoltán Haindrich
> 
> >>>
>
> --
> Zoltán Haindrich

Re: Re-Running CI Tests

[Zoltán Haindrich]

its on cooldown...5 builds/day/branch
http://34.66.156.144:8080/job/hive-precommit/job/PR-1082/

On June 10, 2020 6:55:45 PM GMT+02:00, David Mollitor  wrote:
>Zoltán,
>
>Even a PR against master is only running Travis.
>
>https://github.com/apache/hive/pull/1082
>
>On Wed, Jun 10, 2020 at 12:52 PM David Mollitor 
>wrote:
>
>> Hey Zoltán,
>>
>> Also in regard to that PR, it only ran the travis build  "mvn clean
>> install -DskipTests -q -Pitests"
>>
>> What does this branch require (and 3.x) to enable running of tests?
>>
>> Thanks.
>>
>>
>>
>> On Wed, Jun 10, 2020 at 11:36 AM David Mollitor 
>wrote:
>>
>>> Hey Zoltán,
>>>
>>> Yes.  That is correct.  Community recently put out a 2.x release. 
>This
>>> is in case someone wishes to release a new one.
>>>
>>> Does that have any bearing on re-running tests?
>>>
>>> Thanks.
>>>
>>> On Wed, Jun 10, 2020 at 11:32 AM Zoltán Haindrich 
>wrote:
>>>
 That pr seems to be for branch-2 and not master

 On June 10, 2020 5:17:19 PM GMT+02:00, David Mollitor
>
 wrote:
>
> Zoltan,
>
> I just tried to close/re-open a PR and I don't believe it
>triggered a
> new CI run:
>
> https://github.com/apache/hive/pull/1076
>
> Thanks.
>
> On Wed, Jun 10, 2020 at 10:59 AM David Mollitor
>
> wrote:
>
>> Hey Zoltan,
>>
>> Can you please research a way to initiate it from the GitHub
>> interface?  I have a strong feeling we're going to need such a
>> capability regularly.
>>
>> Thanks.
>>
>> On Wed, Jun 10, 2020 at 9:29 AM Zoltan Haindrich 
>wrote:
>>
>>> Hey
>>>
>>> you could:
>>> * push new commits to the branch
>>>- this will create a new merge with the current master
>>> * login in to the jenkins instance: and launch a new build of
>that PR
>>> * close&re-open the pr: will re-emit the github event triggering
>the
>>> testrun
>>> * login in to the jenkins instance: and press retry button
>>>
>>> I don't know if the last method (retrigger button) will create a
>new
>>> merge with the current master's HEAD or not - I suspect that it
>doesn't.
>>>
>>> cheers,
>>> Zoltan
>>>
>>> On 6/10/20 3:18 PM, David Mollitor wrote:
>>> > Hey Zoltan,
>>> >
>>> > What is the process to trigger a new CI build on GitHub if a
>>> previous one
>>> > failed on a flaky test. timeout, or something of that nature?
>>> >
>>> > Thanks.
>>> >
>>>
>>
 --
 Zoltán Haindrich

>>>

-- 
Zoltán Haindrich

[jira] [Created] (HIVE-23674) Add JenkinsFile to Branch 3

[David Mollitor (Jira)]

David Mollitor created HIVE-23674:
-

 Summary: Add JenkinsFile to Branch 3
 Key: HIVE-23674
 URL: https://issues.apache.org/jira/browse/HIVE-23674
 Project: Hive
  Issue Type: New Feature
Reporter: David Mollitor
Assignee: David Mollitor






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: Re-Running CI Tests

[David Mollitor]

Zoltán,

Even a PR against master is only running Travis.

https://github.com/apache/hive/pull/1082

On Wed, Jun 10, 2020 at 12:52 PM David Mollitor  wrote:

> Hey Zoltán,
>
> Also in regard to that PR, it only ran the travis build  "mvn clean
> install -DskipTests -q -Pitests"
>
> What does this branch require (and 3.x) to enable running of tests?
>
> Thanks.
>
>
>
> On Wed, Jun 10, 2020 at 11:36 AM David Mollitor  wrote:
>
>> Hey Zoltán,
>>
>> Yes.  That is correct.  Community recently put out a 2.x release.  This
>> is in case someone wishes to release a new one.
>>
>> Does that have any bearing on re-running tests?
>>
>> Thanks.
>>
>> On Wed, Jun 10, 2020 at 11:32 AM Zoltán Haindrich  wrote:
>>
>>> That pr seems to be for branch-2 and not master
>>>
>>> On June 10, 2020 5:17:19 PM GMT+02:00, David Mollitor 
>>> wrote:

 Zoltan,

 I just tried to close/re-open a PR and I don't believe it triggered a
 new CI run:

 https://github.com/apache/hive/pull/1076

 Thanks.

 On Wed, Jun 10, 2020 at 10:59 AM David Mollitor 
 wrote:

> Hey Zoltan,
>
> Can you please research a way to initiate it from the GitHub
> interface?  I have a strong feeling we're going to need such a
> capability regularly.
>
> Thanks.
>
> On Wed, Jun 10, 2020 at 9:29 AM Zoltan Haindrich  wrote:
>
>> Hey
>>
>> you could:
>> * push new commits to the branch
>>- this will create a new merge with the current master
>> * login in to the jenkins instance: and launch a new build of that PR
>> * close&re-open the pr: will re-emit the github event triggering the
>> testrun
>> * login in to the jenkins instance: and press retry button
>>
>> I don't know if the last method (retrigger button) will create a new
>> merge with the current master's HEAD or not - I suspect that it doesn't.
>>
>> cheers,
>> Zoltan
>>
>> On 6/10/20 3:18 PM, David Mollitor wrote:
>> > Hey Zoltan,
>> >
>> > What is the process to trigger a new CI build on GitHub if a
>> previous one
>> > failed on a flaky test. timeout, or something of that nature?
>> >
>> > Thanks.
>> >
>>
>
>>> --
>>> Zoltán Haindrich
>>>
>>

Re: Re-Running CI Tests

[David Mollitor]

Hey Zoltán,

Also in regard to that PR, it only ran the travis build  "mvn clean install
-DskipTests -q -Pitests"

What does this branch require (and 3.x) to enable running of tests?

Thanks.



On Wed, Jun 10, 2020 at 11:36 AM David Mollitor  wrote:

> Hey Zoltán,
>
> Yes.  That is correct.  Community recently put out a 2.x release.  This is
> in case someone wishes to release a new one.
>
> Does that have any bearing on re-running tests?
>
> Thanks.
>
> On Wed, Jun 10, 2020 at 11:32 AM Zoltán Haindrich  wrote:
>
>> That pr seems to be for branch-2 and not master
>>
>> On June 10, 2020 5:17:19 PM GMT+02:00, David Mollitor 
>> wrote:
>>>
>>> Zoltan,
>>>
>>> I just tried to close/re-open a PR and I don't believe it triggered a
>>> new CI run:
>>>
>>> https://github.com/apache/hive/pull/1076
>>>
>>> Thanks.
>>>
>>> On Wed, Jun 10, 2020 at 10:59 AM David Mollitor 
>>> wrote:
>>>
 Hey Zoltan,

 Can you please research a way to initiate it from the GitHub
 interface?  I have a strong feeling we're going to need such a
 capability regularly.

 Thanks.

 On Wed, Jun 10, 2020 at 9:29 AM Zoltan Haindrich  wrote:

> Hey
>
> you could:
> * push new commits to the branch
>- this will create a new merge with the current master
> * login in to the jenkins instance: and launch a new build of that PR
> * close&re-open the pr: will re-emit the github event triggering the
> testrun
> * login in to the jenkins instance: and press retry button
>
> I don't know if the last method (retrigger button) will create a new
> merge with the current master's HEAD or not - I suspect that it doesn't.
>
> cheers,
> Zoltan
>
> On 6/10/20 3:18 PM, David Mollitor wrote:
> > Hey Zoltan,
> >
> > What is the process to trigger a new CI build on GitHub if a
> previous one
> > failed on a flaky test. timeout, or something of that nature?
> >
> > Thanks.
> >
>

>> --
>> Zoltán Haindrich
>>
>

Re: Re-Running CI Tests

[David Mollitor]

Hey Zoltán,

Yes.  That is correct.  Community recently put out a 2.x release.  This is
in case someone wishes to release a new one.

Does that have any bearing on re-running tests?

Thanks.

On Wed, Jun 10, 2020 at 11:32 AM Zoltán Haindrich  wrote:

> That pr seems to be for branch-2 and not master
>
> On June 10, 2020 5:17:19 PM GMT+02:00, David Mollitor 
> wrote:
>>
>> Zoltan,
>>
>> I just tried to close/re-open a PR and I don't believe it triggered a new
>> CI run:
>>
>> https://github.com/apache/hive/pull/1076
>>
>> Thanks.
>>
>> On Wed, Jun 10, 2020 at 10:59 AM David Mollitor 
>> wrote:
>>
>>> Hey Zoltan,
>>>
>>> Can you please research a way to initiate it from the GitHub interface?
>>> I have a strong feeling we're going to need such a capability regularly.
>>>
>>> Thanks.
>>>
>>> On Wed, Jun 10, 2020 at 9:29 AM Zoltan Haindrich  wrote:
>>>
 Hey

 you could:
 * push new commits to the branch
- this will create a new merge with the current master
 * login in to the jenkins instance: and launch a new build of that PR
 * close&re-open the pr: will re-emit the github event triggering the
 testrun
 * login in to the jenkins instance: and press retry button

 I don't know if the last method (retrigger button) will create a new
 merge with the current master's HEAD or not - I suspect that it doesn't.

 cheers,
 Zoltan

 On 6/10/20 3:18 PM, David Mollitor wrote:
 > Hey Zoltan,
 >
 > What is the process to trigger a new CI build on GitHub if a previous
 one
 > failed on a flaky test. timeout, or something of that nature?
 >
 > Thanks.
 >

>>>
> --
> Zoltán Haindrich
>

Re: Re-Running CI Tests

[Zoltán Haindrich]

That pr seems to be for branch-2 and not master

On June 10, 2020 5:17:19 PM GMT+02:00, David Mollitor  wrote:
>Zoltan,
>
>I just tried to close/re-open a PR and I don't believe it triggered a
>new
>CI run:
>
>https://github.com/apache/hive/pull/1076
>
>Thanks.
>
>On Wed, Jun 10, 2020 at 10:59 AM David Mollitor 
>wrote:
>
>> Hey Zoltan,
>>
>> Can you please research a way to initiate it from the GitHub
>interface?  I
>> have a strong feeling we're going to need such a capability
>regularly.
>>
>> Thanks.
>>
>> On Wed, Jun 10, 2020 at 9:29 AM Zoltan Haindrich  wrote:
>>
>>> Hey
>>>
>>> you could:
>>> * push new commits to the branch
>>>- this will create a new merge with the current master
>>> * login in to the jenkins instance: and launch a new build of that
>PR
>>> * close&re-open the pr: will re-emit the github event triggering the
>>> testrun
>>> * login in to the jenkins instance: and press retry button
>>>
>>> I don't know if the last method (retrigger button) will create a new
>>> merge with the current master's HEAD or not - I suspect that it
>doesn't.
>>>
>>> cheers,
>>> Zoltan
>>>
>>> On 6/10/20 3:18 PM, David Mollitor wrote:
>>> > Hey Zoltan,
>>> >
>>> > What is the process to trigger a new CI build on GitHub if a
>previous
>>> one
>>> > failed on a flaky test. timeout, or something of that nature?
>>> >
>>> > Thanks.
>>> >
>>>
>>

-- 
Zoltán Haindrich

Re: Re-Running CI Tests

[David Mollitor]

Zoltan,

I just tried to close/re-open a PR and I don't believe it triggered a new
CI run:

https://github.com/apache/hive/pull/1076

Thanks.

On Wed, Jun 10, 2020 at 10:59 AM David Mollitor  wrote:

> Hey Zoltan,
>
> Can you please research a way to initiate it from the GitHub interface?  I
> have a strong feeling we're going to need such a capability regularly.
>
> Thanks.
>
> On Wed, Jun 10, 2020 at 9:29 AM Zoltan Haindrich  wrote:
>
>> Hey
>>
>> you could:
>> * push new commits to the branch
>>- this will create a new merge with the current master
>> * login in to the jenkins instance: and launch a new build of that PR
>> * close&re-open the pr: will re-emit the github event triggering the
>> testrun
>> * login in to the jenkins instance: and press retry button
>>
>> I don't know if the last method (retrigger button) will create a new
>> merge with the current master's HEAD or not - I suspect that it doesn't.
>>
>> cheers,
>> Zoltan
>>
>> On 6/10/20 3:18 PM, David Mollitor wrote:
>> > Hey Zoltan,
>> >
>> > What is the process to trigger a new CI build on GitHub if a previous
>> one
>> > failed on a flaky test. timeout, or something of that nature?
>> >
>> > Thanks.
>> >
>>
>

Re: Re-Running CI Tests

[David Mollitor]

Hey Zoltan,

Can you please research a way to initiate it from the GitHub interface?  I
have a strong feeling we're going to need such a capability regularly.

Thanks.

On Wed, Jun 10, 2020 at 9:29 AM Zoltan Haindrich  wrote:

> Hey
>
> you could:
> * push new commits to the branch
>- this will create a new merge with the current master
> * login in to the jenkins instance: and launch a new build of that PR
> * close&re-open the pr: will re-emit the github event triggering the
> testrun
> * login in to the jenkins instance: and press retry button
>
> I don't know if the last method (retrigger button) will create a new merge
> with the current master's HEAD or not - I suspect that it doesn't.
>
> cheers,
> Zoltan
>
> On 6/10/20 3:18 PM, David Mollitor wrote:
> > Hey Zoltan,
> >
> > What is the process to trigger a new CI build on GitHub if a previous one
> > failed on a flaky test. timeout, or something of that nature?
> >
> > Thanks.
> >
>

Re: Flaky test checker

[Zoltan Haindrich]

One more thing: there should be other builds running while the flaky check is being 
executed (otherwise it will be "alone" on a 12 core system)

On 6/10/20 3:49 PM, Zoltan Haindrich wrote:

Hey All!

I've fiddled around to build this into the main test system or not; but in the end I've concluded that it will be more usefull as a standalone tool (this makes the job a 
bit uglier - but well...it would have made the main one uglier as well - so it doesn't matter which finger I'll bite)


So...if you are suspecting that test is causing trouble for no good reason; you 
could launch a run of this job which will run it a 100 times in a row...if it 
fails...well:
* you could open a jira which references the check you executed which proves 
that the test is low quality
   * please also add the "flaky-test" label to the jira
* add an Ignore to the test referencing the jira ticket
* push the commit which disables the test...

The other use would be when enabling previously unreliable tests back:
* push your branch which supposed to stabilize the test to your own fork on 
github
* visit http://130.211.9.232/job/hive-flaky-check/
* point the job to your user/repo/branch ; and configure to run the test in 
question to validate it


cheers,
Zoltan

[jira] [Created] (HIVE-23673) Maven Standard Directories for accumulo-handler

[David Mollitor (Jira)]

David Mollitor created HIVE-23673:
-

 Summary: Maven Standard Directories for accumulo-handler
 Key: HIVE-23673
 URL: https://issues.apache.org/jira/browse/HIVE-23673
 Project: Hive
  Issue Type: Sub-task
Reporter: David Mollitor
Assignee: David Mollitor






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HIVE-23672) Modules Should Adhere to Maven Default Directory Structure

[David Mollitor (Jira)]

David Mollitor created HIVE-23672:
-

 Summary: Modules Should Adhere to Maven Default Directory Structure
 Key: HIVE-23672
 URL: https://issues.apache.org/jira/browse/HIVE-23672
 Project: Hive
  Issue Type: Improvement
Reporter: David Mollitor
Assignee: David Mollitor






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Flaky test checker

[Zoltan Haindrich]

Hey All!

I've fiddled around to build this into the main test system or not; but in the end I've concluded that it will be more usefull as a standalone tool (this makes the job a 
bit uglier - but well...it would have made the main one uglier as well - so it doesn't matter which finger I'll bite)


So...if you are suspecting that test is causing trouble for no good reason; you 
could launch a run of this job which will run it a 100 times in a row...if it 
fails...well:
* you could open a jira which references the check you executed which proves 
that the test is low quality
  * please also add the "flaky-test" label to the jira
* add an Ignore to the test referencing the jira ticket
* push the commit which disables the test...

The other use would be when enabling previously unreliable tests back:
* push your branch which supposed to stabilize the test to your own fork on 
github
* visit http://130.211.9.232/job/hive-flaky-check/
* point the job to your user/repo/branch ; and configure to run the test in 
question to validate it


cheers,
Zoltan

Re: [DISCUSS] Disable ptest job

[Zoltan Haindrich]

Hey All!

I've disabled the ptest job on builds.apache.org - please open PRs from now on.

cheers,
Zoltan

On 6/5/20 6:12 PM, Jesus Camacho Rodriguez wrote:

+1

Thanks Zoltan.

-Jesús

On Fri, Jun 5, 2020 at 2:13 AM Stamatis Zampetakis 
wrote:


Hi Zoltan,

The sooner we move away from the old system the better. It will also help
to detect and solve faster any kind of problems with the new approach if
there are more people using it.

Also it will be cool to have junit5 :D

Best,
Stamatis


On Fri, Jun 5, 2020 at 10:44 AM Zoltan Haindrich  wrote:


Hey all!

So far I've seen only 1 issue with the new system: there were 2

occurences

in the last week when a build was affected by some kind a kubernetes

issue

which have taken down
an executor - the logs pointed to some kind of kubelet issue; since the
GKE master have upgraded to 1.16 while the node pools were still running
1.15 it could have been the
cause of it. Yesterday I've upgraded all the node-pools.

Because I've seen that people now sometimes open both a PR and upload a
patch to the jira as well; I would like to propose to disable the PTest

job

on builds.apache.org on
Monday.

Note: This will also unblock to apply the junit5 patch - and could open

up

the possibility to sometimes exclude a set of tests from execution - the
total test execution
time is around 24 hours - from which 8 hours is spent running replication
tests. Since most changes will touch replication stuff they could be made
optional.

cheers,
Zoltan

Re: Re-Running CI Tests

[Zoltan Haindrich]

Hey

you could:
* push new commits to the branch
  - this will create a new merge with the current master
* login in to the jenkins instance: and launch a new build of that PR
* close&re-open the pr: will re-emit the github event triggering the testrun
* login in to the jenkins instance: and press retry button

I don't know if the last method (retrigger button) will create a new merge with 
the current master's HEAD or not - I suspect that it doesn't.

cheers,
Zoltan

On 6/10/20 3:18 PM, David Mollitor wrote:

Hey Zoltan,

What is the process to trigger a new CI build on GitHub if a previous one
failed on a flaky test. timeout, or something of that nature?

Thanks.

Re-Running CI Tests

[David Mollitor]

Hey Zoltan,

What is the process to trigger a new CI build on GitHub if a previous one
failed on a flaky test. timeout, or something of that nature?

Thanks.

Re: Review Request 72532: HIVE-23495 AcidUtils.getAcidState cleanup

[Karen Coppage via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72532/#review220990
---


Ship it!




Ship It!

- Karen Coppage


On June 8, 2020, 10:58 a.m., Peter Varga wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72532/
> ---
> 
> (Updated June 8, 2020, 10:58 a.m.)
> 
> 
> Review request for hive, Karen Coppage, Marta Kuczora, and Peter Vary.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> since HIVE-21225 there are two redundant implementation of the 
> AcidUtils.getAcidState.
> 
> The previous implementation (without the recursive listing) can be removed.
> 
> Also the performance can be improved, by removing unnecessary fileStatus 
> calls.
> 
> 
> Diffs
> -
> 
>   ql/src/java/org/apache/hadoop/hive/ql/io/AcidUtils.java 635ed3149c 
>   ql/src/java/org/apache/hadoop/hive/ql/io/HiveInputFormat.java ca234cfb37 
>   ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcInputFormat.java 1059cb227f 
>   ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcRawRecordMerger.java 
> 16c915959c 
>   
> ql/src/java/org/apache/hadoop/hive/ql/io/orc/VectorizedOrcAcidRowBatchReader.java
>  598220b0c4 
>   ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Cleaner.java 2a15913f9f 
>   ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/CompactorMR.java 
> 4e5d5b003b 
>   ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Initiator.java 
> 7913295380 
>   
> ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/MinorQueryCompactor.java 
> d83a50f555 
>   
> ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/MmMajorQueryCompactor.java
>  5e11d8d2d8 
>   
> ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/MmMinorQueryCompactor.java
>  1bdec7df2d 
>   ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Worker.java 75941b3f33 
>   ql/src/test/org/apache/hadoop/hive/ql/TestTxnCommands2.java 337f469d1a 
>   ql/src/test/org/apache/hadoop/hive/ql/io/TestAcidUtils.java f351f04b08 
>   ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestInputOutputFormat.java 
> e4440e9136 
>   ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestOrcRawRecordMerger.java 
> f63c40a7b5 
>   streaming/src/test/org/apache/hive/streaming/TestStreaming.java 3a3b267927 
> 
> 
> Diff: https://reviews.apache.org/r/72532/diff/3/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Peter Varga
> 
>

Re: Review Request 72532: HIVE-23495 AcidUtils.getAcidState cleanup

[Karen Coppage via Review Board]

> On June 5, 2020, 2:04 p.m., Karen Coppage wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/io/AcidUtils.java
> > Line 1411 (original)
> > 
> >
> > Are these originals not needed, or collected elsewhere?
> 
> Peter Varga wrote:
> This one is bothering me, these lines were added when the snapshot way 
> was introduced, but I do not see why. When we calculated the AcidState 
> without the snapshot these files were not added to the originals list. It is 
> explicitly there few lines above, that if we have a base we consider every 
> original files as obsolete. The 
> TestTxnCommandsForMmTable#testInsertOverwriteForPartitionedMmTable breaks for 
> example if these files are added to the list. After an insert-overwrite to a 
> mm table and calling the major compaction, the compaction will create a new 
> base dir, not leaving the perfectly fine base dir generated by the insert 
> overwrite. I did not dig into the compaction to see why the original files 
> are triggering it, but I do not think these files needed in the original list.

Ok. I see that files in a base, even if in original format, should be 
considered base files and not original files. It's still not clear to me why 
this block was added but to me it makes sense to get rid of it.


- Karen


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72532/#review220956
---


On June 8, 2020, 10:58 a.m., Peter Varga wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72532/
> ---
> 
> (Updated June 8, 2020, 10:58 a.m.)
> 
> 
> Review request for hive, Karen Coppage, Marta Kuczora, and Peter Vary.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> since HIVE-21225 there are two redundant implementation of the 
> AcidUtils.getAcidState.
> 
> The previous implementation (without the recursive listing) can be removed.
> 
> Also the performance can be improved, by removing unnecessary fileStatus 
> calls.
> 
> 
> Diffs
> -
> 
>   ql/src/java/org/apache/hadoop/hive/ql/io/AcidUtils.java 635ed3149c 
>   ql/src/java/org/apache/hadoop/hive/ql/io/HiveInputFormat.java ca234cfb37 
>   ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcInputFormat.java 1059cb227f 
>   ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcRawRecordMerger.java 
> 16c915959c 
>   
> ql/src/java/org/apache/hadoop/hive/ql/io/orc/VectorizedOrcAcidRowBatchReader.java
>  598220b0c4 
>   ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Cleaner.java 2a15913f9f 
>   ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/CompactorMR.java 
> 4e5d5b003b 
>   ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Initiator.java 
> 7913295380 
>   
> ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/MinorQueryCompactor.java 
> d83a50f555 
>   
> ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/MmMajorQueryCompactor.java
>  5e11d8d2d8 
>   
> ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/MmMinorQueryCompactor.java
>  1bdec7df2d 
>   ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Worker.java 75941b3f33 
>   ql/src/test/org/apache/hadoop/hive/ql/TestTxnCommands2.java 337f469d1a 
>   ql/src/test/org/apache/hadoop/hive/ql/io/TestAcidUtils.java f351f04b08 
>   ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestInputOutputFormat.java 
> e4440e9136 
>   ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestOrcRawRecordMerger.java 
> f63c40a7b5 
>   streaming/src/test/org/apache/hive/streaming/TestStreaming.java 3a3b267927 
> 
> 
> Diff: https://reviews.apache.org/r/72532/diff/3/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Peter Varga
> 
>

[jira] [Created] (HIVE-23671) MSCK repair should handle transactional tables in certain usecases

[Peter Varga (Jira)]

Peter Varga created HIVE-23671:
--

 Summary: MSCK repair should handle transactional tables in certain 
usecases
 Key: HIVE-23671
 URL: https://issues.apache.org/jira/browse/HIVE-23671
 Project: Hive
  Issue Type: Improvement
  Components: Metastore
Reporter: Peter Varga
Assignee: Peter Varga


The MSCK REPAIR tool does not handle transactional tables too well. It can find 
and add new partitions the same way as for non-transactional tables, but since 
the writeId differences are not handled, the data can not read back from the 
new partitions.

We could handle some usecases when the writeIds in the HMS and the underlying 
data are not conflicting. If the HMS does not contains allocated writes for the 
table we can seed the table with the writeIds read from the directory 
structrure.

Real life use cases could be:
 * Copy data files from one cluster to another with different HMS, create the 
table and call MSCK REPAIR
 * If the HMS db is lost, recreate the table and call MSCK REPAIR

 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HIVE-23670) hive on spark may encounter error when use map join

[zhoukang (Jira)]

zhoukang created HIVE-23670:
---

 Summary: hive on spark may encounter error when use map join
 Key: HIVE-23670
 URL: https://issues.apache.org/jira/browse/HIVE-23670
 Project: Hive
  Issue Type: Bug
Affects Versions: 2.4.0
Reporter: zhoukang


when using spark as hive engine, some task may failed with exception below.
The cause is that when we load table, we may issued with:

{code:java}
if (tableContainer == null) {
tableContainer = useOptimizedContainer ?
new MapJoinBytesTableContainer(hconf, valueContext, -1, 0) :
create(name, metaData);
  }
{code}

then cast exception will issued with code below:

{code:java}
 @Override
public JoinUtil.JoinResult setFromOther(ReusableGetAdaptor other) {
  assert other instanceof GetAdaptor;
  GetAdaptor other2 = (GetAdaptor)other;
  this.key = other2.key;
  this.isFirstKey = other2.isFirstKey;
  this.currentValue = mHash.get(key);
  if (this.currentValue == null) {
return JoinUtil.JoinResult.NOMATCH;
  }
  else {
return JoinUtil.JoinResult.MATCH;
  }
}
{code}


{code:java}
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Unexpected 
exception from MapJoinOperator : 
org.apache.hadoop.hive.ql.exec.persistence.MapJoinBytesTableContainer$GetAdaptor
 cannot be cast to 
org.apache.hadoop.hive.ql.exec.persistence.HashMapWrapper$GetAdaptor
at 
org.apache.hadoop.hive.ql.exec.MapJoinOperator.process(MapJoinOperator.java:465)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:897)
at 
org.apache.hadoop.hive.ql.exec.SelectOperator.process(SelectOperator.java:95)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:897)
at 
org.apache.hadoop.hive.ql.exec.TableScanOperator.process(TableScanOperator.java:130)
at 
org.apache.hadoop.hive.ql.exec.MapOperator$MapOpCtx.forward(MapOperator.java:148)
at 
org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:547)
... 19 more
{code}




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HIVE-23669) Stop using jetty-runner which is an uberjar

[Zoltan Haindrich (Jira)]

Zoltan Haindrich created HIVE-23669:
---

 Summary: Stop using jetty-runner which is an uberjar
 Key: HIVE-23669
 URL: https://issues.apache.org/jira/browse/HIVE-23669
 Project: Hive
  Issue Type: Bug
Reporter: Zoltan Haindrich


we already create our own shaded artifacts ; during it detects class 
duplications
{code}
02:51:03  [WARNING] jetty-server-9.3.27.v20190418.jar, 
jetty-runner-9.3.27.v20190418.jar define 245 overlapping classes: 
02:51:03  [WARNING]   - 
org.eclipse.jetty.server.handler.ThreadLimitHandler$RFC7239
02:51:03  [WARNING]   - 
org.eclipse.jetty.server.handler.ManagedAttributeListener
02:51:03  [WARNING]   - 
org.eclipse.jetty.server.session.JDBCSessionIdManager$SessionTableSchema
02:51:03  [WARNING]   - org.eclipse.jetty.server.ServletRequestHttpWrapper
02:51:03  [WARNING]   - org.eclipse.jetty.server.DebugListener$1
02:51:03  [WARNING]   - 
org.eclipse.jetty.server.session.HashSessionManager$Saver
02:51:03  [WARNING]   - org.eclipse.jetty.server.ShutdownMonitor$1
02:51:03  [WARNING]   - org.eclipse.jetty.server.HttpInput$1
02:51:03  [WARNING]   - org.eclipse.jetty.server.UserIdentity$Scope
02:51:03  [WARNING]   - org.eclipse.jetty.server.AsyncContextState$1
02:51:03  [WARNING]   - 235 more...
02:51:03  [WARNING] jetty-runner-9.3.27.v20190418.jar, ecj-4.4.2.jar define 557 
overlapping classes: 
02:51:03  [WARNING]   - 
org.eclipse.jdt.internal.compiler.apt.model.ErrorTypeElement
02:51:03  [WARNING]   - 
org.eclipse.jdt.internal.compiler.lookup.ReferenceBinding
02:51:03  [WARNING]   - 
org.eclipse.jdt.internal.compiler.lookup.ConstraintFormula
02:51:03  [WARNING]   - 
org.eclipse.jdt.internal.compiler.apt.dispatch.BatchMessagerImpl
02:51:03  [WARNING]   - org.eclipse.jdt.internal.compiler.ast.MarkerAnnotation
02:51:03  [WARNING]   - org.eclipse.jdt.internal.compiler.ast.ThrowStatement
02:51:03  [WARNING]   - 
org.eclipse.jdt.internal.compiler.ast.JavadocSingleNameReference
02:51:03  [WARNING]   - 
org.eclipse.jdt.internal.compiler.lookup.ProblemReferenceBinding
02:51:03  [WARNING]   - org.eclipse.jdt.internal.compiler.ast.IntLiteral
02:51:03  [WARNING]   - org.eclipse.jdt.internal.compiler.ast.Reference
{code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)