Re: Several unit tests failing on Sentry

[Na Li]

Sergio,

Thanks for the summary. It's very helpful.

Lina

On Tue, Nov 21, 2017 at 9:08 AM, Sergio Pena 
wrote:

> Lina,
>
> There were some leads on the Jenkins side about the issues.
> 1. All the tests were failing because the Hive/MR job couldn't find a file
> on the /tmp directory.
> 2. All the tests were failing only during the day (they were passing at
> night).
> 3. Reverting too old patches were sometimes passing or failing with the
> same failures.
> 4. Finally, we found that Jenkins slaves are shared with other non-sentry
> jobs. This was the last confirmation that something was not right with the
> slave environment.
>
> Once I applied those patches to move the /tmp temporary files to a
> different one, then they worked, so the confirmation that something was
> deleting the /tmp space was correct.
>
> - Sergio
>
> On Mon, Nov 20, 2017 at 4:01 PM, Na Li  wrote:
>
> > Sergio,
> >
> > It's great you found the cause. How did you figure it out?
> >
> > Thanks,
> >
> > Lina
> >
> > On Mon, Nov 20, 2017 at 12:46 PM, Kalyan Kumar Kalvagadda <
> > kkal...@cloudera.com> wrote:
> >
> > > Sergio,
> > >
> > > I resolved SENTRY-2050.
> > >
> > > -Kalyan
> > >
> > > On Mon, Nov 20, 2017 at 12:40 PM, Sergio Pena <
> sergio.p...@cloudera.com>
> > > wrote:
> > >
> > > > We finally deciphered the reason for the Jenkins test failures, and
> > they
> > > > were not related to the Sentry codebase. For some unknown reason, the
> > > /tmp
> > > > directory was cleaned up at some point causing the Sentry hive tests
> to
> > > > fail on MR jobs because files on the /tmp weren't found.
> > > >
> > > > The fixes were done through these jiras and now tests are green:
> > > >   SENTRY-2054 Unit tests must create temporary files under the Maven
> > > target
> > > > directory
> > > >   SENTRY-2057: Set hadoop.tmp.dir to the maven build directory
> > configured
> > > > on java.io.tmpdir
> > > >
> > > > Kalyan, you're free to close the SENTRY-2050.
> > > >
> > > > - Sergio
> > > >
> > > > On Thu, Nov 16, 2017 at 1:57 PM, Kalyan Kumar Kalvagadda <
> > > > kkal...@cloudera.com> wrote:
> > > >
> > > > > I have created SENTRY-2050 to investigate the situation. As a first
> > > step
> > > > > I'm reverting couple of commits and submit the patch to see how
> > Jenkins
> > > > > behaves.
> > > > >
> > > > > -Kalyan
> > > > >
> > > > > On Thu, Nov 16, 2017 at 1:27 PM, Sergio Pena <
> > sergio.p...@cloudera.com
> > > >
> > > > > wrote:
> > > > >
> > > > > > There are several tests failing currently and seems they're
> flaky.
> > > > > They're
> > > > > > passing locally but most of the time fail on Jenkins with some
> > > > exceptions
> > > > > > on some patches.
> > > > > >
> > > > > > We're looking at the old commits from the past week to see which
> > > commit
> > > > > is
> > > > > > the culprit of this.
> > > > > >
> > > > > > - Sergio
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: Several unit tests failing on Sentry

[Sergio Pena]

Lina,

There were some leads on the Jenkins side about the issues.
1. All the tests were failing because the Hive/MR job couldn't find a file
on the /tmp directory.
2. All the tests were failing only during the day (they were passing at
night).
3. Reverting too old patches were sometimes passing or failing with the
same failures.
4. Finally, we found that Jenkins slaves are shared with other non-sentry
jobs. This was the last confirmation that something was not right with the
slave environment.

Once I applied those patches to move the /tmp temporary files to a
different one, then they worked, so the confirmation that something was
deleting the /tmp space was correct.

- Sergio

On Mon, Nov 20, 2017 at 4:01 PM, Na Li  wrote:

> Sergio,
>
> It's great you found the cause. How did you figure it out?
>
> Thanks,
>
> Lina
>
> On Mon, Nov 20, 2017 at 12:46 PM, Kalyan Kumar Kalvagadda <
> kkal...@cloudera.com> wrote:
>
> > Sergio,
> >
> > I resolved SENTRY-2050.
> >
> > -Kalyan
> >
> > On Mon, Nov 20, 2017 at 12:40 PM, Sergio Pena 
> > wrote:
> >
> > > We finally deciphered the reason for the Jenkins test failures, and
> they
> > > were not related to the Sentry codebase. For some unknown reason, the
> > /tmp
> > > directory was cleaned up at some point causing the Sentry hive tests to
> > > fail on MR jobs because files on the /tmp weren't found.
> > >
> > > The fixes were done through these jiras and now tests are green:
> > >   SENTRY-2054 Unit tests must create temporary files under the Maven
> > target
> > > directory
> > >   SENTRY-2057: Set hadoop.tmp.dir to the maven build directory
> configured
> > > on java.io.tmpdir
> > >
> > > Kalyan, you're free to close the SENTRY-2050.
> > >
> > > - Sergio
> > >
> > > On Thu, Nov 16, 2017 at 1:57 PM, Kalyan Kumar Kalvagadda <
> > > kkal...@cloudera.com> wrote:
> > >
> > > > I have created SENTRY-2050 to investigate the situation. As a first
> > step
> > > > I'm reverting couple of commits and submit the patch to see how
> Jenkins
> > > > behaves.
> > > >
> > > > -Kalyan
> > > >
> > > > On Thu, Nov 16, 2017 at 1:27 PM, Sergio Pena <
> sergio.p...@cloudera.com
> > >
> > > > wrote:
> > > >
> > > > > There are several tests failing currently and seems they're flaky.
> > > > They're
> > > > > passing locally but most of the time fail on Jenkins with some
> > > exceptions
> > > > > on some patches.
> > > > >
> > > > > We're looking at the old commits from the past week to see which
> > commit
> > > > is
> > > > > the culprit of this.
> > > > >
> > > > > - Sergio
> > > > >
> > > >
> > >
> >
>

Re: Several unit tests failing on Sentry

[Na Li]

Sergio,

It's great you found the cause. How did you figure it out?

Thanks,

Lina

On Mon, Nov 20, 2017 at 12:46 PM, Kalyan Kumar Kalvagadda <
kkal...@cloudera.com> wrote:

> Sergio,
>
> I resolved SENTRY-2050.
>
> -Kalyan
>
> On Mon, Nov 20, 2017 at 12:40 PM, Sergio Pena 
> wrote:
>
> > We finally deciphered the reason for the Jenkins test failures, and they
> > were not related to the Sentry codebase. For some unknown reason, the
> /tmp
> > directory was cleaned up at some point causing the Sentry hive tests to
> > fail on MR jobs because files on the /tmp weren't found.
> >
> > The fixes were done through these jiras and now tests are green:
> >   SENTRY-2054 Unit tests must create temporary files under the Maven
> target
> > directory
> >   SENTRY-2057: Set hadoop.tmp.dir to the maven build directory configured
> > on java.io.tmpdir
> >
> > Kalyan, you're free to close the SENTRY-2050.
> >
> > - Sergio
> >
> > On Thu, Nov 16, 2017 at 1:57 PM, Kalyan Kumar Kalvagadda <
> > kkal...@cloudera.com> wrote:
> >
> > > I have created SENTRY-2050 to investigate the situation. As a first
> step
> > > I'm reverting couple of commits and submit the patch to see how Jenkins
> > > behaves.
> > >
> > > -Kalyan
> > >
> > > On Thu, Nov 16, 2017 at 1:27 PM, Sergio Pena  >
> > > wrote:
> > >
> > > > There are several tests failing currently and seems they're flaky.
> > > They're
> > > > passing locally but most of the time fail on Jenkins with some
> > exceptions
> > > > on some patches.
> > > >
> > > > We're looking at the old commits from the past week to see which
> commit
> > > is
> > > > the culprit of this.
> > > >
> > > > - Sergio
> > > >
> > >
> >
>

Re: Several unit tests failing on Sentry

[Kalyan Kumar Kalvagadda]

Sergio,

I resolved SENTRY-2050.

-Kalyan

On Mon, Nov 20, 2017 at 12:40 PM, Sergio Pena 
wrote:

> We finally deciphered the reason for the Jenkins test failures, and they
> were not related to the Sentry codebase. For some unknown reason, the /tmp
> directory was cleaned up at some point causing the Sentry hive tests to
> fail on MR jobs because files on the /tmp weren't found.
>
> The fixes were done through these jiras and now tests are green:
>   SENTRY-2054 Unit tests must create temporary files under the Maven target
> directory
>   SENTRY-2057: Set hadoop.tmp.dir to the maven build directory configured
> on java.io.tmpdir
>
> Kalyan, you're free to close the SENTRY-2050.
>
> - Sergio
>
> On Thu, Nov 16, 2017 at 1:57 PM, Kalyan Kumar Kalvagadda <
> kkal...@cloudera.com> wrote:
>
> > I have created SENTRY-2050 to investigate the situation. As a first step
> > I'm reverting couple of commits and submit the patch to see how Jenkins
> > behaves.
> >
> > -Kalyan
> >
> > On Thu, Nov 16, 2017 at 1:27 PM, Sergio Pena 
> > wrote:
> >
> > > There are several tests failing currently and seems they're flaky.
> > They're
> > > passing locally but most of the time fail on Jenkins with some
> exceptions
> > > on some patches.
> > >
> > > We're looking at the old commits from the past week to see which commit
> > is
> > > the culprit of this.
> > >
> > > - Sergio
> > >
> >
>

Re: Several unit tests failing on Sentry

[Sergio Pena]

We finally deciphered the reason for the Jenkins test failures, and they
were not related to the Sentry codebase. For some unknown reason, the /tmp
directory was cleaned up at some point causing the Sentry hive tests to
fail on MR jobs because files on the /tmp weren't found.

The fixes were done through these jiras and now tests are green:
  SENTRY-2054 Unit tests must create temporary files under the Maven target
directory
  SENTRY-2057: Set hadoop.tmp.dir to the maven build directory configured
on java.io.tmpdir

Kalyan, you're free to close the SENTRY-2050.

- Sergio

On Thu, Nov 16, 2017 at 1:57 PM, Kalyan Kumar Kalvagadda <
kkal...@cloudera.com> wrote:

> I have created SENTRY-2050 to investigate the situation. As a first step
> I'm reverting couple of commits and submit the patch to see how Jenkins
> behaves.
>
> -Kalyan
>
> On Thu, Nov 16, 2017 at 1:27 PM, Sergio Pena 
> wrote:
>
> > There are several tests failing currently and seems they're flaky.
> They're
> > passing locally but most of the time fail on Jenkins with some exceptions
> > on some patches.
> >
> > We're looking at the old commits from the past week to see which commit
> is
> > the culprit of this.
> >
> > - Sergio
> >
>

Re: Several unit tests failing on Sentry

[Kalyan Kumar Kalvagadda]

I have created SENTRY-2050 to investigate the situation. As a first step
I'm reverting couple of commits and submit the patch to see how Jenkins
behaves.

-Kalyan

On Thu, Nov 16, 2017 at 1:27 PM, Sergio Pena 
wrote:

> There are several tests failing currently and seems they're flaky. They're
> passing locally but most of the time fail on Jenkins with some exceptions
> on some patches.
>
> We're looking at the old commits from the past week to see which commit is
> the culprit of this.
>
> - Sergio
>