Sergio, Thanks for the summary. It's very helpful.
Lina On Tue, Nov 21, 2017 at 9:08 AM, Sergio Pena <sergio.p...@cloudera.com> wrote: > Lina, > > There were some leads on the Jenkins side about the issues. > 1. All the tests were failing because the Hive/MR job couldn't find a file > on the /tmp directory. > 2. All the tests were failing only during the day (they were passing at > night). > 3. Reverting too old patches were sometimes passing or failing with the > same failures. > 4. Finally, we found that Jenkins slaves are shared with other non-sentry > jobs. This was the last confirmation that something was not right with the > slave environment. > > Once I applied those patches to move the /tmp temporary files to a > different one, then they worked, so the confirmation that something was > deleting the /tmp space was correct. > > - Sergio > > On Mon, Nov 20, 2017 at 4:01 PM, Na Li <lina...@cloudera.com> wrote: > > > Sergio, > > > > It's great you found the cause. How did you figure it out? > > > > Thanks, > > > > Lina > > > > On Mon, Nov 20, 2017 at 12:46 PM, Kalyan Kumar Kalvagadda < > > kkal...@cloudera.com> wrote: > > > > > Sergio, > > > > > > I resolved SENTRY-2050. > > > > > > -Kalyan > > > > > > On Mon, Nov 20, 2017 at 12:40 PM, Sergio Pena < > sergio.p...@cloudera.com> > > > wrote: > > > > > > > We finally deciphered the reason for the Jenkins test failures, and > > they > > > > were not related to the Sentry codebase. For some unknown reason, the > > > /tmp > > > > directory was cleaned up at some point causing the Sentry hive tests > to > > > > fail on MR jobs because files on the /tmp weren't found. > > > > > > > > The fixes were done through these jiras and now tests are green: > > > > SENTRY-2054 Unit tests must create temporary files under the Maven > > > target > > > > directory > > > > SENTRY-2057: Set hadoop.tmp.dir to the maven build directory > > configured > > > > on java.io.tmpdir > > > > > > > > Kalyan, you're free to close the SENTRY-2050. > > > > > > > > - Sergio > > > > > > > > On Thu, Nov 16, 2017 at 1:57 PM, Kalyan Kumar Kalvagadda < > > > > kkal...@cloudera.com> wrote: > > > > > > > > > I have created SENTRY-2050 to investigate the situation. As a first > > > step > > > > > I'm reverting couple of commits and submit the patch to see how > > Jenkins > > > > > behaves. > > > > > > > > > > -Kalyan > > > > > > > > > > On Thu, Nov 16, 2017 at 1:27 PM, Sergio Pena < > > sergio.p...@cloudera.com > > > > > > > > > wrote: > > > > > > > > > > > There are several tests failing currently and seems they're > flaky. > > > > > They're > > > > > > passing locally but most of the time fail on Jenkins with some > > > > exceptions > > > > > > on some patches. > > > > > > > > > > > > We're looking at the old commits from the past week to see which > > > commit > > > > > is > > > > > > the culprit of this. > > > > > > > > > > > > - Sergio > > > > > > > > > > > > > > > > > > > > >
