Re: [DISCUSS] Unsustainable situation with ptests

[Siddharth Seth]

Very nice. There was an effort to get fast and green builds back in 2016.
There wasn't any strict "must be a green build" before commit at the time
though. Instead jiras were filed and the expectation was that they'd be
cited / new ones created pre commit(looking at the jiras now - this was
likely followed for a while, many fixes, and eventually got annoying?).
Think the enforcement step is absolutely required to get to, and maintain a
green build. May want to consider performance characteristics of tests as
well - must complete with X seconds.
Jiras for reference (including test infra improvements which were not done
at the time): HIVE-13503, HIVE-15058, HIVE-14547

This will be painful initially, but eventually it'll be great to be able to
commit without having to scan through a bunch of 'known failures', analyze,
document etc.

On Tue, May 15, 2018 at 5:30 PM, Prasanth Jayachandran <
pjayachand...@hortonworks.com> wrote:

> Wow! Awesome. This is the 3rd time I remember seeing green run in >4yrs. :)
>
> Thanks
> Prasanth
>
> > On May 15, 2018, at 5:28 PM, Jesus Camacho Rodriguez <
> jcama...@apache.org> wrote:
> >
> > We have just had the first clean run in a while:
> > https://builds.apache.org/job/PreCommit-HIVE-Build/10971/testReport/
> >
> > I will continue monitoring follow-up runs.
> >
> > Thanks,
> > -Jesús
> >
> >
> > On 5/14/18, 11:28 PM, "Prasanth Jayachandran" <
> pjayachand...@hortonworks.com> wrote:
> >
> >Wondering if we can add a state transition from “Patch Available” to
> “Ready To Commit” which can only be triggered by ptest bot on green test
> run.
> >
> >Thanks
> >Prasanth
> >
> >
> >
> >On Mon, May 14, 2018 at 10:44 PM -0700, "Jesus Camacho Rodriguez" <
> jcama...@apache.org> wrote:
> >
> >
> >I have been working on fixing this situation while commits were still
> coming in.
> >
> >All the tests that have been disabled are in:
> >https://issues.apache.org/jira/browse/HIVE-19509
> >I have created new issues to reenable each of them, they are linked
> to that issue.
> >Maybe I was slightly aggressive disabling some of the tests, however
> that seemed to be the only way to bring the tests failures with age count >
> 1 to zero.
> >
> >Instead of starting a vote to freeze the commits in another thread, I
> will start a vote to be stricter wrt committing to master, i.e., only
> commit if we get a clean QA run.
> >
> >We can discuss more about this issue over there.
> >
> >Thanks,
> >Jesús
> >
> >
> >
> >On 5/14/18, 4:11 PM, "Sergey Shelukhin"  wrote:
> >
> >Can we please make this freeze conditional, i.e. we unfreeze
> automatically
> >after ptest is clean (as evidenced by the clean HiveQA run on a
> given
> >JIRA).
> >
> >On 18/5/14, 15:16, "Alan Gates"  wrote:
> >
> >> We should do it in a separate thread so that people can see it with the
> >> [VOTE] subject.  Some people use that as a filter in their email to know
> >> when to pay attention to things.
> >>
> >> Alan.
> >>
> >> On Mon, May 14, 2018 at 2:36 PM, Prasanth Jayachandran <
> >> pjayachand...@hortonworks.com> wrote:
> >>
> >>> Will there be a separate voting thread? Or the voting on this thread is
> >>> sufficient for lock down?
> >>>
> >>> Thanks
> >>> Prasanth
> >>>
>  On May 14, 2018, at 2:34 PM, Alan Gates  wrote:
> 
>  ​I see there's support for this, but people are still pouring in
> >>> commits.
>  I proposed we have a quick vote on this to lock down the commits
> >>> until we
>  get to green.  That way everyone knows we have drawn the line at a
> >>> specific
>  point.  Any commits after that point would be reverted.  There isn't a
>  category in the bylaws that fits this kind of vote but I suggest lazy
>  majority as the most appropriate one (at least 3 votes, more +1s than
>  -1s).
> 
>  Alan.​
> 
>  On Mon, May 14, 2018 at 10:34 AM, Vihang Karajgaonkar <
> >>> vih...@cloudera.com>
>  wrote:
> 
> > I worked on a few quick-fix optimizations in Ptest infrastructure
> >>> over
> >>> the
> > weekend which reduced the execution run from ~90 min to ~70 min per
> >>> run. I
> > had to restart Ptest multiple times. I was resubmitting the patches
> >>> which
> > were in the queue manually, but I may have missed a few. In case you
> >>> have a
> > patch which is pending pre-commit and you don't see it in the queue,
> >>> please
> > submit it manually or let me know if you don't have access to the
> >>> jenkins
> > job. I will continue to work on the sub-tasks in HIVE-19425 and will
> >>> do
> > some maintenance next weekend as well.
> >
> > On Mon, May 14, 2018 at 7:42 AM, Jesus Camacho Rodriguez <
> > jcama...@apache.org> wrote:
> >
> >> Vineet has already been working on disabling those tests that were
> >>> timing
> >> out. I am working on disabling those that are generating different q
> > 

Re: [DISCUSS] Unsustainable situation with ptests

[Prasanth Jayachandran]

Wow! Awesome. This is the 3rd time I remember seeing green run in >4yrs. :)

Thanks
Prasanth

> On May 15, 2018, at 5:28 PM, Jesus Camacho Rodriguez  
> wrote:
> 
> We have just had the first clean run in a while:
> https://builds.apache.org/job/PreCommit-HIVE-Build/10971/testReport/
> 
> I will continue monitoring follow-up runs.
> 
> Thanks,
> -Jesús
> 
> 
> On 5/14/18, 11:28 PM, "Prasanth Jayachandran" 
>  wrote:
> 
>Wondering if we can add a state transition from “Patch Available” to 
> “Ready To Commit” which can only be triggered by ptest bot on green test run.
> 
>Thanks
>Prasanth
> 
> 
> 
>On Mon, May 14, 2018 at 10:44 PM -0700, "Jesus Camacho Rodriguez" 
> > wrote:
> 
> 
>I have been working on fixing this situation while commits were still 
> coming in.
> 
>All the tests that have been disabled are in:
>https://issues.apache.org/jira/browse/HIVE-19509
>I have created new issues to reenable each of them, they are linked to 
> that issue.
>Maybe I was slightly aggressive disabling some of the tests, however that 
> seemed to be the only way to bring the tests failures with age count > 1 to 
> zero.
> 
>Instead of starting a vote to freeze the commits in another thread, I will 
> start a vote to be stricter wrt committing to master, i.e., only commit if we 
> get a clean QA run.
> 
>We can discuss more about this issue over there.
> 
>Thanks,
>Jesús
> 
> 
> 
>On 5/14/18, 4:11 PM, "Sergey Shelukhin"  wrote:
> 
>Can we please make this freeze conditional, i.e. we unfreeze 
> automatically
>after ptest is clean (as evidenced by the clean HiveQA run on a given
>JIRA).
> 
>On 18/5/14, 15:16, "Alan Gates"  wrote:
> 
>> We should do it in a separate thread so that people can see it with the
>> [VOTE] subject.  Some people use that as a filter in their email to know
>> when to pay attention to things.
>> 
>> Alan.
>> 
>> On Mon, May 14, 2018 at 2:36 PM, Prasanth Jayachandran <
>> pjayachand...@hortonworks.com> wrote:
>> 
>>> Will there be a separate voting thread? Or the voting on this thread is
>>> sufficient for lock down?
>>> 
>>> Thanks
>>> Prasanth
>>> 
 On May 14, 2018, at 2:34 PM, Alan Gates  wrote:
 
 ​I see there's support for this, but people are still pouring in
>>> commits.
 I proposed we have a quick vote on this to lock down the commits
>>> until we
 get to green.  That way everyone knows we have drawn the line at a
>>> specific
 point.  Any commits after that point would be reverted.  There isn't a
 category in the bylaws that fits this kind of vote but I suggest lazy
 majority as the most appropriate one (at least 3 votes, more +1s than
 -1s).
 
 Alan.​
 
 On Mon, May 14, 2018 at 10:34 AM, Vihang Karajgaonkar <
>>> vih...@cloudera.com>
 wrote:
 
> I worked on a few quick-fix optimizations in Ptest infrastructure
>>> over
>>> the
> weekend which reduced the execution run from ~90 min to ~70 min per
>>> run. I
> had to restart Ptest multiple times. I was resubmitting the patches
>>> which
> were in the queue manually, but I may have missed a few. In case you
>>> have a
> patch which is pending pre-commit and you don't see it in the queue,
>>> please
> submit it manually or let me know if you don't have access to the
>>> jenkins
> job. I will continue to work on the sub-tasks in HIVE-19425 and will
>>> do
> some maintenance next weekend as well.
> 
> On Mon, May 14, 2018 at 7:42 AM, Jesus Camacho Rodriguez <
> jcama...@apache.org> wrote:
> 
>> Vineet has already been working on disabling those tests that were
>>> timing
>> out. I am working on disabling those that are generating different q
> files
>> consistently for last ptests n runs. I am keeping track of all these
> tests
>> in https://issues.apache.org/jira/browse/HIVE-19509.
>> 
>> -Jesús
>> 
>> On 5/14/18, 2:25 AM, "Prasanth Jayachandran" <
>> pjayachand...@hortonworks.com> wrote:
>> 
>>   +1 on freezing commits until we get repetitive green tests. We
>>> should
>> probably disable (and remember in a jira to reenable then at later
>>> point)
>> tests that are flaky to get repetitive green test runs.
>> 
>>   Thanks
>>   Prasanth
>> 
>> 
>> 
>>   On Mon, May 14, 2018 at 2:15 AM -0700, "Rui Li" <
> lirui.fu...@gmail.com
>>> wrote:
>> 
>> 
>>   +1 to freezing commits until we stabilize
>> 
>>   On Sat, May 12, 2018 at 6:10 AM, Vihang Karajgaonkar
>>   wrote:
>> 
>>> In order to understand the end-to-end precommit flow I would like
> to
>> get
>>> access to the PreCommit-HIVE-Build jenkins script. Does anyone one
>> know how
>>> can I get that?
>>> 
>>> On Fri, May 11, 2018 at 2:03 PM, Jesus 

Re: [DISCUSS] Unsustainable situation with ptests

[Jesus Camacho Rodriguez]

We have just had the first clean run in a while:
https://builds.apache.org/job/PreCommit-HIVE-Build/10971/testReport/

I will continue monitoring follow-up runs.

Thanks,
-Jesús


On 5/14/18, 11:28 PM, "Prasanth Jayachandran"  
wrote:

Wondering if we can add a state transition from “Patch Available” to “Ready 
To Commit” which can only be triggered by ptest bot on green test run.

Thanks
Prasanth



On Mon, May 14, 2018 at 10:44 PM -0700, "Jesus Camacho Rodriguez" 
> wrote:


I have been working on fixing this situation while commits were still 
coming in.

All the tests that have been disabled are in:
https://issues.apache.org/jira/browse/HIVE-19509
I have created new issues to reenable each of them, they are linked to that 
issue.
Maybe I was slightly aggressive disabling some of the tests, however that 
seemed to be the only way to bring the tests failures with age count > 1 to 
zero.

Instead of starting a vote to freeze the commits in another thread, I will 
start a vote to be stricter wrt committing to master, i.e., only commit if we 
get a clean QA run.

We can discuss more about this issue over there.

Thanks,
Jesús



On 5/14/18, 4:11 PM, "Sergey Shelukhin"  wrote:

Can we please make this freeze conditional, i.e. we unfreeze 
automatically
after ptest is clean (as evidenced by the clean HiveQA run on a given
JIRA).

On 18/5/14, 15:16, "Alan Gates"  wrote:

>We should do it in a separate thread so that people can see it with the
>[VOTE] subject.  Some people use that as a filter in their email to 
know
>when to pay attention to things.
>
>Alan.
>
>On Mon, May 14, 2018 at 2:36 PM, Prasanth Jayachandran <
>pjayachand...@hortonworks.com> wrote:
>
>> Will there be a separate voting thread? Or the voting on this thread 
is
>> sufficient for lock down?
>>
>> Thanks
>> Prasanth
>>
>> > On May 14, 2018, at 2:34 PM, Alan Gates  wrote:
>> >
>> > ​I see there's support for this, but people are still pouring in
>>commits.
>> > I proposed we have a quick vote on this to lock down the commits
>>until we
>> > get to green.  That way everyone knows we have drawn the line at a
>> specific
>> > point.  Any commits after that point would be reverted.  There 
isn't a
>> > category in the bylaws that fits this kind of vote but I suggest 
lazy
>> > majority as the most appropriate one (at least 3 votes, more +1s 
than
>> > -1s).
>> >
>> > Alan.​
>> >
>> > On Mon, May 14, 2018 at 10:34 AM, Vihang Karajgaonkar <
>> vih...@cloudera.com>
>> > wrote:
>> >
>> >> I worked on a few quick-fix optimizations in Ptest infrastructure
>>over
>> the
>> >> weekend which reduced the execution run from ~90 min to ~70 min 
per
>> run. I
>> >> had to restart Ptest multiple times. I was resubmitting the 
patches
>> which
>> >> were in the queue manually, but I may have missed a few. In case 
you
>> have a
>> >> patch which is pending pre-commit and you don't see it in the 
queue,
>> please
>> >> submit it manually or let me know if you don't have access to the
>> jenkins
>> >> job. I will continue to work on the sub-tasks in HIVE-19425 and 
will
>>do
>> >> some maintenance next weekend as well.
>> >>
>> >> On Mon, May 14, 2018 at 7:42 AM, Jesus Camacho Rodriguez <
>> >> jcama...@apache.org> wrote:
>> >>
>> >>> Vineet has already been working on disabling those tests that 
were
>> timing
>> >>> out. I am working on disabling those that are generating 
different q
>> >> files
>> >>> consistently for last ptests n runs. I am keeping track of all 
these
>> >> tests
>> >>> in https://issues.apache.org/jira/browse/HIVE-19509.
>> >>>
>> >>> -Jesús
>> >>>
>> >>> On 5/14/18, 2:25 AM, "Prasanth Jayachandran" <
>> >>> pjayachand...@hortonworks.com> wrote:
>> >>>
>> >>>+1 on freezing commits until we get repetitive green tests. We
>> should
>> >>> probably disable (and remember in a jira to reenable then at 
later
>> point)
>> >>> tests that are flaky to get repetitive green test runs.
>> >>>
>> >>>Thanks
>> >>>Prasanth
>> >>>
>> >>>
>> >>>
>> >>>On Mon, May 14, 2018 at 2:15 AM -0700, "Rui Li" <
>> >> lirui.fu...@gmail.com
>> >>> > wrote:
>> >>>
>> >>>
   

Re: [DISCUSS] Unsustainable situation with ptests

[Prasanth Jayachandran]

Wondering if we can add a state transition from “Patch Available” to “Ready To 
Commit” which can only be triggered by ptest bot on green test run.

Thanks
Prasanth



On Mon, May 14, 2018 at 10:44 PM -0700, "Jesus Camacho Rodriguez" 
> wrote:


I have been working on fixing this situation while commits were still coming in.

All the tests that have been disabled are in:
https://issues.apache.org/jira/browse/HIVE-19509
I have created new issues to reenable each of them, they are linked to that 
issue.
Maybe I was slightly aggressive disabling some of the tests, however that 
seemed to be the only way to bring the tests failures with age count > 1 to 
zero.

Instead of starting a vote to freeze the commits in another thread, I will 
start a vote to be stricter wrt committing to master, i.e., only commit if we 
get a clean QA run.

We can discuss more about this issue over there.

Thanks,
Jesús



On 5/14/18, 4:11 PM, "Sergey Shelukhin"  wrote:

Can we please make this freeze conditional, i.e. we unfreeze automatically
after ptest is clean (as evidenced by the clean HiveQA run on a given
JIRA).

On 18/5/14, 15:16, "Alan Gates"  wrote:

>We should do it in a separate thread so that people can see it with the
>[VOTE] subject.  Some people use that as a filter in their email to know
>when to pay attention to things.
>
>Alan.
>
>On Mon, May 14, 2018 at 2:36 PM, Prasanth Jayachandran <
>pjayachand...@hortonworks.com> wrote:
>
>> Will there be a separate voting thread? Or the voting on this thread is
>> sufficient for lock down?
>>
>> Thanks
>> Prasanth
>>
>> > On May 14, 2018, at 2:34 PM, Alan Gates  wrote:
>> >
>> > ​I see there's support for this, but people are still pouring in
>>commits.
>> > I proposed we have a quick vote on this to lock down the commits
>>until we
>> > get to green.  That way everyone knows we have drawn the line at a
>> specific
>> > point.  Any commits after that point would be reverted.  There isn't a
>> > category in the bylaws that fits this kind of vote but I suggest lazy
>> > majority as the most appropriate one (at least 3 votes, more +1s than
>> > -1s).
>> >
>> > Alan.​
>> >
>> > On Mon, May 14, 2018 at 10:34 AM, Vihang Karajgaonkar <
>> vih...@cloudera.com>
>> > wrote:
>> >
>> >> I worked on a few quick-fix optimizations in Ptest infrastructure
>>over
>> the
>> >> weekend which reduced the execution run from ~90 min to ~70 min per
>> run. I
>> >> had to restart Ptest multiple times. I was resubmitting the patches
>> which
>> >> were in the queue manually, but I may have missed a few. In case you
>> have a
>> >> patch which is pending pre-commit and you don't see it in the queue,
>> please
>> >> submit it manually or let me know if you don't have access to the
>> jenkins
>> >> job. I will continue to work on the sub-tasks in HIVE-19425 and will
>>do
>> >> some maintenance next weekend as well.
>> >>
>> >> On Mon, May 14, 2018 at 7:42 AM, Jesus Camacho Rodriguez <
>> >> jcama...@apache.org> wrote:
>> >>
>> >>> Vineet has already been working on disabling those tests that were
>> timing
>> >>> out. I am working on disabling those that are generating different q
>> >> files
>> >>> consistently for last ptests n runs. I am keeping track of all these
>> >> tests
>> >>> in https://issues.apache.org/jira/browse/HIVE-19509.
>> >>>
>> >>> -Jesús
>> >>>
>> >>> On 5/14/18, 2:25 AM, "Prasanth Jayachandran" <
>> >>> pjayachand...@hortonworks.com> wrote:
>> >>>
>> >>>+1 on freezing commits until we get repetitive green tests. We
>> should
>> >>> probably disable (and remember in a jira to reenable then at later
>> point)
>> >>> tests that are flaky to get repetitive green test runs.
>> >>>
>> >>>Thanks
>> >>>Prasanth
>> >>>
>> >>>
>> >>>
>> >>>On Mon, May 14, 2018 at 2:15 AM -0700, "Rui Li" <
>> >> lirui.fu...@gmail.com
>> >>> > wrote:
>> >>>
>> >>>
>> >>>+1 to freezing commits until we stabilize
>> >>>
>> >>>On Sat, May 12, 2018 at 6:10 AM, Vihang Karajgaonkar
>> >>>wrote:
>> >>>
>>  In order to understand the end-to-end precommit flow I would like
>> >> to
>> >>> get
>>  access to the PreCommit-HIVE-Build jenkins script. Does anyone one
>> >>> know how
>>  can I get that?
>> 
>>  On Fri, May 11, 2018 at 2:03 PM, Jesus Camacho Rodriguez <
>>  jcama...@apache.org> wrote:
>> 
>> > Bq. For the short term green runs, I think we should @Ignore the
>> >>> tests
>> > which
>> > are known to be failing since many runs. They are anyways not
>> >> being
>> > 

Re: [DISCUSS] Unsustainable situation with ptests

[Jesus Camacho Rodriguez]

I have been working on fixing this situation while commits were still coming in.

All the tests that have been disabled are in:
https://issues.apache.org/jira/browse/HIVE-19509
I have created new issues to reenable each of them, they are linked to that 
issue.
Maybe I was slightly aggressive disabling some of the tests, however that 
seemed to be the only way to bring the tests failures with age count > 1 to 
zero.

Instead of starting a vote to freeze the commits in another thread, I will 
start a vote to be stricter wrt committing to master, i.e., only commit if we 
get a clean QA run.

We can discuss more about this issue over there.

Thanks,
Jesús



On 5/14/18, 4:11 PM, "Sergey Shelukhin"  wrote:

Can we please make this freeze conditional, i.e. we unfreeze automatically
after ptest is clean (as evidenced by the clean HiveQA run on a given
JIRA).

On 18/5/14, 15:16, "Alan Gates"  wrote:

>We should do it in a separate thread so that people can see it with the
>[VOTE] subject.  Some people use that as a filter in their email to know
>when to pay attention to things.
>
>Alan.
>
>On Mon, May 14, 2018 at 2:36 PM, Prasanth Jayachandran <
>pjayachand...@hortonworks.com> wrote:
>
>> Will there be a separate voting thread? Or the voting on this thread is
>> sufficient for lock down?
>>
>> Thanks
>> Prasanth
>>
>> > On May 14, 2018, at 2:34 PM, Alan Gates  wrote:
>> >
>> > ​I see there's support for this, but people are still pouring in
>>commits.
>> > I proposed we have a quick vote on this to lock down the commits
>>until we
>> > get to green.  That way everyone knows we have drawn the line at a
>> specific
>> > point.  Any commits after that point would be reverted.  There isn't a
>> > category in the bylaws that fits this kind of vote but I suggest lazy
>> > majority as the most appropriate one (at least 3 votes, more +1s than
>> > -1s).
>> >
>> > Alan.​
>> >
>> > On Mon, May 14, 2018 at 10:34 AM, Vihang Karajgaonkar <
>> vih...@cloudera.com>
>> > wrote:
>> >
>> >> I worked on a few quick-fix optimizations in Ptest infrastructure
>>over
>> the
>> >> weekend which reduced the execution run from ~90 min to ~70 min per
>> run. I
>> >> had to restart Ptest multiple times. I was resubmitting the patches
>> which
>> >> were in the queue manually, but I may have missed a few. In case you
>> have a
>> >> patch which is pending pre-commit and you don't see it in the queue,
>> please
>> >> submit it manually or let me know if you don't have access to the
>> jenkins
>> >> job. I will continue to work on the sub-tasks in HIVE-19425 and will
>>do
>> >> some maintenance next weekend as well.
>> >>
>> >> On Mon, May 14, 2018 at 7:42 AM, Jesus Camacho Rodriguez <
>> >> jcama...@apache.org> wrote:
>> >>
>> >>> Vineet has already been working on disabling those tests that were
>> timing
>> >>> out. I am working on disabling those that are generating different q
>> >> files
>> >>> consistently for last ptests n runs. I am keeping track of all these
>> >> tests
>> >>> in https://issues.apache.org/jira/browse/HIVE-19509.
>> >>>
>> >>> -Jesús
>> >>>
>> >>> On 5/14/18, 2:25 AM, "Prasanth Jayachandran" <
>> >>> pjayachand...@hortonworks.com> wrote:
>> >>>
>> >>>+1 on freezing commits until we get repetitive green tests. We
>> should
>> >>> probably disable (and remember in a jira to reenable then at later
>> point)
>> >>> tests that are flaky to get repetitive green test runs.
>> >>>
>> >>>Thanks
>> >>>Prasanth
>> >>>
>> >>>
>> >>>
>> >>>On Mon, May 14, 2018 at 2:15 AM -0700, "Rui Li" <
>> >> lirui.fu...@gmail.com
>> >>> > wrote:
>> >>>
>> >>>
>> >>>+1 to freezing commits until we stabilize
>> >>>
>> >>>On Sat, May 12, 2018 at 6:10 AM, Vihang Karajgaonkar
>> >>>wrote:
>> >>>
>>  In order to understand the end-to-end precommit flow I would like
>> >> to
>> >>> get
>>  access to the PreCommit-HIVE-Build jenkins script. Does anyone one
>> >>> know how
>>  can I get that?
>> 
>>  On Fri, May 11, 2018 at 2:03 PM, Jesus Camacho Rodriguez <
>>  jcama...@apache.org> wrote:
>> 
>> > Bq. For the short term green runs, I think we should @Ignore the
>> >>> tests
>> > which
>> > are known to be failing since many runs. They are anyways not
>> >> being
>> > addressed as such. If people think they are important to be run
>> >> we
>> >>> should
>> > fix them and only then re-enable them.
>> >
>> > I think that is 

Re: [DISCUSS] Unsustainable situation with ptests

[Sergey Shelukhin]

Can we please make this freeze conditional, i.e. we unfreeze automatically
after ptest is clean (as evidenced by the clean HiveQA run on a given
JIRA).

On 18/5/14, 15:16, "Alan Gates"  wrote:

>We should do it in a separate thread so that people can see it with the
>[VOTE] subject.  Some people use that as a filter in their email to know
>when to pay attention to things.
>
>Alan.
>
>On Mon, May 14, 2018 at 2:36 PM, Prasanth Jayachandran <
>pjayachand...@hortonworks.com> wrote:
>
>> Will there be a separate voting thread? Or the voting on this thread is
>> sufficient for lock down?
>>
>> Thanks
>> Prasanth
>>
>> > On May 14, 2018, at 2:34 PM, Alan Gates  wrote:
>> >
>> > ​I see there's support for this, but people are still pouring in
>>commits.
>> > I proposed we have a quick vote on this to lock down the commits
>>until we
>> > get to green.  That way everyone knows we have drawn the line at a
>> specific
>> > point.  Any commits after that point would be reverted.  There isn't a
>> > category in the bylaws that fits this kind of vote but I suggest lazy
>> > majority as the most appropriate one (at least 3 votes, more +1s than
>> > -1s).
>> >
>> > Alan.​
>> >
>> > On Mon, May 14, 2018 at 10:34 AM, Vihang Karajgaonkar <
>> vih...@cloudera.com>
>> > wrote:
>> >
>> >> I worked on a few quick-fix optimizations in Ptest infrastructure
>>over
>> the
>> >> weekend which reduced the execution run from ~90 min to ~70 min per
>> run. I
>> >> had to restart Ptest multiple times. I was resubmitting the patches
>> which
>> >> were in the queue manually, but I may have missed a few. In case you
>> have a
>> >> patch which is pending pre-commit and you don't see it in the queue,
>> please
>> >> submit it manually or let me know if you don't have access to the
>> jenkins
>> >> job. I will continue to work on the sub-tasks in HIVE-19425 and will
>>do
>> >> some maintenance next weekend as well.
>> >>
>> >> On Mon, May 14, 2018 at 7:42 AM, Jesus Camacho Rodriguez <
>> >> jcama...@apache.org> wrote:
>> >>
>> >>> Vineet has already been working on disabling those tests that were
>> timing
>> >>> out. I am working on disabling those that are generating different q
>> >> files
>> >>> consistently for last ptests n runs. I am keeping track of all these
>> >> tests
>> >>> in https://issues.apache.org/jira/browse/HIVE-19509.
>> >>>
>> >>> -Jesús
>> >>>
>> >>> On 5/14/18, 2:25 AM, "Prasanth Jayachandran" <
>> >>> pjayachand...@hortonworks.com> wrote:
>> >>>
>> >>>+1 on freezing commits until we get repetitive green tests. We
>> should
>> >>> probably disable (and remember in a jira to reenable then at later
>> point)
>> >>> tests that are flaky to get repetitive green test runs.
>> >>>
>> >>>Thanks
>> >>>Prasanth
>> >>>
>> >>>
>> >>>
>> >>>On Mon, May 14, 2018 at 2:15 AM -0700, "Rui Li" <
>> >> lirui.fu...@gmail.com
>> >>> > wrote:
>> >>>
>> >>>
>> >>>+1 to freezing commits until we stabilize
>> >>>
>> >>>On Sat, May 12, 2018 at 6:10 AM, Vihang Karajgaonkar
>> >>>wrote:
>> >>>
>>  In order to understand the end-to-end precommit flow I would like
>> >> to
>> >>> get
>>  access to the PreCommit-HIVE-Build jenkins script. Does anyone one
>> >>> know how
>>  can I get that?
>> 
>>  On Fri, May 11, 2018 at 2:03 PM, Jesus Camacho Rodriguez <
>>  jcama...@apache.org> wrote:
>> 
>> > Bq. For the short term green runs, I think we should @Ignore the
>> >>> tests
>> > which
>> > are known to be failing since many runs. They are anyways not
>> >> being
>> > addressed as such. If people think they are important to be run
>> >> we
>> >>> should
>> > fix them and only then re-enable them.
>> >
>> > I think that is a good idea, as we would minimize the time that
>> >> we
>> >>> halt
>> > development. We can create a JIRA where we list all tests that
>> >> were
>> > failing, and we have disabled to get the clean run. From that
>> >>> moment, we
>> > will have zero tolerance towards committing with failing tests.
>> >>> And we
>>  need
>> > to pick up those tests that should not be ignored and bring them
>> >>> up again
>> > but passing. If there is no disagreement, I can start working on
>> >>> that.
>> >
>> > Once I am done, I can try to help with infra tickets too.
>> >
>> > -Jesús
>> >
>> >
>> > On 5/11/18, 1:57 PM, "Vineet Garg"  wrote:
>> >
>> >+1. I strongly vote for freezing commits and getting our
>> >>> testing
>> > coverage in acceptable state.  We have been struggling to
>> >> stabilize
>> > branch-3 due to test failures and releasing Hive 3.0 in current
>> >>> state
>>  would
>> > be unacceptable.
>> >
>> >Currently there are quite a few test suites which are not
>> >> even
>>  running
>> > and are being timed out. We have been committing patches (to both
>>  branch-3
>> > 

Re: [DISCUSS] Unsustainable situation with ptests

[Alan Gates]

We should do it in a separate thread so that people can see it with the
[VOTE] subject.  Some people use that as a filter in their email to know
when to pay attention to things.

Alan.

On Mon, May 14, 2018 at 2:36 PM, Prasanth Jayachandran <
pjayachand...@hortonworks.com> wrote:

> Will there be a separate voting thread? Or the voting on this thread is
> sufficient for lock down?
>
> Thanks
> Prasanth
>
> > On May 14, 2018, at 2:34 PM, Alan Gates  wrote:
> >
> > ​I see there's support for this, but people are still pouring in commits.
> > I proposed we have a quick vote on this to lock down the commits until we
> > get to green.  That way everyone knows we have drawn the line at a
> specific
> > point.  Any commits after that point would be reverted.  There isn't a
> > category in the bylaws that fits this kind of vote but I suggest lazy
> > majority as the most appropriate one (at least 3 votes, more +1s than
> > -1s).
> >
> > Alan.​
> >
> > On Mon, May 14, 2018 at 10:34 AM, Vihang Karajgaonkar <
> vih...@cloudera.com>
> > wrote:
> >
> >> I worked on a few quick-fix optimizations in Ptest infrastructure over
> the
> >> weekend which reduced the execution run from ~90 min to ~70 min per
> run. I
> >> had to restart Ptest multiple times. I was resubmitting the patches
> which
> >> were in the queue manually, but I may have missed a few. In case you
> have a
> >> patch which is pending pre-commit and you don't see it in the queue,
> please
> >> submit it manually or let me know if you don't have access to the
> jenkins
> >> job. I will continue to work on the sub-tasks in HIVE-19425 and will do
> >> some maintenance next weekend as well.
> >>
> >> On Mon, May 14, 2018 at 7:42 AM, Jesus Camacho Rodriguez <
> >> jcama...@apache.org> wrote:
> >>
> >>> Vineet has already been working on disabling those tests that were
> timing
> >>> out. I am working on disabling those that are generating different q
> >> files
> >>> consistently for last ptests n runs. I am keeping track of all these
> >> tests
> >>> in https://issues.apache.org/jira/browse/HIVE-19509.
> >>>
> >>> -Jesús
> >>>
> >>> On 5/14/18, 2:25 AM, "Prasanth Jayachandran" <
> >>> pjayachand...@hortonworks.com> wrote:
> >>>
> >>>+1 on freezing commits until we get repetitive green tests. We
> should
> >>> probably disable (and remember in a jira to reenable then at later
> point)
> >>> tests that are flaky to get repetitive green test runs.
> >>>
> >>>Thanks
> >>>Prasanth
> >>>
> >>>
> >>>
> >>>On Mon, May 14, 2018 at 2:15 AM -0700, "Rui Li" <
> >> lirui.fu...@gmail.com
> >>> > wrote:
> >>>
> >>>
> >>>+1 to freezing commits until we stabilize
> >>>
> >>>On Sat, May 12, 2018 at 6:10 AM, Vihang Karajgaonkar
> >>>wrote:
> >>>
>  In order to understand the end-to-end precommit flow I would like
> >> to
> >>> get
>  access to the PreCommit-HIVE-Build jenkins script. Does anyone one
> >>> know how
>  can I get that?
> 
>  On Fri, May 11, 2018 at 2:03 PM, Jesus Camacho Rodriguez <
>  jcama...@apache.org> wrote:
> 
> > Bq. For the short term green runs, I think we should @Ignore the
> >>> tests
> > which
> > are known to be failing since many runs. They are anyways not
> >> being
> > addressed as such. If people think they are important to be run
> >> we
> >>> should
> > fix them and only then re-enable them.
> >
> > I think that is a good idea, as we would minimize the time that
> >> we
> >>> halt
> > development. We can create a JIRA where we list all tests that
> >> were
> > failing, and we have disabled to get the clean run. From that
> >>> moment, we
> > will have zero tolerance towards committing with failing tests.
> >>> And we
>  need
> > to pick up those tests that should not be ignored and bring them
> >>> up again
> > but passing. If there is no disagreement, I can start working on
> >>> that.
> >
> > Once I am done, I can try to help with infra tickets too.
> >
> > -Jesús
> >
> >
> > On 5/11/18, 1:57 PM, "Vineet Garg"  wrote:
> >
> >+1. I strongly vote for freezing commits and getting our
> >>> testing
> > coverage in acceptable state.  We have been struggling to
> >> stabilize
> > branch-3 due to test failures and releasing Hive 3.0 in current
> >>> state
>  would
> > be unacceptable.
> >
> >Currently there are quite a few test suites which are not
> >> even
>  running
> > and are being timed out. We have been committing patches (to both
>  branch-3
> > and master) without test coverage for these tests.
> >We should immediately figure out what’s going on before we
> >>> proceed
> > with commits.
> >
> >For reference following test suites are timing out on
> >> master: (
> > https://issues.apache.org/jira/browse/HIVE-19506)
> >
> >
> >TestDbNotificationListener - did 

Re: [DISCUSS] Unsustainable situation with ptests

[Prasanth Jayachandran]

Will there be a separate voting thread? Or the voting on this thread is 
sufficient for lock down?

Thanks
Prasanth

> On May 14, 2018, at 2:34 PM, Alan Gates  wrote:
> 
> ​I see there's support for this, but people are still pouring in commits.
> I proposed we have a quick vote on this to lock down the commits until we
> get to green.  That way everyone knows we have drawn the line at a specific
> point.  Any commits after that point would be reverted.  There isn't a
> category in the bylaws that fits this kind of vote but I suggest lazy
> majority as the most appropriate one (at least 3 votes, more +1s than
> -1s).
> 
> Alan.​
> 
> On Mon, May 14, 2018 at 10:34 AM, Vihang Karajgaonkar 
> wrote:
> 
>> I worked on a few quick-fix optimizations in Ptest infrastructure over the
>> weekend which reduced the execution run from ~90 min to ~70 min per run. I
>> had to restart Ptest multiple times. I was resubmitting the patches which
>> were in the queue manually, but I may have missed a few. In case you have a
>> patch which is pending pre-commit and you don't see it in the queue, please
>> submit it manually or let me know if you don't have access to the jenkins
>> job. I will continue to work on the sub-tasks in HIVE-19425 and will do
>> some maintenance next weekend as well.
>> 
>> On Mon, May 14, 2018 at 7:42 AM, Jesus Camacho Rodriguez <
>> jcama...@apache.org> wrote:
>> 
>>> Vineet has already been working on disabling those tests that were timing
>>> out. I am working on disabling those that are generating different q
>> files
>>> consistently for last ptests n runs. I am keeping track of all these
>> tests
>>> in https://issues.apache.org/jira/browse/HIVE-19509.
>>> 
>>> -Jesús
>>> 
>>> On 5/14/18, 2:25 AM, "Prasanth Jayachandran" <
>>> pjayachand...@hortonworks.com> wrote:
>>> 
>>>+1 on freezing commits until we get repetitive green tests. We should
>>> probably disable (and remember in a jira to reenable then at later point)
>>> tests that are flaky to get repetitive green test runs.
>>> 
>>>Thanks
>>>Prasanth
>>> 
>>> 
>>> 
>>>On Mon, May 14, 2018 at 2:15 AM -0700, "Rui Li" <
>> lirui.fu...@gmail.com
>>> > wrote:
>>> 
>>> 
>>>+1 to freezing commits until we stabilize
>>> 
>>>On Sat, May 12, 2018 at 6:10 AM, Vihang Karajgaonkar
>>>wrote:
>>> 
 In order to understand the end-to-end precommit flow I would like
>> to
>>> get
 access to the PreCommit-HIVE-Build jenkins script. Does anyone one
>>> know how
 can I get that?
 
 On Fri, May 11, 2018 at 2:03 PM, Jesus Camacho Rodriguez <
 jcama...@apache.org> wrote:
 
> Bq. For the short term green runs, I think we should @Ignore the
>>> tests
> which
> are known to be failing since many runs. They are anyways not
>> being
> addressed as such. If people think they are important to be run
>> we
>>> should
> fix them and only then re-enable them.
> 
> I think that is a good idea, as we would minimize the time that
>> we
>>> halt
> development. We can create a JIRA where we list all tests that
>> were
> failing, and we have disabled to get the clean run. From that
>>> moment, we
> will have zero tolerance towards committing with failing tests.
>>> And we
 need
> to pick up those tests that should not be ignored and bring them
>>> up again
> but passing. If there is no disagreement, I can start working on
>>> that.
> 
> Once I am done, I can try to help with infra tickets too.
> 
> -Jesús
> 
> 
> On 5/11/18, 1:57 PM, "Vineet Garg"  wrote:
> 
>+1. I strongly vote for freezing commits and getting our
>>> testing
> coverage in acceptable state.  We have been struggling to
>> stabilize
> branch-3 due to test failures and releasing Hive 3.0 in current
>>> state
 would
> be unacceptable.
> 
>Currently there are quite a few test suites which are not
>> even
 running
> and are being timed out. We have been committing patches (to both
 branch-3
> and master) without test coverage for these tests.
>We should immediately figure out what’s going on before we
>>> proceed
> with commits.
> 
>For reference following test suites are timing out on
>> master: (
> https://issues.apache.org/jira/browse/HIVE-19506)
> 
> 
>TestDbNotificationListener - did not produce a TEST-*.xml
>> file
 (likely
> timed out)
> 
>TestHCatHiveCompatibility - did not produce a TEST-*.xml file
>>> (likely
> timed out)
> 
>TestNegativeCliDriver - did not produce a TEST-*.xml file
>>> (likely
> timed out)
> 
>TestNonCatCallsWithCatalog - did not produce a TEST-*.xml
>> file
 (likely
> timed out)
> 
>TestSequenceFileReadWrite - did not produce a TEST-*.xml file
>>> (likely
> timed out)
> 
>TestTxnExIm - did not 

Re: [DISCUSS] Unsustainable situation with ptests

[Alan Gates]

​I see there's support for this, but people are still pouring in commits.
I proposed we have a quick vote on this to lock down the commits until we
get to green.  That way everyone knows we have drawn the line at a specific
point.  Any commits after that point would be reverted.  There isn't a
category in the bylaws that fits this kind of vote but I suggest lazy
majority as the most appropriate one (at least 3 votes, more +1s than
-1s).

Alan.​

On Mon, May 14, 2018 at 10:34 AM, Vihang Karajgaonkar 
wrote:

> I worked on a few quick-fix optimizations in Ptest infrastructure over the
> weekend which reduced the execution run from ~90 min to ~70 min per run. I
> had to restart Ptest multiple times. I was resubmitting the patches which
> were in the queue manually, but I may have missed a few. In case you have a
> patch which is pending pre-commit and you don't see it in the queue, please
> submit it manually or let me know if you don't have access to the jenkins
> job. I will continue to work on the sub-tasks in HIVE-19425 and will do
> some maintenance next weekend as well.
>
> On Mon, May 14, 2018 at 7:42 AM, Jesus Camacho Rodriguez <
> jcama...@apache.org> wrote:
>
> > Vineet has already been working on disabling those tests that were timing
> > out. I am working on disabling those that are generating different q
> files
> > consistently for last ptests n runs. I am keeping track of all these
> tests
> > in https://issues.apache.org/jira/browse/HIVE-19509.
> >
> > -Jesús
> >
> > On 5/14/18, 2:25 AM, "Prasanth Jayachandran" <
> > pjayachand...@hortonworks.com> wrote:
> >
> > +1 on freezing commits until we get repetitive green tests. We should
> > probably disable (and remember in a jira to reenable then at later point)
> > tests that are flaky to get repetitive green test runs.
> >
> > Thanks
> > Prasanth
> >
> >
> >
> > On Mon, May 14, 2018 at 2:15 AM -0700, "Rui Li" <
> lirui.fu...@gmail.com
> > > wrote:
> >
> >
> > +1 to freezing commits until we stabilize
> >
> > On Sat, May 12, 2018 at 6:10 AM, Vihang Karajgaonkar
> > wrote:
> >
> > > In order to understand the end-to-end precommit flow I would like
> to
> > get
> > > access to the PreCommit-HIVE-Build jenkins script. Does anyone one
> > know how
> > > can I get that?
> > >
> > > On Fri, May 11, 2018 at 2:03 PM, Jesus Camacho Rodriguez <
> > > jcama...@apache.org> wrote:
> > >
> > > > Bq. For the short term green runs, I think we should @Ignore the
> > tests
> > > > which
> > > > are known to be failing since many runs. They are anyways not
> being
> > > > addressed as such. If people think they are important to be run
> we
> > should
> > > > fix them and only then re-enable them.
> > > >
> > > > I think that is a good idea, as we would minimize the time that
> we
> > halt
> > > > development. We can create a JIRA where we list all tests that
> were
> > > > failing, and we have disabled to get the clean run. From that
> > moment, we
> > > > will have zero tolerance towards committing with failing tests.
> > And we
> > > need
> > > > to pick up those tests that should not be ignored and bring them
> > up again
> > > > but passing. If there is no disagreement, I can start working on
> > that.
> > > >
> > > > Once I am done, I can try to help with infra tickets too.
> > > >
> > > > -Jesús
> > > >
> > > >
> > > > On 5/11/18, 1:57 PM, "Vineet Garg"  wrote:
> > > >
> > > > +1. I strongly vote for freezing commits and getting our
> > testing
> > > > coverage in acceptable state.  We have been struggling to
> stabilize
> > > > branch-3 due to test failures and releasing Hive 3.0 in current
> > state
> > > would
> > > > be unacceptable.
> > > >
> > > > Currently there are quite a few test suites which are not
> even
> > > running
> > > > and are being timed out. We have been committing patches (to both
> > > branch-3
> > > > and master) without test coverage for these tests.
> > > > We should immediately figure out what’s going on before we
> > proceed
> > > > with commits.
> > > >
> > > > For reference following test suites are timing out on
> master: (
> > > > https://issues.apache.org/jira/browse/HIVE-19506)
> > > >
> > > >
> > > > TestDbNotificationListener - did not produce a TEST-*.xml
> file
> > > (likely
> > > > timed out)
> > > >
> > > > TestHCatHiveCompatibility - did not produce a TEST-*.xml file
> > (likely
> > > > timed out)
> > > >
> > > > TestNegativeCliDriver - did not produce a TEST-*.xml file
> > (likely
> > > > timed out)
> > > >
> > > > TestNonCatCallsWithCatalog - did not produce a TEST-*.xml
> file
> > > (likely
> > > > timed out)
> > > >
> > > > TestSequenceFileReadWrite - did not 

Re: [DISCUSS] Unsustainable situation with ptests

[Vihang Karajgaonkar]

I worked on a few quick-fix optimizations in Ptest infrastructure over the
weekend which reduced the execution run from ~90 min to ~70 min per run. I
had to restart Ptest multiple times. I was resubmitting the patches which
were in the queue manually, but I may have missed a few. In case you have a
patch which is pending pre-commit and you don't see it in the queue, please
submit it manually or let me know if you don't have access to the jenkins
job. I will continue to work on the sub-tasks in HIVE-19425 and will do
some maintenance next weekend as well.

On Mon, May 14, 2018 at 7:42 AM, Jesus Camacho Rodriguez <
jcama...@apache.org> wrote:

> Vineet has already been working on disabling those tests that were timing
> out. I am working on disabling those that are generating different q files
> consistently for last ptests n runs. I am keeping track of all these tests
> in https://issues.apache.org/jira/browse/HIVE-19509.
>
> -Jesús
>
> On 5/14/18, 2:25 AM, "Prasanth Jayachandran" <
> pjayachand...@hortonworks.com> wrote:
>
> +1 on freezing commits until we get repetitive green tests. We should
> probably disable (and remember in a jira to reenable then at later point)
> tests that are flaky to get repetitive green test runs.
>
> Thanks
> Prasanth
>
>
>
> On Mon, May 14, 2018 at 2:15 AM -0700, "Rui Li"  > wrote:
>
>
> +1 to freezing commits until we stabilize
>
> On Sat, May 12, 2018 at 6:10 AM, Vihang Karajgaonkar
> wrote:
>
> > In order to understand the end-to-end precommit flow I would like to
> get
> > access to the PreCommit-HIVE-Build jenkins script. Does anyone one
> know how
> > can I get that?
> >
> > On Fri, May 11, 2018 at 2:03 PM, Jesus Camacho Rodriguez <
> > jcama...@apache.org> wrote:
> >
> > > Bq. For the short term green runs, I think we should @Ignore the
> tests
> > > which
> > > are known to be failing since many runs. They are anyways not being
> > > addressed as such. If people think they are important to be run we
> should
> > > fix them and only then re-enable them.
> > >
> > > I think that is a good idea, as we would minimize the time that we
> halt
> > > development. We can create a JIRA where we list all tests that were
> > > failing, and we have disabled to get the clean run. From that
> moment, we
> > > will have zero tolerance towards committing with failing tests.
> And we
> > need
> > > to pick up those tests that should not be ignored and bring them
> up again
> > > but passing. If there is no disagreement, I can start working on
> that.
> > >
> > > Once I am done, I can try to help with infra tickets too.
> > >
> > > -Jesús
> > >
> > >
> > > On 5/11/18, 1:57 PM, "Vineet Garg"  wrote:
> > >
> > > +1. I strongly vote for freezing commits and getting our
> testing
> > > coverage in acceptable state.  We have been struggling to stabilize
> > > branch-3 due to test failures and releasing Hive 3.0 in current
> state
> > would
> > > be unacceptable.
> > >
> > > Currently there are quite a few test suites which are not even
> > running
> > > and are being timed out. We have been committing patches (to both
> > branch-3
> > > and master) without test coverage for these tests.
> > > We should immediately figure out what’s going on before we
> proceed
> > > with commits.
> > >
> > > For reference following test suites are timing out on master: (
> > > https://issues.apache.org/jira/browse/HIVE-19506)
> > >
> > >
> > > TestDbNotificationListener - did not produce a TEST-*.xml file
> > (likely
> > > timed out)
> > >
> > > TestHCatHiveCompatibility - did not produce a TEST-*.xml file
> (likely
> > > timed out)
> > >
> > > TestNegativeCliDriver - did not produce a TEST-*.xml file
> (likely
> > > timed out)
> > >
> > > TestNonCatCallsWithCatalog - did not produce a TEST-*.xml file
> > (likely
> > > timed out)
> > >
> > > TestSequenceFileReadWrite - did not produce a TEST-*.xml file
> (likely
> > > timed out)
> > >
> > > TestTxnExIm - did not produce a TEST-*.xml file (likely timed
> out)
> > >
> > >
> > > Vineet
> > >
> > >
> > > On May 11, 2018, at 1:46 PM, Vihang Karajgaonkar <
> > vih...@cloudera.com
> > > > wrote:
> > >
> > > +1 There are many problems with the test infrastructure and in
> my
> > > opinion
> > > it has not become number one bottleneck for the project. I was
> > looking
> > > at
> > > the infrastructure yesterday and I think the current
> infrastructure
> > > (even
> > > its own set of problems) is still under-utilized. I am
> planning to
> > > increase
> > > the number of threads to process 

Re: [DISCUSS] Unsustainable situation with ptests

[Jesus Camacho Rodriguez]

Vineet has already been working on disabling those tests that were timing out. 
I am working on disabling those that are generating different q files 
consistently for last ptests n runs. I am keeping track of all these tests in 
https://issues.apache.org/jira/browse/HIVE-19509.

-Jesús

On 5/14/18, 2:25 AM, "Prasanth Jayachandran"  
wrote:

+1 on freezing commits until we get repetitive green tests. We should 
probably disable (and remember in a jira to reenable then at later point) tests 
that are flaky to get repetitive green test runs.

Thanks
Prasanth



On Mon, May 14, 2018 at 2:15 AM -0700, "Rui Li" 
> wrote:


+1 to freezing commits until we stabilize

On Sat, May 12, 2018 at 6:10 AM, Vihang Karajgaonkar
wrote:

> In order to understand the end-to-end precommit flow I would like to get
> access to the PreCommit-HIVE-Build jenkins script. Does anyone one know 
how
> can I get that?
>
> On Fri, May 11, 2018 at 2:03 PM, Jesus Camacho Rodriguez <
> jcama...@apache.org> wrote:
>
> > Bq. For the short term green runs, I think we should @Ignore the tests
> > which
> > are known to be failing since many runs. They are anyways not being
> > addressed as such. If people think they are important to be run we 
should
> > fix them and only then re-enable them.
> >
> > I think that is a good idea, as we would minimize the time that we halt
> > development. We can create a JIRA where we list all tests that were
> > failing, and we have disabled to get the clean run. From that moment, we
> > will have zero tolerance towards committing with failing tests. And we
> need
> > to pick up those tests that should not be ignored and bring them up 
again
> > but passing. If there is no disagreement, I can start working on that.
> >
> > Once I am done, I can try to help with infra tickets too.
> >
> > -Jesús
> >
> >
> > On 5/11/18, 1:57 PM, "Vineet Garg"  wrote:
> >
> > +1. I strongly vote for freezing commits and getting our testing
> > coverage in acceptable state.  We have been struggling to stabilize
> > branch-3 due to test failures and releasing Hive 3.0 in current state
> would
> > be unacceptable.
> >
> > Currently there are quite a few test suites which are not even
> running
> > and are being timed out. We have been committing patches (to both
> branch-3
> > and master) without test coverage for these tests.
> > We should immediately figure out what’s going on before we proceed
> > with commits.
> >
> > For reference following test suites are timing out on master: (
> > https://issues.apache.org/jira/browse/HIVE-19506)
> >
> >
> > TestDbNotificationListener - did not produce a TEST-*.xml file
> (likely
> > timed out)
> >
> > TestHCatHiveCompatibility - did not produce a TEST-*.xml file 
(likely
> > timed out)
> >
> > TestNegativeCliDriver - did not produce a TEST-*.xml file (likely
> > timed out)
> >
> > TestNonCatCallsWithCatalog - did not produce a TEST-*.xml file
> (likely
> > timed out)
> >
> > TestSequenceFileReadWrite - did not produce a TEST-*.xml file 
(likely
> > timed out)
> >
> > TestTxnExIm - did not produce a TEST-*.xml file (likely timed out)
> >
> >
> > Vineet
> >
> >
> > On May 11, 2018, at 1:46 PM, Vihang Karajgaonkar <
> vih...@cloudera.com
> > > wrote:
> >
> > +1 There are many problems with the test infrastructure and in my
> > opinion
> > it has not become number one bottleneck for the project. I was
> looking
> > at
> > the infrastructure yesterday and I think the current infrastructure
> > (even
> > its own set of problems) is still under-utilized. I am planning to
> > increase
> > the number of threads to process the parallel test batches to start
> > with.
> > It needs a restart on the server side. I can do it now, it folks are
> > okay
> > with it. Else I can do it over weekend when the queue is small.
> >
> > I listed the improvements which I thought would be useful under
> > https://issues.apache.org/jira/browse/HIVE-19425 but frankly
> speaking
> > I am
> > not able to devote as much time as I would like to on it. I would
> > appreciate if folks who have some more time if they can help out.
> >
> > I think to start with https://issues.apache.org/
> jira/browse/HIVE-19429
> > will
> > help a lot. We need to pack more test runs in parallel and 
containers
> > provide good isolation.
> >
> > For the short term green runs, I think we should 

Re: [DISCUSS] Unsustainable situation with ptests

[Prasanth Jayachandran]

+1 on freezing commits until we get repetitive green tests. We should probably 
disable (and remember in a jira to reenable then at later point) tests that are 
flaky to get repetitive green test runs.

Thanks
Prasanth



On Mon, May 14, 2018 at 2:15 AM -0700, "Rui Li" 
> wrote:


+1 to freezing commits until we stabilize

On Sat, May 12, 2018 at 6:10 AM, Vihang Karajgaonkar
wrote:

> In order to understand the end-to-end precommit flow I would like to get
> access to the PreCommit-HIVE-Build jenkins script. Does anyone one know how
> can I get that?
>
> On Fri, May 11, 2018 at 2:03 PM, Jesus Camacho Rodriguez <
> jcama...@apache.org> wrote:
>
> > Bq. For the short term green runs, I think we should @Ignore the tests
> > which
> > are known to be failing since many runs. They are anyways not being
> > addressed as such. If people think they are important to be run we should
> > fix them and only then re-enable them.
> >
> > I think that is a good idea, as we would minimize the time that we halt
> > development. We can create a JIRA where we list all tests that were
> > failing, and we have disabled to get the clean run. From that moment, we
> > will have zero tolerance towards committing with failing tests. And we
> need
> > to pick up those tests that should not be ignored and bring them up again
> > but passing. If there is no disagreement, I can start working on that.
> >
> > Once I am done, I can try to help with infra tickets too.
> >
> > -Jesús
> >
> >
> > On 5/11/18, 1:57 PM, "Vineet Garg"  wrote:
> >
> > +1. I strongly vote for freezing commits and getting our testing
> > coverage in acceptable state.  We have been struggling to stabilize
> > branch-3 due to test failures and releasing Hive 3.0 in current state
> would
> > be unacceptable.
> >
> > Currently there are quite a few test suites which are not even
> running
> > and are being timed out. We have been committing patches (to both
> branch-3
> > and master) without test coverage for these tests.
> > We should immediately figure out what’s going on before we proceed
> > with commits.
> >
> > For reference following test suites are timing out on master: (
> > https://issues.apache.org/jira/browse/HIVE-19506)
> >
> >
> > TestDbNotificationListener - did not produce a TEST-*.xml file
> (likely
> > timed out)
> >
> > TestHCatHiveCompatibility - did not produce a TEST-*.xml file (likely
> > timed out)
> >
> > TestNegativeCliDriver - did not produce a TEST-*.xml file (likely
> > timed out)
> >
> > TestNonCatCallsWithCatalog - did not produce a TEST-*.xml file
> (likely
> > timed out)
> >
> > TestSequenceFileReadWrite - did not produce a TEST-*.xml file (likely
> > timed out)
> >
> > TestTxnExIm - did not produce a TEST-*.xml file (likely timed out)
> >
> >
> > Vineet
> >
> >
> > On May 11, 2018, at 1:46 PM, Vihang Karajgaonkar <
> vih...@cloudera.com
> > > wrote:
> >
> > +1 There are many problems with the test infrastructure and in my
> > opinion
> > it has not become number one bottleneck for the project. I was
> looking
> > at
> > the infrastructure yesterday and I think the current infrastructure
> > (even
> > its own set of problems) is still under-utilized. I am planning to
> > increase
> > the number of threads to process the parallel test batches to start
> > with.
> > It needs a restart on the server side. I can do it now, it folks are
> > okay
> > with it. Else I can do it over weekend when the queue is small.
> >
> > I listed the improvements which I thought would be useful under
> > https://issues.apache.org/jira/browse/HIVE-19425 but frankly
> speaking
> > I am
> > not able to devote as much time as I would like to on it. I would
> > appreciate if folks who have some more time if they can help out.
> >
> > I think to start with https://issues.apache.org/
> jira/browse/HIVE-19429
> > will
> > help a lot. We need to pack more test runs in parallel and containers
> > provide good isolation.
> >
> > For the short term green runs, I think we should @Ignore the tests
> > which
> > are known to be failing since many runs. They are anyways not being
> > addressed as such. If people think they are important to be run we
> > should
> > fix them and only then re-enable them.
> >
> > Also, I feel we need light-weight test run which we can run locally
> > before
> > submitting it for the full-suite. That way minor issues with the
> patch
> > can
> > be handled locally. May be create a profile which runs a subset of
> > important tests which are consistent. We can apply some label that
> > pre-checkin-local tests are runs successful and only then we submit
> > for the
> > full-suite.
> >
> > More thoughts are welcome. Thanks for starting this conversation.
> >
> > On Fri, May 11, 2018 at 1:27 PM, Jesus Camacho Rodriguez <
> > 

Re: [DISCUSS] Unsustainable situation with ptests

[Rui Li]

+1 to freezing commits until we stabilize

On Sat, May 12, 2018 at 6:10 AM, Vihang Karajgaonkar 
wrote:

> In order to understand the end-to-end precommit flow I would like to get
> access to the PreCommit-HIVE-Build jenkins script. Does anyone one know how
> can I get that?
>
> On Fri, May 11, 2018 at 2:03 PM, Jesus Camacho Rodriguez <
> jcama...@apache.org> wrote:
>
> > Bq. For the short term green runs, I think we should @Ignore the tests
> > which
> > are known to be failing since many runs. They are anyways not being
> > addressed as such. If people think they are important to be run we should
> > fix them and only then re-enable them.
> >
> > I think that is a good idea, as we would minimize the time that we halt
> > development. We can create a JIRA where we list all tests that were
> > failing, and we have disabled to get the clean run. From that moment, we
> > will have zero tolerance towards committing with failing tests. And we
> need
> > to pick up those tests that should not be ignored and bring them up again
> > but passing. If there is no disagreement, I can start working on that.
> >
> > Once I am done, I can try to help with infra tickets too.
> >
> > -Jesús
> >
> >
> > On 5/11/18, 1:57 PM, "Vineet Garg"  wrote:
> >
> > +1. I strongly vote for freezing commits and getting our testing
> > coverage in acceptable state.  We have been struggling to stabilize
> > branch-3 due to test failures and releasing Hive 3.0 in current state
> would
> > be unacceptable.
> >
> > Currently there are quite a few test suites which are not even
> running
> > and are being timed out. We have been committing patches (to both
> branch-3
> > and master) without test coverage for these tests.
> > We should immediately figure out what’s going on before we proceed
> > with commits.
> >
> > For reference following test suites are timing out on master: (
> > https://issues.apache.org/jira/browse/HIVE-19506)
> >
> >
> > TestDbNotificationListener - did not produce a TEST-*.xml file
> (likely
> > timed out)
> >
> > TestHCatHiveCompatibility - did not produce a TEST-*.xml file (likely
> > timed out)
> >
> > TestNegativeCliDriver - did not produce a TEST-*.xml file (likely
> > timed out)
> >
> > TestNonCatCallsWithCatalog - did not produce a TEST-*.xml file
> (likely
> > timed out)
> >
> > TestSequenceFileReadWrite - did not produce a TEST-*.xml file (likely
> > timed out)
> >
> > TestTxnExIm - did not produce a TEST-*.xml file (likely timed out)
> >
> >
> > Vineet
> >
> >
> > On May 11, 2018, at 1:46 PM, Vihang Karajgaonkar <
> vih...@cloudera.com
> > > wrote:
> >
> > +1 There are many problems with the test infrastructure and in my
> > opinion
> > it has not become number one bottleneck for the project. I was
> looking
> > at
> > the infrastructure yesterday and I think the current infrastructure
> > (even
> > its own set of problems) is still under-utilized. I am planning to
> > increase
> > the number of threads to process the parallel test batches to start
> > with.
> > It needs a restart on the server side. I can do it now, it folks are
> > okay
> > with it. Else I can do it over weekend when the queue is small.
> >
> > I listed the improvements which I thought would be useful under
> > https://issues.apache.org/jira/browse/HIVE-19425 but frankly
> speaking
> > I am
> > not able to devote as much time as I would like to on it. I would
> > appreciate if folks who have some more time if they can help out.
> >
> > I think to start with https://issues.apache.org/
> jira/browse/HIVE-19429
> > will
> > help a lot. We need to pack more test runs in parallel and containers
> > provide good isolation.
> >
> > For the short term green runs, I think we should @Ignore the tests
> > which
> > are known to be failing since many runs. They are anyways not being
> > addressed as such. If people think they are important to be run we
> > should
> > fix them and only then re-enable them.
> >
> > Also, I feel we need light-weight test run which we can run locally
> > before
> > submitting it for the full-suite. That way minor issues with the
> patch
> > can
> > be handled locally. May be create a profile which runs a subset of
> > important tests which are consistent. We can apply some label that
> > pre-checkin-local tests are runs successful and only then we submit
> > for the
> > full-suite.
> >
> > More thoughts are welcome. Thanks for starting this conversation.
> >
> > On Fri, May 11, 2018 at 1:27 PM, Jesus Camacho Rodriguez <
> > jcama...@apache.org> wrote:
> >
> > I believe we have reached a state (maybe we did reach it a while ago)
> > that
> > is not sustainable anymore, as there are so many tests failing /
> > timing out
> > that it is not possible to verify 

Re: [DISCUSS] Unsustainable situation with ptests

[Vihang Karajgaonkar]

In order to understand the end-to-end precommit flow I would like to get
access to the PreCommit-HIVE-Build jenkins script. Does anyone one know how
can I get that?

On Fri, May 11, 2018 at 2:03 PM, Jesus Camacho Rodriguez <
jcama...@apache.org> wrote:

> Bq. For the short term green runs, I think we should @Ignore the tests
> which
> are known to be failing since many runs. They are anyways not being
> addressed as such. If people think they are important to be run we should
> fix them and only then re-enable them.
>
> I think that is a good idea, as we would minimize the time that we halt
> development. We can create a JIRA where we list all tests that were
> failing, and we have disabled to get the clean run. From that moment, we
> will have zero tolerance towards committing with failing tests. And we need
> to pick up those tests that should not be ignored and bring them up again
> but passing. If there is no disagreement, I can start working on that.
>
> Once I am done, I can try to help with infra tickets too.
>
> -Jesús
>
>
> On 5/11/18, 1:57 PM, "Vineet Garg"  wrote:
>
> +1. I strongly vote for freezing commits and getting our testing
> coverage in acceptable state.  We have been struggling to stabilize
> branch-3 due to test failures and releasing Hive 3.0 in current state would
> be unacceptable.
>
> Currently there are quite a few test suites which are not even running
> and are being timed out. We have been committing patches (to both branch-3
> and master) without test coverage for these tests.
> We should immediately figure out what’s going on before we proceed
> with commits.
>
> For reference following test suites are timing out on master: (
> https://issues.apache.org/jira/browse/HIVE-19506)
>
>
> TestDbNotificationListener - did not produce a TEST-*.xml file (likely
> timed out)
>
> TestHCatHiveCompatibility - did not produce a TEST-*.xml file (likely
> timed out)
>
> TestNegativeCliDriver - did not produce a TEST-*.xml file (likely
> timed out)
>
> TestNonCatCallsWithCatalog - did not produce a TEST-*.xml file (likely
> timed out)
>
> TestSequenceFileReadWrite - did not produce a TEST-*.xml file (likely
> timed out)
>
> TestTxnExIm - did not produce a TEST-*.xml file (likely timed out)
>
>
> Vineet
>
>
> On May 11, 2018, at 1:46 PM, Vihang Karajgaonkar  > wrote:
>
> +1 There are many problems with the test infrastructure and in my
> opinion
> it has not become number one bottleneck for the project. I was looking
> at
> the infrastructure yesterday and I think the current infrastructure
> (even
> its own set of problems) is still under-utilized. I am planning to
> increase
> the number of threads to process the parallel test batches to start
> with.
> It needs a restart on the server side. I can do it now, it folks are
> okay
> with it. Else I can do it over weekend when the queue is small.
>
> I listed the improvements which I thought would be useful under
> https://issues.apache.org/jira/browse/HIVE-19425 but frankly speaking
> I am
> not able to devote as much time as I would like to on it. I would
> appreciate if folks who have some more time if they can help out.
>
> I think to start with https://issues.apache.org/jira/browse/HIVE-19429
> will
> help a lot. We need to pack more test runs in parallel and containers
> provide good isolation.
>
> For the short term green runs, I think we should @Ignore the tests
> which
> are known to be failing since many runs. They are anyways not being
> addressed as such. If people think they are important to be run we
> should
> fix them and only then re-enable them.
>
> Also, I feel we need light-weight test run which we can run locally
> before
> submitting it for the full-suite. That way minor issues with the patch
> can
> be handled locally. May be create a profile which runs a subset of
> important tests which are consistent. We can apply some label that
> pre-checkin-local tests are runs successful and only then we submit
> for the
> full-suite.
>
> More thoughts are welcome. Thanks for starting this conversation.
>
> On Fri, May 11, 2018 at 1:27 PM, Jesus Camacho Rodriguez <
> jcama...@apache.org> wrote:
>
> I believe we have reached a state (maybe we did reach it a while ago)
> that
> is not sustainable anymore, as there are so many tests failing /
> timing out
> that it is not possible to verify whether a patch is breaking some
> critical
> parts of the system or not. It also seems to me that due to the
> timeouts
> (maybe due to infra, maybe not), ptest runs are taking even longer than
> usual, which in turn creates even longer queue of patches.
>
> There is an ongoing effort to improve ptests usability (
> 

Re: [DISCUSS] Unsustainable situation with ptests

[Jesus Camacho Rodriguez]

Bq. For the short term green runs, I think we should @Ignore the tests which
are known to be failing since many runs. They are anyways not being
addressed as such. If people think they are important to be run we should
fix them and only then re-enable them.

I think that is a good idea, as we would minimize the time that we halt 
development. We can create a JIRA where we list all tests that were failing, 
and we have disabled to get the clean run. From that moment, we will have zero 
tolerance towards committing with failing tests. And we need to pick up those 
tests that should not be ignored and bring them up again but passing. If there 
is no disagreement, I can start working on that.

Once I am done, I can try to help with infra tickets too.

-Jesús


On 5/11/18, 1:57 PM, "Vineet Garg"  wrote:

+1. I strongly vote for freezing commits and getting our testing coverage 
in acceptable state.  We have been struggling to stabilize branch-3 due to test 
failures and releasing Hive 3.0 in current state would be unacceptable.

Currently there are quite a few test suites which are not even running and 
are being timed out. We have been committing patches (to both branch-3 and 
master) without test coverage for these tests.
We should immediately figure out what’s going on before we proceed with 
commits.

For reference following test suites are timing out on master: 
(https://issues.apache.org/jira/browse/HIVE-19506)


TestDbNotificationListener - did not produce a TEST-*.xml file (likely 
timed out)

TestHCatHiveCompatibility - did not produce a TEST-*.xml file (likely timed 
out)

TestNegativeCliDriver - did not produce a TEST-*.xml file (likely timed out)

TestNonCatCallsWithCatalog - did not produce a TEST-*.xml file (likely 
timed out)

TestSequenceFileReadWrite - did not produce a TEST-*.xml file (likely timed 
out)

TestTxnExIm - did not produce a TEST-*.xml file (likely timed out)


Vineet


On May 11, 2018, at 1:46 PM, Vihang Karajgaonkar 
> wrote:

+1 There are many problems with the test infrastructure and in my opinion
it has not become number one bottleneck for the project. I was looking at
the infrastructure yesterday and I think the current infrastructure (even
its own set of problems) is still under-utilized. I am planning to increase
the number of threads to process the parallel test batches to start with.
It needs a restart on the server side. I can do it now, it folks are okay
with it. Else I can do it over weekend when the queue is small.

I listed the improvements which I thought would be useful under
https://issues.apache.org/jira/browse/HIVE-19425 but frankly speaking I am
not able to devote as much time as I would like to on it. I would
appreciate if folks who have some more time if they can help out.

I think to start with https://issues.apache.org/jira/browse/HIVE-19429 will
help a lot. We need to pack more test runs in parallel and containers
provide good isolation.

For the short term green runs, I think we should @Ignore the tests which
are known to be failing since many runs. They are anyways not being
addressed as such. If people think they are important to be run we should
fix them and only then re-enable them.

Also, I feel we need light-weight test run which we can run locally before
submitting it for the full-suite. That way minor issues with the patch can
be handled locally. May be create a profile which runs a subset of
important tests which are consistent. We can apply some label that
pre-checkin-local tests are runs successful and only then we submit for the
full-suite.

More thoughts are welcome. Thanks for starting this conversation.

On Fri, May 11, 2018 at 1:27 PM, Jesus Camacho Rodriguez <
jcama...@apache.org> wrote:

I believe we have reached a state (maybe we did reach it a while ago) that
is not sustainable anymore, as there are so many tests failing / timing out
that it is not possible to verify whether a patch is breaking some critical
parts of the system or not. It also seems to me that due to the timeouts
(maybe due to infra, maybe not), ptest runs are taking even longer than
usual, which in turn creates even longer queue of patches.

There is an ongoing effort to improve ptests usability (
https://issues.apache.org/jira/browse/HIVE-19425), but apart from that,
we need to make an effort to stabilize existing tests and bring that
failure count to zero.

Hence, I am suggesting *we stop committing any patch before we get a green
run*. If someone thinks this proposal is too radical, please come up with
an alternative, because I do not think it is OK to have the ptest runs 

Re: [DISCUSS] Unsustainable situation with ptests

[Vineet Garg]

+1. I strongly vote for freezing commits and getting our testing coverage in 
acceptable state.  We have been struggling to stabilize branch-3 due to test 
failures and releasing Hive 3.0 in current state would be unacceptable.

Currently there are quite a few test suites which are not even running and are 
being timed out. We have been committing patches (to both branch-3 and master) 
without test coverage for these tests.
We should immediately figure out what’s going on before we proceed with commits.

For reference following test suites are timing out on master: 
(https://issues.apache.org/jira/browse/HIVE-19506)


TestDbNotificationListener - did not produce a TEST-*.xml file (likely timed 
out)

TestHCatHiveCompatibility - did not produce a TEST-*.xml file (likely timed out)

TestNegativeCliDriver - did not produce a TEST-*.xml file (likely timed out)

TestNonCatCallsWithCatalog - did not produce a TEST-*.xml file (likely timed 
out)

TestSequenceFileReadWrite - did not produce a TEST-*.xml file (likely timed out)

TestTxnExIm - did not produce a TEST-*.xml file (likely timed out)


Vineet


On May 11, 2018, at 1:46 PM, Vihang Karajgaonkar 
> wrote:

+1 There are many problems with the test infrastructure and in my opinion
it has not become number one bottleneck for the project. I was looking at
the infrastructure yesterday and I think the current infrastructure (even
its own set of problems) is still under-utilized. I am planning to increase
the number of threads to process the parallel test batches to start with.
It needs a restart on the server side. I can do it now, it folks are okay
with it. Else I can do it over weekend when the queue is small.

I listed the improvements which I thought would be useful under
https://issues.apache.org/jira/browse/HIVE-19425 but frankly speaking I am
not able to devote as much time as I would like to on it. I would
appreciate if folks who have some more time if they can help out.

I think to start with https://issues.apache.org/jira/browse/HIVE-19429 will
help a lot. We need to pack more test runs in parallel and containers
provide good isolation.

For the short term green runs, I think we should @Ignore the tests which
are known to be failing since many runs. They are anyways not being
addressed as such. If people think they are important to be run we should
fix them and only then re-enable them.

Also, I feel we need light-weight test run which we can run locally before
submitting it for the full-suite. That way minor issues with the patch can
be handled locally. May be create a profile which runs a subset of
important tests which are consistent. We can apply some label that
pre-checkin-local tests are runs successful and only then we submit for the
full-suite.

More thoughts are welcome. Thanks for starting this conversation.

On Fri, May 11, 2018 at 1:27 PM, Jesus Camacho Rodriguez <
jcama...@apache.org> wrote:

I believe we have reached a state (maybe we did reach it a while ago) that
is not sustainable anymore, as there are so many tests failing / timing out
that it is not possible to verify whether a patch is breaking some critical
parts of the system or not. It also seems to me that due to the timeouts
(maybe due to infra, maybe not), ptest runs are taking even longer than
usual, which in turn creates even longer queue of patches.

There is an ongoing effort to improve ptests usability (
https://issues.apache.org/jira/browse/HIVE-19425), but apart from that,
we need to make an effort to stabilize existing tests and bring that
failure count to zero.

Hence, I am suggesting *we stop committing any patch before we get a green
run*. If someone thinks this proposal is too radical, please come up with
an alternative, because I do not think it is OK to have the ptest runs in
their current state. Other projects of certain size (e.g., Hadoop, Spark)
are always green, we should be able to do the same.

Finally, once we get to zero failures, I suggest we are less tolerant with
committing without getting a clean ptests run. If there is a failure, we
need to fix it or revert the patch that caused it, then we continue
developing.

Please, let’s all work together as a community to fix this issue, that is
the only way to get to zero quickly.

Thanks,
Jesús

PS. I assume the flaky tests will come into the discussion. Let´s see
first how many of those we have, then we can work to find a fix.

Re: [DISCUSS] Unsustainable situation with ptests

[Vihang Karajgaonkar]

correction in my email below. I meant "in my opinion it *has now* become
number one bottleneck for the project" (worst place for a typo I guess)

On Fri, May 11, 2018 at 1:46 PM, Vihang Karajgaonkar 
wrote:

> +1 There are many problems with the test infrastructure and in my opinion
> it has not become number one bottleneck for the project. I was looking at
> the infrastructure yesterday and I think the current infrastructure (even
> its own set of problems) is still under-utilized. I am planning to increase
> the number of threads to process the parallel test batches to start with.
> It needs a restart on the server side. I can do it now, it folks are okay
> with it. Else I can do it over weekend when the queue is small.
>
> I listed the improvements which I thought would be useful under
> https://issues.apache.org/jira/browse/HIVE-19425 but frankly speaking I
> am not able to devote as much time as I would like to on it. I would
> appreciate if folks who have some more time if they can help out.
>
> I think to start with https://issues.apache.org/jira/browse/HIVE-19429
> will help a lot. We need to pack more test runs in parallel and containers
> provide good isolation.
>
> For the short term green runs, I think we should @Ignore the tests which
> are known to be failing since many runs. They are anyways not being
> addressed as such. If people think they are important to be run we should
> fix them and only then re-enable them.
>
> Also, I feel we need light-weight test run which we can run locally before
> submitting it for the full-suite. That way minor issues with the patch can
> be handled locally. May be create a profile which runs a subset of
> important tests which are consistent. We can apply some label that
> pre-checkin-local tests are runs successful and only then we submit for the
> full-suite.
>
> More thoughts are welcome. Thanks for starting this conversation.
>
> On Fri, May 11, 2018 at 1:27 PM, Jesus Camacho Rodriguez <
> jcama...@apache.org> wrote:
>
>> I believe we have reached a state (maybe we did reach it a while ago)
>> that is not sustainable anymore, as there are so many tests failing /
>> timing out that it is not possible to verify whether a patch is breaking
>> some critical parts of the system or not. It also seems to me that due to
>> the timeouts (maybe due to infra, maybe not), ptest runs are taking even
>> longer than usual, which in turn creates even longer queue of patches.
>>
>> There is an ongoing effort to improve ptests usability (
>> https://issues.apache.org/jira/browse/HIVE-19425), but apart from that,
>> we need to make an effort to stabilize existing tests and bring that
>> failure count to zero.
>>
>> Hence, I am suggesting *we stop committing any patch before we get a
>> green run*. If someone thinks this proposal is too radical, please come up
>> with an alternative, because I do not think it is OK to have the ptest runs
>> in their current state. Other projects of certain size (e.g., Hadoop,
>> Spark) are always green, we should be able to do the same.
>>
>> Finally, once we get to zero failures, I suggest we are less tolerant
>> with committing without getting a clean ptests run. If there is a failure,
>> we need to fix it or revert the patch that caused it, then we continue
>> developing.
>>
>> Please, let’s all work together as a community to fix this issue, that is
>> the only way to get to zero quickly.
>>
>> Thanks,
>> Jesús
>>
>> PS. I assume the flaky tests will come into the discussion. Let´s see
>> first how many of those we have, then we can work to find a fix.
>>
>>
>>
>

Re: [DISCUSS] Unsustainable situation with ptests

[Vihang Karajgaonkar]

+1 There are many problems with the test infrastructure and in my opinion
it has not become number one bottleneck for the project. I was looking at
the infrastructure yesterday and I think the current infrastructure (even
its own set of problems) is still under-utilized. I am planning to increase
the number of threads to process the parallel test batches to start with.
It needs a restart on the server side. I can do it now, it folks are okay
with it. Else I can do it over weekend when the queue is small.

I listed the improvements which I thought would be useful under
https://issues.apache.org/jira/browse/HIVE-19425 but frankly speaking I am
not able to devote as much time as I would like to on it. I would
appreciate if folks who have some more time if they can help out.

I think to start with https://issues.apache.org/jira/browse/HIVE-19429 will
help a lot. We need to pack more test runs in parallel and containers
provide good isolation.

For the short term green runs, I think we should @Ignore the tests which
are known to be failing since many runs. They are anyways not being
addressed as such. If people think they are important to be run we should
fix them and only then re-enable them.

Also, I feel we need light-weight test run which we can run locally before
submitting it for the full-suite. That way minor issues with the patch can
be handled locally. May be create a profile which runs a subset of
important tests which are consistent. We can apply some label that
pre-checkin-local tests are runs successful and only then we submit for the
full-suite.

More thoughts are welcome. Thanks for starting this conversation.

On Fri, May 11, 2018 at 1:27 PM, Jesus Camacho Rodriguez <
jcama...@apache.org> wrote:

> I believe we have reached a state (maybe we did reach it a while ago) that
> is not sustainable anymore, as there are so many tests failing / timing out
> that it is not possible to verify whether a patch is breaking some critical
> parts of the system or not. It also seems to me that due to the timeouts
> (maybe due to infra, maybe not), ptest runs are taking even longer than
> usual, which in turn creates even longer queue of patches.
>
> There is an ongoing effort to improve ptests usability (
> https://issues.apache.org/jira/browse/HIVE-19425), but apart from that,
> we need to make an effort to stabilize existing tests and bring that
> failure count to zero.
>
> Hence, I am suggesting *we stop committing any patch before we get a green
> run*. If someone thinks this proposal is too radical, please come up with
> an alternative, because I do not think it is OK to have the ptest runs in
> their current state. Other projects of certain size (e.g., Hadoop, Spark)
> are always green, we should be able to do the same.
>
> Finally, once we get to zero failures, I suggest we are less tolerant with
> committing without getting a clean ptests run. If there is a failure, we
> need to fix it or revert the patch that caused it, then we continue
> developing.
>
> Please, let’s all work together as a community to fix this issue, that is
> the only way to get to zero quickly.
>
> Thanks,
> Jesús
>
> PS. I assume the flaky tests will come into the discussion. Let´s see
> first how many of those we have, then we can work to find a fix.
>
>
>

[DISCUSS] Unsustainable situation with ptests

[Jesus Camacho Rodriguez]

I believe we have reached a state (maybe we did reach it a while ago) that is 
not sustainable anymore, as there are so many tests failing / timing out that 
it is not possible to verify whether a patch is breaking some critical parts of 
the system or not. It also seems to me that due to the timeouts (maybe due to 
infra, maybe not), ptest runs are taking even longer than usual, which in turn 
creates even longer queue of patches.

There is an ongoing effort to improve ptests usability 
(https://issues.apache.org/jira/browse/HIVE-19425), but apart from that, we 
need to make an effort to stabilize existing tests and bring that failure count 
to zero.

Hence, I am suggesting *we stop committing any patch before we get a green 
run*. If someone thinks this proposal is too radical, please come up with an 
alternative, because I do not think it is OK to have the ptest runs in their 
current state. Other projects of certain size (e.g., Hadoop, Spark) are always 
green, we should be able to do the same.

Finally, once we get to zero failures, I suggest we are less tolerant with 
committing without getting a clean ptests run. If there is a failure, we need 
to fix it or revert the patch that caused it, then we continue developing.

Please, let’s all work together as a community to fix this issue, that is the 
only way to get to zero quickly.

Thanks,
Jesús

PS. I assume the flaky tests will come into the discussion. Let´s see first how 
many of those we have, then we can work to find a fix.