Re: Precommit unstable?

[Thejas Nair]

Forgot to get back on this one. That delay was actually caused by a
hanging test, it was addressed here -
https://issues.apache.org/jira/browse/HIVE-12560


On Wed, Dec 2, 2015 at 10:08 AM, Sergio Pena  wrote:
> I see the last tests run in average of 1 hour. Should I still restart
> Jenkins?
>
> On Tue, Dec 1, 2015 at 12:11 PM, Thejas Nair  wrote:
>>
>> I killed the current 10+hr test run to see if next test would run
>> faster. (Discussed with Prasanth, the author of patch being tested).
>>
>> However,  I doubt if the next one is going to perform any better. If
>> we could try restarting the ptest2 system, I think that might be
>> better thing to do (instead of waiting several hours to realize that
>> the issue has not gone away).
>>
>>
>> On Tue, Dec 1, 2015 at 10:00 AM, Thejas Nair 
>> wrote:
>> > Would restart of jenkins or the hosts help with this ?
>> >
>> >
>> > On Tue, Dec 1, 2015 at 9:59 AM, Thejas Nair 
>> > wrote:
>> >> The build running right now has been running for almost 10 hours.
>> >> I am wondering if there is something wrong with the hosts being used
>> >> right now. Some commands seem to be taking longer and I get some
>> >> intermittent errors from the jenkins UI.
>> >>
>> >>
>> >> http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6187/
>> >> This rsync command seems to have taken a long time to finish -
>> >>
>> >> 2015-12-01 11:42:41,394  INFO LocalCommand.:35 Starting timeout
>> >> 1h rsync -vaPe "ssh -i /home/hiveptest/.ssh/hive-ptest-user-key"
>> >> --timeout 600
>> >> hiveptest@54.159.202.234:/home/hiveptest/54.159.202.234-hiveptest-0/logs/
>> >>
>> >> /data/hive-ptest/logs/PreCommit-HIVE-TRUNK-Build-6187/succeeded/TestHBaseMinimrCliDriver
>> >> 2015-12-01 12:54:52,522  INFO LocalCommand.:35 Starting timeout
>> >> 1h rsync -vaPe "ssh -i /home/hiveptest/.ssh/hive-ptest-user-key"
>> >> --timeout 600
>> >> hiveptest@54.157.22.221:/home/hiveptest/54.157.22.221-hiveptest-0/logs/
>> >>
>> >> /data/hive-ptest/logs/PreCommit-HIVE-TRUNK-Build-6187/failed/TestMiniTezCliDriver-script_pipe.q-tez_smb_empty.q-mapjoin_decimal.q-and-12-more
>> >> 2015-12-01 12:54:54,240  INFO HostExecutor.executeTests:176 Starting
>> >> isolated execution on 54.157.22.221
>> >>
>> >>
>> >>
>> >>
>> >>
>> >> On Tue, Dec 1, 2015 at 8:57 AM, Sergio Pena 
>> >> wrote:
>> >>> I see HIVE-12541 files were attached 4 times on JIRA while 'Patch
>> >>> Available' status was set. I does not say at what time, but Jenkins
>> >>> runs a
>> >>> job every 5 min to detect which files must be tested for all "Patch
>> >>> Available' JIRAS. That's why you probably saw HIVE-12541 in the queue
>> >>> many
>> >>> times.
>> >>>
>> >>> I would recommend to set the JIRA to 'Patch Available' when the patch
>> >>> is
>> >>> reviewed to avoid many patches to be added to the queue.
>> >>>
>> >>> On Mon, Nov 30, 2015 at 11:09 PM, Prasanth J 
>> >>> wrote:
>> >>>
>>  Hi All
>> 
>>  I am noticing something weird in precommit queue. I can see
>>  HIVE-12541currenly running, but is also waiting next in the queue.
>>  The same
>>  jira ran few more times recently. I cannot see any other patches in
>>  the
>>  queue. Sometime I see bunch of patches in the queue but it will soon
>>  be
>>  removed. But I am seeing HIVE-12541 running quite often. Has anyone
>>  seen
>>  this issue?
>> 
>>  Thanks
>>  Prasanth
>
>

Re: Precommit unstable?

[Sergio Pena]

I see the last tests run in average of 1 hour. Should I still restart
Jenkins?

On Tue, Dec 1, 2015 at 12:11 PM, Thejas Nair  wrote:

> I killed the current 10+hr test run to see if next test would run
> faster. (Discussed with Prasanth, the author of patch being tested).
>
> However,  I doubt if the next one is going to perform any better. If
> we could try restarting the ptest2 system, I think that might be
> better thing to do (instead of waiting several hours to realize that
> the issue has not gone away).
>
>
> On Tue, Dec 1, 2015 at 10:00 AM, Thejas Nair 
> wrote:
> > Would restart of jenkins or the hosts help with this ?
> >
> >
> > On Tue, Dec 1, 2015 at 9:59 AM, Thejas Nair 
> wrote:
> >> The build running right now has been running for almost 10 hours.
> >> I am wondering if there is something wrong with the hosts being used
> >> right now. Some commands seem to be taking longer and I get some
> >> intermittent errors from the jenkins UI.
> >>
> >>
> http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6187/
> >> This rsync command seems to have taken a long time to finish -
> >>
> >> 2015-12-01 11:42:41,394  INFO LocalCommand.:35 Starting timeout
> >> 1h rsync -vaPe "ssh -i /home/hiveptest/.ssh/hive-ptest-user-key"
> >> --timeout 600 hiveptest@54.159.202.234:
> /home/hiveptest/54.159.202.234-hiveptest-0/logs/
> >>
> /data/hive-ptest/logs/PreCommit-HIVE-TRUNK-Build-6187/succeeded/TestHBaseMinimrCliDriver
> >> 2015-12-01 12:54:52,522  INFO LocalCommand.:35 Starting timeout
> >> 1h rsync -vaPe "ssh -i /home/hiveptest/.ssh/hive-ptest-user-key"
> >> --timeout 600 hiveptest@54.157.22.221:/home/hiveptest/54.157.22.221
> -hiveptest-0/logs/
> >>
> /data/hive-ptest/logs/PreCommit-HIVE-TRUNK-Build-6187/failed/TestMiniTezCliDriver-script_pipe.q-tez_smb_empty.q-mapjoin_decimal.q-and-12-more
> >> 2015-12-01 12:54:54,240  INFO HostExecutor.executeTests:176 Starting
> >> isolated execution on 54.157.22.221
> >>
> >>
> >>
> >>
> >>
> >> On Tue, Dec 1, 2015 at 8:57 AM, Sergio Pena 
> wrote:
> >>> I see HIVE-12541 files were attached 4 times on JIRA while 'Patch
> >>> Available' status was set. I does not say at what time, but Jenkins
> runs a
> >>> job every 5 min to detect which files must be tested for all "Patch
> >>> Available' JIRAS. That's why you probably saw HIVE-12541 in the queue
> many
> >>> times.
> >>>
> >>> I would recommend to set the JIRA to 'Patch Available' when the patch
> is
> >>> reviewed to avoid many patches to be added to the queue.
> >>>
> >>> On Mon, Nov 30, 2015 at 11:09 PM, Prasanth J 
> wrote:
> >>>
>  Hi All
> 
>  I am noticing something weird in precommit queue. I can see
>  HIVE-12541currenly running, but is also waiting next in the queue.
> The same
>  jira ran few more times recently. I cannot see any other patches in
> the
>  queue. Sometime I see bunch of patches in the queue but it will soon
> be
>  removed. But I am seeing HIVE-12541 running quite often. Has anyone
> seen
>  this issue?
> 
>  Thanks
>  Prasanth
>

Re: Precommit unstable?

[Sergio Pena]

I see HIVE-12541 files were attached 4 times on JIRA while 'Patch
Available' status was set. I does not say at what time, but Jenkins runs a
job every 5 min to detect which files must be tested for all "Patch
Available' JIRAS. That's why you probably saw HIVE-12541 in the queue many
times.

I would recommend to set the JIRA to 'Patch Available' when the patch is
reviewed to avoid many patches to be added to the queue.

On Mon, Nov 30, 2015 at 11:09 PM, Prasanth J  wrote:

> Hi All
>
> I am noticing something weird in precommit queue. I can see
> HIVE-12541currenly running, but is also waiting next in the queue. The same
> jira ran few more times recently. I cannot see any other patches in the
> queue. Sometime I see bunch of patches in the queue but it will soon be
> removed. But I am seeing HIVE-12541 running quite often. Has anyone seen
> this issue?
>
> Thanks
> Prasanth

Re: Precommit unstable?

[Thejas Nair]

Would restart of jenkins or the hosts help with this ?


On Tue, Dec 1, 2015 at 9:59 AM, Thejas Nair  wrote:
> The build running right now has been running for almost 10 hours.
> I am wondering if there is something wrong with the hosts being used
> right now. Some commands seem to be taking longer and I get some
> intermittent errors from the jenkins UI.
>
> http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6187/
> This rsync command seems to have taken a long time to finish -
>
> 2015-12-01 11:42:41,394  INFO LocalCommand.:35 Starting timeout
> 1h rsync -vaPe "ssh -i /home/hiveptest/.ssh/hive-ptest-user-key"
> --timeout 600 
> hiveptest@54.159.202.234:/home/hiveptest/54.159.202.234-hiveptest-0/logs/
> /data/hive-ptest/logs/PreCommit-HIVE-TRUNK-Build-6187/succeeded/TestHBaseMinimrCliDriver
> 2015-12-01 12:54:52,522  INFO LocalCommand.:35 Starting timeout
> 1h rsync -vaPe "ssh -i /home/hiveptest/.ssh/hive-ptest-user-key"
> --timeout 600 
> hiveptest@54.157.22.221:/home/hiveptest/54.157.22.221-hiveptest-0/logs/
> /data/hive-ptest/logs/PreCommit-HIVE-TRUNK-Build-6187/failed/TestMiniTezCliDriver-script_pipe.q-tez_smb_empty.q-mapjoin_decimal.q-and-12-more
> 2015-12-01 12:54:54,240  INFO HostExecutor.executeTests:176 Starting
> isolated execution on 54.157.22.221
>
>
>
>
>
> On Tue, Dec 1, 2015 at 8:57 AM, Sergio Pena  wrote:
>> I see HIVE-12541 files were attached 4 times on JIRA while 'Patch
>> Available' status was set. I does not say at what time, but Jenkins runs a
>> job every 5 min to detect which files must be tested for all "Patch
>> Available' JIRAS. That's why you probably saw HIVE-12541 in the queue many
>> times.
>>
>> I would recommend to set the JIRA to 'Patch Available' when the patch is
>> reviewed to avoid many patches to be added to the queue.
>>
>> On Mon, Nov 30, 2015 at 11:09 PM, Prasanth J  wrote:
>>
>>> Hi All
>>>
>>> I am noticing something weird in precommit queue. I can see
>>> HIVE-12541currenly running, but is also waiting next in the queue. The same
>>> jira ran few more times recently. I cannot see any other patches in the
>>> queue. Sometime I see bunch of patches in the queue but it will soon be
>>> removed. But I am seeing HIVE-12541 running quite often. Has anyone seen
>>> this issue?
>>>
>>> Thanks
>>> Prasanth

Re: Precommit unstable?

[Thejas Nair]

The build running right now has been running for almost 10 hours.
I am wondering if there is something wrong with the hosts being used
right now. Some commands seem to be taking longer and I get some
intermittent errors from the jenkins UI.

http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6187/
This rsync command seems to have taken a long time to finish -

2015-12-01 11:42:41,394  INFO LocalCommand.:35 Starting timeout
1h rsync -vaPe "ssh -i /home/hiveptest/.ssh/hive-ptest-user-key"
--timeout 600 
hiveptest@54.159.202.234:/home/hiveptest/54.159.202.234-hiveptest-0/logs/
/data/hive-ptest/logs/PreCommit-HIVE-TRUNK-Build-6187/succeeded/TestHBaseMinimrCliDriver
2015-12-01 12:54:52,522  INFO LocalCommand.:35 Starting timeout
1h rsync -vaPe "ssh -i /home/hiveptest/.ssh/hive-ptest-user-key"
--timeout 600 
hiveptest@54.157.22.221:/home/hiveptest/54.157.22.221-hiveptest-0/logs/
/data/hive-ptest/logs/PreCommit-HIVE-TRUNK-Build-6187/failed/TestMiniTezCliDriver-script_pipe.q-tez_smb_empty.q-mapjoin_decimal.q-and-12-more
2015-12-01 12:54:54,240  INFO HostExecutor.executeTests:176 Starting
isolated execution on 54.157.22.221





On Tue, Dec 1, 2015 at 8:57 AM, Sergio Pena  wrote:
> I see HIVE-12541 files were attached 4 times on JIRA while 'Patch
> Available' status was set. I does not say at what time, but Jenkins runs a
> job every 5 min to detect which files must be tested for all "Patch
> Available' JIRAS. That's why you probably saw HIVE-12541 in the queue many
> times.
>
> I would recommend to set the JIRA to 'Patch Available' when the patch is
> reviewed to avoid many patches to be added to the queue.
>
> On Mon, Nov 30, 2015 at 11:09 PM, Prasanth J  wrote:
>
>> Hi All
>>
>> I am noticing something weird in precommit queue. I can see
>> HIVE-12541currenly running, but is also waiting next in the queue. The same
>> jira ran few more times recently. I cannot see any other patches in the
>> queue. Sometime I see bunch of patches in the queue but it will soon be
>> removed. But I am seeing HIVE-12541 running quite often. Has anyone seen
>> this issue?
>>
>> Thanks
>> Prasanth

Re: Precommit unstable?

[Thejas Nair]

fyi, several patches in queue failed to get tested because of some
failures in cloning from github. That could be an unrelated temporary
problem.


On Tue, Dec 1, 2015 at 9:59 AM, Thejas Nair  wrote:
> The build running right now has been running for almost 10 hours.
> I am wondering if there is something wrong with the hosts being used
> right now. Some commands seem to be taking longer and I get some
> intermittent errors from the jenkins UI.
>
> http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6187/
> This rsync command seems to have taken a long time to finish -
>
> 2015-12-01 11:42:41,394  INFO LocalCommand.:35 Starting timeout
> 1h rsync -vaPe "ssh -i /home/hiveptest/.ssh/hive-ptest-user-key"
> --timeout 600 
> hiveptest@54.159.202.234:/home/hiveptest/54.159.202.234-hiveptest-0/logs/
> /data/hive-ptest/logs/PreCommit-HIVE-TRUNK-Build-6187/succeeded/TestHBaseMinimrCliDriver
> 2015-12-01 12:54:52,522  INFO LocalCommand.:35 Starting timeout
> 1h rsync -vaPe "ssh -i /home/hiveptest/.ssh/hive-ptest-user-key"
> --timeout 600 
> hiveptest@54.157.22.221:/home/hiveptest/54.157.22.221-hiveptest-0/logs/
> /data/hive-ptest/logs/PreCommit-HIVE-TRUNK-Build-6187/failed/TestMiniTezCliDriver-script_pipe.q-tez_smb_empty.q-mapjoin_decimal.q-and-12-more
> 2015-12-01 12:54:54,240  INFO HostExecutor.executeTests:176 Starting
> isolated execution on 54.157.22.221
>
>
>
>
>
> On Tue, Dec 1, 2015 at 8:57 AM, Sergio Pena  wrote:
>> I see HIVE-12541 files were attached 4 times on JIRA while 'Patch
>> Available' status was set. I does not say at what time, but Jenkins runs a
>> job every 5 min to detect which files must be tested for all "Patch
>> Available' JIRAS. That's why you probably saw HIVE-12541 in the queue many
>> times.
>>
>> I would recommend to set the JIRA to 'Patch Available' when the patch is
>> reviewed to avoid many patches to be added to the queue.
>>
>> On Mon, Nov 30, 2015 at 11:09 PM, Prasanth J  wrote:
>>
>>> Hi All
>>>
>>> I am noticing something weird in precommit queue. I can see
>>> HIVE-12541currenly running, but is also waiting next in the queue. The same
>>> jira ran few more times recently. I cannot see any other patches in the
>>> queue. Sometime I see bunch of patches in the queue but it will soon be
>>> removed. But I am seeing HIVE-12541 running quite often. Has anyone seen
>>> this issue?
>>>
>>> Thanks
>>> Prasanth

Re: Precommit unstable?

[Thejas Nair]

I killed the current 10+hr test run to see if next test would run
faster. (Discussed with Prasanth, the author of patch being tested).

However,  I doubt if the next one is going to perform any better. If
we could try restarting the ptest2 system, I think that might be
better thing to do (instead of waiting several hours to realize that
the issue has not gone away).


On Tue, Dec 1, 2015 at 10:00 AM, Thejas Nair  wrote:
> Would restart of jenkins or the hosts help with this ?
>
>
> On Tue, Dec 1, 2015 at 9:59 AM, Thejas Nair  wrote:
>> The build running right now has been running for almost 10 hours.
>> I am wondering if there is something wrong with the hosts being used
>> right now. Some commands seem to be taking longer and I get some
>> intermittent errors from the jenkins UI.
>>
>> http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6187/
>> This rsync command seems to have taken a long time to finish -
>>
>> 2015-12-01 11:42:41,394  INFO LocalCommand.:35 Starting timeout
>> 1h rsync -vaPe "ssh -i /home/hiveptest/.ssh/hive-ptest-user-key"
>> --timeout 600 
>> hiveptest@54.159.202.234:/home/hiveptest/54.159.202.234-hiveptest-0/logs/
>> /data/hive-ptest/logs/PreCommit-HIVE-TRUNK-Build-6187/succeeded/TestHBaseMinimrCliDriver
>> 2015-12-01 12:54:52,522  INFO LocalCommand.:35 Starting timeout
>> 1h rsync -vaPe "ssh -i /home/hiveptest/.ssh/hive-ptest-user-key"
>> --timeout 600 
>> hiveptest@54.157.22.221:/home/hiveptest/54.157.22.221-hiveptest-0/logs/
>> /data/hive-ptest/logs/PreCommit-HIVE-TRUNK-Build-6187/failed/TestMiniTezCliDriver-script_pipe.q-tez_smb_empty.q-mapjoin_decimal.q-and-12-more
>> 2015-12-01 12:54:54,240  INFO HostExecutor.executeTests:176 Starting
>> isolated execution on 54.157.22.221
>>
>>
>>
>>
>>
>> On Tue, Dec 1, 2015 at 8:57 AM, Sergio Pena  wrote:
>>> I see HIVE-12541 files were attached 4 times on JIRA while 'Patch
>>> Available' status was set. I does not say at what time, but Jenkins runs a
>>> job every 5 min to detect which files must be tested for all "Patch
>>> Available' JIRAS. That's why you probably saw HIVE-12541 in the queue many
>>> times.
>>>
>>> I would recommend to set the JIRA to 'Patch Available' when the patch is
>>> reviewed to avoid many patches to be added to the queue.
>>>
>>> On Mon, Nov 30, 2015 at 11:09 PM, Prasanth J  wrote:
>>>
 Hi All

 I am noticing something weird in precommit queue. I can see
 HIVE-12541currenly running, but is also waiting next in the queue. The same
 jira ran few more times recently. I cannot see any other patches in the
 queue. Sometime I see bunch of patches in the queue but it will soon be
 removed. But I am seeing HIVE-12541 running quite often. Has anyone seen
 this issue?

 Thanks
 Prasanth

Precommit unstable?

[Prasanth J]

Hi All

I am noticing something weird in precommit queue. I can see HIVE-12541currenly 
running, but is also waiting next in the queue. The same jira ran few more 
times recently. I cannot see any other patches in the queue. Sometime I see 
bunch of patches in the queue but it will soon be removed. But I am seeing 
HIVE-12541 running quite often. Has anyone seen this issue?

Thanks
Prasanth