Re: Build timed out for `branch-2.3 (hadoop-2.7)`

[shane knapp]

this doesn't have anything to do w/the git timeouts...  those will timeout
the build 10 mins after starting (and failing on the initial fetch call).

On Wed, Jan 17, 2018 at 9:51 PM, Sameer Agarwal  wrote:

> FYI, I ended up bumping the build timeouts from 255 to 275 minutes. All
> successful
> 
>  2.3
> (hadoop-2.7) builds last week were already taking 245-250 mins and had
> started timing out earlier today (towards the very end; while making
> consistent progress throughout). Increasing the timeout resolves the issue.
>
> NB: This might be either due to additional tests that were recently added
> or due to the git delays that Shane reported; we haven't investigated the
> root cause yet.
>
> On 12 January 2018 at 16:37, Dongjoon Hyun 
> wrote:
>
>> For this issue, during SPARK-23028, Shane shared that the server limit is
>> already higher.
>>
>> 1. Xiao Li increased the timeout of Spark test script for `master` branch
>> first in the following commit.
>>
>> [SPARK-23028] Bump master branch version to 2.4.0-SNAPSHOT
>> 
>>
>> 2. Marco Gaido reports a flaky test suite and it turns out that the test
>> suite hangs in SPARK-23055
>> 
>>
>> 3. Sameer Agarwal swiftly reverts it.
>>
>> Thank you all!
>>
>> Let's wait and see the dashboard
>> 
>> .
>>
>> Bests,
>> Dongjoon.
>>
>>
>>
>> On Fri, Jan 12, 2018 at 3:22 PM, Shixiong(Ryan) Zhu <
>> shixi...@databricks.com> wrote:
>>
>>> FYI, we reverted a commit in https://github.com/apache/spar
>>> k/commit/55dbfbca37ce4c05f83180777ba3d4fe2d96a02e to fix the issue.
>>>
>>> On Fri, Jan 12, 2018 at 11:45 AM, Xin Lu  wrote:
>>>
 seems like someone should investigate what caused the build time to go
 up an hour and if it's expected or not.

 On Thu, Jan 11, 2018 at 7:37 PM, Dongjoon Hyun  wrote:

> Hi, All and Shane.
>
> Can we increase the build time for `branch-2.3` during 2.3 RC period?
>
> There are two known test issues, but the Jenkins on branch-2.3 with
> hadoop-2.7 fails with build timeout. So, it's difficult to monitor whether
> the branch is healthy or not.
>
> Build timed out (after 255 minutes). Marking the build as aborted.
> Build was aborted
> ...
> Finished: ABORTED
>
> - https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Tes
> t%20(Dashboard)/job/spark-branch-2.3-test-maven-hadoop-2.7/60/console
> - https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Tes
> t%20(Dashboard)/job/spark-branch-2.3-test-sbt-hadoop-2.7/47/console
>
> Bests,
> Dongjoon.
>


>>>
>>
>

Re: Build timed out for `branch-2.3 (hadoop-2.7)`

[Sameer Agarwal]

FYI, I ended up bumping the build timeouts from 255 to 275 minutes. All
successful

2.3
(hadoop-2.7) builds last week were already taking 245-250 mins and had
started timing out earlier today (towards the very end; while making
consistent progress throughout). Increasing the timeout resolves the issue.

NB: This might be either due to additional tests that were recently added
or due to the git delays that Shane reported; we haven't investigated the
root cause yet.

On 12 January 2018 at 16:37, Dongjoon Hyun  wrote:

> For this issue, during SPARK-23028, Shane shared that the server limit is
> already higher.
>
> 1. Xiao Li increased the timeout of Spark test script for `master` branch
> first in the following commit.
>
> [SPARK-23028] Bump master branch version to 2.4.0-SNAPSHOT
> 
>
> 2. Marco Gaido reports a flaky test suite and it turns out that the test
> suite hangs in SPARK-23055
> 
>
> 3. Sameer Agarwal swiftly reverts it.
>
> Thank you all!
>
> Let's wait and see the dashboard
> 
> .
>
> Bests,
> Dongjoon.
>
>
>
> On Fri, Jan 12, 2018 at 3:22 PM, Shixiong(Ryan) Zhu <
> shixi...@databricks.com> wrote:
>
>> FYI, we reverted a commit in https://github.com/apache/spar
>> k/commit/55dbfbca37ce4c05f83180777ba3d4fe2d96a02e to fix the issue.
>>
>> On Fri, Jan 12, 2018 at 11:45 AM, Xin Lu  wrote:
>>
>>> seems like someone should investigate what caused the build time to go
>>> up an hour and if it's expected or not.
>>>
>>> On Thu, Jan 11, 2018 at 7:37 PM, Dongjoon Hyun 
>>> wrote:
>>>
 Hi, All and Shane.

 Can we increase the build time for `branch-2.3` during 2.3 RC period?

 There are two known test issues, but the Jenkins on branch-2.3 with
 hadoop-2.7 fails with build timeout. So, it's difficult to monitor whether
 the branch is healthy or not.

 Build timed out (after 255 minutes). Marking the build as aborted.
 Build was aborted
 ...
 Finished: ABORTED

 - https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Tes
 t%20(Dashboard)/job/spark-branch-2.3-test-maven-hadoop-2.7/60/console
 - https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Tes
 t%20(Dashboard)/job/spark-branch-2.3-test-sbt-hadoop-2.7/47/console

 Bests,
 Dongjoon.

>>>
>>>
>>
>

Re: Build timed out for `branch-2.3 (hadoop-2.7)`

[Shixiong(Ryan) Zhu]

FYI, we reverted a commit in
https://github.com/apache/spark/commit/55dbfbca37ce4c05f83180777ba3d4fe2d96a02e
to fix the issue.

On Fri, Jan 12, 2018 at 11:45 AM, Xin Lu  wrote:

> seems like someone should investigate what caused the build time to go up
> an hour and if it's expected or not.
>
> On Thu, Jan 11, 2018 at 7:37 PM, Dongjoon Hyun 
> wrote:
>
>> Hi, All and Shane.
>>
>> Can we increase the build time for `branch-2.3` during 2.3 RC period?
>>
>> There are two known test issues, but the Jenkins on branch-2.3 with
>> hadoop-2.7 fails with build timeout. So, it's difficult to monitor whether
>> the branch is healthy or not.
>>
>> Build timed out (after 255 minutes). Marking the build as aborted.
>> Build was aborted
>> ...
>> Finished: ABORTED
>>
>> - https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Tes
>> t%20(Dashboard)/job/spark-branch-2.3-test-maven-hadoop-2.7/60/console
>> - https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Tes
>> t%20(Dashboard)/job/spark-branch-2.3-test-sbt-hadoop-2.7/47/console
>>
>> Bests,
>> Dongjoon.
>>
>
>

Re: Build timed out for `branch-2.3 (hadoop-2.7)`

[Xin Lu]

seems like someone should investigate what caused the build time to go up
an hour and if it's expected or not.

On Thu, Jan 11, 2018 at 7:37 PM, Dongjoon Hyun 
wrote:

> Hi, All and Shane.
>
> Can we increase the build time for `branch-2.3` during 2.3 RC period?
>
> There are two known test issues, but the Jenkins on branch-2.3 with
> hadoop-2.7 fails with build timeout. So, it's difficult to monitor whether
> the branch is healthy or not.
>
> Build timed out (after 255 minutes). Marking the build as aborted.
> Build was aborted
> ...
> Finished: ABORTED
>
> - https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%
> 20Test%20(Dashboard)/job/spark-branch-2.3-test-maven-hadoop-2.7/60/console
> - https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%
> 20Test%20(Dashboard)/job/spark-branch-2.3-test-sbt-hadoop-2.7/47/console
>
> Bests,
> Dongjoon.
>

Build timed out for `branch-2.3 (hadoop-2.7)`

[Dongjoon Hyun]

Hi, All and Shane.

Can we increase the build time for `branch-2.3` during 2.3 RC period?

There are two known test issues, but the Jenkins on branch-2.3 with
hadoop-2.7 fails with build timeout. So, it's difficult to monitor whether
the branch is healthy or not.

Build timed out (after 255 minutes). Marking the build as aborted.
Build was aborted
...
Finished: ABORTED

-
https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test%20(Dashboard)/job/spark-branch-2.3-test-maven-hadoop-2.7/60/console
-
https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test%20(Dashboard)/job/spark-branch-2.3-test-sbt-hadoop-2.7/47/console

Bests,
Dongjoon.