Re: Increase Timeout or optimize Spark UT?
BTW, the situation seems to become worse, now we lost two builds. - https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test%20(Dashboard)/job/spark-master-test-sbt-hadoop-2.7/ - https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test%20(Dashboard)/job/spark-master-test-sbt-hadoop-2.6/ Dongjoon. On Thu, Aug 24, 2017 at 2:27 AM, Maciej Szymkiewicz wrote: > It won't be used by PySpark and SparkR, will it? > > On 23 August 2017 at 23:40, Michael Armbrust > wrote: > >> I think we already set the number of partitions to 5 in tests >> <https://github.com/apache/spark/blob/6942aeeb0a0095a1ba85a817eb9e0edc410e5624/sql/core/src/test/scala/org/apache/spark/sql/test/TestSQLContext.scala#L60-L61> >> ? >> >> On Tue, Aug 22, 2017 at 3:25 PM, Maciej Szymkiewicz < >> mszymkiew...@gmail.com> wrote: >> >>> Hi, >>> >>> From my experience it is possible to cut quite a lot by reducing >>> spark.sql.shuffle.partitions to some reasonable value (let's say >>> comparable to the number of cores). 200 is a serious overkill for most of >>> the test cases anyway. >>> >>> >>> Best, >>> Maciej >>> >>> >>> >>> On 21 August 2017 at 03:00, Dong Joon Hyun >>> wrote: >>> >>>> +1 for any efforts to recover Jenkins! >>>> >>>> >>>> >>>> Thank you for the direction. >>>> >>>> >>>> >>>> Bests, >>>> >>>> Dongjoon. >>>> >>>> >>>> >>>> *From: *Reynold Xin >>>> *Date: *Sunday, August 20, 2017 at 5:53 PM >>>> *To: *Dong Joon Hyun >>>> *Cc: *"dev@spark.apache.org" >>>> *Subject: *Re: Increase Timeout or optimize Spark UT? >>>> >>>> >>>> >>>> It seems like it's time to look into how to cut down some of the test >>>> runtimes. Test runtimes will slowly go up given the way development >>>> happens. 3 hr is already a very long time for tests to run. >>>> >>>> >>>> >>>> >>>> >>>> On Sun, Aug 20, 2017 at 5:45 PM, Dong Joon Hyun >>>> wrote: >>>> >>>> Hi, All. >>>> >>>> >>>> >>>> Recently, Apache Spark master branch test (SBT with hadoop-2.7 / 2.6) >>>> has been hitting the build timeout. >>>> >>>> >>>> >>>> Please see the build time trend. >>>> >>>> >>>> >>>> https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Tes >>>> t%20(Dashboard)/job/spark-master-test-sbt-hadoop-2.7/buildTimeTrend >>>> >>>> >>>> >>>> All recent 22 builds fail due to timeout directly/indirectly. The last >>>> success (SBT with Hadoop-2.7) is 15th August. >>>> >>>> >>>> >>>> We may do the followings. >>>> >>>> >>>> >>>>1. Increase Build Timeout (3 hr 30 min) >>>>2. Optimize UTs (Scala/Java/Python/UT) >>>> >>>> >>>> >>>> But, Option 1 will be the immediate solution for now . Could you update >>>> the Jenkins setup? >>>> >>>> >>>> >>>> Bests, >>>> >>>> Dongjoon. >>>> >>>> >>>> >>> >>> >> > > > -- > > Z poważaniem, > Maciej Szymkiewicz > >
Re: Increase Timeout or optimize Spark UT?
It won't be used by PySpark and SparkR, will it? On 23 August 2017 at 23:40, Michael Armbrust wrote: > I think we already set the number of partitions to 5 in tests > <https://github.com/apache/spark/blob/6942aeeb0a0095a1ba85a817eb9e0edc410e5624/sql/core/src/test/scala/org/apache/spark/sql/test/TestSQLContext.scala#L60-L61> > ? > > On Tue, Aug 22, 2017 at 3:25 PM, Maciej Szymkiewicz < > mszymkiew...@gmail.com> wrote: > >> Hi, >> >> From my experience it is possible to cut quite a lot by reducing >> spark.sql.shuffle.partitions to some reasonable value (let's say >> comparable to the number of cores). 200 is a serious overkill for most of >> the test cases anyway. >> >> >> Best, >> Maciej >> >> >> >> On 21 August 2017 at 03:00, Dong Joon Hyun wrote: >> >>> +1 for any efforts to recover Jenkins! >>> >>> >>> >>> Thank you for the direction. >>> >>> >>> >>> Bests, >>> >>> Dongjoon. >>> >>> >>> >>> *From: *Reynold Xin >>> *Date: *Sunday, August 20, 2017 at 5:53 PM >>> *To: *Dong Joon Hyun >>> *Cc: *"dev@spark.apache.org" >>> *Subject: *Re: Increase Timeout or optimize Spark UT? >>> >>> >>> >>> It seems like it's time to look into how to cut down some of the test >>> runtimes. Test runtimes will slowly go up given the way development >>> happens. 3 hr is already a very long time for tests to run. >>> >>> >>> >>> >>> >>> On Sun, Aug 20, 2017 at 5:45 PM, Dong Joon Hyun >>> wrote: >>> >>> Hi, All. >>> >>> >>> >>> Recently, Apache Spark master branch test (SBT with hadoop-2.7 / 2.6) >>> has been hitting the build timeout. >>> >>> >>> >>> Please see the build time trend. >>> >>> >>> >>> https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Tes >>> t%20(Dashboard)/job/spark-master-test-sbt-hadoop-2.7/buildTimeTrend >>> >>> >>> >>> All recent 22 builds fail due to timeout directly/indirectly. The last >>> success (SBT with Hadoop-2.7) is 15th August. >>> >>> >>> >>> We may do the followings. >>> >>> >>> >>>1. Increase Build Timeout (3 hr 30 min) >>>2. Optimize UTs (Scala/Java/Python/UT) >>> >>> >>> >>> But, Option 1 will be the immediate solution for now . Could you update >>> the Jenkins setup? >>> >>> >>> >>> Bests, >>> >>> Dongjoon. >>> >>> >>> >> >> > -- Z poważaniem, Maciej Szymkiewicz
Re: Increase Timeout or optimize Spark UT?
I think we already set the number of partitions to 5 in tests <https://github.com/apache/spark/blob/6942aeeb0a0095a1ba85a817eb9e0edc410e5624/sql/core/src/test/scala/org/apache/spark/sql/test/TestSQLContext.scala#L60-L61> ? On Tue, Aug 22, 2017 at 3:25 PM, Maciej Szymkiewicz wrote: > Hi, > > From my experience it is possible to cut quite a lot by reducing > spark.sql.shuffle.partitions to some reasonable value (let's say > comparable to the number of cores). 200 is a serious overkill for most of > the test cases anyway. > > > Best, > Maciej > > > > On 21 August 2017 at 03:00, Dong Joon Hyun wrote: > >> +1 for any efforts to recover Jenkins! >> >> >> >> Thank you for the direction. >> >> >> >> Bests, >> >> Dongjoon. >> >> >> >> *From: *Reynold Xin >> *Date: *Sunday, August 20, 2017 at 5:53 PM >> *To: *Dong Joon Hyun >> *Cc: *"dev@spark.apache.org" >> *Subject: *Re: Increase Timeout or optimize Spark UT? >> >> >> >> It seems like it's time to look into how to cut down some of the test >> runtimes. Test runtimes will slowly go up given the way development >> happens. 3 hr is already a very long time for tests to run. >> >> >> >> >> >> On Sun, Aug 20, 2017 at 5:45 PM, Dong Joon Hyun >> wrote: >> >> Hi, All. >> >> >> >> Recently, Apache Spark master branch test (SBT with hadoop-2.7 / 2.6) has >> been hitting the build timeout. >> >> >> >> Please see the build time trend. >> >> >> >> https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Tes >> t%20(Dashboard)/job/spark-master-test-sbt-hadoop-2.7/buildTimeTrend >> >> >> >> All recent 22 builds fail due to timeout directly/indirectly. The last >> success (SBT with Hadoop-2.7) is 15th August. >> >> >> >> We may do the followings. >> >> >> >>1. Increase Build Timeout (3 hr 30 min) >>2. Optimize UTs (Scala/Java/Python/UT) >> >> >> >> But, Option 1 will be the immediate solution for now . Could you update >> the Jenkins setup? >> >> >> >> Bests, >> >> Dongjoon. >> >> >> > >
Re: Increase Timeout or optimize Spark UT?
This is another argument for getting the code to the point where this can
default to "true":
SQLConf.scala: val ADAPTIVE_EXECUTION_ENABLED = buildConf("
*spark.sql.adaptive.enabled*")
On Tue, Aug 22, 2017 at 12:27 PM, Reynold Xin wrote:
> +1
>
>
> On Tue, Aug 22, 2017 at 12:25 PM, Maciej Szymkiewicz <
> mszymkiew...@gmail.com> wrote:
>
>> Hi,
>>
>> From my experience it is possible to cut quite a lot by reducing
>> spark.sql.shuffle.partitions to some reasonable value (let's say
>> comparable to the number of cores). 200 is a serious overkill for most of
>> the test cases anyway.
>>
>>
>> Best,
>> Maciej
>>
>>
>>
>> On 21 August 2017 at 03:00, Dong Joon Hyun wrote:
>>
>>> +1 for any efforts to recover Jenkins!
>>>
>>>
>>>
>>> Thank you for the direction.
>>>
>>>
>>>
>>> Bests,
>>>
>>> Dongjoon.
>>>
>>>
>>>
>>> *From: *Reynold Xin
>>> *Date: *Sunday, August 20, 2017 at 5:53 PM
>>> *To: *Dong Joon Hyun
>>> *Cc: *"dev@spark.apache.org"
>>> *Subject: *Re: Increase Timeout or optimize Spark UT?
>>>
>>>
>>>
>>> It seems like it's time to look into how to cut down some of the test
>>> runtimes. Test runtimes will slowly go up given the way development
>>> happens. 3 hr is already a very long time for tests to run.
>>>
>>>
>>>
>>>
>>>
>>> On Sun, Aug 20, 2017 at 5:45 PM, Dong Joon Hyun
>>> wrote:
>>>
>>> Hi, All.
>>>
>>>
>>>
>>> Recently, Apache Spark master branch test (SBT with hadoop-2.7 / 2.6)
>>> has been hitting the build timeout.
>>>
>>>
>>>
>>> Please see the build time trend.
>>>
>>>
>>>
>>> https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Tes
>>> t%20(Dashboard)/job/spark-master-test-sbt-hadoop-2.7/buildTimeTrend
>>>
>>>
>>>
>>> All recent 22 builds fail due to timeout directly/indirectly. The last
>>> success (SBT with Hadoop-2.7) is 15th August.
>>>
>>>
>>>
>>> We may do the followings.
>>>
>>>
>>>
>>>1. Increase Build Timeout (3 hr 30 min)
>>>2. Optimize UTs (Scala/Java/Python/UT)
>>>
>>>
>>>
>>> But, Option 1 will be the immediate solution for now . Could you update
>>> the Jenkins setup?
>>>
>>>
>>>
>>> Bests,
>>>
>>> Dongjoon.
>>>
>>>
>>>
>>
>>
>
Re: Increase Timeout or optimize Spark UT?
+1 On Tue, Aug 22, 2017 at 12:25 PM, Maciej Szymkiewicz wrote: > Hi, > > From my experience it is possible to cut quite a lot by reducing > spark.sql.shuffle.partitions to some reasonable value (let's say > comparable to the number of cores). 200 is a serious overkill for most of > the test cases anyway. > > > Best, > Maciej > > > > On 21 August 2017 at 03:00, Dong Joon Hyun wrote: > >> +1 for any efforts to recover Jenkins! >> >> >> >> Thank you for the direction. >> >> >> >> Bests, >> >> Dongjoon. >> >> >> >> *From: *Reynold Xin >> *Date: *Sunday, August 20, 2017 at 5:53 PM >> *To: *Dong Joon Hyun >> *Cc: *"dev@spark.apache.org" >> *Subject: *Re: Increase Timeout or optimize Spark UT? >> >> >> >> It seems like it's time to look into how to cut down some of the test >> runtimes. Test runtimes will slowly go up given the way development >> happens. 3 hr is already a very long time for tests to run. >> >> >> >> >> >> On Sun, Aug 20, 2017 at 5:45 PM, Dong Joon Hyun >> wrote: >> >> Hi, All. >> >> >> >> Recently, Apache Spark master branch test (SBT with hadoop-2.7 / 2.6) has >> been hitting the build timeout. >> >> >> >> Please see the build time trend. >> >> >> >> https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Tes >> t%20(Dashboard)/job/spark-master-test-sbt-hadoop-2.7/buildTimeTrend >> >> >> >> All recent 22 builds fail due to timeout directly/indirectly. The last >> success (SBT with Hadoop-2.7) is 15th August. >> >> >> >> We may do the followings. >> >> >> >>1. Increase Build Timeout (3 hr 30 min) >>2. Optimize UTs (Scala/Java/Python/UT) >> >> >> >> But, Option 1 will be the immediate solution for now . Could you update >> the Jenkins setup? >> >> >> >> Bests, >> >> Dongjoon. >> >> >> > >
Re: Increase Timeout or optimize Spark UT?
Hi, >From my experience it is possible to cut quite a lot by reducing spark.sql.shuffle.partitions to some reasonable value (let's say comparable to the number of cores). 200 is a serious overkill for most of the test cases anyway. Best, Maciej On 21 August 2017 at 03:00, Dong Joon Hyun wrote: > +1 for any efforts to recover Jenkins! > > > > Thank you for the direction. > > > > Bests, > > Dongjoon. > > > > *From: *Reynold Xin > *Date: *Sunday, August 20, 2017 at 5:53 PM > *To: *Dong Joon Hyun > *Cc: *"dev@spark.apache.org" > *Subject: *Re: Increase Timeout or optimize Spark UT? > > > > It seems like it's time to look into how to cut down some of the test > runtimes. Test runtimes will slowly go up given the way development > happens. 3 hr is already a very long time for tests to run. > > > > > > On Sun, Aug 20, 2017 at 5:45 PM, Dong Joon Hyun > wrote: > > Hi, All. > > > > Recently, Apache Spark master branch test (SBT with hadoop-2.7 / 2.6) has > been hitting the build timeout. > > > > Please see the build time trend. > > > > https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA% > 20Test%20(Dashboard)/job/spark-master-test-sbt-hadoop-2.7/buildTimeTrend > > > > All recent 22 builds fail due to timeout directly/indirectly. The last > success (SBT with Hadoop-2.7) is 15th August. > > > > We may do the followings. > > > >1. Increase Build Timeout (3 hr 30 min) >2. Optimize UTs (Scala/Java/Python/UT) > > > > But, Option 1 will be the immediate solution for now . Could you update > the Jenkins setup? > > > > Bests, > > Dongjoon. > > >
Re: Increase Timeout or optimize Spark UT?
+1 for any efforts to recover Jenkins! Thank you for the direction. Bests, Dongjoon. From: Reynold Xin Date: Sunday, August 20, 2017 at 5:53 PM To: Dong Joon Hyun Cc: "dev@spark.apache.org" Subject: Re: Increase Timeout or optimize Spark UT? It seems like it's time to look into how to cut down some of the test runtimes. Test runtimes will slowly go up given the way development happens. 3 hr is already a very long time for tests to run. On Sun, Aug 20, 2017 at 5:45 PM, Dong Joon Hyun mailto:dh...@hortonworks.com>> wrote: Hi, All. Recently, Apache Spark master branch test (SBT with hadoop-2.7 / 2.6) has been hitting the build timeout. Please see the build time trend. https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test%20(Dashboard)/job/spark-master-test-sbt-hadoop-2.7/buildTimeTrend All recent 22 builds fail due to timeout directly/indirectly. The last success (SBT with Hadoop-2.7) is 15th August. We may do the followings. 1. Increase Build Timeout (3 hr 30 min) 2. Optimize UTs (Scala/Java/Python/UT) But, Option 1 will be the immediate solution for now . Could you update the Jenkins setup? Bests, Dongjoon.
Re: Increase Timeout or optimize Spark UT?
It seems like it's time to look into how to cut down some of the test runtimes. Test runtimes will slowly go up given the way development happens. 3 hr is already a very long time for tests to run. On Sun, Aug 20, 2017 at 5:45 PM, Dong Joon Hyun wrote: > Hi, All. > > > > Recently, Apache Spark master branch test (SBT with hadoop-2.7 / 2.6) has > been hitting the build timeout. > > > > Please see the build time trend. > > > > https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA% > 20Test%20(Dashboard)/job/spark-master-test-sbt-hadoop-2.7/buildTimeTrend > > > > All recent 22 builds fail due to timeout directly/indirectly. The last > success (SBT with Hadoop-2.7) is 15th August. > > > > We may do the followings. > > > >1. Increase Build Timeout (3 hr 30 min) >2. Optimize UTs (Scala/Java/Python/UT) > > > > But, Option 1 will be the immediate solution for now . Could you update > the Jenkins setup? > > > > Bests, > > Dongjoon. >
