This is another argument for getting the code to the point where this can
default to "true":
SQLConf.scala: val ADAPTIVE_EXECUTION_ENABLED = buildConf("
*spark.sql.adaptive.enabled*")
On Tue, Aug 22, 2017 at 12:27 PM, Reynold Xin <r...@databricks.com> wrote:
> +1
>
>
> On Tue, Aug 22, 2017 at 12:25 PM, Maciej Szymkiewicz <
> mszymkiew...@gmail.com> wrote:
>
>> Hi,
>>
>> From my experience it is possible to cut quite a lot by reducing
>> spark.sql.shuffle.partitions to some reasonable value (let's say
>> comparable to the number of cores). 200 is a serious overkill for most of
>> the test cases anyway.
>>
>>
>> Best,
>> Maciej
>>
>>
>>
>> On 21 August 2017 at 03:00, Dong Joon Hyun <dh...@hortonworks.com> wrote:
>>
>>> +1 for any efforts to recover Jenkins!
>>>
>>>
>>>
>>> Thank you for the direction.
>>>
>>>
>>>
>>> Bests,
>>>
>>> Dongjoon.
>>>
>>>
>>>
>>> *From: *Reynold Xin <r...@databricks.com>
>>> *Date: *Sunday, August 20, 2017 at 5:53 PM
>>> *To: *Dong Joon Hyun <dh...@hortonworks.com>
>>> *Cc: *"dev@spark.apache.org" <dev@spark.apache.org>
>>> *Subject: *Re: Increase Timeout or optimize Spark UT?
>>>
>>>
>>>
>>> It seems like it's time to look into how to cut down some of the test
>>> runtimes. Test runtimes will slowly go up given the way development
>>> happens. 3 hr is already a very long time for tests to run.
>>>
>>>
>>>
>>>
>>>
>>> On Sun, Aug 20, 2017 at 5:45 PM, Dong Joon Hyun <dh...@hortonworks.com>
>>> wrote:
>>>
>>> Hi, All.
>>>
>>>
>>>
>>> Recently, Apache Spark master branch test (SBT with hadoop-2.7 / 2.6)
>>> has been hitting the build timeout.
>>>
>>>
>>>
>>> Please see the build time trend.
>>>
>>>
>>>
>>> https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Tes
>>> t%20(Dashboard)/job/spark-master-test-sbt-hadoop-2.7/buildTimeTrend
>>>
>>>
>>>
>>> All recent 22 builds fail due to timeout directly/indirectly. The last
>>> success (SBT with Hadoop-2.7) is 15th August.
>>>
>>>
>>>
>>> We may do the followings.
>>>
>>>
>>>
>>> 1. Increase Build Timeout (3 hr 30 min)
>>> 2. Optimize UTs (Scala/Java/Python/UT)
>>>
>>>
>>>
>>> But, Option 1 will be the immediate solution for now . Could you update
>>> the Jenkins setup?
>>>
>>>
>>>
>>> Bests,
>>>
>>> Dongjoon.
>>>
>>>
>>>
>>
>>
>
