Re: [julia-users] Test code taking forever to start

['Bill Hart' via julia-users]

I did a little bit more bisecting to try and find the most recent 
regression, but I'm not sure if I nailed it or not.

All the data is now summarised in the ticket here:

https://github.com/JuliaLang/julia/issues/15615

Re: [julia-users] Test code taking forever to start

['Bill Hart' via julia-users]

Rather than trying to nail down the remaining regressions, I think I am 
just going to give the data I've collected, which illustrates a problem, 
namely that in a very short space there can be massive improvements, 
immediately followed by massive regressions. This essentially violates the 
git bisect working assumption that there is one bad commit and you are 
trying to find it.

Here is a list of commits I have hit during various commits and the times I 
got for those commits. There was also another commit I hit where the time 
improved by a factor of 2, but all subsequent attempts to build any commits 
of Julia failed and so I had to clone the repository again. Because of 
this, I threw that data point away as likely corrupt.

41fb1ba good 15s
79a0d7b good
5ea20fc good
275c7e8 good
bea07fc good
c5704ed good
0318444 good

6396218 bad1
3b7f18a bad1
63daf4f bad1
1bfabbb bad1
179a439 bad1
0b6cab6 bad1
d8ec4a7 bad1
064b03c bad1
53b02a6 bad1
ca6f253 bad1 42s

8f4238a bad2 59s
f67203c bad2 53s
af81431 bad2 58s

6382116 bad3 73s 
24276a5 bad3 74s
79b08ca bad3 73s
93cb2ae bad3 71s
5baedf4 bad3 72s

70a3120 ??   48s

8af0134 bad4 82s
def50a0 bad4 78s
039f57b bad4 78s
597dc7b bad4 78s
adc1ed8 bad4 78s
fc469b6 bad4 78s
be9f208 bad5 87s
master  bad5 90s

In summary, the commits that I know are serious regressions for sure are:

6396218
8f4238a
6382116
8af0134

There's also a regression somewhere between fc469b6 and cf68f4d3 (the 
latter being master some time today) which I didn't bisect yet. At this 
point I'm not sure how useful this is.

Bill.

Re: [julia-users] Test code taking forever to start

[Stefan Karpinski]

Thanks for taking the time and effort to track these down, Bill. We should
try to distill some tests for each of these regressions so that we can
guard against performance regressions for the same sorts of tasks in the
future.

On Thu, Mar 24, 2016 at 9:20 AM, 'Bill Hart' via julia-users <
julia-users@googlegroups.com> wrote:

> The third regression is:
>
> 6382116 Jameson Nash <...@gmail.com> implement recursive
> type-inference
>
> It's about a 25% regression. There are some other much smaller regressions
> (~7%) before that which I skipped over.
>
> Still trying to track down a potential fourth regression.
>
> Bill.
>

Re: [julia-users] Test code taking forever to start

['Bill Hart' via julia-users]

The third regression is:

6382116 Jameson Nash <...@gmail.com> implement recursive type-inference

It's about a 25% regression. There are some other much smaller regressions 
(~7%) before that which I skipped over.

Still trying to track down a potential fourth regression.

Bill.

Re: [julia-users] Test code taking forever to start

[Kristoffer Carlsson]

These should imo all be on the issue tracker. AFAIU most of these commits 
where supposed to speed up type inference.

On Thursday, March 24, 2016 at 12:46:07 PM UTC+1, Bill Hart wrote:
>
> I'm sorry, please ignore that merge commit. I think that git bisect 
> screwed up due to the fact that there are in fact a couple of major 
> regressions in the range I was bisecting.
>
> The first of these is at the following commit:
>
> 8f4238a do type-inference using a work queue
>
> It's about a 35% regression.
>
> There are still at least two regressions to track down.
>
> Bill.
>

Re: [julia-users] Test code taking forever to start

['Bill Hart' via julia-users]

I'm sorry, please ignore that merge commit. I think that git bisect screwed 
up due to the fact that there are in fact a couple of major regressions in 
the range I was bisecting.

The first of these is at the following commit:

8f4238a do type-inference using a work queue

It's about a 35% regression.

There are still at least two regressions to track down.

Bill.

Re: [julia-users] Test code taking forever to start

['Bill Hart' via julia-users]

The second bad commit is:

7facd20 Merge pull request #15300 from JuliaLang/jn/typeinfq

However this is only about a 60% regression. There is still a third commit 
that is bad. I will try to bisect it today if I can.

There is also a fourth smaller regression of about 10% somewhere, but I 
didn't bother tracking that down as we can live with that.

Bill.

On Thursday, 24 March 2016 02:15:57 UTC+1, Tim Holy wrote:
>
> That's a big help. I'm not certain Jeff reads julia-users, however, so you 
> might want to file an issue. 
>
> Best, 
> --Tim 
>
> On Wednesday, March 23, 2016 04:34:25 PM 'Bill Hart' via julia-users 
> wrote: 
> > On Thursday, 24 March 2016 00:33:04 UTC+1, Bill Hart wrote: 
> > > It seems there may have been two regressions. The first regression 
> with a 
> > > slowdown factor of just over 2 seems to be: 
> > > 
> > > 639621859863609c5f3abbc2ed75c675695b3693 is the first bad commit 
> > > commit 639621859863609c5f3abbc2ed75c675695b3693 
> > > Author: Jeff Bezanson  
> > > Date:   Tue Jan 26 23:33:19 2016 -0500 
> > > 
> > > modify Base.Test not to create a closure for each test 
> > > : 
> > > :04 04 9c84c85afaed99190f3e744123dccc732f2c760e 
> > > 
> > > 486795536d95d1fb14fd9f7f415fb63cd9c6e490 M  base 
> > > 
> > > :04 04 48205a7b1b007692c81b1a8d931cb44f6cc97be8 
> > > 
> > > acb43cd0ecece4237e1834b7a3b577f312884650 M  test 
> > > 
> > > I will try to find time to find the second regression, which occurs 
> > > between 1bfabbb and 1bfabbb I believe. 
> > 
> > Sorry, that should say between 1bfabbb and 9d6e726. 
> > 
> > Bill. 
> > 
> > > On Wednesday, 23 March 2016 15:23:05 UTC+1, Bill Hart wrote: 
> > >> I'll see if it is possible. Currently our code does not work at all 
> with 
> > >> large chunks of the Julia commits in that interval. We had to work 
> around 
> > >> various things and don't know precisely when they were switched on or 
> > >> off. 
> > >> 
> > >> Bill. 
> > >> 
> > >> On Wednesday, 23 March 2016 14:54:22 UTC+1, Tim Holy wrote: 
> > >>> If you can git-bisect the change, it would be a huge help. 
> > >>> 
> > >>> Best, 
> > >>> --Tim 
> > >>> 
> > >>> On Wednesday, March 23, 2016 06:18:23 AM 'Bill Hart' via julia-users 
> > >>> 
> > >>> wrote: 
> > >>> > In very recent Julia-0.5-dev the test code in our Nemo module 
> takes 
> > >>> 
> > >>> forever 
> > >>> 
> > >>> > to start running. It's close to 2 minutes. 
> > >>> > 
> > >>> > This compares with about 15s with older Julia-0.5-dev, say 3 
> months 
> > >>> 
> > >>> ago 
> > >>> 
> > >>> > before the LLVM switchover. 
> > >>> > 
> > >>> > Does anyone know why there is this massive performance regression. 
> Is 
> > >>> 
> > >>> it 
> > >>> 
> > >>> > likely that it can be fixed? It's really killing our development 
> > >>> 
> > >>> cycle. 
> > >>> 
> > >>> > Bill. 
>
>

Re: [julia-users] Test code taking forever to start

[Iain Dunning]

Oh wow, I didn't even know that commit existed. Now BaseTestNext.jl and 
Base.Test are out of sync...
Bill, can you try using BaseTestNext.jl instead of Base.Test? If this 
change was indeed responsible, then that might reveal it.


On Wednesday, March 23, 2016 at 9:15:57 PM UTC-4, Tim Holy wrote:
>
> That's a big help. I'm not certain Jeff reads julia-users, however, so you 
> might want to file an issue. 
>
> Best, 
> --Tim 
>
> On Wednesday, March 23, 2016 04:34:25 PM 'Bill Hart' via julia-users 
> wrote: 
> > On Thursday, 24 March 2016 00:33:04 UTC+1, Bill Hart wrote: 
> > > It seems there may have been two regressions. The first regression 
> with a 
> > > slowdown factor of just over 2 seems to be: 
> > > 
> > > 639621859863609c5f3abbc2ed75c675695b3693 is the first bad commit 
> > > commit 639621859863609c5f3abbc2ed75c675695b3693 
> > > Author: Jeff Bezanson  
> > > Date:   Tue Jan 26 23:33:19 2016 -0500 
> > > 
> > > modify Base.Test not to create a closure for each test 
> > > : 
> > > :04 04 9c84c85afaed99190f3e744123dccc732f2c760e 
> > > 
> > > 486795536d95d1fb14fd9f7f415fb63cd9c6e490 M  base 
> > > 
> > > :04 04 48205a7b1b007692c81b1a8d931cb44f6cc97be8 
> > > 
> > > acb43cd0ecece4237e1834b7a3b577f312884650 M  test 
> > > 
> > > I will try to find time to find the second regression, which occurs 
> > > between 1bfabbb and 1bfabbb I believe. 
> > 
> > Sorry, that should say between 1bfabbb and 9d6e726. 
> > 
> > Bill. 
> > 
> > > On Wednesday, 23 March 2016 15:23:05 UTC+1, Bill Hart wrote: 
> > >> I'll see if it is possible. Currently our code does not work at all 
> with 
> > >> large chunks of the Julia commits in that interval. We had to work 
> around 
> > >> various things and don't know precisely when they were switched on or 
> > >> off. 
> > >> 
> > >> Bill. 
> > >> 
> > >> On Wednesday, 23 March 2016 14:54:22 UTC+1, Tim Holy wrote: 
> > >>> If you can git-bisect the change, it would be a huge help. 
> > >>> 
> > >>> Best, 
> > >>> --Tim 
> > >>> 
> > >>> On Wednesday, March 23, 2016 06:18:23 AM 'Bill Hart' via julia-users 
> > >>> 
> > >>> wrote: 
> > >>> > In very recent Julia-0.5-dev the test code in our Nemo module 
> takes 
> > >>> 
> > >>> forever 
> > >>> 
> > >>> > to start running. It's close to 2 minutes. 
> > >>> > 
> > >>> > This compares with about 15s with older Julia-0.5-dev, say 3 
> months 
> > >>> 
> > >>> ago 
> > >>> 
> > >>> > before the LLVM switchover. 
> > >>> > 
> > >>> > Does anyone know why there is this massive performance regression. 
> Is 
> > >>> 
> > >>> it 
> > >>> 
> > >>> > likely that it can be fixed? It's really killing our development 
> > >>> 
> > >>> cycle. 
> > >>> 
> > >>> > Bill. 
>
>

Re: [julia-users] Test code taking forever to start

[Tim Holy]

That's a big help. I'm not certain Jeff reads julia-users, however, so you 
might want to file an issue.

Best,
--Tim

On Wednesday, March 23, 2016 04:34:25 PM 'Bill Hart' via julia-users wrote:
> On Thursday, 24 March 2016 00:33:04 UTC+1, Bill Hart wrote:
> > It seems there may have been two regressions. The first regression with a
> > slowdown factor of just over 2 seems to be:
> > 
> > 639621859863609c5f3abbc2ed75c675695b3693 is the first bad commit
> > commit 639621859863609c5f3abbc2ed75c675695b3693
> > Author: Jeff Bezanson 
> > Date:   Tue Jan 26 23:33:19 2016 -0500
> > 
> > modify Base.Test not to create a closure for each test
> > :
> > :04 04 9c84c85afaed99190f3e744123dccc732f2c760e
> > 
> > 486795536d95d1fb14fd9f7f415fb63cd9c6e490 M  base
> > 
> > :04 04 48205a7b1b007692c81b1a8d931cb44f6cc97be8
> > 
> > acb43cd0ecece4237e1834b7a3b577f312884650 M  test
> > 
> > I will try to find time to find the second regression, which occurs
> > between 1bfabbb and 1bfabbb I believe.
> 
> Sorry, that should say between 1bfabbb and 9d6e726.
> 
> Bill.
> 
> > On Wednesday, 23 March 2016 15:23:05 UTC+1, Bill Hart wrote:
> >> I'll see if it is possible. Currently our code does not work at all with
> >> large chunks of the Julia commits in that interval. We had to work around
> >> various things and don't know precisely when they were switched on or
> >> off.
> >> 
> >> Bill.
> >> 
> >> On Wednesday, 23 March 2016 14:54:22 UTC+1, Tim Holy wrote:
> >>> If you can git-bisect the change, it would be a huge help.
> >>> 
> >>> Best,
> >>> --Tim
> >>> 
> >>> On Wednesday, March 23, 2016 06:18:23 AM 'Bill Hart' via julia-users
> >>> 
> >>> wrote:
> >>> > In very recent Julia-0.5-dev the test code in our Nemo module takes
> >>> 
> >>> forever
> >>> 
> >>> > to start running. It's close to 2 minutes.
> >>> > 
> >>> > This compares with about 15s with older Julia-0.5-dev, say 3 months
> >>> 
> >>> ago
> >>> 
> >>> > before the LLVM switchover.
> >>> > 
> >>> > Does anyone know why there is this massive performance regression. Is
> >>> 
> >>> it
> >>> 
> >>> > likely that it can be fixed? It's really killing our development
> >>> 
> >>> cycle.
> >>> 
> >>> > Bill.

Re: [julia-users] Test code taking forever to start

['Bill Hart' via julia-users]

On Thursday, 24 March 2016 00:33:04 UTC+1, Bill Hart wrote:
>
> It seems there may have been two regressions. The first regression with a 
> slowdown factor of just over 2 seems to be:
>
> 639621859863609c5f3abbc2ed75c675695b3693 is the first bad commit
> commit 639621859863609c5f3abbc2ed75c675695b3693
> Author: Jeff Bezanson 
> Date:   Tue Jan 26 23:33:19 2016 -0500
>
> modify Base.Test not to create a closure for each test
>
> :04 04 9c84c85afaed99190f3e744123dccc732f2c760e 
> 486795536d95d1fb14fd9f7f415fb63cd9c6e490 M  base
> :04 04 48205a7b1b007692c81b1a8d931cb44f6cc97be8 
> acb43cd0ecece4237e1834b7a3b577f312884650 M  test
>
> I will try to find time to find the second regression, which occurs 
> between 1bfabbb and 1bfabbb I believe.
>
>
Sorry, that should say between 1bfabbb and 9d6e726.

Bill.
>
> On Wednesday, 23 March 2016 15:23:05 UTC+1, Bill Hart wrote:
>>
>> I'll see if it is possible. Currently our code does not work at all with 
>> large chunks of the Julia commits in that interval. We had to work around 
>> various things and don't know precisely when they were switched on or off.
>>
>> Bill.
>>
>> On Wednesday, 23 March 2016 14:54:22 UTC+1, Tim Holy wrote:
>>>
>>> If you can git-bisect the change, it would be a huge help. 
>>>
>>> Best, 
>>> --Tim 
>>>
>>> On Wednesday, March 23, 2016 06:18:23 AM 'Bill Hart' via julia-users 
>>> wrote: 
>>> > In very recent Julia-0.5-dev the test code in our Nemo module takes 
>>> forever 
>>> > to start running. It's close to 2 minutes. 
>>> > 
>>> > This compares with about 15s with older Julia-0.5-dev, say 3 months 
>>> ago 
>>> > before the LLVM switchover. 
>>> > 
>>> > Does anyone know why there is this massive performance regression. Is 
>>> it 
>>> > likely that it can be fixed? It's really killing our development 
>>> cycle. 
>>> > 
>>> > Bill. 
>>>
>>>

Re: [julia-users] Test code taking forever to start

['Bill Hart' via julia-users]

It seems there may have been two regressions. The first regression with a 
slowdown factor of just over 2 seems to be:

639621859863609c5f3abbc2ed75c675695b3693 is the first bad commit
commit 639621859863609c5f3abbc2ed75c675695b3693
Author: Jeff Bezanson 
Date:   Tue Jan 26 23:33:19 2016 -0500

modify Base.Test not to create a closure for each test

:04 04 9c84c85afaed99190f3e744123dccc732f2c760e 
486795536d95d1fb14fd9f7f415fb63cd9c6e490 M  base
:04 04 48205a7b1b007692c81b1a8d931cb44f6cc97be8 
acb43cd0ecece4237e1834b7a3b577f312884650 M  test

I will try to find time to find the second regression, which occurs 
between 1bfabbb and 1bfabbb I believe.

Bill.

On Wednesday, 23 March 2016 15:23:05 UTC+1, Bill Hart wrote:
>
> I'll see if it is possible. Currently our code does not work at all with 
> large chunks of the Julia commits in that interval. We had to work around 
> various things and don't know precisely when they were switched on or off.
>
> Bill.
>
> On Wednesday, 23 March 2016 14:54:22 UTC+1, Tim Holy wrote:
>>
>> If you can git-bisect the change, it would be a huge help. 
>>
>> Best, 
>> --Tim 
>>
>> On Wednesday, March 23, 2016 06:18:23 AM 'Bill Hart' via julia-users 
>> wrote: 
>> > In very recent Julia-0.5-dev the test code in our Nemo module takes 
>> forever 
>> > to start running. It's close to 2 minutes. 
>> > 
>> > This compares with about 15s with older Julia-0.5-dev, say 3 months ago 
>> > before the LLVM switchover. 
>> > 
>> > Does anyone know why there is this massive performance regression. Is 
>> it 
>> > likely that it can be fixed? It's really killing our development cycle. 
>> > 
>> > Bill. 
>>
>>

Re: [julia-users] Test code taking forever to start

['Bill Hart' via julia-users]

I'll see if it is possible. Currently our code does not work at all with 
large chunks of the Julia commits in that interval. We had to work around 
various things and don't know precisely when they were switched on or off.

Bill.

On Wednesday, 23 March 2016 14:54:22 UTC+1, Tim Holy wrote:
>
> If you can git-bisect the change, it would be a huge help. 
>
> Best, 
> --Tim 
>
> On Wednesday, March 23, 2016 06:18:23 AM 'Bill Hart' via julia-users 
> wrote: 
> > In very recent Julia-0.5-dev the test code in our Nemo module takes 
> forever 
> > to start running. It's close to 2 minutes. 
> > 
> > This compares with about 15s with older Julia-0.5-dev, say 3 months ago 
> > before the LLVM switchover. 
> > 
> > Does anyone know why there is this massive performance regression. Is it 
> > likely that it can be fixed? It's really killing our development cycle. 
> > 
> > Bill. 
>
>

Re: [julia-users] Test code taking forever to start

[Tim Holy]

If you can git-bisect the change, it would be a huge help.

Best,
--Tim

On Wednesday, March 23, 2016 06:18:23 AM 'Bill Hart' via julia-users wrote:
> In very recent Julia-0.5-dev the test code in our Nemo module takes forever
> to start running. It's close to 2 minutes.
> 
> This compares with about 15s with older Julia-0.5-dev, say 3 months ago
> before the LLVM switchover.
> 
> Does anyone know why there is this massive performance regression. Is it
> likely that it can be fixed? It's really killing our development cycle.
> 
> Bill.

[julia-users] Test code taking forever to start

['Bill Hart' via julia-users]

In very recent Julia-0.5-dev the test code in our Nemo module takes forever 
to start running. It's close to 2 minutes.

This compares with about 15s with older Julia-0.5-dev, say 3 months ago 
before the LLVM switchover.

Does anyone know why there is this massive performance regression. Is it 
likely that it can be fixed? It's really killing our development cycle.

Bill.