Re: Optimize type streaming

2014-07-11 Thread Jan Hubicka
Are we sure it ever stabilizes? But yes, I had something like this in mind (just do one iteration always) in case we need to improve hashing. Hi, this is bit quick experiment with strengthening the hash by iteration. I don't seem to be able to measure WPA time difference for the patch though

Re: Optimize type streaming

2014-07-11 Thread Richard Biener
On Fri, 11 Jul 2014, Jan Hubicka wrote: Are we sure it ever stabilizes? But yes, I had something like this in mind (just do one iteration always) in case we need to improve hashing. Hi, this is bit quick experiment with strengthening the hash by iteration. I don't seem to be able to

Re: Optimize type streaming

2014-07-11 Thread Jan Hubicka
Well - as we re-use the streamer cache to store the hash value it isn't really possible to do better ... (at least I don't have a clever idea) OK, no cleverness on my side either. Yeah (though you wouldn't need the extra hashing - we only need to know the hash of the SCC). The current

Re: Optimize type streaming

2014-07-11 Thread Richard Biener
On Fri, 11 Jul 2014, Jan Hubicka wrote: Well - as we re-use the streamer cache to store the hash value it isn't really possible to do better ... (at least I don't have a clever idea) OK, no cleverness on my side either. Yeah (though you wouldn't need the extra hashing - we only

Re: Optimize type streaming

2014-07-11 Thread Jan Hubicka
We hash only on outgoing SCC edges. You can easily have main variant type T and variants T1,T2 that are same all used by type T again. Ok, but then the two variants shouldn't be in the same SCC or at least not in the same SCC as the main variant. They will because T refers them

Re: Optimize type streaming

2014-07-11 Thread Richard Biener
On Fri, 11 Jul 2014, Jan Hubicka wrote: We hash only on outgoing SCC edges. You can easily have main variant type T and variants T1,T2 that are same all used by type T again. Ok, but then the two variants shouldn't be in the same SCC or at least not in the same SCC as the

Re: Optimize type streaming

2014-07-11 Thread Jan Hubicka
Hmm, walking everything first and then starting streaming sounds bad idea for locality. Can't I just re-do the walk since I know what the SCC is? I will read the code more after lunch. Might be possible with a 2nd SCC stack set up, yes. I set up hash_map to store the hash values anyway.

Re: Optimize type streaming

2014-07-11 Thread Richard Biener
On Fri, 11 Jul 2014, Jan Hubicka wrote: Hmm, walking everything first and then starting streaming sounds bad idea for locality. Can't I just re-do the walk since I know what the SCC is? I will read the code more after lunch. Might be possible with a 2nd SCC stack set up, yes. I

Re: Optimize type streaming

2014-07-11 Thread Jan Hubicka
On Fri, 11 Jul 2014, Jan Hubicka wrote: Hmm, walking everything first and then starting streaming sounds bad idea for locality. Can't I just re-do the walk since I know what the SCC is? I will read the code more after lunch. Might be possible with a 2nd SCC stack set

Re: Optimize type streaming

2014-07-11 Thread Richard Biener
On Fri, 11 Jul 2014, Jan Hubicka wrote: On Fri, 11 Jul 2014, Jan Hubicka wrote: Hmm, walking everything first and then starting streaming sounds bad idea for locality. Can't I just re-do the walk since I know what the SCC is? I will read the code more after lunch.

Re: Optimize type streaming

2014-07-11 Thread Jan Hubicka
Ah, ok. Well, let's hope walk_tree walks all edges the DFS walk walks ;) A quick look tells me it doesn't walk DECL_VINDEX or DECL_FUNCTION_PERSONALITY for example. I guess it should - will try patch adding that :) Both seems like just ommisions - these fileds were added quite recently

Re: Optimize type streaming

2014-07-09 Thread Jan Hubicka
Hello, perhaps I could write bit more on my longer term plans. At the moment 30% of firefox WPA is taken by straming trees and another roughly 30% is taken by inliner. It is bit anoying but relatively easy to optimize inliner, but trees represent bigger problem. According to the stats average

Re: Optimize type streaming

2014-07-09 Thread Richard Biener
On Wed, Jul 9, 2014 at 10:58 AM, Jan Hubicka hubi...@ucw.cz wrote: Hello, perhaps I could write bit more on my longer term plans. At the moment 30% of firefox WPA is taken by straming trees and another roughly 30% is taken by inliner. It is bit anoying but relatively easy to optimize

Re: Optimize type streaming

2014-07-09 Thread Jan Hubicka
This part can probably be speed up quite a bit by doing the SCC unification before materializing the SCC, that is, doing the on-disk format compare idea. The issue here is that for bigger SCCs that have hash collisions in their entries we need to do the edge walk - but eventually having two

Re: Optimize type streaming

2014-07-09 Thread Richard Biener
On July 9, 2014 2:36:57 PM CEST, Jan Hubicka hubi...@ucw.cz wrote: This part can probably be speed up quite a bit by doing the SCC unification before materializing the SCC, that is, doing the on-disk format compare idea. The issue here is that for bigger SCCs that have hash collisions in their

Re: Optimize type streaming

2014-07-09 Thread Jan Hubicka
Are we sure it ever stabilizes? But yes, I had something like this in mind (just do one iteration always) in case we need to improve hashing. Not necesarily - imagine two identical type variants, no matter how hard we hashrehash, they will be same as they are identical. We can just keep

Re: Optimize type streaming

2014-07-08 Thread Jan Hubicka
We have reverted the patch for now but I note that at least the piece below is a step backward from doing the compare in the on-disk format. Why it is step backward from compare in the on-dist format? All the information is still streamed, just not duplicated. Since we explicitly stream

Re: Optimize type streaming

2014-07-07 Thread Richard Biener
On Sun, 29 Jun 2014, Jan Hubicka wrote: In addition of pr61644 and pr61646, this commit breaks a lot of fortran tests with -flto -O0. Hello, the problem here is that we have POINTER_TYPE that points to array of variable length (not sure why it happens only with -O0). The ICE is introduced

Re: Optimize type streaming

2014-07-07 Thread Richard Biener
On Mon, 30 Jun 2014, Jan Hubicka wrote: Please revert the original patch instead which was not tested properly. I'll get back to this after I return from vacation. OK, I will bootstrap and revert tomorrow morning. Note that testusite passes with the patch; the problem appears only

Re: Optimize type streaming

2014-07-07 Thread Richard Biener
On Mon, 30 Jun 2014, Dominique Dhumieres wrote: Note that testusite passes with the patch; ... Confirmed. There is a typo s/fileds/fields/ I have seen the failures because I now run the gfortran testsuite with the addition of '-g -flto'. Is it worth pushing in this direction? Yes. It

Re: Optimize type streaming

2014-06-30 Thread Richard Biener
On June 29, 2014 9:53:03 PM CEST, Jan Hubicka hubi...@ucw.cz wrote: In addition of pr61644 and pr61646, this commit breaks a lot of fortran tests with -flto -O0. Hello, the problem here is that we have POINTER_TYPE that points to array of variable length (not sure why it happens only with -O0).

Re: Optimize type streaming

2014-06-30 Thread Jan Hubicka
Please revert the original patch instead which was not tested properly. I'll get back to this after I return from vacation. OK, I will bootstrap and revert tomorrow morning. Note that testusite passes with the patch; the problem appears only at -O0 -flto and fortran that we apparently do

Re: Optimize type streaming

2014-06-30 Thread Dominique Dhumieres
Note that testusite passes with the patch; ... Confirmed. There is a typo s/fileds/fields/ I have seen the failures because I now run the gfortran testsuite with the addition of '-g -flto'. Is it worth pushing in this direction? Cheers, Dominique

Re: Optimize type streaming

2014-06-30 Thread Dominique Dhumieres
Note that testusite passes with the patch; ... Confirmed. There is a typo s/fileds/fields/ I have seen the failures because I now run the gfortran testsuite with

Re: Optimize type streaming

2014-06-30 Thread Jan Hubicka
On June 29, 2014 9:53:03 PM CEST, Jan Hubicka hubi...@ucw.cz wrote: In addition of pr61644 and pr61646, this commit breaks a lot of fortran tests with -flto -O0. Hello, the problem here is that we have POINTER_TYPE that points to array of variable length (not sure why it happens only with

Re: Optimize type streaming

2014-06-29 Thread Dominique Dhumieres
In addition of pr61644 and pr61646, this commit breaks a lot of fortran tests with -flto -O0. TIA Dominique FAIL: gfortran.dg/actual_array_constructor_1.f90 -g -flto (internal compiler error) FAIL: gfortran.dg/alloc_comp_assign_6.f90 -g -flto (internal compiler error) FAIL:

Re: Optimize type streaming

2014-06-29 Thread Jan Hubicka
In addition of pr61644 and pr61646, this commit breaks a lot of fortran tests with -flto -O0. Hello, the problem here is that we have POINTER_TYPE that points to array of variable length (not sure why it happens only with -O0). The ICE is introduced by the fact that we stream the type inside of

Re: Optimize type streaming

2014-06-28 Thread Jan Hubicka
Minor comments below, ok with those changes. Thanks! This is version of patch I commited after additional testing on building of some bigger apps with LTO. The main change (in addition to those requested) is that I moved the fixups into lto_copy_fields_not_streamed and added loop before SCC

Optimize type streaming

2014-06-27 Thread Jan Hubicka
Hi, the most common types of tree nodes streamed are declarations (5.4M for Firefox) and types (4.2M for Firefox). This patch makes representation of types smaller by avoiding saving redundent info about type and its variants. About 50% of types are variants and overall this saves about 6% of WPA

Re: Optimize type streaming

2014-06-27 Thread Richard Biener
On Fri, 27 Jun 2014, Jan Hubicka wrote: Hi, the most common types of tree nodes streamed are declarations (5.4M for Firefox) and types (4.2M for Firefox). This patch makes representation of types smaller by avoiding saving redundent info about type and its variants. About 50% of types are