Re: [Yade-dev] High precision Real and minieigen

[Janek Kozicki (yade)]

Yes, I think that we can stay with boost::python.
But we need to put minieigen sources back into yade.

best regards
Janek


Anton Gladky said: (by the date of Sun, 23 Feb 2020 22:08:47 +0100)

> Hi Janek,
> 
> thanks a lot for this quick tests and results! Really impressive!
> 
> As far as I see, there is not benefit for us to replace boost::python,
> at least right now?
> 
> Regards
> 
> Anton
> 
> Am So., 23. Feb. 2020 um 16:02 Uhr schrieb Janek Kozicki (yade)
> :
> >
> > > Another note: I just realized that the compilation benchmarks might
> > > have not been fair, because I think that I added few more
> > > registrations (like Vector2c, or sth. like that) while doing pybind
> > > migration. I will compare again without these extra registrations.
> >
> > OK, I have compared with the exact same registrations. And it seems
> > that pybind has many advantages, but not the one that it is compiling
> > faster. See:
> >
> > pybind without debug info; call `ccache --clear` between every invocation.
> > -O3 -j 1   -O3 -j 10
> > wall clock time: 2:41.14   wall clock time: 0:40.67
> > wall clock time: 2:46.56   wall clock time: 0:41.28
> > wall clock time: 2:47.42   wall clock time: 0:42.54
> > wall clock time: 2:43.79   wall clock time: 0:44.70
> >
> > -O1 -j 1   -O1 -j 10
> > wall clock time: 2:09.88   wall clock time: 0:35.00
> > wall clock time: 2:10.78   wall clock time: 0:33.46
> > wall clock time: 2:08.19   wall clock time: 0:32.62
> > wall clock time: 2:10.10   wall clock time: 0:32.73
> >
> > 
> > boost::python without debug info; call `ccache --clear` between every 
> > invocation.
> > -O3 -j 1   -O3 -j 10
> > wall clock time: 2:16.83   wall clock time: 0:33.88
> > wall clock time: 2:15.96   wall clock time: 0:32.84
> > wall clock time: 2:16.00   wall clock time: 0:34.85
> > wall clock time: 2:16.52   wall clock time: 0:32.32
> >
> > -O1 -j 1   -O1 -j 10
> > wall clock time: 2:01.48   wall clock time: 0:28.56
> > wall clock time: 2:00.70   wall clock time: 0:29.15
> > wall clock time: 2:01.29   wall clock time: 0:28.05
> > wall clock time: 2:00.56   wall clock time: 0:28.35
> >
> > The main pybind disadvantage is unstable API. I don't like writing
> > extra code to support older linux distributions.
> >
> >
> > Also I think that if while copying minieigen into yade we did some
> > extra rebalancing between the .cpp files, so that in each file there's
> > about the same amount of registrations performed, then we could
> > reduce (parallel) compilation time to maybe 25 or 20 seconds.
> >
> >
> > If you wanted to compile and try yourself, the comparison was between
> > branches master and tryPybind. To see how to compile see .gitlab-ci.yml,
> > I'm sorry that it's a bit messy. I was never good at using build
> > systems ;) I just wanted parallel build quickly. That's why I wrote
> > such strange makefile.
> >
> > best regards
> > Janek
> >
> >
> >
> > > Another comparison note:
> > >
> > > 8. boost::python is much more picky about the order of registered
> > >functions, and sometimes does not work if the order is "wrong".
> > >pybind always resolves the function overloads correctly.
> >
> > --
> > --
> > Janek Kozicki, PhD. DSc. Arch. Assoc. Prof.
> > Gdańsk University of Technology
> > Faculty of Applied Physics and Mathematics
> > Department of Theoretical Physics and Quantum Information
> > --
> > http://yade-dem.org/
> > http://pg.edu.pl/jkozicki (click English flag on top right)


-- 
--
Janek Kozicki, PhD. DSc. Arch. Assoc. Prof.
Gdańsk University of Technology
Faculty of Applied Physics and Mathematics
Department of Theoretical Physics and Quantum Information
--
http://yade-dem.org/
http://pg.edu.pl/jkozicki (click English flag on top right)

___
Mailing list: https://launchpad.net/~yade-dev
Post to : yade-dev@lists.launchpad.net
Unsubscribe : https://launchpad.net/~yade-dev
More help   : https://help.launchpad.net/ListHelp

Re: [Yade-dev] High precision Real and minieigen

[Bruno Chareyre]

Hi Janek, thanks for pybind feedback. It is intriguing. It was definitely
after me mentionning compile times with boost for c++ / python, at a
workshop, that someone reached me and advised about pybind.
Rumor?

Bruno



Le dim. 23 févr. 2020 16:03, Janek Kozicki (yade) 
a écrit :

> > Another note: I just realized that the compilation benchmarks might
> > have not been fair, because I think that I added few more
> > registrations (like Vector2c, or sth. like that) while doing pybind
> > migration. I will compare again without these extra registrations.
>
> OK, I have compared with the exact same registrations. And it seems
> that pybind has many advantages, but not the one that it is compiling
> faster. See:
>
> pybind without debug info; call `ccache --clear` between every invocation.
> -O3 -j 1   -O3 -j 10
> wall clock time: 2:41.14   wall clock time: 0:40.67
> wall clock time: 2:46.56   wall clock time: 0:41.28
> wall clock time: 2:47.42   wall clock time: 0:42.54
> wall clock time: 2:43.79   wall clock time: 0:44.70
>
> -O1 -j 1   -O1 -j 10
> wall clock time: 2:09.88   wall clock time: 0:35.00
> wall clock time: 2:10.78   wall clock time: 0:33.46
> wall clock time: 2:08.19   wall clock time: 0:32.62
> wall clock time: 2:10.10   wall clock time: 0:32.73
>
>
> 
> boost::python without debug info; call `ccache --clear` between every
> invocation.
> -O3 -j 1   -O3 -j 10
> wall clock time: 2:16.83   wall clock time: 0:33.88
> wall clock time: 2:15.96   wall clock time: 0:32.84
> wall clock time: 2:16.00   wall clock time: 0:34.85
> wall clock time: 2:16.52   wall clock time: 0:32.32
>
> -O1 -j 1   -O1 -j 10
> wall clock time: 2:01.48   wall clock time: 0:28.56
> wall clock time: 2:00.70   wall clock time: 0:29.15
> wall clock time: 2:01.29   wall clock time: 0:28.05
> wall clock time: 2:00.56   wall clock time: 0:28.35
>
> The main pybind disadvantage is unstable API. I don't like writing
> extra code to support older linux distributions.
>
>
> Also I think that if while copying minieigen into yade we did some
> extra rebalancing between the .cpp files, so that in each file there's
> about the same amount of registrations performed, then we could
> reduce (parallel) compilation time to maybe 25 or 20 seconds.
>
>
> If you wanted to compile and try yourself, the comparison was between
> branches master and tryPybind. To see how to compile see .gitlab-ci.yml,
> I'm sorry that it's a bit messy. I was never good at using build
> systems ;) I just wanted parallel build quickly. That's why I wrote
> such strange makefile.
>
> best regards
> Janek
>
>
>
> > Another comparison note:
> >
> > 8. boost::python is much more picky about the order of registered
> >functions, and sometimes does not work if the order is "wrong".
> >pybind always resolves the function overloads correctly.
>
> --
> --
> Janek Kozicki, PhD. DSc. Arch. Assoc. Prof.
> Gdańsk University of Technology
> Faculty of Applied Physics and Mathematics
> Department of Theoretical Physics and Quantum Information
> --
> http://yade-dem.org/
> http://pg.edu.pl/jkozicki (click English flag on top right)
>
> ___
> Mailing list: https://launchpad.net/~yade-dev
> Post to : yade-dev@lists.launchpad.net
> Unsubscribe : https://launchpad.net/~yade-dev
> More help   : https://help.launchpad.net/ListHelp
>
>
>
___
Mailing list: https://launchpad.net/~yade-dev
Post to : yade-dev@lists.launchpad.net
Unsubscribe : https://launchpad.net/~yade-dev
More help   : https://help.launchpad.net/ListHelp

Re: [Yade-dev] High precision Real and minieigen

[Janek Kozicki (yade)]

> Another note: I just realized that the compilation benchmarks might
> have not been fair, because I think that I added few more
> registrations (like Vector2c, or sth. like that) while doing pybind
> migration. I will compare again without these extra registrations.

OK, I have compared with the exact same registrations. And it seems
that pybind has many advantages, but not the one that it is compiling
faster. See:

pybind without debug info; call `ccache --clear` between every invocation.
-O3 -j 1   -O3 -j 10
wall clock time: 2:41.14   wall clock time: 0:40.67
wall clock time: 2:46.56   wall clock time: 0:41.28
wall clock time: 2:47.42   wall clock time: 0:42.54
wall clock time: 2:43.79   wall clock time: 0:44.70
   
-O1 -j 1   -O1 -j 10
wall clock time: 2:09.88   wall clock time: 0:35.00
wall clock time: 2:10.78   wall clock time: 0:33.46
wall clock time: 2:08.19   wall clock time: 0:32.62
wall clock time: 2:10.10   wall clock time: 0:32.73


boost::python without debug info; call `ccache --clear` between every 
invocation.
-O3 -j 1   -O3 -j 10
wall clock time: 2:16.83   wall clock time: 0:33.88
wall clock time: 2:15.96   wall clock time: 0:32.84
wall clock time: 2:16.00   wall clock time: 0:34.85
wall clock time: 2:16.52   wall clock time: 0:32.32
   
-O1 -j 1   -O1 -j 10
wall clock time: 2:01.48   wall clock time: 0:28.56
wall clock time: 2:00.70   wall clock time: 0:29.15
wall clock time: 2:01.29   wall clock time: 0:28.05
wall clock time: 2:00.56   wall clock time: 0:28.35

The main pybind disadvantage is unstable API. I don't like writing
extra code to support older linux distributions.


Also I think that if while copying minieigen into yade we did some
extra rebalancing between the .cpp files, so that in each file there's
about the same amount of registrations performed, then we could
reduce (parallel) compilation time to maybe 25 or 20 seconds.


If you wanted to compile and try yourself, the comparison was between
branches master and tryPybind. To see how to compile see .gitlab-ci.yml,
I'm sorry that it's a bit messy. I was never good at using build
systems ;) I just wanted parallel build quickly. That's why I wrote
such strange makefile.

best regards
Janek


 
> Another comparison note:
> 
> 8. boost::python is much more picky about the order of registered
>functions, and sometimes does not work if the order is "wrong".
>pybind always resolves the function overloads correctly.

-- 
--
Janek Kozicki, PhD. DSc. Arch. Assoc. Prof.
Gdańsk University of Technology
Faculty of Applied Physics and Mathematics
Department of Theoretical Physics and Quantum Information
--
http://yade-dem.org/
http://pg.edu.pl/jkozicki (click English flag on top right)

___
Mailing list: https://launchpad.net/~yade-dev
Post to : yade-dev@lists.launchpad.net
Unsubscribe : https://launchpad.net/~yade-dev
More help   : https://help.launchpad.net/ListHelp

Re: [Yade-dev] High precision Real and minieigen

[Janek Kozicki (yade)]

Hi Anton

> sorry for delays, "real" life is taking much more time when we get older :)

please don't worry. I really appreciate all your hard work. I got a
little carried away, I'm sorry.

> > * I would rather avoid pybind11 switch in a rush, better give it a few 
> > months
> Maybe I was not clear, pybind11 is just a possible option. If we
> really decide to switch to it, we need to evaluate, whether it is
> suitable for us, whether it really helps to improve compilation times
> and what it costs for us (time).

exactly. The quick migration which I have done yesterday passes the
tests, and seems to work at first glance, but there were some tricky
sections of the code. Which I converted but I didn't test. I feel more
comfortable with old and well tested boost::python version.
Also I see that Vaclav has done this already [1]
 
> > * I would prefer minieigen-src package (no extra compilation time for 
> > people who use `double`)
> > * minieigen-src in ubuntu 20.04 will let more people to test & use high 
> > precision.
> Unfortunately, it is almost impossible for the time being […]

this is clear.

> We should probably consider alternative ways to help you to integrate
> high-precision stuff.

Thank you! :))


Another note: I just realized that the compilation benchmarks might
have not been fair, because I think that I added few more
registrations (like Vector2c, or sth. like that) while doing pybind
migration. I will compare again without these extra registrations.


Another comparison note:

8. boost::python is much more picky about the order of registered
   functions, and sometimes does not work if the order is "wrong".
   pybind always resolves the function overloads correctly.


best regards
Janek

[1] https://github.com/woodem/woo/blob/master/lib/eigen/pybind11/visitors.hpp



___
Mailing list: https://launchpad.net/~yade-dev
Post to : yade-dev@lists.launchpad.net
Unsubscribe : https://launchpad.net/~yade-dev
More help   : https://help.launchpad.net/ListHelp

Re: [Yade-dev] High precision Real and minieigen

[Janek Kozicki (yade)]

Hi All :)

I did an experimental switch to pybind, see this pipeline:

  https://gitlab.com/cosurgi/minieigen-real/pipelines/120246349

My observations about pybind11+minieigen:

1. pybind11 seems better organized and easier to use than boost::python

2. pybind11 has unstable API, what compiles on bullseye, does not
   compile on debian stretch:
   https://gitlab.com/cosurgi/minieigen-real/-/jobs/446852240
   (I only created debian-bullseye-pybind & debian-stretch-pybind
   images, failures on other linux distributions are because there is
   no pybind docker image)

3. compilation time without ccache (on my computer)
   base invocation:
   /usr/bin/time -f "\n\n\twall clock time: %E\n\n" make test_ci_mne_64 -j 1 
CXXFLAGS="-D EIGEN_DONT_ALIGN -D EIGEN_DONT_VECTORIZE -D 
EIGEN_DISABLE_UNALIGNED_ARRAY_ASSERT"

   -O1
 boost::python make -j 10:  31 seconds
 pybind -j 10:  30 seconds
 boost::python make -j 1 : 120 seconds
 pybind -j 1 :  90 seconds

   -O3
 boost::python make -j 10:  32 seconds
 pybind -j 10:  40 seconds
 boost::python make -j 1 : 140 seconds
 pybind -j 1 : 150 seconds

   note: that's compilation time of minieigen only. Not yade ;) It is
   strange to observe that boost::python is easier to be optimized by
   compiler. Please try yourself. I only did two measurements without
   calculating standard deviation ;)

4. pybind does not segfault with Eigen vectorization enabled on older compiler.
   boost::python segfaults with Eigen vectorization on older compilers
   (We only very recently have added make_SSE to the pipeline, and it is
   with the newer compiler version). For someone who really wants SSE
   that's important. But in my opinion struggling for SSE serves no
   purpose without AlignedVector3, which isn't supported (yet).

5. both of them assume that minieigen sources are inside yade
   sources. The only difference is that the content of visitor.hpp is
   a little different e.g. replace py::extract with py::cast,
   and so on.

6. I don't know how many changes have to be done in entire yade code
   to switch to pybind.

7. pybind is more strict about mapping types C++ ↔ python, for pybind
   float is really float.
   For boost::python all three: float, double, long double were
   (partially without consistency) treated as double.
   for pybind a duplicate registration of an already registered type
   is a fatal error.

I understand that minieigen-src has no chance of getting into 20.04.

I conclude that we should put minieigen inside yade. If that will be
a boost::python version or a pybind version is yet to be decided.

best regards
Janek


Anton Gladky said: (by the date of Sat, 22 Feb 2020 20:29:01 +0100)

> Hello Janek,
> 
> sorry for delays, "real" life is taking much more time when we get older :)
> 
> > * I would rather avoid pybind11 switch in a rush, better give it a few 
> > months
> 
> Maybe I was not clear, pybind11 is just a possible option. If we
> really decide to switch to it, we need to evaluate, whether it is
> suitable for us, whether it really helps to improve compilation times
> and what it costs for us (time).
> 
> > * I would prefer minieigen-src package (no extra compilation time for 
> > people who use `double`)
> > * minieigen-src in ubuntu 20.04 will let more people to test & use high 
> > precision.
> 
> Unfortunately, it is almost impossible for the time being. If I add a
> new "minieigen-src" binary package, the package should go through
> "NEW"-queue [1], which has almost 300 packages, waiting for review
> from Debian FTP-masters. It can delay upload for months.
> 
> We should probably consider alternative ways to help you to integrate
> high-precision stuff.
> 
> [1] https://ftp-master.debian.org/new.html
> 
> Best regards
> 
> Anton
> 
> Am Fr., 21. Feb. 2020 um 22:30 Uhr schrieb Janek Kozicki (yade)
> 
> :
> >
> > Hi Anton,
> >
> > Yes, the number of defines in Serialization.hpp is crazy.
> > Changing/removing them to use pybind11 would take a lot of work.
> > I'm sure it is possible, but not quickly.
> >
> > We can open an issue about pybind11 switch. Discuss pros and cons,
> > find a good strategy to deal with crazy defines and complete it later.
> >
> > Doing this switch right now is unrealistic. Working on another moving
> > target (pybind11) on top of an already moving target (minieigen HP
> > not fully integrated) is irresponsible. If possible I would prefer:
> >
> >   * minieigen-src package (slower compilation only for non-double)
> >   * or integrate minieigen with yade sources for now (rather undesirable
> > due to compilation time).
> >
> > Whatever we do I would prefer to integrate HP, because doubles will
> > appear very quickly in the code.
> >
> > Ah, if we talked about pybind11 switch three months ago it would be
> > perfect timing for me :( In fact I was thinking about it that time,
> > but I didn't want to 

Re: [Yade-dev] High precision Real and minieigen

[Anton Gladky]

Hello Janek,

sorry for delays, "real" life is taking much more time when we get older :)

> * I would rather avoid pybind11 switch in a rush, better give it a few months

Maybe I was not clear, pybind11 is just a possible option. If we
really decide to switch to it, we need to evaluate, whether it is
suitable for us, whether it really helps to improve compilation times
and what it costs for us (time).

> * I would prefer minieigen-src package (no extra compilation time for people 
> who use `double`)
> * minieigen-src in ubuntu 20.04 will let more people to test & use high 
> precision.

Unfortunately, it is almost impossible for the time being. If I add a
new "minieigen-src" binary package, the package should go through
"NEW"-queue [1], which has almost 300 packages, waiting for review
from Debian FTP-masters. It can delay upload for months.

We should probably consider alternative ways to help you to integrate
high-precision stuff.

[1] https://ftp-master.debian.org/new.html

Best regards

Anton

Am Fr., 21. Feb. 2020 um 22:30 Uhr schrieb Janek Kozicki (yade)

:
>
> Hi Anton,
>
> Yes, the number of defines in Serialization.hpp is crazy.
> Changing/removing them to use pybind11 would take a lot of work.
> I'm sure it is possible, but not quickly.
>
> We can open an issue about pybind11 switch. Discuss pros and cons,
> find a good strategy to deal with crazy defines and complete it later.
>
> Doing this switch right now is unrealistic. Working on another moving
> target (pybind11) on top of an already moving target (minieigen HP
> not fully integrated) is irresponsible. If possible I would prefer:
>
>   * minieigen-src package (slower compilation only for non-double)
>   * or integrate minieigen with yade sources for now (rather undesirable
> due to compilation time).
>
> Whatever we do I would prefer to integrate HP, because doubles will
> appear very quickly in the code.
>
> Ah, if we talked about pybind11 switch three months ago it would be
> perfect timing for me :( In fact I was thinking about it that time,
> but I didn't want to push into something that wasn't guaranteed to be
> working [*]. Figuring out the correct way to convert high precision
> to/from python with boost was difficult, for a while I prefer to not
> repeat this task with pybind11. But surely we can get back to this
> later.
>
> The extra time for compilation happens only when non-double type is used.
> If minieigen-src package gets to 20.04 it will be possible later for
> people to compile high precision yade from gitlab sources without big hassle.
>
> The build_focal_simple [1] is 5 minutes because it's not parallelized.
> The problem is that pybuild is not accepting -j argument.
> With make -j 6 it is faster [2][3].
>
> Also a fast quad_double (62 decimal places, package libqd-dev)
> integration is in the works with boost::multiprecision developers [4].
>
> To summarize:
>
>  * I would rather avoid pybind11 switch in a rush, better give it a few months
>  * I would prefer minieigen-src package (no extra compilation time for people 
> who use `double`)
>  * minieigen-src in ubuntu 20.04 will let more people to test & use high 
> precision.
>
> I understand that this solution is temporary until we figure a
> proper way to deal with pybind11. The 4 year LTS is more than enough
> to solve pybind11 + Serialization.hpp problem.
>
> It is better to finish HP integration to not lose what is already
> working.
>
> best regards
> Janek
>
>
> [*] pybind11 is a new dependency and a new package, boost is well
> integrated with itself, it was guaranteed to work.
>
> [1] https://gitlab.com/yade-dev/minieigen/-/jobs/444927466
>
> [2] https://gitlab.com/cosurgi/minieigen-real/pipelines/118211208 - I used 
> this to test HP
>
> [3] In the pipeline with ccache it's usually not a problem. On
> debian build servers: it won't be a problem until we decide to
> make yade-float128 package. But if we made an accompanying
> minieigen-float128 this problem would disappear. But pybind11
> alternative is good also, just unrealistic right now.
>
> [4] https://github.com/boostorg/multiprecision/issues/184
>
>
> Anton Gladky said: (by the date of Fri, 21 Feb 2020 20:09:07 +0100)
>
> > Hi Janek,
> >
> > we have misunderstanding here. python3-minieigen is the __binary__
> > package and it is a bad idea to ship the source code with the package.
> >
> > Adding minieigen-src binary package is possible, but it looks like very
> > undesired way.
> >
> > As I see, only Yade is using minieigen in the Debian. So, theoretically we
> > can merge it again with Yade. Not sure, whether it is a good way though
> > Yade compiles too slow. Minieigen adds definitely 4-5 compilation minutes
> > and Gigabytes of used RAM.
> >
> > We should really have a look at pybind11 alternative, if it accelerates the
> > compilation. But when I see Serialization.hpp, I am getting crazy from the
> > number of "defines" there.
> >
> > Anton
> >
> > Am Fr., 21. Feb. 

Re: [Yade-dev] High precision Real and minieigen

[Anton Gladky]

Guys, just to be clear: pybind11 migration was just a possible option
for the future.
I have no experience and no idea, whether it is better option or not.
We can experiment
with minieigen (switching it to pybind11) and see, whether it is
suitable for our needs.

Anton

Am Sa., 22. Feb. 2020 um 13:13 Uhr schrieb Bruno Chareyre
:
>
> It makes sense to me to not rush to pybind right now, although I don't mean 
> to refrain any volunteer...
> B
>
>
>
> On Sat, 22 Feb 2020 at 11:05, Janek Kozicki (yade)  
> wrote:
>>
>> OK, I will experiment a bit and see what it takes to switch to pybind11.
>>
>> Janek
>>
>>
>> Janek Kozicki (yade) said: (by the date of Fri, 21 Feb 2020 22:30:04 
>> +0100)
>>
>> > Hi Anton,
>> >
>> > Yes, the number of defines in Serialization.hpp is crazy.
>> > Changing/removing them to use pybind11 would take a lot of work.
>> > I'm sure it is possible, but not quickly.
>> >
>> > We can open an issue about pybind11 switch. Discuss pros and cons,
>> > find a good strategy to deal with crazy defines and complete it later.
>> >
>> > Doing this switch right now is unrealistic. Working on another moving
>> > target (pybind11) on top of an already moving target (minieigen HP
>> > not fully integrated) is irresponsible. If possible I would prefer:
>> >
>> >   * minieigen-src package (slower compilation only for non-double)
>> >   * or integrate minieigen with yade sources for now (rather undesirable
>> > due to compilation time).
>> >
>> > Whatever we do I would prefer to integrate HP, because doubles will
>> > appear very quickly in the code.
>> >
>> > Ah, if we talked about pybind11 switch three months ago it would be
>> > perfect timing for me :( In fact I was thinking about it that time,
>> > but I didn't want to push into something that wasn't guaranteed to be
>> > working [*]. Figuring out the correct way to convert high precision
>> > to/from python with boost was difficult, for a while I prefer to not
>> > repeat this task with pybind11. But surely we can get back to this
>> > later.
>> >
>> > The extra time for compilation happens only when non-double type is used.
>> > If minieigen-src package gets to 20.04 it will be possible later for
>> > people to compile high precision yade from gitlab sources without big 
>> > hassle.
>> >
>> > The build_focal_simple [1] is 5 minutes because it's not parallelized.
>> > The problem is that pybuild is not accepting -j argument.
>> > With make -j 6 it is faster [2][3].
>> >
>> > Also a fast quad_double (62 decimal places, package libqd-dev)
>> > integration is in the works with boost::multiprecision developers [4].
>> >
>> > To summarize:
>> >
>> >  * I would rather avoid pybind11 switch in a rush, better give it a few 
>> > months
>> >  * I would prefer minieigen-src package (no extra compilation time for 
>> > people who use `double`)
>> >  * minieigen-src in ubuntu 20.04 will let more people to test & use high 
>> > precision.
>> >
>> > I understand that this solution is temporary until we figure a
>> > proper way to deal with pybind11. The 4 year LTS is more than enough
>> > to solve pybind11 + Serialization.hpp problem.
>> >
>> > It is better to finish HP integration to not lose what is already
>> > working.
>> >
>> > best regards
>> > Janek
>> >
>> >
>> > [*] pybind11 is a new dependency and a new package, boost is well
>> > integrated with itself, it was guaranteed to work.
>> >
>> > [1] https://gitlab.com/yade-dev/minieigen/-/jobs/444927466
>> >
>> > [2] https://gitlab.com/cosurgi/minieigen-real/pipelines/118211208 - I used 
>> > this to test HP
>> >
>> > [3] In the pipeline with ccache it's usually not a problem. On
>> > debian build servers: it won't be a problem until we decide to
>> > make yade-float128 package. But if we made an accompanying
>> > minieigen-float128 this problem would disappear. But pybind11
>> > alternative is good also, just unrealistic right now.
>> >
>> > [4] https://github.com/boostorg/multiprecision/issues/184
>> >
>> >
>> > Anton Gladky said: (by the date of Fri, 21 Feb 2020 20:09:07 +0100)
>> >
>> > > Hi Janek,
>> > >
>> > > we have misunderstanding here. python3-minieigen is the __binary__
>> > > package and it is a bad idea to ship the source code with the package.
>> > >
>> > > Adding minieigen-src binary package is possible, but it looks like very
>> > > undesired way.
>> > >
>> > > As I see, only Yade is using minieigen in the Debian. So, theoretically 
>> > > we
>> > > can merge it again with Yade. Not sure, whether it is a good way 
>> > > though
>> > > Yade compiles too slow. Minieigen adds definitely 4-5 compilation minutes
>> > > and Gigabytes of used RAM.
>> > >
>> > > We should really have a look at pybind11 alternative, if it accelerates 
>> > > the
>> > > compilation. But when I see Serialization.hpp, I am getting crazy from 
>> > > the
>> > > number of "defines" there.
>> > >
>> > > Anton
>> > >
>> > > Am Fr., 21. Feb. 2020 um 17:04 Uhr schrieb Janek Kozicki 

Re: [Yade-dev] High precision Real and minieigen

[Bruno Chareyre]

It makes sense to me to not rush to pybind right now, although I don't mean
to refrain any volunteer...
B



On Sat, 22 Feb 2020 at 11:05, Janek Kozicki (yade) 
wrote:

> OK, I will experiment a bit and see what it takes to switch to pybind11.
>
> Janek
>
>
> Janek Kozicki (yade) said: (by the date of Fri, 21 Feb 2020 22:30:04
> +0100)
>
> > Hi Anton,
> >
> > Yes, the number of defines in Serialization.hpp is crazy.
> > Changing/removing them to use pybind11 would take a lot of work.
> > I'm sure it is possible, but not quickly.
> >
> > We can open an issue about pybind11 switch. Discuss pros and cons,
> > find a good strategy to deal with crazy defines and complete it later.
> >
> > Doing this switch right now is unrealistic. Working on another moving
> > target (pybind11) on top of an already moving target (minieigen HP
> > not fully integrated) is irresponsible. If possible I would prefer:
> >
> >   * minieigen-src package (slower compilation only for non-double)
> >   * or integrate minieigen with yade sources for now (rather undesirable
> > due to compilation time).
> >
> > Whatever we do I would prefer to integrate HP, because doubles will
> > appear very quickly in the code.
> >
> > Ah, if we talked about pybind11 switch three months ago it would be
> > perfect timing for me :( In fact I was thinking about it that time,
> > but I didn't want to push into something that wasn't guaranteed to be
> > working [*]. Figuring out the correct way to convert high precision
> > to/from python with boost was difficult, for a while I prefer to not
> > repeat this task with pybind11. But surely we can get back to this
> > later.
> >
> > The extra time for compilation happens only when non-double type is used.
> > If minieigen-src package gets to 20.04 it will be possible later for
> > people to compile high precision yade from gitlab sources without big
> hassle.
> >
> > The build_focal_simple [1] is 5 minutes because it's not parallelized.
> > The problem is that pybuild is not accepting -j argument.
> > With make -j 6 it is faster [2][3].
> >
> > Also a fast quad_double (62 decimal places, package libqd-dev)
> > integration is in the works with boost::multiprecision developers [4].
> >
> > To summarize:
> >
> >  * I would rather avoid pybind11 switch in a rush, better give it a few
> months
> >  * I would prefer minieigen-src package (no extra compilation time for
> people who use `double`)
> >  * minieigen-src in ubuntu 20.04 will let more people to test & use high
> precision.
> >
> > I understand that this solution is temporary until we figure a
> > proper way to deal with pybind11. The 4 year LTS is more than enough
> > to solve pybind11 + Serialization.hpp problem.
> >
> > It is better to finish HP integration to not lose what is already
> > working.
> >
> > best regards
> > Janek
> >
> >
> > [*] pybind11 is a new dependency and a new package, boost is well
> > integrated with itself, it was guaranteed to work.
> >
> > [1] https://gitlab.com/yade-dev/minieigen/-/jobs/444927466
> >
> > [2] https://gitlab.com/cosurgi/minieigen-real/pipelines/118211208 - I
> used this to test HP
> >
> > [3] In the pipeline with ccache it's usually not a problem. On
> > debian build servers: it won't be a problem until we decide to
> > make yade-float128 package. But if we made an accompanying
> > minieigen-float128 this problem would disappear. But pybind11
> > alternative is good also, just unrealistic right now.
> >
> > [4] https://github.com/boostorg/multiprecision/issues/184
> >
> >
> > Anton Gladky said: (by the date of Fri, 21 Feb 2020 20:09:07 +0100)
> >
> > > Hi Janek,
> > >
> > > we have misunderstanding here. python3-minieigen is the __binary__
> > > package and it is a bad idea to ship the source code with the package.
> > >
> > > Adding minieigen-src binary package is possible, but it looks like very
> > > undesired way.
> > >
> > > As I see, only Yade is using minieigen in the Debian. So,
> theoretically we
> > > can merge it again with Yade. Not sure, whether it is a good way
> though
> > > Yade compiles too slow. Minieigen adds definitely 4-5 compilation
> minutes
> > > and Gigabytes of used RAM.
> > >
> > > We should really have a look at pybind11 alternative, if it
> accelerates the
> > > compilation. But when I see Serialization.hpp, I am getting crazy from
> the
> > > number of "defines" there.
> > >
> > > Anton
> > >
> > > Am Fr., 21. Feb. 2020 um 17:04 Uhr schrieb Janek Kozicki (yade)
> > > :
> > > >
> > > > Uh Anton, I'm sorry to be so boring:
> > > >
> > > > I have checked with sid in /etc/apt/sources.list:
> > > >
> > > >   deb-src http://ftp.pl.debian.org/debian/ sid  main non-free
> contrib
> > > >
> > > > and built the python3-minieigen_0.50.3+dfsg1-12_amd64.deb package.
> > > >
> > > > There is no /usr/include/minieigen/*pp files inside :(
> > > >
> > > > For high precision to work, they are necessary. Maybe the proper way
> > > > to do this is to introduce 

Re: [Yade-dev] High precision Real and minieigen

[Janek Kozicki (yade)]

OK, I will experiment a bit and see what it takes to switch to pybind11.

Janek


Janek Kozicki (yade) said: (by the date of Fri, 21 Feb 2020 22:30:04 +0100)

> Hi Anton,
> 
> Yes, the number of defines in Serialization.hpp is crazy.
> Changing/removing them to use pybind11 would take a lot of work.
> I'm sure it is possible, but not quickly.
> 
> We can open an issue about pybind11 switch. Discuss pros and cons,
> find a good strategy to deal with crazy defines and complete it later.
> 
> Doing this switch right now is unrealistic. Working on another moving
> target (pybind11) on top of an already moving target (minieigen HP
> not fully integrated) is irresponsible. If possible I would prefer:
> 
>   * minieigen-src package (slower compilation only for non-double)
>   * or integrate minieigen with yade sources for now (rather undesirable
> due to compilation time).
> 
> Whatever we do I would prefer to integrate HP, because doubles will
> appear very quickly in the code.
> 
> Ah, if we talked about pybind11 switch three months ago it would be
> perfect timing for me :( In fact I was thinking about it that time,
> but I didn't want to push into something that wasn't guaranteed to be
> working [*]. Figuring out the correct way to convert high precision
> to/from python with boost was difficult, for a while I prefer to not
> repeat this task with pybind11. But surely we can get back to this
> later.
> 
> The extra time for compilation happens only when non-double type is used.
> If minieigen-src package gets to 20.04 it will be possible later for
> people to compile high precision yade from gitlab sources without big hassle.
> 
> The build_focal_simple [1] is 5 minutes because it's not parallelized.
> The problem is that pybuild is not accepting -j argument.
> With make -j 6 it is faster [2][3].
> 
> Also a fast quad_double (62 decimal places, package libqd-dev)
> integration is in the works with boost::multiprecision developers [4].
> 
> To summarize:
> 
>  * I would rather avoid pybind11 switch in a rush, better give it a few months
>  * I would prefer minieigen-src package (no extra compilation time for people 
> who use `double`)
>  * minieigen-src in ubuntu 20.04 will let more people to test & use high 
> precision.
> 
> I understand that this solution is temporary until we figure a
> proper way to deal with pybind11. The 4 year LTS is more than enough
> to solve pybind11 + Serialization.hpp problem.
> 
> It is better to finish HP integration to not lose what is already
> working.
> 
> best regards
> Janek
> 
> 
> [*] pybind11 is a new dependency and a new package, boost is well
> integrated with itself, it was guaranteed to work.
> 
> [1] https://gitlab.com/yade-dev/minieigen/-/jobs/444927466
> 
> [2] https://gitlab.com/cosurgi/minieigen-real/pipelines/118211208 - I used 
> this to test HP
> 
> [3] In the pipeline with ccache it's usually not a problem. On
> debian build servers: it won't be a problem until we decide to
> make yade-float128 package. But if we made an accompanying
> minieigen-float128 this problem would disappear. But pybind11
> alternative is good also, just unrealistic right now.
> 
> [4] https://github.com/boostorg/multiprecision/issues/184
> 
> 
> Anton Gladky said: (by the date of Fri, 21 Feb 2020 20:09:07 +0100)
> 
> > Hi Janek,
> > 
> > we have misunderstanding here. python3-minieigen is the __binary__
> > package and it is a bad idea to ship the source code with the package.
> > 
> > Adding minieigen-src binary package is possible, but it looks like very
> > undesired way.
> > 
> > As I see, only Yade is using minieigen in the Debian. So, theoretically we
> > can merge it again with Yade. Not sure, whether it is a good way though
> > Yade compiles too slow. Minieigen adds definitely 4-5 compilation minutes
> > and Gigabytes of used RAM.
> > 
> > We should really have a look at pybind11 alternative, if it accelerates the
> > compilation. But when I see Serialization.hpp, I am getting crazy from the
> > number of "defines" there.
> > 
> > Anton
> > 
> > Am Fr., 21. Feb. 2020 um 17:04 Uhr schrieb Janek Kozicki (yade)
> > :
> > >
> > > Uh Anton, I'm sorry to be so boring:
> > >
> > > I have checked with sid in /etc/apt/sources.list:
> > >
> > >   deb-src http://ftp.pl.debian.org/debian/ sid  main non-free contrib
> > >
> > > and built the python3-minieigen_0.50.3+dfsg1-12_amd64.deb package.
> > >
> > > There is no /usr/include/minieigen/*pp files inside :(
> > >
> > > For high precision to work, they are necessary. Maybe the proper way
> > > to do this is to introduce python3-minieigen-dev package? I'm not
> > > sure. These sources are needed because of these include [1] statements.
> > >
> > > I am attaching once again the file python3-minieigen.install which
> > > installs *pp files. Even the *.cpp files are used. If you feel
> > > that *.cpp files are too much, we could duplicate the .cpp files in
> > > yade (they are rather short) and only 

Re: [Yade-dev] High precision Real and minieigen

[Janek Kozicki (yade)]

Hi Anton,

Yes, the number of defines in Serialization.hpp is crazy.
Changing/removing them to use pybind11 would take a lot of work.
I'm sure it is possible, but not quickly.

We can open an issue about pybind11 switch. Discuss pros and cons,
find a good strategy to deal with crazy defines and complete it later.

Doing this switch right now is unrealistic. Working on another moving
target (pybind11) on top of an already moving target (minieigen HP
not fully integrated) is irresponsible. If possible I would prefer:

  * minieigen-src package (slower compilation only for non-double)
  * or integrate minieigen with yade sources for now (rather undesirable
due to compilation time).

Whatever we do I would prefer to integrate HP, because doubles will
appear very quickly in the code.

Ah, if we talked about pybind11 switch three months ago it would be
perfect timing for me :( In fact I was thinking about it that time,
but I didn't want to push into something that wasn't guaranteed to be
working [*]. Figuring out the correct way to convert high precision
to/from python with boost was difficult, for a while I prefer to not
repeat this task with pybind11. But surely we can get back to this
later.

The extra time for compilation happens only when non-double type is used.
If minieigen-src package gets to 20.04 it will be possible later for
people to compile high precision yade from gitlab sources without big hassle.

The build_focal_simple [1] is 5 minutes because it's not parallelized.
The problem is that pybuild is not accepting -j argument.
With make -j 6 it is faster [2][3].

Also a fast quad_double (62 decimal places, package libqd-dev)
integration is in the works with boost::multiprecision developers [4].

To summarize:

 * I would rather avoid pybind11 switch in a rush, better give it a few months
 * I would prefer minieigen-src package (no extra compilation time for people 
who use `double`)
 * minieigen-src in ubuntu 20.04 will let more people to test & use high 
precision.

I understand that this solution is temporary until we figure a
proper way to deal with pybind11. The 4 year LTS is more than enough
to solve pybind11 + Serialization.hpp problem.

It is better to finish HP integration to not lose what is already
working.

best regards
Janek


[*] pybind11 is a new dependency and a new package, boost is well
integrated with itself, it was guaranteed to work.

[1] https://gitlab.com/yade-dev/minieigen/-/jobs/444927466

[2] https://gitlab.com/cosurgi/minieigen-real/pipelines/118211208 - I used this 
to test HP

[3] In the pipeline with ccache it's usually not a problem. On
debian build servers: it won't be a problem until we decide to
make yade-float128 package. But if we made an accompanying
minieigen-float128 this problem would disappear. But pybind11
alternative is good also, just unrealistic right now.

[4] https://github.com/boostorg/multiprecision/issues/184


Anton Gladky said: (by the date of Fri, 21 Feb 2020 20:09:07 +0100)

> Hi Janek,
> 
> we have misunderstanding here. python3-minieigen is the __binary__
> package and it is a bad idea to ship the source code with the package.
> 
> Adding minieigen-src binary package is possible, but it looks like very
> undesired way.
> 
> As I see, only Yade is using minieigen in the Debian. So, theoretically we
> can merge it again with Yade. Not sure, whether it is a good way though
> Yade compiles too slow. Minieigen adds definitely 4-5 compilation minutes
> and Gigabytes of used RAM.
> 
> We should really have a look at pybind11 alternative, if it accelerates the
> compilation. But when I see Serialization.hpp, I am getting crazy from the
> number of "defines" there.
> 
> Anton
> 
> Am Fr., 21. Feb. 2020 um 17:04 Uhr schrieb Janek Kozicki (yade)
> :
> >
> > Uh Anton, I'm sorry to be so boring:
> >
> > I have checked with sid in /etc/apt/sources.list:
> >
> >   deb-src http://ftp.pl.debian.org/debian/ sid  main non-free contrib
> >
> > and built the python3-minieigen_0.50.3+dfsg1-12_amd64.deb package.
> >
> > There is no /usr/include/minieigen/*pp files inside :(
> >
> > For high precision to work, they are necessary. Maybe the proper way
> > to do this is to introduce python3-minieigen-dev package? I'm not
> > sure. These sources are needed because of these include [1] statements.
> >
> > I am attaching once again the file python3-minieigen.install which
> > installs *pp files. Even the *.cpp files are used. If you feel
> > that *.cpp files are too much, we could duplicate the .cpp files in
> > yade (they are rather short) and only include the *.hpp files (these
> > are quite long). But all *pp files in /usr/include/minieigen/ would
> > be perfect.
> >
> > Best regards
> > Janek


-- 
--
Janek Kozicki, PhD. DSc. Arch. Assoc. Prof.
Gdańsk University of Technology
Faculty of Applied Physics and Mathematics
Department of Theoretical Physics and Quantum Information
--
http://yade-dem.org/
http://pg.edu.pl/jkozicki (click 

Re: [Yade-dev] High precision Real and minieigen

[Anton Gladky]

Hi Janek,

we have misunderstanding here. python3-minieigen is the __binary__
package and it is a bad idea to ship the source code with the package.

Adding minieigen-src binary package is possible, but it looks like very
undesired way.

As I see, only Yade is using minieigen in the Debian. So, theoretically we
can merge it again with Yade. Bot sure, whether it is a good way though
Yade compiles too slow. Minieigen adds definitely 4-5 compilation minutes
and Gigabytes of used RAM.

We should really have a look at pybind11 alternative, if it accelerates the
compilation. But when I see Serialization.hpp, I am getting crazy from the
number of "defines" there.

Anton

Am Fr., 21. Feb. 2020 um 17:04 Uhr schrieb Janek Kozicki (yade)
:
>
> Uh Anton, I'm sorry to be so boring:
>
> I have checked with sid in /etc/apt/sources.list:
>
>   deb-src http://ftp.pl.debian.org/debian/ sid  main non-free contrib
>
> and built the python3-minieigen_0.50.3+dfsg1-12_amd64.deb package.
>
> There is no /usr/include/minieigen/*pp files inside :(
>
> For high precision to work, they are necessary. Maybe the proper way
> to do this is to introduce python3-minieigen-dev package? I'm not
> sure. These sources are needed because of these include [1] statements.
>
> I am attaching once again the file python3-minieigen.install which
> installs *pp files. Even the *.cpp files are used. If you feel
> that *.cpp files are too much, we could duplicate the .cpp files in
> yade (they are rather short) and only include the *.hpp files (these
> are quite long). But all *pp files in /usr/include/minieigen/ would
> be perfect.
>
> Best regards
> Janek

___
Mailing list: https://launchpad.net/~yade-dev
Post to : yade-dev@lists.launchpad.net
Unsubscribe : https://launchpad.net/~yade-dev
More help   : https://help.launchpad.net/ListHelp

Re: [Yade-dev] High precision Real and minieigen

[Janek Kozicki (yade)]

Uh Anton, I'm sorry to be so boring:

I have checked with sid in /etc/apt/sources.list:

  deb-src http://ftp.pl.debian.org/debian/ sid  main non-free contrib

and built the python3-minieigen_0.50.3+dfsg1-12_amd64.deb package.

There is no /usr/include/minieigen/*pp files inside :(

For high precision to work, they are necessary. Maybe the proper way
to do this is to introduce python3-minieigen-dev package? I'm not
sure. These sources are needed because of these include [1] statements.

I am attaching once again the file python3-minieigen.install which
installs *pp files. Even the *.cpp files are used. If you feel
that *.cpp files are too much, we could duplicate the .cpp files in
yade (they are rather short) and only include the *.hpp files (these
are quite long). But all *pp files in /usr/include/minieigen/ would
be perfect.

Best regards
Janek


[1] 
https://gitlab.com/yade-dev/trunk/-/blob/master/py/high-precision/_ExposeMatrices.cpp


Anton Gladky said: (by the date of Thu, 20 Feb 2020 20:17:25 +0100)

> Hi Janek,
> 
> I habe backported both of your patches [1], [2] into the
> existing in Debian minieigen-package and uploaded into
> the Debian.
> 
> The newer minieigen can now be polished, new version released and
> uploaded with no rush.
> 
> [1] https://github.com/eudoxos/minieigen/pull/24
> [2] https://github.com/eudoxos/minieigen/pull/25
> 
> Best regards
> 
> Anton
> 
> Am Mi., 19. Feb. 2020 um 22:37 Uhr schrieb Janek Kozicki (yade)
> :
> >
> > I have asked Vaclav to transfer maintenance of miniEigen to us:
> >
> > https://github.com/eudoxos/minieigen/pull/26
> >
> > cheers
> > Janek


-- 
--
Janek Kozicki, PhD. DSc. Arch. Assoc. Prof.
Gdańsk University of Technology
Faculty of Applied Physics and Mathematics
Department of Theoretical Physics and Quantum Information
--
http://yade-dem.org/
http://pg.edu.pl/jkozicki (click English flag on top right)
src/*pp usr/include/minieigen/
___
Mailing list: https://launchpad.net/~yade-dev
Post to : yade-dev@lists.launchpad.net
Unsubscribe : https://launchpad.net/~yade-dev
More help   : https://help.launchpad.net/ListHelp

Re: [Yade-dev] High precision Real and minieigen

[Bruno Chareyre]

Hi, this is great. Thanks a lot Anton.
I removed yade-daily from yade-dev project on Gitlab, btw.
Bruno

p.s. As a side note, Vaçlav dropped minieigen support because of pybind
 move mainly. That package
seems to reduce compilation times significantly compared to boost - it was
suggested to me for this reason at least.





On Thu, 20 Feb 2020 at 21:57, Janek Kozicki (yade) 
wrote:

> Thank you Anton, this is very considerate of you.
>
> best regards
> Janek
>
> Anton Gladky said: (by the date of Thu, 20 Feb 2020 20:17:25 +0100)
>
> > Hi Janek,
> >
> > I have backported both of your patches [1], [2] into the
> > existing in Debian minieigen-package and uploaded into
> > the Debian.
> >
> > The newer minieigen can now be polished, new version released and
> > uploaded with no rush.
> >
> > [1] https://github.com/eudoxos/minieigen/pull/24
> > [2] https://github.com/eudoxos/minieigen/pull/25
> >
> > Best regards
> >
> > Anton
> >
> > Am Mi., 19. Feb. 2020 um 22:37 Uhr schrieb Janek Kozicki (yade)
> > :
> > >
> > > I have asked Vaclav to transfer maintenance of miniEigen to us:
> > >
> > > https://github.com/eudoxos/minieigen/pull/26
> > >
> > > cheers
> > > Janek
>
>
> --
> --
> Janek Kozicki, PhD. DSc. Arch. Assoc. Prof.
> Gdańsk University of Technology
> Faculty of Applied Physics and Mathematics
> Department of Theoretical Physics and Quantum Information
> --
> http://yade-dem.org/
> http://pg.edu.pl/jkozicki (click English flag on top right)
>
>
>

-- 
-- 
___
Bruno Chareyre
Associate Professor
ENSE³ - Grenoble INP
Lab. 3SR
BP 53
38041 Grenoble cedex 9
Tél : +33 4 56 52 86 21


Email too brief?
Here's why: email charter

___
Mailing list: https://launchpad.net/~yade-dev
Post to : yade-dev@lists.launchpad.net
Unsubscribe : https://launchpad.net/~yade-dev
More help   : https://help.launchpad.net/ListHelp

Re: [Yade-dev] High precision Real and minieigen

[Janek Kozicki (yade)]

Anton

did you remember about `./debian/python3-minieigen.install` file with content:

   src/*pp usr/include/minieigen/

? It is not a part of upstream repository. I guess It's part of
packaging process. At least that's how I understand it. Without
sources in /usr/include/minieigen/
yade high precision is not going to compile.

best regards
Janek

Janek Kozicki (yade) said: (by the date of Thu, 20 Feb 2020 21:57:54 +0100)

> Thank you Anton, this is very considerate of you.
> 
> best regards
> Janek
> 
> Anton Gladky said: (by the date of Thu, 20 Feb 2020 20:17:25 +0100)
> 
> > Hi Janek,
> > 
> > I have backported both of your patches [1], [2] into the
> > existing in Debian minieigen-package and uploaded into
> > the Debian.
> > 
> > The newer minieigen can now be polished, new version released and
> > uploaded with no rush.
> > 
> > [1] https://github.com/eudoxos/minieigen/pull/24
> > [2] https://github.com/eudoxos/minieigen/pull/25
> > 
> > Best regards
> > 
> > Anton
> > 
> > Am Mi., 19. Feb. 2020 um 22:37 Uhr schrieb Janek Kozicki (yade)
> > :
> > >
> > > I have asked Vaclav to transfer maintenance of miniEigen to us:
> > >
> > > https://github.com/eudoxos/minieigen/pull/26
> > >
> > > cheers
> > > Janek
> 
Janek Kozicki (yade) said: (by the date of Mon, 17 Feb 2020 13:18:42 +0100)

> Hi,
> 
> Vaclav has accepted my pull request to minieigen. The macro name is
> now `_HIGH_PRECISION_SUPPORT`, instead of `MINIEIGEN_OVERRIDE`.
> I will update the code.
> 
> Anton, what are the perspectives of updating debian/ubuntu package?
> 
> Please remember about `python3-minieigen.install` file with content:
> 
>src/*pp usr/include/minieigen/
> 
> it is not part of [2], but belongs to packaging process.
> 
> Vaclav suggests that we make minieigen a part of yade build.
> Personally I like the fact that it is a separate pacakge. However
> maybe we need to think about forking it into gitlab [3] as a fourth
> repository (we have yadedaily, docker-yade, trunk). And change the
> upstream for minieigen package.
> 
> I don't expect more changes in minieigen. It all works now. However
> some code from py/wrapper/customConverters.cpp could be safely moved
> there. So moving minieigen to yade-dev would allow some more cleanup.
> 
> best regards
> Janek
> 
> [1] https://github.com/eudoxos/minieigen/pull/24
> [2] https://github.com/eudoxos/minieigen
> [3] https://gitlab.com/yade-dev


-- 
--
Janek Kozicki, PhD. DSc. Arch. Assoc. Prof.
Gdańsk University of Technology
Faculty of Applied Physics and Mathematics
Department of Theoretical Physics and Quantum Information
--
http://yade-dem.org/
http://pg.edu.pl/jkozicki (click English flag on top right)

___
Mailing list: https://launchpad.net/~yade-dev
Post to : yade-dev@lists.launchpad.net
Unsubscribe : https://launchpad.net/~yade-dev
More help   : https://help.launchpad.net/ListHelp

Re: [Yade-dev] High precision Real and minieigen

[Janek Kozicki (yade)]

Thank you Anton, this is very considerate of you.

best regards
Janek

Anton Gladky said: (by the date of Thu, 20 Feb 2020 20:17:25 +0100)

> Hi Janek,
> 
> I have backported both of your patches [1], [2] into the
> existing in Debian minieigen-package and uploaded into
> the Debian.
> 
> The newer minieigen can now be polished, new version released and
> uploaded with no rush.
> 
> [1] https://github.com/eudoxos/minieigen/pull/24
> [2] https://github.com/eudoxos/minieigen/pull/25
> 
> Best regards
> 
> Anton
> 
> Am Mi., 19. Feb. 2020 um 22:37 Uhr schrieb Janek Kozicki (yade)
> :
> >
> > I have asked Vaclav to transfer maintenance of miniEigen to us:
> >
> > https://github.com/eudoxos/minieigen/pull/26
> >
> > cheers
> > Janek


-- 
--
Janek Kozicki, PhD. DSc. Arch. Assoc. Prof.
Gdańsk University of Technology
Faculty of Applied Physics and Mathematics
Department of Theoretical Physics and Quantum Information
--
http://yade-dem.org/
http://pg.edu.pl/jkozicki (click English flag on top right)

___
Mailing list: https://launchpad.net/~yade-dev
Post to : yade-dev@lists.launchpad.net
Unsubscribe : https://launchpad.net/~yade-dev
More help   : https://help.launchpad.net/ListHelp

Re: [Yade-dev] High precision Real and minieigen

[Anton Gladky]

Hi Janek,

I habe backported both of your patches [1], [2] into the
existing in Debian minieigen-package and uploaded into
the Debian.

The newer minieigen can now be polished, new version released and
uploaded with no rush.

[1] https://github.com/eudoxos/minieigen/pull/24
[2] https://github.com/eudoxos/minieigen/pull/25

Best regards

Anton

Am Mi., 19. Feb. 2020 um 22:37 Uhr schrieb Janek Kozicki (yade)
:
>
> I have asked Vaclav to transfer maintenance of miniEigen to us:
>
> https://github.com/eudoxos/minieigen/pull/26
>
> cheers
> Janek

___
Mailing list: https://launchpad.net/~yade-dev
Post to : yade-dev@lists.launchpad.net
Unsubscribe : https://launchpad.net/~yade-dev
More help   : https://help.launchpad.net/ListHelp

Re: [Yade-dev] High precision Real and minieigen

[Janek Kozicki (yade)]

I have asked Vaclav to transfer maintenance of miniEigen to us:

https://github.com/eudoxos/minieigen/pull/26

cheers
Janek


Anton Gladky said: (by the date of Tue, 18 Feb 2020 20:22:31 +0100)

> Hi Janek,
> 
> if Vaclav is not against it, I agree to move the miniegen back to
> Yade community, create yade/minieigen, release new version and
> upload it to Debian.
> 
> If we want to get a newer version in Ubuntu 20.04, it should be done
> as soon as possible.
> Automatic merge window is closing on 27.02.
> 
> PS Also yade/yadedaily can be removed. I have no enough permissions to do it.
> 
> Regards
> 
> Anton
> 
> Am Mo., 17. Feb. 2020 um 13:18 Uhr schrieb Janek Kozicki (yade)
> :
> >
> > Hi,
> >
> > Vaclav has accepted my pull request to minieigen. The macro name is
> > now `_HIGH_PRECISION_SUPPORT`, instead of `MINIEIGEN_OVERRIDE`.
> > I will update the code.
> >
> > Anton, what are the perspectives of updating debian/ubuntu package?
> >
> > Please remember about `python3-minieigen.install` file with content:
> >
> >src/*pp usr/include/minieigen/
> >
> > it is not part of [2], but belongs to packaging process.
> >
> > Vaclav suggests that we make minieigen a part of yade build.
> > Personally I like the fact that it is a separate pacakge. However
> > maybe we need to think about forking it into gitlab [3] as a fourth
> > repository (we have yadedaily, docker-yade, trunk). And change the
> > upstream for minieigen package.
> >
> > I don't expect more changes in minieigen. It all works now. However
> > some code from py/wrapper/customConverters.cpp could be safely moved
> > there. So moving minieigen to yade-dev would allow some more cleanup.
> >
> > best regards
> > Janek
> >
> > [1] https://github.com/eudoxos/minieigen/pull/24
> > [2] https://github.com/eudoxos/minieigen
> > [3] https://gitlab.com/yade-dev
> >
> > Janek Kozicki (yade) said: (by the date of Mon, 3 Feb 2020 15:48:19 
> > +0100)
> >
> > > In the attachment are the minimal minieigen patches:
> > >
> > > These are the super-minimal necessary patches:
> > >
> > > 1. 50_fix_minieigen_agnostic_to_Real.patch
> > > 2. 52_fix_convertible_Scalar_types.patch
> > > 3. python3-minieigen.install
> > >
> > > This one is not strictly necessary. It's just a bugfix. The Vector4r
> > > was broken, because 4th argument to the constructor was missing:
> > >
> > > 4. 51_fix_Vector4r.patch
> > >
> > > Sorry for the mixed numbering of patches, I preferred to keep the
> > > original (old) patch numbers.
> > >
> > >
> > > I am nicely surprised that these patches turned out to be so small.
> > > I just learned that when compiling with #included external library
> > > files the 'unused variable' warning is muted by g++. And so the
> > > largest patch is not necessary. Other patches which I had here
> > > initially were concerning the fixes of AlignedVector3. But for now we
> > > don't use it. We have make_SSE vectorization in [4] without it.
> > >
> > >
> > > I am really sorry for the delay. I wanted to be 100% sure that all
> > > pipelines pass. (there were some problems with the build servers).
> > >
> > > Anton, can you apply these patches to the minieigen package?
> > >
> > >
> > > In a couple of days I should also submit a merge request to minieigen 
> > > upstream.
> > >
> > > The latest pipeline [1] passes with a super-minimal set of patches [2][3].
> > >
> > > The MR 383 [1] can be merged after all the others are merged. It will
> > > then have only couple of commits. Also then I will rearrange the
> > > pipeline a bit, so that there are fewer asan tests, but they will be 
> > > rotating.
> > > Currently they use [5] 'debian-bullseye-SMALLEST-PATCH-minieigen' build 
> > > image.
> > >
> > > best regards
> > > Janek
> > >
> > >
> > > [1] https://gitlab.com/yade-dev/trunk/-/merge_requests/383/pipelines
> > > [2] 
> > > https://gitlab.com/yade-dev/docker-yade/commit/a3accab668a4896bff4e5ec1fdf0df6b741d638c
> > > [3] 
> > > https://gitlab.com/yade-dev/trunk/-/merge_requests/383/diffs?commit_id=fff0dc8fda163d57f8816e61201660cd3b6a0a38
> > > [4] https://gitlab.com/yade-dev/trunk/-/merge_requests/362
> > > [5] https://gitlab.com/yade-dev/docker-yade/container_registry
> > >
> > > Janek Kozicki (yade) said: (by the date of Mon, 27 Jan 2020 21:38:09 
> > > +0100)
> > >
> > > > Hi Anton,
> > > >
> > > > > Can we avoid it somehow? minieigen sources were in Yade tree many 
> > > > > years
> > > > > ago and we managed to drop them, introducing minieigen packages.
> > > > > It would not be the best variant to put them in the source again.
> > > >
> > > > Another solution is that you will include these patches [1] in the 
> > > > debian
> > > > and ubuntu package until Vaclav accepts my merge request (which I
> > > > will prepare in two or three days).
> > > >
> > > > I will review these patches again to see if they can be made smaller
> > > > than they currently are. And I will let you know by Wednesday evening.
> > > > (tomorrow I have lectures and a 

Re: [Yade-dev] High precision Real and minieigen

[Janek Kozicki (yade)]

Hi,

Vaclav has accepted my pull request to minieigen. The macro name is
now `_HIGH_PRECISION_SUPPORT`, instead of `MINIEIGEN_OVERRIDE`.
I will update the code.

Anton, what are the perspectives of updating debian/ubuntu package?

Please remember about `python3-minieigen.install` file with content:

   src/*pp usr/include/minieigen/

it is not part of [2], but belongs to packaging process.

Vaclav suggests that we make minieigen a part of yade build.
Personally I like the fact that it is a separate pacakge. However
maybe we need to think about forking it into gitlab [3] as a fourth
repository (we have yadedaily, docker-yade, trunk). And change the
upstream for minieigen package.

I don't expect more changes in minieigen. It all works now. However
some code from py/wrapper/customConverters.cpp could be safely moved
there. So moving minieigen to yade-dev would allow some more cleanup.

best regards
Janek

[1] https://github.com/eudoxos/minieigen/pull/24
[2] https://github.com/eudoxos/minieigen
[3] https://gitlab.com/yade-dev

Janek Kozicki (yade) said: (by the date of Mon, 3 Feb 2020 15:48:19 +0100)

> In the attachment are the minimal minieigen patches:
> 
> These are the super-minimal necessary patches:
> 
> 1. 50_fix_minieigen_agnostic_to_Real.patch
> 2. 52_fix_convertible_Scalar_types.patch
> 3. python3-minieigen.install
> 
> This one is not strictly necessary. It's just a bugfix. The Vector4r
> was broken, because 4th argument to the constructor was missing:
> 
> 4. 51_fix_Vector4r.patch
> 
> Sorry for the mixed numbering of patches, I preferred to keep the
> original (old) patch numbers.
> 
> 
> I am nicely surprised that these patches turned out to be so small.
> I just learned that when compiling with #included external library
> files the 'unused variable' warning is muted by g++. And so the
> largest patch is not necessary. Other patches which I had here
> initially were concerning the fixes of AlignedVector3. But for now we
> don't use it. We have make_SSE vectorization in [4] without it.
> 
> 
> I am really sorry for the delay. I wanted to be 100% sure that all
> pipelines pass. (there were some problems with the build servers).
> 
> Anton, can you apply these patches to the minieigen package?
> 
> 
> In a couple of days I should also submit a merge request to minieigen 
> upstream.
> 
> The latest pipeline [1] passes with a super-minimal set of patches [2][3].
> 
> The MR 383 [1] can be merged after all the others are merged. It will
> then have only couple of commits. Also then I will rearrange the
> pipeline a bit, so that there are fewer asan tests, but they will be rotating.
> Currently they use [5] 'debian-bullseye-SMALLEST-PATCH-minieigen' build image.
> 
> best regards
> Janek
> 
> 
> [1] https://gitlab.com/yade-dev/trunk/-/merge_requests/383/pipelines
> [2] 
> https://gitlab.com/yade-dev/docker-yade/commit/a3accab668a4896bff4e5ec1fdf0df6b741d638c
> [3] 
> https://gitlab.com/yade-dev/trunk/-/merge_requests/383/diffs?commit_id=fff0dc8fda163d57f8816e61201660cd3b6a0a38
> [4] https://gitlab.com/yade-dev/trunk/-/merge_requests/362
> [5] https://gitlab.com/yade-dev/docker-yade/container_registry
> 
> Janek Kozicki (yade) said: (by the date of Mon, 27 Jan 2020 21:38:09 
> +0100)
> 
> > Hi Anton,
> > 
> > > Can we avoid it somehow? minieigen sources were in Yade tree many years
> > > ago and we managed to drop them, introducing minieigen packages.
> > > It would not be the best variant to put them in the source again.
> > 
> > Another solution is that you will include these patches [1] in the debian
> > and ubuntu package until Vaclav accepts my merge request (which I
> > will prepare in two or three days).
> > 
> > I will review these patches again to see if they can be made smaller
> > than they currently are. And I will let you know by Wednesday evening.
> > (tomorrow I have lectures and a meeting).
> > 
> > Please note that file [2] installs the source files
> > in /usr/include/minieigen, they are needed. And also that's why [3]
> > is necessary. To partially avoid this we could work on preparing
> > `long double` and `float128` versions of minieigen package.
> > But for MPFR and boost::multiprecision::cpp_bin_float these sources are
> > still necessary because the precision is specified during compilation.
> > 
> > best regards
> > Janek
> > 
> > BTW: you have write access to [4]
> > 
> > [1] https://gitlab.com/cosurgi/minieigen-real/tree/master/patches
> > [2] 
> > https://gitlab.com/cosurgi/minieigen-real/blob/master/patches/python3-minieigen.install
> > [3] 
> > https://gitlab.com/cosurgi/minieigen-real/blob/master/patches/54_fix_compilation_warnings.patch
> > [4] https://gitlab.com/cosurgi/minieigen-real
> > 
> > 
> > 
> > 
> > Anton Gladky said: (by the date of Mon, 27 Jan 2020 21:07:15 +0100)
> > 
> > > Hi Janek,
> > > 
> > > > BTW, Anton did you notice that building debian package fails on
> > > > master due to missing python-gts?
> > > >