I remember this hang as well, it was pretty hard to reproduce IIRC. I
believe the stacks for the hang are here:
https://gist.github.com/KellenSunderland/893d11165e19d1efcf5c0fe8e8584600 and
the trick was we could only debug it up to the point that we hit:
#0 0x00007fec6df1ba4f in futex_wait (private=0, expected=1,
futex_word=0x7fec60843758)
at ../sysdeps/unix/sysv/linux/futex-internal.h:61
#1 futex_wait_simple (private=0, expected=1, futex_word=0x7fec60843758)
at ../sysdeps/nptl/futex-internal.h:135
#2 __pthread_once_slow (once_control=0x7fec60843758,
init_routine=0x7fec605f38f0)
at pthread_once.c:105
...
#6 0x00007fec6061c577 in cudaSetDevice () from
/usr/local/cuda/lib64/libcudart.so.9.0
because the code in libcudart is obviously closed source we couldn't dig
into what threading work was going on when we called cudaSetDevice.
On Mon, Jun 24, 2019 at 6:13 PM Pedro Larroy <pedro.larroy.li...@gmail.com>
wrote:
> If you check initialize.cc we seem to be explicitly disabling that
> behaviour in pthread_at_fork which seems to cause thread contention
> during multiprocessing. Why do we need this major advantage for the
> library if that's the case?
>
> Related PRs:
>
> https://github.com/apache/incubator-mxnet/pull/10820
> https://github.com/apache/incubator-mxnet/issues/14396
>
> The original code was authored in this PR:
>
> https://github.com/apache/incubator-mxnet/pull/8677
>
> I actually remember this fix, it was done during a release as the cuda
> runtime was forking and the engine was being re-entered. If that
> situation is now happening anymore it might not be needed any longer.
> I don't think we know the cause why there was a fork inside cuda, so
> the code has grown around a fix for an issue which its root cause was
> not understood, and side effects which this fix caused afterwards.
>
> My build uses MKL+LLVM OMP+DEBUG as seen in the container provided in
> the link above, no libgomp.
>
> I didn't try the Make build.
>
> I would refactor the code linked above and stop using pthread_at_fork,
> since OMP assumes it won't be initialized twice, but needs to be very
> well tested to make sure it doesn't cause bugs or affect the fixes
> done on the linked PRs above.
>
> Pedro.
>
> On Mon, Jun 24, 2019 at 5:38 PM Chris Olivier <cjolivie...@gmail.com>
> wrote:
> >
> > one major advantage of intel/llvm omp is that it spawns a new thread pool
> > after fork if a thread pool was already created. this is so that omp can
> be
> > used in the forked processes. libgomp doesn’t do this so it’ll just lock
> up
> > if you try to do omp in the forked process.
> >
> > is your build linking libgomp as well?
> >
> > standard mkl build (from Makefile) uses same omp library. are there
> > problems with that build?
> >
> > what changes need to be made to make the assertion not fire?
> >
> > On Mon, Jun 24, 2019 at 5:32 PM Pedro Larroy <
> pedro.larroy.li...@gmail.com>
> > wrote:
> >
> > > There's an assertion which is easily reproducible, and also there's a
> > > crash including core dump, the latter is not easy to reproduce for me
> > > in different environments. I have also seen mxnet getting stuck
> > > without progressing with this build configuration and using no CPU at
> > > all when running unit tests.
> > >
> > > In my view, the root cause of the assertion is that we are re-entering
> > > OMP initialization when spawning threads on the following code through
> > > pthread_at_fork
> > >
> > >
> https://github.com/apache/incubator-mxnet/blob/master/src/initialize.cc#L58
> > >
> > > This causes double initialization of the OMP engine, including the
> > > assertion which you are asking about, and I suspect some additional
> > > overhead. That's the shady forking part you are asking for.
> > >
> > > A question for you: What is the cause of runtime differences between
> > > OMP runtimes? Shouldn't the implementation overhead diminish as
> > > threads run longer?
> > >
> > > Pedro.
> > >
> > > On Mon, Jun 24, 2019 at 5:10 PM Chris Olivier <cjolivie...@gmail.com>
> > > wrote:
> > > >
> > > > What’s the reason for the assertion failure? btw classifying an
> assertion
> > > > failure a “crash” is debatable. As I stated in the original issue a
> long
> > > > time ago, it’s possible something shady is being done with when
> forking
> > > > that should be fixed. The assertion should be root caused.
> > > >
> > > >
> > > >
> > > > On Mon, Jun 24, 2019 at 1:22 PM Pedro Larroy <
> > > pedro.larroy.li...@gmail.com>
> > > > wrote:
> > > >
> > > > > Added a dockerfile, and reports of a crash in my local machine when
> > > > > running MKL+OMP+DEBUG, with Anton's branch the crash happened as
> well.
> > > > > I couldn't reproduce the crash on my EC2 machine:
> > > > > Added the backtrace of the crash as well.
> > > > >
> > > > > https://github.com/apache/incubator-mxnet/issues/10856
> > > > >
> > > > > Dockerfile here:
> > > > >
> > > > > https://github.com/larroy/mxnet_omp
> > > > >
> > > > > Kind regards.
> > > > >
> > > > > Pedro.
> > > > >
> > > > > On Thu, Jun 20, 2019 at 5:29 PM Marco de Abreu <
> > > marco.g.ab...@gmail.com>
> > > > > wrote:
> > > > > >
> > > > > > As already proposed, I think the easiest way to get a common
> > > > > understanding
> > > > > > is if we start with a few docker containers. Pedro, would it be
> > > possible
> > > > > > for you to wrap your benchmarks into a few containers that will
> > > produce
> > > > > > your shown results? That way, we can avoid possible
> > > misunderstandings and
> > > > > > also pinpoint the exact parts where people disagree or
> misunderstood
> > > each
> > > > > > other.
> > > > > >
> > > > > > -Marco
> > > > > >
> > > > > > Pedro Larroy <pedro.larroy.li...@gmail.com> schrieb am Do., 20.
> Juni
> > > > > 2019,
> > > > > > 21:47:
> > > > > >
> > > > > > > I can confirm that we are linking with two versions of omp, I'm
> > > > > > > gaining more clarity into this topic, but I have still
> questions,
> > > the
> > > > > > > facts that I got so far are the folllowing:
> > > > > > >
> > > > > > > * #1: We are linking with two versions of omp, intel's omp and
> llvm
> > > > > > > openmp when building with MKL enabled.
> > > > > > > * #2: We have 3 different possible OMP versions: Intel OMP
> (comes
> > > with
> > > > > > > MKL), LLVM OpenMP (3rdparty/openmp), libgomp (comes with gcc)
> (This
> > > > > > > one is used on the PR proposed by Anton).
> > > > > > >
> > > > > > > Questions:
> > > > > > >
> > > > > > > * #1 Is it ok to have two versions of openmp linked at the
> same
> > > time?
> > > > > > > * #2 Which implementation of OMP gives the best performance?
> (See
> > > > > > > total training time of my measurement for a partial answer)
> > > > > > > * #3 Should we have a build flag so we can choose the OMP
> version
> > > at
> > > > > > > runtime?
> > > > > > > * #4 Which Compiler and build flags did Chris use to get 10x
> > > slowdown?
> > > > > > > * #5 @Stas: is there a script to replicate your benchmarks
> > > easily? If
> > > > > > > so could you provide a link? I think we would need to
> reproduce
> > > your
> > > > > > > benchmarks and verify which versions are being linked. It's
> > > possible
> > > > > > > that while compiling with MKL intel's omp was pulled in
> instead of
> > > > > > > GNU OpenMP.
> > > > > > > * #6 @Chris: how to maintain the copy of LLVM's Openmp?
> Should we
> > > > > > > update the subrepo regularly?
> > > > > > >
> > > > > > > My conclusion so far:
> > > > > > >
> > > > > > > * #1 We should avoid linking two versions of omp if possible
> and
> > > > > > > allow users to choose one in the build as we do for BLAS.
> > > > > > > * #2 For performance reasons and more control vs different
> > > compiler
> > > > > > > versions seems it makes indeed sense to keep the LLVM OpenMP
> > > version
> > > > > > > in 3rdparty for now. So unless some more data is gathered, it
> makes
> > > > > > > sense not to remove it as of now.
> > > > > > > * #3 We should provide build options to choose which openmp
> > > library
> > > > > > > is to be used from the three options available, including
> libgomp.
> > > > > > > * #4 Refining the build we could also enable OpenMP in mac
> without
> > > > > > > additional contortions (doesn't work as of today):
> > > > > > > https://iscinumpy.gitlab.io/post/omp-on-high-sierra/
> > > > > > > * #5 We should add different omp versions to our benchmarks
> and
> > > track
> > > > > > > the performance, so this data is available for prescribing the
> best
> > > > > > > build options and for binary releases.
> > > > > > >
> > > > > > > This is also an interesting related gh issue posted in the
> mkl-dnn
> > > > > > > repository: https://github.com/intel/mkl-dnn/issues/230
> > > > > > >
> > > > > > >
> > > > > > > I don't observe the order of magnitude divergence reported by
> > > Chris in
> > > > > > > vanilla Ubuntu 18.04 in samples / s but the full training
> finishes
> > > > > > > indeed faster with the OMP from 3rdparty (LLVM openmp) vs
> libgomp.
> > > > > > >
> > > > > > > There's also differences in training time when using MKL and
> the ,
> > > > > > > it's actually a bit slower, I don't know if it's related to
> OMP.
> > > > > > >
> > > > > > > gcc version 7.4.0 (Ubuntu 7.4.0-1ubuntu1~18.04.1)
> > > > > > >
> > > > > > > Anton's branch: g...@github.com:lebeg/incubator-mxnet.git
> branch
> > > > > 'omp'
> > > > > > > (py3_venv) piotr@ec2 cpu:0: ~/mxnet_openmp [omp]> ldd
> > > > > > > build/libmxnet.so |grep -i omp
> > > > > > > libgomp.so.1 => /usr/lib/x86_64-linux-gnu/libgomp.so.1
> > > > > > > (0x00007fd99a51d000)
> > > > > > >
> > > > > > > time python train_mnist.py
> > > > > > >
> > > > > > > INFO:root:Epoch[18] Validation-accuracy=0.984176
> > > > > > > INFO:root:Epoch[19] Batch [0-100] Speed: 41617.00
> samples/sec
> > > > > > > accuracy=1.000000
> > > > > > > INFO:root:Epoch[19] Batch [100-200] Speed: 47990.69
> samples/sec
> > > > > > > accuracy=0.999531
> > > > > > > INFO:root:Epoch[19] Batch [200-300] Speed: 47517.01
> samples/sec
> > > > > > > accuracy=0.999687
> > > > > > > INFO:root:Epoch[19] Batch [300-400] Speed: 47430.53
> samples/sec
> > > > > > > accuracy=1.000000
> > > > > > > INFO:root:Epoch[19] Batch [400-500] Speed: 47649.77
> samples/sec
> > > > > > > accuracy=0.999687
> > > > > > > INFO:root:Epoch[19] Batch [500-600] Speed: 51708.12
> samples/sec
> > > > > > > accuracy=0.999687
> > > > > > > INFO:root:Epoch[19] Batch [600-700] Speed: 57228.63
> samples/sec
> > > > > > > accuracy=0.999375
> > > > > > > INFO:root:Epoch[19] Batch [700-800] Speed: 50887.85
> samples/sec
> > > > > > > accuracy=0.999844
> > > > > > > INFO:root:Epoch[19] Batch [800-900] Speed: 53947.98
> samples/sec
> > > > > > > accuracy=0.999531
> > > > > > > INFO:root:Epoch[19] Train-accuracy=0.999717
> > > > > > > INFO:root:Epoch[19] Time cost=1.219
> > > > > > > INFO:root:Epoch[19] Validation-accuracy=0.983977
> > > > > > > 1011.98user 26.78system 0:31.54elapsed 3292%CPU
> (0avgtext+0avgdata
> > > > > > > 1146052maxresident)k
> > > > > > > 0inputs+0outputs (0major+3496364minor)pagefaults 0swaps
> > > > > > >
> > > > > > > Master, MKL ON:
> > > > > > >
> > > > > > > (py3_venv) piotr@ec2 cpu:1: ~/m/e/image-classification
> [master]>
> > > ldd
> > > > > > > ../../build/libmxnet.so | grep -i omp
> > > > > > > libomp.so =>
> > > > > > >
> > > /home/piotr/mxnet_master/build/3rdparty/openmp/runtime/src/libomp.so
> > > > > > > (0x00007f05ba38f000)
> > > > > > > libiomp5.so =>
> > > > > > >
> > > > > > >
> > > > >
> > >
> /home/piotr/mxnet_master/build/mklml/mklml_lnx_2019.0.5.20190502/lib/libiomp5.so
> > > > > > > (0x00007f05b09f4000)
> > > > > > >
> > > > > > > INFO:root:Epoch[18] Validation-accuracy=0.982484
> > > > > > > INFO:root:Epoch[19] Batch [0-100] Speed: 36651.63
> samples/sec
> > > > > > > accuracy=0.999691
> > > > > > > INFO:root:Epoch[19] Batch [100-200] Speed: 45093.98
> samples/sec
> > > > > > > accuracy=0.999844
> > > > > > > INFO:root:Epoch[19] Batch [200-300] Speed: 45146.84
> samples/sec
> > > > > > > accuracy=0.999687
> > > > > > > INFO:root:Epoch[19] Batch [300-400] Speed: 45119.90
> samples/sec
> > > > > > > accuracy=0.999687
> > > > > > > INFO:root:Epoch[19] Batch [400-500] Speed: 44998.96
> samples/sec
> > > > > > > accuracy=0.999531
> > > > > > > INFO:root:Epoch[19] Batch [500-600] Speed: 45072.25
> samples/sec
> > > > > > > accuracy=0.999844
> > > > > > > INFO:root:Epoch[19] Batch [600-700] Speed: 44969.79
> samples/sec
> > > > > > > accuracy=0.999844
> > > > > > > INFO:root:Epoch[19] Batch [700-800] Speed: 44962.78
> samples/sec
> > > > > > > accuracy=0.999844
> > > > > > > INFO:root:Epoch[19] Batch [800-900] Speed: 44945.47
> samples/sec
> > > > > > > accuracy=0.999375
> > > > > > > INFO:root:Epoch[19] Train-accuracy=0.999717
> > > > > > > INFO:root:Epoch[19] Time cost=1.367
> > > > > > > INFO:root:Epoch[19] Validation-accuracy=0.982783
> > > > > > > 854.97user 847.21system 0:41.44elapsed 4106%CPU
> (0avgtext+0avgdata
> > > > > > > 1154348maxresident)k
> > > > > > > 0inputs+0outputs (0major+3624361minor)pagefaults 0swaps
> > > > > > >
> > > > > > >
> > > > > > > MKL OFF:
> > > > > > > (py3_venv) piotr@ec2 cpu:0: ~/mxnet_master [master]> grep -i
> MKL
> > > > > > > cmake_options.yml
> > > > > > > USE_MKL_IF_AVAILABLE: "OFF" # Use MKL if found
> > > > > > > USE_MKLML_MKL: "OFF" # Use MKLDNN variant of MKL (if MKL
> found) IF
> > > > > > > USE_MKL_IF_AVAILABLE AND (NOT APPLE)
> > > > > > > USE_MKLDNN: "OFF" # Use MKLDNN variant of MKL (if MKL found) IF
> > > > > > > USE_MKL_IF_AVAILABLE AND (NOT APPLE)
> > > > > > > (py3_venv) piotr@ec2 cpu:0: ~/mxnet_master [master]> ldd
> > > > > > > build/libmxnet.so |grep -i omp
> > > > > > > libomp.so =>
> > > > > > >
> > > /home/piotr/mxnet_master/build/3rdparty/openmp/runtime/src/libomp.so
> > > > > > > (0x00007fb720c54000)
> > > > > > >
> > > > > > > INFO:root:Epoch[18] Validation-accuracy=0.983479
> > > > > > > INFO:root:Epoch[19] Batch [0-100] Speed: 46784.02
> samples/sec
> > > > > > > accuracy=1.000000
> > > > > > > INFO:root:Epoch[19] Batch [100-200] Speed: 48824.29
> samples/sec
> > > > > > > accuracy=0.999687
> > > > > > > INFO:root:Epoch[19] Batch [200-300] Speed: 49190.31
> samples/sec
> > > > > > > accuracy=0.999687
> > > > > > > INFO:root:Epoch[19] Batch [300-400] Speed: 51518.77
> samples/sec
> > > > > > > accuracy=0.999844
> > > > > > > INFO:root:Epoch[19] Batch [400-500] Speed: 51551.62
> samples/sec
> > > > > > > accuracy=0.999844
> > > > > > > INFO:root:Epoch[19] Batch [500-600] Speed: 49026.35
> samples/sec
> > > > > > > accuracy=0.999844
> > > > > > > INFO:root:Epoch[19] Batch [600-700] Speed: 49002.46
> samples/sec
> > > > > > > accuracy=0.999375
> > > > > > > INFO:root:Epoch[19] Batch [700-800] Speed: 48980.55
> samples/sec
> > > > > > > accuracy=0.999687
> > > > > > > INFO:root:Epoch[19] Batch [800-900] Speed: 47402.56
> samples/sec
> > > > > > > accuracy=0.999844
> > > > > > > INFO:root:Epoch[19] Train-accuracy=0.999767
> > > > > > > INFO:root:Epoch[19] Time cost=1.259
> > > > > > > INFO:root:Epoch[19] Validation-accuracy=0.983181
> > > > > > > 755.36user 754.94system 0:35.89elapsed 4207%CPU
> (0avgtext+0avgdata
> > > > > > > 1147008maxresident)k
> > > > > > > 0inputs+3112outputs (0major+3568826minor)pagefaults 0swaps
> > > > > > >
> > > > > > > Let me know what you think.
> > > > > > >
> > > > > > > Link to the original PR:
> > > > > > > https://github.com/apache/incubator-mxnet/pull/12160
> > > > > > >
> > > > > > > Thanks.
> > > > > > >
> > > > > > > On Wed, Jun 19, 2019 at 5:35 PM kellen sunderland
> > > > > > > <kellen.sunderl...@gmail.com> wrote:
> > > > > > > >
> > > > > > > > "if you’re linking in two then you’re doing something wrong."
> > > > > Correct,
> > > > > > > > that's one thing I believe we've got consensus on. So let's
> call
> > > > > that
> > > > > > > out
> > > > > > > > as a bug to be fixed.
> > > > > > > >
> > > > > > > > Let's move forward with some reproducible numbers and then
> > > discuss
> > > > > the
> > > > > > > pros
> > > > > > > > / cons of which particular OMP implementation we should use.
> > > > > > > >
> > > > > > > > On Wed, Jun 19, 2019 at 3:06 PM Pedro Larroy <
> > > > > > > pedro.larroy.li...@gmail.com>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hi Chris
> > > > > > > > >
> > > > > > > > > I would ask you to have a bit of patience and help us with
> your
> > > > > > > > > experience in this matter. Nobody is ignoring anything, I
> > > think we
> > > > > are
> > > > > > > > > individually gathering feedbacks and trying to understand
> the
> > > > > multiple
> > > > > > > > > contributions done to this topic including yours, then go
> step
> > > by
> > > > > > > > > step, understand what is going on and run experiments and
> > > report
> > > > > back
> > > > > > > > > to the list or the corresponding github item. It was
> suggested
> > > by
> > > > > > > > > Kellen to prepare some containers, this takes effort.
> > > > > > > > >
> > > > > > > > > Regarding your final comment, most of us also have many
> other
> > > > > things
> > > > > > > > > to do and responsibilities even if our daytime jobs might
> > > involve
> > > > > > > > > MXNet in some form or another. I think that's part of the
> > > privilege
> > > > > > > > > and responsibility of working close with an open source
> > > project and
> > > > > > > > > the magic of collaboration across organizations. Let's all
> be
> > > > > patient
> > > > > > > > > and take some time to understand and reason about this
> topic
> > > which
> > > > > is
> > > > > > > > > not simple. Since we decided to step back and gather more
> data
> > > > > let's
> > > > > > > > > take time and do it properly.
> > > > > > > > >
> > > > > > > > > Personally I hope to find time to look again into this
> issue
> > > before
> > > > > > > > > the end of the week.
> > > > > > > > >
> > > > > > > > > Thanks.
> > > > > > > > >
> > > > > > > > > Pedro.
> > > > > > > > >
> > > > > > > > > On Wed, Jun 19, 2019 at 2:43 PM Chris Olivier <
> > > > > cjolivie...@apache.org>
> > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > if you’re linking in two then you’re doing something
> wrong.
> > > You
> > > > > can
> > > > > > > see
> > > > > > > > > by
> > > > > > > > > > my email yesterday that only one is linked in. This is
> also
> > > the
> > > > > case
> > > > > > > with
> > > > > > > > > > the mkl version built by the Makefile — only the Intel
> OMP
> > > > > library is
> > > > > > > > > used
> > > > > > > > > > (no libgomp).
> > > > > > > > > >
> > > > > > > > > > That being said, Do you have clear evidence that using
> Intel
> > > OMP
> > > > > is
> > > > > > > both
> > > > > > > > > > problematic and the situation isn’t fixable? The burden
> of
> > > > > proof is
> > > > > > > on
> > > > > > > > > the
> > > > > > > > > > ones requesting the change — it is not my responsibility
> to
> > > > > justify
> > > > > > > the
> > > > > > > > > > current state. There must be something “terrible” and
> > > unfixable
> > > > > to
> > > > > > > > > justify
> > > > > > > > > > a change. I have seen no proof of this in all this time.
> > > > > > > > > >
> > > > > > > > > > On a side note, I mentioned a couple of things in my
> email
> > > > > yesterday
> > > > > > > that
> > > > > > > > > > still are not being responded to (they were also ignored
> in
> > > the
> > > > > last
> > > > > > > > > > incarnation of this “discussion” — I have much
> experience in
> > > this
> > > > > > > matter
> > > > > > > > > to
> > > > > > > > > > assume “discussion” is a waste of my time, seeing and I
> am
> > > not
> > > > > paid
> > > > > > > to
> > > > > > > > > > “work on” mxnet like y’all are).
> > > > > > > > > >
> > > > > > > > > > -C
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > On Wed, Jun 19, 2019 at 10:28 AM kellen sunderland <
> > > > > > > > > > kellen.sunderl...@gmail.com> wrote:
> > > > > > > > > >
> > > > > > > > > > > I've also quite often seen two versions of OpenMP
> linked.
> > > I
> > > > > think
> > > > > > > we
> > > > > > > > > can
> > > > > > > > > > > all agree we probably want to avoid linking in two
> > > libraries
> > > > > that
> > > > > > > do
> > > > > > > > > > > effectively the same thing.
> > > > > > > > > > >
> > > > > > > > > > > The performance questions should be fairly straight
> > > forward to
> > > > > > > > > demonstrate
> > > > > > > > > > > right? Could we just collaborate on a few minimal
> > > Dockerfiles
> > > > > that
> > > > > > > > > show
> > > > > > > > > > > (or don't show) Intel OpenMP performance speedups with
> the
> > > > > > > workloads
> > > > > > > > > Chris
> > > > > > > > > > > is referencing?
> > > > > > > > > > >
> > > > > > > > > > > On Wed, Jun 19, 2019 at 4:44 AM Tsukrov, Stanislav <
> > > > > > > > > > > stanislav.tsuk...@gmail.com> wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Hi, Chris!
> > > > > > > > > > > >
> > > > > > > > > > > > Stas here - I've gathered that performance data.
> > > > > > > > > > > > Sure thing, I can be wrong, but please elaborate a
> bit on
> > > > > what
> > > > > > > we are
> > > > > > > > > > > > missing.
> > > > > > > > > > > > Be assured, intentional misdirection was never a
> case.
> > > > > > > > > > > >
> > > > > > > > > > > > Thanks a lot for being constructive.
> > > > > > > > > > > >
> > > > > > > > > > > > > Turning Intel OMP on and off (and MKL as well,
> since it
> > > > > tends
> > > > > > > to
> > > > > > > > > pull
> > > > > > > > > > > in
> > > > > > > > > > > > omp, depending which one is linked in).
> > > > > > > > > > > >
> > > > > > > > > > > > We never ever considered turning MKL off. We are on
> the
> > > same
> > > > > page
> > > > > > > > > here -
> > > > > > > > > > > > MKL is crucial for the performance.
> > > > > > > > > > > > Why should we? There's a GOMP-linked version of MKL,
> > > that we
> > > > > can
> > > > > > > use.
> > > > > > > > > > > >
> > > > > > > > > > > > What we did - we measured, if using compilers default
> > > OpenMP
> > > > > > > > > > > > implementation instead of referenced source code
> > > > > distribution of
> > > > > > > > > OpenMP
> > > > > > > > > > > > makes anything slower.
> > > > > > > > > > > > We have found the impact to be hardly measurable.
> > > > > > > > > > > > The difference between GOMP and iOMP is <5% on our
> > > > > benchmarks,
> > > > > > > most
> > > > > > > > > of
> > > > > > > > > > > the
> > > > > > > > > > > > time less than that.
> > > > > > > > > > > >
> > > > > > > > > > > > We just suggest to simplify the build of mxnet, by
> > > removing
> > > > > the
> > > > > > > > > > > > unnecessary dependency.
> > > > > > > > > > > >
> > > > > > > > > > > > During that we discovered for example the following
> > > amazing
> > > > > > > issue:
> > > > > > > > > > > >
> https://github.com/apache/incubator-mxnet/issues/14087
> > > > > > > > > > > >
> > > > > > > > > > > > Best Regards
> > > > > > > > > > > >
> > > > > > > > > > > > Stas
> > > > > > > > > > > >
> > > > > > > > > > > > On 18.06.19, 18:24, "Chris Olivier" <
> > > cjolivie...@gmail.com>
> > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > I am very reluctant to feed the trolls again, and
> > > this
> > > > > will
> > > > > > > be
> > > > > > > > > teh
> > > > > > > > > > > last
> > > > > > > > > > > > time I address Pedro or Anton on the subject, but
> > > since I
> > > > > > > think
> > > > > > > > > the
> > > > > > > > > > > > numbers
> > > > > > > > > > > > being presented are incorrect (either by te
> builders
> > > not
> > > > > > > really
> > > > > > > > > > > > understanding what they are building, or possibly
> > > > > intentional
> > > > > > > > > > > > misdirection):
> > > > > > > > > > > >
> > > > > > > > > > > > Turning Intel OMP on and off (and MKL as well,
> since
> > > it
> > > > > > > tends to
> > > > > > > > > pull
> > > > > > > > > > > > in
> > > > > > > > > > > > omp, depending which one is linked in).
> > > > > > > > > > > > There is a HUGE difference. This is consistent
> with
> > > my
> > > > > > > > > experience
> > > > > > > > > > > > before
> > > > > > > > > > > > when it was added.
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > default mnist:
> > > > > > > > > > > >
> > > > > > > > > > > > python
> ../example/image-classification/train_mnist.py
> > > > > > > > > > > > INFO:root:start with arguments
> > > Namespace(add_stn=False,
> > > > > > > > > > > batch_size=64,
> > > > > > > > > > > > disp_batches=100, dtype='float32',
> gc_threshold=0.5,
> > > > > > > > > gc_type='none',
> > > > > > > > > > > > gpus=None, image_shape='1, 28, 28',
> > > > > initializer='default',
> > > > > > > > > > > > kv_store='device', load_epoch=None, loss='',
> lr=0.05,
> > > > > > > > > lr_factor=0.1,
> > > > > > > > > > > > lr_step_epochs='10', macrobatch_size=0,
> > > > > model_prefix=None,
> > > > > > > > > mom=0.9,
> > > > > > > > > > > > monitor=0, network='mlp', num_classes=10,
> > > num_epochs=20,
> > > > > > > > > > > > num_examples=60000, num_layers=None,
> optimizer='sgd',
> > > > > > > > > > > > profile_server_suffix='',
> profile_worker_suffix='',
> > > > > > > > > save_period=1,
> > > > > > > > > > > > test_io=0, top_k=0, warmup_epochs=5,
> > > > > > > warmup_strategy='linear',
> > > > > > > > > > > > wd=0.0001)
> > > > > > > > > > > >
> > > > > > > > > > > > INTEL OMP:
> > > > > > > > > > > >
> > > > > > > > > > > > ldd libmxnet.so | grep omp
> > > > > > > > > > > > libomp.so =>
> > > > > > > > > > > >
> > > > > > > > >
> > > > >
> /home/chris/src/mxnet/cmake_omp/3rdparty/openmp/runtime/src/libomp.so
> > > > > > > > > > > > (0x00007f978fde7000)
> > > > > > > > > > > >
> > > > > > > > > > > > :root:Epoch[0] Batch [0-100] Speed:
> 31548.09
> > > > > > > samples/sec
> > > > > > > > > > > > accuracy=0.780012
> > > > > > > > > > > > INFO:root:Epoch[0] Batch [100-200] Speed:
> > > 16073.21
> > > > > > > > > samples/sec
> > > > > > > > > > > > accuracy=0.920469
> > > > > > > > > > > > INFO:root:Epoch[0] Batch [200-300] Speed:
> > > 19075.91
> > > > > > > > > samples/sec
> > > > > > > > > > > > accuracy=0.928281
> > > > > > > > > > > > INFO:root:Epoch[0] Batch [300-400] Speed:
> > > 23211.36
> > > > > > > > > samples/sec
> > > > > > > > > > > > accuracy=0.942813
> > > > > > > > > > > > INFO:root:Epoch[0] Batch [400-500] Speed:
> > > 22139.79
> > > > > > > > > samples/sec
> > > > > > > > > > > > accuracy=0.938750
> > > > > > > > > > > > INFO:root:Epoch[0] Batch [500-600] Speed:
> > > 23225.52
> > > > > > > > > samples/sec
> > > > > > > > > > > > accuracy=0.946562
> > > > > > > > > > > > INFO:root:Epoch[0] Batch [600-700] Speed:
> > > 19547.41
> > > > > > > > > samples/sec
> > > > > > > > > > > > accuracy=0.953281
> > > > > > > > > > > > INFO:root:Epoch[0] Batch [700-800] Speed:
> > > 24111.73
> > > > > > > > > samples/sec
> > > > > > > > > > > > accuracy=0.951562
> > > > > > > > > > > > INFO:root:Epoch[0] Batch [800-900] Speed:
> > > 13959.88
> > > > > > > > > samples/sec
> > > > > > > > > > > > accuracy=0.957500
> > > > > > > > > > > > INFO:root:Epoch[0] Train-accuracy=0.925423
> > > > > > > > > > > > INFO:root:Epoch[0] Time cost=3.806
> > > > > > > > > > > > INFO:root:Epoch[0] Validation-accuracy=0.962580
> > > > > > > > > > > > INFO:root:Epoch[1] Batch [0-100] Speed:
> > > 24560.21
> > > > > > > > > samples/sec
> > > > > > > > > > > > accuracy=0.968131
> > > > > > > > > > > > INFO:root:Epoch[1] Batch [100-200] Speed:
> > > 23457.03
> > > > > > > > > samples/sec
> > > > > > > > > > > > accuracy=0.966250
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > LIBGOMP:
> > > > > > > > > > > >
> > > > > > > > > > > > ldd libmxnet.so | grep omp
> > > > > > > > > > > > libgomp.so.1 =>
> > > > > > > /usr/lib/x86_64-linux-gnu/libgomp.so.1
> > > > > > > > > > > > (0x00007f25c25dd000)
> > > > > > > > > > > >
> > > > > > > > > > > > INFO:root:Epoch[0] Batch [0-100] Speed:
> > > 1731.01
> > > > > > > > > samples/sec
> > > > > > > > > > > > accuracy=0.782488
> > > > > > > > > > > > INFO:root:Epoch[0] Batch [100-200] Speed:
> > > 3551.32
> > > > > > > > > samples/sec
> > > > > > > > > > > > accuracy=0.907813
> > > > > > > > > > > > INFO:root:Epoch[0] Batch [200-300] Speed:
> > > 1991.00
> > > > > > > > > samples/sec
> > > > > > > > > > > > accuracy=0.927188
> > > > > > > > > > > > INFO:root:Epoch[0] Batch [300-400] Speed:
> > > 2175.45
> > > > > > > > > samples/sec
> > > > > > > > > > > > accuracy=0.937969
> > > > > > > > > > > > INFO:root:Epoch[0] Batch [400-500] Speed:
> > > 1644.95
> > > > > > > > > samples/sec
> > > > > > > > > > > > accuracy=0.942187
> > > > > > > > > > > > INFO:root:Epoch[0] Batch [500-600] Speed:
> > > 6444.58
> > > > > > > > > samples/sec
> > > > > > > > > > > > accuracy=0.950156
> > > > > > > > > > > > INFO:root:Epoch[0] Batch [600-700] Speed:
> > > 7842.16
> > > > > > > > > samples/sec
> > > > > > > > > > > > accuracy=0.947969
> > > > > > > > > > > > INFO:root:Epoch[0] Batch [700-800] Speed:
> > > 9412.07
> > > > > > > > > samples/sec
> > > > > > > > > > > > accuracy=0.953750
> > > > > > > > > > > > INFO:root:Epoch[0] Batch [800-900] Speed:
> > > 12707.58
> > > > > > > > > samples/sec
> > > > > > > > > > > > accuracy=0.953125
> > > > > > > > > > > >
> > > > > > > > > > > > That being said, there's other issued beyond
> speed.
> > > The
> > > > > > > DEFAULT
> > > > > > > > > > > build
> > > > > > > > > > > > from
> > > > > > > > > > > > makefile (not CMake) uses Intel OMP mkl (I showed
> > > > > before) and
> > > > > > > > > > > > mysteriously
> > > > > > > > > > > > it has no issues? This seems highly suspicious.
> > > All I
> > > > > see
> > > > > > > is a
> > > > > > > > > lot
> > > > > > > > > > > of
> > > > > > > > > > > > hand-waving and conjecture and pointing to
> > > StackOverflow
> > > > > > > posts
> > > > > > > > > made
> > > > > > > > > > > by
> > > > > > > > > > > > people who may be of questionable pedigree to
> begin
> > > with.
> > > > > > > This
> > > > > > > > > > > smells
> > > > > > > > > > > > of a
> > > > > > > > > > > > Pedro-ego-fight rather than one of purely
> technical
> > > > > merit.
> > > > > > > > > Also, if
> > > > > > > > > > > > one
> > > > > > > > > > > > knows how OMP works, they would be very
> suspicious
> > > of
> > > > > the
> > > > > > > > > > > > "intermittent
> > > > > > > > > > > > hangs" claim -- that's probably just broken race
> > > > > conditions
> > > > > > > > > elsewhere
> > > > > > > > > > > > until
> > > > > > > > > > > > proven differently. It'd tend freeze on the
> first
> > > use if
> > > > > > > > > something
> > > > > > > > > > > is
> > > > > > > > > > > > wrong (try using libgomp after a fork and see),
> since
> > > > > worker
> > > > > > > > > threads"
> > > > > > > > > > > > wouldn't be assigned/joined properly. IntelOMP
> is
> > > > > faster,
> > > > > > > but
> > > > > > > > > also
> > > > > > > > > > > has
> > > > > > > > > > > > other advantages, such as allowing OMP after a
> fork.
> > > > > > > > > > > >
> > > > > > > > > > > > I actually addressed a lot of issues and ask for
> > > > > > > clarification
> > > > > > > > > in the
> > > > > > > > > > > > original PR's way back when, but they're all just
> > > > > ignored.
> > > > > > > > > > > >
> > > > > > > > > > > > -Chris
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > >
> > > > > > >
> > > > >
> > >
>
