Re: subsequent rebuilds of notmuch always re-build sphinx and ruby

2019-04-23 Thread Daniel Kahn Gillmor
On Tue 2019-04-23 22:09:23 -0300, David Bremner wrote:
> [dkg wrote:]
>> I do note that (independent of this series), if i run the following
>> loop:
>>
>>
>>  while make -j4 --trace; do
>> python3 -c 'print("="*100)'
>> touch doc/man1/notmuch-reply.rst
>>  done
>>
>> then only every second run of make contains this info:
>
> I think I see this also, but no idea yet what is going on.

I've tagged the previous mail with notmuch::bug so that we can keep
track of it, but i don't think it's a high priority.

thanks for having looked into it!

   --dkg
___
notmuch mailing list
notmuch@notmuchmail.org
https://notmuchmail.org/mailman/listinfo/notmuch


Re: subsequent rebuilds of notmuch always re-build sphinx and ruby

2019-04-23 Thread David Bremner
Daniel Kahn Gillmor  writes:

> On Mon 2019-04-22 21:03:05 -0300, David Bremner wrote:
>> There was a problem with the first patch, which I replaced with two more.
>
> thanks.  i've reviewed and published my review on that series.  I think
> it should probably be merged.
>
>> I'm open to ideas, but keep in mind we want to support parallel make,
>> which means we have to be careful not to trigger multiple invocations of
>> sphinx-build in parallel.

> hm, i'm not entirely sure why sphinx-build can't be run in parallel, if
> it could target the creation of specific files (but maybe it can't).

It can target specific files according to the documentation, but the
main issue is that it caches a bunch of state under
doc/_build/doctrees. It doesn't do any kind of locking, so multiple
writers leads to build failures.

>
> I do note that (independent of this series), if i run the following
> loop:
>
>
>  while make -j4 --trace; do
> python3 -c 'print("="*100)'
> touch doc/man1/notmuch-reply.rst
>  done
>
> then only every second run of make contains this info:
>

I think I see this also, but no idea yet what is going on.
___
notmuch mailing list
notmuch@notmuchmail.org
https://notmuchmail.org/mailman/listinfo/notmuch


Re: subsequent rebuilds of notmuch always re-build sphinx and ruby

2019-04-23 Thread Daniel Kahn Gillmor
On Mon 2019-04-22 21:03:05 -0300, David Bremner wrote:
> There was a problem with the first patch, which I replaced with two more.

thanks.  i've reviewed and published my review on that series.  I think
it should probably be merged.

> I'm open to ideas, but keep in mind we want to support parallel make,
> which means we have to be careful not to trigger multiple invocations of
> sphinx-build in parallel.

hm, i'm not entirely sure why sphinx-build can't be run in parallel, if
it could target the creation of specific files (but maybe it can't).

I do note that (independent of this series), if i run the following
loop:


 while make -j4 --trace; do
python3 -c 'print("="*100)'
touch doc/man1/notmuch-reply.rst
 done

then only every second run of make contains this info:


doc/Makefile.local:38: update target 'doc/_build/man/man1/notmuch-reply.1.gz' 
due to: doc/_build/man/man1/notmuch-reply.1
rm -f doc/_build/man/man1/notmuch-reply.1.gz && gzip --stdout 
doc/_build/man/man1/notmuch-reply.1 > doc/_build/man/man1/notmuch-reply.1.gz
doc/Makefile.local:38: update target 'doc/_build/man/man1/notmuch-reindex.1.gz' 
due to: doc/_build/man/man1/notmuch-reindex.1
rm -f doc/_build/man/man1/notmuch-reindex.1.gz && gzip --stdout 
doc/_build/man/man1/notmuch-reindex.1 > doc/_build/man/man1/notmuch-reindex.1.gz
doc/Makefile.local:38: update target 'doc/_build/man/man1/notmuch-restore.1.gz' 
due to: doc/_build/man/man1/notmuch-restore.1
rm -f doc/_build/man/man1/notmuch-restore.1.gz && gzip --stdout 
doc/_build/man/man1/notmuch-restore.1 > doc/_build/man/man1/notmuch-restore.1.gz
doc/Makefile.local:38: update target 'doc/_build/man/man1/notmuch-new.1.gz' due 
to: doc/_build/man/man1/notmuch-new.1
rm -f doc/_build/man/man1/notmuch-new.1.gz && gzip --stdout 
doc/_build/man/man1/notmuch-new.1 > doc/_build/man/man1/notmuch-new.1.gz
doc/Makefile.local:38: update target 'doc/_build/man/man1/notmuch-dump.1.gz' 
due to: doc/_build/man/man1/notmuch-dump.1
rm -f doc/_build/man/man1/notmuch-dump.1.gz && gzip --stdout 
doc/_build/man/man1/notmuch-dump.1 > doc/_build/man/man1/notmuch-dump.1.gz
doc/Makefile.local:38: update target 'doc/_build/man/man1/notmuch-address.1.gz' 
due to: doc/_build/man/man1/notmuch-address.1
rm -f doc/_build/man/man1/notmuch-address.1.gz && gzip --stdout 
doc/_build/man/man1/notmuch-address.1 > doc/_build/man/man1/notmuch-address.1.gz
doc/Makefile.local:38: update target 'doc/_build/man/man1/notmuch-tag.1.gz' due 
to: doc/_build/man/man1/notmuch-tag.1
rm -f doc/_build/man/man1/notmuch-tag.1.gz && gzip --stdout 
doc/_build/man/man1/notmuch-tag.1 > doc/_build/man/man1/notmuch-tag.1.gz
doc/Makefile.local:38: update target 'doc/_build/man/man1/notmuch-count.1.gz' 
due to: doc/_build/man/man1/notmuch-count.1
rm -f doc/_build/man/man1/notmuch-count.1.gz && gzip --stdout 
doc/_build/man/man1/notmuch-count.1 > doc/_build/man/man1/notmuch-count.1.gz
doc/Makefile.local:38: update target 'doc/_build/man/man1/notmuch-search.1.gz' 
due to: doc/_build/man/man1/notmuch-search.1
rm -f doc/_build/man/man1/notmuch-search.1.gz && gzip --stdout 
doc/_build/man/man1/notmuch-search.1 > doc/_build/man/man1/notmuch-search.1.gz
doc/Makefile.local:38: update target 
'doc/_build/man/man1/notmuch-emacs-mua.1.gz' due to: 
doc/_build/man/man1/notmuch-emacs-mua.1
rm -f doc/_build/man/man1/notmuch-emacs-mua.1.gz && gzip --stdout 
doc/_build/man/man1/notmuch-emacs-mua.1 > 
doc/_build/man/man1/notmuch-emacs-mua.1.gz
doc/Makefile.local:38: update target 'doc/_build/man/man1/notmuch-show.1.gz' 
due to: doc/_build/man/man1/notmuch-show.1
rm -f doc/_build/man/man1/notmuch-show.1.gz && gzip --stdout 
doc/_build/man/man1/notmuch-show.1 > doc/_build/man/man1/notmuch-show.1.gz
doc/Makefile.local:38: update target 'doc/_build/man/man1/notmuch-insert.1.gz' 
due to: doc/_build/man/man1/notmuch-insert.1
rm -f doc/_build/man/man1/notmuch-insert.1.gz && gzip --stdout 
doc/_build/man/man1/notmuch-insert.1 > doc/_build/man/man1/notmuch-insert.1.gz
doc/Makefile.local:38: update target 'doc/_build/man/man1/notmuch.1.gz' due to: 
doc/_build/man/man1/notmuch.1
rm -f doc/_build/man/man1/notmuch.1.gz && gzip --stdout 
doc/_build/man/man1/notmuch.1 > doc/_build/man/man1/notmuch.1.gz
doc/Makefile.local:38: update target 'doc/_build/man/man1/notmuch-compact.1.gz' 
due to: doc/_build/man/man1/notmuch-compact.1
rm -f doc/_build/man/man1/notmuch-compact.1.gz && gzip --stdout 
doc/_build/man/man1/notmuch-compact.1 > doc/_build/man/man1/notmuch-compact.1.gz
doc/Makefile.local:38: update target 'doc/_build/man/man1/notmuch-config.1.gz' 
due to: doc/_build/man/man1/notmuch-config.1
rm -f doc/_build/man/man1/notmuch-config.1.gz && gzip --stdout 
doc/_build/man/man1/notmuch-config.1 > doc/_build/man/man1/notmuch-config.1.gz
doc/Makefile.local:38: update target 'doc/_build/man/man5/notmuch-hooks.5.gz' 
due to: doc/_build/man/man5/notmuch-hooks.5
rm -f doc/_build/man/man5/notmuch-hooks.5.gz && gzip --stdout 
doc/_build/man/man5/notmuch-hooks.5 > 

Re: subsequent rebuilds of notmuch always re-build sphinx and ruby

2019-04-22 Thread David Bremner
Daniel Kahn Gillmor  writes:

> On Sun 2019-04-21 16:29:02 -0300, David Bremner wrote:
>> the html rebuild is much faster than the texinfo + info rebuilds.
>
> agreed, in the runs that i've been doing as well.  I was concerned that
> the html rebuild itself may have been *triggering* the rebuild of the
> texinfo stuff, though.  Sounds like you don't think that's the case.
>
>> I've posted some patches for the sphinx-doc issues a couple of hours ago
>> (id:20190421171245.19729-1-da...@tethera.net). 
>
> thanks!  your own commentary on that series seems to acknowledge that
> there are problems with it (though i don't understand the tradeoffs
> well).

There was a problem with the first patch, which I replaced with two more.


> Is there no way to give make itself full visibility into the specific
> generated files so it can do its comparisons directly?  I'm obviously
> not asking you to rewrite the entire native sphinx build system, i'm
> just observing that at present it seems suboptimal, though i don't know
> how to fix it either :/

I'm open to ideas, but keep in mind we want to support parallel make,
which means we have to be careful not to trigger multiple invocations of
sphinx-build in parallel.
___
notmuch mailing list
notmuch@notmuchmail.org
https://notmuchmail.org/mailman/listinfo/notmuch


Re: subsequent rebuilds of notmuch always re-build sphinx and ruby

2019-04-22 Thread Daniel Kahn Gillmor
On Sun 2019-04-21 16:29:02 -0300, David Bremner wrote:
> the html rebuild is much faster than the texinfo + info rebuilds.

agreed, in the runs that i've been doing as well.  I was concerned that
the html rebuild itself may have been *triggering* the rebuild of the
texinfo stuff, though.  Sounds like you don't think that's the case.

> I've posted some patches for the sphinx-doc issues a couple of hours ago
> (id:20190421171245.19729-1-da...@tethera.net). 

thanks!  your own commentary on that series seems to acknowledge that
there are problems with it (though i don't understand the tradeoffs
well).  i'm not super comfortable with make-style "stamping": my
experience with it is that it creates a synchronization problem, and
it's easy for the "sync" between the stamp and the generated artifacts
to break, at which point the safest thing is to "make clean" and start
fully over.

Is there no way to give make itself full visibility into the specific
generated files so it can do its comparisons directly?  I'm obviously
not asking you to rewrite the entire native sphinx build system, i'm
just observing that at present it seems suboptimal, though i don't know
how to fix it either :/

> Currently the ruby rebuild doesn't seem to be slowing things down much
> for me.
>
> ╭─ convex:~/software/upstream/notmuch 
> ╰─ (git)-[wip/make-docs]-% /usr/bin/time make ruby-bindings 1>/dev/null
> 0.13user 0.02system 0:00.15elapsed 101%CPU (0avgtext+0avgdata 
> 11504maxresident)k
> 0inputs+208outputs (0major+6523minor)pagefaults 0swaps
>
> That's with an SSD, so maybe there's more of hit for other environments.

I agree, this one isn't particularly slow, and it appears to be a leaf
dependency (for now), so it's not the worst thing.  But it's still
pretty clearly a bug that this thing loops.

   --dkg


signature.asc
Description: PGP signature
___
notmuch mailing list
notmuch@notmuchmail.org
https://notmuchmail.org/mailman/listinfo/notmuch


Re: subsequent rebuilds of notmuch always re-build sphinx and ruby

2019-04-21 Thread David Bremner
Daniel Kahn Gillmor  writes:

> but it's not just texinfo, right?  it starts with the html build
> itself.  can we at least diagnose why that's happening?
>

Yes, although the html rebuild is much faster than the texinfo + info rebuilds.

>> This Makefile is generated by "ruby extconf.rb --vendor". It includes a
>> dependency on itself, so it always fires after running "ruby
>> extconf.rb". It might be only running "ruby extconf.rb" if
>> bindings/ruby/Makefile does not exist would fix this particular
>> issue. That sounds more gnu make specific than ruby specific.

> I'd be happy to test any proposed patches.  I don't really understand
> this toolchain, or why anyone would build a makefile that rewrites
> itself :/

I've posted some patches for the sphinx-doc issues a couple of hours ago
(id:20190421171245.19729-1-da...@tethera.net). 

Currently the ruby rebuild doesn't seem to be slowing things down much
for me.

╭─ convex:~/software/upstream/notmuch 
╰─ (git)-[wip/make-docs]-% /usr/bin/time make ruby-bindings 1>/dev/null
0.13user 0.02system 0:00.15elapsed 101%CPU (0avgtext+0avgdata 11504maxresident)k
0inputs+208outputs (0major+6523minor)pagefaults 0swaps

That's with an SSD, so maybe there's more of hit for other environments.

d
___
notmuch mailing list
notmuch@notmuchmail.org
https://notmuchmail.org/mailman/listinfo/notmuch


Re: subsequent rebuilds of notmuch always re-build sphinx and ruby

2019-04-21 Thread Daniel Kahn Gillmor
thanks for the review, Bremner!

On Sat 2019-04-20 21:12:07 -0300, David Bremner wrote:
> Daniel Kahn Gillmor  writes:
>> 0 dkg@alice:~/src/notmuch/notmuch$ ./configure && make
>>  […]
>> make[1]: Leaving directory '/home/dkg/src/notmuch/notmuch/bindings/ruby'
>> 0 dkg@alice:~/src/notmuch/notmuch$ make --trace
>> doc/Makefile.local:53: update target 'sphinx-html' due to: docstring.stamp
>> sphinx-build -b html -d doc/_build/doctrees -q ./doc doc/_build/html
>> doc/Makefile.local:56: update target 'sphinx-texinfo' due to: docstring.stamp
>> sphinx-build -b texinfo -d doc/_build/doctrees -q ./doc doc/_build/texinfo
>> doc/Makefile.local:59: update target 'sphinx-info' due to: sphinx-texinfo
>> make -C doc/_build/texinfo info
>> make[1]: Entering directory 
>> '/home/dkg/src/notmuch/notmuch/doc/_build/texinfo'
>> Makefile:32: update target 'notmuch-search-terms.info' due to: 
>> notmuch-search-terms.texi
>
> This is not our Makefile, but something generated by sphinx; it would
> not be that hard to replace if the problem was there. Alas I think the
> underlying problem seems to be that "sphinx-build -b texinfo" is
> regenerating (or at least touching) all of the texi files. I suspect
> that's a limitation of sphinx-builder texinfo output.

but it's not just texinfo, right?  it starts with the html build
itself.  can we at least diagnose why that's happening?

>> cd bindings/ruby && \
>>  EXTRA_LDFLAGS="-Wl,--no-undefined" \
>>  LIBNOTMUCH="../../lib/libnotmuch.so" \
>>  NOTMUCH_SRCDIR='/home/dkg/src/notmuch/notmuch' \
>>  ruby extconf.rb --vendor
>> creating Makefile
>> make -C bindings/ruby
>> make[1]: Entering directory '/home/dkg/src/notmuch/notmuch/bindings/ruby'
>> Makefile:258: update target 'notmuch.so' due to: Makefile
>> echo linking shared-object notmuch.so
>> linking shared-object notmuch.so
>> rm -f notmuch.so
>> gcc -shared -o notmuch.so database.o directory.o filenames.o init.o 
>> message.o messages.o query.o status.o tags.o thread.o threads.o -L. 
>> -L/usr/lib/x86_64-linux-gnu -L. -Wl,-z,relro -Wl,-z,now -fstack-protector 
>> -rdynamic -Wl,-export-dynamic -Wl,--no-undefined -Wl,-z,relro -Wl,-z,now 
>> -Wl,--compress-debug-sections=zlib   ../../lib/libnotmuch.so -lruby-2.5  
>> -lpthread -lgmp -ldl -lcrypt -lm   -lc
>> make[1]: Leaving directory '/home/dkg/src/notmuch/notmuch/bindings/ruby'
>> 0 dkg@alice:~/src/notmuch/notmuch$
>
> This Makefile is generated by "ruby extconf.rb --vendor". It includes a
> dependency on itself, so it always fires after running "ruby
> extconf.rb". It might be only running "ruby extconf.rb" if
> bindings/ruby/Makefile does not exist would fix this particular
> issue. That sounds more gnu make specific than ruby specific.

I'd be happy to test any proposed patches.  I don't really understand
this toolchain, or why anyone would build a makefile that rewrites
itself :/

   --dkg


signature.asc
Description: PGP signature
___
notmuch mailing list
notmuch@notmuchmail.org
https://notmuchmail.org/mailman/listinfo/notmuch


Re: subsequent rebuilds of notmuch always re-build sphinx and ruby

2019-04-20 Thread David Bremner
Daniel Kahn Gillmor  writes:

> Hi folks--
>
> when i run "make" from my source tree, and it succeeds, i typically
> expect that running "make" again will show that nothing needs to be
> done.
>
> but that's not the case.  Both the sphinx-based documentation and the
> ruby notmuch.so are always rebuilt, i think due to some sort of
> dependency loop.
>
> But i don't really understand the dependencies there.  Maybe someone who
> understands either ruby or sphinx better than i do can clean it up?
> Having clean dependency tracking and quick rebuilds makes a project a
> lot more fun to hack on.
>
> Here's a trace of the rebuild process:
>
> 0 dkg@alice:~/src/notmuch/notmuch$ ./configure && make
>  […]
> make[1]: Leaving directory '/home/dkg/src/notmuch/notmuch/bindings/ruby'
> 0 dkg@alice:~/src/notmuch/notmuch$ make --trace
> doc/Makefile.local:53: update target 'sphinx-html' due to: docstring.stamp
> sphinx-build -b html -d doc/_build/doctrees -q ./doc doc/_build/html
> doc/Makefile.local:56: update target 'sphinx-texinfo' due to: docstring.stamp
> sphinx-build -b texinfo -d doc/_build/doctrees -q ./doc doc/_build/texinfo
> doc/Makefile.local:59: update target 'sphinx-info' due to: sphinx-texinfo
> make -C doc/_build/texinfo info
> make[1]: Entering directory '/home/dkg/src/notmuch/notmuch/doc/_build/texinfo'
> Makefile:32: update target 'notmuch-search-terms.info' due to: 
> notmuch-search-terms.texi

This is not our Makefile, but something generated by sphinx; it would
not be that hard to replace if the problem was there. Alas I think the
underlying problem seems to be that "sphinx-build -b texinfo" is
regenerating (or at least touching) all of the texi files. I suspect
that's a limitation of sphinx-builder texinfo output.

> cd bindings/ruby && \
>   EXTRA_LDFLAGS="-Wl,--no-undefined" \
>   LIBNOTMUCH="../../lib/libnotmuch.so" \
>   NOTMUCH_SRCDIR='/home/dkg/src/notmuch/notmuch' \
>   ruby extconf.rb --vendor
> creating Makefile
> make -C bindings/ruby
> make[1]: Entering directory '/home/dkg/src/notmuch/notmuch/bindings/ruby'
> Makefile:258: update target 'notmuch.so' due to: Makefile
> echo linking shared-object notmuch.so
> linking shared-object notmuch.so
> rm -f notmuch.so
> gcc -shared -o notmuch.so database.o directory.o filenames.o init.o message.o 
> messages.o query.o status.o tags.o thread.o threads.o -L. 
> -L/usr/lib/x86_64-linux-gnu -L. -Wl,-z,relro -Wl,-z,now -fstack-protector 
> -rdynamic -Wl,-export-dynamic -Wl,--no-undefined -Wl,-z,relro -Wl,-z,now 
> -Wl,--compress-debug-sections=zlib   ../../lib/libnotmuch.so -lruby-2.5  
> -lpthread -lgmp -ldl -lcrypt -lm   -lc
> make[1]: Leaving directory '/home/dkg/src/notmuch/notmuch/bindings/ruby'
> 0 dkg@alice:~/src/notmuch/notmuch$

This Makefile is generated by "ruby extconf.rb --vendor". It includes a
dependency on itself, so it always fires after running "ruby
extconf.rb". It might be only running "ruby extconf.rb" if
bindings/ruby/Makefile does not exist would fix this particular
issue. That sounds more gnu make specific than ruby specific.

d



___
notmuch mailing list
notmuch@notmuchmail.org
https://notmuchmail.org/mailman/listinfo/notmuch