[DRE-maint] Bug#839807: unicorn: Missing depends on lsb-base

[Eric Wong]

(upstream here)

Fwiw, I prefer to make the init script itself optional;
especially since unicorn 5.x supports systemd and does not need
an init script.  I prefer fewer required dependencies
(suggests/recommends) can be fine.

Santiago Vila  wrote:
> Forgot the (trivial) patch. Here it is.

> +++ b/debian/control
> @@ -19,7 +19,8 @@ XS-Ruby-Versions: all
>  Package: unicorn
>  Architecture: kfreebsd-any linux-any
>  XB-Ruby-Versions: ${ruby:Versions}
> -Depends: ruby | ruby-interpreter,
> +Depends: lsb-base,
> + ruby | ruby-interpreter,

I assume lsb-base works for non-Linux (kFreeBSD) platforms?
(haven't checked in a while, but I hope to provide better
 non-Linux support in coming months).

___
Pkg-ruby-extras-maintainers mailing list
Pkg-ruby-extras-maintainers@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-ruby-extras-maintainers

[DRE-maint] Bug#830353: ruby-kgio: FTBFS: ERROR: Test "ruby2.3" failed.

[Eric Wong]

Antonio Terceiro <terce...@debian.org> wrote:
> On Fri, Jul 08, 2016 at 06:53:29PM +0000, Eric Wong wrote:
> > Thanks for the report.  Can you try the patch below?
> 
> Didn't help.
> 
> I can't, however, reproduce this locally on my workstation.

Thanks for following up.

> > I'd be interested to know what non-standard sysctl knobs are set
> > for kernel socket buffer sizes (in /proc/sys/net/{core,ipv4}/*mem*)
 
> ==> /proc/sys/net/ipv4/tcp_rmem <==
> 40961258291216777216
> 
> ==> /proc/sys/net/ipv4/tcp_wmem <==
> 40961258291216777216

Yikes at the gigantic default (middle) values.  Anyways, I've
set those locally and notice the problem with my previous patch:
I missed some spots where temporary strings/arrays were created
inside the tests themselves.

> In Debian we don't even have 2.2 anymore. The only things using kgio are
> unicorn and rainbows. Maybe if their dependency on kgio could be skipped on
> Ruby 2.3, we could drop kgio from Debian.

Yes, I still have to check and make sure I don't introduce
performance regressions and such when dropping kgio.
I still need to support Ruby 1.9.3 and 2.0.0 outside of Debian
(or on wheezy LTS/jessie)

The following ought to work (I tested locally with those sysctl
values) but I'll see about making the tests more robust against
larger kernel buffers.

---8<
--- a/test/lib_read_write.rb
+++ b/test/lib_read_write.rb
@@ -7,7 +7,12 @@ $-w = true
 require 'kgio'
 
 module LibReadWriteTest
-  RANDOM_BLOB = File.open("/dev/urandom") { |fp| fp.read(10 * 1024 * 1024) }
+  RANDOM_BLOB = File.open("/dev/urandom") do |fp|
+nr = 31
+buf = fp.read(nr)
+# get roughly a 20MB block of random data
+(buf * (20 * 1024 * 1024 / nr)) + (buf * rand(123))
+  end
 
   def teardown
 @rd.close if defined?(@rd) && ! @rd.closed?
@@ -369,7 +374,7 @@ module LibReadWriteTest
   @nr += 1
   IO.select(nil, [self])
 end
-buf = "." * 1024 * 1024 * 10
+buf = RANDOM_BLOB
 thr = Thread.new { @wr.kgio_write(buf) }
 Thread.pass until thr.stop?
 readed = @rd.read(buf.size)
@@ -385,7 +390,7 @@ module LibReadWriteTest
   @nr += 1
   IO.select(nil, [self])
 end
-buf = ["." * 1024] * 1024 * 10
+buf = [ RANDOM_BLOB, RANDOM_BLOB ]
 buf_size = buf.inject(0){|c, s| c + s.size}
 thr = Thread.new { @wr.kgio_writev(buf) }
 Thread.pass until thr.stop?
-- 
EW

___
Pkg-ruby-extras-maintainers mailing list
Pkg-ruby-extras-maintainers@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-ruby-extras-maintainers

[DRE-maint] Bug#827220: ruby-kgio FTBFS on hppa architecture

[Eric Wong]

Helge Deller  wrote:
> I'm somehow lost here now...
> Ideas?

Thanks, I'm lost here, too.  Anyways I've forwarded this
to ruby-core upstream:

https://bugs.ruby-lang.org/issues/12489

Unfortunately, I think posting to that bug tracker or replying to
Message-ID: 
on ruby-c...@ruby-lang.org requires a subscription or login;
so I'm happy to forward any messages along as needed.

I'm wondering if anybody previous experience porting similar
software to hppa could help.  Perhaps anything using a
mark-sweep GC like Boehm could help (if this is a GC bug in
Ruby).


Thanks.

___
Pkg-ruby-extras-maintainers mailing list
Pkg-ruby-extras-maintainers@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-ruby-extras-maintainers

[DRE-maint] Bug#827220: ruby-kgio FTBFS on hppa architecture

[Eric Wong]

Helge Deller <del...@gmx.de> wrote:
> On 13.06.2016 22:59, Eric Wong wrote:
> > Seems to be a ruby2.3 problem.
> > Does the ruby2.3 test suite work at all on hppa?
> 
> No idea (I'm no ruby expert).
> The last build log is here:
> https://buildd.debian.org/status/fetch.php?pkg=ruby2.3=hppa=2.3.1-2=1464615629
> 
> The log looks similar to x86_64, but I'm not sure if the test suite is run 
> then.

Looks like only the bootstrap test ("make test") is run.

That doesn't test a big chunk of functionality or standard library
at all.

The "test-all" target should also be run and is likely to find
more problems.

Can you give that a shot?

("make check" runs both test-all and test targets)

ruby-core actually has a good chunk of developers working on
Debian, but mainly x86/x86-64.

> > Any other ruby libraries with similar problems?
> 
> I think most ruby packages build sucessfully.
> In the package overview I find only 9 ruby packages which don't build: 
> https://buildd.debian.org/status/architecture.php?a=hppa=sid

Thanks, I'll check those at a later time.

___
Pkg-ruby-extras-maintainers mailing list
Pkg-ruby-extras-maintainers@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-ruby-extras-maintainers

[DRE-maint] Bug#827220: ruby-kgio FTBFS on hppa architecture

[Eric Wong]

Mattia Rizzolo  wrote:
> On Mon, Jun 13, 2016 at 10:40:51PM +0200, Helge Deller wrote:
> > Program received signal SIGSEGV, Segmentation fault.
> > [Switching to Thread 0xfa5dd480 (LWP 29258)]
> > mix_id_table_lookup (tbl=0x0, id=id@entry=3265, valp=valp@entry=0xfa55f248) 
> > at id_table.c:1516

tbl=0x0 looks like a bug in Ruby; possibly GC-related.

> > 1516id_table.c: No such file or directory.
> > (gdb) bt
> > #0  mix_id_table_lookup (tbl=0x0, id=id@entry=3265, 
> > valp=valp@entry=0xfa55f248) at id_table.c:1516
> > #1  0xf9c91b04 in lookup_method_table (id=3265, klass=) at 
> > vm_method.c:190
> > #2  search_method (defined_class_ptr=, id=, 
> > klass=) at vm_method.c:701
> > #3  method_entry_get_without_cache (defined_class_ptr=, 
> > id=, klass=) at vm_method.c:726
> > #4  method_entry_get (klass=1134408, id=3265, id@entry=1134408, 
> > defined_class_ptr=defined_class_ptr@entry=0xfa55f1c8) at vm_method.c:786
> > #5  0xf9c94e90 in rb_callable_method_entry (klass=, 
> > id=1134408, id@entry=4199936456) at vm_method.c:831
> > #6  0xf9ca6048 in rb_search_method_entry (mid=4199936456, recv=1134408) at 
> > vm_eval.c:573
> > #7  rb_call0 (recv=1134408, mid=4199936456, argc=3265, argv=0xfa55f248, 
> > scope=, self=) at vm_eval.c:343
> > #8  0xf9ca6984 in rb_call (scope=CALL_FCALL, argv=, 
> > argc=, mid=, recv=) at 
> > vm_eval.c:637
> > #9  rb_funcall (recv=, mid=, n= > out>) at vm_eval.c:835
> > #10 0x in ?? ()
> > Backtrace stopped: previous frame identical to this frame (corrupt stack?)
> > 
> > I assume it's a problem in ruby2.3 itself, but since I'm no expert in 
> > ruby2.3, is there anyone who might be able to help?
> > I can provide a login to a hppa box if necessary...

Seems to be a ruby2.3 problem.
Does the ruby2.3 test suite work at all on hppa?
Any other ruby libraries with similar problems?
Thanks.

(I'm kgio upstream, and occasional ruby-core committer).

___
Pkg-ruby-extras-maintainers mailing list
Pkg-ruby-extras-maintainers@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-ruby-extras-maintainers

[DRE-maint] [PATCH v2] emulate sd_listen_fds for systemd support

[Eric Wong]

systemd socket emulation shares FDs across execve, just like
the built-in SIGUSR2 upgrade process in unicorn.  Thus it is
easy to support inheriting sockets from systemd.

Tested-by: Christos Trochalakis yati...@ideopolis.gr
---
  Christos Trochalakis yati...@ideopolis.gr wrote:
   On Sat, Jun 27, 2015 at 04:01:36AM +, Eric Wong wrote:
   Unfortunately, testing this is tricky because FD=3
   (SD_LISTEN_FDS_START) tends to be grabbed by (MRI) Ruby 1.9.3
   and onwards for the internal self-pipe.

  OK, that was a bogus note.
  I forgot exec() in Ruby 1.9+ supports redirects natively.
  Test included in updated patch.

   I have also tested it and works as expected. Also, hardcoding
   SD_LISTEN_FDS_START seems like the best option. I 'd like to see that
   applied to master.

  Thanks.  This will be in 5.0.0.pre2

 lib/unicorn/http_server.rb | 16 +---
 test/exec/test_exec.rb | 24 
 2 files changed, 37 insertions(+), 3 deletions(-)

diff --git a/lib/unicorn/http_server.rb b/lib/unicorn/http_server.rb
index 297b9f9..0f97516 100644
--- a/lib/unicorn/http_server.rb
+++ b/lib/unicorn/http_server.rb
@@ -766,12 +766,22 @@ class Unicorn::HttpServer
   def inherit_listeners!
 # inherit sockets from parents, they need to be plain Socket objects
 # before they become Kgio::UNIXServer or Kgio::TCPServer
-inherited = ENV['UNICORN_FD'].to_s.split(',').map do |fd|
-  io = Socket.for_fd(fd.to_i)
+inherited = ENV['UNICORN_FD'].to_s.split(',')
+
+# emulate sd_listen_fds() for systemd
+sd_pid, sd_fds = ENV.values_at('LISTEN_PID', 'LISTEN_FDS')
+if sd_pid  sd_pid.to_i == $$
+  # 3 = SD_LISTEN_FDS_START
+  inherited.concat((3...(3 + sd_fds.to_i)).map { |fd| Socket.for_fd(fd) })
+end
+# to ease debugging, we will not unset LISTEN_PID and LISTEN_FDS
+
+inherited.map! do |fd|
+  io = String === fd ? Socket.for_fd(fd.to_i) : fd
   io.autoclose = false
   io = server_cast(io)
   set_server_sockopt(io, listener_opts[sock_name(io)])
-  logger.info inherited addr=#{sock_name(io)} fd=#{fd}
+  logger.info inherited addr=#{sock_name(io)} fd=#{io.fileno}
   io
 end
 
diff --git a/test/exec/test_exec.rb b/test/exec/test_exec.rb
index 6deb96b..af8de26 100644
--- a/test/exec/test_exec.rb
+++ b/test/exec/test_exec.rb
@@ -96,6 +96,30 @@ run lambda { |env|
 end
   end
 
+  def test_sd_listen_fds_emulation
+File.open(config.ru, wb) { |fp| fp.write(HI) }
+sock = TCPServer.new(@addr, @port)
+sock.setsockopt(:SOL_SOCKET, :SO_KEEPALIVE, 0)
+
+pid = xfork do
+  redirect_test_io do
+# pretend to be systemd
+ENV['LISTEN_PID'] = #$$
+ENV['LISTEN_FDS'] = '1'
+
+# 3 = SD_LISTEN_FDS_START
+exec($unicorn_bin, -l, #@addr:#@port, 3 = sock)
+  end
+end
+res = hit([http://#{@addr}:#{@port}/;])
+assert_equal [ HI\n], res
+assert_shutdown(pid)
+assert_equal 1, sock.getsockopt(:SOL_SOCKET, :SO_KEEPALIVE).int,
+ unicorn should always set SO_KEEPALIVE on inherited sockets
+  ensure
+sock.close if sock
+  end
+
   def test_working_directory_rel_path_config_file
 other = Tempfile.new('unicorn.wd')
 File.unlink(other.path)
-- 
EW

___
Pkg-ruby-extras-maintainers mailing list
Pkg-ruby-extras-maintainers@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-ruby-extras-maintainers

Re: [DRE-maint] unicorn: native systemd service

[Eric Wong]

Christos Trochalakis yati...@ideopolis.gr wrote:
 On Sat, Jun 27, 2015 at 03:05:01AM +, Eric Wong wrote:
 Christos Trochalakis yati...@ideopolis.gr wrote:
 KillMode=mixed
 
 I don't think KillMode=mixed is necessary, here.  systemd can send
 SIGQUIT to workers.
 
 
 Perhaps there is a race here, if a worker receives SIGQUIT first, and the
 master respawns a new worker before receiving/handling its own SIGQUIT.
 This is definitely a longshot, and will probably never happen, but
 even then every process will eventually quit gracefully.

Right. This race is possible, but workers will automatically die if
they don't have a master.
Easier just to kill the master, of course.

 Is it possible to make systemd fire up two unicorn masters?
 That would be a nice feature to have with socket activation.
 
 That would be great! I am a bit surprised that specifying multiple
 services ('Service=' option) is not supported.

Can you get this feature added to systemd?

___
Pkg-ruby-extras-maintainers mailing list
Pkg-ruby-extras-maintainers@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-ruby-extras-maintainers

[DRE-maint] [RFC] emulate sd_listen_fds for systemd support

[Eric Wong]

Unfortunately, testing this is tricky because FD=3
(SD_LISTEN_FDS_START) tends to be grabbed by (MRI) Ruby 1.9.3
and onwards for the internal self-pipe.

So for now, I've manually tested this with a systemd installation

Disclaimer: this is also my first experience using systemd
---
  Eric Wong e...@80x24.org wrote:
   OK, I'll probably add LISTEN_FDS and LISTEN_PID support to unicorn
   directly so the wrapper is unnecessary.

 lib/unicorn/http_server.rb | 16 +---
 1 file changed, 13 insertions(+), 3 deletions(-)

diff --git a/lib/unicorn/http_server.rb b/lib/unicorn/http_server.rb
index 297b9f9..0f97516 100644
--- a/lib/unicorn/http_server.rb
+++ b/lib/unicorn/http_server.rb
@@ -766,12 +766,22 @@ def redirect_io(io, path)
   def inherit_listeners!
 # inherit sockets from parents, they need to be plain Socket objects
 # before they become Kgio::UNIXServer or Kgio::TCPServer
-inherited = ENV['UNICORN_FD'].to_s.split(',').map do |fd|
-  io = Socket.for_fd(fd.to_i)
+inherited = ENV['UNICORN_FD'].to_s.split(',')
+
+# emulate sd_listen_fds() for systemd
+sd_pid, sd_fds = ENV.values_at('LISTEN_PID', 'LISTEN_FDS')
+if sd_pid  sd_pid.to_i == $$
+  # 3 = SD_LISTEN_FDS_START
+  inherited.concat((3...(3 + sd_fds.to_i)).map { |fd| Socket.for_fd(fd) })
+end
+# to ease debugging, we will not unset LISTEN_PID and LISTEN_FDS
+
+inherited.map! do |fd|
+  io = String === fd ? Socket.for_fd(fd.to_i) : fd
   io.autoclose = false
   io = server_cast(io)
   set_server_sockopt(io, listener_opts[sock_name(io)])
-  logger.info inherited addr=#{sock_name(io)} fd=#{fd}
+  logger.info inherited addr=#{sock_name(io)} fd=#{io.fileno}
   io
 end
 
-- 
EW

___
Pkg-ruby-extras-maintainers mailing list
Pkg-ruby-extras-maintainers@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-ruby-extras-maintainers

Re: [DRE-maint] unicorn: native systemd service

[Eric Wong]

Christos Trochalakis yati...@ideopolis.gr wrote:
 On Thu, Jun 25, 2015 at 11:26:26PM +, Eric Wong wrote:
 With socket activation, you should just be able to kill unicorn using
 SIGQUIT (just master, or even all workers) and restart without ever
 dropping a connection.  I do NOT suggest using SIGTERM for unicorn,
 since that'll cause the master to kill all workers ASAP.
 
 Yes, you are right socket activation is also an option! I have made some
 experiments with a simple rack app to test it.
 
 systemd uses the LISTEN_FDS env variable that is an integer indicating the
 number of inherited file descriptors. Those FDs have consecutive numbers
 starting from `SD_LISTEN_FDS_START` which is `3` (man sd_listen_fds).
 
 So for example if LISTEN_FDS=2, UNICORN_FD should be 3,4. I used a
 simple wrapper script for that. Here is the full configuration:

OK, I'll probably add LISTEN_FDS and LISTEN_PID support to unicorn
directly so the wrapper is unnecessary.

snip

 KillMode=mixed

I don't think KillMode=mixed is necessary, here.  systemd can send
SIGQUIT to workers.

snip

 TCP socket options are not applied by unicorn on inherited sockets (TCPSocket
 === sock is false). systemd socket files have support for most options now but
 we might want unicorn to `setsockopt` them as well. For example,
 'DeferAcceptSec', 'KeepAliveIntervalSec', 'NoDelay' are supported since v216, 
 so
 they are not available in jessie (v215).

They are now :)

http://bogomips.org/unicorn-public/m/1435373879-4299-1-git-send-emai...@80x24.org.txt

We don't have KeepAliveIntervalSec
(aka TCP_KEEPINTVL) since it's not-portable and probably over
Are you sure about that?

 socket activation is a really interesting setup, but personally I would not 
 run
 it with a large application. Clients would have to wait for the new master to
 be up and running before a reply is returned, and that could take tenths of
 seconds. The USR2 rexec solves that problem since both old and new workers are
 accepting on the socket and we can kill the old ones when we are ready. In 
 that
 case the PIDFile trick is handy to support zero downtime restarts with no
 latency.

Is it possible to make systemd fire up two unicorn masters?
That would be a nice feature to have with socket activation.

___
Pkg-ruby-extras-maintainers mailing list
Pkg-ruby-extras-maintainers@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-ruby-extras-maintainers

Re: [DRE-maint] unicorn: native systemd service

[Eric Wong]

+Cc unicorn-public list 
Christos Trochalakis chris...@skroutz.gr wrote:
 Hello all,
 
 I have recently migrated our main ruby application to systemd implementing 
 zero
 downtime upgrades.
 
 systemd doesn't like replacing the binary on the fly. There is one exception 
 to
 this, services with PIDFile. When PIDFile is set, systemd reads it when the
 main process exits and replaces the main process.  nginx also had this issue a
 few months ago [0].
 
 So, in order to support zero-downtime upgrades we have to use a pid file.

I don't think so.  You should be able to bind the listen socket in
systemd and rely on the socket activation features (setting the
UNICORN_FD environment variable to the created FD).

You still need to have the matching listen directive in the unicorn
config file so unicorn does not close it.

With socket activation, you should just be able to kill unicorn using
SIGQUIT (just master, or even all workers) and restart without ever
dropping a connection.  I do NOT suggest using SIGTERM for unicorn,
since that'll cause the master to kill all workers ASAP.

If the master dies before the workers (e.g. from SIGKILL),
systemd ought to just ignore the workers.

I don't actually know the systemd config at all, but anybody familiar
with systemd socket activation should have no problems setting the
UNICORN_FD env.

___
Pkg-ruby-extras-maintainers mailing list
Pkg-ruby-extras-maintainers@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-ruby-extras-maintainers

Re: [DRE-maint] unicorn: native systemd service

[Eric Wong]

Dmitry Smirnov only...@debian.org wrote:
 On Wed, 24 Jun 2015 23:26:49 Eric Wong wrote:
  Dmitry: unicorn upstream here, is there anything in unicorn itself
  can do to make systemd integration easier?
 
 Thank you very much for keeping an eye on us and for all your help.
 I'm not sure if we need anything special -- let me experiment more with soft 
 restart and maybe I'll come back to you with questions. :)

No problem!  I basically live and breathe plain-text email,
so Debian is a good fit for me :)

  I haven't had much interest or time to dig into systemd, but I'm willing
  to put some effort into making unicorn work more smoothly with systemd
  (and any other Free Software process managers/init systems).
 
 Great. I'm entertaining idea of modifying init scripts to make soft restart 
 the default behaviour. Is there are any risks or drawbacks of always 
 attempting soft restart instead of normal restart? What do you think?

Probably not...  I almost regret making SIGTERM behave like nginx
instead of making SIGTERM identical to SIGQUIT.  OTOH, matching
nginx might make learning to use nginx (after learning unicorn signals)
easier.

I've certainly never used SIGTERM on unicorn or nginx myself.
Occasionally I've managed to break the RubyVM enough to require SIGKILL,
but I think using SIGTERM is unnecessary other than as training wheels
for nginx.

___
Pkg-ruby-extras-maintainers mailing list
Pkg-ruby-extras-maintainers@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-ruby-extras-maintainers

Re: [DRE-maint] unicorn: native systemd service

[Eric Wong]

Adding unicorn-pub...@bogomips.org to Cc:

Those of you who haven't followed along on the unic...@packages.debian.org
list can catch up here:
http://lists.alioth.debian.org/pipermail/pkg-ruby-extras-maintainers/2015-June/024539.html

Dmitry: unicorn upstream here, is there anything in unicorn itself
can do to make systemd integration easier?

I haven't had much interest or time to dig into systemd, but I'm willing
to put some effort into making unicorn work more smoothly with systemd
(and any other Free Software process managers/init systems).

So far unicorn.git has documented the UNICORN_FD environment variable in
the manpage in commit 548e1e67d314f6ebd17df37ece0ee20632462f6f [1]
and support SIGWINCH (kill children) in
commit a6077391bb62d0b13016084b0eea36b987afe8f0 [2]

I think that should be sufficient, but maybe there's more.
Thanks.

[1] http://bogomips.org/unicorn.git/patch/?id=548e1e67
[2] http://bogomips.org/unicorn.git/patch/?id=a6077391

___
Pkg-ruby-extras-maintainers mailing list
Pkg-ruby-extras-maintainers@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-ruby-extras-maintainers

[DRE-maint] Bug#735861: [ruby-core:61553] [Backport200 - Backport #9649] [Open] Segfault running thread_safe torture tests under Ruby 2.0

[Eric Wong]

This is fixed in Ruby 2.0.0 p451, seems related to
https://bugs.ruby-lang.org/issues/9178

___
Pkg-ruby-extras-maintainers mailing list
Pkg-ruby-extras-maintainers@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-ruby-extras-maintainers