Bug#972702: ruby-bundler: dependency resolution fails for compiled gems
On Thu, Oct 29, 2020 at 01:21:03PM +0100, David Rodríguez wrote: > El 29/10/20 a las 13:14, Antonio Terceiro escribió: > > > On Thu, Oct 29, 2020 at 11:17:59AM +0100, David Rodríguez wrote: > > > Hi! > > > > > > Just to clarify why I prefer the second solution, I think what debian > > > does is shipping precompiled versions of extensions, so technically the > > > gemspec shipped in the debian should include no extensions at all. This > > > is something some upstream gems already do. Take, for example, > > > google-protobuf. It has a precompiled version for linux: > > > https://rubygems.org/gems/google-protobuf/versions/3.13.0-x86_64-linux. > > > If we fetch and unpack this package, we can see it includes the prebuilt > > > `.so` extension, but no extensions in its gemspec: > > > > > > $ gem fetch google-protobuf > > > Fetching google-protobuf-3.13.0-x86_64-linux.gem > > > Downloaded google-protobuf-3.13.0-x86_64-linux > > > > > > $ gem unpack google-protobuf-3.13.0-x86_64-linux.gem > > > Unpacked gem: > > > '/home/deivid/Code/playground/google-protobuf-3.13.0-x86_64-linux' > > > > > > $ find google-protobuf-3.13.0-x86_64-linux -name '*.so' > > > google-protobuf-3.13.0-x86_64-linux/lib/google/2.6/protobuf_c.so > > > google-protobuf-3.13.0-x86_64-linux/lib/google/2.4/protobuf_c.so > > > google-protobuf-3.13.0-x86_64-linux/lib/google/2.7/protobuf_c.so > > > google-protobuf-3.13.0-x86_64-linux/lib/google/2.5/protobuf_c.so > > > google-protobuf-3.13.0-x86_64-linux/lib/google/2.3/protobuf_c.so > > > > > > $ gem unpack google-protobuf-3.13.0-x86_64-linux.gem --spec && grep > > > extensions google-protobuf-3.13.0-x86_64-linux.gemspec > > > extensions: [] > > > > > > I think the cleanest solution would be for debian to do the same thing. > > Fair enough. Now that I think about it, extensions is supposed to be > > a list of extensions that need to be built, so indeed dropping it from > > the gemspec included in the Debian packages make sense. > Yeah, that's exactly how I expect the `extensions` field to be interpreted. On the other hand, our packages that already use the rubygems machinery to install, already contain the gem_build.complete file: $ dpkg -L ruby-byebug | grep complete /usr/lib/x86_64-linux-gnu/rubygems-integration/2.7.0/extensions/x86_64-linux/2.7.0/byebug-11.1.3/gem.build_complete 8<8<8<- $ cat > Gemfile source "https://rubygems.org/; gem "byebug" 8<8<8<- $ bundle --local Resolving dependencies... Using byebug 11.1.3 Using bundler 2.2.0.rc.1 Following files may not be writable, so sudo is needed: /usr/local/bin /var/lib/gems/2.7.0 Bundle complete! 1 Gemfile dependency, 2 gems now installed. Use `bundle info [gemname]` to see where a bundled gem is installed. 8<8<8<- $ echo 'gem "ffi"' >> Gemfile 8<8<8<- $ bundle --local Could not find gem 'ffi' in any of the gem sources listed in your Gemfile. 8<8<8<- So this is only a problem for the "old-style" packages that install their stuff to vendor_ruby and don't install as gems: $ dpkg -L ruby-ffi | grep gem.build_complete (nothing) and they are quite a few packages yet: $ apt-file search vendor_ruby/ | grep '\.so$' | wc -l 164 signature.asc Description: PGP signature
Bug#972702: ruby-bundler: dependency resolution fails for compiled gems
On Thu, Oct 29, 2020 at 11:17:59AM +0100, David Rodríguez wrote: > Hi! > > Just to clarify why I prefer the second solution, I think what debian does is > shipping precompiled versions of extensions, so technically the gemspec > shipped in the debian should include no extensions at all. This is something > some upstream gems already do. Take, for example, google-protobuf. It has a > precompiled version for linux: > https://rubygems.org/gems/google-protobuf/versions/3.13.0-x86_64-linux. If we > fetch and unpack this package, we can see it includes the prebuilt `.so` > extension, but no extensions in its gemspec: > > $ gem fetch google-protobuf > Fetching google-protobuf-3.13.0-x86_64-linux.gem > Downloaded google-protobuf-3.13.0-x86_64-linux > > $ gem unpack google-protobuf-3.13.0-x86_64-linux.gem > Unpacked gem: > '/home/deivid/Code/playground/google-protobuf-3.13.0-x86_64-linux' > > $ find google-protobuf-3.13.0-x86_64-linux -name '*.so' > google-protobuf-3.13.0-x86_64-linux/lib/google/2.6/protobuf_c.so > google-protobuf-3.13.0-x86_64-linux/lib/google/2.4/protobuf_c.so > google-protobuf-3.13.0-x86_64-linux/lib/google/2.7/protobuf_c.so > google-protobuf-3.13.0-x86_64-linux/lib/google/2.5/protobuf_c.so > google-protobuf-3.13.0-x86_64-linux/lib/google/2.3/protobuf_c.so > > $ gem unpack google-protobuf-3.13.0-x86_64-linux.gem --spec && grep > extensions google-protobuf-3.13.0-x86_64-linux.gemspec > extensions: [] > > I think the cleanest solution would be for debian to do the same thing. Fair enough. Now that I think about it, extensions is supposed to be a list of extensions that need to be built, so indeed dropping it from the gemspec included in the Debian packages make sense. Thanks! signature.asc Description: PGP signature
Bug#972702: ruby-bundler: dependency resolution fails for compiled gems
El 29/10/20 a las 13:14, Antonio Terceiro escribió: On Thu, Oct 29, 2020 at 11:17:59AM +0100, David Rodríguez wrote: Hi! Just to clarify why I prefer the second solution, I think what debian does is shipping precompiled versions of extensions, so technically the gemspec shipped in the debian should include no extensions at all. This is something some upstream gems already do. Take, for example, google-protobuf. It has a precompiled version for linux: https://rubygems.org/gems/google-protobuf/versions/3.13.0-x86_64-linux. If we fetch and unpack this package, we can see it includes the prebuilt `.so` extension, but no extensions in its gemspec: $ gem fetch google-protobuf Fetching google-protobuf-3.13.0-x86_64-linux.gem Downloaded google-protobuf-3.13.0-x86_64-linux $ gem unpack google-protobuf-3.13.0-x86_64-linux.gem Unpacked gem: '/home/deivid/Code/playground/google-protobuf-3.13.0-x86_64-linux' $ find google-protobuf-3.13.0-x86_64-linux -name '*.so' google-protobuf-3.13.0-x86_64-linux/lib/google/2.6/protobuf_c.so google-protobuf-3.13.0-x86_64-linux/lib/google/2.4/protobuf_c.so google-protobuf-3.13.0-x86_64-linux/lib/google/2.7/protobuf_c.so google-protobuf-3.13.0-x86_64-linux/lib/google/2.5/protobuf_c.so google-protobuf-3.13.0-x86_64-linux/lib/google/2.3/protobuf_c.so $ gem unpack google-protobuf-3.13.0-x86_64-linux.gem --spec && grep extensions google-protobuf-3.13.0-x86_64-linux.gemspec extensions: [] I think the cleanest solution would be for debian to do the same thing. Fair enough. Now that I think about it, extensions is supposed to be a list of extensions that need to be built, so indeed dropping it from the gemspec included in the Debian packages make sense. Yeah, that's exactly how I expect the `extensions` field to be interpreted. Thanks! No problem, if you run into other issues, let me know!
Bug#972702: ruby-bundler: dependency resolution fails for compiled gems
Hi! Just to clarify why I prefer the second solution, I think what debian does is shipping precompiled versions of extensions, so technically the gemspec shipped in the debian should include no extensions at all. This is something some upstream gems already do. Take, for example, google-protobuf. It has a precompiled version for linux: https://rubygems.org/gems/google-protobuf/versions/3.13.0-x86_64-linux. If we fetch and unpack this package, we can see it includes the prebuilt `.so` extension, but no extensions in its gemspec: $ gem fetch google-protobuf Fetching google-protobuf-3.13.0-x86_64-linux.gem Downloaded google-protobuf-3.13.0-x86_64-linux $ gem unpack google-protobuf-3.13.0-x86_64-linux.gem Unpacked gem: '/home/deivid/Code/playground/google-protobuf-3.13.0-x86_64-linux' $ find google-protobuf-3.13.0-x86_64-linux -name '*.so' google-protobuf-3.13.0-x86_64-linux/lib/google/2.6/protobuf_c.so google-protobuf-3.13.0-x86_64-linux/lib/google/2.4/protobuf_c.so google-protobuf-3.13.0-x86_64-linux/lib/google/2.7/protobuf_c.so google-protobuf-3.13.0-x86_64-linux/lib/google/2.5/protobuf_c.so google-protobuf-3.13.0-x86_64-linux/lib/google/2.3/protobuf_c.so $ gem unpack google-protobuf-3.13.0-x86_64-linux.gem --spec && grep extensions google-protobuf-3.13.0-x86_64-linux.gemspec extensions: [] I think the cleanest solution would be for debian to do the same thing.
Bug#972702: ruby-bundler: dependency resolution fails for compiled gems
On Wed, Oct 28, 2020 at 07:02:06PM -0300, Lucas Kanashiro wrote: > Thanks to David Rodriguez (our dear bundler upstream maintainer :) I was > able to workaround the bug locally and understand a bit better what is going > on. > > The source of this problem is this commit: > > https://github.com/rubygems/rubygems/commit/226ec115fe503bcc93bffdf5cd3e8a668890b4d8 > > Quoting David: > > "Anyways, the actual check making bundler believe the ffi gem is missing > extensions is `File.exist?(gem_build_complete_path)`. I believe the idea of > this check is that bundler/rubygems are not able to know which artifacts an > extension will generate, so what rubygems does is generating this dummy file > at a well known path after a successful extension compilation, so that it > can later check the existence of this path when it needs to know whether a > gem is missing extensions." > > There are 2 workarounds to fix this issue: > > 1) Create this dummy file. > > $ mkdir -p > /usr/share/rubygems-integration/2.7.0/extensions/x86_64-linux/2.7.0/ffi-1.12.2/ > > $ touch > /usr/share/rubygems-integration/2.7.0/extensions/x86_64-linux/2.7.0/ffi-1.12.2/gem.build_complete > > > After that 'bundle --local' will find the ffi gem. > > 2) Remove the extension from gemspec. > > $ sed -i.bak -e 3d > /usr/share/rubygems-integration/2.7.0/specifications/ffi-1.12.2.gemspec > > After removing the header line containing "# stub: ext/ffi_c/extconf.rb", > 'bundler --local' also works fine and find the ffi gem. > > I think the second approach is what we want, and the proper fix would be to > not include extensions in our gemspecs. We could try to achieve that via > gem2deb. Any thoughts? I think we should diverge less, not more, from upstream. So I would go for 1). We already use the Rubygems classes that do builds, but it seems that this new feature was added around the method we use. I think we should change gem2deb to include that file, ideally reusing more of the code from rubygems. The relevant code is lib/rubygems/ext/builder.rb in rubygems, and lib/gem2deb/extension_builder.rb in gem2deb. signature.asc Description: PGP signature
Bug#972702: ruby-bundler: dependency resolution fails for compiled gems
Thanks to David Rodriguez (our dear bundler upstream maintainer :) I was able to workaround the bug locally and understand a bit better what is going on. The source of this problem is this commit: https://github.com/rubygems/rubygems/commit/226ec115fe503bcc93bffdf5cd3e8a668890b4d8 Quoting David: "Anyways, the actual check making bundler believe the ffi gem is missing extensions is `File.exist?(gem_build_complete_path)`. I believe the idea of this check is that bundler/rubygems are not able to know which artifacts an extension will generate, so what rubygems does is generating this dummy file at a well known path after a successful extension compilation, so that it can later check the existence of this path when it needs to know whether a gem is missing extensions." There are 2 workarounds to fix this issue: 1) Create this dummy file. $ mkdir -p /usr/share/rubygems-integration/2.7.0/extensions/x86_64-linux/2.7.0/ffi-1.12.2/ $ touch /usr/share/rubygems-integration/2.7.0/extensions/x86_64-linux/2.7.0/ffi-1.12.2/gem.build_complete After that 'bundle --local' will find the ffi gem. 2) Remove the extension from gemspec. $ sed -i.bak -e 3d /usr/share/rubygems-integration/2.7.0/specifications/ffi-1.12.2.gemspec After removing the header line containing "# stub: ext/ffi_c/extconf.rb", 'bundler --local' also works fine and find the ffi gem. I think the second approach is what we want, and the proper fix would be to not include extensions in our gemspecs. We could try to achieve that via gem2deb. Any thoughts? -- Lucas Kanashiro
Bug#972702: ruby-bundler: dependency resolution fails for compiled gems
Running the same command with verbose enabled tells us that bundler ignores the ffi gem because it's missing extensions: root@rubygems-test:~/test# bundle --local --verbose Running `bundle install --local --verbose` with bundler 2.2.0.rc.1 Don't run Bundler as root. Bundler can ask for sudo if it is needed, and installing your bundle as root will break this application for all non-root users on this machine. Found changes from the lockfile, re-resolving dependencies because the list of sources changed, the dependencies in your gemfile changed, you added a new platform to your gemfile Source rubygems repository https://rubygems.org/ or installed locally is ignoring # because it is missing extensions Could not find gem 'ffi' in any of the gem sources listed in your Gemfile. Bundler::GemNotFound: Could not find gem 'ffi' in any of the gem sources listed in your Gemfile. /usr/share/rubygems-integration/all/gems/bundler-2.2.0.rc.1/lib/bundler/resolver.rb:313:in `block in verify_gemfile_dependencies_are_found!' /usr/share/rubygems-integration/all/gems/bundler-2.2.0.rc.1/lib/bundler/resolver.rb:281:in `each' /usr/share/rubygems-integration/all/gems/bundler-2.2.0.rc.1/lib/bundler/resolver.rb:281:in `verify_gemfile_dependencies_are_found!' /usr/share/rubygems-integration/all/gems/bundler-2.2.0.rc.1/lib/bundler/resolver.rb:49:in `start' /usr/share/rubygems-integration/all/gems/bundler-2.2.0.rc.1/lib/bundler/resolver.rb:22:in `resolve' /usr/share/rubygems-integration/all/gems/bundler-2.2.0.rc.1/lib/bundler/definition.rb:264:in `resolve' /usr/share/rubygems-integration/all/gems/bundler-2.2.0.rc.1/lib/bundler/definition.rb:175:in `specs' /usr/share/rubygems-integration/all/gems/bundler-2.2.0.rc.1/lib/bundler/definition.rb:156:in `resolve_with_cache!' /usr/share/rubygems-integration/all/gems/bundler-2.2.0.rc.1/lib/bundler/installer.rb:304:in `resolve_if_needed' /usr/share/rubygems-integration/all/gems/bundler-2.2.0.rc.1/lib/bundler/installer.rb:84:in `block in run' /usr/share/rubygems-integration/all/gems/bundler-2.2.0.rc.1/lib/bundler/process_lock.rb:12:in `block in lock' /usr/share/rubygems-integration/all/gems/bundler-2.2.0.rc.1/lib/bundler/process_lock.rb:9:in `open' /usr/share/rubygems-integration/all/gems/bundler-2.2.0.rc.1/lib/bundler/process_lock.rb:9:in `lock' /usr/share/rubygems-integration/all/gems/bundler-2.2.0.rc.1/lib/bundler/installer.rb:73:in `run' /usr/share/rubygems-integration/all/gems/bundler-2.2.0.rc.1/lib/bundler/installer.rb:25:in `install' /usr/share/rubygems-integration/all/gems/bundler-2.2.0.rc.1/lib/bundler/cli/install.rb:64:in `run' /usr/share/rubygems-integration/all/gems/bundler-2.2.0.rc.1/lib/bundler/cli.rb:261:in `block in install' /usr/share/rubygems-integration/all/gems/bundler-2.2.0.rc.1/lib/bundler/settings.rb:121:in `temporary' /usr/share/rubygems-integration/all/gems/bundler-2.2.0.rc.1/lib/bundler/cli.rb:260:in `install' /usr/share/rubygems-integration/all/gems/bundler-2.2.0.rc.1/lib/bundler/vendor/thor/lib/thor/command.rb:27:in `run' /usr/share/rubygems-integration/all/gems/bundler-2.2.0.rc.1/lib/bundler/vendor/thor/lib/thor/invocation.rb:127:in `invoke_command' /usr/share/rubygems-integration/all/gems/bundler-2.2.0.rc.1/lib/bundler/vendor/thor/lib/thor.rb:392:in `dispatch' /usr/share/rubygems-integration/all/gems/bundler-2.2.0.rc.1/lib/bundler/cli.rb:30:in `dispatch' /usr/share/rubygems-integration/all/gems/bundler-2.2.0.rc.1/lib/bundler/vendor/thor/lib/thor/base.rb:485:in `start' /usr/share/rubygems-integration/all/gems/bundler-2.2.0.rc.1/lib/bundler/cli.rb:24:in `start' /usr/share/rubygems-integration/all/gems/bundler-2.2.0.rc.1/exe/bundle:49:in `block in ' /usr/share/rubygems-integration/all/gems/bundler-2.2.0.rc.1/lib/bundler/friendly_errors.rb:117:in `with_friendly_errors' /usr/share/rubygems-integration/all/gems/bundler-2.2.0.rc.1/exe/bundle:37:in `' /usr/bin/bundle:23:in `load' /usr/bin/bundle:23:in `' -- Lucas Kanashiro
Bug#972702: ruby-bundler: dependency resolution fails for compiled gems
It seems the search path is not right in this newer ruby-bundler version. Using the following Gemfile and running 'bundle --local' works fine. root@rubygems-test:~/test# cat Gemfile path "/usr/share/rubygems-integration/2.7.0" do gem 'ffi' end root@rubygems-test:~/test# bundle --local Don't run Bundler as root. Bundler can ask for sudo if it is needed, and installing your bundle as root will break this application for all non-root users on this machine. Using bundler 2.2.0.rc.1 Using ffi 1.12.2 from source at `/usr/share/rubygems-integration/2.7.0` Bundle complete! 1 Gemfile dependency, 2 gems now installed. Use `bundle info [gemname]` to see where a bundled gem is installed. Interestingly, I have 2 containers (one with ruby-bundler 2.2.0.rc.1 and another with version 2.1.4) and both of them have the same set of directories in GEM PATHS, so both should be looking at the same directories. root@rubygems-test-old:~/test# bundler version Bundler version 2.1.4 (2020-01-05 commit 32a4159325) root@rubygems-test-old:~/test# gem env RubyGems Environment: - RUBYGEMS VERSION: 3.2.0.rc.1 - RUBY VERSION: 2.7.2 (2020-10-01 patchlevel 137) [x86_64-linux-gnu] - INSTALLATION DIRECTORY: /var/lib/gems/2.7.0 - USER INSTALLATION DIRECTORY: /root/.local/share/gem/ruby/2.7.0 - RUBY EXECUTABLE: /usr/bin/ruby2.7 - GIT EXECUTABLE: - EXECUTABLE DIRECTORY: /usr/local/bin - SPEC CACHE DIRECTORY: /root/.local/share/gem/specs - SYSTEM CONFIGURATION DIRECTORY: /etc - RUBYGEMS PLATFORMS: - ruby - x86_64-linux - GEM PATHS: - /var/lib/gems/2.7.0 - /root/.local/share/gem/ruby/2.7.0 - /usr/local/lib/ruby/gems/2.7.0 - /usr/lib/ruby/gems/2.7.0 - /usr/lib/x86_64-linux-gnu/ruby/gems/2.7.0 - /usr/share/rubygems-integration/2.7.0 - /usr/share/rubygems-integration/all - /usr/lib/x86_64-linux-gnu/rubygems-integration/2.7.0 - GEM CONFIGURATION: - :update_sources => true - :verbose => true - :backtrace => false - :bulk_threshold => 1000 - REMOTE SOURCES: - https://rubygems.org/ - SHELL PATH: - /usr/local/sbin - /usr/local/bin - /usr/sbin - /usr/bin - /sbin - /bin root@rubygems-test:~/test# bundler version Bundler version 2.2.0.rc.1 (2020-10-24 commit unknown) root@rubygems-test:~/test# gem env RubyGems Environment: - RUBYGEMS VERSION: 3.2.0.rc.1 - RUBY VERSION: 2.7.2 (2020-10-01 patchlevel 137) [x86_64-linux-gnu] - INSTALLATION DIRECTORY: /var/lib/gems/2.7.0 - USER INSTALLATION DIRECTORY: /root/.local/share/gem/ruby/2.7.0 - RUBY EXECUTABLE: /usr/bin/ruby2.7 - GIT EXECUTABLE: - EXECUTABLE DIRECTORY: /usr/local/bin - SPEC CACHE DIRECTORY: /root/.local/share/gem/specs - SYSTEM CONFIGURATION DIRECTORY: /etc - RUBYGEMS PLATFORMS: - ruby - x86_64-linux - GEM PATHS: - /var/lib/gems/2.7.0 - /root/.local/share/gem/ruby/2.7.0 - /usr/local/lib/ruby/gems/2.7.0 - /usr/lib/ruby/gems/2.7.0 - /usr/lib/x86_64-linux-gnu/ruby/gems/2.7.0 - /usr/share/rubygems-integration/2.7.0 - /usr/share/rubygems-integration/all - /usr/lib/x86_64-linux-gnu/rubygems-integration/2.7.0 - GEM CONFIGURATION: - :update_sources => true - :verbose => true - :backtrace => false - :bulk_threshold => 1000 - REMOTE SOURCES: - https://rubygems.org/ - SHELL PATH: - /usr/local/sbin - /usr/local/bin - /usr/sbin - /usr/bin - /sbin - /bin I'll keep investigating to understand what is going on. -- Lucas Kanashiro
Bug#972702: ruby-bundler: dependency resolution fails for compiled gems
Package: ruby-bundler Version: 2.2.0~rc.1 Severity: grave Justification: renders package unusable 8<8<8<- $ cat Gemfile source 'https://rubygems.org' gem 'ffi' $ bundle --local Could not find gem 'ffi' in any of the gem sources listed in your Gemfile. [7]$ gem list ffi *** LOCAL GEMS *** ffi (1.12.2) ffi-bit_masks (0.1.1) ffi-compiler (1.0.1) ffi-libarchive (1.0.1) gir_ffi (0.15.3) public_suffix (3.0.3) 8<8<8<- Reverting to ruby-bundler (from the previous source package bundler) from testing fixes it. -- System Information: Debian Release: bullseye/sid APT prefers unstable-debug APT policy: (500, 'unstable-debug'), (500, 'testing-debug'), (500, 'unstable'), (500, 'testing'), (1, 'experimental-debug'), (1, 'experimental') Architecture: amd64 (x86_64) Foreign Architectures: arm64, i386 Kernel: Linux 5.9.0-1-amd64 (SMP w/4 CPU threads) Locale: LANG=pt_BR.UTF-8, LC_CTYPE=pt_BR.UTF-8 (charmap=UTF-8), LANGUAGE=pt_BR:pt:en Shell: /bin/sh linked to /usr/bin/dash Init: systemd (via /run/systemd/system) LSM: AppArmor: enabled Versions of packages ruby-bundler depends on: ii ruby 1:2.7+2 ii ruby-molinillo0.6.4-1 ii ruby-net-http-persistent 3.1.0-2 ii ruby-rubygems 3.2.0~rc.1-1 ii ruby-thor 0.20.3-2 ii rubygems-integration 1.17.2 ruby-bundler recommends no packages. ruby-bundler suggests no packages. -- no debconf information signature.asc Description: PGP signature
