Bug#863927: qtwebengine-opensource-src: FTBFS: memory exhausted

[Sandro Knauß]

Moin,

thanks for your work! I'll added the NMU patch to the git repo too.

Best regards,

sandro

--
On Montag, 5. Juni 2017 22:28:10 CEST Adrian Bunk wrote:
> On Mon, Jun 05, 2017 at 10:44:59AM +0200, Rene Engelhard wrote:
> >...
> >
> > This looks unnecessarily ugly to me (or I miss something). Why not use
> > dpkg-buildflags' mechanism for this? See e.g.
> >
> >...
> >
> > In this case maybe _APPEND -g1 (if you don't do the _STRIP)
> 
> Thanks, your _APPEND suggestion is much better than my original patch.
> 
> Sandro, the debdiff of my upload is attached.
> 
> > Regards,
> > 
> > Rene
> 
> cu
> Adrian



signature.asc
Description: This is a digitally signed message part.

Bug#863927: qtwebengine-opensource-src: FTBFS: memory exhausted

[Adrian Bunk]

On Mon, Jun 05, 2017 at 10:44:59AM +0200, Rene Engelhard wrote:
>...
> This looks unnecessarily ugly to me (or I miss something). Why not use
> dpkg-buildflags' mechanism for this? See e.g.
>...
> In this case maybe _APPEND -g1 (if you don't do the _STRIP)

Thanks, your _APPEND suggestion is much better than my original patch.

Sandro, the debdiff of my upload is attached.

> Regards,
> 
> Rene

cu
Adrian

-- 

   "Is there not promise of rain?" Ling Tan asked suddenly out
of the darkness. There had been need of rain for many days.
   "Only a promise," Lao Er said.
   Pearl S. Buck - Dragon Seed

diffstat for qtwebengine-opensource-src-5.7.1+dfsg 
qtwebengine-opensource-src-5.7.1+dfsg

 changelog |8 
 rules |8 
 2 files changed, 16 insertions(+)

diff -Nru qtwebengine-opensource-src-5.7.1+dfsg/debian/changelog 
qtwebengine-opensource-src-5.7.1+dfsg/debian/changelog
--- qtwebengine-opensource-src-5.7.1+dfsg/debian/changelog  2017-01-18 
20:08:01.0 +0200
+++ qtwebengine-opensource-src-5.7.1+dfsg/debian/changelog  2017-06-05 
16:16:54.0 +0300
@@ -1,3 +1,11 @@
+qtwebengine-opensource-src (5.7.1+dfsg-6.1) unstable; urgency=medium
+
+  * Non-maintainer upload.
+  * Build with -g1 instead of -g on 32bit architectures.
+(Closes: #863927)
+
+ -- Adrian Bunk   Mon, 05 Jun 2017 16:16:54 +0300
+
 qtwebengine-opensource-src (5.7.1+dfsg-6) unstable; urgency=medium
 
   [ Sandro Knauß ]
diff -Nru qtwebengine-opensource-src-5.7.1+dfsg/debian/rules 
qtwebengine-opensource-src-5.7.1+dfsg/debian/rules
--- qtwebengine-opensource-src-5.7.1+dfsg/debian/rules  2017-01-18 
20:00:38.0 +0200
+++ qtwebengine-opensource-src-5.7.1+dfsg/debian/rules  2017-06-05 
16:16:54.0 +0300
@@ -6,6 +6,14 @@
 export NINJAFLAGS=-v
 include /usr/share/dpkg/default.mk
 
+DEB_HOST_ARCH_BITS ?= $(shell dpkg-architecture -qDEB_HOST_ARCH_BITS)
+
+# TODO: properly integrate with the other debug setting
+ifeq (32,$(DEB_HOST_ARCH_BITS))
+   export DEB_CFLAGS_MAINT_APPEND = -g1
+   export DEB_CXXFLAGS_MAINT_APPEND = -g1
+endif
+
 export CFLAGS := $(shell dpkg-buildflags --get CFLAGS) $(shell dpkg-buildflags 
--get CPPFLAGS)
 export CXXFLAGS := $(shell dpkg-buildflags --get CXXFLAGS) $(shell 
dpkg-buildflags --get CPPFLAGS)
 export LDFLAGS := $(shell dpkg-buildflags --get LDFLAGS) -Wl,--as-needed

Bug#863927: qtwebengine-opensource-src: FTBFS: memory exhausted

[Rene Engelhard]

Hi,

On Mon, Jun 05, 2017 at 12:07:48AM +0300, Adrian Bunk wrote:
> --- qtwebengine-opensource-src-5.7.1+dfsg/debian/rules2017-01-18 
> 20:00:38.0 +0200
> +++ qtwebengine-opensource-src-5.7.1+dfsg/debian/rules2017-06-04 
> 22:12:35.0 +0300
> @@ -6,16 +6,22 @@
>  export NINJAFLAGS=-v
>  include /usr/share/dpkg/default.mk
>  
> -export CFLAGS := $(shell dpkg-buildflags --get CFLAGS) $(shell 
> dpkg-buildflags --get CPPFLAGS)
> -export CXXFLAGS := $(shell dpkg-buildflags --get CXXFLAGS) $(shell 
> dpkg-buildflags --get CPPFLAGS)
> +DEB_HOST_ARCH ?= $(shell dpkg-architecture -qDEB_HOST_ARCH)
> +DEB_HOST_MULTIARCH ?= $(shell dpkg-architecture -qDEB_HOST_MULTIARCH)
> +
> +# TODO: properly integrate with the other debug setting for buster
> +ifneq (,$(filter $(DEB_HOST_ARCH), armel armhf i386 mips mipsel))
> + export CFLAGS := $(shell dpkg-buildflags --get CFLAGS) $(shell 
> dpkg-buildflags --get CPPFLAGS) -g1
> + export CXXFLAGS := $(shell dpkg-buildflags --get CXXFLAGS) $(shell 
> dpkg-buildflags --get CPPFLAGS) -g1
> +else
> + export CFLAGS := $(shell dpkg-buildflags --get CFLAGS) $(shell 
> dpkg-buildflags --get CPPFLAGS)
> + export CXXFLAGS := $(shell dpkg-buildflags --get CXXFLAGS) $(shell 
> dpkg-buildflags --get CPPFLAGS)
> +endif
>  export LDFLAGS := $(shell dpkg-buildflags --get LDFLAGS) -Wl,--as-needed
>  export QT_SELECT := qt5
>  
>  VERSION_CLEAN_UPSTREAM = $(call dpkg_late_eval,VERSION_CLEAN_UPSTREAM,echo 
> '$(DEB_VERSION_UPSTREAM)' | sed -e 's/\(~\|+\).*//')
>  
> -DEB_HOST_ARCH ?= $(shell dpkg-architecture -qDEB_HOST_ARCH)
> -DEB_HOST_MULTIARCH ?= $(shell dpkg-architecture -qDEB_HOST_MULTIARCH)
> -
>  gstab_architectures :=
>  fulldebug_architectures :=
>  disabled_jit_architectures := armel mips mipsel

This looks unnecessarily ugly to me (or I miss something). Why not use
dpkg-buildflags' mechanism for this? See e.g.

https://anonscm.debian.org/cgit/pkg-openoffice/libreoffice.git/tree/rules#n878:

ifeq "$(ENABLE_SYMBOLS)" "y"
  # Small symbols?
  ifeq "$(SMALL_SYMBOLS)" "y"
CONFIGURE_FLAGS += --enable-symbols=SMALL
DEB_CFLAGS_MAINT_STRIP := -g
DEB_CXXFLAGS_MAINT_STRIP := -g
DEB_CFLAGS_MAINT_PREPEND += -g1
DEB_CXXFLAGS_MAINT_PREPEND += -g1
export DEB_CFLAGS_MAINT_STRIP DEB_CXXFLAGS_MAINT_STRIP
export DEB_CFLAGS_MAINT_PREPEND DEB_CXXFLAGS_MAINT_PREPEND
  else
CONFIGURE_FLAGS += --enable-symbols
  endif
endif
export DPKG_EXPORT_BUILDFLAGS=y 
include /usr/share/dpkg/buildflags.mk

In this case maybe _APPEND -g1 (if you don't do the _STRIP)

Regards,

Rene

Bug#863927: qtwebengine-opensource-src: FTBFS: memory exhausted

[Sandro Knauß]

Hey,

thanks for your help.

Please go for an NMU (delay 0 is fine for me). I'll add the patch to the git 
repo, if you don't have access to it.

Best Regards,

sandro

--
> I found the problem:
> The -g is coming from dpkg-buildflags.
> 
> Below is the smallest patch I see for this, a proper reshuffle of the
> debug settings in debian/rules looked too heavy so near to the release.
> 
> Could you upload this, or can I do an NMU?
> 
> > Best Regards,
> > 
> > sandro
> 
> cu
> Adrian
> 
> 
> --- qtwebengine-opensource-src-5.7.1+dfsg/debian/rules2017-01-18
> 20:00:38.0 +0200 +++
> qtwebengine-opensource-src-5.7.1+dfsg/debian/rules2017-06-04
> 22:12:35.0 +0300 @@ -6,16 +6,22 @@
>  export NINJAFLAGS=-v
>  include /usr/share/dpkg/default.mk
> 
> -export CFLAGS := $(shell dpkg-buildflags --get CFLAGS) $(shell
> dpkg-buildflags --get CPPFLAGS) -export CXXFLAGS := $(shell dpkg-buildflags
> --get CXXFLAGS) $(shell dpkg-buildflags --get CPPFLAGS) +DEB_HOST_ARCH ?=
> $(shell dpkg-architecture -qDEB_HOST_ARCH)
> +DEB_HOST_MULTIARCH ?= $(shell dpkg-architecture -qDEB_HOST_MULTIARCH)
> +
> +# TODO: properly integrate with the other debug setting for buster
> +ifneq (,$(filter $(DEB_HOST_ARCH), armel armhf i386 mips mipsel))
> + export CFLAGS := $(shell dpkg-buildflags --get CFLAGS) $(shell
> dpkg-buildflags --get CPPFLAGS) -g1 + export CXXFLAGS := $(shell
> dpkg-buildflags --get CXXFLAGS) $(shell dpkg-buildflags --get CPPFLAGS) -g1
> +else
> + export CFLAGS := $(shell dpkg-buildflags --get CFLAGS) $(shell
> dpkg-buildflags --get CPPFLAGS) + export CXXFLAGS := $(shell
> dpkg-buildflags --get CXXFLAGS) $(shell dpkg-buildflags --get CPPFLAGS)
> +endif
>  export LDFLAGS := $(shell dpkg-buildflags --get LDFLAGS) -Wl,--as-needed
>  export QT_SELECT := qt5
> 
>  VERSION_CLEAN_UPSTREAM = $(call dpkg_late_eval,VERSION_CLEAN_UPSTREAM,echo
> '$(DEB_VERSION_UPSTREAM)' | sed -e 's/\(~\|+\).*//')
> 
> -DEB_HOST_ARCH ?= $(shell dpkg-architecture -qDEB_HOST_ARCH)
> -DEB_HOST_MULTIARCH ?= $(shell dpkg-architecture -qDEB_HOST_MULTIARCH)
> -
>  gstab_architectures :=
>  fulldebug_architectures :=
>  disabled_jit_architectures := armel mips mipsel



signature.asc
Description: This is a digitally signed message part.

Bug#863927: qtwebengine-opensource-src: FTBFS: memory exhausted

[Adrian Bunk]

On Sat, Jun 03, 2017 at 09:33:38PM +0200, Sandro Knauß wrote:
> Hey,
> 
> > No, the fix would be to not produce 1 GB of debug info for this library:
> 
> > The build log [1] confirms that -g is used in the i386 build.
> > 
> > -g0 instead of -g (or no -g option) would surely solve this problem.
> > 
> > -g1 would likely be sufficient to fix the problem on i386,
> > while still providing enough debug information for backtraces.
> > 
> > A similar -g0/-g1 fix could also fix the armhf and mipsel builds (untested).
> 
> interessting point and a year ago I also ask for ideas and nothing came. I 
> tested a lot to get it building on other platforms without success. At the 
> moment I'm hardy using my computer, so I can't test it at the moment, sorry.

I found the problem:
The -g is coming from dpkg-buildflags.

Below is the smallest patch I see for this, a proper reshuffle of the 
debug settings in debian/rules looked too heavy so near to the release.

Could you upload this, or can I do an NMU?

> Best Regards,
> 
> sandro

cu
Adrian


--- qtwebengine-opensource-src-5.7.1+dfsg/debian/rules  2017-01-18 
20:00:38.0 +0200
+++ qtwebengine-opensource-src-5.7.1+dfsg/debian/rules  2017-06-04 
22:12:35.0 +0300
@@ -6,16 +6,22 @@
 export NINJAFLAGS=-v
 include /usr/share/dpkg/default.mk
 
-export CFLAGS := $(shell dpkg-buildflags --get CFLAGS) $(shell dpkg-buildflags 
--get CPPFLAGS)
-export CXXFLAGS := $(shell dpkg-buildflags --get CXXFLAGS) $(shell 
dpkg-buildflags --get CPPFLAGS)
+DEB_HOST_ARCH ?= $(shell dpkg-architecture -qDEB_HOST_ARCH)
+DEB_HOST_MULTIARCH ?= $(shell dpkg-architecture -qDEB_HOST_MULTIARCH)
+
+# TODO: properly integrate with the other debug setting for buster
+ifneq (,$(filter $(DEB_HOST_ARCH), armel armhf i386 mips mipsel))
+   export CFLAGS := $(shell dpkg-buildflags --get CFLAGS) $(shell 
dpkg-buildflags --get CPPFLAGS) -g1
+   export CXXFLAGS := $(shell dpkg-buildflags --get CXXFLAGS) $(shell 
dpkg-buildflags --get CPPFLAGS) -g1
+else
+   export CFLAGS := $(shell dpkg-buildflags --get CFLAGS) $(shell 
dpkg-buildflags --get CPPFLAGS)
+   export CXXFLAGS := $(shell dpkg-buildflags --get CXXFLAGS) $(shell 
dpkg-buildflags --get CPPFLAGS)
+endif
 export LDFLAGS := $(shell dpkg-buildflags --get LDFLAGS) -Wl,--as-needed
 export QT_SELECT := qt5
 
 VERSION_CLEAN_UPSTREAM = $(call dpkg_late_eval,VERSION_CLEAN_UPSTREAM,echo 
'$(DEB_VERSION_UPSTREAM)' | sed -e 's/\(~\|+\).*//')
 
-DEB_HOST_ARCH ?= $(shell dpkg-architecture -qDEB_HOST_ARCH)
-DEB_HOST_MULTIARCH ?= $(shell dpkg-architecture -qDEB_HOST_MULTIARCH)
-
 gstab_architectures :=
 fulldebug_architectures :=
 disabled_jit_architectures := armel mips mipsel

Bug#863927: qtwebengine-opensource-src: FTBFS: memory exhausted

[Sandro Knauß]

Hey,

> No, the fix would be to not produce 1 GB of debug info for this library:

> The build log [1] confirms that -g is used in the i386 build.
> 
> -g0 instead of -g (or no -g option) would surely solve this problem.
> 
> -g1 would likely be sufficient to fix the problem on i386,
> while still providing enough debug information for backtraces.
> 
> A similar -g0/-g1 fix could also fix the armhf and mipsel builds (untested).

interessting point and a year ago I also ask for ideas and nothing came. I 
tested a lot to get it building on other platforms without success. At the 
moment I'm hardy using my computer, so I can't test it at the moment, sorry.

Best Regards,

sandro

signature.asc
Description: This is a digitally signed message part.

Bug#863927: qtwebengine-opensource-src: FTBFS: memory exhausted

[Adrian Bunk]

On Fri, Jun 02, 2017 at 11:32:39AM +0200, Sandro Knauß wrote:
>...
> I know this is not ideal but we can't fix this. If this is a nogo for the 
> release than we need to remove the i386 packages, that used the fact, that 
> debian buildds for i386 are running on amd64 systems.

No, the fix would be to not produce 1 GB of debug info for this library:

Package: libqt5webenginecore5-dbgsym
Source: qtwebengine-opensource-src
Version: 5.7.1+dfsg-6
Installed-Size: 916272
Maintainer: Debian Qt/KDE Maintainers 
Architecture: i386

The build log [1] confirms that -g is used in the i386 build.

-g0 instead of -g (or no -g option) would surely solve this problem.

-g1 would likely be sufficient to fix the problem on i386,
while still providing enough debug information for backtraces.

A similar -g0/-g1 fix could also fix the armhf and mipsel builds (untested).

> Best regards,
> 
> sandro

cu
Adrian

[1] 
https://buildd.debian.org/status/fetch.php?pkg=qtwebengine-opensource-src=i386=5.7.1%2Bdfsg-6=1484781991=0

-- 

   "Is there not promise of rain?" Ling Tan asked suddenly out
of the darkness. There had been need of rain for many days.
   "Only a promise," Lao Er said.
   Pearl S. Buck - Dragon Seed

Bug#863927: qtwebengine-opensource-src: FTBFS: memory exhausted

[Lucas Nussbaum]

Hi,

On 02/06/17 at 11:32 +0200, Sandro Knauß wrote:
> Hey,
>  
> > During a rebuild of all packages in stretch (in a stretch chroot, not a
> > sid chroot), your package failed to build on i386.
> > 
> > Note that the build node has a lot of RAM, so if memory is really
> > exhausted, it could mean that it can no longer be built on i386.
> 
> Well the memory consumptions are really at on limit for all porterboxes and 
> it 
> is only possible to build QtWebEngine on a 64bit systems or that the host 
> system is a 64bit on. Otherwise the 32bit system can't access enough RAM/
> diskspace, because it hits otherwise the 4GB limit for files. We had 
> discussed 
> this topic in a thread already:
> https://lists.alioth.debian.org/pipermail/pkg-kde-talk/2016-December/
> 002455.html
> 
> I know this is not ideal but we can't fix this. If this is a nogo for the 
> release than we need to remove the i386 packages, that used the fact, that 
> debian buildds for i386 are running on amd64 systems.

That strange, because my configuration is similar: amd64 box with a lot
of RAM, with a 32b chroot (using linux32). So I don't understand how the
build works on buildds but not there

Lucas

Bug#863927: qtwebengine-opensource-src: FTBFS: memory exhausted

[Sandro Knauß]

Hey,
 
> During a rebuild of all packages in stretch (in a stretch chroot, not a
> sid chroot), your package failed to build on i386.
> 
> Note that the build node has a lot of RAM, so if memory is really
> exhausted, it could mean that it can no longer be built on i386.

Well the memory consumptions are really at on limit for all porterboxes and it 
is only possible to build QtWebEngine on a 64bit systems or that the host 
system is a 64bit on. Otherwise the 32bit system can't access enough RAM/
diskspace, because it hits otherwise the 4GB limit for files. We had discussed 
this topic in a thread already:
https://lists.alioth.debian.org/pipermail/pkg-kde-talk/2016-December/
002455.html

I know this is not ideal but we can't fix this. If this is a nogo for the 
release than we need to remove the i386 packages, that used the fact, that 
debian buildds for i386 are running on amd64 systems.

Best regards,

sandro

--
> 
> Relevant part (hopefully):
> > make[1]: Entering directory
> > '/<>/qtwebengine-opensource-src-5.7.1+dfsg' dh_link
> > --package=qt5webengine-examples \
> > 
> > usr/share/javascript/jquery/jquery.min.js
> > usr/lib/i386-linux-gnu/qt5/examples/webenginewidgets/contentmanipulation
> > /jquery.min.js \ usr/share/javascript/marked/marked.min.js
> > usr/lib/i386-linux-gnu/qt5/examples/webenginewidgets/markdowneditor/reso
> > urces/marked.min.js> 
> > dh_link --remaining-packages
> > make[1]: Leaving directory
> > '/<>/qtwebengine-opensource-src-5.7.1+dfsg'> 
> >dh_strip_nondeterminism
> >dh_compress
> >dh_fixperms
> >dh_strip
> > 
> > objcopy:
> > debian/libqt5webenginecore5/usr/lib/i386-linux-gnu/libQt5WebEngineCore.so
> > .5.7.1: unable to initialize compress status for section .debug_info
> > objcopy:debian/libqt5webenginecore5/usr/lib/i386-linux-gnu/libQt5WebEngin
> > eCore.so.5.7.1: Memory exhausted dh_strip: objcopy --only-keep-debug
> > --compress-debug-sections
> > debian/libqt5webenginecore5/usr/lib/i386-linux-gnu/libQt5WebEngineCore.so
> > .5.7.1
> > debian/.debhelper/libqt5webenginecore5/dbgsym-root/usr/lib/debug/.build-i
> > d/d4/c82995202e374ab64d53e8f6f30fd1f5f1866f.debug returned exit code 1
> > debian/rules:68: recipe for target 'binary' failed
> > make: *** [binary] Error 2
> 
> The full build log is available from:
>   
> http://aws-logs.debian.net/2017/06/01/qtwebengine-opensource-src_5.7.1+dfsg
> -6_testing-i386.log
> 
> A list of current common problems and possible solutions is available at
> http://wiki.debian.org/qa.debian.org/FTBFS . You're welcome to contribute!
> 
> About the archive rebuild: The rebuild was done on EC2 VM instances from
> Amazon Web Services, using a clean, minimal and up-to-date chroot. Every
> failed build was retried once to eliminate random failures.



signature.asc
Description: This is a digitally signed message part.

Bug#863927: qtwebengine-opensource-src: FTBFS: memory exhausted

[Lucas Nussbaum]

Source: qtwebengine-opensource-src
Version: 5.7.1+dfsg-6
Severity: serious
Tags: stretch sid
User: debian...@lists.debian.org
Usertags: qa-ftbfs-20170601-i386 qa-ftbfs
Justification: FTBFS in stretch on i386

Hi,

During a rebuild of all packages in stretch (in a stretch chroot, not a
sid chroot), your package failed to build on i386.

Note that the build node has a lot of RAM, so if memory is really
exhausted, it could mean that it can no longer be built on i386.

Relevant part (hopefully):
> make[1]: Entering directory 
> '/<>/qtwebengine-opensource-src-5.7.1+dfsg'
> dh_link --package=qt5webengine-examples \
>   usr/share/javascript/jquery/jquery.min.js 
> usr/lib/i386-linux-gnu/qt5/examples/webenginewidgets/contentmanipulation/jquery.min.js
>  \
>   usr/share/javascript/marked/marked.min.js 
> usr/lib/i386-linux-gnu/qt5/examples/webenginewidgets/markdowneditor/resources/marked.min.js
> dh_link --remaining-packages
> make[1]: Leaving directory 
> '/<>/qtwebengine-opensource-src-5.7.1+dfsg'
>dh_strip_nondeterminism
>dh_compress
>dh_fixperms
>dh_strip
> objcopy: 
> debian/libqt5webenginecore5/usr/lib/i386-linux-gnu/libQt5WebEngineCore.so.5.7.1:
>  unable to initialize compress status for section .debug_info
> objcopy:debian/libqt5webenginecore5/usr/lib/i386-linux-gnu/libQt5WebEngineCore.so.5.7.1:
>  Memory exhausted
> dh_strip: objcopy --only-keep-debug --compress-debug-sections 
> debian/libqt5webenginecore5/usr/lib/i386-linux-gnu/libQt5WebEngineCore.so.5.7.1
>  
> debian/.debhelper/libqt5webenginecore5/dbgsym-root/usr/lib/debug/.build-id/d4/c82995202e374ab64d53e8f6f30fd1f5f1866f.debug
>  returned exit code 1
> debian/rules:68: recipe for target 'binary' failed
> make: *** [binary] Error 2

The full build log is available from:
   
http://aws-logs.debian.net/2017/06/01/qtwebengine-opensource-src_5.7.1+dfsg-6_testing-i386.log

A list of current common problems and possible solutions is available at
http://wiki.debian.org/qa.debian.org/FTBFS . You're welcome to contribute!

About the archive rebuild: The rebuild was done on EC2 VM instances from
Amazon Web Services, using a clean, minimal and up-to-date chroot. Every
failed build was retried once to eliminate random failures.