----- Original Message ----- > From: "Sandro Bonazzola" <[email protected]> > To: "Vojtech Szocs" <[email protected]> > Cc: "Tomas Jelinek" <[email protected]>, "Juan Hernandez" > <[email protected]>, "Kiril Nesenko" > <[email protected]>, "arch" <[email protected]> > Sent: Wednesday, February 26, 2014 5:37:20 PM > Subject: Re: [urgent] Re: ovirt-engine build failure > > Il 26/02/2014 16:40, Vojtech Szocs ha scritto: > > (top posting) > > > > I did some preliminary tests on my local machine, invoking "mvn clean > > install -Pgwt-admin" command to build (only) WebAdmin from source. > > > > Default permutation settings: all (5) browsers, one locale = 5 total > > permutations to compile > > > > Tested two different strategies (machine with 16G RAM / 8G swap, quad-core > > CPU): > > A, localWorkers=X (default external-JVM worker factory) > > -Dgwt.compiler.localWorkers=X > > B, localWorkers=X + maxThreads=X (thread-based worker factory) > > -Dgwt.compiler.localWorkers=X > > > > -Dgwt.jjs.permutationWorkerFactory=com.google.gwt.dev.ThreadedPermutationWorkerFactory > > -Dgwt.jjs.maxThreads=X > > > > A -> this is what we use today > > B -> this is possible alternative > > > > X=4 [5 perms] > > A, initial 2.6 GiB, peak 13.5 GiB > > Total time: 5:25.851s > > B, initial 2.6 GiB, peak 13.1 GiB > > Total time: 5:11.238s > > > > X=1 [5 perms] > > A, initial 2.6 GiB, peak 7.1 GiB > > Total time: 7:00.601s > > B, initial 2.6 GiB, peak 7.4 GiB > > Total time: 7:07.458s > > > > Now switched to permutation settings: all (5) browsers, 3 locales = 15 > > total permutations to compile > > > > (Used swapoff/swapon to force swap clear after each test run.) > > > > X=4 [15 perms] > > A, initial 2.6 GiB, peak physical max + swap ~4.3 GiB > > Total time: 11:39.086s > > B, initial 2.6 GiB, peak physical max + swap ~3.5 GiB > > Total time: 10:59.966s > > > > X=1 [15 perms] > > A, initial 2.6 GiB, peak 7.7 GiB > > Total time: 17:39.103s > > > > B, initial 2.6 GiB. peak 7.9 GiB > > Total time: 17:16.314s > > > > For X=1, both strategies (A & B) have very similar results. > > > > For X=4 (more than one worker), thread-based strategy seems slightly > > better, but still we have huge RAM consumption + max opened files quite > > high. > > > > I think the only real solution is to cut off (eliminate) locale permutation > > vector to reduce amount of GWT permutations. It's on my task list. > > > > > Great! > BTW, raising max opened files to 10240 worked, thanks.
Submitted patch for master: http://gerrit.ovirt.org/#/c/25163/ Details in commit message, short summary: - GWT compiler will spawn thread(s) instead of Java sub-process(es) to compile specific permutation(s) - max worker thread upper bound is set to 4 - you can use gwt.compiler.localWorkers to control max worker thread value, i.e. -Dgwt.compiler.localWorkers=2 means max 2 worker threads -> min(2,4) However, I don't expect too much out of this optimization; eliminating GWT permutations is something we'll consider as long-term improvement. Vojtech > > > > Vojtech > > > > > > ----- Original Message ----- > >> From: "Sandro Bonazzola" <[email protected]> > >> To: "Vojtech Szocs" <[email protected]> > >> Cc: "Tomas Jelinek" <[email protected]>, "Juan Hernandez" > >> <[email protected]>, "Kiril Nesenko" > >> <[email protected]>, "arch" <[email protected]> > >> Sent: Wednesday, February 26, 2014 11:58:29 AM > >> Subject: Re: [urgent] Re: ovirt-engine build failure > >> > >> Il 26/02/2014 11:51, Vojtech Szocs ha scritto: > >>> > >>> > >>> ----- Original Message ----- > >>>> From: "Sandro Bonazzola" <[email protected]> > >>>> To: "Tomas Jelinek" <[email protected]> > >>>> Cc: "Vojtech Szocs" <[email protected]>, "Juan Hernandez" > >>>> <[email protected]>, "Kiril Nesenko" > >>>> <[email protected]>, "arch" <[email protected]> > >>>> Sent: Wednesday, February 26, 2014 10:22:22 AM > >>>> Subject: Re: [urgent] Re: ovirt-engine build failure > >>>> > >>>> Il 26/02/2014 09:06, Tomas Jelinek ha scritto: > >>>>> Hey, > >>>>> > >>>>> from this logs it seems that the root cause is: > >>>>> [ERROR] java.io.FileNotFoundException: > >>>>> /home/jenkins/workspace/manual-build-tarball/label/centos6-host/rpmbuild/BUILD/ovirt-engine-3.5.0/frontend/webadmin/modules/webadmin/target/tmp/ImageResourceGenerator4201639982561998434.png > >>>>> (Too many open files) > >>>>> > >>>>> e.g. the OS does not allow to open all the files the gwt compilation > >>>>> needs. > >>>>> Could you please try to increase the limit of max opened files on the > >>>>> host > >>>>> and try to rebuild? > >>>> > >>>> David is increasing max open files. We'll give it another run. > >>>> I really think that having to reduce localWorkers to 1 for having decent > >>>> build requirements on RAM and still hit too many opened files is really > >>>> bad. > >>>> Any chance of lowering build requirements in terms of RAM and opened > >>>> files? > >>> > >>> In general, GWT compiler generates/reads lots of intermediate files > >>> during > >>> its runtime. IIRC, this file amount is proportional to number of > >>> permutations > >>> being compiled, i.e. num_of_browsers x num_of_locales. This is why we > >>> don't > >>> hit "Too many open files" in small-scale builds (i.e. development) but we > >>> do > >>> hit it on large-scale builds (i.e. all_browsers x all_locales nightly > >>> build). > >>> > >>> But is increasing max open files for GWT compiler really that big of a > >>> deal? > >>> (i.e. we know that compiler needs to process lots of files per each > >>> permutation) > >> > >> No, it's not that big of a deal *for our packaging* but since we've people > >> on > >> Gentoo building engine > >> on their own systems, it may impact all of them... > >> > >>> > >>> Alternatively, we could implement "remote GWT worker grid" -> offload > >>> compilation > >>> for each specific (browser x locale) permutation to a remote machine, > >>> this > >>> would > >>> require some extra Java implementation though. > >>> > >>> I agree on excessive RAM usage, I'm currently experimenting with some > >>> compiler > >>> options, I'll share results when I have something. > >>> > >>> Another approach we could take (something I'll discuss with my peers) is > >>> the > >>> idea of eliminating locale vector entirely -> re-implement i18n support > >>> in > >>> UI > >>> in a way that for X browsers, there will be only X total permutations > >>> (i.e. > >>> not X x num_of_locales). This would reduce RAM usage && compilation time > >>> && > >>> perhaps max files opened too. > >> > >> +1 on whatever can reduce build time and build requirements. > >> > >> > >>> > >>>> > >>>> > >>>>> > >>>>> ----- Original Message ----- > >>>>>> From: "Sandro Bonazzola" <[email protected]> > >>>>>> To: "Vojtech Szocs" <[email protected]> > >>>>>> Cc: "Juan Hernandez" <[email protected]>, "Kiril Nesenko" > >>>>>> <[email protected]>, "arch" <[email protected]> > >>>>>> Sent: Wednesday, February 26, 2014 8:37:07 AM > >>>>>> Subject: Re: [urgent] Re: ovirt-engine build failure > >>>>>> > >>>>>> Il 25/02/2014 16:52, Vojtech Szocs ha scritto: > >>>>>>> > >>>>>>> > >>>>>>> ----- Original Message ----- > >>>>>>>> From: "Vojtech Szocs" <[email protected]> > >>>>>>>> To: "Sandro Bonazzola" <[email protected]> > >>>>>>>> Cc: "Juan Hernandez" <[email protected]>, "Kiril Nesenko" > >>>>>>>> <[email protected]>, "arch" <[email protected]> > >>>>>>>> Sent: Tuesday, February 25, 2014 1:05:35 PM > >>>>>>>> Subject: Re: [urgent] Re: ovirt-engine build failure > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> ----- Original Message ----- > >>>>>>>>> From: "Sandro Bonazzola" <[email protected]> > >>>>>>>>> To: "Vojtech Szocs" <[email protected]> > >>>>>>>>> Cc: "Juan Hernandez" <[email protected]>, "arch" > >>>>>>>>> <[email protected]>, > >>>>>>>>> "Kiril > >>>>>>>>> Nesenko" <[email protected]> > >>>>>>>>> Sent: Tuesday, February 25, 2014 9:02:11 AM > >>>>>>>>> Subject: Re: [urgent] Re: ovirt-engine build failure > >>>>>>>>> > >>>>>>>>> Il 25/02/2014 08:47, Sandro Bonazzola ha scritto: > >>>>>>>>>> Il 24/02/2014 18:28, Vojtech Szocs ha scritto: > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> ----- Original Message ----- > >>>>>>>>>>>> From: "Sandro Bonazzola" <[email protected]> > >>>>>>>>>>>> To: "Vojtech Szocs" <[email protected]> > >>>>>>>>>>>> Cc: "Eyal Edri" <[email protected]>, "Juan Hernandez" > >>>>>>>>>>>> <[email protected]>, "Kiril Nesenko" <[email protected]>, > >>>>>>>>>>>> "arch" <[email protected]> > >>>>>>>>>>>> Sent: Monday, February 24, 2014 9:08:09 AM > >>>>>>>>>>>> Subject: [urgent] Re: ovirt-engine build failure > >>>>>>>>>>>> > >>>>>>>>>>>> Top posting, > >>>>>>>>>>>> > >>>>>>>>>>>> Please give this task really urgent priority, as you can see: > >>>>>>>>>>>> > >>>>>>>>>>>> http://jenkins.ovirt.org/job/ovirt-engine_3.4_create_rpms/ > >>>>>>>>>>>> and > >>>>>>>>>>>> http://jenkins.ovirt.org/job/ovirt-engine_master_create_rpms/ > >>>>>>>>>>>> > >>>>>>>>>>>> are failing consistently so we don't have nightly builds. > >>>>>>>>>>> > >>>>>>>>>>> (CC'ing Einav) > >>>>>>>>>>> > >>>>>>>>>>> Attached TEST patch, please apply & rebuild ovirt-engine as > >>>>>>>>>>> usual. > >>>>>>>>>>> > >>>>>>>>>>> This patch should cause following output during GWT compilation: > >>>>>>>>>>> > >>>>>>>>>>> [ERROR] ######### Util.readURLAsBytes > >>>>>>>>>>> file:/home/vszocs/work/ovirt-engine/frontend/webadmin/modules/webadmin/target/generated-sources/gwt/org/ovirt/engine/ui/webadmin/images/search_button.png > >>>>>>>>>>> > >>>>>>>>>>> If IOException occurs, exception stacktrace will be printed right > >>>>>>>>>>> below > >>>>>>>>>>> that line. > >>>>>>>>>>> > >>>>>>>>>>> Note: please ignore the [ERROR] prefix, it's a technical issue, > >>>>>>>>>>> more > >>>>>>>>>>> details at > >>>>>>>>>>> https://www.mail-archive.com/[email protected]/msg05735.html > >>>>>>>>>>> > >>>>>>>>>>> I just did a local upstream build with TEST patch applied, GWT > >>>>>>>>>>> compilation > >>>>>>>>>>> passed OK. > >>>>>>>>>> > >>>>>>>>>> build job running with your patch here: > >>>>>>>>>> http://jenkins.ovirt.org/job/manual-build-tarball/235/ > >>>>>>>>>> waiting for a free executor right now. > >>>>>>>>>> Let's see how it goes. > >>>>>>>>> > >>>>>>>>> Failed on CentOS: > >>>>>>>>> http://jenkins.ovirt.org/job/manual-build-tarball/235/label=centos6-host/consoleFull > >>>>>>>> > >>>>>>>> Hm, in that ^^ build, following exceptions occured: > >>>>>>>> > >>>>>>>> 1. > >>>>>>>> [INFO] java.lang.NullPointerException > >>>>>>>> [INFO] at > >>>>>>>> com.google.gwt.dev.util.Util.computeStrongName(Util.java:172) > >>>>>>>> > >>>>>>>> 2. > >>>>>>>> [INFO] java.lang.NullPointerException: No ImageRect ever computed > >>>>>>>> for > >>>>>>>> downDisabledImage > >>>>>>>> [INFO] at > >>>>>>>> com.google.gwt.resources.rg.ImageResourceGenerator.createAssignment(ImageResourceGenerator.java:475 > >>>>>>>> > >>>>>>>> 1. is caused by Util#readURLAsBytes but I don't see any IOException > >>>>>>>> in > >>>>>>>> the > >>>>>>>> logs > >>>>>>>> 2. is just follow-up consequence of 1. (i.e. image metadata missing > >>>>>>>> in > >>>>>>>> GWT > >>>>>>>> image cache) > >>>>>>>> > >>>>>>>> Now I noticed I missed one possible IOException inside > >>>>>>>> Util#readURLConnectionAsBytes ... > >>>>>>>> > >>>>>>>> Long story short, I'll send updated TEST patch soon, apologies for > >>>>>>>> the > >>>>>>>> inconvenience. > >>>>>>> > >>>>>>> TEST-v2 patch attached. Please apply & rebuild ovirt-engine as usual. > >>>>>> > >>>>>> Done, here is the console output: > >>>>>> http://jenkins.ovirt.org/job/manual-build-tarball/label=centos6-host/239/consoleFull > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>>> > >>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> Il 20/02/2014 13:34, Vojtech Szocs ha scritto: > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> ----- Original Message ----- > >>>>>>>>>>>>>> From: "Sandro Bonazzola" <[email protected]> > >>>>>>>>>>>>>> To: "Vojtech Szocs" <[email protected]> > >>>>>>>>>>>>>> Cc: "Eyal Edri" <[email protected]>, "Juan Hernandez" > >>>>>>>>>>>>>> <[email protected]>, "Kiril Nesenko" <[email protected]>, > >>>>>>>>>>>>>> "arch" <[email protected]> > >>>>>>>>>>>>>> Sent: Thursday, February 20, 2014 10:02:00 AM > >>>>>>>>>>>>>> Subject: Re: ovirt-engine build failure > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> Il 19/02/2014 16:51, Vojtech Szocs ha scritto: > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> ----- Original Message ----- > >>>>>>>>>>>>>>>> From: "Sandro Bonazzola" <[email protected]> > >>>>>>>>>>>>>>>> To: "Vojtech Szocs" <[email protected]> > >>>>>>>>>>>>>>>> Cc: "Eyal Edri" <[email protected]>, "Juan Hernandez" > >>>>>>>>>>>>>>>> <[email protected]>, "Kiril Nesenko" > >>>>>>>>>>>>>>>> <[email protected]>, > >>>>>>>>>>>>>>>> "arch" <[email protected]> > >>>>>>>>>>>>>>>> Sent: Wednesday, February 19, 2014 2:43:44 PM > >>>>>>>>>>>>>>>> Subject: Re: ovirt-engine build failure > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> Il 19/02/2014 14:41, Vojtech Szocs ha scritto: > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> ----- Original Message ----- > >>>>>>>>>>>>>>>>>> From: "Sandro Bonazzola" <[email protected]> > >>>>>>>>>>>>>>>>>> To: "Vojtech Szocs" <[email protected]> > >>>>>>>>>>>>>>>>>> Cc: "Eyal Edri" <[email protected]>, "Juan Hernandez" > >>>>>>>>>>>>>>>>>> <[email protected]>, "Kiril Nesenko" > >>>>>>>>>>>>>>>>>> <[email protected]>, > >>>>>>>>>>>>>>>>>> "arch" <[email protected]> > >>>>>>>>>>>>>>>>>> Sent: Wednesday, February 19, 2014 2:23:16 PM > >>>>>>>>>>>>>>>>>> Subject: Re: ovirt-engine build failure > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> Il 19/02/2014 14:19, Vojtech Szocs ha scritto: > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> ----- Original Message ----- > >>>>>>>>>>>>>>>>>>>> From: "Sandro Bonazzola" <[email protected]> > >>>>>>>>>>>>>>>>>>>> To: "Vojtech Szocs" <[email protected]> > >>>>>>>>>>>>>>>>>>>> Cc: "Eyal Edri" <[email protected]>, "Juan Hernandez" > >>>>>>>>>>>>>>>>>>>> <[email protected]>, "Kiril Nesenko" > >>>>>>>>>>>>>>>>>>>> <[email protected]>, > >>>>>>>>>>>>>>>>>>>> "arch" <[email protected]> > >>>>>>>>>>>>>>>>>>>> Sent: Wednesday, February 19, 2014 1:18:28 PM > >>>>>>>>>>>>>>>>>>>> Subject: Re: ovirt-engine build failure > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> Il 19/02/2014 12:45, Vojtech Szocs ha scritto: > >>>>>>>>>>>>>>>>>>>>> Hi, I checked GWT sources: > >>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> [INFO] java.lang.NullPointerException: No ImageRect > >>>>>>>>>>>>>>>>>>>>>> ever > >>>>>>>>>>>>>>>>>>>>>> computed > >>>>>>>>>>>>>>>>>>>>>> for > >>>>>>>>>>>>>>>>>>>>>> removeBrickCommitRequired > >>>>>>>>>>>>>>>>>>>>>> [INFO] at > >>>>>>>>>>>>>>>>>>>>>> com.google.gwt.resources.rg.ImageResourceGenerator.createAssignment(ImageResourceGenerator.java:475) > >>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>> This happened because internal image cache for bundled > >>>>>>>>>>>>>>>>>>>>> (composite > >>>>>>>>>>>>>>>>>>>>> PNG) > >>>>>>>>>>>>>>>>>>>>> images was corrput. > >>>>>>>>>>>>>>>>>>>>> It seems to be caused by disk I/O error while reading > >>>>>>>>>>>>>>>>>>>>> file > >>>>>>>>>>>>>>>>>>>>> images/glusterAsyncTasks/remove_brick_commit_required.png > >>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> [INFO] java.lang.NullPointerException > >>>>>>>>>>>>>>>>>>>>>> [INFO] at > >>>>>>>>>>>>>>>>>>>>>> com.google.gwt.resources.rebind.context.InlineResourceContext.deploy(InlineResourceContext.java:40) > >>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>> This happened because image data was null, again > >>>>>>>>>>>>>>>>>>>>> possible > >>>>>>>>>>>>>>>>>>>>> disk > >>>>>>>>>>>>>>>>>>>>> I/O > >>>>>>>>>>>>>>>>>>>>> error > >>>>>>>>>>>>>>>>>>>>> while reading image file. > >>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>> Can we enable "TRACE" logging during GWT compilation? > >>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>> -Dgwt.logLevel=TRACE > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> Building master right now with your suggestions here: > >>>>>>>>>>>>>>>>>>>> http://jenkins.ovirt.org/job/manual-build-tarball/227/ > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> Looking at > >>>>>>>>>>>>>>>>>>> http://jenkins.ovirt.org/job/manual-build-tarball/227/label=fedora20-host/console > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> So far I see no errors.. There should be total 5 browsers > >>>>>>>>>>>>>>>>>>> x > >>>>>>>>>>>>>>>>>>> 8 > >>>>>>>>>>>>>>>>>>> locales > >>>>>>>>>>>>>>>>>>> = > >>>>>>>>>>>>>>>>>>> 40 > >>>>>>>>>>>>>>>>>>> GWT permutations. > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> So no changes with and without > >>>>>>>>>>>>>>>>>> -Dgwt.userAgent=ie8,ie9,gecko1_8,safari,opera. > >>>>>>>>>>>>>>>>>> We've still 40 permutations. Maybe you defined > >>>>>>>>>>>>>>>>>> userAgent=ie8,ie9,gecko1_8,safari,opera somewhere in the > >>>>>>>>>>>>>>>>>> POM > >>>>>>>>>>>>>>>>>> files? > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> You are right. I checked frontend/webadmin/modules/pom.xml > >>>>>>>>>>>>>>>>> -> > >>>>>>>>>>>>>>>>> the > >>>>>>>>>>>>>>>>> "gwt.userAgent" > >>>>>>>>>>>>>>>>> property is copied into "locale" GWT property in > >>>>>>>>>>>>>>>>> {WebAdmin,UserPortal}.gwt.xml > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> Ok, so I'll remove > >>>>>>>>>>>>>>>> -Dgwt.userAgent=ie8,ie9,gecko1_8,safari,opera > >>>>>>>>>>>>>>>> from > >>>>>>>>>>>>>>>> build > >>>>>>>>>>>>>>>> command. > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> Agreed. > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> This: > >>>>>>>>>>>>>>>>>> http://jenkins.ovirt.org/job/manual-build-tarball/227/label=centos6-host/console > >>>>>>>>>>>>>>>>>> failed, you can look at it, it has -Dgwt.logLevel=TRACE > >>>>>>>>>>>>>>>>>> -X. > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> Hm, it contains: > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> [DEBUG] (f) logLevel = INFO > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> Somehow the TRACE log level for GWT wasn't applied.. not > >>>>>>>>>>>>>>>>> sure > >>>>>>>>>>>>>>>>> why. > >>>>>>>>>>>>>>>>> Let > >>>>>>>>>>>>>>>>> me > >>>>>>>>>>>>>>>>> check > >>>>>>>>>>>>>>>>> this on local build. > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> Ok, let me know if I've to change command line. > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> I wrote upstream patch: http://gerrit.ovirt.org/#/c/24789/ > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> When it's merged, setting "gwt.logLevel" will work (as you > >>>>>>>>>>>>>>> did > >>>>>>>>>>>>>>> before) > >>>>>>>>>>>>>>> and we'll be able to see more details in the build log. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> Here is the new build with your patch merged: > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> http://jenkins.ovirt.org/job/manual-build-tarball/228/label=centos6-host/consoleFull > >>>>>>>>>>>>> > >>>>>>>>>>>>> Thanks. I did some more investigation on this. > >>>>>>>>>>>>> > >>>>>>>>>>>>> Again it's NullPointerException when rebinding > >>>>>>>>>>>>> ApplicationResources, > >>>>>>>>>>>>> but > >>>>>>>>>>>>> this time for different image file: > >>>>>>>>>>>>> ApplicationResources.wrenchImage > >>>>>>>>>>>>> ("images/wrench.png"). > >>>>>>>>>>>>> > >>>>>>>>>>>>> Looking at stacktrace & GWT sources, the problematic code seems > >>>>>>>>>>>>> to > >>>>>>>>>>>>> be > >>>>>>>>>>>>> (AbstractResourceContext:74): > >>>>>>>>>>>>> > >>>>>>>>>>>>> byte[] bytes = Util.readURLAsBytes(resource); > >>>>>>>>>>>>> > >>>>>>>>>>>>> where "resource" is (ImageResourceGenerator:311): > >>>>>>>>>>>>> > >>>>>>>>>>>>> localized.getUrl(); // LocalizedImage localized > >>>>>>>>>>>>> > >>>>>>>>>>>>> Note: in oVirt UI we don't use localized images (i.e. > >>>>>>>>>>>>> "wrench_fr_FR.png") > >>>>>>>>>>>>> so "localized" in this case points to "wrench.png". > >>>>>>>>>>>>> > >>>>>>>>>>>>> So the problem seems to be reading this file: > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> frontend/webadmin/modules/webadmin/src/main/resources/org/ovirt/engine/ui/webadmin/images/wrench.png > >>>>>>>>>>>>> > >>>>>>>>>>>>> Since we're having many GWT permutations, this file ^^ (along > >>>>>>>>>>>>> with > >>>>>>>>>>>>> similar > >>>>>>>>>>>>> ones) is attempted to be read multiple times *from the same > >>>>>>>>>>>>> (single) > >>>>>>>>>>>>> GWT > >>>>>>>>>>>>> compiler Java process, before actual permutation worker Java > >>>>>>>>>>>>> sub-processes > >>>>>>>>>>>>> are spawned*. (More precisely, above NullPointerException > >>>>>>>>>>>>> happens > >>>>>>>>>>>>> at > >>>>>>>>>>>>> deferred binding phase which precedes actual GWT compilation > >>>>>>>>>>>>> per > >>>>>>>>>>>>> each > >>>>>>>>>>>>> permutation.) > >>>>>>>>>>>>> > >>>>>>>>>>>>> I think the root cause is Util.readURLAsBytes method used > >>>>>>>>>>>>> internally > >>>>>>>>>>>>> by > >>>>>>>>>>>>> GWT > >>>>>>>>>>>>> compiler: > >>>>>>>>>>>>> > >>>>>>>>>>>>> public static byte[] readURLAsBytes(URL url) { > >>>>>>>>>>>>> try { > >>>>>>>>>>>>> URLConnection conn = url.openConnection(); > >>>>>>>>>>>>> conn.setUseCaches(false); > >>>>>>>>>>>>> return readURLConnectionAsBytes(conn); > >>>>>>>>>>>>> } catch (IOException e) { > >>>>>>>>>>>>> return null; // This causes NullPointerException, but > >>>>>>>>>>>>> this > >>>>>>>>>>>>> exception > >>>>>>>>>>>>> isn't logged (wtf, GWT!) > >>>>>>>>>>>>> } > >>>>>>>>>>>>> } > >>>>>>>>>>>>> > >>>>>>>>>>>>> To summarize, I'll need to write another patch which replaces > >>>>>>>>>>>>> Util.readURLAsBytes implementation via AOP and prints > >>>>>>>>>>>>> IOException > >>>>>>>>>>>>> details. > >>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> # > >>>>>>>>>>>>>>>>>>>> CI_MAVEN_SETTINGS=$WORKSPACE/artifactory-ovirt-org-settings.xml > >>>>>>>>>>>>>>>>>>>> # export MAVEN_OPTS="${MAVEN_OPTS} -Xmx2048M > >>>>>>>>>>>>>>>>>>>> -XX:MaxPermSize=1024m > >>>>>>>>>>>>>>>>>>>> -Dgwt.compiler.localWorkers=1" > >>>>>>>>>>>>>>>>>>>> # rpmbuild -D "_topdir $(pwd)/rpmbuild" -ts "${TARBALL}" > >>>>>>>>>>>>>>>>>>>> # rpmbuild -D "_topdir $(pwd)/rpmbuild" -D > >>>>>>>>>>>>>>>>>>>> "ovirt_build_extra_flags > >>>>>>>>>>>>>>>>>>>> -gs > >>>>>>>>>>>>>>>>>>>> ${CI_MAVEN_SETTINGS} > >>>>>>>>>>>>>>>>>>>> -Dgwt.userAgent=ie8,ie9,gecko1_8,safari,opera > >>>>>>>>>>>>>>>>>>>> -Dgwt.logLevel=TRACE -X" --rebuild > >>>>>>>>>>>>>>>>>>>> rpmbuild/SRPMS/*.src.rpm > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> Let me know if you need anything else. > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>> Vojtech > >>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>> ----- Original Message ----- > >>>>>>>>>>>>>>>>>>>>>> From: "Vojtech Szocs" <[email protected]> > >>>>>>>>>>>>>>>>>>>>>> To: "Sandro Bonazzola" <[email protected]> > >>>>>>>>>>>>>>>>>>>>>> Cc: "Eyal Edri" <[email protected]>, "Juan Hernandez" > >>>>>>>>>>>>>>>>>>>>>> <[email protected]>, "Kiril Nesenko" > >>>>>>>>>>>>>>>>>>>>>> <[email protected]>, > >>>>>>>>>>>>>>>>>>>>>> "arch" <[email protected]> > >>>>>>>>>>>>>>>>>>>>>> Sent: Tuesday, February 18, 2014 6:54:20 PM > >>>>>>>>>>>>>>>>>>>>>> Subject: Re: ovirt-engine build failure > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> In your log I see following error: > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> ... > >>>>>>>>>>>>>>>>>>>>>> [INFO] Computing all possible rebind results for > >>>>>>>>>>>>>>>>>>>>>> 'org.ovirt.engine.ui.webadmin.ApplicationResources' > >>>>>>>>>>>>>>>>>>>>>> [INFO] Rebinding > >>>>>>>>>>>>>>>>>>>>>> org.ovirt.engine.ui.webadmin.ApplicationResources > >>>>>>>>>>>>>>>>>>>>>> [INFO] Invoking generator > >>>>>>>>>>>>>>>>>>>>>> com.google.gwt.resources.rebind.context.InlineClientBundleGenerator > >>>>>>>>>>>>>>>>>>>>>> [INFO] [ERROR] Generator > >>>>>>>>>>>>>>>>>>>>>> 'com.google.gwt.resources.rebind.context.InlineClientBundleGenerator' > >>>>>>>>>>>>>>>>>>>>>> threw > >>>>>>>>>>>>>>>>>>>>>> an exception while rebinding > >>>>>>>>>>>>>>>>>>>>>> 'org.ovirt.engine.ui.webadmin.ApplicationResources' > >>>>>>>>>>>>>>>>>>>>>> [INFO] java.lang.NullPointerException: No ImageRect > >>>>>>>>>>>>>>>>>>>>>> ever > >>>>>>>>>>>>>>>>>>>>>> computed > >>>>>>>>>>>>>>>>>>>>>> for > >>>>>>>>>>>>>>>>>>>>>> removeBrickCommitRequired > >>>>>>>>>>>>>>>>>>>>>> [INFO] at > >>>>>>>>>>>>>>>>>>>>>> com.google.gwt.resources.rg.ImageResourceGenerator.createAssignment(ImageResourceGenerator.java:475) > >>>>>>>>>>>>>>>>>>>>>> ... > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> I checked again logs at > >>>>>>>>>>>>>>>>>>>>>> http://jenkins.ovirt.org/job/manual-build-tarball/212/label=fedora19-host/consoleFull > >>>>>>>>>>>>>>>>>>>>>> and indeed there is a similar error: > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> ... > >>>>>>>>>>>>>>>>>>>>>> [INFO] Computing all possible rebind results for > >>>>>>>>>>>>>>>>>>>>>> 'org.ovirt.engine.ui.webadmin.ApplicationResources' > >>>>>>>>>>>>>>>>>>>>>> [INFO] Rebinding > >>>>>>>>>>>>>>>>>>>>>> org.ovirt.engine.ui.webadmin.ApplicationResources > >>>>>>>>>>>>>>>>>>>>>> [INFO] Invoking generator > >>>>>>>>>>>>>>>>>>>>>> com.google.gwt.resources.rebind.context.InlineClientBundleGenerator > >>>>>>>>>>>>>>>>>>>>>> [INFO] [ERROR] Generator > >>>>>>>>>>>>>>>>>>>>>> 'com.google.gwt.resources.rebind.context.InlineClientBundleGenerator' > >>>>>>>>>>>>>>>>>>>>>> threw > >>>>>>>>>>>>>>>>>>>>>> an exception while rebinding > >>>>>>>>>>>>>>>>>>>>>> 'org.ovirt.engine.ui.webadmin.ApplicationResources' > >>>>>>>>>>>>>>>>>>>>>> [INFO] java.lang.NullPointerException > >>>>>>>>>>>>>>>>>>>>>> [INFO] at > >>>>>>>>>>>>>>>>>>>>>> com.google.gwt.resources.rebind.context.InlineResourceContext.deploy(InlineResourceContext.java:40) > >>>>>>>>>>>>>>>>>>>>>> ... > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> NullPointerException when processing > >>>>>>>>>>>>>>>>>>>>>> ApplicationResources > >>>>>>>>>>>>>>>>>>>>>> but > >>>>>>>>>>>>>>>>>>>>>> occuring > >>>>>>>>>>>>>>>>>>>>>> from > >>>>>>>>>>>>>>>>>>>>>> different places (ImageResourceGenerator vs. > >>>>>>>>>>>>>>>>>>>>>> InlineResourceContext) > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> Now I understand why Maven build fails without any > >>>>>>>>>>>>>>>>>>>>>> details: > >>>>>>>>>>>>>>>>>>>>>> - GWT deferred binding fails for ApplicationResources > >>>>>>>>>>>>>>>>>>>>>> (i.e. > >>>>>>>>>>>>>>>>>>>>>> uncaught > >>>>>>>>>>>>>>>>>>>>>> NPE) > >>>>>>>>>>>>>>>>>>>>>> - GWT compilation proceeds, fails to link with > >>>>>>>>>>>>>>>>>>>>>> ApplicationResources > >>>>>>>>>>>>>>>>>>>>>> - GWT compilation task fails with non-OK status code > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> So the issue here is GWT deferred binding generator > >>>>>>>>>>>>>>>>>>>>>> (InlineClientBundleGenerator) failure during GWT > >>>>>>>>>>>>>>>>>>>>>> compilation. > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> GWT compiler writes extra Java files during deferred > >>>>>>>>>>>>>>>>>>>>>> binding > >>>>>>>>>>>>>>>>>>>>>> phase, so this could point to issue with local > >>>>>>>>>>>>>>>>>>>>>> environment.. > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> I will try to investigate this and look at GWT source > >>>>>>>>>>>>>>>>>>>>>> code. > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> -- > >>>>>>>>>>>>>>>>>>>> Sandro Bonazzola > >>>>>>>>>>>>>>>>>>>> Better technology. Faster innovation. Powered by > >>>>>>>>>>>>>>>>>>>> community > >>>>>>>>>>>>>>>>>>>> collaboration. > >>>>>>>>>>>>>>>>>>>> See how it works at redhat.com > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> -- > >>>>>>>>>>>>>>>>>> Sandro Bonazzola > >>>>>>>>>>>>>>>>>> Better technology. Faster innovation. Powered by community > >>>>>>>>>>>>>>>>>> collaboration. > >>>>>>>>>>>>>>>>>> See how it works at redhat.com > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> -- > >>>>>>>>>>>>>>>> Sandro Bonazzola > >>>>>>>>>>>>>>>> Better technology. Faster innovation. Powered by community > >>>>>>>>>>>>>>>> collaboration. > >>>>>>>>>>>>>>>> See how it works at redhat.com > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> -- > >>>>>>>>>>>>>> Sandro Bonazzola > >>>>>>>>>>>>>> Better technology. Faster innovation. Powered by community > >>>>>>>>>>>>>> collaboration. > >>>>>>>>>>>>>> See how it works at redhat.com > >>>>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> -- > >>>>>>>>>>>> Sandro Bonazzola > >>>>>>>>>>>> Better technology. Faster innovation. Powered by community > >>>>>>>>>>>> collaboration. > >>>>>>>>>>>> See how it works at redhat.com > >>>>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> -- > >>>>>>>>> Sandro Bonazzola > >>>>>>>>> Better technology. Faster innovation. Powered by community > >>>>>>>>> collaboration. > >>>>>>>>> See how it works at redhat.com > >>>>>>>>> > >>>>>>>> _______________________________________________ > >>>>>>>> Arch mailing list > >>>>>>>> [email protected] > >>>>>>>> http://lists.ovirt.org/mailman/listinfo/arch > >>>>>>>> > >>>>>> > >>>>>> > >>>>>> -- > >>>>>> Sandro Bonazzola > >>>>>> Better technology. Faster innovation. Powered by community > >>>>>> collaboration. > >>>>>> See how it works at redhat.com > >>>>>> _______________________________________________ > >>>>>> Arch mailing list > >>>>>> [email protected] > >>>>>> http://lists.ovirt.org/mailman/listinfo/arch > >>>>>> > >>>> > >>>> > >>>> -- > >>>> Sandro Bonazzola > >>>> Better technology. Faster innovation. Powered by community > >>>> collaboration. > >>>> See how it works at redhat.com > >>>> > >> > >> > >> -- > >> Sandro Bonazzola > >> Better technology. Faster innovation. Powered by community collaboration. > >> See how it works at redhat.com > >> > > > -- > Sandro Bonazzola > Better technology. Faster innovation. Powered by community collaboration. > See how it works at redhat.com > _______________________________________________ Arch mailing list [email protected] http://lists.ovirt.org/mailman/listinfo/arch
