Il 26/02/2014 16:40, Vojtech Szocs ha scritto: > (top posting) > > I did some preliminary tests on my local machine, invoking "mvn clean install > -Pgwt-admin" command to build (only) WebAdmin from source. > > Default permutation settings: all (5) browsers, one locale = 5 total > permutations to compile > > Tested two different strategies (machine with 16G RAM / 8G swap, quad-core > CPU): > A, localWorkers=X (default external-JVM worker factory) > -Dgwt.compiler.localWorkers=X > B, localWorkers=X + maxThreads=X (thread-based worker factory) > -Dgwt.compiler.localWorkers=X > -Dgwt.jjs.permutationWorkerFactory=com.google.gwt.dev.ThreadedPermutationWorkerFactory > -Dgwt.jjs.maxThreads=X > > A -> this is what we use today > B -> this is possible alternative > > X=4 [5 perms] > A, initial 2.6 GiB, peak 13.5 GiB > Total time: 5:25.851s > B, initial 2.6 GiB, peak 13.1 GiB > Total time: 5:11.238s > > X=1 [5 perms] > A, initial 2.6 GiB, peak 7.1 GiB > Total time: 7:00.601s > B, initial 2.6 GiB, peak 7.4 GiB > Total time: 7:07.458s > > Now switched to permutation settings: all (5) browsers, 3 locales = 15 total > permutations to compile > > (Used swapoff/swapon to force swap clear after each test run.) > > X=4 [15 perms] > A, initial 2.6 GiB, peak physical max + swap ~4.3 GiB > Total time: 11:39.086s > B, initial 2.6 GiB, peak physical max + swap ~3.5 GiB > Total time: 10:59.966s > > X=1 [15 perms] > A, initial 2.6 GiB, peak 7.7 GiB > Total time: 17:39.103s > > B, initial 2.6 GiB. peak 7.9 GiB > Total time: 17:16.314s > > For X=1, both strategies (A & B) have very similar results. > > For X=4 (more than one worker), thread-based strategy seems slightly better, > but still we have huge RAM consumption + max opened files quite high. > > I think the only real solution is to cut off (eliminate) locale permutation > vector to reduce amount of GWT permutations. It's on my task list. >
Great! BTW, raising max opened files to 10240 worked, thanks. > Vojtech > > > ----- Original Message ----- >> From: "Sandro Bonazzola" <[email protected]> >> To: "Vojtech Szocs" <[email protected]> >> Cc: "Tomas Jelinek" <[email protected]>, "Juan Hernandez" >> <[email protected]>, "Kiril Nesenko" >> <[email protected]>, "arch" <[email protected]> >> Sent: Wednesday, February 26, 2014 11:58:29 AM >> Subject: Re: [urgent] Re: ovirt-engine build failure >> >> Il 26/02/2014 11:51, Vojtech Szocs ha scritto: >>> >>> >>> ----- Original Message ----- >>>> From: "Sandro Bonazzola" <[email protected]> >>>> To: "Tomas Jelinek" <[email protected]> >>>> Cc: "Vojtech Szocs" <[email protected]>, "Juan Hernandez" >>>> <[email protected]>, "Kiril Nesenko" >>>> <[email protected]>, "arch" <[email protected]> >>>> Sent: Wednesday, February 26, 2014 10:22:22 AM >>>> Subject: Re: [urgent] Re: ovirt-engine build failure >>>> >>>> Il 26/02/2014 09:06, Tomas Jelinek ha scritto: >>>>> Hey, >>>>> >>>>> from this logs it seems that the root cause is: >>>>> [ERROR] java.io.FileNotFoundException: >>>>> /home/jenkins/workspace/manual-build-tarball/label/centos6-host/rpmbuild/BUILD/ovirt-engine-3.5.0/frontend/webadmin/modules/webadmin/target/tmp/ImageResourceGenerator4201639982561998434.png >>>>> (Too many open files) >>>>> >>>>> e.g. the OS does not allow to open all the files the gwt compilation >>>>> needs. >>>>> Could you please try to increase the limit of max opened files on the >>>>> host >>>>> and try to rebuild? >>>> >>>> David is increasing max open files. We'll give it another run. >>>> I really think that having to reduce localWorkers to 1 for having decent >>>> build requirements on RAM and still hit too many opened files is really >>>> bad. >>>> Any chance of lowering build requirements in terms of RAM and opened >>>> files? >>> >>> In general, GWT compiler generates/reads lots of intermediate files during >>> its runtime. IIRC, this file amount is proportional to number of >>> permutations >>> being compiled, i.e. num_of_browsers x num_of_locales. This is why we don't >>> hit "Too many open files" in small-scale builds (i.e. development) but we >>> do >>> hit it on large-scale builds (i.e. all_browsers x all_locales nightly >>> build). >>> >>> But is increasing max open files for GWT compiler really that big of a >>> deal? >>> (i.e. we know that compiler needs to process lots of files per each >>> permutation) >> >> No, it's not that big of a deal *for our packaging* but since we've people on >> Gentoo building engine >> on their own systems, it may impact all of them... >> >>> >>> Alternatively, we could implement "remote GWT worker grid" -> offload >>> compilation >>> for each specific (browser x locale) permutation to a remote machine, this >>> would >>> require some extra Java implementation though. >>> >>> I agree on excessive RAM usage, I'm currently experimenting with some >>> compiler >>> options, I'll share results when I have something. >>> >>> Another approach we could take (something I'll discuss with my peers) is >>> the >>> idea of eliminating locale vector entirely -> re-implement i18n support in >>> UI >>> in a way that for X browsers, there will be only X total permutations (i.e. >>> not X x num_of_locales). This would reduce RAM usage && compilation time && >>> perhaps max files opened too. >> >> +1 on whatever can reduce build time and build requirements. >> >> >>> >>>> >>>> >>>>> >>>>> ----- Original Message ----- >>>>>> From: "Sandro Bonazzola" <[email protected]> >>>>>> To: "Vojtech Szocs" <[email protected]> >>>>>> Cc: "Juan Hernandez" <[email protected]>, "Kiril Nesenko" >>>>>> <[email protected]>, "arch" <[email protected]> >>>>>> Sent: Wednesday, February 26, 2014 8:37:07 AM >>>>>> Subject: Re: [urgent] Re: ovirt-engine build failure >>>>>> >>>>>> Il 25/02/2014 16:52, Vojtech Szocs ha scritto: >>>>>>> >>>>>>> >>>>>>> ----- Original Message ----- >>>>>>>> From: "Vojtech Szocs" <[email protected]> >>>>>>>> To: "Sandro Bonazzola" <[email protected]> >>>>>>>> Cc: "Juan Hernandez" <[email protected]>, "Kiril Nesenko" >>>>>>>> <[email protected]>, "arch" <[email protected]> >>>>>>>> Sent: Tuesday, February 25, 2014 1:05:35 PM >>>>>>>> Subject: Re: [urgent] Re: ovirt-engine build failure >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> ----- Original Message ----- >>>>>>>>> From: "Sandro Bonazzola" <[email protected]> >>>>>>>>> To: "Vojtech Szocs" <[email protected]> >>>>>>>>> Cc: "Juan Hernandez" <[email protected]>, "arch" <[email protected]>, >>>>>>>>> "Kiril >>>>>>>>> Nesenko" <[email protected]> >>>>>>>>> Sent: Tuesday, February 25, 2014 9:02:11 AM >>>>>>>>> Subject: Re: [urgent] Re: ovirt-engine build failure >>>>>>>>> >>>>>>>>> Il 25/02/2014 08:47, Sandro Bonazzola ha scritto: >>>>>>>>>> Il 24/02/2014 18:28, Vojtech Szocs ha scritto: >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> ----- Original Message ----- >>>>>>>>>>>> From: "Sandro Bonazzola" <[email protected]> >>>>>>>>>>>> To: "Vojtech Szocs" <[email protected]> >>>>>>>>>>>> Cc: "Eyal Edri" <[email protected]>, "Juan Hernandez" >>>>>>>>>>>> <[email protected]>, "Kiril Nesenko" <[email protected]>, >>>>>>>>>>>> "arch" <[email protected]> >>>>>>>>>>>> Sent: Monday, February 24, 2014 9:08:09 AM >>>>>>>>>>>> Subject: [urgent] Re: ovirt-engine build failure >>>>>>>>>>>> >>>>>>>>>>>> Top posting, >>>>>>>>>>>> >>>>>>>>>>>> Please give this task really urgent priority, as you can see: >>>>>>>>>>>> >>>>>>>>>>>> http://jenkins.ovirt.org/job/ovirt-engine_3.4_create_rpms/ >>>>>>>>>>>> and >>>>>>>>>>>> http://jenkins.ovirt.org/job/ovirt-engine_master_create_rpms/ >>>>>>>>>>>> >>>>>>>>>>>> are failing consistently so we don't have nightly builds. >>>>>>>>>>> >>>>>>>>>>> (CC'ing Einav) >>>>>>>>>>> >>>>>>>>>>> Attached TEST patch, please apply & rebuild ovirt-engine as usual. >>>>>>>>>>> >>>>>>>>>>> This patch should cause following output during GWT compilation: >>>>>>>>>>> >>>>>>>>>>> [ERROR] ######### Util.readURLAsBytes >>>>>>>>>>> file:/home/vszocs/work/ovirt-engine/frontend/webadmin/modules/webadmin/target/generated-sources/gwt/org/ovirt/engine/ui/webadmin/images/search_button.png >>>>>>>>>>> >>>>>>>>>>> If IOException occurs, exception stacktrace will be printed right >>>>>>>>>>> below >>>>>>>>>>> that line. >>>>>>>>>>> >>>>>>>>>>> Note: please ignore the [ERROR] prefix, it's a technical issue, >>>>>>>>>>> more >>>>>>>>>>> details at >>>>>>>>>>> https://www.mail-archive.com/[email protected]/msg05735.html >>>>>>>>>>> >>>>>>>>>>> I just did a local upstream build with TEST patch applied, GWT >>>>>>>>>>> compilation >>>>>>>>>>> passed OK. >>>>>>>>>> >>>>>>>>>> build job running with your patch here: >>>>>>>>>> http://jenkins.ovirt.org/job/manual-build-tarball/235/ >>>>>>>>>> waiting for a free executor right now. >>>>>>>>>> Let's see how it goes. >>>>>>>>> >>>>>>>>> Failed on CentOS: >>>>>>>>> http://jenkins.ovirt.org/job/manual-build-tarball/235/label=centos6-host/consoleFull >>>>>>>> >>>>>>>> Hm, in that ^^ build, following exceptions occured: >>>>>>>> >>>>>>>> 1. >>>>>>>> [INFO] java.lang.NullPointerException >>>>>>>> [INFO] at >>>>>>>> com.google.gwt.dev.util.Util.computeStrongName(Util.java:172) >>>>>>>> >>>>>>>> 2. >>>>>>>> [INFO] java.lang.NullPointerException: No ImageRect ever computed for >>>>>>>> downDisabledImage >>>>>>>> [INFO] at >>>>>>>> com.google.gwt.resources.rg.ImageResourceGenerator.createAssignment(ImageResourceGenerator.java:475 >>>>>>>> >>>>>>>> 1. is caused by Util#readURLAsBytes but I don't see any IOException in >>>>>>>> the >>>>>>>> logs >>>>>>>> 2. is just follow-up consequence of 1. (i.e. image metadata missing in >>>>>>>> GWT >>>>>>>> image cache) >>>>>>>> >>>>>>>> Now I noticed I missed one possible IOException inside >>>>>>>> Util#readURLConnectionAsBytes ... >>>>>>>> >>>>>>>> Long story short, I'll send updated TEST patch soon, apologies for the >>>>>>>> inconvenience. >>>>>>> >>>>>>> TEST-v2 patch attached. Please apply & rebuild ovirt-engine as usual. >>>>>> >>>>>> Done, here is the console output: >>>>>> http://jenkins.ovirt.org/job/manual-build-tarball/label=centos6-host/239/consoleFull >>>>>> >>>>>> >>>>>> >>>>>> >>>>>>> >>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> Il 20/02/2014 13:34, Vojtech Szocs ha scritto: >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> ----- Original Message ----- >>>>>>>>>>>>>> From: "Sandro Bonazzola" <[email protected]> >>>>>>>>>>>>>> To: "Vojtech Szocs" <[email protected]> >>>>>>>>>>>>>> Cc: "Eyal Edri" <[email protected]>, "Juan Hernandez" >>>>>>>>>>>>>> <[email protected]>, "Kiril Nesenko" <[email protected]>, >>>>>>>>>>>>>> "arch" <[email protected]> >>>>>>>>>>>>>> Sent: Thursday, February 20, 2014 10:02:00 AM >>>>>>>>>>>>>> Subject: Re: ovirt-engine build failure >>>>>>>>>>>>>> >>>>>>>>>>>>>> Il 19/02/2014 16:51, Vojtech Szocs ha scritto: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> ----- Original Message ----- >>>>>>>>>>>>>>>> From: "Sandro Bonazzola" <[email protected]> >>>>>>>>>>>>>>>> To: "Vojtech Szocs" <[email protected]> >>>>>>>>>>>>>>>> Cc: "Eyal Edri" <[email protected]>, "Juan Hernandez" >>>>>>>>>>>>>>>> <[email protected]>, "Kiril Nesenko" <[email protected]>, >>>>>>>>>>>>>>>> "arch" <[email protected]> >>>>>>>>>>>>>>>> Sent: Wednesday, February 19, 2014 2:43:44 PM >>>>>>>>>>>>>>>> Subject: Re: ovirt-engine build failure >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Il 19/02/2014 14:41, Vojtech Szocs ha scritto: >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> ----- Original Message ----- >>>>>>>>>>>>>>>>>> From: "Sandro Bonazzola" <[email protected]> >>>>>>>>>>>>>>>>>> To: "Vojtech Szocs" <[email protected]> >>>>>>>>>>>>>>>>>> Cc: "Eyal Edri" <[email protected]>, "Juan Hernandez" >>>>>>>>>>>>>>>>>> <[email protected]>, "Kiril Nesenko" >>>>>>>>>>>>>>>>>> <[email protected]>, >>>>>>>>>>>>>>>>>> "arch" <[email protected]> >>>>>>>>>>>>>>>>>> Sent: Wednesday, February 19, 2014 2:23:16 PM >>>>>>>>>>>>>>>>>> Subject: Re: ovirt-engine build failure >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Il 19/02/2014 14:19, Vojtech Szocs ha scritto: >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> ----- Original Message ----- >>>>>>>>>>>>>>>>>>>> From: "Sandro Bonazzola" <[email protected]> >>>>>>>>>>>>>>>>>>>> To: "Vojtech Szocs" <[email protected]> >>>>>>>>>>>>>>>>>>>> Cc: "Eyal Edri" <[email protected]>, "Juan Hernandez" >>>>>>>>>>>>>>>>>>>> <[email protected]>, "Kiril Nesenko" >>>>>>>>>>>>>>>>>>>> <[email protected]>, >>>>>>>>>>>>>>>>>>>> "arch" <[email protected]> >>>>>>>>>>>>>>>>>>>> Sent: Wednesday, February 19, 2014 1:18:28 PM >>>>>>>>>>>>>>>>>>>> Subject: Re: ovirt-engine build failure >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> Il 19/02/2014 12:45, Vojtech Szocs ha scritto: >>>>>>>>>>>>>>>>>>>>> Hi, I checked GWT sources: >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> [INFO] java.lang.NullPointerException: No ImageRect ever >>>>>>>>>>>>>>>>>>>>>> computed >>>>>>>>>>>>>>>>>>>>>> for >>>>>>>>>>>>>>>>>>>>>> removeBrickCommitRequired >>>>>>>>>>>>>>>>>>>>>> [INFO] at >>>>>>>>>>>>>>>>>>>>>> com.google.gwt.resources.rg.ImageResourceGenerator.createAssignment(ImageResourceGenerator.java:475) >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> This happened because internal image cache for bundled >>>>>>>>>>>>>>>>>>>>> (composite >>>>>>>>>>>>>>>>>>>>> PNG) >>>>>>>>>>>>>>>>>>>>> images was corrput. >>>>>>>>>>>>>>>>>>>>> It seems to be caused by disk I/O error while reading >>>>>>>>>>>>>>>>>>>>> file >>>>>>>>>>>>>>>>>>>>> images/glusterAsyncTasks/remove_brick_commit_required.png >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> [INFO] java.lang.NullPointerException >>>>>>>>>>>>>>>>>>>>>> [INFO] at >>>>>>>>>>>>>>>>>>>>>> com.google.gwt.resources.rebind.context.InlineResourceContext.deploy(InlineResourceContext.java:40) >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> This happened because image data was null, again possible >>>>>>>>>>>>>>>>>>>>> disk >>>>>>>>>>>>>>>>>>>>> I/O >>>>>>>>>>>>>>>>>>>>> error >>>>>>>>>>>>>>>>>>>>> while reading image file. >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> Can we enable "TRACE" logging during GWT compilation? >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> -Dgwt.logLevel=TRACE >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> Building master right now with your suggestions here: >>>>>>>>>>>>>>>>>>>> http://jenkins.ovirt.org/job/manual-build-tarball/227/ >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> Looking at >>>>>>>>>>>>>>>>>>> http://jenkins.ovirt.org/job/manual-build-tarball/227/label=fedora20-host/console >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> So far I see no errors.. There should be total 5 browsers x >>>>>>>>>>>>>>>>>>> 8 >>>>>>>>>>>>>>>>>>> locales >>>>>>>>>>>>>>>>>>> = >>>>>>>>>>>>>>>>>>> 40 >>>>>>>>>>>>>>>>>>> GWT permutations. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> So no changes with and without >>>>>>>>>>>>>>>>>> -Dgwt.userAgent=ie8,ie9,gecko1_8,safari,opera. >>>>>>>>>>>>>>>>>> We've still 40 permutations. Maybe you defined >>>>>>>>>>>>>>>>>> userAgent=ie8,ie9,gecko1_8,safari,opera somewhere in the POM >>>>>>>>>>>>>>>>>> files? >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> You are right. I checked frontend/webadmin/modules/pom.xml -> >>>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>> "gwt.userAgent" >>>>>>>>>>>>>>>>> property is copied into "locale" GWT property in >>>>>>>>>>>>>>>>> {WebAdmin,UserPortal}.gwt.xml >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Ok, so I'll remove >>>>>>>>>>>>>>>> -Dgwt.userAgent=ie8,ie9,gecko1_8,safari,opera >>>>>>>>>>>>>>>> from >>>>>>>>>>>>>>>> build >>>>>>>>>>>>>>>> command. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Agreed. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> This: >>>>>>>>>>>>>>>>>> http://jenkins.ovirt.org/job/manual-build-tarball/227/label=centos6-host/console >>>>>>>>>>>>>>>>>> failed, you can look at it, it has -Dgwt.logLevel=TRACE -X. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Hm, it contains: >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> [DEBUG] (f) logLevel = INFO >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Somehow the TRACE log level for GWT wasn't applied.. not sure >>>>>>>>>>>>>>>>> why. >>>>>>>>>>>>>>>>> Let >>>>>>>>>>>>>>>>> me >>>>>>>>>>>>>>>>> check >>>>>>>>>>>>>>>>> this on local build. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Ok, let me know if I've to change command line. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> I wrote upstream patch: http://gerrit.ovirt.org/#/c/24789/ >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> When it's merged, setting "gwt.logLevel" will work (as you did >>>>>>>>>>>>>>> before) >>>>>>>>>>>>>>> and we'll be able to see more details in the build log. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Here is the new build with your patch merged: >>>>>>>>>>>>>> >>>>>>>>>>>>>> http://jenkins.ovirt.org/job/manual-build-tarball/228/label=centos6-host/consoleFull >>>>>>>>>>>>> >>>>>>>>>>>>> Thanks. I did some more investigation on this. >>>>>>>>>>>>> >>>>>>>>>>>>> Again it's NullPointerException when rebinding >>>>>>>>>>>>> ApplicationResources, >>>>>>>>>>>>> but >>>>>>>>>>>>> this time for different image file: >>>>>>>>>>>>> ApplicationResources.wrenchImage >>>>>>>>>>>>> ("images/wrench.png"). >>>>>>>>>>>>> >>>>>>>>>>>>> Looking at stacktrace & GWT sources, the problematic code seems >>>>>>>>>>>>> to >>>>>>>>>>>>> be >>>>>>>>>>>>> (AbstractResourceContext:74): >>>>>>>>>>>>> >>>>>>>>>>>>> byte[] bytes = Util.readURLAsBytes(resource); >>>>>>>>>>>>> >>>>>>>>>>>>> where "resource" is (ImageResourceGenerator:311): >>>>>>>>>>>>> >>>>>>>>>>>>> localized.getUrl(); // LocalizedImage localized >>>>>>>>>>>>> >>>>>>>>>>>>> Note: in oVirt UI we don't use localized images (i.e. >>>>>>>>>>>>> "wrench_fr_FR.png") >>>>>>>>>>>>> so "localized" in this case points to "wrench.png". >>>>>>>>>>>>> >>>>>>>>>>>>> So the problem seems to be reading this file: >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> frontend/webadmin/modules/webadmin/src/main/resources/org/ovirt/engine/ui/webadmin/images/wrench.png >>>>>>>>>>>>> >>>>>>>>>>>>> Since we're having many GWT permutations, this file ^^ (along >>>>>>>>>>>>> with >>>>>>>>>>>>> similar >>>>>>>>>>>>> ones) is attempted to be read multiple times *from the same >>>>>>>>>>>>> (single) >>>>>>>>>>>>> GWT >>>>>>>>>>>>> compiler Java process, before actual permutation worker Java >>>>>>>>>>>>> sub-processes >>>>>>>>>>>>> are spawned*. (More precisely, above NullPointerException happens >>>>>>>>>>>>> at >>>>>>>>>>>>> deferred binding phase which precedes actual GWT compilation per >>>>>>>>>>>>> each >>>>>>>>>>>>> permutation.) >>>>>>>>>>>>> >>>>>>>>>>>>> I think the root cause is Util.readURLAsBytes method used >>>>>>>>>>>>> internally >>>>>>>>>>>>> by >>>>>>>>>>>>> GWT >>>>>>>>>>>>> compiler: >>>>>>>>>>>>> >>>>>>>>>>>>> public static byte[] readURLAsBytes(URL url) { >>>>>>>>>>>>> try { >>>>>>>>>>>>> URLConnection conn = url.openConnection(); >>>>>>>>>>>>> conn.setUseCaches(false); >>>>>>>>>>>>> return readURLConnectionAsBytes(conn); >>>>>>>>>>>>> } catch (IOException e) { >>>>>>>>>>>>> return null; // This causes NullPointerException, but this >>>>>>>>>>>>> exception >>>>>>>>>>>>> isn't logged (wtf, GWT!) >>>>>>>>>>>>> } >>>>>>>>>>>>> } >>>>>>>>>>>>> >>>>>>>>>>>>> To summarize, I'll need to write another patch which replaces >>>>>>>>>>>>> Util.readURLAsBytes implementation via AOP and prints IOException >>>>>>>>>>>>> details. >>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> # >>>>>>>>>>>>>>>>>>>> CI_MAVEN_SETTINGS=$WORKSPACE/artifactory-ovirt-org-settings.xml >>>>>>>>>>>>>>>>>>>> # export MAVEN_OPTS="${MAVEN_OPTS} -Xmx2048M >>>>>>>>>>>>>>>>>>>> -XX:MaxPermSize=1024m >>>>>>>>>>>>>>>>>>>> -Dgwt.compiler.localWorkers=1" >>>>>>>>>>>>>>>>>>>> # rpmbuild -D "_topdir $(pwd)/rpmbuild" -ts "${TARBALL}" >>>>>>>>>>>>>>>>>>>> # rpmbuild -D "_topdir $(pwd)/rpmbuild" -D >>>>>>>>>>>>>>>>>>>> "ovirt_build_extra_flags >>>>>>>>>>>>>>>>>>>> -gs >>>>>>>>>>>>>>>>>>>> ${CI_MAVEN_SETTINGS} >>>>>>>>>>>>>>>>>>>> -Dgwt.userAgent=ie8,ie9,gecko1_8,safari,opera >>>>>>>>>>>>>>>>>>>> -Dgwt.logLevel=TRACE -X" --rebuild >>>>>>>>>>>>>>>>>>>> rpmbuild/SRPMS/*.src.rpm >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> Let me know if you need anything else. >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> Vojtech >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> ----- Original Message ----- >>>>>>>>>>>>>>>>>>>>>> From: "Vojtech Szocs" <[email protected]> >>>>>>>>>>>>>>>>>>>>>> To: "Sandro Bonazzola" <[email protected]> >>>>>>>>>>>>>>>>>>>>>> Cc: "Eyal Edri" <[email protected]>, "Juan Hernandez" >>>>>>>>>>>>>>>>>>>>>> <[email protected]>, "Kiril Nesenko" >>>>>>>>>>>>>>>>>>>>>> <[email protected]>, >>>>>>>>>>>>>>>>>>>>>> "arch" <[email protected]> >>>>>>>>>>>>>>>>>>>>>> Sent: Tuesday, February 18, 2014 6:54:20 PM >>>>>>>>>>>>>>>>>>>>>> Subject: Re: ovirt-engine build failure >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> In your log I see following error: >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> ... >>>>>>>>>>>>>>>>>>>>>> [INFO] Computing all possible rebind results for >>>>>>>>>>>>>>>>>>>>>> 'org.ovirt.engine.ui.webadmin.ApplicationResources' >>>>>>>>>>>>>>>>>>>>>> [INFO] Rebinding >>>>>>>>>>>>>>>>>>>>>> org.ovirt.engine.ui.webadmin.ApplicationResources >>>>>>>>>>>>>>>>>>>>>> [INFO] Invoking generator >>>>>>>>>>>>>>>>>>>>>> com.google.gwt.resources.rebind.context.InlineClientBundleGenerator >>>>>>>>>>>>>>>>>>>>>> [INFO] [ERROR] Generator >>>>>>>>>>>>>>>>>>>>>> 'com.google.gwt.resources.rebind.context.InlineClientBundleGenerator' >>>>>>>>>>>>>>>>>>>>>> threw >>>>>>>>>>>>>>>>>>>>>> an exception while rebinding >>>>>>>>>>>>>>>>>>>>>> 'org.ovirt.engine.ui.webadmin.ApplicationResources' >>>>>>>>>>>>>>>>>>>>>> [INFO] java.lang.NullPointerException: No ImageRect ever >>>>>>>>>>>>>>>>>>>>>> computed >>>>>>>>>>>>>>>>>>>>>> for >>>>>>>>>>>>>>>>>>>>>> removeBrickCommitRequired >>>>>>>>>>>>>>>>>>>>>> [INFO] at >>>>>>>>>>>>>>>>>>>>>> com.google.gwt.resources.rg.ImageResourceGenerator.createAssignment(ImageResourceGenerator.java:475) >>>>>>>>>>>>>>>>>>>>>> ... >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> I checked again logs at >>>>>>>>>>>>>>>>>>>>>> http://jenkins.ovirt.org/job/manual-build-tarball/212/label=fedora19-host/consoleFull >>>>>>>>>>>>>>>>>>>>>> and indeed there is a similar error: >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> ... >>>>>>>>>>>>>>>>>>>>>> [INFO] Computing all possible rebind results for >>>>>>>>>>>>>>>>>>>>>> 'org.ovirt.engine.ui.webadmin.ApplicationResources' >>>>>>>>>>>>>>>>>>>>>> [INFO] Rebinding >>>>>>>>>>>>>>>>>>>>>> org.ovirt.engine.ui.webadmin.ApplicationResources >>>>>>>>>>>>>>>>>>>>>> [INFO] Invoking generator >>>>>>>>>>>>>>>>>>>>>> com.google.gwt.resources.rebind.context.InlineClientBundleGenerator >>>>>>>>>>>>>>>>>>>>>> [INFO] [ERROR] Generator >>>>>>>>>>>>>>>>>>>>>> 'com.google.gwt.resources.rebind.context.InlineClientBundleGenerator' >>>>>>>>>>>>>>>>>>>>>> threw >>>>>>>>>>>>>>>>>>>>>> an exception while rebinding >>>>>>>>>>>>>>>>>>>>>> 'org.ovirt.engine.ui.webadmin.ApplicationResources' >>>>>>>>>>>>>>>>>>>>>> [INFO] java.lang.NullPointerException >>>>>>>>>>>>>>>>>>>>>> [INFO] at >>>>>>>>>>>>>>>>>>>>>> com.google.gwt.resources.rebind.context.InlineResourceContext.deploy(InlineResourceContext.java:40) >>>>>>>>>>>>>>>>>>>>>> ... >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> NullPointerException when processing >>>>>>>>>>>>>>>>>>>>>> ApplicationResources >>>>>>>>>>>>>>>>>>>>>> but >>>>>>>>>>>>>>>>>>>>>> occuring >>>>>>>>>>>>>>>>>>>>>> from >>>>>>>>>>>>>>>>>>>>>> different places (ImageResourceGenerator vs. >>>>>>>>>>>>>>>>>>>>>> InlineResourceContext) >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> Now I understand why Maven build fails without any >>>>>>>>>>>>>>>>>>>>>> details: >>>>>>>>>>>>>>>>>>>>>> - GWT deferred binding fails for ApplicationResources >>>>>>>>>>>>>>>>>>>>>> (i.e. >>>>>>>>>>>>>>>>>>>>>> uncaught >>>>>>>>>>>>>>>>>>>>>> NPE) >>>>>>>>>>>>>>>>>>>>>> - GWT compilation proceeds, fails to link with >>>>>>>>>>>>>>>>>>>>>> ApplicationResources >>>>>>>>>>>>>>>>>>>>>> - GWT compilation task fails with non-OK status code >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> So the issue here is GWT deferred binding generator >>>>>>>>>>>>>>>>>>>>>> (InlineClientBundleGenerator) failure during GWT >>>>>>>>>>>>>>>>>>>>>> compilation. >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> GWT compiler writes extra Java files during deferred >>>>>>>>>>>>>>>>>>>>>> binding >>>>>>>>>>>>>>>>>>>>>> phase, so this could point to issue with local >>>>>>>>>>>>>>>>>>>>>> environment.. >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> I will try to investigate this and look at GWT source >>>>>>>>>>>>>>>>>>>>>> code. >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>>>>>> Sandro Bonazzola >>>>>>>>>>>>>>>>>>>> Better technology. Faster innovation. Powered by community >>>>>>>>>>>>>>>>>>>> collaboration. >>>>>>>>>>>>>>>>>>>> See how it works at redhat.com >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>>>> Sandro Bonazzola >>>>>>>>>>>>>>>>>> Better technology. Faster innovation. Powered by community >>>>>>>>>>>>>>>>>> collaboration. >>>>>>>>>>>>>>>>>> See how it works at redhat.com >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>> Sandro Bonazzola >>>>>>>>>>>>>>>> Better technology. Faster innovation. Powered by community >>>>>>>>>>>>>>>> collaboration. >>>>>>>>>>>>>>>> See how it works at redhat.com >>>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> -- >>>>>>>>>>>>>> Sandro Bonazzola >>>>>>>>>>>>>> Better technology. Faster innovation. Powered by community >>>>>>>>>>>>>> collaboration. >>>>>>>>>>>>>> See how it works at redhat.com >>>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> -- >>>>>>>>>>>> Sandro Bonazzola >>>>>>>>>>>> Better technology. Faster innovation. Powered by community >>>>>>>>>>>> collaboration. >>>>>>>>>>>> See how it works at redhat.com >>>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> Sandro Bonazzola >>>>>>>>> Better technology. Faster innovation. Powered by community >>>>>>>>> collaboration. >>>>>>>>> See how it works at redhat.com >>>>>>>>> >>>>>>>> _______________________________________________ >>>>>>>> Arch mailing list >>>>>>>> [email protected] >>>>>>>> http://lists.ovirt.org/mailman/listinfo/arch >>>>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> Sandro Bonazzola >>>>>> Better technology. Faster innovation. Powered by community >>>>>> collaboration. >>>>>> See how it works at redhat.com >>>>>> _______________________________________________ >>>>>> Arch mailing list >>>>>> [email protected] >>>>>> http://lists.ovirt.org/mailman/listinfo/arch >>>>>> >>>> >>>> >>>> -- >>>> Sandro Bonazzola >>>> Better technology. Faster innovation. Powered by community collaboration. >>>> See how it works at redhat.com >>>> >> >> >> -- >> Sandro Bonazzola >> Better technology. Faster innovation. Powered by community collaboration. >> See how it works at redhat.com >> -- Sandro Bonazzola Better technology. Faster innovation. Powered by community collaboration. See how it works at redhat.com _______________________________________________ Arch mailing list [email protected] http://lists.ovirt.org/mailman/listinfo/arch
