----- Original Message ----- > From: "Sandro Bonazzola" <[email protected]> > To: "Vojtech Szocs" <[email protected]> > Cc: "Juan Hernandez" <[email protected]>, "arch" <[email protected]>, "Kiril > Nesenko" <[email protected]> > Sent: Tuesday, February 25, 2014 9:02:11 AM > Subject: Re: [urgent] Re: ovirt-engine build failure > > Il 25/02/2014 08:47, Sandro Bonazzola ha scritto: > > Il 24/02/2014 18:28, Vojtech Szocs ha scritto: > >> > >> > >> ----- Original Message ----- > >>> From: "Sandro Bonazzola" <[email protected]> > >>> To: "Vojtech Szocs" <[email protected]> > >>> Cc: "Eyal Edri" <[email protected]>, "Juan Hernandez" > >>> <[email protected]>, "Kiril Nesenko" <[email protected]>, > >>> "arch" <[email protected]> > >>> Sent: Monday, February 24, 2014 9:08:09 AM > >>> Subject: [urgent] Re: ovirt-engine build failure > >>> > >>> Top posting, > >>> > >>> Please give this task really urgent priority, as you can see: > >>> > >>> http://jenkins.ovirt.org/job/ovirt-engine_3.4_create_rpms/ > >>> and > >>> http://jenkins.ovirt.org/job/ovirt-engine_master_create_rpms/ > >>> > >>> are failing consistently so we don't have nightly builds. > >> > >> (CC'ing Einav) > >> > >> Attached TEST patch, please apply & rebuild ovirt-engine as usual. > >> > >> This patch should cause following output during GWT compilation: > >> > >> [ERROR] ######### Util.readURLAsBytes > >> file:/home/vszocs/work/ovirt-engine/frontend/webadmin/modules/webadmin/target/generated-sources/gwt/org/ovirt/engine/ui/webadmin/images/search_button.png > >> > >> If IOException occurs, exception stacktrace will be printed right below > >> that line. > >> > >> Note: please ignore the [ERROR] prefix, it's a technical issue, more > >> details at > >> https://www.mail-archive.com/[email protected]/msg05735.html > >> > >> I just did a local upstream build with TEST patch applied, GWT compilation > >> passed OK. > > > > build job running with your patch here: > > http://jenkins.ovirt.org/job/manual-build-tarball/235/ > > waiting for a free executor right now. > > Let's see how it goes. > > Failed on CentOS: > http://jenkins.ovirt.org/job/manual-build-tarball/235/label=centos6-host/consoleFull
Hm, in that ^^ build, following exceptions occured: 1. [INFO] java.lang.NullPointerException [INFO] at com.google.gwt.dev.util.Util.computeStrongName(Util.java:172) 2. [INFO] java.lang.NullPointerException: No ImageRect ever computed for downDisabledImage [INFO] at com.google.gwt.resources.rg.ImageResourceGenerator.createAssignment(ImageResourceGenerator.java:475 1. is caused by Util#readURLAsBytes but I don't see any IOException in the logs 2. is just follow-up consequence of 1. (i.e. image metadata missing in GWT image cache) Now I noticed I missed one possible IOException inside Util#readURLConnectionAsBytes ... Long story short, I'll send updated TEST patch soon, apologies for the inconvenience. > > > > > > > > > > > > > >> > >>> > >>> > >>> > >>> > >>> Il 20/02/2014 13:34, Vojtech Szocs ha scritto: > >>>> > >>>> > >>>> ----- Original Message ----- > >>>>> From: "Sandro Bonazzola" <[email protected]> > >>>>> To: "Vojtech Szocs" <[email protected]> > >>>>> Cc: "Eyal Edri" <[email protected]>, "Juan Hernandez" > >>>>> <[email protected]>, "Kiril Nesenko" <[email protected]>, > >>>>> "arch" <[email protected]> > >>>>> Sent: Thursday, February 20, 2014 10:02:00 AM > >>>>> Subject: Re: ovirt-engine build failure > >>>>> > >>>>> Il 19/02/2014 16:51, Vojtech Szocs ha scritto: > >>>>>> > >>>>>> > >>>>>> ----- Original Message ----- > >>>>>>> From: "Sandro Bonazzola" <[email protected]> > >>>>>>> To: "Vojtech Szocs" <[email protected]> > >>>>>>> Cc: "Eyal Edri" <[email protected]>, "Juan Hernandez" > >>>>>>> <[email protected]>, "Kiril Nesenko" <[email protected]>, > >>>>>>> "arch" <[email protected]> > >>>>>>> Sent: Wednesday, February 19, 2014 2:43:44 PM > >>>>>>> Subject: Re: ovirt-engine build failure > >>>>>>> > >>>>>>> Il 19/02/2014 14:41, Vojtech Szocs ha scritto: > >>>>>>>> > >>>>>>>> > >>>>>>>> ----- Original Message ----- > >>>>>>>>> From: "Sandro Bonazzola" <[email protected]> > >>>>>>>>> To: "Vojtech Szocs" <[email protected]> > >>>>>>>>> Cc: "Eyal Edri" <[email protected]>, "Juan Hernandez" > >>>>>>>>> <[email protected]>, "Kiril Nesenko" <[email protected]>, > >>>>>>>>> "arch" <[email protected]> > >>>>>>>>> Sent: Wednesday, February 19, 2014 2:23:16 PM > >>>>>>>>> Subject: Re: ovirt-engine build failure > >>>>>>>>> > >>>>>>>>> Il 19/02/2014 14:19, Vojtech Szocs ha scritto: > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> ----- Original Message ----- > >>>>>>>>>>> From: "Sandro Bonazzola" <[email protected]> > >>>>>>>>>>> To: "Vojtech Szocs" <[email protected]> > >>>>>>>>>>> Cc: "Eyal Edri" <[email protected]>, "Juan Hernandez" > >>>>>>>>>>> <[email protected]>, "Kiril Nesenko" <[email protected]>, > >>>>>>>>>>> "arch" <[email protected]> > >>>>>>>>>>> Sent: Wednesday, February 19, 2014 1:18:28 PM > >>>>>>>>>>> Subject: Re: ovirt-engine build failure > >>>>>>>>>>> > >>>>>>>>>>> Il 19/02/2014 12:45, Vojtech Szocs ha scritto: > >>>>>>>>>>>> Hi, I checked GWT sources: > >>>>>>>>>>>> > >>>>>>>>>>>>> [INFO] java.lang.NullPointerException: No ImageRect ever > >>>>>>>>>>>>> computed > >>>>>>>>>>>>> for > >>>>>>>>>>>>> removeBrickCommitRequired > >>>>>>>>>>>>> [INFO] at > >>>>>>>>>>>>> com.google.gwt.resources.rg.ImageResourceGenerator.createAssignment(ImageResourceGenerator.java:475) > >>>>>>>>>>>> > >>>>>>>>>>>> This happened because internal image cache for bundled > >>>>>>>>>>>> (composite > >>>>>>>>>>>> PNG) > >>>>>>>>>>>> images was corrput. > >>>>>>>>>>>> It seems to be caused by disk I/O error while reading file > >>>>>>>>>>>> images/glusterAsyncTasks/remove_brick_commit_required.png > >>>>>>>>>>>> > >>>>>>>>>>>>> [INFO] java.lang.NullPointerException > >>>>>>>>>>>>> [INFO] at > >>>>>>>>>>>>> com.google.gwt.resources.rebind.context.InlineResourceContext.deploy(InlineResourceContext.java:40) > >>>>>>>>>>>> > >>>>>>>>>>>> This happened because image data was null, again possible disk > >>>>>>>>>>>> I/O > >>>>>>>>>>>> error > >>>>>>>>>>>> while reading image file. > >>>>>>>>>>>> > >>>>>>>>>>>> Can we enable "TRACE" logging during GWT compilation? > >>>>>>>>>>>> > >>>>>>>>>>>> -Dgwt.logLevel=TRACE > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> Building master right now with your suggestions here: > >>>>>>>>>>> http://jenkins.ovirt.org/job/manual-build-tarball/227/ > >>>>>>>>>> > >>>>>>>>>> Looking at > >>>>>>>>>> http://jenkins.ovirt.org/job/manual-build-tarball/227/label=fedora20-host/console > >>>>>>>>>> > >>>>>>>>>> So far I see no errors.. There should be total 5 browsers x 8 > >>>>>>>>>> locales > >>>>>>>>>> = > >>>>>>>>>> 40 > >>>>>>>>>> GWT permutations. > >>>>>>>>> > >>>>>>>>> So no changes with and without > >>>>>>>>> -Dgwt.userAgent=ie8,ie9,gecko1_8,safari,opera. > >>>>>>>>> We've still 40 permutations. Maybe you defined > >>>>>>>>> userAgent=ie8,ie9,gecko1_8,safari,opera somewhere in the POM files? > >>>>>>>> > >>>>>>>> You are right. I checked frontend/webadmin/modules/pom.xml -> the > >>>>>>>> "gwt.userAgent" > >>>>>>>> property is copied into "locale" GWT property in > >>>>>>>> {WebAdmin,UserPortal}.gwt.xml > >>>>>>> > >>>>>>> Ok, so I'll remove -Dgwt.userAgent=ie8,ie9,gecko1_8,safari,opera from > >>>>>>> build > >>>>>>> command. > >>>>>> > >>>>>> Agreed. > >>>>>> > >>>>>>> > >>>>>>> > >>>>>>>> > >>>>>>>>> > >>>>>>>>> This: > >>>>>>>>> http://jenkins.ovirt.org/job/manual-build-tarball/227/label=centos6-host/console > >>>>>>>>> failed, you can look at it, it has -Dgwt.logLevel=TRACE -X. > >>>>>>>> > >>>>>>>> Hm, it contains: > >>>>>>>> > >>>>>>>> [DEBUG] (f) logLevel = INFO > >>>>>>>> > >>>>>>>> Somehow the TRACE log level for GWT wasn't applied.. not sure why. > >>>>>>>> Let > >>>>>>>> me > >>>>>>>> check > >>>>>>>> this on local build. > >>>>>>> > >>>>>>> Ok, let me know if I've to change command line. > >>>>>> > >>>>>> I wrote upstream patch: http://gerrit.ovirt.org/#/c/24789/ > >>>>>> > >>>>>> When it's merged, setting "gwt.logLevel" will work (as you did before) > >>>>>> and we'll be able to see more details in the build log. > >>>>> > >>>>> Here is the new build with your patch merged: > >>>>> > >>>>> http://jenkins.ovirt.org/job/manual-build-tarball/228/label=centos6-host/consoleFull > >>>> > >>>> Thanks. I did some more investigation on this. > >>>> > >>>> Again it's NullPointerException when rebinding ApplicationResources, but > >>>> this time for different image file: ApplicationResources.wrenchImage > >>>> ("images/wrench.png"). > >>>> > >>>> Looking at stacktrace & GWT sources, the problematic code seems to be > >>>> (AbstractResourceContext:74): > >>>> > >>>> byte[] bytes = Util.readURLAsBytes(resource); > >>>> > >>>> where "resource" is (ImageResourceGenerator:311): > >>>> > >>>> localized.getUrl(); // LocalizedImage localized > >>>> > >>>> Note: in oVirt UI we don't use localized images (i.e. > >>>> "wrench_fr_FR.png") > >>>> so "localized" in this case points to "wrench.png". > >>>> > >>>> So the problem seems to be reading this file: > >>>> > >>>> > >>>> frontend/webadmin/modules/webadmin/src/main/resources/org/ovirt/engine/ui/webadmin/images/wrench.png > >>>> > >>>> Since we're having many GWT permutations, this file ^^ (along with > >>>> similar > >>>> ones) is attempted to be read multiple times *from the same (single) GWT > >>>> compiler Java process, before actual permutation worker Java > >>>> sub-processes > >>>> are spawned*. (More precisely, above NullPointerException happens at > >>>> deferred binding phase which precedes actual GWT compilation per each > >>>> permutation.) > >>>> > >>>> I think the root cause is Util.readURLAsBytes method used internally by > >>>> GWT > >>>> compiler: > >>>> > >>>> public static byte[] readURLAsBytes(URL url) { > >>>> try { > >>>> URLConnection conn = url.openConnection(); > >>>> conn.setUseCaches(false); > >>>> return readURLConnectionAsBytes(conn); > >>>> } catch (IOException e) { > >>>> return null; // This causes NullPointerException, but this > >>>> exception > >>>> isn't logged (wtf, GWT!) > >>>> } > >>>> } > >>>> > >>>> To summarize, I'll need to write another patch which replaces > >>>> Util.readURLAsBytes implementation via AOP and prints IOException > >>>> details. > >>>> > >>>>> > >>>>> > >>>>> > >>>>> > >>>>>> > >>>>>>> > >>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> # CI_MAVEN_SETTINGS=$WORKSPACE/artifactory-ovirt-org-settings.xml > >>>>>>>>>>> # export MAVEN_OPTS="${MAVEN_OPTS} -Xmx2048M > >>>>>>>>>>> -XX:MaxPermSize=1024m > >>>>>>>>>>> -Dgwt.compiler.localWorkers=1" > >>>>>>>>>>> # rpmbuild -D "_topdir $(pwd)/rpmbuild" -ts "${TARBALL}" > >>>>>>>>>>> # rpmbuild -D "_topdir $(pwd)/rpmbuild" -D > >>>>>>>>>>> "ovirt_build_extra_flags > >>>>>>>>>>> -gs > >>>>>>>>>>> ${CI_MAVEN_SETTINGS} > >>>>>>>>>>> -Dgwt.userAgent=ie8,ie9,gecko1_8,safari,opera > >>>>>>>>>>> -Dgwt.logLevel=TRACE -X" --rebuild rpmbuild/SRPMS/*.src.rpm > >>>>>>>>>>> > >>>>>>>>>>> Let me know if you need anything else. > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> Vojtech > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> ----- Original Message ----- > >>>>>>>>>>>>> From: "Vojtech Szocs" <[email protected]> > >>>>>>>>>>>>> To: "Sandro Bonazzola" <[email protected]> > >>>>>>>>>>>>> Cc: "Eyal Edri" <[email protected]>, "Juan Hernandez" > >>>>>>>>>>>>> <[email protected]>, "Kiril Nesenko" <[email protected]>, > >>>>>>>>>>>>> "arch" <[email protected]> > >>>>>>>>>>>>> Sent: Tuesday, February 18, 2014 6:54:20 PM > >>>>>>>>>>>>> Subject: Re: ovirt-engine build failure > >>>>>>>>>>>>> > >>>>>>>>>>>>> In your log I see following error: > >>>>>>>>>>>>> > >>>>>>>>>>>>> ... > >>>>>>>>>>>>> [INFO] Computing all possible rebind results for > >>>>>>>>>>>>> 'org.ovirt.engine.ui.webadmin.ApplicationResources' > >>>>>>>>>>>>> [INFO] Rebinding > >>>>>>>>>>>>> org.ovirt.engine.ui.webadmin.ApplicationResources > >>>>>>>>>>>>> [INFO] Invoking generator > >>>>>>>>>>>>> com.google.gwt.resources.rebind.context.InlineClientBundleGenerator > >>>>>>>>>>>>> [INFO] [ERROR] Generator > >>>>>>>>>>>>> 'com.google.gwt.resources.rebind.context.InlineClientBundleGenerator' > >>>>>>>>>>>>> threw > >>>>>>>>>>>>> an exception while rebinding > >>>>>>>>>>>>> 'org.ovirt.engine.ui.webadmin.ApplicationResources' > >>>>>>>>>>>>> [INFO] java.lang.NullPointerException: No ImageRect ever > >>>>>>>>>>>>> computed > >>>>>>>>>>>>> for > >>>>>>>>>>>>> removeBrickCommitRequired > >>>>>>>>>>>>> [INFO] at > >>>>>>>>>>>>> com.google.gwt.resources.rg.ImageResourceGenerator.createAssignment(ImageResourceGenerator.java:475) > >>>>>>>>>>>>> ... > >>>>>>>>>>>>> > >>>>>>>>>>>>> I checked again logs at > >>>>>>>>>>>>> http://jenkins.ovirt.org/job/manual-build-tarball/212/label=fedora19-host/consoleFull > >>>>>>>>>>>>> and indeed there is a similar error: > >>>>>>>>>>>>> > >>>>>>>>>>>>> ... > >>>>>>>>>>>>> [INFO] Computing all possible rebind results for > >>>>>>>>>>>>> 'org.ovirt.engine.ui.webadmin.ApplicationResources' > >>>>>>>>>>>>> [INFO] Rebinding > >>>>>>>>>>>>> org.ovirt.engine.ui.webadmin.ApplicationResources > >>>>>>>>>>>>> [INFO] Invoking generator > >>>>>>>>>>>>> com.google.gwt.resources.rebind.context.InlineClientBundleGenerator > >>>>>>>>>>>>> [INFO] [ERROR] Generator > >>>>>>>>>>>>> 'com.google.gwt.resources.rebind.context.InlineClientBundleGenerator' > >>>>>>>>>>>>> threw > >>>>>>>>>>>>> an exception while rebinding > >>>>>>>>>>>>> 'org.ovirt.engine.ui.webadmin.ApplicationResources' > >>>>>>>>>>>>> [INFO] java.lang.NullPointerException > >>>>>>>>>>>>> [INFO] at > >>>>>>>>>>>>> com.google.gwt.resources.rebind.context.InlineResourceContext.deploy(InlineResourceContext.java:40) > >>>>>>>>>>>>> ... > >>>>>>>>>>>>> > >>>>>>>>>>>>> NullPointerException when processing ApplicationResources but > >>>>>>>>>>>>> occuring > >>>>>>>>>>>>> from > >>>>>>>>>>>>> different places (ImageResourceGenerator vs. > >>>>>>>>>>>>> InlineResourceContext) > >>>>>>>>>>>>> > >>>>>>>>>>>>> Now I understand why Maven build fails without any details: > >>>>>>>>>>>>> - GWT deferred binding fails for ApplicationResources (i.e. > >>>>>>>>>>>>> uncaught > >>>>>>>>>>>>> NPE) > >>>>>>>>>>>>> - GWT compilation proceeds, fails to link with > >>>>>>>>>>>>> ApplicationResources > >>>>>>>>>>>>> - GWT compilation task fails with non-OK status code > >>>>>>>>>>>>> > >>>>>>>>>>>>> So the issue here is GWT deferred binding generator > >>>>>>>>>>>>> (InlineClientBundleGenerator) failure during GWT compilation. > >>>>>>>>>>>>> > >>>>>>>>>>>>> GWT compiler writes extra Java files during deferred binding > >>>>>>>>>>>>> phase, so this could point to issue with local environment.. > >>>>>>>>>>>>> > >>>>>>>>>>>>> I will try to investigate this and look at GWT source code. > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> -- > >>>>>>>>>>> Sandro Bonazzola > >>>>>>>>>>> Better technology. Faster innovation. Powered by community > >>>>>>>>>>> collaboration. > >>>>>>>>>>> See how it works at redhat.com > >>>>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> -- > >>>>>>>>> Sandro Bonazzola > >>>>>>>>> Better technology. Faster innovation. Powered by community > >>>>>>>>> collaboration. > >>>>>>>>> See how it works at redhat.com > >>>>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> -- > >>>>>>> Sandro Bonazzola > >>>>>>> Better technology. Faster innovation. Powered by community > >>>>>>> collaboration. > >>>>>>> See how it works at redhat.com > >>>>>>> > >>>>> > >>>>> > >>>>> -- > >>>>> Sandro Bonazzola > >>>>> Better technology. Faster innovation. Powered by community > >>>>> collaboration. > >>>>> See how it works at redhat.com > >>>>> > >>> > >>> > >>> -- > >>> Sandro Bonazzola > >>> Better technology. Faster innovation. Powered by community collaboration. > >>> See how it works at redhat.com > >>> > > > > > > > -- > Sandro Bonazzola > Better technology. Faster innovation. Powered by community collaboration. > See how it works at redhat.com > _______________________________________________ Arch mailing list [email protected] http://lists.ovirt.org/mailman/listinfo/arch
