Il 25/02/2014 08:47, Sandro Bonazzola ha scritto: > Il 24/02/2014 18:28, Vojtech Szocs ha scritto: >> >> >> ----- Original Message ----- >>> From: "Sandro Bonazzola" <[email protected]> >>> To: "Vojtech Szocs" <[email protected]> >>> Cc: "Eyal Edri" <[email protected]>, "Juan Hernandez" <[email protected]>, >>> "Kiril Nesenko" <[email protected]>, >>> "arch" <[email protected]> >>> Sent: Monday, February 24, 2014 9:08:09 AM >>> Subject: [urgent] Re: ovirt-engine build failure >>> >>> Top posting, >>> >>> Please give this task really urgent priority, as you can see: >>> >>> http://jenkins.ovirt.org/job/ovirt-engine_3.4_create_rpms/ >>> and >>> http://jenkins.ovirt.org/job/ovirt-engine_master_create_rpms/ >>> >>> are failing consistently so we don't have nightly builds. >> >> (CC'ing Einav) >> >> Attached TEST patch, please apply & rebuild ovirt-engine as usual. >> >> This patch should cause following output during GWT compilation: >> >> [ERROR] ######### Util.readURLAsBytes >> file:/home/vszocs/work/ovirt-engine/frontend/webadmin/modules/webadmin/target/generated-sources/gwt/org/ovirt/engine/ui/webadmin/images/search_button.png >> >> If IOException occurs, exception stacktrace will be printed right below that >> line. >> >> Note: please ignore the [ERROR] prefix, it's a technical issue, more details >> at https://www.mail-archive.com/[email protected]/msg05735.html >> >> I just did a local upstream build with TEST patch applied, GWT compilation >> passed OK. > > build job running with your patch here: > http://jenkins.ovirt.org/job/manual-build-tarball/235/ > waiting for a free executor right now. > Let's see how it goes.
Failed on CentOS: http://jenkins.ovirt.org/job/manual-build-tarball/235/label=centos6-host/consoleFull > > > >> >>> >>> >>> >>> >>> Il 20/02/2014 13:34, Vojtech Szocs ha scritto: >>>> >>>> >>>> ----- Original Message ----- >>>>> From: "Sandro Bonazzola" <[email protected]> >>>>> To: "Vojtech Szocs" <[email protected]> >>>>> Cc: "Eyal Edri" <[email protected]>, "Juan Hernandez" >>>>> <[email protected]>, "Kiril Nesenko" <[email protected]>, >>>>> "arch" <[email protected]> >>>>> Sent: Thursday, February 20, 2014 10:02:00 AM >>>>> Subject: Re: ovirt-engine build failure >>>>> >>>>> Il 19/02/2014 16:51, Vojtech Szocs ha scritto: >>>>>> >>>>>> >>>>>> ----- Original Message ----- >>>>>>> From: "Sandro Bonazzola" <[email protected]> >>>>>>> To: "Vojtech Szocs" <[email protected]> >>>>>>> Cc: "Eyal Edri" <[email protected]>, "Juan Hernandez" >>>>>>> <[email protected]>, "Kiril Nesenko" <[email protected]>, >>>>>>> "arch" <[email protected]> >>>>>>> Sent: Wednesday, February 19, 2014 2:43:44 PM >>>>>>> Subject: Re: ovirt-engine build failure >>>>>>> >>>>>>> Il 19/02/2014 14:41, Vojtech Szocs ha scritto: >>>>>>>> >>>>>>>> >>>>>>>> ----- Original Message ----- >>>>>>>>> From: "Sandro Bonazzola" <[email protected]> >>>>>>>>> To: "Vojtech Szocs" <[email protected]> >>>>>>>>> Cc: "Eyal Edri" <[email protected]>, "Juan Hernandez" >>>>>>>>> <[email protected]>, "Kiril Nesenko" <[email protected]>, >>>>>>>>> "arch" <[email protected]> >>>>>>>>> Sent: Wednesday, February 19, 2014 2:23:16 PM >>>>>>>>> Subject: Re: ovirt-engine build failure >>>>>>>>> >>>>>>>>> Il 19/02/2014 14:19, Vojtech Szocs ha scritto: >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> ----- Original Message ----- >>>>>>>>>>> From: "Sandro Bonazzola" <[email protected]> >>>>>>>>>>> To: "Vojtech Szocs" <[email protected]> >>>>>>>>>>> Cc: "Eyal Edri" <[email protected]>, "Juan Hernandez" >>>>>>>>>>> <[email protected]>, "Kiril Nesenko" <[email protected]>, >>>>>>>>>>> "arch" <[email protected]> >>>>>>>>>>> Sent: Wednesday, February 19, 2014 1:18:28 PM >>>>>>>>>>> Subject: Re: ovirt-engine build failure >>>>>>>>>>> >>>>>>>>>>> Il 19/02/2014 12:45, Vojtech Szocs ha scritto: >>>>>>>>>>>> Hi, I checked GWT sources: >>>>>>>>>>>> >>>>>>>>>>>>> [INFO] java.lang.NullPointerException: No ImageRect ever computed >>>>>>>>>>>>> for >>>>>>>>>>>>> removeBrickCommitRequired >>>>>>>>>>>>> [INFO] at >>>>>>>>>>>>> com.google.gwt.resources.rg.ImageResourceGenerator.createAssignment(ImageResourceGenerator.java:475) >>>>>>>>>>>> >>>>>>>>>>>> This happened because internal image cache for bundled (composite >>>>>>>>>>>> PNG) >>>>>>>>>>>> images was corrput. >>>>>>>>>>>> It seems to be caused by disk I/O error while reading file >>>>>>>>>>>> images/glusterAsyncTasks/remove_brick_commit_required.png >>>>>>>>>>>> >>>>>>>>>>>>> [INFO] java.lang.NullPointerException >>>>>>>>>>>>> [INFO] at >>>>>>>>>>>>> com.google.gwt.resources.rebind.context.InlineResourceContext.deploy(InlineResourceContext.java:40) >>>>>>>>>>>> >>>>>>>>>>>> This happened because image data was null, again possible disk I/O >>>>>>>>>>>> error >>>>>>>>>>>> while reading image file. >>>>>>>>>>>> >>>>>>>>>>>> Can we enable "TRACE" logging during GWT compilation? >>>>>>>>>>>> >>>>>>>>>>>> -Dgwt.logLevel=TRACE >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Building master right now with your suggestions here: >>>>>>>>>>> http://jenkins.ovirt.org/job/manual-build-tarball/227/ >>>>>>>>>> >>>>>>>>>> Looking at >>>>>>>>>> http://jenkins.ovirt.org/job/manual-build-tarball/227/label=fedora20-host/console >>>>>>>>>> >>>>>>>>>> So far I see no errors.. There should be total 5 browsers x 8 locales >>>>>>>>>> = >>>>>>>>>> 40 >>>>>>>>>> GWT permutations. >>>>>>>>> >>>>>>>>> So no changes with and without >>>>>>>>> -Dgwt.userAgent=ie8,ie9,gecko1_8,safari,opera. >>>>>>>>> We've still 40 permutations. Maybe you defined >>>>>>>>> userAgent=ie8,ie9,gecko1_8,safari,opera somewhere in the POM files? >>>>>>>> >>>>>>>> You are right. I checked frontend/webadmin/modules/pom.xml -> the >>>>>>>> "gwt.userAgent" >>>>>>>> property is copied into "locale" GWT property in >>>>>>>> {WebAdmin,UserPortal}.gwt.xml >>>>>>> >>>>>>> Ok, so I'll remove -Dgwt.userAgent=ie8,ie9,gecko1_8,safari,opera from >>>>>>> build >>>>>>> command. >>>>>> >>>>>> Agreed. >>>>>> >>>>>>> >>>>>>> >>>>>>>> >>>>>>>>> >>>>>>>>> This: >>>>>>>>> http://jenkins.ovirt.org/job/manual-build-tarball/227/label=centos6-host/console >>>>>>>>> failed, you can look at it, it has -Dgwt.logLevel=TRACE -X. >>>>>>>> >>>>>>>> Hm, it contains: >>>>>>>> >>>>>>>> [DEBUG] (f) logLevel = INFO >>>>>>>> >>>>>>>> Somehow the TRACE log level for GWT wasn't applied.. not sure why. Let >>>>>>>> me >>>>>>>> check >>>>>>>> this on local build. >>>>>>> >>>>>>> Ok, let me know if I've to change command line. >>>>>> >>>>>> I wrote upstream patch: http://gerrit.ovirt.org/#/c/24789/ >>>>>> >>>>>> When it's merged, setting "gwt.logLevel" will work (as you did before) >>>>>> and we'll be able to see more details in the build log. >>>>> >>>>> Here is the new build with your patch merged: >>>>> >>>>> http://jenkins.ovirt.org/job/manual-build-tarball/228/label=centos6-host/consoleFull >>>> >>>> Thanks. I did some more investigation on this. >>>> >>>> Again it's NullPointerException when rebinding ApplicationResources, but >>>> this time for different image file: ApplicationResources.wrenchImage >>>> ("images/wrench.png"). >>>> >>>> Looking at stacktrace & GWT sources, the problematic code seems to be >>>> (AbstractResourceContext:74): >>>> >>>> byte[] bytes = Util.readURLAsBytes(resource); >>>> >>>> where "resource" is (ImageResourceGenerator:311): >>>> >>>> localized.getUrl(); // LocalizedImage localized >>>> >>>> Note: in oVirt UI we don't use localized images (i.e. "wrench_fr_FR.png") >>>> so "localized" in this case points to "wrench.png". >>>> >>>> So the problem seems to be reading this file: >>>> >>>> >>>> frontend/webadmin/modules/webadmin/src/main/resources/org/ovirt/engine/ui/webadmin/images/wrench.png >>>> >>>> Since we're having many GWT permutations, this file ^^ (along with similar >>>> ones) is attempted to be read multiple times *from the same (single) GWT >>>> compiler Java process, before actual permutation worker Java sub-processes >>>> are spawned*. (More precisely, above NullPointerException happens at >>>> deferred binding phase which precedes actual GWT compilation per each >>>> permutation.) >>>> >>>> I think the root cause is Util.readURLAsBytes method used internally by GWT >>>> compiler: >>>> >>>> public static byte[] readURLAsBytes(URL url) { >>>> try { >>>> URLConnection conn = url.openConnection(); >>>> conn.setUseCaches(false); >>>> return readURLConnectionAsBytes(conn); >>>> } catch (IOException e) { >>>> return null; // This causes NullPointerException, but this exception >>>> isn't logged (wtf, GWT!) >>>> } >>>> } >>>> >>>> To summarize, I'll need to write another patch which replaces >>>> Util.readURLAsBytes implementation via AOP and prints IOException details. >>>> >>>>> >>>>> >>>>> >>>>> >>>>>> >>>>>>> >>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> # CI_MAVEN_SETTINGS=$WORKSPACE/artifactory-ovirt-org-settings.xml >>>>>>>>>>> # export MAVEN_OPTS="${MAVEN_OPTS} -Xmx2048M -XX:MaxPermSize=1024m >>>>>>>>>>> -Dgwt.compiler.localWorkers=1" >>>>>>>>>>> # rpmbuild -D "_topdir $(pwd)/rpmbuild" -ts "${TARBALL}" >>>>>>>>>>> # rpmbuild -D "_topdir $(pwd)/rpmbuild" -D "ovirt_build_extra_flags >>>>>>>>>>> -gs >>>>>>>>>>> ${CI_MAVEN_SETTINGS} -Dgwt.userAgent=ie8,ie9,gecko1_8,safari,opera >>>>>>>>>>> -Dgwt.logLevel=TRACE -X" --rebuild rpmbuild/SRPMS/*.src.rpm >>>>>>>>>>> >>>>>>>>>>> Let me know if you need anything else. >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> Vojtech >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> ----- Original Message ----- >>>>>>>>>>>>> From: "Vojtech Szocs" <[email protected]> >>>>>>>>>>>>> To: "Sandro Bonazzola" <[email protected]> >>>>>>>>>>>>> Cc: "Eyal Edri" <[email protected]>, "Juan Hernandez" >>>>>>>>>>>>> <[email protected]>, "Kiril Nesenko" <[email protected]>, >>>>>>>>>>>>> "arch" <[email protected]> >>>>>>>>>>>>> Sent: Tuesday, February 18, 2014 6:54:20 PM >>>>>>>>>>>>> Subject: Re: ovirt-engine build failure >>>>>>>>>>>>> >>>>>>>>>>>>> In your log I see following error: >>>>>>>>>>>>> >>>>>>>>>>>>> ... >>>>>>>>>>>>> [INFO] Computing all possible rebind results for >>>>>>>>>>>>> 'org.ovirt.engine.ui.webadmin.ApplicationResources' >>>>>>>>>>>>> [INFO] Rebinding >>>>>>>>>>>>> org.ovirt.engine.ui.webadmin.ApplicationResources >>>>>>>>>>>>> [INFO] Invoking generator >>>>>>>>>>>>> com.google.gwt.resources.rebind.context.InlineClientBundleGenerator >>>>>>>>>>>>> [INFO] [ERROR] Generator >>>>>>>>>>>>> 'com.google.gwt.resources.rebind.context.InlineClientBundleGenerator' >>>>>>>>>>>>> threw >>>>>>>>>>>>> an exception while rebinding >>>>>>>>>>>>> 'org.ovirt.engine.ui.webadmin.ApplicationResources' >>>>>>>>>>>>> [INFO] java.lang.NullPointerException: No ImageRect ever computed >>>>>>>>>>>>> for >>>>>>>>>>>>> removeBrickCommitRequired >>>>>>>>>>>>> [INFO] at >>>>>>>>>>>>> com.google.gwt.resources.rg.ImageResourceGenerator.createAssignment(ImageResourceGenerator.java:475) >>>>>>>>>>>>> ... >>>>>>>>>>>>> >>>>>>>>>>>>> I checked again logs at >>>>>>>>>>>>> http://jenkins.ovirt.org/job/manual-build-tarball/212/label=fedora19-host/consoleFull >>>>>>>>>>>>> and indeed there is a similar error: >>>>>>>>>>>>> >>>>>>>>>>>>> ... >>>>>>>>>>>>> [INFO] Computing all possible rebind results for >>>>>>>>>>>>> 'org.ovirt.engine.ui.webadmin.ApplicationResources' >>>>>>>>>>>>> [INFO] Rebinding >>>>>>>>>>>>> org.ovirt.engine.ui.webadmin.ApplicationResources >>>>>>>>>>>>> [INFO] Invoking generator >>>>>>>>>>>>> com.google.gwt.resources.rebind.context.InlineClientBundleGenerator >>>>>>>>>>>>> [INFO] [ERROR] Generator >>>>>>>>>>>>> 'com.google.gwt.resources.rebind.context.InlineClientBundleGenerator' >>>>>>>>>>>>> threw >>>>>>>>>>>>> an exception while rebinding >>>>>>>>>>>>> 'org.ovirt.engine.ui.webadmin.ApplicationResources' >>>>>>>>>>>>> [INFO] java.lang.NullPointerException >>>>>>>>>>>>> [INFO] at >>>>>>>>>>>>> com.google.gwt.resources.rebind.context.InlineResourceContext.deploy(InlineResourceContext.java:40) >>>>>>>>>>>>> ... >>>>>>>>>>>>> >>>>>>>>>>>>> NullPointerException when processing ApplicationResources but >>>>>>>>>>>>> occuring >>>>>>>>>>>>> from >>>>>>>>>>>>> different places (ImageResourceGenerator vs. >>>>>>>>>>>>> InlineResourceContext) >>>>>>>>>>>>> >>>>>>>>>>>>> Now I understand why Maven build fails without any details: >>>>>>>>>>>>> - GWT deferred binding fails for ApplicationResources (i.e. >>>>>>>>>>>>> uncaught >>>>>>>>>>>>> NPE) >>>>>>>>>>>>> - GWT compilation proceeds, fails to link with >>>>>>>>>>>>> ApplicationResources >>>>>>>>>>>>> - GWT compilation task fails with non-OK status code >>>>>>>>>>>>> >>>>>>>>>>>>> So the issue here is GWT deferred binding generator >>>>>>>>>>>>> (InlineClientBundleGenerator) failure during GWT compilation. >>>>>>>>>>>>> >>>>>>>>>>>>> GWT compiler writes extra Java files during deferred binding >>>>>>>>>>>>> phase, so this could point to issue with local environment.. >>>>>>>>>>>>> >>>>>>>>>>>>> I will try to investigate this and look at GWT source code. >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> -- >>>>>>>>>>> Sandro Bonazzola >>>>>>>>>>> Better technology. Faster innovation. Powered by community >>>>>>>>>>> collaboration. >>>>>>>>>>> See how it works at redhat.com >>>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> Sandro Bonazzola >>>>>>>>> Better technology. Faster innovation. Powered by community >>>>>>>>> collaboration. >>>>>>>>> See how it works at redhat.com >>>>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> Sandro Bonazzola >>>>>>> Better technology. Faster innovation. Powered by community >>>>>>> collaboration. >>>>>>> See how it works at redhat.com >>>>>>> >>>>> >>>>> >>>>> -- >>>>> Sandro Bonazzola >>>>> Better technology. Faster innovation. Powered by community collaboration. >>>>> See how it works at redhat.com >>>>> >>> >>> >>> -- >>> Sandro Bonazzola >>> Better technology. Faster innovation. Powered by community collaboration. >>> See how it works at redhat.com >>> > > -- Sandro Bonazzola Better technology. Faster innovation. Powered by community collaboration. See how it works at redhat.com _______________________________________________ Arch mailing list [email protected] http://lists.ovirt.org/mailman/listinfo/arch
