So back to a divide and conquer approach I think.
I just created a trivial "example" [1] and it works as expected (i.e. no
exception was generated) (on the same servers I was testing the complex JAR
file) -- so possibly something else modifying the behaviour -- a JAR on the
classpath, or something in the JSP file...
[1]
$ cat src/main/webapp/index.jsp
<%@page contentType="text/html" pageEncoding="UTF-8" isELIgnored="false" %>
<%@page import="com.example.rt.Failing"%>
<%
final Failing failing = null;
pageContext.setAttribute("failing", failing);
%>
<html>
<body>
<h2>Hello World!</h2>
<p>field1=${failing.field1}</p>
</body>
</html>
$ cat src/main/java/com/example/rt/Failing.java
package com.example.rt;
public class Failing {
private final String field1 = "field1_value";
public String getField1() {
return field1;
}
}
On Mon, Feb 7, 2022 at 9:51 PM Robert Turner <rtur...@e-djuster.ca> wrote:
> I'm just avoiding sharing product details or things I think only serves to
> confuse the problem. Sorry if you felt I wasn't sharing. It wasn't my
> intention to be obtuse. I didn't believe they added any value for the
> diagnostics (of course that assumes I know enough about the problem).
>
> However, since you think they might be useful, here they the name mapping
> from the exception and the Java and JSP code excerpts:
>
> "package1" -> "model"
> "Class1" -> "Organization"
> "class1" -> "organization"
>
> The class is present in the package (see [1]) -- otherwise it wouldn't
> work in one environment and not the other -- I believe I have confirmed
> that carefully. Full class paths (with path-names relativised to compare
> between environments) listed below [2], and the code that emitted the
> listing is here [3]. I post-processed the log lines as follows [4]. Docker
> file for the Tomcat container provided [5]. JDK details listed in [6].
> Tomcat version is 9.0.58 in all cases.
>
> What seems to be different is:
> - the way the EL resolver is working
> OR
> - the behaviour of the class loader differs in the different environments.
> OR
> - something else I do not understand is relevant
>
> The working environments are MacOS and Windows, and the failing
> environment is either Debian or AWS Linux 2 running in a docker container.
> If the class loaders behaviour differently, then that could explain the
> issues, however, that would surprise me if they differed in any material
> way on the different platforms.
>
> I hope that helps provide more detail that might be useful...
>
> Robert
>
>
> [1]
> $ unzip -l target/app.war | grep "model\/Organization\.class"
> 66246 02-07-2022 20:17 WEB-INF/classes/model/Organization.class
>
>
> [2]
> java.class.path=/bin/bootstrap.jar:/bin/tomcat-juli.jar
> Class loader URLs:
> Class loader name=org.apache.catalina.loader.ParallelWebappClassLoader,
> URL count=146
> URL=file:/WEB-INF/classes/
> URL=file:/WEB-INF/lib/FastInfoset-1.2.18.jar
> URL=file:/WEB-INF/lib/SparseBitSet-1.2.jar
> URL=file:/WEB-INF/lib/VeracodeAnnotations-1.2.1.jar
> URL=file:/WEB-INF/lib/activation-1.1.jar
> URL=file:/WEB-INF/lib/animal-sniffer-annotations-1.20.jar
> URL=file:/WEB-INF/lib/annotations-4.1.1.4.jar
> URL=file:/WEB-INF/lib/api-common-2.0.1.jar
> URL=file:/WEB-INF/lib/auto-value-annotations-1.8.2.jar
> URL=file:/WEB-INF/lib/avatax-rest-v2-api-java_2.11-21.12.1.jar
> URL=file:/WEB-INF/lib/aws-java-sdk-core-1.12.145.jar
> URL=file:/WEB-INF/lib/aws-java-sdk-kms-1.12.145.jar
> URL=file:/WEB-INF/lib/aws-java-sdk-s3-1.12.145.jar
> URL=file:/WEB-INF/lib/bcmail-jdk15on-1.70.jar
> URL=file:/WEB-INF/lib/bcpkix-jdk15on-1.70.jar
> URL=file:/WEB-INF/lib/bcprov-jdk15on-1.70.jar
> URL=file:/WEB-INF/lib/bcutil-jdk15on-1.70.jar
> URL=file:/WEB-INF/lib/castor-core-1.4.1.jar
> URL=file:/WEB-INF/lib/castor-xml-1.4.1.jar
> URL=file:/WEB-INF/lib/checker-qual-3.5.0.jar
> URL=file:/WEB-INF/lib/commons-beanutils-1.9.4.jar
> URL=file:/WEB-INF/lib/commons-codec-1.15.jar
> URL=file:/WEB-INF/lib/commons-collections-3.2.2.jar
> URL=file:/WEB-INF/lib/commons-collections4-4.4.jar
> URL=file:/WEB-INF/lib/commons-compress-1.21.jar
> URL=file:/WEB-INF/lib/commons-digester-2.1.jar
> URL=file:/WEB-INF/lib/commons-fileupload-1.4.jar
> URL=file:/WEB-INF/lib/commons-io-2.11.0.jar
> URL=file:/WEB-INF/lib/commons-lang3-3.12.0.jar
> URL=file:/WEB-INF/lib/commons-logging-1.2.jar
> URL=file:/WEB-INF/lib/commons-math3-3.6.1.jar
> URL=file:/WEB-INF/lib/commons-text-1.9.jar
> URL=file:/WEB-INF/lib/conscrypt-openjdk-uber-2.5.1.jar
> URL=file:/WEB-INF/lib/curvesapi-1.06.jar
> URL=file:/WEB-INF/lib/ecj-3.21.0.jar
> URL=file:/WEB-INF/lib/encoder-1.2.3.jar
> URL=file:/WEB-INF/lib/encoder-jsp-1.2.3.jar
> URL=file:/WEB-INF/lib/error_prone_annotations-2.9.0.jar
> URL=file:/WEB-INF/lib/failureaccess-1.0.1.jar
> URL=file:/WEB-INF/lib/fontbox-2.0.25.jar
> URL=file:/WEB-INF/lib/gax-2.3.0.jar
> URL=file:/WEB-INF/lib/gax-grpc-2.3.0.jar
> URL=file:/WEB-INF/lib/gmbal-api-only-4.0.3.jar
> URL=file:/WEB-INF/lib/google-auth-library-credentials-1.1.0.jar
> URL=file:/WEB-INF/lib/google-auth-library-oauth2-http-1.1.0.jar
> URL=file:/WEB-INF/lib/google-cloud-speech-1.30.5.jar
> URL=file:/WEB-INF/lib/google-http-client-1.39.2.jar
> URL=file:/WEB-INF/lib/google-http-client-gson-1.39.2.jar
> URL=file:/WEB-INF/lib/googleauth-1.5.0.jar
> URL=file:/WEB-INF/lib/grpc-alts-1.40.0.jar
> URL=file:/WEB-INF/lib/grpc-api-1.40.0.jar
> URL=file:/WEB-INF/lib/grpc-auth-1.40.0.jar
> URL=file:/WEB-INF/lib/grpc-context-1.40.0.jar
> URL=file:/WEB-INF/lib/grpc-core-1.40.0.jar
> URL=file:/WEB-INF/lib/grpc-grpclb-1.40.0.jar
> URL=file:/WEB-INF/lib/grpc-netty-shaded-1.40.0.jar
> URL=file:/WEB-INF/lib/grpc-protobuf-1.40.0.jar
> URL=file:/WEB-INF/lib/grpc-protobuf-lite-1.40.0.jar
> URL=file:/WEB-INF/lib/grpc-stub-1.40.0.jar
> URL=file:/WEB-INF/lib/gson-2.8.9.jar
> URL=file:/WEB-INF/lib/guava-31.0.1-jre.jar
> URL=file:/WEB-INF/lib/ha-api-3.1.13.jar
> URL=file:/WEB-INF/lib/handlebars-4.3.0.jar
> URL=file:/WEB-INF/lib/httpclient-4.5.13.jar
> URL=file:/WEB-INF/lib/httpcore-4.4.15.jar
> URL=file:/WEB-INF/lib/ion-java-1.0.2.jar
> URL=file:/WEB-INF/lib/ipaddress-5.3.3.jar
> URL=file:/WEB-INF/lib/itext-2.1.7.js9.jar
> URL=file:/WEB-INF/lib/j2objc-annotations-1.3.jar
> URL=file:/WEB-INF/lib/jackson-annotations-2.13.1.jar
> URL=file:/WEB-INF/lib/jackson-core-2.13.1.jar
> URL=file:/WEB-INF/lib/jackson-databind-2.13.1.jar
> URL=file:/WEB-INF/lib/jackson-dataformat-cbor-2.13.1.jar
> URL=file:/WEB-INF/lib/jai-imageio-core-1.4.0.jar
> URL=file:/WEB-INF/lib/jakarta.activation-2.0.1.jar
> URL=file:/WEB-INF/lib/jakarta.annotation-api-1.3.5.jar
> URL=file:/WEB-INF/lib/jakarta.jws-api-2.1.0.jar
> URL=file:/WEB-INF/lib/jakarta.mail-1.6.7.jar
> URL=file:/WEB-INF/lib/jakarta.xml.bind-api-2.3.3.jar
> URL=file:/WEB-INF/lib/jakarta.xml.soap-api-1.4.2.jar
> URL=file:/WEB-INF/lib/jakarta.xml.ws-api-2.3.3.jar
> URL=file:/WEB-INF/lib/jasperreports-6.18.1.jar
> URL=file:/WEB-INF/lib/jasperreports-fonts-6.18.1.jar
> URL=file:/WEB-INF/lib/java-saml-2.8.0.jar
> URL=file:/WEB-INF/lib/java-saml-core-2.8.0.jar
> URL=file:/WEB-INF/lib/javax.activation-api-1.2.0.jar
> URL=file:/WEB-INF/lib/javax.annotation-api-1.3.2.jar
> URL=file:/WEB-INF/lib/javax.inject-1.jar
> URL=file:/WEB-INF/lib/javax.mail-1.6.2.jar
> URL=file:/WEB-INF/lib/javax.xml.soap-api-1.4.0.jar
> URL=file:/WEB-INF/lib/jaxb-api-2.3.1.jar
> URL=file:/WEB-INF/lib/jaxb-core-2.3.0.1.jar
> URL=file:/WEB-INF/lib/jaxb-impl-2.3.5.jar
> URL=file:/WEB-INF/lib/jaxws-api-2.3.1.jar
> URL=file:/WEB-INF/lib/jaxws-rt-2.3.5.jar
> URL=file:/WEB-INF/lib/jbig2-imageio-3.0.3.jar
> URL=file:/WEB-INF/lib/jcommon-1.0.24.jar
> URL=file:/WEB-INF/lib/jempbox-1.8.16.jar
> URL=file:/WEB-INF/lib/jettison-1.4.1.jar
> URL=file:/WEB-INF/lib/jfreechart-1.0.19.jar
> URL=file:/WEB-INF/lib/jmespath-java-1.12.145.jar
> URL=file:/WEB-INF/lib/joda-time-2.10.6.jar
> URL=file:/WEB-INF/lib/jsch-0.1.55.jar
> URL=file:/WEB-INF/lib/jsoup-1.14.3.jar
> URL=file:/WEB-INF/lib/jsr181-api-1.0-MR1.jar
> URL=file:/WEB-INF/lib/jsr305-3.0.2.jar
>
> URL=file:/WEB-INF/lib/listenablefuture-9999.0-empty-to-avoid-conflict-with-guava.jar
> URL=file:/WEB-INF/lib/log4j-api-2.17.1.jar
> URL=file:/WEB-INF/lib/log4j-core-2.17.1.jar
> URL=file:/WEB-INF/lib/log4j-slf4j-impl-2.17.1.jar
> URL=file:/WEB-INF/lib/log4j-web-2.17.1.jar
> URL=file:/WEB-INF/lib/management-api-3.2.3.jar
> URL=file:/WEB-INF/lib/metadata-extractor-2.16.0.jar
> URL=file:/WEB-INF/lib/mimepull-1.9.15.jar
> URL=file:/WEB-INF/lib/opencensus-api-0.28.0.jar
> URL=file:/WEB-INF/lib/opencensus-contrib-http-util-0.28.0.jar
> URL=file:/WEB-INF/lib/owasp-java-html-sanitizer-20211018.2.jar
> URL=file:/WEB-INF/lib/pdfbox-2.0.25.jar
> URL=file:/WEB-INF/lib/perfmark-api-0.23.0.jar
> URL=file:/WEB-INF/lib/poi-4.1.2.jar
> URL=file:/WEB-INF/lib/poi-ooxml-4.1.2.jar
> URL=file:/WEB-INF/lib/poi-ooxml-schemas-4.1.2.jar
> URL=file:/WEB-INF/lib/policy-2.7.10.jar
> URL=file:/WEB-INF/lib/postgresql-42.3.1.jar
> URL=file:/WEB-INF/lib/proto-google-cloud-speech-v1-1.30.5.jar
> URL=file:/WEB-INF/lib/proto-google-cloud-speech-v1p1beta1-0.83.5.jar
> URL=file:/WEB-INF/lib/proto-google-common-protos-2.3.2.jar
> URL=file:/WEB-INF/lib/protobuf-java-3.17.3.jar
> URL=file:/WEB-INF/lib/protobuf-java-util-3.17.3.jar
> URL=file:/WEB-INF/lib/saaj-impl-1.5.3.jar
> URL=file:/WEB-INF/lib/scala-library-2.11.12.jar
> URL=file:/WEB-INF/lib/slf4j-api-1.7.33.jar
> URL=file:/WEB-INF/lib/stax-api-1.0.1.jar
> URL=file:/WEB-INF/lib/stax-ex-1.8.3.jar
> URL=file:/WEB-INF/lib/stax2-api-4.2.1.jar
> URL=file:/WEB-INF/lib/streambuffer-2.0.2.jar
> URL=file:/WEB-INF/lib/taglibs-standard-impl-1.2.5.jar
> URL=file:/WEB-INF/lib/taglibs-standard-spec-1.2.5.jar
> URL=file:/WEB-INF/lib/threetenbp-1.3.5.jar
> URL=file:/WEB-INF/lib/tika-core-2.2.1.jar
> URL=file:/WEB-INF/lib/tika-parser-image-module-2.2.1.jar
> URL=file:/WEB-INF/lib/tika-parser-xmp-commons-2.2.1.jar
> URL=file:/WEB-INF/lib/woodstox-core-6.2.6.jar
> URL=file:/WEB-INF/lib/xmlbeans-3.1.0.jar
> URL=file:/WEB-INF/lib/xmlsec-2.2.3.jar
> URL=file:/WEB-INF/lib/xmpcore-6.1.11.jar
> Class loader name=java.net.URLClassLoader, URL count=33
> URL=file:/lib/
> URL=file:/lib/annotations-api.jar
> URL=file:/lib/catalina-ant.jar
> URL=file:/lib/catalina-ha.jar
> URL=file:/lib/catalina-ssi.jar
> URL=file:/lib/catalina-storeconfig.jar
> URL=file:/lib/catalina-tribes.jar
> URL=file:/lib/catalina.jar
> URL=file:/lib/ecj-4.20.jar
> URL=file:/lib/el-api.jar
> URL=file:/lib/jasper-el.jar
> URL=file:/lib/jasper.jar
> URL=file:/lib/jaspic-api.jar
> URL=file:/lib/jsp-api.jar
> URL=file:/lib/servlet-api.jar
> URL=file:/lib/tomcat-api.jar
> URL=file:/lib/tomcat-coyote.jar
> URL=file:/lib/tomcat-dbcp.jar
> URL=file:/lib/tomcat-i18n-cs.jar
> URL=file:/lib/tomcat-i18n-de.jar
> URL=file:/lib/tomcat-i18n-es.jar
> URL=file:/lib/tomcat-i18n-fr.jar
> URL=file:/lib/tomcat-i18n-ja.jar
> URL=file:/lib/tomcat-i18n-ko.jar
> URL=file:/lib/tomcat-i18n-pt-BR.jar
> URL=file:/lib/tomcat-i18n-ru.jar
> URL=file:/lib/tomcat-i18n-zh-CN.jar
> URL=file:/lib/tomcat-jdbc.jar
> URL=file:/lib/tomcat-jni.jar
> URL=file:/lib/tomcat-util-scan.jar
> URL=file:/lib/tomcat-util.jar
> URL=file:/lib/tomcat-websocket.jar
> URL=file:/lib/websocket-api.jar
> Class loader name=jdk.internal.loader.ClassLoaders$AppClassLoader
> Class loader name=jdk.internal.loader.ClassLoaders$PlatformClassLoader
>
>
> [3]
> private static void logClassLoaderUrls(org.apache.logging.log4j.Level
> logLevel, ClassLoader cl) {
> if (null == cl) {
> return;
> }
> boolean loaderNameLogged = false;
> if (cl instanceof URLClassLoader) {
> final URLClassLoader urlCl = (URLClassLoader) cl;
> final URL[] urls = urlCl.getURLs();
> if (null != urls) {
> LOG.log(logLevel, " Class loader name={}, URL count={}",
> cl.getClass().getName(), urls.length);
> loaderNameLogged = true;
> for (int idx = 0; idx < urls.length; idx++) {
> final URL url = urls[idx];
> LOG.log(logLevel, " URL[{}]={}", idx, url);
> }
> }
> }
> if (!loaderNameLogged) {
> LOG.log(logLevel, " Class loader name={}",
> cl.getClass().getName());
> }
> logClassLoaderUrls(logLevel, cl.getParent());
> }
>
> ..snip..
>
> LOG.info("java.class.path={}",
> System.getProperty("java.class.path"));
> LOG.debug("Class loader URLs:");
> logClassLoaderUrls(org.apache.logging.log4j.Level.DEBUG,
> ctx.getClassLoader());
>
>
> [4]
> cat log-extract.txt | cut -c137- | sed -e "s+/usr/local/tomcat++g" -e
> "s+/usr/local/tomcat/webapps/exclaim++g" -e "s+\[[0-9]*\]++"
> and I manually sorted the "sections" using vim, with the "sort" command.
>
>
> [5]
> # Debian with OpenJDK
> #FROM tomcat:9.0.58-jre11-openjdk AS tomcat_package
> # AWS Linux 2 with Corretto
> FROM tomcat:9.0.58-jdk11-corretto AS tomcat_package
>
> # Set the timezone to Canada/Eastern
> # For Debian
> #RUN ln -sf /usr/share/zoneinfo/Canada/Eastern /etc/localtime && echo
> "Canada/Eastern" > /etc/timezone && dpkg-reconfigure -f noninteractive
> tzdata
> # For AWS Linux
> RUN ln -sf /usr/share/zoneinfo/Canada/Eastern /etc/localtime && sed -i
> 's/\"UTC\"/\"Canada\/Eastern\"/' /etc/sysconfig/clock # and reboot, but
> will skip as container will be created and ran separately
>
> # Install a postgresql client for testing and the required font packages
> needed by JasperReports
> # For Debian:
> #RUN echo "deb http://deb.debian.org/debian bullseye contrib non-free" >>
> /etc/apt/sources.list && apt-get update && apt-get install -y
> postgresql-client ttf-mscorefonts-installer
> # For AWS Linux:
> RUN yum -q list installed epel-release.noarch &>/dev/null && echo
> 'epel-release already installed' || yum install -y
> https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm
> RUN yum install -y cabextract xorg-x11-font-utils fontconfig
> RUN rpm -q msttcore-fonts-installer-2.6-1 &>/dev/null && echo
> 'msttcore-fonts-installer-2.6-1 already installed' || rpm -i
> https://iweb.dl.sourceforge.net/project/mscorefonts2/rpms/msttcore-fonts-installer-2.6-1.noarch.rpm
>
> # Add a health-check script to wait for exclaim to be running -- allow at
> least 10 minutes for it to start
> COPY health_check.sh /usr/local/bin/
> HEALTHCHECK --interval=10s --timeout=5s --start-period=10m --retries=10
> CMD /usr/local/bin/health_check.sh
>
> # Set up the resource folders needed by application and make it a
> # volume so it persists across container restarts
> COPY resources/ /resources/
> VOLUME /resources
>
> # Map some of the Tomcat folders to volumes so they persist between
> restarts of the container
> VOLUME /usr/local/tomcat/logs
> VOLUME /usr/local/tomcat/webapps
> VOLUME /usr/local/tomcat/work
> VOLUME /usr/local/tomcat/conf
>
> # Enable connecting from anywhere for JPDA (req'd for Java 9+)
> ENV JPDA_ADDRESS=*:8000
>
> CMD ["catalina.sh","jpda","run"]
>
>
> [6]
> JDK on MacOS is:
> openjdk version "11.0.13" 2021-10-19 LTS
> OpenJDK Runtime Environment Corretto-11.0.13.8.1 (build 11.0.13+8-LTS)
> OpenJDK 64-Bit Server VM Corretto-11.0.13.8.1 (build 11.0.13+8-LTS, mixed
> mode)
>
> JDK on Windows is:
> openjdk 11.0.14 2022-01-18 LTS
> OpenJDK Runtime Environment Corretto-11.0.14.9.1 (build 11.0.14+9-LTS)
> OpenJDK 64-Bit Server VM Corretto-11.0.14.9.1 (build 11.0.14+9-LTS, mixed
> mode)
>
> JDK on Docker with AWS Linux 2 is:
> openjdk 11.0.14 2022-01-18 LTS
> OpenJDK Runtime Environment Corretto-11.0.14.9.1 (build 11.0.14+9-LTS)
> OpenJDK 64-Bit Server VM Corretto-11.0.14.9.1 (build 11.0.14+9-LTS, mixed
> mode)
>
>
>
> On Mon, Feb 7, 2022 at 9:24 PM Rob Sargent <rsarg...@xmission.com> wrote:
>
>>
>>
>> On 2/7/22 19:13, Robert Turner wrote:
>> > So, I've gone back and double-checked as much as I can (Tomcat version,
>> JDK
>> > version), and the classpath, and I have identical classpaths in both
>> > environments (except the sort order of the URLs per "level" of
>> > ClassLoader), and I've re-verified the behaviour:
>> > - fails in the docker environment
>> > - works locally
>> > (Same WAR file on both).
>> >
>> > I guess I'm in to one of the following approaches next:
>> > - build a debug version of Tomcat, and step through the code that
>> breaks
>> > and see if I can figure out why (not that I have time to do this of
>> > course...but might be necessary)
>> > - construct a trivial application reproduction, along with docker
>> layout,
>> > and see if anyone else can reproduce... (assuming anyone else has time
>> to
>> > do that of course...)
>> >
>> > Anyone got any suggestions of what to look into next?
>> >
>> >
>> > On Mon, Feb 7, 2022 at 5:05 PM Rob Sargent <rsarg...@xmission.com>
>> wrote:
>> >
>> >
>> As you appear averse to sharing names of things, seems you will need to
>> do a lot of extra clean-up before you can share a docker image. Make
>> sure you're clear on what NoClassDefFoundError indicates (present at
>> compile time, absent at runtime) and how that translates to docker-ness
>> (of which I know naught).
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
>> For additional commands, e-mail: users-h...@tomcat.apache.org
>>
>>
