Mark,
Thanks for the quick follow up.
Based on your comments I have some ideas on what it might be, and I should
be able to narrow that down further for you, and provide all the details
that you requested.
A few notes on the environment I'm using:
Docker (the host) is running on my Mac (12.2), not in AWS. AWS Linux is
only involved in the Docker image (as per the Dockerfile I shared earlier).
Moreover, it happens with Debian in the Docker image as well.
What I think is important is that I have "<tomcat>/work" and
"<tomcat>/webapps" mounted to file folders on the Mac, so I am suspecting
it's something to do with that "mounting" of the volume to the host OS, and
how Docker is mapping those and handling filename casing, etc. My MacOS
file system is APFS, Encrypted (and I thought I had case sensitivity
enabled, but I can no longer see that option -- maybe not an option for
APFS).
I will try to confirm suspicions and provide details in a few hours
(hopefully -- got a few meetings today that will get in the way).
Thanks again,
Robert
On Tue, Feb 8, 2022 at 8:51 AM Mark Thomas <ma...@apache.org> wrote:
> Robert,
>
> I agree this is something to do with the Docker environment.
>
> I think case insensitivity is involved somewhere as I can trigger the
> error if I copy my equivalent of Failure.class to failure.class and then
> call the JSP.
>
> I understand why it only occurs for * imports. In that instance, Tomcat
> has to check if the class can be loaded from each of the imported
> packages. Tomcat checks the file system first as that is faster than
> trying (and failing) to load a class. The file system checks are
> designed to be case sensitive - even on case insensitive file systems.
> Something seems to be going wrong here and I'm still not sure what.
>
> I have tried to recreate this on AWS without success. If you have time,
> I think we'd need the following to dig into this further:
>
> - Source for the simplest possible test WAR that demonstrates this
> issue. I think a single class and a single JSP page should be
> sufficient.
>
> - The WAR you created from the above (to rule out any build issues).
>
> - A minimal Dockerfile to create a Tomcat instance that demonstrates
> this issue. Should just copy the WAR to the container and start it
> with JPDA enabled.
>
> - Which AMI you used to create the AWS instance. I'm using the AWS free
> tier so I used a t2.micro instance with
> amzn2-ami-kernel-5.10-hvm-2.0.20220121.0-x86_64-gp2
>
> Thanks,
>
> Mark
>
>
> On 08/02/2022 13:24, Robert Turner wrote:
> > One thing to add is that my "conclusion" about OS variances I believe to
> be
> > incorrect. Our tests typically run on Linux, so I think it's still
> > something to do with a difference in the Docker-based environment.
> >
> > Let me know if you need any more details on anything...(but I suspect
> with
> > a debugger up on the Expression Resolvers, you will at least narrow it
> down
> > quickly...)
> >
> > On Tue, Feb 8, 2022 at 7:55 AM Robert Turner <rtur...@e-djuster.ca>
> wrote:
> >
> >> Thanks Mark. Much appreciated.
> >>
> >> On Tue., Feb. 8, 2022, 04:06 Mark Thomas, <ma...@apache.org> wrote:
> >>
> >>> Robert,
> >>>
> >>> Thank you for putting the effort in to debugging this. Narrowing down
> >>> the issue to a simple test case is extremely helpful.
> >>>
> >>> The behaviour you describe looks odd to me. I'd expect consistent
> >>> behaviour across platforms irrespective of the case sensitivity of the
> >>> file system in use.
> >>>
> >>> I'm going to use your test case to investigate this further. I'll
> report
> >>> back here with my findings - hopefully later today.
> >>>
> >>> Mark
> >>>
> >>>
> >>> On 08/02/2022 03:29, Robert Turner wrote:
> >>>> Okay, so I have finally narrowed it down the trivial failure case,
> and I
> >>>> think I have an explanation as a result:
> >>>>
> >>>> [1] works (in docker), and [2] fails (in docker) but works outside.
> The
> >>>> difference between the two is the import directive being a wildcard
> >>> (ugly,
> >>>> but historical in our app in some places we haven't yet cleaned up).
> >>>>
> >>>> I am therefore speculating based on the Expression Language
> >>> specification
> >>>> that because the class wasn't explicitly imported, it's not in the
> list
> >>> of
> >>>> available classes for static class resolution, and thus it fails.
> >>>> Combine this with MacOS and Windows not caring about filename cases,
> and
> >>>> Linux caring, then I suspect it's just matching differently in both
> >>> cases.
> >>>>
> >>>> Workaround/fix would be:
> >>>> - to ensure we explicitly import the class (instead or in addition to
> >>> the
> >>>> wildcard)
> >>>> OR
> >>>> - rename the attribute so it doesn't map directly to the class name.
> >>>>
> >>>>
> >>>> So I think I can bring my overly-detailed thread to an end...unless my
> >>>> guess at the reasoning is incorrect and someone has a better
> >>> explanation.
> >>>>
> >>>> Thanks Rob S and Neil for having a look and providing suggestions --
> in
> >>>> part, it was something related to what you both said, but I believe in
> >>>> different contexts than you expected.
> >>>>
> >>>> Robert
> >>>>
> >>>>
> >>>> [1]
> >>>> $ cat src/main/webapp/index.jsp
> >>>> <%@page contentType="text/html" pageEncoding="UTF-8"
> >>> isELIgnored="false" %>
> >>>> <%@page import="com.example.rt.Failing" %>
> >>>> <%
> >>>> final Failing failing = null;
> >>>> pageContext.setAttribute("failing", failing);
> >>>> %>
> >>>> <html>
> >>>> <body>
> >>>> <h2>Hello World!</h2>
> >>>>
> >>>> <p>field1=${failing.field1}</p>
> >>>> </body>
> >>>>
> >>>> [2]
> >>>> $ cat src/main/webapp/index.jsp
> >>>> <%@page contentType="text/html" pageEncoding="UTF-8"
> >>> isELIgnored="false" %>
> >>>> <%@page import="com.example.rt.*" %>
> >>>> <%
> >>>> final Failing failing = null;
> >>>> pageContext.setAttribute("failing", failing);
> >>>> %>
> >>>> <html>
> >>>> <body>
> >>>> <h2>Hello World!</h2>
> >>>>
> >>>> <p>field1=${failing.field1}</p>
> >>>> </body>
> >>>>
> >>>>
> >>>>
> >>>> On Mon, Feb 7, 2022 at 10:14 PM Robert Turner <rtur...@e-djuster.ca>
> >>> wrote:
> >>>>
> >>>>> So back to a divide and conquer approach I think.
> >>>>>
> >>>>> I just created a trivial "example" [1] and it works as expected (i.e.
> >>> no
> >>>>> exception was generated) (on the same servers I was testing the
> >>> complex JAR
> >>>>> file) -- so possibly something else modifying the behaviour -- a JAR
> >>> on the
> >>>>> classpath, or something in the JSP file...
> >>>>>
> >>>>> [1]
> >>>>>
> >>>>> $ cat src/main/webapp/index.jsp
> >>>>> <%@page contentType="text/html" pageEncoding="UTF-8"
> >>> isELIgnored="false" %>
> >>>>> <%@page import="com.example.rt.Failing"%>
> >>>>> <%
> >>>>> final Failing failing = null;
> >>>>> pageContext.setAttribute("failing", failing);
> >>>>> %>
> >>>>> <html>
> >>>>> <body>
> >>>>> <h2>Hello World!</h2>
> >>>>>
> >>>>> <p>field1=${failing.field1}</p>
> >>>>> </body>
> >>>>> </html>
> >>>>>
> >>>>> $ cat src/main/java/com/example/rt/Failing.java
> >>>>> package com.example.rt;
> >>>>>
> >>>>> public class Failing {
> >>>>>
> >>>>> private final String field1 = "field1_value";
> >>>>>
> >>>>> public String getField1() {
> >>>>> return field1;
> >>>>> }
> >>>>>
> >>>>> }
> >>>>>
> >>>>>
> >>>>>
> >>>>> On Mon, Feb 7, 2022 at 9:51 PM Robert Turner <rtur...@e-djuster.ca>
> >>> wrote:
> >>>>>
> >>>>>> I'm just avoiding sharing product details or things I think only
> >>> serves
> >>>>>> to confuse the problem. Sorry if you felt I wasn't sharing. It
> wasn't
> >>> my
> >>>>>> intention to be obtuse. I didn't believe they added any value for
> the
> >>>>>> diagnostics (of course that assumes I know enough about the
> problem).
> >>>>>>
> >>>>>> However, since you think they might be useful, here they the name
> >>> mapping
> >>>>>> from the exception and the Java and JSP code excerpts:
> >>>>>>
> >>>>>> "package1" -> "model"
> >>>>>> "Class1" -> "Organization"
> >>>>>> "class1" -> "organization"
> >>>>>>
> >>>>>> The class is present in the package (see [1]) -- otherwise it
> wouldn't
> >>>>>> work in one environment and not the other -- I believe I have
> >>> confirmed
> >>>>>> that carefully. Full class paths (with path-names relativised to
> >>> compare
> >>>>>> between environments) listed below [2], and the code that emitted
> the
> >>>>>> listing is here [3]. I post-processed the log lines as follows [4].
> >>> Docker
> >>>>>> file for the Tomcat container provided [5]. JDK details listed in
> [6].
> >>>>>> Tomcat version is 9.0.58 in all cases.
> >>>>>>
> >>>>>> What seems to be different is:
> >>>>>> - the way the EL resolver is working
> >>>>>> OR
> >>>>>> - the behaviour of the class loader differs in the different
> >>>>>> environments.
> >>>>>> OR
> >>>>>> - something else I do not understand is relevant
> >>>>>>
> >>>>>> The working environments are MacOS and Windows, and the failing
> >>>>>> environment is either Debian or AWS Linux 2 running in a docker
> >>> container.
> >>>>>> If the class loaders behaviour differently, then that could explain
> >>> the
> >>>>>> issues, however, that would surprise me if they differed in any
> >>> material
> >>>>>> way on the different platforms.
> >>>>>>
> >>>>>> I hope that helps provide more detail that might be useful...
> >>>>>>
> >>>>>> Robert
> >>>>>>
> >>>>>>
> >>>>>> [1]
> >>>>>> $ unzip -l target/app.war | grep "model\/Organization\.class"
> >>>>>> 66246 02-07-2022 20:17
> >>> WEB-INF/classes/model/Organization.class
> >>>>>>
> >>>>>>
> >>>>>> [2]
> >>>>>> java.class.path=/bin/bootstrap.jar:/bin/tomcat-juli.jar
> >>>>>> Class loader URLs:
> >>>>>> Class loader
> >>> name=org.apache.catalina.loader.ParallelWebappClassLoader,
> >>>>>> URL count=146
> >>>>>> URL=file:/WEB-INF/classes/
> >>>>>> URL=file:/WEB-INF/lib/FastInfoset-1.2.18.jar
> >>>>>> URL=file:/WEB-INF/lib/SparseBitSet-1.2.jar
> >>>>>> URL=file:/WEB-INF/lib/VeracodeAnnotations-1.2.1.jar
> >>>>>> URL=file:/WEB-INF/lib/activation-1.1.jar
> >>>>>> URL=file:/WEB-INF/lib/animal-sniffer-annotations-1.20.jar
> >>>>>> URL=file:/WEB-INF/lib/annotations-4.1.1.4.jar
> >>>>>> URL=file:/WEB-INF/lib/api-common-2.0.1.jar
> >>>>>> URL=file:/WEB-INF/lib/auto-value-annotations-1.8.2.jar
> >>>>>> URL=file:/WEB-INF/lib/avatax-rest-v2-api-java_2.11-21.12.1.jar
> >>>>>> URL=file:/WEB-INF/lib/aws-java-sdk-core-1.12.145.jar
> >>>>>> URL=file:/WEB-INF/lib/aws-java-sdk-kms-1.12.145.jar
> >>>>>> URL=file:/WEB-INF/lib/aws-java-sdk-s3-1.12.145.jar
> >>>>>> URL=file:/WEB-INF/lib/bcmail-jdk15on-1.70.jar
> >>>>>> URL=file:/WEB-INF/lib/bcpkix-jdk15on-1.70.jar
> >>>>>> URL=file:/WEB-INF/lib/bcprov-jdk15on-1.70.jar
> >>>>>> URL=file:/WEB-INF/lib/bcutil-jdk15on-1.70.jar
> >>>>>> URL=file:/WEB-INF/lib/castor-core-1.4.1.jar
> >>>>>> URL=file:/WEB-INF/lib/castor-xml-1.4.1.jar
> >>>>>> URL=file:/WEB-INF/lib/checker-qual-3.5.0.jar
> >>>>>> URL=file:/WEB-INF/lib/commons-beanutils-1.9.4.jar
> >>>>>> URL=file:/WEB-INF/lib/commons-codec-1.15.jar
> >>>>>> URL=file:/WEB-INF/lib/commons-collections-3.2.2.jar
> >>>>>> URL=file:/WEB-INF/lib/commons-collections4-4.4.jar
> >>>>>> URL=file:/WEB-INF/lib/commons-compress-1.21.jar
> >>>>>> URL=file:/WEB-INF/lib/commons-digester-2.1.jar
> >>>>>> URL=file:/WEB-INF/lib/commons-fileupload-1.4.jar
> >>>>>> URL=file:/WEB-INF/lib/commons-io-2.11.0.jar
> >>>>>> URL=file:/WEB-INF/lib/commons-lang3-3.12.0.jar
> >>>>>> URL=file:/WEB-INF/lib/commons-logging-1.2.jar
> >>>>>> URL=file:/WEB-INF/lib/commons-math3-3.6.1.jar
> >>>>>> URL=file:/WEB-INF/lib/commons-text-1.9.jar
> >>>>>> URL=file:/WEB-INF/lib/conscrypt-openjdk-uber-2.5.1.jar
> >>>>>> URL=file:/WEB-INF/lib/curvesapi-1.06.jar
> >>>>>> URL=file:/WEB-INF/lib/ecj-3.21.0.jar
> >>>>>> URL=file:/WEB-INF/lib/encoder-1.2.3.jar
> >>>>>> URL=file:/WEB-INF/lib/encoder-jsp-1.2.3.jar
> >>>>>> URL=file:/WEB-INF/lib/error_prone_annotations-2.9.0.jar
> >>>>>> URL=file:/WEB-INF/lib/failureaccess-1.0.1.jar
> >>>>>> URL=file:/WEB-INF/lib/fontbox-2.0.25.jar
> >>>>>> URL=file:/WEB-INF/lib/gax-2.3.0.jar
> >>>>>> URL=file:/WEB-INF/lib/gax-grpc-2.3.0.jar
> >>>>>> URL=file:/WEB-INF/lib/gmbal-api-only-4.0.3.jar
> >>>>>>
> URL=file:/WEB-INF/lib/google-auth-library-credentials-1.1.0.jar
> >>>>>>
> URL=file:/WEB-INF/lib/google-auth-library-oauth2-http-1.1.0.jar
> >>>>>> URL=file:/WEB-INF/lib/google-cloud-speech-1.30.5.jar
> >>>>>> URL=file:/WEB-INF/lib/google-http-client-1.39.2.jar
> >>>>>> URL=file:/WEB-INF/lib/google-http-client-gson-1.39.2.jar
> >>>>>> URL=file:/WEB-INF/lib/googleauth-1.5.0.jar
> >>>>>> URL=file:/WEB-INF/lib/grpc-alts-1.40.0.jar
> >>>>>> URL=file:/WEB-INF/lib/grpc-api-1.40.0.jar
> >>>>>> URL=file:/WEB-INF/lib/grpc-auth-1.40.0.jar
> >>>>>> URL=file:/WEB-INF/lib/grpc-context-1.40.0.jar
> >>>>>> URL=file:/WEB-INF/lib/grpc-core-1.40.0.jar
> >>>>>> URL=file:/WEB-INF/lib/grpc-grpclb-1.40.0.jar
> >>>>>> URL=file:/WEB-INF/lib/grpc-netty-shaded-1.40.0.jar
> >>>>>> URL=file:/WEB-INF/lib/grpc-protobuf-1.40.0.jar
> >>>>>> URL=file:/WEB-INF/lib/grpc-protobuf-lite-1.40.0.jar
> >>>>>> URL=file:/WEB-INF/lib/grpc-stub-1.40.0.jar
> >>>>>> URL=file:/WEB-INF/lib/gson-2.8.9.jar
> >>>>>> URL=file:/WEB-INF/lib/guava-31.0.1-jre.jar
> >>>>>> URL=file:/WEB-INF/lib/ha-api-3.1.13.jar
> >>>>>> URL=file:/WEB-INF/lib/handlebars-4.3.0.jar
> >>>>>> URL=file:/WEB-INF/lib/httpclient-4.5.13.jar
> >>>>>> URL=file:/WEB-INF/lib/httpcore-4.4.15.jar
> >>>>>> URL=file:/WEB-INF/lib/ion-java-1.0.2.jar
> >>>>>> URL=file:/WEB-INF/lib/ipaddress-5.3.3.jar
> >>>>>> URL=file:/WEB-INF/lib/itext-2.1.7.js9.jar
> >>>>>> URL=file:/WEB-INF/lib/j2objc-annotations-1.3.jar
> >>>>>> URL=file:/WEB-INF/lib/jackson-annotations-2.13.1.jar
> >>>>>> URL=file:/WEB-INF/lib/jackson-core-2.13.1.jar
> >>>>>> URL=file:/WEB-INF/lib/jackson-databind-2.13.1.jar
> >>>>>> URL=file:/WEB-INF/lib/jackson-dataformat-cbor-2.13.1.jar
> >>>>>> URL=file:/WEB-INF/lib/jai-imageio-core-1.4.0.jar
> >>>>>> URL=file:/WEB-INF/lib/jakarta.activation-2.0.1.jar
> >>>>>> URL=file:/WEB-INF/lib/jakarta.annotation-api-1.3.5.jar
> >>>>>> URL=file:/WEB-INF/lib/jakarta.jws-api-2.1.0.jar
> >>>>>> URL=file:/WEB-INF/lib/jakarta.mail-1.6.7.jar
> >>>>>> URL=file:/WEB-INF/lib/jakarta.xml.bind-api-2.3.3.jar
> >>>>>> URL=file:/WEB-INF/lib/jakarta.xml.soap-api-1.4.2.jar
> >>>>>> URL=file:/WEB-INF/lib/jakarta.xml.ws-api-2.3.3.jar
> >>>>>> URL=file:/WEB-INF/lib/jasperreports-6.18.1.jar
> >>>>>> URL=file:/WEB-INF/lib/jasperreports-fonts-6.18.1.jar
> >>>>>> URL=file:/WEB-INF/lib/java-saml-2.8.0.jar
> >>>>>> URL=file:/WEB-INF/lib/java-saml-core-2.8.0.jar
> >>>>>> URL=file:/WEB-INF/lib/javax.activation-api-1.2.0.jar
> >>>>>> URL=file:/WEB-INF/lib/javax.annotation-api-1.3.2.jar
> >>>>>> URL=file:/WEB-INF/lib/javax.inject-1.jar
> >>>>>> URL=file:/WEB-INF/lib/javax.mail-1.6.2.jar
> >>>>>> URL=file:/WEB-INF/lib/javax.xml.soap-api-1.4.0.jar
> >>>>>> URL=file:/WEB-INF/lib/jaxb-api-2.3.1.jar
> >>>>>> URL=file:/WEB-INF/lib/jaxb-core-2.3.0.1.jar
> >>>>>> URL=file:/WEB-INF/lib/jaxb-impl-2.3.5.jar
> >>>>>> URL=file:/WEB-INF/lib/jaxws-api-2.3.1.jar
> >>>>>> URL=file:/WEB-INF/lib/jaxws-rt-2.3.5.jar
> >>>>>> URL=file:/WEB-INF/lib/jbig2-imageio-3.0.3.jar
> >>>>>> URL=file:/WEB-INF/lib/jcommon-1.0.24.jar
> >>>>>> URL=file:/WEB-INF/lib/jempbox-1.8.16.jar
> >>>>>> URL=file:/WEB-INF/lib/jettison-1.4.1.jar
> >>>>>> URL=file:/WEB-INF/lib/jfreechart-1.0.19.jar
> >>>>>> URL=file:/WEB-INF/lib/jmespath-java-1.12.145.jar
> >>>>>> URL=file:/WEB-INF/lib/joda-time-2.10.6.jar
> >>>>>> URL=file:/WEB-INF/lib/jsch-0.1.55.jar
> >>>>>> URL=file:/WEB-INF/lib/jsoup-1.14.3.jar
> >>>>>> URL=file:/WEB-INF/lib/jsr181-api-1.0-MR1.jar
> >>>>>> URL=file:/WEB-INF/lib/jsr305-3.0.2.jar
> >>>>>>
> >>>>>>
> >>>
> URL=file:/WEB-INF/lib/listenablefuture-9999.0-empty-to-avoid-conflict-with-guava.jar
> >>>>>> URL=file:/WEB-INF/lib/log4j-api-2.17.1.jar
> >>>>>> URL=file:/WEB-INF/lib/log4j-core-2.17.1.jar
> >>>>>> URL=file:/WEB-INF/lib/log4j-slf4j-impl-2.17.1.jar
> >>>>>> URL=file:/WEB-INF/lib/log4j-web-2.17.1.jar
> >>>>>> URL=file:/WEB-INF/lib/management-api-3.2.3.jar
> >>>>>> URL=file:/WEB-INF/lib/metadata-extractor-2.16.0.jar
> >>>>>> URL=file:/WEB-INF/lib/mimepull-1.9.15.jar
> >>>>>> URL=file:/WEB-INF/lib/opencensus-api-0.28.0.jar
> >>>>>> URL=file:/WEB-INF/lib/opencensus-contrib-http-util-0.28.0.jar
> >>>>>> URL=file:/WEB-INF/lib/owasp-java-html-sanitizer-20211018.2.jar
> >>>>>> URL=file:/WEB-INF/lib/pdfbox-2.0.25.jar
> >>>>>> URL=file:/WEB-INF/lib/perfmark-api-0.23.0.jar
> >>>>>> URL=file:/WEB-INF/lib/poi-4.1.2.jar
> >>>>>> URL=file:/WEB-INF/lib/poi-ooxml-4.1.2.jar
> >>>>>> URL=file:/WEB-INF/lib/poi-ooxml-schemas-4.1.2.jar
> >>>>>> URL=file:/WEB-INF/lib/policy-2.7.10.jar
> >>>>>> URL=file:/WEB-INF/lib/postgresql-42.3.1.jar
> >>>>>> URL=file:/WEB-INF/lib/proto-google-cloud-speech-v1-1.30.5.jar
> >>>>>>
> >>> URL=file:/WEB-INF/lib/proto-google-cloud-speech-v1p1beta1-0.83.5.jar
> >>>>>> URL=file:/WEB-INF/lib/proto-google-common-protos-2.3.2.jar
> >>>>>> URL=file:/WEB-INF/lib/protobuf-java-3.17.3.jar
> >>>>>> URL=file:/WEB-INF/lib/protobuf-java-util-3.17.3.jar
> >>>>>> URL=file:/WEB-INF/lib/saaj-impl-1.5.3.jar
> >>>>>> URL=file:/WEB-INF/lib/scala-library-2.11.12.jar
> >>>>>> URL=file:/WEB-INF/lib/slf4j-api-1.7.33.jar
> >>>>>> URL=file:/WEB-INF/lib/stax-api-1.0.1.jar
> >>>>>> URL=file:/WEB-INF/lib/stax-ex-1.8.3.jar
> >>>>>> URL=file:/WEB-INF/lib/stax2-api-4.2.1.jar
> >>>>>> URL=file:/WEB-INF/lib/streambuffer-2.0.2.jar
> >>>>>> URL=file:/WEB-INF/lib/taglibs-standard-impl-1.2.5.jar
> >>>>>> URL=file:/WEB-INF/lib/taglibs-standard-spec-1.2.5.jar
> >>>>>> URL=file:/WEB-INF/lib/threetenbp-1.3.5.jar
> >>>>>> URL=file:/WEB-INF/lib/tika-core-2.2.1.jar
> >>>>>> URL=file:/WEB-INF/lib/tika-parser-image-module-2.2.1.jar
> >>>>>> URL=file:/WEB-INF/lib/tika-parser-xmp-commons-2.2.1.jar
> >>>>>> URL=file:/WEB-INF/lib/woodstox-core-6.2.6.jar
> >>>>>> URL=file:/WEB-INF/lib/xmlbeans-3.1.0.jar
> >>>>>> URL=file:/WEB-INF/lib/xmlsec-2.2.3.jar
> >>>>>> URL=file:/WEB-INF/lib/xmpcore-6.1.11.jar
> >>>>>> Class loader name=java.net.URLClassLoader, URL count=33
> >>>>>> URL=file:/lib/
> >>>>>> URL=file:/lib/annotations-api.jar
> >>>>>> URL=file:/lib/catalina-ant.jar
> >>>>>> URL=file:/lib/catalina-ha.jar
> >>>>>> URL=file:/lib/catalina-ssi.jar
> >>>>>> URL=file:/lib/catalina-storeconfig.jar
> >>>>>> URL=file:/lib/catalina-tribes.jar
> >>>>>> URL=file:/lib/catalina.jar
> >>>>>> URL=file:/lib/ecj-4.20.jar
> >>>>>> URL=file:/lib/el-api.jar
> >>>>>> URL=file:/lib/jasper-el.jar
> >>>>>> URL=file:/lib/jasper.jar
> >>>>>> URL=file:/lib/jaspic-api.jar
> >>>>>> URL=file:/lib/jsp-api.jar
> >>>>>> URL=file:/lib/servlet-api.jar
> >>>>>> URL=file:/lib/tomcat-api.jar
> >>>>>> URL=file:/lib/tomcat-coyote.jar
> >>>>>> URL=file:/lib/tomcat-dbcp.jar
> >>>>>> URL=file:/lib/tomcat-i18n-cs.jar
> >>>>>> URL=file:/lib/tomcat-i18n-de.jar
> >>>>>> URL=file:/lib/tomcat-i18n-es.jar
> >>>>>> URL=file:/lib/tomcat-i18n-fr.jar
> >>>>>> URL=file:/lib/tomcat-i18n-ja.jar
> >>>>>> URL=file:/lib/tomcat-i18n-ko.jar
> >>>>>> URL=file:/lib/tomcat-i18n-pt-BR.jar
> >>>>>> URL=file:/lib/tomcat-i18n-ru.jar
> >>>>>> URL=file:/lib/tomcat-i18n-zh-CN.jar
> >>>>>> URL=file:/lib/tomcat-jdbc.jar
> >>>>>> URL=file:/lib/tomcat-jni.jar
> >>>>>> URL=file:/lib/tomcat-util-scan.jar
> >>>>>> URL=file:/lib/tomcat-util.jar
> >>>>>> URL=file:/lib/tomcat-websocket.jar
> >>>>>> URL=file:/lib/websocket-api.jar
> >>>>>> Class loader
> name=jdk.internal.loader.ClassLoaders$AppClassLoader
> >>>>>> Class loader
> >>> name=jdk.internal.loader.ClassLoaders$PlatformClassLoader
> >>>>>>
> >>>>>>
> >>>>>> [3]
> >>>>>> private static void
> >>> logClassLoaderUrls(org.apache.logging.log4j.Level
> >>>>>> logLevel, ClassLoader cl) {
> >>>>>> if (null == cl) {
> >>>>>> return;
> >>>>>> }
> >>>>>> boolean loaderNameLogged = false;
> >>>>>> if (cl instanceof URLClassLoader) {
> >>>>>> final URLClassLoader urlCl = (URLClassLoader) cl;
> >>>>>> final URL[] urls = urlCl.getURLs();
> >>>>>> if (null != urls) {
> >>>>>> LOG.log(logLevel, " Class loader name={}, URL
> >>> count={}",
> >>>>>> cl.getClass().getName(), urls.length);
> >>>>>> loaderNameLogged = true;
> >>>>>> for (int idx = 0; idx < urls.length; idx++) {
> >>>>>> final URL url = urls[idx];
> >>>>>> LOG.log(logLevel, " URL[{}]={}", idx, url);
> >>>>>> }
> >>>>>> }
> >>>>>> }
> >>>>>> if (!loaderNameLogged) {
> >>>>>> LOG.log(logLevel, " Class loader name={}",
> >>>>>> cl.getClass().getName());
> >>>>>> }
> >>>>>> logClassLoaderUrls(logLevel, cl.getParent());
> >>>>>> }
> >>>>>>
> >>>>>> ..snip..
> >>>>>>
> >>>>>> LOG.info("java.class.path={}",
> >>>>>> System.getProperty("java.class.path"));
> >>>>>> LOG.debug("Class loader URLs:");
> >>>>>>
> logClassLoaderUrls(org.apache.logging.log4j.Level.DEBUG,
> >>>>>> ctx.getClassLoader());
> >>>>>>
> >>>>>>
> >>>>>> [4]
> >>>>>> cat log-extract.txt | cut -c137- | sed -e
> >>> "s+/usr/local/tomcat++g" -e
> >>>>>> "s+/usr/local/tomcat/webapps/exclaim++g" -e "s+\[[0-9]*\]++"
> >>>>>> and I manually sorted the "sections" using vim, with the "sort"
> >>> command.
> >>>>>>
> >>>>>>
> >>>>>> [5]
> >>>>>> # Debian with OpenJDK
> >>>>>> #FROM tomcat:9.0.58-jre11-openjdk AS tomcat_package
> >>>>>> # AWS Linux 2 with Corretto
> >>>>>> FROM tomcat:9.0.58-jdk11-corretto AS tomcat_package
> >>>>>>
> >>>>>> # Set the timezone to Canada/Eastern
> >>>>>> # For Debian
> >>>>>> #RUN ln -sf /usr/share/zoneinfo/Canada/Eastern /etc/localtime &&
> echo
> >>>>>> "Canada/Eastern" > /etc/timezone && dpkg-reconfigure -f
> noninteractive
> >>>>>> tzdata
> >>>>>> # For AWS Linux
> >>>>>> RUN ln -sf /usr/share/zoneinfo/Canada/Eastern /etc/localtime && sed
> -i
> >>>>>> 's/\"UTC\"/\"Canada\/Eastern\"/' /etc/sysconfig/clock # and reboot,
> >>> but
> >>>>>> will skip as container will be created and ran separately
> >>>>>>
> >>>>>> # Install a postgresql client for testing and the required font
> >>> packages
> >>>>>> needed by JasperReports
> >>>>>> # For Debian:
> >>>>>> #RUN echo "deb http://deb.debian.org/debian bullseye contrib
> >>> non-free"
> >>>>>>>> /etc/apt/sources.list && apt-get update && apt-get install -y
> >>>>>> postgresql-client ttf-mscorefonts-installer
> >>>>>> # For AWS Linux:
> >>>>>> RUN yum -q list installed epel-release.noarch &>/dev/null && echo
> >>>>>> 'epel-release already installed' || yum install -y
> >>>>>>
> >>> https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm
> >>>>>> RUN yum install -y cabextract xorg-x11-font-utils fontconfig
> >>>>>> RUN rpm -q msttcore-fonts-installer-2.6-1 &>/dev/null && echo
> >>>>>> 'msttcore-fonts-installer-2.6-1 already installed' || rpm -i
> >>>>>>
> >>>
> https://iweb.dl.sourceforge.net/project/mscorefonts2/rpms/msttcore-fonts-installer-2.6-1.noarch.rpm
> >>>>>>
> >>>>>> # Add a health-check script to wait for exclaim to be running --
> >>> allow at
> >>>>>> least 10 minutes for it to start
> >>>>>> COPY health_check.sh /usr/local/bin/
> >>>>>> HEALTHCHECK --interval=10s --timeout=5s --start-period=10m
> >>> --retries=10
> >>>>>> CMD /usr/local/bin/health_check.sh
> >>>>>>
> >>>>>> # Set up the resource folders needed by application and make it a
> >>>>>> # volume so it persists across container restarts
> >>>>>> COPY resources/ /resources/
> >>>>>> VOLUME /resources
> >>>>>>
> >>>>>> # Map some of the Tomcat folders to volumes so they persist between
> >>>>>> restarts of the container
> >>>>>> VOLUME /usr/local/tomcat/logs
> >>>>>> VOLUME /usr/local/tomcat/webapps
> >>>>>> VOLUME /usr/local/tomcat/work
> >>>>>> VOLUME /usr/local/tomcat/conf
> >>>>>>
> >>>>>> # Enable connecting from anywhere for JPDA (req'd for Java 9+)
> >>>>>> ENV JPDA_ADDRESS=*:8000
> >>>>>>
> >>>>>> CMD ["catalina.sh","jpda","run"]
> >>>>>>
> >>>>>>
> >>>>>> [6]
> >>>>>> JDK on MacOS is:
> >>>>>> openjdk version "11.0.13" 2021-10-19 LTS
> >>>>>> OpenJDK Runtime Environment Corretto-11.0.13.8.1 (build
> 11.0.13+8-LTS)
> >>>>>> OpenJDK 64-Bit Server VM Corretto-11.0.13.8.1 (build 11.0.13+8-LTS,
> >>> mixed
> >>>>>> mode)
> >>>>>>
> >>>>>> JDK on Windows is:
> >>>>>> openjdk 11.0.14 2022-01-18 LTS
> >>>>>> OpenJDK Runtime Environment Corretto-11.0.14.9.1 (build
> 11.0.14+9-LTS)
> >>>>>> OpenJDK 64-Bit Server VM Corretto-11.0.14.9.1 (build 11.0.14+9-LTS,
> >>> mixed
> >>>>>> mode)
> >>>>>>
> >>>>>> JDK on Docker with AWS Linux 2 is:
> >>>>>> openjdk 11.0.14 2022-01-18 LTS
> >>>>>> OpenJDK Runtime Environment Corretto-11.0.14.9.1 (build
> 11.0.14+9-LTS)
> >>>>>> OpenJDK 64-Bit Server VM Corretto-11.0.14.9.1 (build 11.0.14+9-LTS,
> >>> mixed
> >>>>>> mode)
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> On Mon, Feb 7, 2022 at 9:24 PM Rob Sargent <rsarg...@xmission.com>
> >>> wrote:
> >>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>> On 2/7/22 19:13, Robert Turner wrote:
> >>>>>>>> So, I've gone back and double-checked as much as I can (Tomcat
> >>>>>>> version, JDK
> >>>>>>>> version), and the classpath, and I have identical classpaths in
> both
> >>>>>>>> environments (except the sort order of the URLs per "level" of
> >>>>>>>> ClassLoader), and I've re-verified the behaviour:
> >>>>>>>> - fails in the docker environment
> >>>>>>>> - works locally
> >>>>>>>> (Same WAR file on both).
> >>>>>>>>
> >>>>>>>> I guess I'm in to one of the following approaches next:
> >>>>>>>> - build a debug version of Tomcat, and step through the code
> that
> >>>>>>> breaks
> >>>>>>>> and see if I can figure out why (not that I have time to do this
> of
> >>>>>>>> course...but might be necessary)
> >>>>>>>> - construct a trivial application reproduction, along with
> docker
> >>>>>>> layout,
> >>>>>>>> and see if anyone else can reproduce... (assuming anyone else has
> >>> time
> >>>>>>> to
> >>>>>>>> do that of course...)
> >>>>>>>>
> >>>>>>>> Anyone got any suggestions of what to look into next?
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> On Mon, Feb 7, 2022 at 5:05 PM Rob Sargent <rsarg...@xmission.com
> >
> >>>>>>> wrote:
> >>>>>>>>
> >>>>>>>>
> >>>>>>> As you appear averse to sharing names of things, seems you will
> need
> >>> to
> >>>>>>> do a lot of extra clean-up before you can share a docker image.
> Make
> >>>>>>> sure you're clear on what NoClassDefFoundError indicates (present
> at
> >>>>>>> compile time, absent at runtime) and how that translates to
> >>> docker-ness
> >>>>>>> (of which I know naught).
> >>>>>>>
> >>>>>>>
> >>>>>>>
> ---------------------------------------------------------------------
> >>>>>>> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> >>>>>>> For additional commands, e-mail: users-h...@tomcat.apache.org
> >>>>>>>
> >>>>>>>
> >>>>
> >>>
> >>> ---------------------------------------------------------------------
> >>> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> >>> For additional commands, e-mail: users-h...@tomcat.apache.org
> >>>
> >>>
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org
>
>
