Re: JVM crashing with caCertificatePath in server.xml

[Michael Osipov]

On 2024/05/22 00:05:18 Andy Arismendi wrote:
> Hi Micheal, you had asked to try these - 
> http://home.apache.org/~michaelo/issues/tomcat/openssl-crash/. I replaced my 
> files with these but Tomcat failed to start at this point with this message -
> 
> 22-May-2024 00:02:30.808 INFO [main] org.apache.coyote.AbstractProtocol.init 
> Initializing ProtocolHandler ["https-openssl-nio2-10.232.115.117-443"]
> OPENSSL_Uplink(7FFEEBF10C88,08): no OPENSSL_Applink

Copied and compiled applnk.c into the library, and updated at the given 
location. Please try again.

M

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: JVM crashing with caCertificatePath in server.xml

[Michael Osipov]

On 2024/05/22 00:05:18 Andy Arismendi wrote:
> Hi Micheal, you had asked to try these - 
> http://home.apache.org/~michaelo/issues/tomcat/openssl-crash/. I replaced my 
> files with these but Tomcat failed to start at this point with this message -
> 
> 22-May-2024 00:02:30.808 INFO [main] org.apache.coyote.AbstractProtocol.init 
> Initializing ProtocolHandler ["https-openssl-nio2-10.232.115.117-443"]
> OPENSSL_Uplink(7FFEEBF10C88,08): no OPENSSL_Applink

Darn, the dreaded applink.c. Though, I do not understand why it works for me, 
but not for you. I need to see how to add the object into tcnative's build. 
Give me some time.

M

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: JVM crashing with caCertificatePath in server.xml

[Michael Osipov]

On 2024/05/21 18:04:18 Christopher Schultz wrote:
> Michael,
> 
> On 5/21/24 03:32, Michael Osipov wrote:
> > On 2024/05/20 13:30:43 Christopher Schultz wrote:
> >> Michael,
> >>
> >> On 5/20/24 06:52, Michael Osipov wrote:
> >>> On 2024/05/17 15:11:58 Christopher Schultz wrote:
> >>>> Michael,
> >>>>
> >>>> On 5/17/24 03:42, Michael Osipov wrote:
> >>>>> On 2024/05/16 21:37:34 Christopher Schultz wrote:
> >>>>>> Michael,
> >>>>>>
> >>>>>> On 5/16/24 12:00, Michael Osipov wrote:
> >>>>>>> On 2024/05/16 15:55:04 Andy Arismendi wrote:
> >>>>>>>> Ok great! Thank you for taking the time and making the effort to 
> >>>>>>>> look into this Michael, much appreciated!
> >>>>>>>
> >>>>>>> Here is a dynamically linked, patched version until there is an 
> >>>>>>> official release: 
> >>>>>>> http://home.apache.org/~michaelo/issues/tomcat/openssl-crash/
> >>>>>>>
> >>>>>>> Please give it a try.
> >>>>>>
> >>>>>> Since you have produced a debug build of tcnative (and other
> >>>>>> components?) could you post the debug trace of the native stack?
> >>>>>
> >>>>> Unfortunately I can't. While I have the files with debug symbols I am 
> >>>>> limited by https://github.com/mturk/cmsc?tab=readme-ov-file#warning. I 
> >>>>> don't have a full blown Visual Studio installed.
> >>>>
> >>>> Okay. If you did build with VS, can you get a debug build with a 
> >>>> backtrace?
> >>>
> >>> Unfortunately not. Currently, I don't have the capacity to do so.
> >>>
> >>>> I guess you already tracked the crash to openssl_fopen. When I did a
> >>>> decompile of the official binary, I can see the code but it's very
> >>>> difficult to read:
> >>>>
> >>>> void FUN_1800cccd0(char *param_1,char *param_2)
> >>>>
> >>>> {
>  >>>>
> >>>> }
> >>>>
> >>>> Thanks for helping to at least link it to this openssl source:
> >>>>
> >>>> https://github.com/openssl/openssl/blob/45f5d51b72a262bf85c4461fbded91485ce6b9da/crypto/o_fopen.c#L38
> >>>>
> >>>> Since libtcnative.dll is statically-linked, it doesn't even need a
> >>>> symbol table for internal calls so the openssl_fopen token is completely
> >>>> lost. Also, libtcnative contains all of TCN, APR, and OpenSSL. TCN
> >>>> doesn't make direct Win32 calls so that leaves ... all of APR and
> >>>> OpenSSL to search for this pattern of calls.
> >>>>
> >>>> Since you know where the fault is occurring, do you know the native
> >>>> call-trace being performed? I'd love to know which component along the
> >>>> way is not properly checking for NULL.
> >>>
> >>> Yes, sure:
> >>> * 
> >>> https://github.com/apache/tomcat-native/blob/6a6a6b2395036c6a6cabb2b8af22aa329e438436/native/src/sslcontext.c#L711
> >>
> >> So, 'file' is null on this line? If so, I guess the bug is in tcnative.
> > 
> > Yes, it is NULL. I don't think that the bug in necessarily in tcnative 
> > because othe functions in OpenSSL do support NULL and fail appropriately. I 
> > consider this either bad documenation or missing input validation.
> > 
> >>> * 
> >>> https://github.com/openssl/openssl/blob/45f5d51b72a262bf85c4461fbded91485ce6b9da/ssl/ssl_cert.c#L834
> >>> * 
> >>> https://github.com/openssl/openssl/blob/45f5d51b72a262bf85c4461fbded91485ce6b9da/crypto/o_fopen.c#L42
> >>
> >> Where does the call go from BIO_read_filename to openssl_fopen?
> > 
> > Here: 
> > https://github.com/openssl/openssl/blob/45f5d51b72a262bf85c4461fbded91485ce6b9da/crypto/bio/bss_file.c#L267-L295
> 
> That's in file_ctrl(). What's the full call trace from tcnative's 
> setCACertificate through to openssl_fopen?

BIO_read_filename() is a macro, if you trace down the resolution manually you 
will end up in the resources I have provided you. Took me some time.

M

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: JVM crashing with caCertificatePath in server.xml

[Michael Osipov]

On 2024/05/20 13:30:43 Christopher Schultz wrote:
> Michael,
> 
> On 5/20/24 06:52, Michael Osipov wrote:
> > On 2024/05/17 15:11:58 Christopher Schultz wrote:
> >> Michael,
> >>
> >> On 5/17/24 03:42, Michael Osipov wrote:
> >>> On 2024/05/16 21:37:34 Christopher Schultz wrote:
> >>>> Michael,
> >>>>
> >>>> On 5/16/24 12:00, Michael Osipov wrote:
> >>>>> On 2024/05/16 15:55:04 Andy Arismendi wrote:
> >>>>>> Ok great! Thank you for taking the time and making the effort to look 
> >>>>>> into this Michael, much appreciated!
> >>>>>
> >>>>> Here is a dynamically linked, patched version until there is an 
> >>>>> official release: 
> >>>>> http://home.apache.org/~michaelo/issues/tomcat/openssl-crash/
> >>>>>
> >>>>> Please give it a try.
> >>>>
> >>>> Since you have produced a debug build of tcnative (and other
> >>>> components?) could you post the debug trace of the native stack?
> >>>
> >>> Unfortunately I can't. While I have the files with debug symbols I am 
> >>> limited by https://github.com/mturk/cmsc?tab=readme-ov-file#warning. I 
> >>> don't have a full blown Visual Studio installed.
> >>
> >> Okay. If you did build with VS, can you get a debug build with a backtrace?
> > 
> > Unfortunately not. Currently, I don't have the capacity to do so.
> > 
> >> I guess you already tracked the crash to openssl_fopen. When I did a
> >> decompile of the official binary, I can see the code but it's very
> >> difficult to read:
> >>
> >> void FUN_1800cccd0(char *param_1,char *param_2)
> >>
> >> {
> >> char cVar1;
> >> longlong lVar2;
> >> int iVar3;
> >> DWORD DVar4;
> >> char *pcVar5;
> >> FILE *pFVar6;
> >> int *piVar7;
> >> ulonglong uVar8;
> >> uint uVar9;
> >> int cbMultiByte;
> >> undefined *puVar10;
> >> undefined *puVar11;
> >> uint uVar12;
> >> undefined8 uStackY_80;
> >> undefined auStackY_78 [32];
> >> wchar_t local_48 [8];
> >> ulonglong local_38;
> >> undefined8 uStack_30;
> >>
> >> uStack_30 = 0x1800ccce3;
> >> local_38 = DAT_18033f868 ^ (ulonglong)local_48;
> >> cVar1 = *param_1;
> >> uVar12 = 0;
> >> pcVar5 = param_1;
> >> for (uVar9 = uVar12; (cVar1 != '\0' && (uVar9 < 0x8000)); uVar9 =
> >> uVar9 + 1) {
> >>   pcVar5 = pcVar5 + 1;
> >>   cVar1 = *pcVar5;
> >> }
> >> cbMultiByte = (uVar9 & 0x7fff) + 1;
> >> uStackY_80 = 0x1800ccd50;
> >> iVar3 = 
> >> MultiByteToWideChar(0xfde9,8,param_1,cbMultiByte,(LPWSTR)0x0,0);
> >> DVar4 = 8;
> >> if (iVar3 < 1) {
> >>   uStackY_80 = 0x1800ccd5d;
> >>   DVar4 = GetLastError();
> >>   if (DVar4 == 0x3ec) {
> >> uStackY_80 = 0x1800ccd84;
> >> iVar3 =
> >> MultiByteToWideChar(0xfde9,0,param_1,cbMultiByte,(LPWSTR)0x0,0);
> >> DVar4 = 0;
> >> if (0 < iVar3) goto LAB_1800ccdac;
> >>   }
> >>   uStackY_80 = 0x1800ccd91;
> >>   DVar4 = GetLastError();
> >>   puVar10 = auStackY_78;
> >>   puVar11 = auStackY_78;
> >>   if (DVar4 != 0x459) goto LAB_1800cce89;
> >> }
> >> else {
> >> LAB_1800ccdac:
> >>   uVar8 = (longlong)iVar3 * 2 + 0xf;
> >>   if (uVar8 <= (ulonglong)((longlong)iVar3 * 2)) {
> >> uVar8 = 0xff0;
> >>   }
> >>   uStackY_80 = 0x1800ccdd1;
> >>   lVar2 = -(uVar8 & 0xfff0);
> >>   *(int *)( + lVar2) = iVar3;
> >>   *(wchar_t **)( + lVar2) = (wchar_t
> >> *)((longlong)local_48 + lVar2);
> >>   *(undefined8 *)(auStackY_78 + lVar2 + -8) = 0x1800ccdf7;
> >>   iVar3 = MultiByteToWideChar(0xfde9,DVar4,param_1,cbMultiByte,
> >>   *(LPWSTR *)( +
> >> lVar2),
> >>   *(int *)( +
> >> lVar2));
> >>   puVar11 = auStackY_78 + lVar2;
> >>   if (iVar3 == 0) goto LAB_1800cce89;
> >>   cVar1 = *param_2;
> >>   

Re: JVM crashing with caCertificatePath in server.xml

[Michael Osipov]

On 2024/05/17 15:11:58 Christopher Schultz wrote:
> Michael,
> 
> On 5/17/24 03:42, Michael Osipov wrote:
> > On 2024/05/16 21:37:34 Christopher Schultz wrote:
> >> Michael,
> >>
> >> On 5/16/24 12:00, Michael Osipov wrote:
> >>> On 2024/05/16 15:55:04 Andy Arismendi wrote:
> >>>> Ok great! Thank you for taking the time and making the effort to look 
> >>>> into this Michael, much appreciated!
> >>>
> >>> Here is a dynamically linked, patched version until there is an official 
> >>> release: http://home.apache.org/~michaelo/issues/tomcat/openssl-crash/
> >>>
> >>> Please give it a try.
> >>
> >> Since you have produced a debug build of tcnative (and other
> >> components?) could you post the debug trace of the native stack?
> > 
> > Unfortunately I can't. While I have the files with debug symbols I am 
> > limited by https://github.com/mturk/cmsc?tab=readme-ov-file#warning. I 
> > don't have a full blown Visual Studio installed.
> 
> Okay. If you did build with VS, can you get a debug build with a backtrace?

Unfortunately not. Currently, I don't have the capacity to do so.

> I guess you already tracked the crash to openssl_fopen. When I did a 
> decompile of the official binary, I can see the code but it's very 
> difficult to read:
> 
> void FUN_1800cccd0(char *param_1,char *param_2)
> 
> {
>char cVar1;
>longlong lVar2;
>int iVar3;
>DWORD DVar4;
>char *pcVar5;
>FILE *pFVar6;
>int *piVar7;
>ulonglong uVar8;
>uint uVar9;
>int cbMultiByte;
>undefined *puVar10;
>undefined *puVar11;
>uint uVar12;
>undefined8 uStackY_80;
>undefined auStackY_78 [32];
>wchar_t local_48 [8];
>ulonglong local_38;
>undefined8 uStack_30;
> 
>uStack_30 = 0x1800ccce3;
>local_38 = DAT_18033f868 ^ (ulonglong)local_48;
>cVar1 = *param_1;
>uVar12 = 0;
>pcVar5 = param_1;
>for (uVar9 = uVar12; (cVar1 != '\0' && (uVar9 < 0x8000)); uVar9 = 
> uVar9 + 1) {
>  pcVar5 = pcVar5 + 1;
>  cVar1 = *pcVar5;
>}
>cbMultiByte = (uVar9 & 0x7fff) + 1;
>uStackY_80 = 0x1800ccd50;
>iVar3 = MultiByteToWideChar(0xfde9,8,param_1,cbMultiByte,(LPWSTR)0x0,0);
>DVar4 = 8;
>if (iVar3 < 1) {
>  uStackY_80 = 0x1800ccd5d;
>  DVar4 = GetLastError();
>  if (DVar4 == 0x3ec) {
>uStackY_80 = 0x1800ccd84;
>iVar3 = 
> MultiByteToWideChar(0xfde9,0,param_1,cbMultiByte,(LPWSTR)0x0,0);
>DVar4 = 0;
>if (0 < iVar3) goto LAB_1800ccdac;
>  }
>  uStackY_80 = 0x1800ccd91;
>  DVar4 = GetLastError();
>  puVar10 = auStackY_78;
>  puVar11 = auStackY_78;
>  if (DVar4 != 0x459) goto LAB_1800cce89;
>}
>else {
> LAB_1800ccdac:
>  uVar8 = (longlong)iVar3 * 2 + 0xf;
>  if (uVar8 <= (ulonglong)((longlong)iVar3 * 2)) {
>uVar8 = 0xff0;
>  }
>  uStackY_80 = 0x1800ccdd1;
>  lVar2 = -(uVar8 & 0xfff0);
>  *(int *)( + lVar2) = iVar3;
>  *(wchar_t **)( + lVar2) = (wchar_t 
> *)((longlong)local_48 + lVar2);
>  *(undefined8 *)(auStackY_78 + lVar2 + -8) = 0x1800ccdf7;
>  iVar3 = MultiByteToWideChar(0xfde9,DVar4,param_1,cbMultiByte,
>  *(LPWSTR *)( + 
> lVar2),
>  *(int *)( + 
> lVar2));
>  puVar11 = auStackY_78 + lVar2;
>  if (iVar3 == 0) goto LAB_1800cce89;
>  cVar1 = *param_2;
>  pcVar5 = param_2;
>  for (; (cVar1 != '\0' && (uVar12 < 0x8000)); uVar12 = uVar12 + 1) {
>pcVar5 = pcVar5 + 1;
>cVar1 = *pcVar5;
>  }
>  *(undefined4 *)( + lVar2) = 8;
>  *(wchar_t **)( + lVar2) = local_48;
>  *(undefined8 *)(auStackY_78 + lVar2 + -8) = 0x1800cce4d;
>  iVar3 = MultiByteToWideChar(0xfde9,0,param_2,(uVar12 & 0x7fff) + 1,
>  *(LPWSTR *)( + 
> lVar2),
>  *(int *)( + 
> lVar2));
>  puVar11 = auStackY_78 + lVar2;
>  if (iVar3 == 0) goto LAB_1800cce89;
>  *(undefined8 *)(auStackY_78 + lVar2 + -8) = 0x1800cce5d;
>  pFVar6 = _wfopen((wchar_t *)((longlong)local_48 + lVar2),local_48);
>  puVar11 = auStackY_78 + lVar2;
>  if (pFVar6 != (FILE *)0x0) goto LAB_1800cce89;
>  *(undefined8 *)(auStackY_78 + lVar2 + -8) = 0x1800cce6a;
>  piVar7 = _errno();
>  puVar10 = auStackY_78 + lVar2;
>  if (*piVar7 != 2) {
>*(undefined8 *)(auStackY_78 + lVar2 + -8) = 0x1800cce78

Re: JVM crashing with caCertificatePath in server.xml

[Michael Osipov]

On 2024/05/16 21:37:34 Christopher Schultz wrote:
> Michael,
> 
> On 5/16/24 12:00, Michael Osipov wrote:
> > On 2024/05/16 15:55:04 Andy Arismendi wrote:
> >> Ok great! Thank you for taking the time and making the effort to look into 
> >> this Michael, much appreciated!
> > 
> > Here is a dynamically linked, patched version until there is an official 
> > release: http://home.apache.org/~michaelo/issues/tomcat/openssl-crash/
> > 
> > Please give it a try.
> 
> Since you have produced a debug build of tcnative (and other 
> components?) could you post the debug trace of the native stack?

Unfortunately I can't. While I have the files with debug symbols I am limited 
by https://github.com/mturk/cmsc?tab=readme-ov-file#warning. I don't have a 
full blown Visual Studio installed.

M

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: JVM crashing with caCertificatePath in server.xml

[Michael Osipov]

On 2024/05/16 15:55:04 Andy Arismendi wrote:
> Ok great! Thank you for taking the time and making the effort to look into 
> this Michael, much appreciated!

Here is a dynamically linked, patched version until there is an official 
release: http://home.apache.org/~michaelo/issues/tomcat/openssl-crash/

Please give it a try.

Michael

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: JVM crashing with caCertificatePath in server.xml

[Michael Osipov]

On 2024/05/15 20:35:08 Michael Osipov wrote:
> On 2024/05/15 14:41:43 Michael Osipov wrote:
> > Good news. I can reproduce on Windows:
> > 15-May-2024 16:40:31.092 INFORMATION [main] 
> > org.apache.coyote.AbstractProtocol.init Initialisiere 
> > ProtocolHandler["https-openssl-apr-18444"]
> > 15-May-2024 16:40:31.144 WARNUNG [main] 
> > org.apache.tomcat.util.net.SSLUtilBase.getEnabled Tomcat interprets the 
> > [ciphers] attribute in a manner consistent with the latest OpenSSL 
> > development branch. Some of the specified [ciphers] are not supported by 
> > the configured SSL engine for this connector (which may use JSSE or an 
> > older OpenSSL version) and have been skipped: 
> > [[TLS_DH_DSS_WITH_AES_256_GCM_SHA384, TLS_DH_RSA_WITH_AES_256_GCM_SHA384, 
> > TLS_ECDH_RSA_WITH_AES_256_GCM_SHA384, 
> > TLS_ECDH_ECDSA_WITH_AES_256_GCM_SHA384, TLS_AES_128_CCM_SHA256, 
> > TLS_DH_DSS_WITH_AES_128_GCM_SHA256, TLS_DH_RSA_WITH_AES_128_GCM_SHA256, 
> > TLS_ECDH_RSA_WITH_AES_128_GCM_SHA256, 
> > TLS_ECDH_ECDSA_WITH_AES_128_GCM_SHA256]]
> > #
> > # A fatal error has been detected by the Java Runtime Environment:
> > #
> > #  EXCEPTION_ACCESS_VIOLATION (0xc005) at pc=0x024928d5cd10, 
> > pid=33136, tid=0x55b8
> > #
> > # JRE version: OpenJDK Runtime Environment (Zulu 8.68.0.21-CA-win64) 
> > (8.0_362-b09) (build 1.8.0_362-b09)
> > # Java VM: OpenJDK 64-Bit Server VM (25.362-b09 mixed mode windows-amd64 
> > compressed oops)
> > # Problematic frame:
> > # C  [tcnative-1.dll+0xccd10]
> > #
> > # Failed to write core dump. Minidumps are not enabled by default on client 
> > versions of Windows
> > #
> > # An error report file with more information is saved as:
> > # C:\Temp\apache-tomcat-9.0.89\hs_err_pid33136.log
> > #
> > # If you would like to submit a bug report, please visit:
> > #   http://www.azul.com/support/
> > # The crash happened outside the Java Virtual Machine in native code.
> > # See problematic frame for where to report the bug.
> > #
> > 
> > I will do a custom build of Tomcat Native and see where it crashes. Stay 
> > tuned.
> 
> Found the bug: It is either a flaw or uncertainty in OpenSSL. Details follow 
> tomorrow.

Details:

Reported the issue upstream: https://github.com/openssl/openssl/issues/24416
I will push a temporary fix until upstream does properly handle NULL input.

Partially OT: After testing here in and out I am convinced that the code after 
SSL_CTX_load_verify_locations() does absolutely not that what the author 
intended to do. The code block messes up CA certification for client 
verification with the request DNs for client cert auth. I will report a 
separate issue because it is unrelated.

Michael

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: JVM crashing with caCertificatePath in server.xml

[Michael Osipov]

On 2024/05/15 14:41:43 Michael Osipov wrote:
> Good news. I can reproduce on Windows:
> 15-May-2024 16:40:31.092 INFORMATION [main] 
> org.apache.coyote.AbstractProtocol.init Initialisiere 
> ProtocolHandler["https-openssl-apr-18444"]
> 15-May-2024 16:40:31.144 WARNUNG [main] 
> org.apache.tomcat.util.net.SSLUtilBase.getEnabled Tomcat interprets the 
> [ciphers] attribute in a manner consistent with the latest OpenSSL 
> development branch. Some of the specified [ciphers] are not supported by the 
> configured SSL engine for this connector (which may use JSSE or an older 
> OpenSSL version) and have been skipped: [[TLS_DH_DSS_WITH_AES_256_GCM_SHA384, 
> TLS_DH_RSA_WITH_AES_256_GCM_SHA384, TLS_ECDH_RSA_WITH_AES_256_GCM_SHA384, 
> TLS_ECDH_ECDSA_WITH_AES_256_GCM_SHA384, TLS_AES_128_CCM_SHA256, 
> TLS_DH_DSS_WITH_AES_128_GCM_SHA256, TLS_DH_RSA_WITH_AES_128_GCM_SHA256, 
> TLS_ECDH_RSA_WITH_AES_128_GCM_SHA256, TLS_ECDH_ECDSA_WITH_AES_128_GCM_SHA256]]
> #
> # A fatal error has been detected by the Java Runtime Environment:
> #
> #  EXCEPTION_ACCESS_VIOLATION (0xc005) at pc=0x024928d5cd10, 
> pid=33136, tid=0x55b8
> #
> # JRE version: OpenJDK Runtime Environment (Zulu 8.68.0.21-CA-win64) 
> (8.0_362-b09) (build 1.8.0_362-b09)
> # Java VM: OpenJDK 64-Bit Server VM (25.362-b09 mixed mode windows-amd64 
> compressed oops)
> # Problematic frame:
> # C  [tcnative-1.dll+0xccd10]
> #
> # Failed to write core dump. Minidumps are not enabled by default on client 
> versions of Windows
> #
> # An error report file with more information is saved as:
> # C:\Temp\apache-tomcat-9.0.89\hs_err_pid33136.log
> #
> # If you would like to submit a bug report, please visit:
> #   http://www.azul.com/support/
> # The crash happened outside the Java Virtual Machine in native code.
> # See problematic frame for where to report the bug.
> #
> 
> I will do a custom build of Tomcat Native and see where it crashes. Stay 
> tuned.

Found the bug: It is either a flaw or uncertainty in OpenSSL. Details follow 
tomorrow.

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: JVM crashing with caCertificatePath in server.xml

[Michael Osipov]

Good news. I can reproduce on Windows:
15-May-2024 16:40:31.092 INFORMATION [main] 
org.apache.coyote.AbstractProtocol.init Initialisiere 
ProtocolHandler["https-openssl-apr-18444"]
15-May-2024 16:40:31.144 WARNUNG [main] 
org.apache.tomcat.util.net.SSLUtilBase.getEnabled Tomcat interprets the 
[ciphers] attribute in a manner consistent with the latest OpenSSL development 
branch. Some of the specified [ciphers] are not supported by the configured SSL 
engine for this connector (which may use JSSE or an older OpenSSL version) and 
have been skipped: [[TLS_DH_DSS_WITH_AES_256_GCM_SHA384, 
TLS_DH_RSA_WITH_AES_256_GCM_SHA384, TLS_ECDH_RSA_WITH_AES_256_GCM_SHA384, 
TLS_ECDH_ECDSA_WITH_AES_256_GCM_SHA384, TLS_AES_128_CCM_SHA256, 
TLS_DH_DSS_WITH_AES_128_GCM_SHA256, TLS_DH_RSA_WITH_AES_128_GCM_SHA256, 
TLS_ECDH_RSA_WITH_AES_128_GCM_SHA256, TLS_ECDH_ECDSA_WITH_AES_128_GCM_SHA256]]
#
# A fatal error has been detected by the Java Runtime Environment:
#
#  EXCEPTION_ACCESS_VIOLATION (0xc005) at pc=0x024928d5cd10, pid=33136, 
tid=0x55b8
#
# JRE version: OpenJDK Runtime Environment (Zulu 8.68.0.21-CA-win64) 
(8.0_362-b09) (build 1.8.0_362-b09)
# Java VM: OpenJDK 64-Bit Server VM (25.362-b09 mixed mode windows-amd64 
compressed oops)
# Problematic frame:
# C  [tcnative-1.dll+0xccd10]
#
# Failed to write core dump. Minidumps are not enabled by default on client 
versions of Windows
#
# An error report file with more information is saved as:
# C:\Temp\apache-tomcat-9.0.89\hs_err_pid33136.log
#
# If you would like to submit a bug report, please visit:
#   http://www.azul.com/support/
# The crash happened outside the Java Virtual Machine in native code.
# See problematic frame for where to report the bug.
#

I will do a custom build of Tomcat Native and see where it crashes. Stay tuned.

M

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: JVM crashing with caCertificatePath in server.xml

[Michael Osipov]

On 2024/05/15 01:51:41 Andy Arismendi wrote:
> ADDITIONAL ENVIRONMENT INFO UPDATE:
> 
> libtcnative: org.apache.catalina.core.AprLifecycleListener.lifecycleEvent 
> Loaded Apache Tomcat Native library [1.3.0] using APR version [1.7.4].
> 
> CRASH LOG
> 
> See enclosed: hs_err_pid4464.log
> 
> c_rehash.pl
> 
> I didn’t have perl, tried strawberry perl, it didn’t seem to create symlinks 
> on Windows so I do it with a powershell using "openssl x509 -subject_hash 
> -fingerprint -noout -in " making symlinks in the same directory for 
> each CA cert PEM e.g. a655d288.0 (link) -> cert.pem (file). This didn’t seem 
> to make a difference though, JVM still crashed.

To make sure I have just tried:
> 8981 2024-05-15T10:26:58.717 INFORMATION [main] 
> org.apache.catalina.startup.VersionLoggerListener.log Server version name:   
> Apache Tomcat/9.0.89
> 8982 2024-05-15T10:26:58.722 INFORMATION [main] 
> org.apache.catalina.startup.VersionLoggerListener.log Server built:  
> May 3 2024 20:22:11 UTC
> 8983 2024-05-15T10:26:58.722 INFORMATION [main] 
> org.apache.catalina.startup.VersionLoggerListener.log Server version number: 
> 9.0.89.0
> 8984 2024-05-15T10:26:58.722 INFORMATION [main] 
> org.apache.catalina.startup.VersionLoggerListener.log OS Name:   
> HP-UX
> 8985 2024-05-15T10:26:58.723 INFORMATION [main] 
> org.apache.catalina.startup.VersionLoggerListener.log OS Version:
> B.11.31
> 8986 2024-05-15T10:26:58.723 INFORMATION [main] 
> org.apache.catalina.startup.VersionLoggerListener.log Architecture:  
> IA64N
> 8987 2024-05-15T10:26:58.723 INFORMATION [main] 
> org.apache.catalina.startup.VersionLoggerListener.log Java Home: 
> /opt/java8/jre
> 8988 2024-05-15T10:26:58.723 INFORMATION [main] 
> org.apache.catalina.startup.VersionLoggerListener.log JVM Version:   
> 1.8.0.27-hp-ux-b1
> 8989 2024-05-15T10:26:58.724 INFORMATION [main] 
> org.apache.catalina.startup.VersionLoggerListener.log JVM Vendor:
> Hewlett Packard Enterprise Company
> 8990 2024-05-15T10:26:58.724 INFORMATION [main] 
> org.apache.catalina.startup.VersionLoggerListener.log CATALINA_BASE: 
> /var/opt/tomcat-services
> 8991 2024-05-15T10:26:58.724 INFORMATION [main] 
> org.apache.catalina.startup.VersionLoggerListener.log CATALINA_HOME: 
> /opt/ports/apache-tomcat-9.0.89
> 9015 2024-05-15T10:26:58.733 INFORMATION [main] 
> org.apache.catalina.core.AprLifecycleListener.lifecycleEvent Loaded Apache 
> Tomcat Native library [1.3.0] using APR version [1.7.4].
> 9016 2024-05-15T10:26:58.733 INFORMATION [main] 
> org.apache.catalina.core.AprLifecycleListener.lifecycleEvent APR 
> capabilities: IPv6 [true], sendfile [true], accept filters [false], random 
> [true], UDS [true].
> 9017 2024-05-15T10:26:58.733 INFORMATION [main] 
> org.apache.catalina.core.AprLifecycleListener.lifecycleEvent APR/OpenSSL 
> configuration: useAprConnector [true], useOpenSSL [true]
> 9018 2024-05-15T10:26:58.816 INFORMATION [main] 
> org.apache.catalina.core.AprLifecycleListener.initializeSSL OpenSSL 
> successfully initialized [OpenSSL 3.0.13 30 Jan 2024]

With my smartcard it just works:
>  maxParameterCount="1000"
> maxHttpHeaderSize="24576" maxThreads="250"
> SSLEnabled="true" scheme="https" secure="true"
> defaultSSLHostConfigName="...">
>  honorCipherOrder="true" disableSessionTickets="true"
> certificateVerification="optional" certificateVerificationDepth="5"
> 
> ciphers="HIGH:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!3DES:!MD5:!PSK:!DSS:!SHA1:!SHA256:!SHA384"
> caCertificatePath="/opt/openssl/certs">
>  certificateKeyFile="/opt/openssl/.../key.crt"
> certificateKeyPasswordFile="/opt/openssl/.../password" type="RSA" 
> />
> 
>  value="/opt/openssl/siemens-medium+strong-clientcert-cacerts.crt" />
> 
> 
> 
> 

M

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: JVM crashing with caCertificatePath in server.xml

[Michael Osipov]

On 2024/05/14 20:27:02 Christopher Schultz wrote:
> 
> 
> On 5/14/24 15:23, Andy Arismendi wrote:
> > Sure thing -
> > 
> > ADDITIONAL ENVIRONMENT INFO:
> > 
> > libtcnative: tcnative-1.dll is included in the Tomcat 9.0.89 64-bit Windows 
> > zip download, not sure about the version...
> > OpenSSL version: 3.0.13 30 Jan 2024 (Library: OpenSSL 3.0.13 30 Jan 2024) 
> > (with FIPS 140-2)
> > 
> > Regarding expecting a directory of certificate hash files, I wasn’t
> > aware of this, assumed it would pick up CA cert PEM files in a
> > directory.
> 
> The Tomcat documentation does say this just needs to be a directory full 
> of PEM files. I can trace through the code to see if it's more like what 
> Michael-O posted. Honestly, the whole idea of having to run c_rehash is 
> a stupid hack for stupid programs. You should never have to do that. :/

If the docs say so, then we need to fix the docs because all path input in 
OpenSSL expects simplified subject hashes. Anything else will not work/will be 
ignored. Use strace/truss/etc. and you will see what OpenSSL will try to read. 
"openssl s_server" will do the trick here for you.

Michael

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: JVM crashing with caCertificatePath in server.xml

[Michael Osipov]

On 2024/05/15 01:51:41 Andy Arismendi wrote:
> ADDITIONAL ENVIRONMENT INFO UPDATE:
> 
> libtcnative: org.apache.catalina.core.AprLifecycleListener.lifecycleEvent 
> Loaded Apache Tomcat Native library [1.3.0] using APR version [1.7.4].
> 
> CRASH LOG
> 
> See enclosed: hs_err_pid4464.log

Attachments are stripped. You not to upload it somewhere or send via email.

Michael

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: JVM crashing with caCertificatePath in server.xml

[Michael Osipov]

On 2024/05/14 19:23:47 Andy Arismendi wrote:
> Sure thing - 
> 
> ADDITIONAL ENVIRONMENT INFO:
> 
> libtcnative: tcnative-1.dll is included in the Tomcat 9.0.89 64-bit Windows 
> zip download, not sure about the version...
> OpenSSL version: 3.0.13 30 Jan 2024 (Library: OpenSSL 3.0.13 30 Jan 2024) 
> (with FIPS 140-2)

Have a look at catalina.out, it should be 1.3.0, I guess.

> Regarding expecting a directory of certificate hash files, I wasn’t aware of 
> this, assumed it would pick up CA cert PEM files in a directory. I would 
> however not expect this or an empty directory to crash the JVM however…

Nope, it won't. See SSL_CTX_load_verify_locations at 
https://www.openssl.org/docs/manmaster/man3/SSL_CTX_load_verify_locations.html:
If CApath is not NULL, it points to a directory containing CA certificates in 
PEM format. The files each contain one CA certificate. The files are looked up 
by the CA subject name hash value, which must hence be available. If more than 
one CA certificate with the same name hash value exist, the extension must be 
different (e.g. 9d66eef0.0, 9d66eef0.1 etc). The search is performed in the 
ordering of the extension number, regardless of other properties of the 
certificates. Use the c_rehash utility to create the necessary links.

Please don't forget the log file. The issue is somewhere here: 
https://github.com/apache/tomcat-native/blob/43ddd1e8059528454110198ca0d7d191322beeaf/native/src/sslcontext.c#L673-L738

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: JVM crashing with caCertificatePath in server.xml

[Michael Osipov]

On 2024/05/14 18:21:36 Andy Arismendi wrote:
> Hi, just ran into this today. The JVM is crashing when caCertificatePath is 
> added to server.xml. I tried the latest Zulu JRE 8 and 11 but still had the 
> crash.
> 
> 
> ENVIRONMENT
> 
> Tomcat: 9.0.89 (64-bit Windows zip)
> OS: Windows Server 2019
> JVM:
> openjdk version "1.8.0_322"
> OpenJDK Runtime Environment (Zulu 8.60.0.21-CA-win64) (build 1.8.0_322-b06)
> OpenJDK 64-Bit Server VM (Zulu 8.60.0.21-CA-win64) (build 25.322-b06, mixed 
> mode)
> 
> 
> CRASH INFO
> 
> When caCertificatePath is present in server.xml and points to a valid 
> directory (empty or with PEM files) the JVM crashes during Tomcat startup. 
> This is the JVM console output:
> 
> 14-May-2024 17:34:58.443 INFO [main] org.apache.coyote.AbstractProtocol.init 
> Initializing ProtocolHandler ["https-openssl-nio2-1.2.3.4-443"]
> #
> # A fatal error has been detected by the Java Runtime Environment:
> #
> #  EXCEPTION_ACCESS_VIOLATION (0xc005) at pc=0x0001800ccd10, 
> pid=1244, tid=0x0ab0
> #
> # JRE version: OpenJDK Runtime Environment (Zulu 8.60.0.21-CA-win64) 
> (8.0_322-b06) (build 1.8.0_322-b06)
> # Java VM: OpenJDK 64-Bit Server VM (25.322-b06 mixed mode windows-amd64 
> compressed oops)
> # Problematic frame:
> # C  [tcnative-1.dll+0xccd10]
> #
> # Core dump written. Default location: D:\Program 
> Files\apache-tomcat\bin\hs_err_pid1244.mdmp
> #
> # An error report file with more information is saved as:
> # D:\Program Files\apache-tomcat\bin\hs_err_pid1244.log
> #
> # If you would like to submit a bug report, please visit:
> #   http://www.azul.com/support/
> # The crash happened outside the Java Virtual Machine in native code.
> # See problematic frame for where to report the bug.
> #
> 
> 
> CONFIG INFO
> 
> Here’s the server.xml that causes the JVM crash. 
> 
>  maxThreads="1000" port="443" scheme="https" secure="true" SSLEnabled="true" 
> allowTrace="false" xpoweredBy="false" address="1.2.3.4" acceptCount="1" 
> socket.rxBufSize="131072" socket.txBufSize="131072" minSpareThreads="100" 
> maxConnections="1">
>ciphers="TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_CBC_SHA256,TLS_RSA_WITH_AES_256_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256,TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA,TLS_RSA_WITH_AES_128_GCM_SHA256,TLS_RSA_WITH_AES_128_CBC_SHA256,TLS_RSA_WITH_AES_128_CBC_SHA"
>  disableCompression="true" disableSessionTickets="false" 
> honorCipherOrder="true" caCertificatePath="C:\PKI\CA">
>  certificateKeyFile="C:\PKI\server.key" 
> certificateChainFile="C:\PKI\server-chain.pem"/>
>   
> 

Please provide the log file, the OpenSSL version used and the libtcnative 
version used. 
Please note that caCertificatePath expects a directory with certificate hash 
files. Plain certs won't work.

M

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: SPNEGO GSSCaller {UNKNOWN} No Delegated Creds

[Michael Osipov]

We need to split between constrained and unconstrained delegation. Let's stay 
with uncontrained, simplest one.

For that to happen you need:
* Enable it for the service account (acceptor side)
* Set the delegate flag (also there is a policy) on the security context 
(initiator side)

Try again. The best thing would be to do with with gss-client/gss-server which 
comes with MIT Kerberos otherwise  you have too many variables in the game.

Alternatively, use https://github.com/pythongssapi/python-gssapi. It has a 
fantastic interface to MIT Kerberos or Heimdal to exactly evaluate your 
environment. I use it as well.

Michael

On 2024/05/03 13:42:39 Tom Delaney wrote:
> Thanks for the reply Michael,
> 
> I'm trying to achieve retrieving delegated credentials. I'm confused by the
> debug output because I'm being told that authentication succeeded but no
> indication of why I'm not receiving delegated credentials other than there
> are none.I have looked over the delegation rules for the service account
> and SPN multiple times. When you mentioned "S4U is tried, but not
> configured for that account. Totally fine" What does that mean? Is there a
> specific place on Tomcat or Windows I need to look for this?
> 
> What I'm expecting to see outputted "Delegated Creds have pname=
> tdela...@subdomain.domain.com sname=krbtgt/SUBDOMAIN.DOMAIN.COM
> authtime=null starttime={date/timestamp} endtime={date/timestamp}"
> 
> P.S
> I see in my ktpass command I made a typo and meant to put SA_EX_VAISSO
> instead of "SA_EX_SSO"
> 
> On Fri, May 3, 2024 at 8:26 AM Michael Osipov  wrote:
> 
> > On 2024/05/02 19:20:59 Tom Delaney wrote:
> > > Hi All,
> > >
> > > Sorry for the duplicate requests. The first one was accidentally flagged
> > > for Google's new Confidential Mode which happened to be flagged.
> > > I have a red hat 9.2 server hosting a web application on a single
> > instance
> > > of Apache Tomcat. This instance is behind an apache HTTP server on
> > version
> > > 2.4.57.The application is hosted on Tomcat 9.0.54.
> > >
> > > Domain: subdomain.domain.com
> > > Site: devexample.domain.com
> > >
> > > URL hit: https://example.subdomain.domain.com/webclient/
> > > <https://devexample.domain.com/webclient_devex/exclient.jsp>exclient.jsp
> > >
> > > *I keep getting this in the Tomcat Logs when accessing the application:*
> > > *>>> Constrained deleg from GSSCaller{UNKNOWN}*
> >
> > You should first try to describe what you are trying to achieve and not
> > what the debug output is. The debug message comes from:
> > https://github.com/openjdk/jdk8u-dev/blob/6b53212ef78ad50f9eede829c5ff87cadcdb434b/jdk/src/share/classes/sun/security/jgss/krb5/Krb5Context.java#L540
> > The message is obviously caused by this call:
> > https://github.com/openjdk/jdk8u-dev/blob/6b53212ef78ad50f9eede829c5ff87cadcdb434b/jdk/src/share/classes/sun/security/jgss/krb5/Krb5Context.java#L254-L263
> >
> > S4U is tried, but not configured for that account. Totally fine.
> >
> > BTW: The filter you use isn't from us.
> >
> > M
> >
> > -
> > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> > For additional commands, e-mail: users-h...@tomcat.apache.org
> >
> >
> 

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: SPNEGO GSSCaller {UNKNOWN} No Delegated Creds

[Michael Osipov]

On 2024/05/02 19:20:59 Tom Delaney wrote:
> Hi All,
> 
> Sorry for the duplicate requests. The first one was accidentally flagged
> for Google's new Confidential Mode which happened to be flagged.
> I have a red hat 9.2 server hosting a web application on a single instance
> of Apache Tomcat. This instance is behind an apache HTTP server on version
> 2.4.57.The application is hosted on Tomcat 9.0.54.
> 
> Domain: subdomain.domain.com
> Site: devexample.domain.com
> 
> URL hit: https://example.subdomain.domain.com/webclient/
> exclient.jsp
> 
> *I keep getting this in the Tomcat Logs when accessing the application:*
> *>>> Constrained deleg from GSSCaller{UNKNOWN}*

You should first try to describe what you are trying to achieve and not what 
the debug output is. The debug message comes from: 
https://github.com/openjdk/jdk8u-dev/blob/6b53212ef78ad50f9eede829c5ff87cadcdb434b/jdk/src/share/classes/sun/security/jgss/krb5/Krb5Context.java#L540
 The message is obviously caused by this call: 
https://github.com/openjdk/jdk8u-dev/blob/6b53212ef78ad50f9eede829c5ff87cadcdb434b/jdk/src/share/classes/sun/security/jgss/krb5/Krb5Context.java#L254-L263

S4U is tried, but not configured for that account. Totally fine.

BTW: The filter you use isn't from us.

M

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: [ANN] Apache Tomcat Native 1.3.0 released

[Michael Osipov]

On 2024/02/13 08:46:42 Mark Thomas wrote:
> The Apache Tomcat team announces the immediate availability of Apache
> Tomcat Native 1.3.0 stable.
> 
> The key features of this release are:
> 
> - The minimum supported OpenSSL version is 1.1.1
> - The minimum supported APR version in 1.6.3
> - The windows binaries in this release have been built with OpenSSL
>3.0.13
> 
> Please refer to the change log for the complete list of changes:
> http://tomcat.apache.org/native-doc/miscellaneous/changelog.html

This link is wrong, it leads me to Tomcat Native 2.0.x.

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Tomcat Instance unable to connect to DB with TCPS

[Kebret, Michael]

Tomcat version 9.0.83 running on Linux  redhat 7 java 11.0.20.

When changing the protocol from TCP to TCPS in Catalina.properties and in 
server.xml we have attribute truststorePassword= (tested with both cleartext 
and encrypted) password connection is refused to the DB and get the below 
exceptions. However, when we add -Djavax.net.ssl.trustStorePassword=cleartext 
to setenv.sh the connection is made successfully. Wanted to see if anyone has 
faced something similar or have any suggestions on how I can get TCPS working 
without having to use -D option in setenv.sh

java.sql.SQLException: Unable to start the Universal Connection Pool: 
oracle.ucp.UniversalConnectionPoolException: Cannot get Connection from 
Datasource: java.sql.SQLRecoverableException: IO Error: The Network Adapter 
could not establish
 the connection
at 
oracle.ucp.util.UCPErrorHandler.newSQLException(UCPErrorHandler.java:456)
at 
oracle.ucp.util.UCPErrorHandler.throwSQLException(UCPErrorHandler.java:133)
at 
oracle.ucp.jdbc.PoolDataSourceImpl.startPool(PoolDataSourceImpl.java:928)
at 
oracle.ucp.jdbc.PoolDataSourceImpl.getConnection(PoolDataSourceImpl.java:1961)
at 
oracle.ucp.jdbc.PoolDataSourceImpl.access$400(PoolDataSourceImpl.java:201)
at 
oracle.ucp.jdbc.PoolDataSourceImpl$31.build(PoolDataSourceImpl.java:4279)
at 
oracle.ucp.jdbc.PoolDataSourceImpl.getConnection(PoolDataSourceImpl.java:1917)
at 
oracle.ucp.jdbc.PoolDataSourceImpl.getConnection(PoolDataSourceImpl.java:1880)
at 
oracle.ucp.jdbc.PoolDataSourceImpl.getConnection(PoolDataSourceImpl.java:1865)
at 
com.wellsfargo.trust.SampleTrust.SampleTrust.doGet(SampleTrust.java:49)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:529)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:623)
at 
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:209)
at 
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:153)
at 
org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:51)
at 
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:178)
at 
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:153)
at 
org.apache.catalina.filters.HttpHeaderSecurityFilter.doFilter(HttpHeaderSecurityFilter.java:129)
at 
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:178)
at 
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:153)
at 
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:168)
at 
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:90)
at 
org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:481)
at 
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:130)
at 
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:93)
at 
org.apache.catalina.valves.StuckThreadDetectionValve.invoke(StuckThreadDetectionValve.java:185)
at 
org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:670)
at 
org.apache.catalina.valves.RemoteIpValve.invoke(RemoteIpValve.java:765)
at 
org.apache.catalina.valves.RequestFilterValve.process(RequestFilterValve.java:355)
at 
org.apache.catalina.valves.RemoteAddrValve.invoke(RemoteAddrValve.java:54)
at 
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:74)
at 
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:342)
at 
org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:390)
at 
org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:63)
at 
org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:928)
at 
org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1794)
at 
org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:52)
at 
org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1191)
at 
org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659)
at 
org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: oracle.ucp.UniversalConnectionPoolException: Cannot get Connection 
from Datasource: java.sql.SQLRecoverableException: IO Error: The Network 
Adapter could not establish the connection
at 
oracle.ucp.util.UCPErrorHandler.newUniversalConnectionPoolException(UCPErrorHandler.java:336)
at 

Re: Rotating/archiving catalina.out

[Michael Osipov]

On 2024/01/29 20:28:05 Aryeh Friedman wrote:
> We need to shrink the size of catalina.out but looking at the logging
> documentation I do not see any way to do this with the standard
> logging.properties (or else where).   Due to the nature of the
> production site we never bring it completely down unless we must (life
> critical 24/7/365)
> 
> Specifically we have a fair number of System.out.println's with
> debugging information to it and we dumb stack traces into it also.
> And without stopping and restarting tomcat we want to make it so there
> is periodic rotation of catalina.out to some other file?   (note stack
> traces do go to the dated one but not the System.out.println's)

If you use the FreeBSD port, it will automatically use commons-daemon. JSVC 
does support SIGUSR1 with reopening logs: 
https://github.com/apache/commons-daemon/blob/6a95175ba875977a59fa4be2e99d80d2fa8d053a/src/native/unix/native/jsvc-unix.c#L124-L127
Then, you can use newsyslogd to rotate with SIGUSR1.  This works for me for 
ages now:
/usr/local/ldadocgen/prod/tomcat-1/logs/catalina.outldadocgen:ldadocgen
064030*@T00GBXP/var/run/tomcat_ldadocgen_prod_1.pid
SIGUSR1

Michael

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Problems with cross app uploads (migrating from 9.0.35 to 9.0.84)

[Michael Osipov]

Aryeh,

a general advice: The current Tomcat ports on FreeBSD are a bit unfortunate 
since they don't motivate to use separate catalina bases. I do not recommend 
using it as-is. Leverage the multi-instance options provided by its rc script 
to split out and the do:
> tomcat_{name}_enable="YES"
> tomcat_{name}_catalina_user="your-user"
> tomcat_{name}_catalina_base="/your/path"
> tomcat_{name}_umask="0027"

I do *not* recommend to spill your deployment into CATALINA_HOME [1]
I have discussed this once briefly with the port maintainer, but simply didn't 
have the time to work on a tomcatX-split port.

Michael

[1] 
https://github.com/freebsd/freebsd-ports/blob/4430a994eea5d6b2ed84c940963788069d6fd1d6/www/tomcat9/Makefile#L42

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: 9.0.83 addSslHostConfig failures?

[Michael Osipov]

On 2023/11/21 11:25:11 Michael Osipov wrote:
> On 2023/11/20 22:14:14 Daniel Skiles wrote:
> > Was there a change to the addSslHostConfig JMX mbean operation between
> > 9.0.82 and 9.0.83?  I have some code that works in 82, but fails with an
> > MBeanException: Cannot find operation [addSslHostConfig] in 9.0.83.
> > 
> > When I attempt to look at the available operations on ProtocolHandler in
> > jconsole, it throws an exception in 83 that opens a new window, but works
> > in 82.
> 
> I have the following with 8.5.x:
> > Error setting Operation panel :org.apache.coyote.Request
> > Error setting Operation panel :org.apache.tomcat.util.net.SSLHostConfig
> 
> addSslHostConfig is greyed out for me...let me go back a patch version...

Tried on 8.5.92, same behavior. You should share your connector config.

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: 9.0.83 addSslHostConfig failures?

[Michael Osipov]

On 2023/11/20 22:14:14 Daniel Skiles wrote:
> Was there a change to the addSslHostConfig JMX mbean operation between
> 9.0.82 and 9.0.83?  I have some code that works in 82, but fails with an
> MBeanException: Cannot find operation [addSslHostConfig] in 9.0.83.
> 
> When I attempt to look at the available operations on ProtocolHandler in
> jconsole, it throws an exception in 83 that opens a new window, but works
> in 82.

I have the following with 8.5.x:
> Error setting Operation panel :org.apache.coyote.Request
> Error setting Operation panel :org.apache.tomcat.util.net.SSLHostConfig

addSslHostConfig is greyed out for me...let me go back a patch version...

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

DBCP exception after Tomcat 10.1.13 -> 10.1.14 upgrade

[Michael Hayes]

I have just upgraded a working Tomcat 10.1.13 installation to Tomcat 10.1.14,
on Rocky Linux 9.2, PostgreSQL 11.20, JDBC driver 42.6.0, Java 17.0.8.

During startup I get this exception:

java.sql.SQLException
at 
org.apache.tomcat.jdbc.pool.ConnectionPool.setupConnection(ConnectionPool.java:351)
at 
org.apache.tomcat.jdbc.pool.ConnectionPool.getConnection(ConnectionPool.java:200)
at 
org.apache.tomcat.jdbc.pool.DataSourceProxy.getConnection(DataSourceProxy.java:131)
at fms.lib.Jdbc.open(Jdbc.java:139)
at fms.lib.Startup.contextInitialized(Startup.java:38)
at 
org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4422)
at 
org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:4860)
... lots more
Caused by: java.lang.IllegalArgumentException: 
org.apache.tomcat.jdbc.pool.PooledConnection is not an interface
at 
java.base/java.lang.reflect.Proxy$ProxyBuilder.validateProxyInterfaces(Proxy.java:706)
at java.base/java.lang.reflect.Proxy$ProxyBuilder.(Proxy.java:648)
at 
java.base/java.lang.reflect.Proxy.lambda$getProxyConstructor$1(Proxy.java:440)
at 
java.base/jdk.internal.loader.AbstractClassLoaderValue$Memoizer.get(AbstractClassLoaderValue.java:329)
at 
java.base/jdk.internal.loader.AbstractClassLoaderValue.computeIfAbsent(AbstractClassLoaderValue.java:205)
at java.base/java.lang.reflect.Proxy.getProxyConstructor(Proxy.java:438)
at java.base/java.lang.reflect.Proxy.getProxyClass(Proxy.java:398)
at 
org.apache.tomcat.jdbc.pool.ConnectionPool.getProxyConstructor(ConnectionPool.java:377)
at 
org.apache.tomcat.jdbc.pool.ConnectionPool.setupConnection(ConnectionPool.java:339)
... 45 more

Any ideas?

Thanks...

--Mike

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: AW: Solution to "Invalid keystore format" (cross-posted to Tomcat Users List at Apache, and Java 400 List at Midrange)

[Michael Osipov]

On 2023/09/12 07:06:52 "Thomas Hoffmann (Speed4Trade GmbH)" wrote:
> Hallo James,
> 
> > -Ursprüngliche Nachricht-
> > Von: James H. H. Lampert 
> > Gesendet: Montag, 11. September 2023 18:31
> > An: Java 400 List ; Tomcat Users List
> > 
> > Betreff: Solution to "Invalid keystore format" (cross-posted to Tomcat Users
> > List at Apache, and Java 400 List at Midrange)
> > 
> > Ladies and Gentlemen of Both Lists:
> > 
> > Last Friday evening, I ran into a problem updating SSL/TLS keystores on two
> > customer boxes, and spent three hours yesterday, finding the cause, doping
> > out a way to salvage the certs they'd paid for, and doping out a solution to
> > keep it from happening in the future.
> > 
> > It seems that with the new keystores (generated on my Mac, initially created
> > with Keytool, and then maintained with Keystore Explorer), they were
> > getting:
> > 
> >  >   Throwable occurred: java.io.IOException: Invalid keystore format
> > >   at com.ibm.crypto.provider.JavaKeyStore.engineLoad(Unknown Source)
> > >   at java.security.KeyStore.load(KeyStore.java:414)
> > 
> > I put them back on their old keystores, and cycled Tomcat again, to get them
> > back up, and then spent three hours working the problem yesterday
> > (Sunday) afternoon.
> > 
> > It turns out that the default keytool on my new Mac is the one from Java 17.
> > And the customer boxes are running Tomcat under much older JVMs,
> > because there's always a significant time lag before any given JVM makes it
> > to an IBM Midrange box.
> > 
> > So I was able to salvage one of the certs (and its CA reply, and its
> > chain) by moving the cert to a keystore generated on my *old* Mac (with
> > Java 8 as the default JVM), and then re-signing and re-chaining it in KSE. 
> > And I
> > tested the KS on our V6 box, to make *sure* it worked.
> > 
> > I then looked for a way, since my new Mac *has* a Java 8 JVM (it's just not
> > the default), to conveniently use that JVM's Keytool, and came up with a
> > wrapper BASH script to do the job. I tested the wrapper script by using it 
> > to
> > generate their new keystore.
> > 
> > Key takeaway (no pun intended) here: if you get an "Invalid keystore
> > format" in Tomcat (or presumably anything else that uses Java Keystores),
> > when generating a keystore on one box for use on another, *look for a
> > difference in JVM.*
> > 
> > --
> > JHHL
> > 
> 
> I moved away from using the proprietary java keystore format.
> I switched to using Base64 PEM format. This is usually also the format you 
> get from the certificate issuer.
> No need to convert it into Java format any more and you can also open it with 
> any text editor.

This is exactly the same what I have been doing for the past 10 years. No 
pointless fiddling with Java keystores.

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Tomcat 9.0.76 Memory leak with Java 17

[Michael Osipov]

On 2023/07/11 18:16:24 Christopher Schultz wrote:
> You should report all of the previous issues to Oracle against their 
> ORDS version 22.1 and ask them to fix them. It's why you write those 
> big, fat checks in the first place ;)

This doesn't really matter. I have reported a memory leak in OJDBC many years 
ago where a background thread pins the WebappClassLoader. Answer from Oracle: 
This happens only once, does not repeat. We won't address that...

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Apache Tomcat 9.x end of life

[Osipov, Michael (SMD IT IN)]

On 2023-06-08 18:17, Mark Thomas wrote:

On 08/06/2023 16:32, Zoltan Balogh wrote:

Dear Apache Team,

I just would like to ask is there any planned EOL for Tomcat 9?


Best guess at the moment (and it is only a guess - no decisions have 
been made) is 3 years after 8.5.x is EOL which gives a date of 31 March 
2027.


It is worth noting that Tomcat major versions track releases of the 
Jakarta Servlet specification and there has been an effort from the 
Jakarta project to increase the release cadence. Delays are also not 
unheard of. The estimate above could easily be a year out in either 
direction.


We are heavily using Java 8 that is supported until 2030, so we expect 
Apache Tomcat 9 will be also support while Java 8 is supported.


That is extremely unlikely at this point.

Important note that Tomcat 9 is the last release that works fine with 
Java 8 along with Java EE. Therefore, it is highly important for us to 
have a maintained Tomcat 9 for Java 8.


Once Tomcat 9.0.x reaches EOL we will provide, assuming there is demand, 
Tomcat 9.10.x which will be Tomcat 10.1.x but with the Java EE 8 API. 
The minimum Java requirement for 9.10.x will be Java 11.


Once Tomcat 10.1.x reaches EOL we will provide Tomcat 9.11.x which would 
be Tomcat 11.0.x but with the Java EE 8 API.


I wonder to what extent this is necessary at all. Baseline is Java EE 8 
and Java 8 and it should stay that way, I guess. If some code is back 
ported, sure it needs to be refitted to Java 8 if necessary, otherwise 
there would no more Java EE 8 capable container on Java 8. Moving to a 
new Java version is one thing, but replacing namespaces is a completely 
different one for people.


M

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: AbstractEndpoint#logCertificate() prints incorrect information

[Osipov, Michael (SMD IT IN)]

On 2023-06-08 15:36, Mark Thomas wrote:

On 08/06/2023 14:22, Osipov, Michael (SMD IT IN) wrote:

On 2023-06-08 14:45, Mark Thomas wrote:

On 08/06/2023 12:25, Osipov, Michael (SMD IT IN) wrote:

Folks,

I am running of 8.5.88 and noticed these lines in my catalina.out:
2023-06-08T12:38:54.938 INFORMATION [main] 
org.apache.tomcat.util.net.AbstractEndpoint.logCertificate 
Connector [https-openssl-apr-8444], TLS virtual host 
[deblndw024v.ad001.siemens.net], certificate type [RSA] configured 
from [/net/home/smartld/.keystore] using alias [tomcat] and with 
trust store [null]
2023-06-08T12:38:55.036 INFORMATION [main] 
org.apache.tomcat.util.net.AbstractEndpoint.logCertificate 
Connector [https-openssl-apr-18444], TLS virtual host 
[deblndw024v.ad001.siemens.net], certificate type [RSA] configured 
from [/net/home/smartld/.keystore] using alias [tomcat] and with 
trust store [/opt/openssl/certs]


I have two connectors configured (both APR and OpenSSL), the only 
difference is that the second one (18444) requires TLS client 
certificates:
keepAliveTimeout="30" maxParameterCount="1000"

  maxHttpHeaderSize="24576" maxThreads="250"
  SSLEnabled="true" scheme="https" secure="true"
  defaultSSLHostConfigName="deblndw024v.ad001.siemens.net">
  protocols="TLSv1.2+TLSv1.3"

    honorCipherOrder="true" disableSessionTickets="true"
ciphers="HIGH:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!3DES:!MD5:!PSK:!DSS:!SHA1:!SHA256:!SHA384">
    certificateFile="/opt/openssl/deblndw024v.ad001.siemens.net/cert.crt"

certificateKeyFile="/opt/openssl/deblndw024v.ad001.siemens.net/key.crt"
  certificateKeyPassword="..." type="RSA" />
  



and

keepAliveTimeout="720" maxParameterCount="1000"

  maxHttpHeaderSize="24576" maxThreads="250"
  SSLEnabled="true" scheme="https" secure="true"
  defaultSSLHostConfigName="deblndw024v.ad001.siemens.net">
  protocols="TLSv1.2+TLSv1.3"

    honorCipherOrder="true" disableSessionTickets="true"
    certificateVerification="optional" 
certificateVerificationDepth="5"

    caCertificatePath="/opt/openssl/certs"
ciphers="HIGH:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!3DES:!MD5:!PSK:!DSS:!SHA1:!SHA256:!SHA384">
    certificateFile="/opt/openssl/deblndw024v.ad001.siemens.net/cert.crt"

certificateKeyFile="/opt/openssl/deblndw024v.ad001.siemens.net/key.crt"
  certificateKeyPassword="..." type="RSA" />
    
  value="/opt/openssl/siemens-medium+strong-clientcert-cacerts.crt" />

    
  



The information displayed is partially wrong. It missed to 
differentiate between store types 
(org.apache.tomcat.util.net.SSLHostConfigCertificate.StoreType).


I disagree with the characterization that the data is wrong. There 
are many TLS configuration attributes. A few are displayed, most are 
not. StoreType is one of those that is not.


The aim isn't to display the full configuration (admins can look at 
server.xml for that). The intention is to make clear, for each 
configured server certificate, which configuration files are being 
used. The idea being that you can then examine the relevant files 
and/or server.xml if there is an issue.


I'm not seeing much of a case for adding StoreType.


..and that's the problem. Neither [/net/home/smartld/.keystore] nor 
[tomcat] is used because I have never configured it. I don't expect 
full display, but correct display. I rather see no information than 
incorrect information. I don't insist on printing the storeType at 
all, it needs to be used in Java code to determine what configuration 
has to be printed.


Got it. Yep - open a bug report for that.


Done: https://bz.apache.org/bugzilla/show_bug.cgi?id=66635


-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: AbstractEndpoint#logCertificate() prints incorrect information

[Osipov, Michael (SMD IT IN)]

On 2023-06-08 14:45, Mark Thomas wrote:

On 08/06/2023 12:25, Osipov, Michael (SMD IT IN) wrote:

Folks,

I am running of 8.5.88 and noticed these lines in my catalina.out:
2023-06-08T12:38:54.938 INFORMATION [main] 
org.apache.tomcat.util.net.AbstractEndpoint.logCertificate Connector 
[https-openssl-apr-8444], TLS virtual host 
[deblndw024v.ad001.siemens.net], certificate type [RSA] configured 
from [/net/home/smartld/.keystore] using alias [tomcat] and with 
trust store [null]
2023-06-08T12:38:55.036 INFORMATION [main] 
org.apache.tomcat.util.net.AbstractEndpoint.logCertificate Connector 
[https-openssl-apr-18444], TLS virtual host 
[deblndw024v.ad001.siemens.net], certificate type [RSA] configured 
from [/net/home/smartld/.keystore] using alias [tomcat] and with 
trust store [/opt/openssl/certs]


I have two connectors configured (both APR and OpenSSL), the only 
difference is that the second one (18444) requires TLS client 
certificates:
keepAliveTimeout="30" maxParameterCount="1000"

  maxHttpHeaderSize="24576" maxThreads="250"
  SSLEnabled="true" scheme="https" secure="true"
  defaultSSLHostConfigName="deblndw024v.ad001.siemens.net">
  protocols="TLSv1.2+TLSv1.3"

    honorCipherOrder="true" disableSessionTickets="true"
ciphers="HIGH:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!3DES:!MD5:!PSK:!DSS:!SHA1:!SHA256:!SHA384">
    certificateFile="/opt/openssl/deblndw024v.ad001.siemens.net/cert.crt"

certificateKeyFile="/opt/openssl/deblndw024v.ad001.siemens.net/key.crt"
  certificateKeyPassword="..." type="RSA" />
  



and

keepAliveTimeout="720" maxParameterCount="1000"

  maxHttpHeaderSize="24576" maxThreads="250"
  SSLEnabled="true" scheme="https" secure="true"
  defaultSSLHostConfigName="deblndw024v.ad001.siemens.net">
  protocols="TLSv1.2+TLSv1.3"

    honorCipherOrder="true" disableSessionTickets="true"
    certificateVerification="optional" certificateVerificationDepth="5"
    caCertificatePath="/opt/openssl/certs"
ciphers="HIGH:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!3DES:!MD5:!PSK:!DSS:!SHA1:!SHA256:!SHA384">
    certificateFile="/opt/openssl/deblndw024v.ad001.siemens.net/cert.crt"

certificateKeyFile="/opt/openssl/deblndw024v.ad001.siemens.net/key.crt"
  certificateKeyPassword="..." type="RSA" />
    
  value="/opt/openssl/siemens-medium+strong-clientcert-cacerts.crt" />

    
  



The information displayed is partially wrong. It missed to 
differentiate between store types 
(org.apache.tomcat.util.net.SSLHostConfigCertificate.StoreType).


I disagree with the characterization that the data is wrong. There are 
many TLS configuration attributes. A few are displayed, most are not. 
StoreType is one of those that is not.


The aim isn't to display the full configuration (admins can look at 
server.xml for that). The intention is to make clear, for each 
configured server certificate, which configuration files are being used. 
The idea being that you can then examine the relevant files and/or 
server.xml if there is an issue.


I'm not seeing much of a case for adding StoreType.


..and that's the problem. Neither [/net/home/smartld/.keystore] nor 
[tomcat] is used because I have never configured it. I don't expect full 
display, but correct display. I rather see no information than incorrect 
information. I don't insist on printing the storeType at all, it needs 
to be used in Java code to determine what configuration has to be printed.


Moreover, from my PoV it makes little sense to print "trust store 
[null]" if no verification is requested, no? It causes, at least, 
confusion that a user/admin has missed to configure something.


Only if the admin doesn't understand the purpose of the trust store. I'd 
prefer to keep it. It makes it explicit that one hasn't been configured.


I see your point, yet only a fraction will use TLS client cert, so for 
those who don't this is just (confusion) log junk.


While the latter is debatable, I consider the former wrong when not 
KEYSTORE is used.


M


-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

AbstractEndpoint#logCertificate() prints incorrect information

[Osipov, Michael (SMD IT IN)]

Folks,

I am running of 8.5.88 and noticed these lines in my catalina.out:

2023-06-08T12:38:54.938 INFORMATION [main] 
org.apache.tomcat.util.net.AbstractEndpoint.logCertificate Connector 
[https-openssl-apr-8444], TLS virtual host [deblndw024v.ad001.siemens.net], 
certificate type [RSA] configured from [/net/home/smartld/.keystore] using 
alias [tomcat] and with trust store [null]
2023-06-08T12:38:55.036 INFORMATION [main] 
org.apache.tomcat.util.net.AbstractEndpoint.logCertificate Connector 
[https-openssl-apr-18444], TLS virtual host [deblndw024v.ad001.siemens.net], 
certificate type [RSA] configured from [/net/home/smartld/.keystore] using 
alias [tomcat] and with trust store [/opt/openssl/certs]


I have two connectors configured (both APR and OpenSSL), the only 
difference is that the second one (18444) requires TLS client certificates:


  

  



and



  


  

  



The information displayed is partially wrong. It missed to differentiate 
between store types 
(org.apache.tomcat.util.net.SSLHostConfigCertificate.StoreType).


Moreover, from my PoV it makes little sense to print "trust store 
[null]" if no verification is requested, no? It causes, at least, 
confusion that a user/admin has missed to configure something.


Should I file a bug?

Michael

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: [solved] How to debug 404s / How to enable error log

[Michael B Allen]

On Tue, Jan 31, 2023 at 5:48 PM Konstantin Kolinko 
wrote:

> and your configuration is missing
> catalina.properties
> context.xml
> logging.properties
>
> files that are required for proper operation.
>

Thanks.

I have confirmed that also copying logging.properties results in a
logs/catalina.2023-02-01.log file being created.

For posterity (mostly for me actually), the complete procedure for creating
a proper CATALINA_BASE is the following:

Directories that need to be created in CATALINA_BASE are:

bin
conf
logs
work
webapps
temp

Files that need to be copied from CATALINA_HOME\conf to CATALINA_BASE\conf
are:

server.xml
tomcat-users.xml
web.xml
catalina.properties
context.xml
logging.properties

A minimal bin/run.sh for Linux might look like:

#!/bin/sh

export JRE_HOME=/usr/local/java11
export CATALINA_HOME=/path/to/tomcat10
export CATALINA_BASE=/path/to/myapp-base

${CATALINA_HOME}/bin/catalina.sh run "$@"

A minimal bin\run.bat for Windows might look like:

setlocal

set "JRE_HOME=C:\tmp\java17"
set "CATALINA_HOME=C:\tmp\tomcat10"
set "CATALINA_BASE=C:\tmp\myapp-base"

%CATALINA_HOME%\bin\catalina.bat start

Mike

--
Michael B Allen
Java AD DS Integration
http://www.ioplex.com/

Re: [solved] How to debug 404s / How to enable error log

[Michael B Allen]

On Tue, Jan 31, 2023 at 4:33 PM Konstantin Kolinko 
wrote:

> > INFO: No global web.xml found
>
> The above message is odd.
> Essentially, it means that the file %CATALINA_BASE%\conf\web.xml does not
> exist.
>

Solved.

That was it. After copying over the conf/web.xml into myapp-base/conf, the
app now works.

The REALLY embarrassing thing is that I just asked about this on Nov 15,
2021 and even after taking notes about it, I failed to include the web.xml
in my notes.

How have you configured your %CATALINA_BASE% directory?
>

These are all of the files that give me what appear to be a working config:

C:\tmp\myapp-base>tree /f
Folder PATH listing
Volume serial number is BAF1-D00E
C:.
│   cmd.lnk
│
├───bin
│   start.bat
│
├───conf
│   │   server.xml
│   │   tomcat-users.xml
│   │   web.xml
│   │
│   └───Catalina
│   └───localhost
├───logs
│   myapp.log
│   localhost_access_log.2023-01-31.txt
│
├───temp
├───webapps
│   └───myapp
│   │   index.jsp
│   │
│   └───WEB-INF
│   │   example.prp
│   │   web.xml
│   │
│   └───lib
│   myapp-jakarta-1.2.3.jar
│
└───work
└───Catalina
└───localhost
└───myapp
└───org
└───apache
└───jsp
index_jsp.class
index_jsp.java


>
>
> The conf/web.xml configures all the defaults, shared by all web
> applications.
> That includes
> - declaration of DefaultServlet
> - declaration of JspServlet
> - "welcome-file-list"
> - "mime-mapping"s
>
> > How does one enable a debug log? The documentation about such things is
> > pretty cryptic.
>
> It assumes that you are familiar with java.util.logging.
>

That's unfortunate.  It would be so much nicer if it was a simple one-liner.

Mike

How to debug 404s / How to enable error log

[Michael B Allen]

This is embarrassing. I have apparently forgotten how to operate Tomcat.

I just downloaded openjdk 17, Tomcat 10.1.5 to a clean Windows 10, upacked
everything, created a myapp-base with a .bat like:

--8<-- bin\start.bat --8<--

setlocal

set "JRE_HOME=C:\tmp\java17"
set "CATALINA_HOME=C:\tmp\tomcat10"
set "CATALINA_BASE=C:\tmp\myapp-base"

%CATALINA_HOME%\bin\catalina.bat start

--

Copied webapp files into the webapps dir.

On bin\start I get some INFO in the console like:

Jan 31, 2023 3:46:02 PM org.apache.coyote.AbstractProtocol init
INFO: Initializing ProtocolHandler ["http-nio-8080"]
Jan 31, 2023 3:46:02 PM org.apache.catalina.startup.Catalina load
INFO: Server initialization in [797] milliseconds
Jan 31, 2023 3:46:02 PM org.apache.catalina.core.StandardService
startInternal
INFO: Starting service [Catalina]
Jan 31, 2023 3:46:02 PM org.apache.catalina.core.StandardEngine
startInternal
INFO: Starting Servlet engine: [Apache Tomcat/10.1.5]
Jan 31, 2023 3:46:02 PM org.apache.catalina.startup.HostConfig
deployDirectory
INFO: Deploying web application directory [C:\tmp\myapp-base\webapps\myapp]
Jan 31, 2023 3:46:02 PM org.apache.catalina.startup.ContextConfig
getDefaultWebXmlFragment
INFO: No global web.xml found
Jan 31, 2023 3:46:03 PM org.apache.jasper.servlet.TldScanner scanJars
INFO: At least one JAR was scanned for TLDs yet contained no TLDs. Enable
debug logging for this logger for a complete list of JARs that were scanned
but no TLDs were found in them. Skipping unneeded JARs during scanning can
improve startup time and JSP compilation time.
Jan 31, 2023 3:46:03 PM org.apache.catalina.startup.HostConfig
deployDirectory
INFO: Deployment of web application directory
[C:\tmp\myapp-base\webapps\myapp] has finished in [656] ms
Jan 31, 2023 3:46:03 PM org.apache.coyote.AbstractProtocol start
INFO: Starting ProtocolHandler ["http-nio-8080"]
Jan 31, 2023 3:46:03 PM org.apache.catalina.startup.Catalina start
INFO: Server startup in [765] milliseconds

The myapp servlet filter init runs. A localhost access log is created in
myapp-base\logs.

But trying to access anything through localhost:8080 or remotely by
hostname fails with nothing but 404:

10.22.10.30 - - [31/Jan/2023:15:14:49 -0500] "GET /myapp/ HTTP/1.1" 404 682
10.22.10.30 - - [31/Jan/2023:15:14:49 -0500] "GET /favicon.ico HTTP/1.1"
404 682
0:0:0:0:0:0:0:1 - - [31/Jan/2023:15:22:43 -0500] "GET / HTTP/1.1" 404 682
0:0:0:0:0:0:0:1 - - [31/Jan/2023:15:22:43 -0500] "GET /favicon.ico
HTTP/1.1" 404 682
0:0:0:0:0:0:0:1 - - [31/Jan/2023:15:22:49 -0500] "GET /myapp HTTP/1.1" 302 -
0:0:0:0:0:0:0:1 - - [31/Jan/2023:15:22:49 -0500] "GET /myapp/ HTTP/1.1" 404
682
0:0:0:0:0:0:0:1 - - [31/Jan/2023:15:22:53 -0500] "GET /myapp/index.jsp
HTTP/1.1" 404 682

So now I'm just trying to find out how to enable an error log and for the
life of me I'm completely blanking.

How does one enable a debug log? The documentation about such things is
pretty cryptic.

How can I diagnose this further?

Thanks,
Mike

Re: javax.servlet vs jakarta.servlet?

[Michael B Allen]

On Thu, Dec 30, 2021 at 10:57 AM Christopher Schultz
 wrote:
>
> You mean like ServletContext.getRealPath?

Honestly I'm not sure how I feel about getRealPath. On the one hand, I
don't think it's reasonable to just pretend that applications only
exist in the vacuum of space. There are many practical reasons why an
application might want to interact with the filesystem but without
requiring absolute paths. Just because there might not be a file
system is a weak excuse to not properly account for one. Being able to
update a file of properties for example and have the application see
that the file is updated without reloading the webapp (maybe even if
the app is packaged as a war) is very useful. On the other hand I
don't think I would want another 10 classes just to create some kind
of ancillary webapp storage abstraction.

Mike

-- 
Michael B Allen
Java Active Directory Integration
http://www.ioplex.com/

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: javax.servlet vs jakarta.servlet?

[Michael B Allen]

On Wed, Dec 29, 2021 at 2:07 PM Mark Thomas  wrote:
> One of the advantages of moving to Eclipse is that everyone involved in
> the spec, not just the spec lead, has an equal say in what goes into the
> spec.

That sounds like design by committee which is my concern. IMO the only
way to design a really good API is for one individual with a vision to
just write it, use it, exercise it and then see what works and what
does not. I routinely re-write things multiple times very slowly
before settling on an API. It's exponentially harder for multiple
people to have the same vision. Design by committee can easily
degenerate into an overly-OOP set of interfaces and classes and enums
to define an API that doesn't actually do that much. Once you lock
into an API, that could be used for decades. So even the slightest
mistake can create seams that transcend every codebase that uses it
with long term consequences.

But I suppose this is thoroughly off topic at this point.

Thanks for your help,

Mike

--
Java Active Directory Integration
http://www.ioplex.com/

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: javax.servlet vs jakarta.servlet?

[Michael B Allen]

On Tue, Dec 28, 2021 at 10:52 AM Mark Thomas  wrote:
>
> On 28/12/2021 14:58, Michael B Allen wrote:
> > On Tue, Dec 28, 2021 at 3:29 AM Johan Compagner  
> > wrote:
> >>
> >> Will that really work?
> >
> > No.
> >
> > Clearly I'm not paying attention because after reading surprisingly
> > little information about this fundamental incompatibility and
> > downloading and trying Tomcat 10, there is no way to use a
> > javax.servlet library or application with a jakarta.servlet app server
> > like Tomcat 10.
>
> Yes there is. Just place the Java EE web application in the
> webapps-javaee directory and Tomcat will convert automatically to
> Jakarta EE and then deploy it.

Hi Mark,

Thanks for the detailed answer.

I will try this (actually I already did but I'm getting 404 so I
probably have some other unrelated problem).

Actually it seems the migration tool behind this feature:

  https://github.com/apache/tomcat-jakartaee-migration

is a better and more general solution. If it's just re-writing package
names, can any jar converted using the tomcat-jakartaee-migration tool
be used with another app server like Jetty 11?

> > So now I REALLY want to know if people actually use
> > Tomcat 10 and what the release policy will be going forward for Tomcat
> > 9. Is Tomcat 9 going to be maintained separately with security fixes
> > and such?
>
> It is the intention of the Tomcat community to continue to support
> Tomcat 9 for as long as there are users that want a supported Java EE
> platform.
>
> Normally, we support 3 versions in parallel with a major version
> reaching EOL every ~3 years. Typically a major version will be supported
> for ~10 years.
>
> Tomcat 9 will be different. When Tomcat 9 reaches the normal EOL point
> (roughly 2027) 9.0.x releases will stop but Tomcat 9.10.x releases will
> start. These will essentially track Tomcat 10.x releases but with the
> Java EE API rather than the Jakarta EE API.
>
> When Tomcat 10.x reaches end of life (~2030) 9.10.x releases will stop
> and 9.11.x releases will start. These will essentially track Tomcat 11.x
> releases but with the Java EE API rather than the Jakarta EE API.
>
> And so on for as long as there is community demand for the Java EE
> support and there are 3 Tomcat PMC members willing to vote for the release.

That is encouraging. So just to be crystal clear, Tomcat 9 will get
security fixes as if Tomcat 10 didn't exist?

Is there a page on apache.org that explains what you say or are you
speaking authoritatively?

> If anyone has an app that the conversion tool doesn't handle correctly,
> open a bug, tell us what it isn't getting right and we should be able to
> fix it.

Will do.

> No need for any build process changes if you don't want to. Just use
> webapps-javaee and the automatic conversion.

Well the webapps-javaee dir is good for an existing application that
you don't want to touch but maybe there should also be a
WEB-INF/lib-javaee/ directory to convert individual jars? That seems
like a more likely scenario where people want to use JakartaEE but
there are a few libs that haven't made the leap.

> >package javax.servlet.http;
> >
> >public class HttpServletRequestWrapper extends
> > jakarta.servlet.http.ServletRequestWrapper implements
> > HttpServletRequest
> >{
> >  // nothing to see here
> >}
>
> This illustrates the my point. You would not be able to cast an instance
> of javax.servlet.http.HttpServletRequestWrapper to
> javax.servlet.http.ServletRequestWrapper

Ok so the classes don't cascade effortlessly. That doesn't mean you
couldn't create a wrapper that just operates on a member.

> Alternatively, just try and write a class (that compiles) that
> implements javax.servlet.http.HttpServletRequest and
> jakarta.servlet.http.HttpServletRequest

I wasn't suggesting that the javax classes had to actually be jakarta
instances under the hood.

But no matter. If the migration tool actually does what it looks like
it's supposed to do, I would say that's the best option. People can
either call it in their build to make two jars or the operator can use
it to convert a jar at their end.

> Ignoring the package change, the biggest difference in Servlet 6 and the
> other Jakarta EE 10 APIs (Tomcat 10.1.x) is that a lot of the deprecated
> code has been removed.

Just an aside, I must say the community process worries me a little.
Say what you want about Oracle but they have provided a stable
platform for many years. Contrarily, there have historically been
community driven efforts that have produced some, lets say,
"extraneous" results. Look into the history of X.500. Or Oasis
standards like CORBA. HttpServletRequest.login() is misplace

Re: javax.servlet vs jakarta.servlet?

[Michael B Allen]

On Tue, Dec 28, 2021 at 3:29 AM Johan Compagner  wrote:
>
> Will that really work?

No.

Clearly I'm not paying attention because after reading surprisingly
little information about this fundamental incompatibility and
downloading and trying Tomcat 10, there is no way to use a
javax.servlet library or application with a jakarta.servlet app server
like Tomcat 10. So now I REALLY want to know if people actually use
Tomcat 10 and what the release policy will be going forward for Tomcat
9. Is Tomcat 9 going to be maintained separately with security fixes
and such?

There must be thousands of libraries and applications that cannot be
converted for various reasons. Even active open source projects are
probably going to punt on this. Think about how many github projects
there are with one guy using it for some thing at work and so he
doesn't want to make up some screwy build process that creates two
jars. This basically guarantees that people will be using Tomcat 9 for
many years.

Also, correct me if I'm wrong but it seems to me that the two
interfaces should easily coexist. The Servlet interface is just that -
an interface. So just create an implementation of javax.servlet that
uses the jakarta.servlet code as it's implementation. In many cases
the code could literally be like:

  package javax.servlet.http;

  public class HttpServletRequestWrapper extends
jakarta.servlet.http.ServletRequestWrapper implements
HttpServletRequest
  {
// nothing to see here
  }

The only code that might be different is details about sessions or
something that jakarta.servlet wants to do a little differently. But
in that case, the ability of OOP to override methods to selectively
change some behavior can be utilized well here. Unless something in
jakarta.servlet changes that fundamentally breaks compatibility with
javax.servlet, code using javax.servlet shouldn't know the difference.
And if something does change that breaks backward compatibility, that
would be a slightly frightening change in policy regarding backward
compatibility of Java code in general since just about any piece of
Java code written in literally the last 20 years will run on the
latest platform with only minor tweaks.

The effort required for every product and project in the universe to
create a build process that generates two jars is massive compared to
the effort that would be required for Tomcat to support both
javax.servlet and jakarta.servlet even if it were difficult to do for
some reason I'm not seeing.

Mike

--
Michael B Allen
Java Active Directory Integration
http://www.ioplex.com/

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

javax.servlet vs jakarta.servlet?

[Michael B Allen]

Hello Tomcat Users,

Does anyone have a sense of how many people are using the jakarta.servlet API?

I have a product that includes some classes that use javax.servlet and
I'm trying to figure out how to support people who want to use
jakarta.servlet. I plan to install Tomcat 10 momentarily but I assume
one just has to add whatever javax.servlet jar to make Tomcat 10 work
with a javax.servlet lib or app?

Mike

-- 
Michael B Allen
Java Active Directory Integration
http://www.ioplex.com/

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Authentication with Browser stopped working / missing exception handling in getRemainingLifetime

[Michael B Allen]

On Tue, Nov 23, 2021 at 2:59 PM Thomas Hoffmann (Speed4Trade GmbH)
 wrote:
>
> Short Addendum:
>
> The "destroyed" flag gets set, when the dispose-method of the 
> GSSCredentialImpl was invoked.
> Currently, I have no clue when and how it happens, but I have seen this 
> problem every few months.
> So it is only occurring sometimes. Maybe if the Kerberos ticket expires and 
> the http session is still alive (?)
>
> Nevertheless, the application should be able to recover from this situation 
> and handles it like "not authenticated".

So as suspected it may actually be an invalid credential that maybe
Tomcat had a hand in. If Tomcat disposed the credential and then
subsequently tried to use it for any reason, that would be "invalid".
So that might warrant investigation before submitting a bug report.

But I would still argue that a JGSS implementation should not throw
exceptions that are not defined by the API and currently only
GSSException is defined.

Correction: This is not a bug in the JGSS API, it is (almost
certainly) a bug in the *Oracle / Sun implementation* of JGSS.

Mike

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Authentication with Browser stopped working / missing exception handling in getRemainingLifetime

[Michael B Allen]

On Mon, Nov 22, 2021 at 2:39 AM Thomas Hoffmann (Speed4Trade GmbH)
 wrote:
> Would it be better to also catch IllegalStateException and instead of 
> checking left == 0 to change it to left <= 0 ?

I would argue that this is a bug in JGSS. JGSS has been a comedy of
errors over the years. I thought it had mostly stabilized over the
last 5-10 years but this is a good example of the sort of bad behavior
from that lib. Throwing an IllegalStateException there is a bad API
choice. I have to wonder if that was not the designers intention. The
getRemainingLifetime API documentation does not say anything about it
throwing an IllegalStateException when your cred expires. You might
want to try the latest JRE if you're using something old. Or maybe
there's something screwy about the cred and it's tripping up an
unexpected code path. I assume you mean Kerberos and not LDAP BTW.

But I think the only real short term solution for now would be to
catch the IllegalStateException and just set left = 0.

Mike

-- 
Michael B Allen
Java Active Directory Integration
http://www.ioplex.com/

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: How to *properly* create and use a CATALINA_BASE installation

[Michael B Allen]

On Thu, Nov 18, 2021 at 12:06 PM Mark Eggers
 wrote:
> How well do mklinks work with services? I am not really a Windows
> administrator, so I have no idea.

I have no idea either as I have never used them together. But mklink
is a filesystem level feature AFAIK so I don't see why a Windows
service would know the difference between a link and the directory it
points to. If it did, that would make mklink pretty useless.

Mike

-- 
Michael B Allen
Java Active Directory Integration
http://www.ioplex.com/

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: How to *properly* create and use a CATALINA_BASE installation

[Michael B Allen]

On Wed, Nov 17, 2021 at 9:05 PM Mark Eggers
 wrote:
> CATALINA_HOME and CATALINA_BASE are links to an appropriate Tomcat
> installation, and one configured for that particular service.
>
> Then to upgrade to a new Tomcat, you do the following:
>
> 1. Unpack the new reference version of Tomcat somewhere which becomes
> CATALINA_HOME.
>
> 2. Create the new service-specific installation of Tomcat which becomes
> CATALINA_BASE.
>
> All of the above can be done without disturbing the existing service.
>
> To upgrade, do the following:
>
> 1. Shut down the service
> 2. Move the links
> 3. Start up the service
>
> If things blow up in your face, then the roll back is really easy:
>
> 1. Shut down the service
> 2. Restore the links
> 3. Start up the service

This makes me realize my proposed bin/run.sh method is not really
tuned for production. Indeed links could be used to great effect here.
Windows has mklink /d   which is essentially the same
as ln on *nix near as I can tell. Might help with issues like the
catalina.policy file path in the registry when using the Windows
service.

Mike

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: How to *properly* create and use a CATALINA_BASE installation

[Michael B Allen]

On Wed, Nov 17, 2021 at 11:04 AM  wrote:
> I, in my opinion, find it far easier to set my BASE in the setenv.sh for the 
> instance I'm using. As Chris said, you can have multiple instances (BASEs) on 
> a server.

Jon,

If you mean you're setting $CATALINA_BASE in setenv.sh, I don't think
that will work because when you run $CATALINA_BASE/bin/startup.sh,
$CATALINA_BASE won't be set yet and so it's going to set
$CATALINA_BASE to $CATALINA_HOME and then source the
$CATALINA_HOME/bin/setenv.sh and not the $CATALINA_BASE specific
setenv.sh. I think that's what Chris was referring to in his first
reply to you.

Although if I'm using a $CATALINA_BASE/bin/run.sh alternative to
startup.sh like in my previous post, I would probably just put ALL the
various environment variables (listed at the top of bin/catalina.sh)
in there too. Using setenv.sh at that point would just be needlessly
spreading config around.

The documentation currently seems to be targeting installations shared
by multiple different users where $CATALINA_HOME and $CATALINA_BASE
get set in the users environment such that different users can have
different bases but share a common installation like in /usr/local or
some such. Then you *would* want to put base specific stuff in
$CATALINA_BASE/bin/setenv.sh. But IMO that's kind of a late 90's way
of doing things. Nowadays, not only do people have their own machines,
but they have multiple instances in VMs and private servers and docker
and so on. So I think the self-contained $CATALINA_BASE/bin/run.sh
method is probably a little better for most cases (although I still
need to study the Windows service use-case which is probably
important).

Mike

-- 
Michael B Allen
Java Active Directory Integration
http://www.ioplex.com/

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: How to *properly* create and use a CATALINA_BASE installation

[Michael B Allen]

> https://people.apache.org/~schultz/presentations/ApacheCon%20NA%202020/Splitting%20Your%20Tomcat%20Installation.pdf

Thanks. That's what I'm looking for.

Slightly related: Instead of setting environment variables in your
profile and running $CATALINA_HOME/bin/startup.sh, is there any reason
why we should not create a $CATALINA_BASE/bin/startup.sh like the
following?

#!/bin/sh
export JRE_HOME=/path/to/jre/
export CATALINA_HOME=/path/to/tomcat/
export CATALINA_BASE=/path/to/tomcat-base/
${CATALINA_HOME}/bin/catalina.sh run "$@"

and use this instead?

It seems to me this would completely separate the installation from
the host and environment.

Mike

-- 
Michael B Allen
Java Active Directory Integration
http://www.ioplex.com/

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: How to *properly* create and use a CATALINA_BASE installation

[Michael B Allen]

On Mon, Nov 15, 2021 at 4:31 PM Christopher Schultz
 wrote:
> > conf
> > All of the conf files.
>
> Specifically, you'll want server.xml and web.xml. You can also choose to
> customize context.xml, and put any [engine]/[host]/[webapp].xml
> deployment descriptors there.

Hi chros,

Ok, so then the minimum required is server.xml and web.xml?

And then there are optional files like context.xml? Are all of the
other files also optional and completely overwrite settings like
tomcat-users.xml or only some?

I have confirmed that adding only web.xml alone resolves the issue
completely. I no longer get 404's and the application works. But I
would like to further understand this as much as possible to document
it for my customers.

Thanks,
Mike

-- 
Michael B Allen
Java Active Directory Integration
http://www.ioplex.com/

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

How to *properly* create and use a CATALINA_BASE installation

[Michael B Allen]

Hi,

What $CATALINA_HOME/conf/ files should be copied into $CATALINA_BASE?

RUNNING.txt just says:

* conf - Server configuration files (including server.xml)

So it's multiple fileS but not necessarily all if server.xml is
explicitly included?

Ideally I would think it should be only files that need to be modified
since that seems to be the point of using $CATALINA_BASE. Is this
correct?

I'm trying to use $CATALINA_BASE just because it seems like the proper
way to setup Tomcat in general.

Without $CATALINA_BASE everything works as near as I can tell.

But if I change $CATALINA_BASE to be different from $CATALINA_HOME in
my startup bat like:

$CATALINA_HOME/bin/xstart.bat:
SETLOCAL

set JRE_HOME=%ProgramFiles%\Java\jre1.8.0_311
set CATALINA_HOME=C:\path\to\tomcat
set CATALINA_BASE=C:\path\to\tomcat-base

"%CATALINA_HOME%\bin\catalina.bat" run %1 %2 %3 %4 %5 %6 %7 %8 %9

And then in tomcat-base I have:

bin\tomcat-juli.jar
conf\keystore.jks
conf\server.xml

The server.xml is stock except for the following:







conf\tomcat-users.xml
conf\Catalina\localhost\manager.xml
logs\localhost_access_log.2021-11-15.txt
temp\
webapps\myapp\

Note: There is no myapp\WEB-INF\context.xml

webapps\manager\

Tomcat starts up ok and Tomcat Manager works. I can see myapp in the
manager which claims it's deployed and running.

But trying to access /myapp results in:

404 Not Found: The origin server did not find a current
representation for the target resource or is not willing to
disclose that one exists.

I can un-deploy /myapp and re-deploy it through the manager and again,
nothing but 404.

Doesn't work under HTTPS either (and HTTPS works without using $CATALINA_BASE).

What could be the problem here?

I used the following to create a symbolic link to the tomcat directory:

cmd>mklink /d tomcat apache-tomcat-9.0.54

Is this ok?

I'm using Tomcat 9.0.54 32 bit on Windows Server 2016 64 bit. The
native runtime DLL fails to load because it's built for 32bit. But
this seems to fallback to the Java runtime just fine. Is this somehow
a problem?

Do I need a deployment context xml?

I'm a little stumped by this. I don't normally use Tomcat but I just
wanted to create an "Application Note" about how to properly use my
product with Tomcat. So I'm really interested in how this all is
supposed to work and not so much just seeing it work.

Thanks,

Mike
-- 
Michael B Allen
Java Active Directory Integration
http://www.ioplex.com/

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Mimicking two distinct virtual hosts just like in HTTPd

[Michael Osipov]

Am 2021-11-08 um 17:36 schrieb Mark Thomas:

On 08/11/2021 15:36, Michael Osipov wrote:

Folks,

consider the following in httpd.conf:

Listen {IP}:8443
Listen {IP}:8444

later:

   DocumentRoot /www/webapps1
   ServerName {hostname}
   mod_ssl config...


   DocumentRoot /www/webapps2
   ServerName {hostname}
   mod_ssl config2...


The second virtual host shall deliver only a subset of webapps1, but 
configured client cert auth to avoid issues with PHA and renotiation.


Now, I am looking for the same in Tomcat. Given that I have one 
Server, one Service, two Connectors one Engine and two Hosts there is 
no way to bind a Host in Tomcat to an listen address, but only to a 
hostname/IP address.
 From my understanding of the server.xml I would need set up *two* 
Service elements with one Engine, Connector and Host each.


Is my understanding correct?


Yes. Connector elements are associated with a Service so if you want 
different Hosts on different ports you need different Services.



Thanks, I already thought so. This is a solution I can happily live with.

Maybe this could be simplified in Tomcat 11 from an admin/developer 
point of view.


M

Mimicking two distinct virtual hosts just like in HTTPd

[Michael Osipov]

Folks,

consider the following in httpd.conf:

Listen {IP}:8443
Listen {IP}:8444

later:

  DocumentRoot /www/webapps1
  ServerName {hostname}
  mod_ssl config...


  DocumentRoot /www/webapps2
  ServerName {hostname}
  mod_ssl config2...


The second virtual host shall deliver only a subset of webapps1, but 
configured client cert auth to avoid issues with PHA and renotiation.


Now, I am looking for the same in Tomcat. Given that I have one Server, 
one Service, two Connectors one Engine and two Hosts there is no way to 
bind a Host in Tomcat to an listen address, but only to a hostname/IP 
address.
From my understanding of the server.xml I would need set up *two* 
Service elements with one Engine, Connector and Host each.


Is my understanding correct?

Michael

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: JASPIC Plugin for OIDC/JWT/OAuth

[Michael Kolenda]

Thanks Mark! Will take a look

On Fri, Oct 8, 2021, 5:01 AM Mark Thomas  wrote:

> On 07/10/2021 18:37, Michael Kolenda wrote:
> > Hey Tomcat Users,
> >
> > I've run into an interesting behavior with a custom JASPIC provider. When
> > there is an existing session i.e. JSESSIONID cookie, It appears the
> > groups/roles are not checked again... even when the new groups are
> provided
> > in the client Subject (JASPIC's validate() ). When attempting stateless
> > authentication via JWT/OAuth how can I ignore a previously set session
> for
> > an individual request?
> >
> > It appears to be based around equals() on my Principal object. I can make
> > it so Principal's generated via stateless authentication protocols are
> > never equal, but then I get a new session id in the response. I don't
> want
> > a session id at all for this request
>
> I'm only basing this on looking at Tomcat's source code so I may be on
> the wrong track.
>
> You probably want to set cache="false" on your authenticator. That will
> stop Tomcat trying to cache the authenticated principal in the session.
>
>  From your description and looking at the source for AuthenticatorBase,
> I think that should address the issue you are seeing.
>
> You might also want to check if alwaysUseSession has been set. If not,
> the default of false is fine but I don't think you want this set to true.
>
> Mark
>
> -
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org
>
>

JASPIC Plugin for OIDC/JWT/OAuth

[Michael Kolenda]

Hey Tomcat Users,

I've run into an interesting behavior with a custom JASPIC provider. When
there is an existing session i.e. JSESSIONID cookie, It appears the
groups/roles are not checked again... even when the new groups are provided
in the client Subject (JASPIC's validate() ). When attempting stateless
authentication via JWT/OAuth how can I ignore a previously set session for
an individual request?

It appears to be based around equals() on my Principal object. I can make
it so Principal's generated via stateless authentication protocols are
never equal, but then I get a new session id in the response. I don't want
a session id at all for this request

Any ideas?

Thanks,
Mike

Re: Specifying a Custom Authenticator Class

[Michael Osipov]

Am 2021-10-02 um 02:48 schrieb Jerry Malcolm:
I need to write a custom BasicAuthenticator class to decode a 
specialized encoding of the authToken.  I have been scouring google for 
info.  I found one post where the answer included the statement:


This would clearly violate Basic auth scheme and the according RFC. I 
highly recommend against. Don't abuse Basic. Create your own 
scheme/header and solve your problem with it.


M


-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: reporting a problem with LDAP auth to Windows Active Directory with Kerberos using the default spnegoDelegationQop="auth-conf" value for Tomcat 9.0.31 and 9.0.52

[Michael Osipov]

Am 2021-09-14 um 18:23 schrieb Tim Miller Dyck:

Hello,

I wanted to report an issue with Tomcat LDAP user authentication lookups with 
Tomcat container Kerberos security that I found in our environment when 
upgrading to version Tomcat 9.0.52 from 9.0.30 and what configuration settings 
bypassed the problem.

Here is our base pre-Tomcat upgrade configuration which was working for both 
LDAP user authentication and with Kerberos-enabled authentication (e.g. with 
the Tomcat Manager app).


   *   Tomcat 9.0.30 running on Windows Server 2019
   *   Microsoft OpenJDK packaging, build 11.0.12+7 64-bit
   *   Tomcat is authenticating users using LDAP queries to Windows Server 
2012R2 Active Directory controllers
   *   Tomcat server.xml LDAP configuration:

 


   *   We have some web apps that use Tomcat container Kerberos authentication, 
e.g. we configure the Tomcat Manager app WEB-INF\web.xml with


   

   

 

 SPNEGO

 Tomcat Manager Application

   

The same configuration was then upgraded to Tomcat 9.0.52 with otherwise all 
the same overall configuration in place (Kerberos settings, Java settings, cert 
settings, etc.).

The following problem then appeared when trying to log into the Manager app 
using SPNEGO/Kerberos (and also our web apps using Tomcat container Kerberos 
authentication).


From the Tomcat stderr file:



14-Sep-2021 07:00:32.196 SEVERE [https-jsse-nio-443-exec-10] 
org.apache.catalina.realm.JNDIRealm.getPrincipal Exception performing 
authentication

javax.naming.NamingException: LDAP connection has been closed; remaining name 
'dc=myorg,dc=com'

at java.naming/com.sun.jndi.ldap.LdapRequest.getReplyBer(LdapRequest.java:133)

at java.naming/com.sun.jndi.ldap.Connection.readReply(Connection.java:443)

at java.naming/com.sun.jndi.ldap.LdapClient.getSearchReply(LdapClient.java:639)

at java.naming/com.sun.jndi.ldap.LdapClient.search(LdapClient.java:562)

at java.naming/com.sun.jndi.ldap.LdapCtx.doSearch(LdapCtx.java:2014)

at java.naming/com.sun.jndi.ldap.LdapCtx.searchAux(LdapCtx.java:1873)

at java.naming/com.sun.jndi.ldap.LdapCtx.c_search(LdapCtx.java:1798)

at 
java.naming/com.sun.jndi.toolkit.ctx.ComponentDirContext.p_search(ComponentDirContext.java:392)

at 
java.naming/com.sun.jndi.toolkit.ctx.PartialCompositeDirContext.search(PartialCompositeDirContext.java:358)

at 
java.naming/com.sun.jndi.toolkit.ctx.PartialCompositeDirContext.search(PartialCompositeDirContext.java:341)

at 
java.naming/javax.naming.directory.InitialDirContext.search(InitialDirContext.java:267)

at org.apache.catalina.realm.JNDIRealm.getUserBySearch(JNDIRealm.java:1610)

at org.apache.catalina.realm.JNDIRealm.getUser(JNDIRealm.java:1447)

at org.apache.catalina.realm.JNDIRealm.getUser(JNDIRealm.java:1376)

at org.apache.catalina.realm.JNDIRealm.getPrincipal(JNDIRealm.java:2352)

at org.apache.catalina.realm.JNDIRealm.getPrincipal(JNDIRealm.java:2288)

at org.apache.catalina.realm.JNDIRealm.getPrincipal(JNDIRealm.java:2269)

at org.apache.catalina.realm.RealmBase.authenticate(RealmBase.java:504)

at org.apache.catalina.realm.CombinedRealm.authenticate(CombinedRealm.java:365)

at org.apache.catalina.realm.LockOutRealm.authenticate(LockOutRealm.java:194)

...

In testing with Tomcat ver. 9.0.52, I found the following:


   *   Using Manager with basic auth but still with LDAP user lookups worked - 
only Kerberos-enabled apps had problems with LDAP auth
   *   Switching our connectionURLs to use non-encrypted LDAP instead of LDAPS 
caused the problem to be resolved and LDAP+Kerberos to work again so it was 
something to do with use of SSL with LDAP (but we only want to use LDAPS, of 
course, for security)
   *   In the LDAP server.xml configuration, as per Tomcat documentation, these 
two settings are the defaults if not otherwise specified:

useDelegatedCredential="true"

spnegoDelegationQop="auth-conf "

  *   As per documentation, these settings cause a users's Kerberos 
credentials to be used for the LDAP lookup (if Kerberos credentials are 
available to Tomcat) instead of the specific LDAP connection username and 
password in the JNDIRealm configuration.
   *   I tried setting stripRealmForGss="false" (the default is true, where the 
domain is stripped from the username) - but has the same problem with both true and false 
settings
   *   I tried adding only the line spnegoDelegationQop="auth" and this caused 
the problem to be resolved with Kerberos+LDAP working again
  *   The options for spnegoDelegationQop are auth-conf (default, "authentication plus integrity and 
confidentiality protection"), auth-int ("authentication plus integrity") and auth 
("authentication only").
  *   I also tried auth-int, but this had the same 
"org.apache.catalina.realm.JNDIRealm.getPrincipal Exception performing 
authentication" error
   *   I then tested by adding only the line useDelegatedCredential="false" and 
this also caused the problem to be resolved with Kerberos+LDAP working again 

Re: JNDIRealm does not retry on read timeouts or closed connections

[Osipov, Michael (LDA IT PLM)]

Am 2021-09-06 um 11:21 schrieb Mark Thomas:

On 06/09/2021 09:52, Osipov, Michael (LDA IT PLM) wrote:



My question is: Mark, you have direct access to JBS, would you be 
willing to file this issue directly or do you want me to file through 
bugreport.java.com first and when it arrives in JBS you could drop a 
comment that this also affects Tomcat?


Best if you create it and I comment.


It is public: https://bugs.openjdk.java.net/projects/JDK/issues/JDK-8273402

and partially related 
https://bugs.openjdk.java.net/projects/JDK/issues/JDK-8273385 for error 
analysis purposes.


Michael

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: JNDIRealm does not retry on read timeouts or closed connections

[Osipov, Michael (LDA IT PLM)]

Am 2021-09-06 um 11:21 schrieb Mark Thomas:

On 06/09/2021 09:52, Osipov, Michael (LDA IT PLM) wrote:



My question is: Mark, you have direct access to JBS, would you be 
willing to file this issue directly or do you want me to file through 
bugreport.java.com first and when it arrives in JBS you could drop a 
comment that this also affects Tomcat?


Best if you create it and I comment.


Done. Internal review id: 9071381
Will wait until this goes public.

Michael

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

JNDIRealm does not retry on read timeouts or closed connections

[Osipov, Michael (LDA IT PLM)]

Hi,

disclaimer: this has not been explicitly verified on the JNDIRealm, but 
the code is very similar to my ActiveDirectoryRealm wich exposes the 
same behavior.


The code in question:

try {

// Authenticate the specified username if possible
principal = getPrincipal(connection, username, gssCredential);

} catch (CommunicationException | ServiceUnavailableException e) {
// log the exception so we know it's there.
containerLog.info(sm.getString("jndiRealm.exception.retry"), e);

// close the connection so we know it will be reopened.
close(connection);
closePooledConnections();

// open a new directory context.
connection = get();

// Try the authentication again.
principal = getPrincipal(connection, username, gssCredential);
}

// Release this context
release(connection);

// Return the authenticated Principal (if any)
return principal;

} catch (NamingException e) {
// Log the problem for posterity
containerLog.error(sm.getString("jndiRealm.exception"), e);

// close the connection so we know it will be reopened.
close(connection);
closePooledConnections();

// Return "not authenticated" for this request
return null;
}


The connection has been verified to be valid, getPrincipal() is 
performed, but the LDAP request can still fail in flight. Therefore 
CommunicationException and ServiceUnavailableException shall handle this 
case, drop the connection, open a new one and retry. This does not happen.

The issue is rooted in Sun's JDNI code. From LdapRequest#getReplyBer(long):


// poll from 'replies' blocking queue ended-up with timeout
if (result == null) {
throw new NamingException(String.format(TIMEOUT_MSG_FMT, millis));
}
// Unexpected EOF can be caused by connection closure or cancellation
if (result == EOF) {
throw new NamingException(CLOSE_MSG);
}


Both are actually communication issues which -- for some strange reason 
-- do not make use of CommunicationException. Therefore, the retry never 
happens also this is a valid request.

(I can provide a verbose log if necessary).

I see two ways to solve this:
1. Short term: catch NamingException and parse out those two messages:
  * LDAP connection has been closed
  * LDAP response read timed out
2. Report a bug/RFE with Java to use new CommunicationException instead 
of NamingException. I believe this can easily be backported to Java 8 
and 11 since the external signature does not change.


My question is: Mark, you have direct access to JBS, would you be 
willing to file this issue directly or do you want me to file through 
bugreport.java.com first and when it arrives in JBS you could drop a 
comment that this also affects Tomcat?


Michael

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: how to tune cacheMaxSize

[Michael Richardson]

Christopher Schultz  wrote:
>> NOTE: Picked up JDK_JAVA_OPTIONS:  
--add-opens=java.base/java.lang=ALL-UNNAMED 
--add-opens=java.base/java.io=ALL-UNNAMED 
--add-opens=java.base/java.util=ALL-UNNAMED --add-opens=java.ba
>> se/java.util.concurrent=ALL-UNNAMED 
--add-opens=java.rmi/sun.rmi.transport=ALL-UNNAMED
>> ^Z
>> [1]+  Stopped $CATALINA_HOME/bin/catalina.sh run

> How long did you wait before hitting CTRL-Z there?

at least 5 minutes.

>> tomcat@oscar-serv03:/opt/apache-tomcat-9.0.52$ bg
>> [1]+ $CATALINA_HOME/bin/catalina.sh run &

> Well, now things are complicated because you backgrounded the process.

Well, either it tries to write to stdout and gets blocked (and suspended) on
SIGTOUT, or it doesn't.
Background processes still get to write to stdout by default.

--
]   Never tell me the odds! | ipv6 mesh networks [
]   Michael Richardson, Sandelman Software Works|IoT architect   [
] m...@sandelman.ca  http://www.sandelman.ca/|   ruby on rails[



signature.asc
Description: PGP signature

Re: how to tune cacheMaxSize

[Michael Richardson]

Christopher Schultz  wrote:
> That's quite a long time (1643381ms), and the timestamps match-up.

> So you tried to get a thread-dump during this 27-minute period and the
> process simply wouldn't respond?

Correct.

> It is during this 27-minute period that the CPU is pegged?

Correct.

> Do you see anything in the kernel logs? Like dmesg.log or kernel.log or
> whatever Ubuntu logs to?

Nothing.

> Well, the application deployment(s) are happening somewhat quickly. 13
> seconds is a long time IMO to wait for manager and the other rather 
trivial
> web applications to load, but that points to something more like a slow 
disk
> or CPU or something.

I can investigate this a bit.
I spent several days this week cleaning out a data center, so some different
hardware has come my way.  Perhaps I'll try that out.



signature.asc
Description: PGP signature

Re: how to tune cacheMaxSize

[Michael Richardson]

Let's deploy the oscar.war, what's the worst that can happen?

20-Aug-2021 01:36:10.129 WARNING [Catalina-utility-1] 
org.apache.catalina.webresources.Cache.getResource Unable to add the resource 
at [/WEB-INF/classes/oscar/ocan/testdata/ocan_v1.0.1.xml] to the cache for web 
application [/oscar] because there was insufficient free space available after 
evicting expired cache entries - consider increasing the maximum size of the 
cache
... repeated many times.

This was among the original message that lead to this original subject line.

--
]   Never tell me the odds! | ipv6 mesh networks [
]   Michael Richardson, Sandelman Software Works|IoT architect   [
] m...@sandelman.ca  http://www.sandelman.ca/|   ruby on rails[



-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: how to tune cacheMaxSize

[Michael Richardson]

Aha.  Well, I left it running after the last email and went on to more
important things.  Then the window just jumped:

The previous log line:
20-Aug-2021 01:02:42.315 INFO [main] 
org.apache.catalina.core.AprLifecycleListener.initializeSSL OpenSSL 
successfully initialized [OpenSSL 1.1.1  11 Sep 2018]

Then it acted on the ^\ I had sent.  And then it printed:
Could it have spent that time initializing OpenSSL?


20-Aug-2021 01:29:58.867 INFO [main] org.apache.coyote.AbstractProtocol.init 
Initializing ProtocolHandler ["http-nio-8080"]
20-Aug-2021 01:29:59.917 INFO [main] org.apache.catalina.startup.Catalina.load 
Server initialization in [1643381] milliseconds
20-Aug-2021 01:30:01.832 INFO [main] 
org.apache.catalina.core.StandardService.startInternal Starting service 
[Catalina]
20-Aug-2021 01:30:01.837 INFO [main] 
org.apache.catalina.core.StandardEngine.startInternal Starting Servlet engine: 
[Apache Tomcat/9.0.52]
20-Aug-2021 01:30:01.997 INFO [main] 
org.apache.catalina.startup.HostConfig.deployDirectory Deploying web 
application directory [/opt/apache-tomcat-9.0.52/webapps/manager]
20-Aug-2021 01:30:06.559 WARNING [main] 
org.apache.catalina.util.SessionIdGeneratorBase.createSecureRandom Creation of 
SecureRandom instance for session ID generation using [SHA1PRNG] took [248] 
milliseconds.
20-Aug-2021 01:30:07.109 INFO [main] 
org.apache.catalina.startup.HostConfig.deployDirectory Deployment of web 
application directory [/opt/apache-tomcat-9.0.52/webapps/manager] has finished 
in [5,111] ms
20-Aug-2021 01:30:07.117 INFO [main] 
org.apache.catalina.startup.HostConfig.deployDirectory Deploying web 
application directory [/opt/apache-tomcat-9.0.52/webapps/examples]
20-Aug-2021 01:30:12.192 INFO [main] 
org.apache.catalina.startup.HostConfig.deployDirectory Deployment of web 
application directory [/opt/apache-tomcat-9.0.52/webapps/examples] has finished 
in [5,074] ms
20-Aug-2021 01:30:12.195 INFO [main] 
org.apache.catalina.startup.HostConfig.deployDirectory Deploying web 
application directory [/opt/apache-tomcat-9.0.52/webapps/docs]
20-Aug-2021 01:30:12.449 INFO [main] 
org.apache.catalina.startup.HostConfig.deployDirectory Deployment of web 
application directory [/opt/apache-tomcat-9.0.52/webapps/docs] has finished in 
[253] ms
20-Aug-2021 01:30:12.451 INFO [main] 
org.apache.catalina.startup.HostConfig.deployDirectory Deploying web 
application directory [/opt/apache-tomcat-9.0.52/webapps/ROOT]
20-Aug-2021 01:30:12.697 INFO [main] 
org.apache.catalina.startup.HostConfig.deployDirectory Deployment of web 
application directory [/opt/apache-tomcat-9.0.52/webapps/ROOT] has finished in 
[245] ms
20-Aug-2021 01:30:12.700 INFO [main] 
org.apache.catalina.startup.HostConfig.deployDirectory Deploying web 
application directory [/opt/apache-tomcat-9.0.52/webapps/host-manager]
20-Aug-2021 01:30:12.990 INFO [main] 
org.apache.catalina.startup.HostConfig.deployDirectory Deployment of web 
application directory [/opt/apache-tomcat-9.0.52/webapps/host-manager] has 
finished in [290] ms
20-Aug-2021 01:30:13.095 INFO [main] org.apache.coyote.AbstractProtocol.start 
Starting ProtocolHandler ["http-nio-8080"]
20-Aug-2021 01:30:13.254 INFO [main] org.apache.catalina.startup.Catalina.start 
Server startup in [1] milliseconds

--
]   Never tell me the odds! | ipv6 mesh networks [
]   Michael Richardson, Sandelman Software Works|IoT architect   [
] m...@sandelman.ca  http://www.sandelman.ca/|   ruby on rails[



signature.asc
Description: PGP signature

Re: how to tune cacheMaxSize

[Michael Richardson]

/usr/lib/jvm/java-11-openjdk-amd64/bin/java 
-Djava.util.logging.config.file=/opt/apache-tomcat-9.0.52/conf/logging.properties

No logs.
^\ nothing.
^C nothing.

Needs yee-old kill-9 to exit.

--
]   Never tell me the odds! | ipv6 mesh networks [
]   Michael Richardson, Sandelman Software Works|IoT architect   [
] m...@sandelman.ca  http://www.sandelman.ca/|   ruby on rails[



signature.asc
Description: PGP signature

Re: how to tune cacheMaxSize

[Michael Richardson]

Christopher Schultz  wrote:
> Not at all. EC2 is entirely reasonable for such purposes. Amazon will
> even grant you a signed BAA if you ask for one.

Canada is not the US, and OHIP has rules differently than others.
And the clinic would like to survive having our third-world telco go down.

>> No, I'm not building myself.  There is an oscar.deb, which btw,
>> results in the same 100%.

> Okay, good. Does OSCAR distribute the .deb file? I'm ... weirdly
> surprised by that.

OSCAR has a .deb that tries to be all singing, all dancing.
It's not the only solution.

>> It also does other stuff like configure mariadb locally (which I don't
>> want).

> Interesting. You want control over your own database, or you don't want
> the db to be local (e.g. you want a remote db)?

Exactly.

>> This is the only time I've seen this error, I've also used the manager
>> web application before.

> Hmm. Same error there? Or same 100% CPU circumstances there?

Same 100% CPU.

>> I tried that.  It says something about there being no socket open.  My
>> impression is that "jstack" was an Oracle Java only tool.

> Nope, it's in all of the OpenJDK builds. You might need a "JDK" instead
> of a"JRE".

okay, I'll try again and post the result.

> When you issue "kill -3", the thread dump should appear on stdout,
> which should be captured in logs/catalina.[date].log. If that's not
> happening, then I'm a bit confused...

It's not happening.

> Okay, try this:

> 1. Stop Tomcat, clear all logs, delete your oscar.war file and the
> exploded directory in CATALINA_BASE/webapps/oscar (or wherever your
> appBase points to).  2. Copy your oscar.war file into appBase, making
> sure that operation completes 3. Start Tomcat, but like this instead of
> what you usually do:

>$ sudo -iu tomcatuser $ $CATALINA_BASE/bin/catalina.sh run

> That "run" is important: it will run Tomcat in the current console
> instead of in the background as a service. You'll get stdout directly
> on your console, no log files to worry about.

okay.

--
]   Never tell me the odds! | ipv6 mesh networks [
]   Michael Richardson, Sandelman Software Works|IoT architect   [
] m...@sandelman.ca  http://www.sandelman.ca/|   ruby on rails[



signature.asc
Description: PGP signature

Re: trying to deploy oscar (was Re: how to tune cacheMaxSize )

[Michael Richardson]

I grabbed apache-tomcat-9.0.52.tar.gz from
https://downloads.apache.org/tomcat/tomcat-9/v9.0.52/bin/apache-tomcat-9.0.52.tar.gz
I unpacked it to /opt/apache-tomcat-9.0.52.tar.gz.
I uninstalled tomcat9 (purged, and rm -rf /var/lib/tomcat/webapps)
I tweaked CATALINA_HOME and JRE_HOME appropriately.

I ran $CATALINA_HOME/bin/start.sh.  As root, which clearly was a mistake.
I thought it would su, but clearly I should start it as mortal user.
Perhaps there is a better script, but clearly the tomcat package took care of 
that.
(I wouldn't want to run it as root in production at all)

Before fixing that mistake, I notice:

  PID USER  PRI  NI  VIRT   RES   SHR S CPU% MEM%   TIME+  Command
 6246 root   20   0 3417M 37392 17752 S 201.  0.9 22:34.04 /usr/bin/java 
-Djava.util.logging.config.file=/opt/apache-tomcat-9.0.52/conf/logging.properties
 -Djava.util.logging.manag
 6252 root   20   0 3417M 37392 17752 R 100.  0.9 11:10.02 /usr/bin/java 
-Djava.util.logging.config.file=/opt/apache-tomcat-9.0.52/conf/logging.properties
 -Djava.util.logging.manag
 6247 root   20   0 3417M 37392 17752 R 100.  0.9 11:15.75 /usr/bin/java 
-Djava.util.logging.config.file=/opt/apache-tomcat-9.0.52/conf/logging.properties
 -Djava.util.logging.manag


oscar-serv03-[~] root 55 #ls -l /opt/apache-tomcat-9.0.52/logs
total 0
-rw-r- 1 root root 0 Aug 16 20:34 catalina.2021-08-16.log
-rw-r- 1 root root 0 Aug 16 20:34 catalina.out
-rw-r- 1 root root 0 Aug 16 20:34 host-manager.2021-08-16.log
-rw-r- 1 root root 0 Aug 16 20:34 localhost.2021-08-16.log
-rw-r- 1 root root 0 Aug 16 20:34 manager.2021-08-16.log

I've done nothing.  No deploy.  Of course, the manager is included, so
perhaps it is compiling or deploying that?

Port 8080 not open:

oscar-serv03-[~] root 56 #netstat -tan
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address   Foreign Address State
tcp0  0 127.0.0.53:53   0.0.0.0:*   LISTEN
tcp0  0 0.0.0.0:22  0.0.0.0:*   LISTEN
tcp0  0 192.168.2.13:22 192.168.2.3:56962   ESTABLISHED
tcp0  0 192.168.2.13:22 192.168.2.3:56968   ESTABLISHED
tcp6   0  0 :::22   :::*LISTEN

oscar-serv03-[~] mcr 10002 %free
  totalusedfree  shared  buff/cache   available
Mem:4030104  169540 2835356 792 1025208 3603084
Swap:   1048572   0 1048572

So, 169M of ~4G used.  No swapping.  Lots of ram free.

--
]   Never tell me the odds! | ipv6 mesh networks [
]   Michael Richardson, Sandelman Software Works|IoT architect   [
] m...@sandelman.ca  http://www.sandelman.ca/|   ruby on rails[



signature.asc
Description: PGP signature

Re: how to tune cacheMaxSize

[Michael Richardson]

ished
>> deploying this war.
>>
>> I've tried kill-3 repeatedly, on the main PID of the process, and
>> gotten no output.

> Anything in the other log files?

Nope.

> Instead of kill -3, what if you use the "jstack" utility and provide
> the PID of the running process? (You'll want to provide the PID of the
> parent JVM process, not a specific thread.)

I tried that.  It says something about there being no socket open.
My impression is that "jstack" was an Oracle Java only tool.

--
]   Never tell me the odds! | ipv6 mesh networks [
]   Michael Richardson, Sandelman Software Works|IoT architect   [
] m...@sandelman.ca  http://www.sandelman.ca/|   ruby on rails[




signature.asc
Description: PGP signature

trying to deploy oscar (was Re: how to tune cacheMaxSize )

[Michael Richardson]

<#secure method=pgpmime mode=sign>

Extracts of log, full log at URL below. 18k.

14-Aug-2021 17:16:43.821 INFO [Catalina-utility-1] 
org.apache.catalina.startup.HostConfig.deployWAR Deploying web application 
archive [/var/lib/tomcat9/webapps/oscar.war]
14-Aug-2021 17:16:44.005 SEVERE [Catalina-utility-1] 
org.apache.catalina.startup.ContextConfig.beforeStart Exception fixing docBase 
for context [/oscar]
 java.util.zip.ZipException: zip END header not found
 ...

14-Aug-2021 17:16:45.464 SEVERE [Catalina-utility-1] 
org.apache.catalina.startup.HostConfig.deployWAR Error deploying web 
application archive [/var/lib/tomcat9/webapps/oscar.war]
 java.lang.IllegalStateException: Error starting child
at 
org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:716)
...

14-Aug-2021 17:16:45.506 INFO [Catalina-utility-1] 
org.apache.catalina.startup.HostConfig.deployWAR Deployment of web application 
archive [/var/lib/tomcat9/webapps/oscar.war] has finished in [1,684] ms
14-Aug-2021 17:16:55.661 INFO [Catalina-utility-2] 
org.apache.catalina.startup.HostConfig.undeploy Undeploying context [/oscar]
14-Aug-2021 17:16:58.286 INFO [Catalina-utility-2] 
org.apache.catalina.startup.HostConfig.deployWAR Deploying web application 
archive [/var/lib/tomcat9/webapps/oscar.war]

My guess is that the cp oscar.war /var/lib/tomcat9/webapps hadn't actually
finished yet when it tried to deploy it.  So the first 1.684ms attempt fails,
and then it tried again. (Times are UTC)

oscar-serv03-[~] root 30 #netstat -tan
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address   Foreign Address State
tcp0  0 127.0.0.53:53   0.0.0.0:*   LISTEN
tcp0  0 0.0.0.0:22  0.0.0.0:*   LISTEN
tcp0  0 127.0.0.1:3306  0.0.0.0:*   LISTEN
tcp0  0 192.168.2.13:22 192.168.2.3:56896   ESTABLISHED
tcp0  0 192.168.2.13:22 192.168.2.3:56898   ESTABLISHED
tcp6   0  0 :::22   :::*LISTEN
tcp6   0  0 :::8080 :::*LISTEN

So the port is open, so I attempt to visit the /manage  URL

oscar-serv03-[~] root 31 #netstat -tan
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address   Foreign Address State
tcp0  0 127.0.0.53:53   0.0.0.0:*   LISTEN
tcp0  0 0.0.0.0:22  0.0.0.0:*   LISTEN
tcp0  0 127.0.0.1:3306  0.0.0.0:*   LISTEN
tcp0  0 192.168.2.13:22 192.168.2.3:56896   ESTABLISHED
tcp0  0 192.168.2.13:22 192.168.2.3:56898   ESTABLISHED
tcp6   0  0 :::22   :::*LISTEN
tcp6   0  0 :::8080 :::*LISTEN
tcp6 559  0 192.168.2.9:8080192.168.2.3:37486   ESTABLISHED

And it just stalls.

I went to eat lunch and watch TV, so waited a few hours, and it's the same
afterwards.  Note that it never told me that it had finished deploying this
war.

I've tried kill-3 repeatedly, on the main PID of the process, and gotten no 
output.

Complete log is now at:
  https://www.sandelman.ca/tmp/terapia9/catalina.2021-08-14.log-2


--
]   Never tell me the odds! | ipv6 mesh networks [
]   Michael Richardson, Sandelman Software Works|IoT architect   [
] m...@sandelman.ca  http://www.sandelman.ca/|   ruby on rails[


-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: how to tune cacheMaxSize

[Michael Richardson]

This time, after apt-get purge tomcat9 and re-install, I seem to be in 100%
CPU before I've deployed oscar or even updated the tomcat-users.xml to
enable the manager login.

HTOP view attached at bottom.
(4 virtual CPUs, 4G of ram. KVM VM running ubuntu 18.04, on a server running
ubuntu core 20)

oscar-serv03-[~] root 276 #netstat -tan
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address   Foreign Address State
tcp0  0 127.0.0.53:53   0.0.0.0:*   LISTEN
tcp0  0 0.0.0.0:22  0.0.0.0:*   LISTEN
tcp0  0 127.0.0.1:3306  0.0.0.0:*   LISTEN
tcp0  0 192.168.2.13:22 192.168.2.3:56874   ESTABLISHED
tcp0  0 192.168.2.13:22 192.168.2.3:56556   ESTABLISHED
tcp6   2  0 :::8080 :::*LISTEN
tcp6   0  0 :::22   :::*LISTEN
tcp6 747  0 192.168.2.9:8080192.168.2.3:37450   CLOSE_WAIT
tcp6 489  0 192.168.2.9:8080192.168.2.3:37452   ESTABLISHED

I find it weird that I have no catalina.out file:

oscar-serv03-[~] root 274 #ls -l /var/log/tomcat9
total 8
-rw-r- 1 tomcat tomcat 5233 Aug 14 16:38 catalina.2021-08-14.log
-rw-r- 1 tomcat tomcat0 Aug 14 16:37 localhost.2021-08-14.log
-rw-r- 1 tomcat tomcat0 Aug 14 16:38 localhost_access_log.2021-08-14.txt

I include cataline.2021-08-14.log here.
Also now at: https://www.sandelman.ca/tmp/terapia9/catalina.2021-08-14.log

ubuntu says that I have a new kernel and I should reboot, so I'll do another
purge, reboot, and then reinstall again.

14-Aug-2021 16:38:02.279 INFO [main] 
org.apache.catalina.startup.VersionLoggerListener.log Server version name:   
Apache Tomcat/9.0.16 (Ubuntu)
14-Aug-2021 16:38:02.302 INFO [main] 
org.apache.catalina.startup.VersionLoggerListener.log Server built:  
Sep 11 2019 19:47:51 UTC
14-Aug-2021 16:38:02.320 INFO [main] 
org.apache.catalina.startup.VersionLoggerListener.log Server version number: 
9.0.16.0
14-Aug-2021 16:38:02.343 INFO [main] 
org.apache.catalina.startup.VersionLoggerListener.log OS Name:   
Linux
14-Aug-2021 16:38:02.345 INFO [main] 
org.apache.catalina.startup.VersionLoggerListener.log OS Version:
5.4.0-80-generic
14-Aug-2021 16:38:02.348 INFO [main] 
org.apache.catalina.startup.VersionLoggerListener.log Architecture:  
amd64
14-Aug-2021 16:38:02.374 INFO [main] 
org.apache.catalina.startup.VersionLoggerListener.log Java Home: 
/usr/lib/jvm/java-11-openjdk-amd64
14-Aug-2021 16:38:02.380 INFO [main] 
org.apache.catalina.startup.VersionLoggerListener.log JVM Version:   
11.0.11+9-Ubuntu-0ubuntu2.18.04
14-Aug-2021 16:38:02.382 INFO [main] 
org.apache.catalina.startup.VersionLoggerListener.log JVM Vendor:
Ubuntu
14-Aug-2021 16:38:02.384 INFO [main] 
org.apache.catalina.startup.VersionLoggerListener.log CATALINA_BASE: 
/var/lib/tomcat9
14-Aug-2021 16:38:02.385 INFO [main] 
org.apache.catalina.startup.VersionLoggerListener.log CATALINA_HOME: 
/usr/share/tomcat9
14-Aug-2021 16:38:02.472 INFO [main] 
org.apache.catalina.startup.VersionLoggerListener.log Command line argument: 
--add-opens=java.base/java.lang=ALL-UNNAMED
14-Aug-2021 16:38:02.478 INFO [main] 
org.apache.catalina.startup.VersionLoggerListener.log Command line argument: 
--add-opens=java.base/java.io=ALL-UNNAMED
14-Aug-2021 16:38:02.499 INFO [main] 
org.apache.catalina.startup.VersionLoggerListener.log Command line argument: 
--add-opens=java.rmi/sun.rmi.transport=ALL-UNNAMED
14-Aug-2021 16:38:02.501 INFO [main] 
org.apache.catalina.startup.VersionLoggerListener.log Command line argument: 
-Djava.util.logging.config.file=/var/lib/tomcat9/conf/logging.properties
14-Aug-2021 16:38:02.504 INFO [main] 
org.apache.catalina.startup.VersionLoggerListener.log Command line argument: 
-Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager
14-Aug-2021 16:38:02.509 INFO [main] 
org.apache.catalina.startup.VersionLoggerListener.log Command line argument: 
-Djava.awt.headless=true
14-Aug-2021 16:38:02.512 INFO [main] 
org.apache.catalina.startup.VersionLoggerListener.log Command line argument: 
-XX:+UseG1GC
14-Aug-2021 16:38:02.513 INFO [main] 
org.apache.catalina.startup.VersionLoggerListener.log Command line argument: 
-Djdk.tls.ephemeralDHKeySize=2048
14-Aug-2021 16:38:02.529 INFO [main] 
org.apache.catalina.startup.VersionLoggerListener.log Command line argument: 
-Djava.protocol.handler.pkgs=org.apache.catalina.webresources
14-Aug-2021 16:38:02.531 INFO [main] 
org.apache.catalina.startup.VersionLoggerListener.log Command line argument: 
-Dorg.apache.catalina.security.SecurityListener.UMASK=0027
14-Aug-2021 16:38:02.532 INFO [main] 
org.apache.catalina.startup.VersionLoggerListener.log Command line argument: 
-Dignore.endorsed.dirs=
14-Aug-2021 16:38:02.534 

Re: how to tune cacheMaxSize

[Michael Richardson]

Thank you for the reply.

Christopher Schultz  wrote:
> On 8/12/21 11:05, Michael Richardson wrote:
>> I am trying to deploy OSCAR-EMR

> Wow, that still exists? I remember more than a decade ago being asked
> to integrate a product at $work with that thing as a demo. We never
> did, because the market seemed not to really exist. I see you are in
> Canada. I know OSCAR same from McMaster University. Does it have a good
> ecosystem and install base in CA?

Yes, it's still quite popular among many Ontario (and I'm told BC) doctors,
because it has good integration with the provincial billing system.  Doctor
like it because the price appears right, and doctors don't get rich by
spending money.

There are a number of support companies that install it, but my experience
(via my wife) is that their do a rather poor job of installation.
No understanding of database encryption, replication, or even how to enable 
https.

There seems to be developer still working on it, according to the bitbucket
git commit lot.  But, the public community around it seems to have died, as
far as I can tell.

>> I then deploy the oscar.war to /var/lib/tomcat19/webapps, and observe
>> in htop that I get 5-6 threads that get very busy, as it extracts the
>> app.

> Hmm. I wouldn't expect more than 1 thread to get busy extracting that
> application.

Sure, that's what I figured too.

>> If I stop the java processes, and restart them, then they seem to
>> continue the "deploy", even if I have removed the oscar.war and
>> /var/lib/tomcat19/webcaps/oscar directory.  Clearly something else
>> remembers it.  When that happens, I get *no output* in catalina.out.

> What makes you think those threads are continuing to deploy the WAR
> file? If the WAR file is missing, deployment would be most
> difficult. When you remove the WAR file, are you also removing the
> exploded-WAR-directory that Tomcat creates during the deployment of the
> WAR?

Yes, I also remove the exploded WAR-directory, and also the oscar.properties.

>> (I initially suspected the whole /dev/random issue, so I moved the TLS
>> to a proxy-pass Apache, but also I have virtio /dev/random, and I also
>> ln /dev/urandom to /dev/random just in case...)

> Okay, I don't mean to sound mean, but it sounds like you have no idea
> what's going on, here. There are probably several issues, but let's
> start with the first one: what are those threads doing at 100% CPU.

I am not a JAVA/Tomcat guy.
I do Rails for 15 years, postgres, maintain tcpdump and IPsec, and still
occasionally debug kernel issues.  But, mostly I write and implement IoT
security RFCs.

> You don't have a /dev/random problem, because really nobody has that
> problem anymore. Also, the /dev/random problem manifests as a "hung"
> server with *zero* CPU usage.

Yes, I noticed that egd setting is no more.  I mention this to be clear that
I'd debugged that.

> Reset your environment back to the beginning: fresh Tomcat, no log
> files, no WAR deployed or anything like that. Start up Tomcat and make
> sure everything looks good so far. Then deploy the WAR file and let
> everything go to hell. After things settle down and you are in your "2
> threads eating the CPU" situation, do the following:

> 1. Grab the log file in CATALINA_BASE/logs/catalina.out (or similar)
> 2. Take a thread-dump of the running JVM[1]

Before doing the purge, I have tried kill -3 as instructed, and gotten
nothing (no output to catalina.out).

I am resetting now, and I'll kill -3 just after the war deploy.

--
]   Never tell me the odds! | ipv6 mesh networks [
]   Michael Richardson, Sandelman Software Works|IoT architect   [
] m...@sandelman.ca  http://www.sandelman.ca/|   ruby on rails[





signature.asc
Description: PGP signature

Re: how to tune cacheMaxSize

[Michael Richardson]

Michael Richardson  wrote:
> Then, port 8080 stops responding.  And also I see new connections just
> stall.

to be clear, this is what I mean:

oscar-serv03-[~] root 213 #netstat -tan
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address   Foreign Address State
tcp0  0 127.0.0.53:53   0.0.0.0:*   LISTEN
tcp0  0 0.0.0.0:22  0.0.0.0:*   LISTEN
tcp0  0 127.0.0.1:3306  0.0.0.0:*   LISTEN
tcp0  0 192.168.2.13:22 192.168.2.3:56556   ESTABLISHED
tcp6   1  0 :::8080 :::*LISTEN
tcp6   0  0 :::22   :::*LISTEN
tcp6 699  0 192.168.2.9:8080192.168.2.3:37404   ESTABLISHED

{192.168.2.3:37404 is an ssh port forward from my desktop.
I wish stupid ISP had IPv6}

The process is clearly too busy to bother calling accept(2).


htop says:

  PID USER  PRI  NI  VIRT   RES   SHR S CPU% MEM%   TIME+  Command
11266 tomcat 20   0 3746M  169M 21252 S 201.  4.3  7:31.82 
/usr/lib/jvm/java-8-openjdk-amd64/bin/java 
-Djava.util.logging.config.file=/var/lib/tomcat9/conf/logging.propert
11283 tomcat 20   0 3746M  169M 21252 R 100.  4.3  3:13.27 
/usr/lib/jvm/java-8-openjdk-amd64/bin/java 
-Djava.util.logging.config.file=/var/lib/tomcat9/conf/logging.propert
11285 tomcat 20   0 3746M  169M 21252 R 100.  4.3  3:13.77 
/usr/lib/jvm/java-8-openjdk-amd64/bin/java 
-Djava.util.logging.config.file=/var/lib/tomcat9/conf/logging.propert
...and a bunch of 0% threads.

oscar-serv03-[~] root 218 #strace -p 11266
strace: Process 11266 attached
futex(0x7fa28312b9d0, FUTEX_WAIT, 11273, NULL^Cstrace: Process 11266 detached
 
oscar-serv03-[~] root 219 #strace -p 11283 2>&1 | sed 10q
strace: Process 11283 attached
sched_yield()   = 0
sched_yield()   = 0
sched_yield()   = 0
sched_yield()   = 0
sched_yield()   = 0
sched_yield()   = 0
sched_yield()   = 0
sched_yield()   = 0
sched_yield()   = 0
...forever...
oscar-serv03-[~] root 220 #strace -p 11285 2>&1 | sed 10q
strace: Process 11285 attached
strace: [ Process PID=11285 runs in x32 mode. ]
^C

I guess some JIT codes to 32-bit rather than amd64 codes?

--
]   Never tell me the odds! | ipv6 mesh networks [
]   Michael Richardson, Sandelman Software Works|IoT architect   [
] m...@sandelman.ca  http://www.sandelman.ca/|   ruby on rails[



signature.asc
Description: PGP signature

how to tune cacheMaxSize

[Michael Richardson]

I am trying to deploy OSCAR-EMR in a new location (for my wife's office),
with better encryption and database replication.  We have an existing
bare-metal instance running that her boss assembled. It's fragile, but it
runs.

In both ubuntu 18.04 and devuan beowulf VMs, I can start up tomcat9
and openjdk (8 or 11) and talk to the tomcat9-admin interface.

I then deploy the oscar.war to /var/lib/tomcat19/webapps, and observe
in htop that I get 5-6 threads that get very busy, as it extracts the app.

Then, port 8080 stops responding.  And also I see new connections just stall.

I wind up with two threads of java each chewing 100% of the CPU.
strace on it doesn't help, telling me that it looks like 32-bit code.
I think that the JIT might confuse it.  Are these threads doing I/O or system
calls? I can't tell.

(I can see the new connections stalling in netstat -tan, with unprocessed
bytes in the recv Q)

If I stop the java processes, and restart them, then they seem to continue
the "deploy", even if I have removed the oscar.war and
/var/lib/tomcat19/webcaps/oscar directory.  Clearly something else remembers
it.  When that happens, I get *no output* in catalina.out.

And _apt-get purge tomcat9_, and re-install, gets me back to the start, but I'd
like to understand where the deploy is being recorded and how I can undo only
that.

I will get one message that I am trying to look at the cacheMaxSize
value in /etc/tomcat9/context.xml.
  

My question is, if the cacheMaxSize is too small, will it lead to some kind
of continual cache replacement where it never actually starts up?
If so, is there some debug that would let me estimate what an appropriate
cache size is?  The running machine has:
   

(I initially suspected the whole /dev/random issue, so I moved the TLS
to a proxy-pass Apache, but also I have virtio /dev/random, and I also
ln /dev/urandom to /dev/random just in case...)

--
]   Never tell me the odds! | ipv6 mesh networks [
]   Michael Richardson, Sandelman Software Works|IoT architect   [
] m...@sandelman.ca  http://www.sandelman.ca/|   ruby on rails[




signature.asc
Description: PGP signature

Re: Updating Oracle JDK or JRE from 1.8.0-1.8.0_291 to 1.8.0-1.8.0_301, Breaks Apache Tomcat 9 Connector Http11AprProtocol

[Michael Osipov]

https://bugs.openjdk.java.net/browse/JDK-8202837
https://www.oracle.com/java/technologies/javase/8u301-bugfixes.html

What now?

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Enhancement: Additional user attributes queried by (some) realms

[Michael Osipov]

You read my mind. I always wanted to report this, but never find time. 
Implemented this for our realm and principal years ago:

http://tomcatspnegoad.sourceforge.net/apidocs/net/sf/michaelo/tomcat/realm/ActiveDirectoryPrincipal.html#getAdditionalAttributes--

The entire principal should be immutable, thus return only read-only 
objects. Always.


-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

AJAX value %27 results in 403 forbidden

[Michael Duffy]

I need help from the very capable and experienced IT professionals on this
list.

Can you help solve this problem:

https://stackoverflow.com/questions/66715576/ajax-value-27-results-in-403-forbidden

Thx!

Re: HTTP thread pool size remained high after database failover

[Michael Osipov]

Am 2020-11-11 um 12:40 schrieb Suvendu Sekhar Mondal:

Hello Everyone,

During database failover resiliency testing we noticed something
unusual which we still cannot explain. So, reaching out to the
community for help.

Application Setup:
AdoptOpenJDK 1.8.192, Tomcat 7.0.55, Apache httpd, Windows 2012, SQL Server 2016


Almost of them them are outdated, you should upgrade first.

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: No thread name in AccessLogValve printed

[Michael Osipov]

Am 2020-10-20 um 11:44 schrieb Mark Thomas:

On 20/10/2020 10:10, Michael Osipov wrote:

Folks,

I have seen recently entried like this on our access logs:

2020-10-19T20:00:05.591 [null] xyz - "-" 400 - 0
2020-10-19T20:00:05.591 [null] abc- "-" 400 - 0
2020-10-19T20:00:05.592 [null] abc - "-" 400 - 0
2020-10-19T20:00:05.593 [null] abc - "-" 400 - 0
2020-10-19T20:00:05.616 [null] abc - "-" 400 - 0


with pattern:

%{-MM-dd'T'HH:mm:ss.SSS}t [%I] %h %u %r %s %b %D


While I am quite certain that these ary "security" scans at work
I wonder why RequestInfo#getWorkerThreadName() is null.
Is the request rejected because it is malformed because before it is
handled to a worker?


Have you tried looking at the source code? Callers of
RequestInfo.setWorkerThreadName() are likely to be enlightening.


Yes, I did. It happens before the thread local is used. It happens in 
org.apache.coyote.http11.Http11Processor.prepareRequest() before a 
servlet is invoked. So the worker name has to be null here.



I am not sure whether "info != null &&
info.getgetWorkerThreadName() != null" would be the right change here.


That would address the symptom. There may be a better fix that addresses
(or at least gets closer) to the cause.


Given the information above I am inclined to say that Thread#getName() 
would make sense here. Writing '-' does not make sense becuase every 
request is processed in some thread, isn't it?


Michael

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

No thread name in AccessLogValve printed

[Michael Osipov]

Folks,

I have seen recently entried like this on our access logs:

2020-10-19T20:00:05.591 [null] xyz - "-" 400 - 0
2020-10-19T20:00:05.591 [null] abc- "-" 400 - 0
2020-10-19T20:00:05.592 [null] abc - "-" 400 - 0
2020-10-19T20:00:05.593 [null] abc - "-" 400 - 0
2020-10-19T20:00:05.616 [null] abc - "-" 400 - 0


with pattern:

%{-MM-dd'T'HH:mm:ss.SSS}t [%I] %h %u %r %s %b %D


While I am quite certain that these ary "security" scans at work
I wonder why RequestInfo#getWorkerThreadName() is null.
Is the request rejected because it is malformed because before it is 
handled to a worker? I am not sure whether "info != null && 
info.getgetWorkerThreadName() != null" would be the right change here.


Does someone know better?

Michael

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: OpenSSL prompts for key password

[Michael Osipov]

Am 2020-10-15 um 17:34 schrieb Christopher Schultz:

Michael,

On 10/15/20 08:12, Michael Osipov wrote:

Michael,

On 10/14/20 12:46, Michael Osipov wrote:

Folks,

I have recently upgrade a cert and left out the last char of the key
password by accident.


# /sbin/init.d/tomcat-smartld start
Starting Apache Tomcat 8.5...
Using CATALINA_BASE:   /var/opt/tomcat-smartld
Using CATALINA_HOME:   /opt/ports/apache-tomcat-8.5.57
Using CATALINA_TMPDIR: /var/opt/tomcat-smartld/temp
Using JRE_HOME:    /opt/java8
Using CLASSPATH:
/opt/ports/apache-tomcat-8.5.57/bin/bootstrap.jar:/opt/ports/apache-tomcat-8.5.57/bin/tomcat-juli.jar

Tomcat started.
Apache Tomcat 8.5 started.
# Some of your private key files are encrypted for security reasons.
In order to read them you have to provide the pass phrases.
Enter password :
  


I have seen similar with HTTPd in the past. Since the start is asyncI
have no option to react on that and it will block the entire config.I
looked briefly in the OpenSSL API, but wasn't really able to find a flag
to inhibit the interactive prompt.

Does someone know whether we can make this better with libtcnative?


What kind of behavior were you hoping for? I'm assuming that some kind
of exception would be best for this case (incorrect password).


Correct. As long as stdin is is not attached to a tty, this must be
non-interactive.


Suppressing the interactive prompt is likely to simply cause the
connector to fail to initialize; basically the same thing as throwingan
exception in the above case.


Correct.


I searched the Tomcat code and I don't see that sting anywhere, so I
suspect it's coming directly from OpenSSL (which is very weird IMHO).


It comes from us (tomcat-native/native/include/ssl_private.h):

#define SSL_DEFAULT_PASS_PROMPT "Some of your private key files are encrypted for 
security reasons.\n"  \
 "In order to read them you have to provide the pass 
phrases.\n" \
 "Enter password :"


*facepalm* I was only looking at the Java code.


Used by us in SSL_password_prompt() called by SSL_password_callback().
I haven't yet found out which object is called to obtain the password.


It seems reasonable that when we *know* that we are supplying a
password, we should install a null-callback for that and let OpenSSL's
engine fail to initialize.


Exactly!


mod_ssl has a configurable way to gather this passphrase, presumably to
pass it into OpenSSL's read-key function. It would surprise me greatly
if an incorrect passphrase would cause the same kind of prompt in httpd.


It does when your don't provide a method how you will supply the password.
See SSLPassPhraseDialog builtin in

./modules/ssl/ssl_engine_config.c ./modules/ssl/ssl_engine_pphrase.c


See also Javadoc of SSLContext#setCertificate(long, String, String, String, int)


What version of OpenSSL are you using? Have you tried any other versions?


I am on

# openssl version
OpenSSL 1.1.1d  10 Sep 2019


supplied by HPE on HP-UX.

A change in OpenSSL version won't make a difference because this is how it is
configured from tcnative for now.


+1


I will file an issue to track this, especially because it is not documented
compared to SSLPassPhraseDialog.


+1

I didn't even know that tcnative would prompt for a password, *ever*.


Worse than that, I does not even tell for what certificate key it 
expects one..


Michael



-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Re: OpenSSL prompts for key password

[Michael Osipov]

> Michael,
> 
> On 10/14/20 12:46, Michael Osipov wrote:
> > Folks,
> > 
> > I have recently upgrade a cert and left out the last char of the key
> > password by accident.
> > 
> >> # /sbin/init.d/tomcat-smartld start
> >> Starting Apache Tomcat 8.5...
> >> Using CATALINA_BASE:   /var/opt/tomcat-smartld
> >> Using CATALINA_HOME:   /opt/ports/apache-tomcat-8.5.57
> >> Using CATALINA_TMPDIR: /var/opt/tomcat-smartld/temp
> >> Using JRE_HOME:    /opt/java8
> >> Using CLASSPATH:  
> >> /opt/ports/apache-tomcat-8.5.57/bin/bootstrap.jar:/opt/ports/apache-tomcat-8.5.57/bin/tomcat-juli.jar
> >>
> >> Tomcat started.
> >> Apache Tomcat 8.5 started.
> >> # Some of your private key files are encrypted for security reasons.
> >> In order to read them you have to provide the pass phrases.
> >> Enter password :
> >>  
> > 
> > I have seen similar with HTTPd in the past. Since the start is async I
> > have no option to react on that and it will block the entire config. I
> > looked briefly in the OpenSSL API, but wasn't really able to find a flag
> > to inhibit the interactive prompt.
> > 
> > Does someone know whether we can make this better with libtcnative?
> 
> What kind of behavior were you hoping for? I'm assuming that some kind
> of exception would be best for this case (incorrect password).

Correct. As long as stdin is is not attached to a tty, this must be
non-interactive.

> Suppressing the interactive prompt is likely to simply cause the
> connector to fail to initialize; basically the same thing as throwing an
> exception in the above case.

Correct.

> I searched the Tomcat code and I don't see that sting anywhere, so I
> suspect it's coming directly from OpenSSL (which is very weird IMHO).

It comes from us (tomcat-native/native/include/ssl_private.h):
> #define SSL_DEFAULT_PASS_PROMPT "Some of your private key files are encrypted 
> for security reasons.\n"  \
> "In order to read them you have to provide 
> the pass phrases.\n" \
> "Enter password :"

Used by us in SSL_password_prompt() called by SSL_password_callback().
I haven't yet found out which oboect is called to obtain the password.

> mod_ssl has a configurable way to gather this passphrase, presumably to
> pass it into OpenSSL's read-key function. It would surprise me greatly
> if an incorrect passphrase would cause the same kind of prompt in httpd.

It does when your don't provide a method how you will supply the password.
See SSLPassPhraseDialog builtin in
> ./modules/ssl/ssl_engine_config.c ./modules/ssl/ssl_engine_pphrase.c

See also Javadoc of SSLContext#setCertificate(long, String, String, String, int)

> What version of OpenSSL are you using? Have you tried any other versions?

I am on
> # openssl version
> OpenSSL 1.1.1d  10 Sep 2019

supplied by HPE on HP-UX.

A change in OpenSSL version won't make a difference because this is how it is
configured from tcnative for now.

I will file an issue to track this, especially because it is not documented
compared to SSLPassPhraseDialog.

Michael

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

OpenSSL prompts for key password

[Michael Osipov]

Folks,

I have recently upgrade a cert and left out the last char of the key 
password by accident.



# /sbin/init.d/tomcat-smartld start
Starting Apache Tomcat 8.5...
Using CATALINA_BASE:   /var/opt/tomcat-smartld
Using CATALINA_HOME:   /opt/ports/apache-tomcat-8.5.57
Using CATALINA_TMPDIR: /var/opt/tomcat-smartld/temp
Using JRE_HOME:/opt/java8
Using CLASSPATH:   
/opt/ports/apache-tomcat-8.5.57/bin/bootstrap.jar:/opt/ports/apache-tomcat-8.5.57/bin/tomcat-juli.jar
Tomcat started.
Apache Tomcat 8.5 started.
# Some of your private key files are encrypted for security reasons.
In order to read them you have to provide the pass phrases.
Enter password :
 


I have seen similar with HTTPd in the past. Since the start is async I 
have no option to react on that and it will block the entire config. I 
looked briefly in the OpenSSL API, but wasn't really able to find a flag 
to inhibit the interactive prompt.


Does someone know whether we can make this better with libtcnative?

Michael

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Tomcat JDBC Pool Cleaner Deadlock Problem

[Michael Ng]

I had a similar issue with JDBC and it got fixed by adding this
parameter   'numTestsPerEvictionRun'=> '3',

'auth'  => 'Container',
'driverClassName'   => 'oracle.jdbc.OracleDriver',
'maxActive' => '30',
'maxIdle'   => '8',
'maxWait'   => '-1',
'minIdle'   => '1',
'factory'   =>
'org.apache.tomcat.jdbc.pool.DataSourceFactory',
'testWhileIdle' => 'true',
'testOnReturn'  => 'false',
'validationQuery'   => 'SELECT 1 FROM DUAL',
'testOnBorrow'  => 'true',
'maxWait'   => '-1',
'initialSize'   => '1',
'removeAbandoned'   => 'true',
'logAbandoned'  => 'true',
'jmxEnabled'=> 'true',
'jdbcInterceptors'  =>
'org.apache.tomcat.jdbc.pool.interceptor.ConnectionState;org.apache.tomcat.jdbc.pool.interceptor.StatementFinalizer',
'validationInterval'=> '3',
'timeBetweenEvictionRunsMillis' => '3',
'minEvictableIdleTimeMillis'=> '3',
'removeAbandonedTimeout'=> '60',
'numTestsPerEvictionRun'=> '3',

On Thu, Aug 27, 2020 at 1:44 PM Christopher Schultz
 wrote:
>
> -BEGIN PGP SIGNED MESSAGE-
> Hash: SHA256
>
> Gokhan,
>
> On 8/27/20 05:47, Gokhan Akgul wrote:
> > Hi ,
> >
> > I have been facing the deadlock issue for the last 2 months  about
> > JDBCPoolCleaner Thread .
> >
> > Following config set in context.xml
> >
> >  > type="javax.sql.DataSource"
> > factory="org.apache.tomcat.jdbc.pool.DataSourceFactory"
> > driverClassName="com.mysql.jdbc.Driver"
> > url="jdbc:mysql://adress:3306/db?useUnicode=truecharacterEncoding
> =latin5characterResultSet=latin5zeroDateTimeBehavior=convertTo
> NullautoReconnect=trueinteractiveClient=true"
> >
> >
> username="user"
> > password="pass" initialSize="10" maxActive="30" maxIdle="15"
> > minIdle="10" maxWait="3" timeBetweenEvictionRunsMillis="5000"
> > minEvictableIdleTimeMillis="6" removeAbandonedTimeout="600"
> > removeAbandoned="true" logAbandoned="false" testWhileIdle="true"
> > testOnBorrow="true" testOnReturn="false" validationQuery="/* ping
> > */ SELECT 1" validationInterval="3" jmxEnabled="true"
> > jdbcInterceptors="ConnectionState;StatementFinalizer;ResetAbandonedTim
> er;SlowQueryReport"
> >
> >
> />
> >
> >
> >
> > Thread dump
> >
> > Tomcat JDBC Pool Cleaner[63445188:1598345711425] id=16
> > state=BLOCKED - waiting to lock <0x57dcb0b7> (a
> > com.mysql.jdbc.JDBC4PreparedStatement) owned by
> > http-nio-8080-exec-8 id=25 at
> > com.mysql.jdbc.PreparedStatement.realClose(PreparedStatement.java:3078
> )
> >
> >
> at
> com.mysql.jdbc.ConnectionImpl.closeAllOpenStatements(ConnectionImpl.java
> :1584)
> > at
> > com.mysql.jdbc.ConnectionImpl.realClose(ConnectionImpl.java:4364)
> > at com.mysql.jdbc.ConnectionImpl.close(ConnectionImpl.java:1556) at
> > org.apache.tomcat.jdbc.pool.PooledConnection.disconnect(PooledConnecti
> on.java:388)
> >
> >
> at
> org.apache.tomcat.jdbc.pool.PooledConnection.release(PooledConnection.ja
> va:618)
> > at
> > org.apache.tomcat.jdbc.pool.ConnectionPool.release(ConnectionPool.java
> :612)
> >
> >
> at
> org.apache.tomcat.jdbc.pool.ConnectionPool.abandon(ConnectionPool.java:5
> 69)
> > at
> > org.apache.tomcat.jdbc.pool.ConnectionPool.checkAbandoned(ConnectionPo
> ol.java:999)
> >
> >
> at
> org.apache.tomcat.jdbc.pool.ConnectionPool$PoolCleaner.run(ConnectionPoo
> l.java:1468)
> > at java.util.TimerThread.mainLoop(Timer.java:555) at
> > java.util.TimerThread.run(Timer.java:505)
> >
> > Locked synchronizers: count = 1 -
> > java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync@58039868
> >
> >
> >
> >
> > http-nio-8080-exec-8 id=25 state=BLOCKED - waiting to lock
> > <0x42de9995> (a com.mysql.jdbc.JDBC4Connection) owned by Tomcat
> > JDBC Pool Cleaner[63445188:1598345711425] id=16 at
> > com.mysql.jdbc.ConnectionImpl.useAnsiQuotedIdentifiers(ConnectionImpl.
> java:5435)
> >
> >
> at
> com.mysql.jdbc.DatabaseMetaData.getIdentifierQuoteString(DatabaseMetaDat
> a.java:3269)
> > at
> > com.mysql.jdbc.DatabaseMetaData.(DatabaseMetaData.java:675)
> > at
> > com.mysql.jdbc.JDBC4DatabaseMetaData.(JDBC4DatabaseMetaData.java
> :39)
> >
> >
> at sun.reflect.GeneratedConstructorAccessor24.newInstance(Unknown Source
> )
> > at
> > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingCo
> nstructorAccessorImpl.java:45)
> >
> >
> at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
> > at com.mysql.jdbc.Util.handleNewInstance(Util.java:411) at
> > 

Re: [tomcat-users] Re: getting web application version string?

[Michael Osipov]

Am 2020-08-17 um 17:57 schrieb Jason Pyeron:

-Original Message-
From: Michael Osipov
Sent: Sunday, August 16, 2020 1:27 PM

Am 2020-08-16 um 18:16 schrieb Jason Pyeron:

Is there a better way than this?

Specifically - detect running Tomcat, then if under Tomcat (today only 
interested in v7 and v9)

obtain the version string as described [1] and shown on the Manager web 
application.

At least for the version, you can use my listener, it will expose all
all context-related information via JNDI. Give it a try:
http://mo-tomcat-ext.sourceforge.net/user-guide.html#ContextNamingInfoListener


Cute. I like how you use org.apache.catalina.LifecycleListener to have 
legitimate access to the org.apache.catalina.Context.

The only gotcha I have is it requires advanced knowledge that you are on 
Tomcat. Specifically, it requires Operations to configure the context.xml for 
the web application.


I see no portable way to solve this. In fact, from an app's POV. JNDI is 
portable through containers. If you want the version then you need to 
accept that this is Tomcat-specific information.


M

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: getting web application version string?

[Michael Osipov]

Am 2020-08-16 um 18:16 schrieb Jason Pyeron:

Is there a better way than this?

Specifically - detect running Tomcat, then if under Tomcat (today only 
interested in v7 and v9) obtain the version string as described [1] and shown 
on the Manager web application.


At least for the version, you can use my listener, it will expose all 
all context-related information via JNDI. Give it a try: 
http://mo-tomcat-ext.sourceforge.net/user-guide.html#ContextNamingInfoListener


Michael

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: [Tomcat 9.0.37] Https / SSL on Windows server 2016 with windows certificate store

[Michael Osipov]

Am 2020-07-11 um 23:52 schrieb Valentin:

Hello,

I try to configure my tomcat 9.0.37 installed on a windows server 2016 to
use a certificate located in *cert:LocalMachine\My*

I mention that I am an administrator of this machine.
This certificate is also used by IIS.

What I did was to configure my server.xml file like this :



The error I got in tomcat logs was that the keyAlias doesn't exist but I
used the CN mentioned in the description of my certificate.

Is it possible for tomcat to use the windows certificate store ?
The only link I found about this was :
https://bz.apache.org/bugzilla/show_bug.cgi?id=56021


I have used Windows-MY several times now with HttpClient, curl and 
OpenSSL. The native Crypto API of the Windows Cert Store provides 
several name formats for the key alias.

First of all, set CAPI_TRACE env var to see more output.
Native does this: 
https://github.com/AdoptOpenJDK/openjdk-jdk8u/blob/master/jdk/src/windows/native/sun/security/mscapi/security.cpp#L561-L563
CERT_NAME_FRIENDLY_DISPLAY_TYPE (fallback CERT_NAME_SIMPLE_DISPLAY_TYPE) 
from 
https://docs.microsoft.com/en-us/windows/win32/api/wincrypt/nf-wincrypt-certgetnamestringa


My recommendation is to write the simplest code, open Windows-MY iterate 
over all keys, print keys and then you will know what these display 
names are. The DNS name you use is obviously not the right one since it 
had to be CERT_NAME_DNS_TYPE.


Good luck,

Michael

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Looking for mod_proxy_ajp or mod_proxy_http users

[Michael Osipov]

Am 2020-07-08 um 23:52 schrieb Christopher Schultz:

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256

All,

I'm (likely) giving a presentation at this year's ApacheCon North
America (which is *virtual* this year, so the "North America" just
means "it was supposed to be in New Orleans in September" but anyone
can attend ... FOR FREE!) about migrating from mod_jk ->
mod_proxy_http and I'd like some feedback on my draft slides.

Is anyone using mod_proxy_ajp or mod_proxy_http in a production
setting and has a lot of experience with getting it all to work well?
My 20 years or so of experience with proxying to Tomcat has always
used mod_jk so I'm sure there are some obvious things I'm not thinking
about because mod_jk was designed to pretty much work right out of the
box.


I wouldn't say I have a lot of experience, but I am using mod_proxy_http 
for quite a while. I wish I could use Tomcat directly, there are several 
annoying bugs in mod_proxy_http discussing with httpd devs.


Michael

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: RFC7807 ErrorReportValve

[Michael Osipov]

Am 2020-07-06 um 16:04 schrieb Mark Thomas:

On 05/07/2020 10:28, Michael Osipov wrote:

Am 2020-07-02 um 21:30 schrieb Thomas Meyer:

Hi,

What are your opinions on providing a RFC7807 based ErrorReportValve
as part of Tomcat default distribution?


Thomas, this has been bugging me for a while. Let me share some thoughts
on this, I'll limit my experiences with Tomcat, Spring Web and Zalando
Problem (including it's web module):

Mark, please correct me if my citation of the Servlet API is wrong.

* The Servlet API has been designed where the only clients where browsers


I don't think anything in the Servlet API is browser specific. That use
case is certainly supported but I don't think that support is exclusive.


* The Servlet API mandates that all invocations of
HttpServletResponse#setError() must yield in a HTML page and this
*cannot* be changed by defult


I don't recall any requirement to that effect and I can't find anything
looking in the spec.


As for these two, Javadoc says:
Sends an error response to the client using the specified status code and clears the output buffer. The server defaults to creating the response to look like an HTML-formatted server error page containing the specified message, setting the content type to "text/html", leaving cookies and other headers unmodified. If an error-page declaration has been made for the web application corresponding to the status code passed in, it will be served back in preference to the suggested msg parameter. 


This is what I see as default.


The requirement is that the error handling mechanism is invoked. Nothing
stops an application providing an error servlet configured to handle all
errors. That could return HTML, JSON, XML, text/plain, etc.

Tomcat's default implementation returns HTML.


Correct, but we are talking about default behavior.


* Even if you write a REST API or explicitly use @RestController Spring
will still invoke #setError() although it makes no sense. I consider
this to be a conceptual flaw in the Spring framework.


Why? That using sendError() just invokes the error handling mechanism.
What matters is what is configured to handle those errors.


Again here, for REST API this doesn't make sense. #setStatus() would be 
better here.


M

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: RFC7807 ErrorReportValve

[Michael Osipov]

Am 2020-07-02 um 21:30 schrieb Thomas Meyer:

Hi,

What are your opinions on providing a RFC7807 based ErrorReportValve as part of 
Tomcat default distribution?


Thomas, this has been bugging me for a while. Let me share some thoughts 
on this, I'll limit my experiences with Tomcat, Spring Web and Zalando 
Problem (including it's web module):


Mark, please correct me if my citation of the Servlet API is wrong.

* The Servlet API has been designed where the only clients where browsers
* The Servlet API mandates that all invocations of 
HttpServletResponse#setError() must yield in a HTML page and this 
*cannot* be changed by defult
* Even if you write a REST API or explicitly use @RestController Spring 
will still invoke #setError() although it makes no sense. I consider 
this to be a conceptual flaw in the Spring framework.


Before we continue which issue do you want to solve? Tomcat produced 
errors or by a framework?  As for the framework, I would prefer to file 
an issue with Spring Framework first and see what the devs say because 
this would solely solve a symptom.


Michael

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: jsvc - non root - log as root

[Michael Osipov]

Am 2020-06-26 um 09:21 schrieb Jürgen Weber:

Hi,

when you run tomcat with jsvc and have jsvc drop privileges to a
different user, stdout and stderr log files are still created with
root as owner.
Can you make jsvc create them as the -user ?


What is you actual problem with that?

I have talked about this to Mark two years ago. I think this is 
logically not that easy. jsvc needs to start as root to bind priviledged 
ports, it inits Tomcat, then downgrades and starts the rest of the 
container. While it is in init state stdout/stderr need to be opened 
already.


If you are have problem with log rotation. I use newsyslogd for that 
which does send SIGUSR1 and works flawlessly. You may use similar tools.


M



-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Connection Closure due to Fatal Stream with HTTP2

[Michael Osipov]

Am 2020-06-13 um 08:42 schrieb Chirag Dewan:

Hi,

We are observing that under high load, my clients start receiving a GoAway
frame with error:

*Connection[{id}], Stream[{id}] an error occurred during processing that
was fatal to the connection.*

Background : We have implemented our clients to close connections after
every 500-1000 requests (streams). This is a load balancer requirement that
we are working on and hence such a behavior. So with a throughput of around
19k, almost 40 connections are closed and recreated every second.

After we receive this frame, my clients start behaving erroneously. Before
this as well, my clients start sending RST_STREAM with canceled for each
request. Could this be due to the number of connections we open? Is it
related to the version of Tomcat? Or maybe my clients are misbehaving?

Now since I only receive this under heavy load, I can't quite picture
enough reasons for this to happen.

Any possible clues on where I should start looking?

My Stack:
Server - Tomcat 9.0.29
Client - Jetty 9.x


Does the same happen w/o the load balancer?

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Should Tomcat 10 enable response compression by default?

[Michael Osipov]

Am 2020-06-09 um 22:20 schrieb Mark Thomas:

Hi all,

An enhancement has been opened to enable response compression by default:
https://bz.apache.org/bugzilla/show_bug.cgi?id=64431

In short, the proposal is to change the default for the Connector's
compression attribute from "off" to "on".

This would be for Tomcat 10 onwards only.

The following would be unchanged:
- compressibleMimeType
- compressionMinSize



- noCompressionStrongETag


Shouldn't this be gone in 10?



It would be helpful to know what the range of views of the user
community are on this proposal.

So, thoughts?


Don't enable by default. Carsten Klein brought up a few good points. 
Another is when used in corporate networks, where clients and servers 
are close, I see no benefit for compression here.


-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Tomcat 9.0.27 loads incorrect openssl version

[Michael Osipov]

Am 2020-06-07 um 21:31 schrieb Norbert Elbanbuena:

Hi Michael,

Interesting I find multiple files.


You have managed to mess up your system. The loader hierarchy looks wrong.


# ldd -v /usr/lib/libtcnative-1.so


I assume that this is provided by your package manager. If so


 linux-vdso.so.1 =>  (0x7ffd767f9000)
 libssl.so.1.1 => /usr/local/ssl/lib/libssl.so.1.1 (0x7f33f497b000)
 libcrypto.so.1.1 => /usr/local/ssl/lib/libcrypto.so.1.1 
(0x7f33f4491000)
 libapr-1.so.0 => /usr/local/apr/lib/libapr-1.so.0 (0x7f33f4256000)


This is wrong. All deps should reside in /usr from other packages 
managed by your package manager.



 libuuid.so.1 => /lib64/libuuid.so.1 (0x7f33f4051000)
 librt.so.1 => /lib64/librt.so.1 (0x7f33f3e49000)
 libcrypt.so.1 => /lib64/libcrypt.so.1 (0x7f33f3c12000)
 libpthread.so.0 => /lib64/libpthread.so.0 (0x7f33f39f6000)
 libdl.so.2 => /lib64/libdl.so.2 (0x7f33f37f2000)
 libc.so.6 => /lib64/libc.so.6 (0x7f33f3424000)
 libz.so.1 => /lib64/libz.so.1 (0x7f33f320e000)
 /lib64/ld-linux-x86-64.so.2 (0x7f33f4e39000)
 libfreebl3.so => /lib64/libfreebl3.so (0x7f33f300b000)

# ldd -v /usr/local/apr/lib/libtcnative-1.so
 linux-vdso.so.1 =>  (0x7ffdad59d000)
 libssl.so.10 => /lib64/libssl.so.10 (0x7fcee1505000)


this is obviously not what you wanted. Either RPATH is wrong in your 
shared objects or LD_LIBRARY_PATH leads to a weird state.



 libcrypto.so.10 => /lib64/libcrypto.so.10 (0x7fcee10a2000)
 libapr-1.so.0 => /usr/local/apr/lib/libapr-1.so.0 (0x7fcee0e67000)
 libuuid.so.1 => /lib64/libuuid.so.1 (0x7fcee0c62000)
 librt.so.1 => /lib64/librt.so.1 (0x7fcee0a5a000)
 libcrypt.so.1 => /lib64/libcrypt.so.1 (0x7fcee0823000)
 libpthread.so.0 => /lib64/libpthread.so.0 (0x7fcee0607000)
 libdl.so.2 => /lib64/libdl.so.2 (0x7fcee0403000)
 libc.so.6 => /lib64/libc.so.6 (0x7fcee0035000)
 libgssapi_krb5.so.2 => /lib64/libgssapi_krb5.so.2 (0x7fcedfde8000)
 libkrb5.so.3 => /lib64/libkrb5.so.3 (0x7fcedfaff000)
 libcom_err.so.2 => /lib64/libcom_err.so.2 (0x7fcedf8fb000)
 libk5crypto.so.3 => /lib64/libk5crypto.so.3 (0x7fcedf6c8000)
 libz.so.1 => /lib64/libz.so.1 (0x7fcedf4b2000)
 /lib64/ld-linux-x86-64.so.2 (0x7fcee19a4000)
 libfreebl3.so => /lib64/libfreebl3.so (0x7fcedf2af000)
 libkrb5support.so.0 => /lib64/libkrb5support.so.0 (0x7fcedf09f000)
 libkeyutils.so.1 => /lib64/libkeyutils.so.1 (0x7fcedee9b000)
 libresolv.so.2 => /lib64/libresolv.so.2 (0x7fcedec81000)
 libselinux.so.1 => /lib64/libselinux.so.1 (0x7fcedea5a000)
 libpcre.so.1 => /lib64/libpcre.so.1 (0x7fcede7f8000)



As a side node, there is *no* /usr/local on Linux or System V UNIX, 
/usr/local is a BSD approach. You should put custom software into /opt. 
(I use /opt/ports on System 5 UNIX)


What you should do now:

* Separate clearly software from your package manager and manually 
compiled with GNU autoconf/make/cmake. Don't mix and match package 
provided software with custom compiled unless you perfectly know what 
you are doing, this can lead to subtile bugs.
* Find out which of the libtcnative-1.so files Tomcat is loading. (use 
strace for this, you'll see dlopen(3))
* Install custom software into /opt with --prefix=/opt and ALWAYS 
provide linker and runtime path (rpath) to ./configure to have the 
exected result

* Verify with ldd that linking is correct
* Don't use LD_LIBRARY_PATH globally. This is merely a workaround for a 
missing RPATH and some other situations


You can use Python's ctypes.CDLL() function to perform the same 
operation quick and easy in Python's REPL.


Please also provide the ./configure args and output of libtcnative as 
well as config.log after configure.


Michael


-Original Message-
From: Michael Osipov 
Sent: Sunday, June 7, 2020 2:19 PM
To: users@tomcat.apache.org
Subject: Re: Tomcat 9.0.27 loads incorrect openssl version

Am 2020-06-07 um 20:16 schrieb Norbert Elbanbuena:

Hi,

I removed the previous version of OpenSSL 1.0.2k-fips from yum. Then I 
installed OpenSSL 1.1.1g from source and made a clean install of tomcat-native 
1.2.24 pointing to the correct OpenSSL path.
When I start Tomcat, it still shows OpenSSL 1.0.2k-fips being loaded on startup.

07-Jun-2020 18:09:20.357 INFO [main] 
org.apache.catalina.core.AprLifecycleListener.lifecycleEvent Loaded APR based 
Apache Tomcat Native library [1.2.24] using APR version [1.7.0].
07-Jun-2020 18:09:20.357 INFO [main] 
org.apache.catalina.core.AprLifecycleListener.lifecycleEvent APR capabilities: 
IPv6 

Re: Tomcat 9.0.27 loads incorrect openssl version

[Michael Osipov]

Am 2020-06-07 um 20:16 schrieb Norbert Elbanbuena:

Hi,

I removed the previous version of OpenSSL 1.0.2k-fips from yum. Then I 
installed OpenSSL 1.1.1g from source and made a clean install of tomcat-native 
1.2.24 pointing to the correct OpenSSL path.
When I start Tomcat, it still shows OpenSSL 1.0.2k-fips being loaded on startup.

07-Jun-2020 18:09:20.357 INFO [main] 
org.apache.catalina.core.AprLifecycleListener.lifecycleEvent Loaded APR based 
Apache Tomcat Native library [1.2.24] using APR version [1.7.0].
07-Jun-2020 18:09:20.357 INFO [main] 
org.apache.catalina.core.AprLifecycleListener.lifecycleEvent APR capabilities: 
IPv6 [true], sendfile [true], accept filters [false], random [true].
07-Jun-2020 18:09:20.357 INFO [main] 
org.apache.catalina.core.AprLifecycleListener.lifecycleEvent APR/OpenSSL 
configuration: useAprConnector [true], useOpenSSL [true]
07-Jun-2020 18:09:20.361 INFO [main] 
org.apache.catalina.core.AprLifecycleListener.initializeSSL OpenSSL 
successfully initialized [OpenSSL 1.0.2k-fips  26 Jan 2017]

I verified latest version of OpenSSL reflected

$openssl version -a
OpenSSL 1.1.1g  21 Apr 2020
built on: Sun Jun  7 15:15:04 2020 UTC
platform: linux-x86_64
options:  bn(64,64) rc4(16x,int) des(int) idea(int) blowfish(ptr)
compiler: gcc -fPIC -pthread -m64 -Wa,--noexecstack -Wall -O3 
-DOPENSSL_USE_NODELETE -DL_ENDIAN -DOPENSSL_PIC -DOPENSSL_CPUID_OBJ 
-DOPENSSL_IA32_SSE2 -DOPENSSL_BN_ASM_MONT -DOPENSSL_BN_ASM_MONT5 
-DOPENSSL_BN_ASM_GF2m -DSHA1_ASM -DSHA256_ASM -DSHA512_ASM -DKECCAK1600_ASM 
-DRC4_ASM -DMD5_ASM -DAESNI_ASM -DVPAES_ASM -DGHASH_ASM -DECP_NISTZ256_ASM 
-DX25519_ASM -DPOLY1305_ASM -DZLIB -DNDEBUG
OPENSSLDIR: "/usr/local/ssl"
ENGINESDIR: "/usr/local/ssl/lib/engines-1.1"
Seeding source: os-specific

Can somebody help me understand why Tomcat still points to the old fips 
version? Or do I need to install OpenSSL fips instead?


Run 'ldd .../libtcnative-1.so' and paste results.

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: maxSwallowSize and misbehaving clients (e.g., mod_proxy_http)

[Osipov, Michael]

Am 2020-05-27 um 12:35 schrieb Mark Thomas:
> This then opens up an interesting question of whether to bother reading
> *any* of the request body if Tomcat knows it is going to close the
> connection before reading all of it. Based on what you have observed,
> would earlier closure of the connection (after a response with a
> "Connection: close" header) create additional difficulties? Would those
> clients that currently read the response while Tomcat is swallowing the
> first 2MB of upload still read the response if Tomcat didn't swallow any
> of the request? I think you can test this with maxSwallowSize="0"

As requested, reconfigured to 0 plain HTTP and HTTPS. For clients which 
can handle early responses at anytime, e.g. curl, it does not matter. 
They will stop sending as soon as possible. So does curl here.


For clients which do not handle this, they will fail earlier, but not as 
soon as possible because they will send until the socket is broken and a 
SocketException is raised. They don't see the early response at all. If 
and only if the entire body is swallowed the final response is seen. 
Personally, I prefer to fail fast.


I see no silver bullet here. It seriously depends on the usecase. If the 
usecase contains a slow link between client and server, I prefer 0. For 
a fast link, it depends on the average size of the request payload.


Can provide output for both, if required.

Michael

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: maxSwallowSize and misbehaving clients (e.g., mod_proxy_http)

[Osipov, Michael]

Am 2020-05-27 um 12:35 schrieb Mark Thomas:

On 27/05/2020 10:59, Osipov, Michael wrote:



Am 2020-05-27 um 10:51 schrieb Mark Thomas:

On 22/05/2020 22:59, Osipov, Michael wrote:




I found one issue with HttpClient and Tomcat via HTTP/1.1. I have
decrypted the TLS traffic [1]. I can see that HttpClient sends the
headers also with a 4 KiB large chunk of the ZIP file. In return a
Tomcats send the 401 response with:

Keep-Alive: timeout=300
Connection: keep-alive


The client keeps sending in 8 KiB large blocks. After 2 134 016 written
bytes there is a TLS alert: Close Notify. A few packets later: RST.

I would expect to see here: "Connection: close".


Why?

Does the client provide a content-length header and no expectation? If
so, Tomcat could determine that the maxSwallowSize is going to be
exceeded and close the connection early.


Yes, it does. CL present and not expectation. A transcript is here [1].
We have already a working patch for the early response, also Oleg is
already working on the early close not happening which is a bug in
HttpClient.

As for the connection header: I do not understand why you assume the
header is no necessary. Your reasoning is that if the server closes the
physical connection before the client has spooled the entire body and it
is imperative for the client to close the connection too regard of the
presense of the header?


I didn't assume the header is not necessary. I asked for further
explanation from you of why you thought it was necessary. I also asked
some questions relevant to a potential justification for adding the header.


Sorry if I caused confusion or missed something.

This is my understanding from section 6.5 the last paragraph. The 
justification will follow below.



Tomcat has enough information to know that:
- it needs to return a 401
- the request body won't be read (by the application)
- the request body is larger than maxSwallowSize so once that many bytes
   have been swallowed, Tomcat is going to close the connection

At this point I think we are heading outside of behaviour defined by the
HTTP/1.1 specification. I haven't found language covering aborted uploads.

RFC 7230, 6.6 states a "Connection: close" header SHOULD be sent if
Tomcat knows it is going to close the connection. I think that applies
in this case so I'd be in favour of adding that behaviour.


That's the first paragraph, I agree. See also this example from the now 
working HttpClient:



5921 [main] DEBUG 
org.apache.hc.client5.http.impl.io.PoolingHttpClientConnectionManager - 
ep-0001: connected http-outgoing-1
5921 [main] DEBUG org.apache.hc.client5.http.impl.classic.InternalHttpClient - 
ep-0001: endpoint connected
5921 [main] DEBUG org.apache.hc.client5.http.impl.classic.MainClientExec - 
ex-0002: executing POST /content-dev/api/documents HTTP/1.1
5921 [main] DEBUG org.apache.hc.client5.http.impl.classic.InternalHttpClient - 
ep-0001: start execution ex-0002
5921 [main] DEBUG 
org.apache.hc.client5.http.impl.io.PoolingHttpClientConnectionManager - 
ep-0001: executing exchange ex-0002 over http-outgoing-1
5921 [main] DEBUG org.apache.hc.client5.http.headers - http-outgoing-1 >> POST 
/content-dev/api/documents HTTP/1.1
5921 [main] DEBUG org.apache.hc.client5.http.headers - http-outgoing-1 >> 
Accept-Encoding: gzip, x-gzip, deflate
5921 [main] DEBUG org.apache.hc.client5.http.headers - http-outgoing-1 >> 
Content-Length: 7664149
5921 [main] DEBUG org.apache.hc.client5.http.headers - http-outgoing-1 >> 
Content-Type: application/zip
5921 [main] DEBUG org.apache.hc.client5.http.headers - http-outgoing-1 >> Host: 
:1
5921 [main] DEBUG org.apache.hc.client5.http.headers - http-outgoing-1 >> 
Connection: keep-alive
5921 [main] DEBUG org.apache.hc.client5.http.headers - http-outgoing-1 >> 
User-Agent: Apache-HttpClient/5.0.1-SNAPSHOT (Java/13.0.2)
5949 [main] DEBUG org.apache.hc.client5.http.headers - http-outgoing-1 << HTTP/1.1 401 
5949 [main] DEBUG org.apache.hc.client5.http.headers - http-outgoing-1 << WWW-Authenticate: Negotiate

5949 [main] DEBUG org.apache.hc.client5.http.headers - http-outgoing-1 << 
Content-Type: text/html;charset=utf-8
5949 [main] DEBUG org.apache.hc.client5.http.headers - http-outgoing-1 << 
Content-Language: en
5949 [main] DEBUG org.apache.hc.client5.http.headers - http-outgoing-1 << 
Content-Length: 437
5949 [main] DEBUG org.apache.hc.client5.http.headers - http-outgoing-1 << Date: 
Wed, 27 May 2020 12:15:29 GMT
5949 [main] DEBUG org.apache.hc.client5.http.headers - http-outgoing-1 << 
Keep-Alive: timeout=300
5949 [main] DEBUG org.apache.hc.client5.http.headers - http-outgoing-1 << 
Connection: keep-alive
5952 [main] DEBUG org.apache.hc.client5.http.impl.classic.MainClientExec - 
ex-0002: connection can be kept alive for 300 SECONDS
5954 [main] DEBUG org.apache.hc.client5.http.impl.auth.HttpAuthenticator

Re: maxSwallowSize and misbehaving clients (e.g., mod_proxy_http)

[Osipov, Michael]

Am 2020-05-27 um 10:51 schrieb Mark Thomas:

On 22/05/2020 22:59, Osipov, Michael wrote:




I found one issue with HttpClient and Tomcat via HTTP/1.1. I have
decrypted the TLS traffic [1]. I can see that HttpClient sends the
headers also with a 4 KiB large chunk of the ZIP file. In return a
Tomcats send the 401 response with:

Keep-Alive: timeout=300
Connection: keep-alive


The client keeps sending in 8 KiB large blocks. After 2 134 016 written
bytes there is a TLS alert: Close Notify. A few packets later: RST.

I would expect to see here: "Connection: close".


Why?

Does the client provide a content-length header and no expectation? If
so, Tomcat could determine that the maxSwallowSize is going to be
exceeded and close the connection early.


Yes, it does. CL present and not expectation. A transcript is here [1].
We have already a working patch for the early response, also Oleg is 
already working on the early close not happening which is a bug in 
HttpClient.


As for the connection header: I do not understand why you assume the 
header is no necessary. Your reasoning is that if the server closes the 
physical connection before the client has spooled the entire body and it 
is imperative for the client to close the connection too regard of the 
presense of the header?


Michael

[1] 
https://www.mail-archive.com/httpclient-users@hc.apache.org/msg09927.html


-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: maxSwallowSize and misbehaving clients (e.g., mod_proxy_http)

[Osipov, Michael]

Am 2020-05-22 um 13:26 schrieb Mark Thomas:

On 21/05/2020 23:30, Osipov, Michael wrote:




Output will be sent privately.


Got it. Tx.

Looking at the direct case.

It looks like you have debug logging enabled for everything. You only
need it for the org.apache.coyote.http2 package.

grep "http2" catalina.2020-05-21.log | less

gives a nice clear view of what is happening which is roughly:

22:34:29 Client sends request headers and ~64k of body on stream 1
22:34:29 Server sends 401 response and resets stream 1
22:34:29 Client resends request with authorization header but NO body
22:34:31 Server tries to read request body
22:34:51 Server times out trying to read request body
22:34:51 Server sends 500 response due to timeout and resets stream 3

It looks like the response to stream 3 includes an authorization challenge.

I think something isn't quite right with the authentication process.

Which authenticator are you using?

Would expect to see a second challenge form the server or should the
client be authenticated once the first auth header has been processed?

What is triggering the read of the body here?

Why isn't the client sending the body?


I have made some progress on the topic with curl and HttpClient.

RFC 7230, 6.5 says:

   A client sending a message body SHOULD monitor the network connection
   for an error response while it is transmitting the request.  If the
   client sees a response that indicates the server does not wish to
   receive the message body and is closing the connection, the client
   SHOULD immediately cease transmitting the body and close its side of
   the connection.


curl perfectly manages the situation of 6.5 by using non-blocking I/O 
[1]. As for HttpClient, it does not handle this because of [2], [3], 
blocking I/O limitation. It then suffers from section 6.6:



   If a server performs an immediate close of a TCP connection, there is
   a significant risk that the client will not be able to read the last
   HTTP response.  If the server receives additional data from the
   client on a fully closed connection, such as another request that was
   sent by the client before receiving the server's response, the
   server's TCP stack will send a reset packet to the client;
   unfortunately, the reset packet might erase the client's
   unacknowledged input buffers before they can be read and interpreted
   by the client's HTTP parser.


So I at least believe that according to 6.5 Tomcat has to include 
"Connection: close" on the response.


As for the HTTP/2 issues, I will create separate threads when I have 
gathered the appropriate data.


Michael

[1] https://curl.haxx.se/mail/archive-2020-05/0054.html
[2] https://stackoverflow.com/a/9176445/696632
[3] 
http://mail-archives.apache.org/mod_mbox/hc-httpclient-users/202005.mbox/%3Cc43e2056-a95c-2070-3b9c-dd081a5f5235%40apache.org%3E


-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: maxSwallowSize and misbehaving clients (e.g., mod_proxy_http)

[Osipov, Michael]

Am 2020-05-22 um 18:51 schrieb [ext] Osipov, Michael:


Am 2020-05-22 um 13:26 schrieb Mark Thomas:

On 21/05/2020 23:30, Osipov, Michael wrote:




Output will be sent privately.


Got it. Tx.

Looking at the direct case.

It looks like you have debug logging enabled for everything. You only
need it for the org.apache.coyote.http2 package.

grep "http2" catalina.2020-05-21.log | less

gives a nice clear view of what is happening which is roughly:

22:34:29 Client sends request headers and ~64k of body on stream 1
22:34:29 Server sends 401 response and resets stream 1
22:34:29 Client resends request with authorization header but NO body
22:34:31 Server tries to read request body
22:34:51 Server times out trying to read request body
22:34:51 Server sends 500 response due to timeout and resets stream 3

It looks like the response to stream 3 includes an authorization 
challenge.


I think something isn't quite right with the authentication process.

Which authenticator are you using?


I am using my SPNEGO authenticator which is publically available [1] and 
based on the code which I have donated many years ago.



Would expect to see a second challenge form the server or should the
client be authenticated once the first auth header has been processed?


No, from a server's perspective the authentication has been completed 
already, but the client failed to provide the body.



What is triggering the read of the body here?

 >

Why isn't the client sending the body?


That's a good question. I need to ask fellow curl committers, this maybe 
a bug in curl.


I am looking at this with another client now, HttpClient 5.0. Even w/o 
authentication. the situation is even worse:

...


I found one issue with HttpClient and Tomcat via HTTP/1.1. I have 
decrypted the TLS traffic [1]. I can see that HttpClient sends the 
headers also with a 4 KiB large chunk of the ZIP file. In return a 
Tomcats send the 401 response with:

> Keep-Alive: timeout=300
> Connection: keep-alive

The client keeps sending in 8 KiB large blocks. After 2 134 016 written 
bytes there is a TLS alert: Close Notify. A few packets later: RST.


I would expect to see here: "Connection: close".

If necessary, I can provide the pcap and the keylog file.


[1] https://github.com/neykov/extract-tls-secrets

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: maxSwallowSize and misbehaving clients (e.g., mod_proxy_http)

[Osipov, Michael]

Am 2020-05-22 um 13:26 schrieb Mark Thomas:

On 21/05/2020 23:30, Osipov, Michael wrote:




Output will be sent privately.


Got it. Tx.

Looking at the direct case.

It looks like you have debug logging enabled for everything. You only
need it for the org.apache.coyote.http2 package.

grep "http2" catalina.2020-05-21.log | less

gives a nice clear view of what is happening which is roughly:

22:34:29 Client sends request headers and ~64k of body on stream 1
22:34:29 Server sends 401 response and resets stream 1
22:34:29 Client resends request with authorization header but NO body
22:34:31 Server tries to read request body
22:34:51 Server times out trying to read request body
22:34:51 Server sends 500 response due to timeout and resets stream 3

It looks like the response to stream 3 includes an authorization challenge.

I think something isn't quite right with the authentication process.

Which authenticator are you using?


I am using my SPNEGO authenticator which is publically available [1] and 
based on the code which I have donated many years ago.



Would expect to see a second challenge form the server or should the
client be authenticated once the first auth header has been processed?


No, from a server's perspective the authentication has been completed 
already, but the client failed to provide the body.



What is triggering the read of the body here?

>

Why isn't the client sending the body?


That's a good question. I need to ask fellow curl committers, this maybe 
a bug in curl.


I am looking at this with another client now, HttpClient 5.0. Even w/o 
authentication. the situation is even worse:



69 [main] DEBUG org.apache.hc.client5.http.impl.classic.InternalHttpClient - 
ex-0001: preparing request execution
145 [main] DEBUG org.apache.hc.client5.http.protocol.RequestAddCookies - Cookie 
spec selected: strict
150 [main] DEBUG org.apache.hc.client5.http.protocol.RequestAuthCache - Auth 
cache not set in the context
150 [main] DEBUG org.apache.hc.client5.http.impl.classic.ProtocolExec - 
ex-0001: target auth state: UNCHALLENGED
151 [main] DEBUG org.apache.hc.client5.http.impl.classic.ProtocolExec - 
ex-0001: proxy auth state: UNCHALLENGED
151 [main] DEBUG org.apache.hc.client5.http.impl.classic.ConnectExec - ex-0001: 
acquiring connection with route {s}->https://:1
151 [main] DEBUG org.apache.hc.client5.http.impl.classic.InternalHttpClient - 
ex-0001: acquiring endpoint (3 MINUTES)
152 [main] DEBUG org.apache.hc.client5.http.impl.io.PoolingHttpClientConnectionManager - 
ex-0001: endpoint lease request (3 MINUTES) [route: 
{s}->https://:1][total available: 0; route allocated: 0 of 5; 
total allocated: 0 of 25]
175 [main] DEBUG org.apache.hc.client5.http.impl.io.PoolingHttpClientConnectionManager - 
ex-0001: endpoint leased [route: {s}->https://:1][total 
available: 0; route allocated: 1 of 5; total allocated: 1 of 25]
183 [main] DEBUG 
org.apache.hc.client5.http.impl.io.PoolingHttpClientConnectionManager - 
ex-0001: acquired ep-
183 [main] DEBUG org.apache.hc.client5.http.impl.classic.InternalHttpClient - 
ex-0001: acquired endpoint ep-
183 [main] DEBUG org.apache.hc.client5.http.impl.classic.ConnectExec - ex-0001: 
opening connection {s}->https://:1
184 [main] DEBUG org.apache.hc.client5.http.impl.classic.InternalHttpClient - 
ep-: connecting endpoint (3 MINUTES)
184 [main] DEBUG 
org.apache.hc.client5.http.impl.io.PoolingHttpClientConnectionManager - ep-: 
connecting endpoint to https://:1 (3 MINUTES)
201 [main] DEBUG org.apache.hc.client5.http.impl.io.DefaultHttpClientConnectionOperator - 
http-outgoing-0: connecting to /:1
201 [main] DEBUG org.apache.hc.client5.http.ssl.SSLConnectionSocketFactory - Connecting 
socket to /:1 with timeout 3 MINUTES
237 [main] DEBUG org.apache.hc.client5.http.ssl.SSLConnectionSocketFactory - 
Enabled protocols: [TLSv1.3, TLSv1.2]
237 [main] DEBUG org.apache.hc.client5.http.ssl.SSLConnectionSocketFactory - 
Enabled cipher suites:[TLS_AES_256_GCM_SHA384, TLS_AES_128_GCM_SHA256, 
TLS_CHACHA20_POLY1305_SHA256, TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384, 
TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256, 
TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256, 
TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384, 
TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256, 
TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256, TLS_DHE_RSA_WITH_AES_256_GCM_SHA384, 
TLS_DHE_RSA_WITH_CHACHA20_POLY1305_SHA256, TLS_DHE_DSS_WITH_AES_256_GCM_SHA384, 
TLS_DHE_RSA_WITH_AES_128_GCM_SHA256, TLS_DHE_DSS_WITH_AES_128_GCM_SHA256, 
TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384, 
TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256, 
TLS_DHE_RSA_WITH_AES_256_CBC_SHA256, TLS_DHE_DSS_WITH_AES_256_CBC_SHA256, 
TLS_DHE_RSA_WITH_AES_128_CBC_SHA256, TLS_DHE_DSS_WITH_AES_128_CBC_SHA256, 
TLS_ECDH_ECDSA_WITH_AES_256_GCM_SHA384, TLS_ECDH_RS

Re: maxSwallowSize and misbehaving clients (e.g., mod_proxy_http)

[Osipov, Michael]

Am 2020-05-21 um 23:36 schrieb Mark Thomas:

On 21/05/2020 21:45, Osipov, Michael wrote:



Am 2020-05-21 um 11:02 schrieb Mark Thomas:

On 20/05/2020 17:57, Osipov, Michael wrote:


Is there an option to proxy via HTTP/2? That should be able to handle
these sorts of situations more gracefully.


This seems to be broken for some reason in another way:





What now?


Turn on debug logging for o.a.coyote.http2 for both.

For httpd, disable/relax the overhead protection although it would we
worth looking at what httpd is doing to trigger that.


I did now the tests with h2/h2c direct and via proxy. Will send output
privately.

What explicit overhead protection in HTTPd are you referring?
mod_proxy_http2 does not provide any options. Please note that clients
talk to HTTPd via HTTP/1.1 only.

Plese tell the exact tunable and I will retry.


Sorry, I meant for the httpd as a proxy test, relax/disable Tomcat's
HTTP/2 overhead protection.


Done. Set overheadCountFactor="0".

$ time curl -X POST  --anyauth -u : --upload-file inputs.zip -H "Content-Type: application/zip" 
https:///content-dev/api/documents --verbose -H "Expect:"  --trace trace.proxy
Warning: --trace overrides an earlier trace/verbose option


503 Service Unavailable

Service Unavailable
The server is temporarily unable to service your
request due to maintenance downtime or capacity
problems. Please try again later.

Apache Server at  Port 443


real0m53,683s
user0m9,948s
sys 0m0,821s


The execution time has exploded for real. Direct was with HTTP/1.1 6 
seconds, via proxy 12 s, now a minute.


The worst I can see is that requests are physically replayed twice:

2020-05-22T00:11:29 WARNUNG [http-apr-127.0.1.2-8083-exec-9] 
net.sf.michaelo.tomcat.authenticator.SpnegoAuthenticator.doAuthenticate The 
Negotiate (SPNEGO) authentication token is invalid: YIIRjAY...
GSSException: Failure unspecified at GSS-API level (Mechanism level: 
Request is a replay (34))
2020-05-22T00:11:55 WARNUNG [http-apr-127.0.1.2-8083-exec-5] 
net.sf.michaelo.tomcat.authenticator.SpnegoAuthenticator.doAuthenticate The 
Negotiate (SPNEGO) authentication token is invalid: YIIRjQYGKwYBBQU...
GSSException: Failure unspecified at GSS-API level (Mechanism level: 
Request is a replay (34))


Output will be sent privately.

Thanks,

Michael

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: maxSwallowSize and misbehaving clients (e.g., mod_proxy_http)

[Osipov, Michael]

Am 2020-05-21 um 11:02 schrieb Mark Thomas:

On 20/05/2020 17:57, Osipov, Michael wrote:


Is there an option to proxy via HTTP/2? That should be able to handle
these sorts of situations more gracefully.


This seems to be broken for some reason in another way:





What now?


Turn on debug logging for o.a.coyote.http2 for both.

For httpd, disable/relax the overhead protection although it would we
worth looking at what httpd is doing to trigger that.


I did now the tests with h2/h2c direct and via proxy. Will send output 
privately.


What explicit overhead protection in HTTPd are you referring? 
mod_proxy_http2 does not provide any options. Please note that clients 
talk to HTTPd via HTTP/1.1 only.


Plese tell the exact tunable and I will retry.

Regards,

Michael

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: maxSwallowSize and misbehaving clients (e.g., mod_proxy_http)

[Osipov, Michael]

Am 2020-05-20 um 15:07 schrieb Mark Thomas:

On 20/05/2020 11:05, Osipov, Michael wrote:




My questions are:
* Why is parseRequestLine printing the request body?


Because it is the contents of the input buffer that is being logged, not
just the headers.


* Can someone confirm the erratic behavior of HTTPd's module?


I'm not sure it is erratic. httpd is writing a request and Tomcat closes
the connection before httpd has finished. Most HTTP user agents don't
try reading the response until they have finished writing the request.
That is what appears to be happening here.


It seems that it is not ideal, far from it. When I look at the case with 
curl and Tomcat directly, curl handles the "premature" close perfectly 
as you can see with:

NEGOTIATE send, close instead of sending 7598613 bytes



maxSwallowSize was created to address this.


* Is there anything we need to improve on the Tomcat side?


I don't see anything Tomcat can do differently here.

Tomcat has to close the connection if it isn't going to read/swallow the
rest of the request body. There is no other choice.


I didn't even doubt that there is something wrong with Tomcat. I thougt 
there might be some more gracefully way. E.g., detect that client is 
still sending. Just a wild guess.



You could set maxSwallowSize to -1 and rely on httpd to enforce upload
limits. Looking at the httpd docs that might be easier said than done
when mod_proxy is used.

It is beginning to look like you need to pick a maxSwallowSize that is
high enough for your users but low enough to prevent DoS.


Obviously :-(


I guess mod_proxy could try to read the response sooner but it doesn't
know it needs to that until the connection is dropped by which time it
is too late. And how to ensure mod_proxy has read the response before
dropping the connection? I can of ways around this but they are all more
complex and that may impact.


There must be a way, does't it? If can do too?


Is there an option to proxy via HTTP/2? That should be able to handle
these sorts of situations more gracefully.


This seems to be broken for some reason in another way:

I have enabled the UpgradeProtocol on both connectors, localhost and 
remote IP. Remote IP is:

$ curl -X POST  --anyauth -u : --upload-file inputs.zip -H "Content-Type: application/zip" 
https://:1/content-dev/api/documents --verbose -H "Expect:"
* Uses proxy env variable NO_PROXY == 'localhost .siemens.net .siemens.com 
.siemens.de'
*   Trying :1...
* Connected to  () port 1 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: none
  CApath: /etc/ssl/certs/
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server accepted to use h2
* Server certificate:
*  subject: C=DE; O=Siemens; OU=LDA DW; CN=
*  start date: Mar 12 09:50:16 2020 GMT
*  expire date: Mar 19 13:10:13 2021 GMT
*  subjectAltName: host "" matched cert's ""
*  issuer: C=DE; ST=Bayern; L=Muenchen; O=Siemens; serialNumber=ZZB7; 
OU=Siemens Trust Center; CN=Siemens Issuing CA Intranet Server 2017
*  SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x800d91800)

POST /content-dev/api/documents HTTP/2
Host: :1
user-agent: curl/7.70.0
accept: */*
content-type: application/zip
content-length: 7664149


* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* old SSL session ID is stale, removing
* Connection state changed (MAX_CONCURRENT_STREAMS == 100)!
< HTTP/2 401
< www-authenticate: Negotiate
< content-type: text/html;charset=utf-8
< content-language: en
< content-length: 437
< date: Wed, 20 May 2020 16:31:37 GMT
* NEGOTIATE send, close instead of sending 7598614 bytes
* HTTP error before end of send, stop sending
<
* Ignoring the response-body
* Connection #0 to host  left intact
* Issue another request to this URL: 
'https://:1/content-dev/api/documents'
* Uses proxy env variable NO_PROXY == 'localhost .siemens.net .siemens.com 
.siemens.de'
* Found bundle for host : 0x800cba3c0 [can multiplex]
* Re-using existing connection! (#0) with host 
* Connected to  () port 1 (#0)
* Server auth using Negotiate with user ''
* Using Stream ID: 3 (easy handle 0x800d91800)

POST /content-dev/api/documents HTTP/2
Host: :1
authorization: Negotiate YIIRjQYGK...
user-agen

maxSwallowSize and misbehaving clients (e.g., mod_proxy_http)

[Osipov, Michael]

LSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* old SSL session ID is stale, removing
* Mark bundle as not supporting multiuse
< HTTP/1.1 401
< WWW-Authenticate: Negotiate
< Content-Type: text/html;charset=utf-8
< Content-Language: en
< Content-Length: 437
< Date: Wed, 20 May 2020 08:28:16 GMT
* NEGOTIATE send, close instead of sending 7598613 bytes
<
* Excess found: excess = 437 url = /content-dev/api/documents (zero-length body)
* Closing connection 0
* Issue another request to this URL: 
'https://:1/content-dev/api/documents'
* Uses proxy env variable NO_PROXY == 'localhost .siemens.net .siemens.com 
.siemens.de'
* Hostname  was found in DNS cache
*   Trying :1...
* Connected to  () port 1 (#1)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: none
  CApath: /etc/ssl/certs/
* SSL re-using session ID
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server did not agree to a protocol
* Server certificate:
*  subject: C=DE; O=Siemens; OU=LDA DW; CN=
*  start date: Mar 12 09:50:16 2020 GMT
*  expire date: Mar 19 13:10:13 2021 GMT
*  subjectAltName: host "" matched cert's ""
*  issuer: C=DE; ST=Bayern; L=Muenchen; O=Siemens; serialNumber=ZZB7; 
OU=Siemens Trust Center; CN=Siemens Issuing CA Intranet Server 2017
*  SSL certificate verify ok.
* Server auth using Negotiate with user ''

POST /content-dev/api/documents HTTP/1.1
Host: :1
Authorization: Negotiate YIIRjQYG^...
User-Agent: curl/7.70.0
Accept: */*
Content-Type: application/zip
Content-Length: 7664149


* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* old SSL session ID is stale, removing
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* old SSL session ID is stale, removing
* We are completely uploaded and fine
* Mark bundle as not supporting multiuse
< HTTP/1.1 201
< WWW-Authenticate: Negotiate oYG3MIG0o...
< vary: Origin
< Access-Control-Allow-Credentials: true
< Access-Control-Expose-Headers: 
Correlation-Id,Content-Disposition,Content-Length,Location
< Location: 
https://:1/content-dev/api/documents/68552d0c-8f0b-4631-a14a-614d7bd7469d
< Content-Length: 0
< Date: Wed, 20 May 2020 08:28:17 GMT
<
* Connection #1 to host  left intact


As you can see the request is failing via mod_proxy_http with a 502. 
After a few hours of investigation and reading log files it came to me 
that mod_proxy_http does not read the response from Tomcat while it 
sends the request body because Tomcat has already decided to abort the 
request with 401.
See log files: What suprises me that 
Http11InputBuffer.parseRequestLine() not only prints the request 
headers, but also the entire body (misbehavior of mod_proxy_http?!). As 
soon as I increase maxSwallowSize the symptom is gone until the new 
threshold.


Similar issues:
* https://bz.apache.org/bugzilla/show_bug.cgi?id=64016
* https://bz.apache.org/bugzilla/show_bug.cgi?id=61090
* https://bz.apache.org/bugzilla/show_bug.cgi?id=60717
* https://github.com/eclipse/jetty.project/issues/651

My questions are:
* Why is parseRequestLine printing the request body?
* Can someone confirm the erratic behavior of HTTPd's module?
* Is there anyting we need to improve on the Tomcat side?

The more I use HTTPd with mod_proxy_http the more I notice that the has 
been focus on websites and not real API usages.


Regards,

Michael

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Is ARM64 architecture officially supported ?

[Michael Osipov]

Am 2020-04-19 um 19:51 schrieb Martin Grigorov:

Hi Emilio,

On Fri, Apr 17, 2020 at 2:14 PM Emilio Fernandes <
emilio.fernande...@gmail.com> wrote:


Hola Tomcat community!

We consider using AWS Graviton [1] based instances which use ARM64
processors for our backend services.
I've googled around and found [2] saying that Tomcat is being tested on
ARM64 architecture at TravisCI! This is great!
Does this mean that Tomcat is officially supported on ARM64 ? I was not
able to find any specific documentation listing which platforms are
officially supported.

Does anyone from the community have any experience with Tomcat/HTTPD on
ARM64 in production ?



I work a lot with ARM64 servers lately!
So far we didn't face any issue related to ARM64 & Tomcat in our
application!
I've introduced the testing on TravisCI that you have found. In addition
for my daily job we have created a nightly test setup:
- build and test Tomcat 9.x (as at Travis)
- run some smoke tests with our application
- run some performance tests on x86_64 and ARM64 VMs with similar hardware
specs. The results for both architectures are very similar. One thing that
still bothers me is that the throughput on HTTPS is 3 times less than HTTP.


OpenSSL or SunJSSE?

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org