Re: Expect: 100-continue not working with curl and HTTP/2
Am 2019-03-29 um 22:07 schrieb Mark Thomas: On 29/03/2019 12:28, Michael Osipov wrote: Am 2019-03-29 um 12:14 schrieb Mark Thomas: On 28/03/2019 15:14, Osipov, Michael wrote: Hi folks, right away, I don't know whether it is us (Tomcat) or curl. I'd lke to narrow down the cause. It seems to be related to the use of kerberos. I don't see any errors when I provide the user name and password on the command line. I wonder how because it is only one roundtrip.. * Server auth using Negotiate with user '' The above looks odd. Shouldn't that show the user name being used? This is a curl shortcoming, but not a bug. It simply indicates that no user (-u :, known issue #10) has been provided. It does not obtain the GSS name from the default credential. I've got some a Kerberos test environment in some VMs. I'll spin that up and see if I can reproduce the issue. Great, let me know if you need further information or a test via HTTP/1.1. I can't reproduce this. I was using a Windows 7.64.1 client. verbose logging confirms both HTTP/2 and kerberos are being used. Maybe re-test with 7.64.1 ? It looks like a curl bug at this point although there are enough moving parts that it could be elsewhere. Oh well, I've got a step further I do think that there is a bug in curl and in Tomcat somewhere. I have updated the port to 7.64.1 and recompiled. It does work now, in fact it exactly behaves like the Windows version of curl 7.64.1. Two issues arise here: 1. Why did it fail with 7.64.0 and Tomcat while it worked with HTTPd?! The dependencies of curl did not change. 2. If you change update=false both on Windows and FreeBSD curl hangs forever and has to be killed. Make it update=true and it passes flawlessly. (assuming that version 003 is already deployed). Can you confirm? Do you need log outout? Michael - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Isapi_Redirect Issue Page Not Found 404.0
Issue: When trying to open a JSP page in IIS it appears that the isapi_redirect.dll cannot find the file to convert. I have checked several sights that confirm my configuration below but none could find that had this condition, Has anybody got this to work in the below configuration. Below is my configuration. Windows 2016 Server 64-bit IIS 10.0 Tomcat 9.0.16 Page Requests http:/// Tomcat Default Page http://:8080/ Tomcat Default Page http:///test.jsp 404.0 Page not found http:///iisstart.htm 404.0 Page not found Isapi Redirect Configuration [according to https://tomcat.apache.org/connectors-doc/webserver_howto/iis.html ] 1) Path: D:\Tomcat\isapi 2) Ran ACL on D:\Tomcat\isapi icacls "D:\Tomcat\isapi" /grant "IIS APPPOOL\DefaultAppPool":(OI)(CI)M 3) Copied isapi_redirect.dll (64-bit, 1.2.46) [Tried all versions all the way down to 27.] 4) Created isapi_redirect.properties file (have tried registry as well) - D:\Tomcat\isapi\isapi_redirect.properties extension_uri=/jakarta/isapi_redirect.dll log_file=D:\Tomcat\isapi\isapi_redirect.log log_level=trace worker_file=D:\Tomcat\isapi\workers.properties worker_mount_file=D:\Tomcat\isapi\uriworkermap.properties 5) Created workers.properties - D:\Tomcat\isapi\workers.properties worker.list=tomcat01 worker.tomcat01.type=ajp13 worker.tomcat01.host=localhost worker.tomcat01.port=8009 6) Created uriworkermap.properties - D:\Tomcat\isapi\ \*=tomcat01 7) Creatd Virtual Directory in IIS under Default Web Site Alias: jakarta path: D:\Tomcat\isapi 8) Select newly created virtual directory. (jakarta) - Selected ISAPI-dll (checked all to enable. 9) Selected Default Web Site - Isapi Filters - Filter Name: tomcat - Path: D:\Tomcat\isapi\isapi_redirect.dll 10) Selected Server --> ISAPI and CGI Restrictions - Isapi or CGI path: D:\Tomcat\isapi\isapi_redirect.dll - Description: tomcat - Checked Allow extension path to execute Notes Other Notes: - Application Pool - Integrated, V4.0 - Enable 32-bit Application = false - FailedReqLogFiles - 233. MODULE_SET_RESPONSE_ERROR_STATUS Warning ModuleName="IsapiModule", Notification="EXECUTE_REQUEST_HANDLER", HttpStatus="404", HttpReason="404", HttpSubStatus="0", ErrorCode="The operation completed successfully. (0x0)", ConfigExceptionInfo="" Thanks for any help in advance, been trying to fix this for a couple of weeks now. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Expect: 100-continue not working with curl and HTTP/2
Am 2019-03-29 um 22:07 schrieb Mark Thomas: On 29/03/2019 12:28, Michael Osipov wrote: Am 2019-03-29 um 12:14 schrieb Mark Thomas: On 28/03/2019 15:14, Osipov, Michael wrote: Hi folks, right away, I don't know whether it is us (Tomcat) or curl. I'd lke to narrow down the cause. It seems to be related to the use of kerberos. I don't see any errors when I provide the user name and password on the command line. I wonder how because it is only one roundtrip.. * Server auth using Negotiate with user '' The above looks odd. Shouldn't that show the user name being used? This is a curl shortcoming, but not a bug. It simply indicates that no user (-u :, known issue #10) has been provided. It does not obtain the GSS name from the default credential. I've got some a Kerberos test environment in some VMs. I'll spin that up and see if I can reproduce the issue. Great, let me know if you need further information or a test via HTTP/1.1. I can't reproduce this. I was using a Windows 7.64.1 client. verbose logging confirms both HTTP/2 and kerberos are being used. Maybe re-test with 7.64.1 ? It looks like a curl bug at this point although there are enough moving parts that it could be elsewhere. Can you provide provide the verbose log of curl as well as your curl --version output?! I will try to go down to h2c and see what I get. I do not really understand why it works with HTTPd directly. The flow is exactly the same as in Tomcat. I will try to narrow it down on Monday. Regards, Micheal - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat 8.5.39 failed to initialize connector
On Fri, Mar 29, 2019, 3:09 PM Mark Thomas wrote: > On 29/03/2019 18:52, Ethan Jensen wrote: > > On Thu, Mar 28, 2019 at 5:05 PM Mark Thomas wrote: > > > >> On 28/03/2019 17:18, Ethan Jensen wrote: > >>> On Thu, Mar 28, 2019 at 11:11 AM Mark Thomas wrote: > >> > >> > >> > Can you post the header of your private key file? It should look > something like: > > -BEGIN RSA PRIVATE KEY- > Proc-Type: 4,ENCRYPTED > DEK-Info: AES-256-CBC,D02DE734A8C2DBA625FC4180E7AECC78 > > Thanks, > > Mark > > > >>> Here you are: > >>> > >>> Bag Attributes > >>> localKeyID: 14 A3 77 23 14 44 3E 99 FD 7D A4 BE C3 4C 10 D0 DD 5A > DA > >> 0B > >>> friendlyName: mydomain.com > >>> Key Attributes: > >>> -BEGIN ENCRYPTED PRIVATE KEY- > >> > >> Bingo. That is a PKCS#8 format file that OpenSSL understands but JSSE > >> does not. The fix I had in mind does work. Now I understand why the > >> problem occurred and can confirm that the fix works I'll apply it for > >> the next release. A a workaround you can convert that private key to > >> PKCS#1 format. > >> > >> Mark > >> > >> > > Mark, > > > > I can confirm that this does work! I converted the key and when starting > > up Tomcat am greeted with this message in the log: > > > > ... > > 29-Mar-2019 14:43:30.865 INFO [main] > > org.apache.tomcat.util.net.openssl.OpenSSLUtil.getKeyManagers The > > certificate [conf/tls_config/20200411/star_mydomain_com.pem] or its > private > > key [conf/tls_config/20200411/star_mydomain_com.key] could not be > processed > > using a JSSE key manager and will be given directly to OpenSSL > > ... > > > > For future reference, can you share how you determined the key was in a > > PKCS#8 format? I had tried to ascertain that ahead of time, but didn't > see > > anything readily identifiable (to me), though I'm not terribly familiar > > with particular key formats and perhaps it was just a recognition thing > > (for you). > > I googled for "-BEGIN ENCRYPTED PRIVATE KEY-" > > Mark > > Haha, guess that'll do it. Thanks again. -- Ethan
Re: Tomcat 8 not responding - socket data not being read from TCP socket queue
On 29/03/2019 15:37, Inder wrote: > On Fri, Mar 29, 2019 at 1:21 PM Mark Thomas wrote: >> Its 8.5.20 > >>> INFO: Loaded APR based Apache Tomcat Native library [1.2.14] using APR >>> version [1.4.8]. >> >> Those are quite old. There is a fix that may be relevant to APR poll >> behaviour in APR 1.6.4. >> > OK... will consider upgrading or moving to NIO Connector. Do you see any > issue or comparable dis-advantage of using NIO connector? NIO + JSSE will be slower than NIO+OpenSSL. NIO+OpenSSL should be about the same as APR/Native. (You can construct performance 'tests' that show significant differences either way but in the real world Id expect them to be about the same.) >> What is the CPU utilisation like at this point? >> > Didn't take much note of that. Will check the next time issue happens. A spinning thread would confirm the analsysis. Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat 8.5.39 failed to initialize connector
On 29/03/2019 18:52, Ethan Jensen wrote: > On Thu, Mar 28, 2019 at 5:05 PM Mark Thomas wrote: > >> On 28/03/2019 17:18, Ethan Jensen wrote: >>> On Thu, Mar 28, 2019 at 11:11 AM Mark Thomas wrote: >> >> >> Can you post the header of your private key file? It should look something like: -BEGIN RSA PRIVATE KEY- Proc-Type: 4,ENCRYPTED DEK-Info: AES-256-CBC,D02DE734A8C2DBA625FC4180E7AECC78 Thanks, Mark >>> Here you are: >>> >>> Bag Attributes >>> localKeyID: 14 A3 77 23 14 44 3E 99 FD 7D A4 BE C3 4C 10 D0 DD 5A DA >> 0B >>> friendlyName: mydomain.com >>> Key Attributes: >>> -BEGIN ENCRYPTED PRIVATE KEY- >> >> Bingo. That is a PKCS#8 format file that OpenSSL understands but JSSE >> does not. The fix I had in mind does work. Now I understand why the >> problem occurred and can confirm that the fix works I'll apply it for >> the next release. A a workaround you can convert that private key to >> PKCS#1 format. >> >> Mark >> >> > Mark, > > I can confirm that this does work! I converted the key and when starting > up Tomcat am greeted with this message in the log: > > ... > 29-Mar-2019 14:43:30.865 INFO [main] > org.apache.tomcat.util.net.openssl.OpenSSLUtil.getKeyManagers The > certificate [conf/tls_config/20200411/star_mydomain_com.pem] or its private > key [conf/tls_config/20200411/star_mydomain_com.key] could not be processed > using a JSSE key manager and will be given directly to OpenSSL > ... > > For future reference, can you share how you determined the key was in a > PKCS#8 format? I had tried to ascertain that ahead of time, but didn't see > anything readily identifiable (to me), though I'm not terribly familiar > with particular key formats and perhaps it was just a recognition thing > (for you). I googled for "-BEGIN ENCRYPTED PRIVATE KEY-" Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Expect: 100-continue not working with curl and HTTP/2
On 29/03/2019 12:28, Michael Osipov wrote: > Am 2019-03-29 um 12:14 schrieb Mark Thomas: >> On 28/03/2019 15:14, Osipov, Michael wrote: >>> Hi folks, >>> >>> right away, I don't know whether it is us (Tomcat) or curl. I'd lke to >>> narrow down the cause. >> >> It seems to be related to the use of kerberos. I don't see any errors >> when I provide the user name and password on the command line. > > I wonder how because it is only one roundtrip.. > * Server auth using Negotiate with user '' >> >> The above looks odd. Shouldn't that show the user name being used? > > This is a curl shortcoming, but not a bug. It simply indicates that no > user (-u :, known issue #10) has been provided. It does not obtain the > GSS name from the default credential. > >> I've got some a Kerberos test environment in some VMs. I'll spin that up >> and see if I can reproduce the issue. > > Great, let me know if you need further information or a test via HTTP/1.1. I can't reproduce this. I was using a Windows 7.64.1 client. verbose logging confirms both HTTP/2 and kerberos are being used. Maybe re-test with 7.64.1 ? It looks like a curl bug at this point although there are enough moving parts that it could be elsewhere. Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat 8.5.39 failed to initialize connector
On Thu, Mar 28, 2019 at 5:05 PM Mark Thomas wrote: > On 28/03/2019 17:18, Ethan Jensen wrote: > > On Thu, Mar 28, 2019 at 11:11 AM Mark Thomas wrote: > > > > >> Can you post the header of your private key file? It should look > >> something like: > >> > >> -BEGIN RSA PRIVATE KEY- > >> Proc-Type: 4,ENCRYPTED > >> DEK-Info: AES-256-CBC,D02DE734A8C2DBA625FC4180E7AECC78 > >> > >> Thanks, > >> > >> Mark > >> > >> > > Here you are: > > > > Bag Attributes > > localKeyID: 14 A3 77 23 14 44 3E 99 FD 7D A4 BE C3 4C 10 D0 DD 5A DA > 0B > > friendlyName: mydomain.com > > Key Attributes: > > -BEGIN ENCRYPTED PRIVATE KEY- > > Bingo. That is a PKCS#8 format file that OpenSSL understands but JSSE > does not. The fix I had in mind does work. Now I understand why the > problem occurred and can confirm that the fix works I'll apply it for > the next release. A a workaround you can convert that private key to > PKCS#1 format. > > Mark > > Mark, I can confirm that this does work! I converted the key and when starting up Tomcat am greeted with this message in the log: ... 29-Mar-2019 14:43:30.865 INFO [main] org.apache.tomcat.util.net.openssl.OpenSSLUtil.getKeyManagers The certificate [conf/tls_config/20200411/star_mydomain_com.pem] or its private key [conf/tls_config/20200411/star_mydomain_com.key] could not be processed using a JSSE key manager and will be given directly to OpenSSL ... For future reference, can you share how you determined the key was in a PKCS#8 format? I had tried to ascertain that ahead of time, but didn't see anything readily identifiable (to me), though I'm not terribly familiar with particular key formats and perhaps it was just a recognition thing (for you). Thanks, Mark! Appreciate the assistance. -- Ethan
Re: Tomcat 8 not responding - socket data not being read from TCP socket queue
Many Thanks Mark for the super quick response. More details inline. On Fri, Mar 29, 2019 at 1:21 PM Mark Thomas wrote: > On 29/03/2019 01:40, Inder wrote: > > > > > INFO: Server version: > > INFO: Server built: unknown > > INFO: Server number: 8.5.x > > That doesn't look like a standard Tomcat build. What version of Tomcat > is this / is this based on? > > Its 8.5.20 > > INFO: Loaded APR based Apache Tomcat Native library [1.2.14] using APR > > version [1.4.8]. > > Those are quite old. There is a fix that may be relevant to APR poll > behaviour in APR 1.6.4. > > OK... will consider upgrading or moving to NIO Connector. Do you see any issue or comparable dis-advantage of using NIO connector? > > INFO: APR capabilities: IPv6 [true], sendfile [true], accept filters > > [false], random [true]. > > INFO: APR/OpenSSL configuration: useAprConnector [true], useOpenSSL > [true] > > INFO: OpenSSL successfully initialized [OpenSSL 1.0.2k-fips 26 Jan 2017] > > > > Mar 15, 2019 7:59:35 AM org.apache.coyote.AbstractProtocol init > > INFO: Initializing ProtocolHandler ["http-apr-8081"] > > Mar 15, 2019 7:59:35 AM org.apache.coyote.AbstractProtocol init > > INFO: Initializing ProtocolHandler > > ["https-openssl-apr-0:0:0:0:0:0:0:0-8443"] > > What is the connector configuration here? > > > > After some troubleshooting, we found that when this happens, Java worker > > threads are idle and waiting for tasks, while sockets are in ESTABLISHED > > and CLOSE_WAIT state. All the TCP sockets at the OS level indicate they > > have bytes in the TCP buffer that is not been read by the application. > > so looks like something went wrong, and the APR connector library is no > > more reading from the socket, or some issue between java socket Acceptor > > call and APR library. > > What is the CPU utilisation like at this point? > Didn't take much note of that. Will check the next time issue happens. > > > Has anyone encountered any problem like this or have any suggestion on > what > > should be checked that may help to isolate the problem. > > Possibly: > https://tomcat.markmail.org/thread/4wdcgluibejormye Thanks... Looks like I hit into the same issue. I think if I change the Connector configuration to use NIO, I should not hit into this issue. Any idea what causes the issue above? Is that some race condition scenario? > > > > "https-openssl-apr-0:0:0:0:0:0:0:0-8443-Poller" #404 daemon prio=5 > > os_prio=0 tid=0x7fe938168320 nid=0x2991 runnable [0x7fe661063000] > >java.lang.Thread.State: RUNNABLE > > at > > org.apache.tomcat.util.net.AprEndpoint$Poller.run(AprEndpoint.java:1642) > > at java.lang.Thread.run(Thread.java:748) > > This is consistent with the problem described above. > > It looks like a Tomcat upgrade to at least 8.5.30 should address this. I > recommend the latest 8.5.x > > OK... Will consider the upgrade. Many Thanks again for the quick inputs and pointers. > Mark > > - > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > >
Re: Expect: 100-continue not working with curl and HTTP/2
Am 2019-03-29 um 12:14 schrieb Mark Thomas: On 28/03/2019 15:14, Osipov, Michael wrote: Hi folks, right away, I don't know whether it is us (Tomcat) or curl. I'd lke to narrow down the cause. It seems to be related to the use of kerberos. I don't see any errors when I provide the user name and password on the command line. I wonder how because it is only one roundtrip.. * Server auth using Negotiate with user '' The above looks odd. Shouldn't that show the user name being used? This is a curl shortcoming, but not a bug. It simply indicates that no user (-u :, known issue #10) has been provided. It does not obtain the GSS name from the default credential. I've got some a Kerberos test environment in some VMs. I'll spin that up and see if I can reproduce the issue. Great, let me know if you need further information or a test via HTTP/1.1. Michael - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Expect: 100-continue not working with curl and HTTP/2
On 28/03/2019 15:14, Osipov, Michael wrote: > Hi folks, > > right away, I don't know whether it is us (Tomcat) or curl. I'd lke to > narrow down the cause. It seems to be related to the use of kerberos. I don't see any errors when I provide the user name and password on the command line. >> * Server auth using Negotiate with user '' The above looks odd. Shouldn't that show the user name being used? >> * Using Stream ID: 3 (easy handle 0x800d65000) >> } [5 bytes data] >> > PUT > /backend-dev/manager-1/text/deploy?path=/backend-dev&update=false&version=003 > HTTP/2 >> > Host: sitex-ldadw.ad001.siemens.net:8445 >> > Authorization: Negotiate YIISIwYGKwY... >> > User-Agent: curl/7.64.0 >> > Accept: */* >> > Expect: 100-continue >> > Content-Length: 6502195 >> > >> } [0 bytes data] >> 0 6349k 0 0 0 0 0 0 --:--:-- --:--:-- > --:--:-- 0< HTTP/2 100 >> } [0 bytes data] >> 0 6349k 0 0 0 0 0 0 --:--:-- 0:00:20 > --:--:-- 0< HTTP/2 200 >> < cache-control: private >> < expires: Thu, 01 Jan 1970 00:00:00 GMT >> < www-authenticate: Negotiate oYHzMIHwo... >> < x-content-type-options: nosniff >> < content-type: text/plain;charset=utf-8 >> < date: Thu, 28 Mar 2019 13:48:39 GMT >> < >> } [0 bytes data] >> * HTTP/2 stream 0 was not closed cleanly: ENHANCE_YOUR_CALM (err 11) That is a result of the timeout. I've got some a Kerberos test environment in some VMs. I'll spin that up and see if I can reproduce the issue. Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat 8 not responding - socket data not being read from TCP socket queue
On 29/03/2019 01:40, Inder wrote: > INFO: Server version: > INFO: Server built: unknown > INFO: Server number: 8.5.x That doesn't look like a standard Tomcat build. What version of Tomcat is this / is this based on? > INFO: Loaded APR based Apache Tomcat Native library [1.2.14] using APR > version [1.4.8]. Those are quite old. There is a fix that may be relevant to APR poll behaviour in APR 1.6.4. > INFO: APR capabilities: IPv6 [true], sendfile [true], accept filters > [false], random [true]. > INFO: APR/OpenSSL configuration: useAprConnector [true], useOpenSSL [true] > INFO: OpenSSL successfully initialized [OpenSSL 1.0.2k-fips 26 Jan 2017] > Mar 15, 2019 7:59:35 AM org.apache.coyote.AbstractProtocol init > INFO: Initializing ProtocolHandler ["http-apr-8081"] > Mar 15, 2019 7:59:35 AM org.apache.coyote.AbstractProtocol init > INFO: Initializing ProtocolHandler > ["https-openssl-apr-0:0:0:0:0:0:0:0-8443"] What is the connector configuration here? > After some troubleshooting, we found that when this happens, Java worker > threads are idle and waiting for tasks, while sockets are in ESTABLISHED > and CLOSE_WAIT state. All the TCP sockets at the OS level indicate they > have bytes in the TCP buffer that is not been read by the application. > so looks like something went wrong, and the APR connector library is no > more reading from the socket, or some issue between java socket Acceptor > call and APR library. What is the CPU utilisation like at this point? > Has anyone encountered any problem like this or have any suggestion on what > should be checked that may help to isolate the problem. Possibly: https://tomcat.markmail.org/thread/4wdcgluibejormye > "https-openssl-apr-0:0:0:0:0:0:0:0-8443-Poller" #404 daemon prio=5 > os_prio=0 tid=0x7fe938168320 nid=0x2991 runnable [0x7fe661063000] >java.lang.Thread.State: RUNNABLE > at > org.apache.tomcat.util.net.AprEndpoint$Poller.run(AprEndpoint.java:1642) > at java.lang.Thread.run(Thread.java:748) This is consistent with the problem described above. It looks like a Tomcat upgrade to at least 8.5.30 should address this. I recommend the latest 8.5.x Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Thread pools
Hi Rajendra, On Fri, Mar 29, 2019 at 2:44 AM Rajendra Popuri wrote: > > Hi, > > Is any parameter to recycle Tomcat connector thread pools like Datasource > pools? Or we can only define maxThreads value to connector element. Please > advise. > Can you please elaborate on "recycle Tomcat connector thread pools" part? Exactly what you wanted to do, to purge idle threads or something else. There are different Connectors properties. Please see[1]. [1] https://tomcat.apache.org/tomcat-7.0-doc/config/http.html Thanks! Suvendu > Thanks, > Rajendra > -- > Thanks&Regards Rajendra - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
