Re: completely automated (for real) Let's Encrypt on embedded Tomcat
I'm coming into this conversation late, so what I say could be completely irrelevant, but when I recently set up an independent (i.e., not behind httpd) Tomcat server on one of our AWS EC2 instances, and could not get certbot to function at all, to save my life, I ended up using something called "LEGO." It *does* require one to shut the Tomcat server down during the renewal process (because it has to take over the port briefly), but it also *does* play nicely with a Tomcat server that's doing its own SSL. -- James H. H. Lampert - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: completely automated (for real) Let's Encrypt on embedded Tomcat
On 10/5/2020 2:42 PM, Christopher Schultz wrote: … Sure, it can contain S3 credentials and you can pick-up your key and certificate (or, better yet, the whole keystore) there, but at that point you have "moved" the problem outside of Tomcat, right? No, not at all. The major problems are: 1. Generating the certificate automatically. 2. Feeding the certificate to Tomcat automatically. If the "extra" thing I have to do is specify an environment variable with the name of the S3 bucket to use as a certificate state, then that is a teeny, tiny problem. That is not really even a problem. Can you imagine if in my spring boot application I could run it using "java -jar my-app.jar --cert-work-bucket my-bucket" and it would just go get a certificate automatically? You can have a "certificate renewal service" that writes to S3. I want it built into my application as a module. You just include the module, specify the domain name (and maybe a couple of other details needed by RFC 8555, such as a contact email), and boom, it all happens automatically. Why does it have to be more difficult than that? It shouldn't be. I suppose you could put that directly into Tomcat, but Tomcat is not likely to ship with an Amazon-specific feature built-into it. Read my original email. I never intended to put this directly into standalone Tomcat (although if you want to put it into Tomcat you're welcome to). I want to use this with an application running on embedded Tomcat. Spring Boot is a prime example. I'll just extend the Spring Boot embedded Tomcat module and extend/modify that as needed. If that requires modifying Tomcat code, fine. Another idea would be to use embedded Tomcat (or, at your suggestion, Spring Boot) and fetch the keystore from some "standard" location of your choosing. Again, that would be (appropriately, IMO) outside Tomcat code. That was always the intention. In my introduction to my original email I explained that the modern approach is moving away from something like standalone Tomcat to self-contained executables that run their own servers, whether embedded Tomcat or whatever. (I was late to the party, and even two years ago I wasn't getting it.) 1. Does acme4j allow me to verify my certificate behind another port? (e.g. ElasticBeanstalk deploys a JAR behind NGINX port 5000 by default. I'm still reading RFC 8555 to find out if the ACME server has to connect back on a certain port for verification.) 2. Once I have the Let's Encrypt certificate, can I convert to PKCS12 for Tomcat completely in the application without shelling out to openssl or keytool? I'm hoping Bouncy Castle and/or acme4j-util will allow me to do that. This can be done, but it's non-trivial. For example, Tomcat contains code to package PEM-encoded DER files (good old OpenSSL-style =BEGIN CERTIFICATE= things) into an in-memory keystore to configure JSSE. It seems like it would be straightforward, but it turns out not to be in all cases. YMMV. Non-trivial as it may be, it /only needs to be done once/. If I have a converter, then I can use it a thousand times. A million times. And suddenly the deployment becomes a piece of cake. In reality, today's style of handling SSL is what matches your description of "It seems like it would be straightforward, but it turns out not to be …". So why do we keep doing all this difficult, manual, tedious, not-trivial stuff to deploy certificates the hard way, when we could put our efforts into a single no-trivial task of making a converter so that Tomcat can use the Let's Encrypt certificates directly? We do it once. It's hard, but then everything else is easy. I don't get why we want to spend another decade doing it the hard way when we can spend one year on a different hard task and then do SSL the easy way for the other nine. (The frustration isn't directed at you. It's just in general in software development I see the industry—why does the most basic of things have to be so difficult?) 3. Once I have the PKCS12, how do I feed it to the embedded Tomcat? If it's a file on the disk, it's easy: just use the path. Can I pass it in memory? If not, why not? How is memory less accessible than a file? Chris, where can I get more information on the latter questions about getting this certificate to Tomcat once I have it? This mailing list is a good place to start (and likely finish). Of course I really super-appreciate the help on this mailing list, and I'll be asking lots more questions. But I also don't want to ask things you've already answered elsewhere. I thought sure in one of your ApacheCon 2019 presentations you mentioned you had made more progress than the ApacheCon 2018 slides, such as auto-reloading or something. If there is further documentation let me know. … Go back to my presentation on Let's Encrypt and you'll see how to use openssl to convert to a keystore if that's what you want. I want
Re: completely automated (for real) Let's Encrypt on embedded Tomcat
Garret, On 10/5/20 12:21, Garret Wilson wrote: > Thank you so much for replying, Chris. Responses below. > > On 10/5/2020 8:53 AM, Christopher Schultz wrote: >> Microservices won't work the way you want with Let's Encrypt. You have >> two options: >> >> 1. Hit Let's Encrypt every time you launch a new instance of the >> microservice to deploy a new certificate >> >> 2. Handle the certificate provisioning elsewhere (e.g. ELB) >> >> #1 just won't work. LE won't re-issue a certificate more frequently than >> every 6 weeks or something like that. So that really leaves you with #2. > > It's good to know about the six-week limit, but you discarded #1 too > quickly. Can't the microservice simply store the credentials in S3 or > one of a hundred other data stores? (Note that I care less about > "microservices" as such at the moment. I just want a turnkey deployment > of a single application for now. But the idea is the same.) Sure, it can contain S3 credentials and you can pick-up your key and certificate (or, better yet, the whole keystore) there, but at that point you have "moved" the problem outside of Tomcat, right? You can have a "certificate renewal service" that writes to S3. I suppose you could put that directly into Tomcat, but Tomcat is not likely to ship with an Amazon-specific feature built-into it. I could imagine a LifecycleListener which has been written with this kind of thing in mind. (I'm not sure that a LifecycleListener would be able to intervene early enough in the process of a non-embedded Tomcat startup to do this, btw; just an idea.) Another idea would be to use embedded Tomcat (or, at your suggestion, Spring Boot) and fetch the keystore from some "standard" location of your choosing. Again, that would be (appropriately, IMO) outside Tomcat code. >> What you really want is for the orchestrator to provide the certificate >> and key to the nodes as they come on-line. > > Look these "orchestrators" are a configuration nightmare at the moment. > They end up being worse than my just configuring a CentOS machine from > scratch. Plus I have to pay for all those extra services. Read > https://leebriggs.co.uk/blog/2019/04/13/the-fargate-illusion.html and > try not to shudder. > >> So instead of trying to get LE to work with Tomcat (which does work, but >> requires some care and feeding), maybe we should try to get Tomcat to >> load its crypto material from other places. > > There is already a Java library (https://github.com/shred/acme4j) for > talking to Let's Encrypt. It sounds like it does everything I need. I'll > need to investigate more, but here are my initial doubts: > > 1. Does acme4j allow me to verify my certificate behind another port? > (e.g. ElasticBeanstalk deploys a JAR behind NGINX port 5000 by > default. I'm still reading RFC 8555 to find out if the ACME server > has to connect back on a certain port for verification.) > 2. Once I have the Let's Encrypt certificate, can I convert to PKCS12 > for Tomcat completely in the application without shelling out to > openssl or keytool? I'm hoping Bouncy Castle and/or acme4j-util will > allow me to do that. This can be done, but it's non-trivial. For example, Tomcat contains code to package PEM-encoded DER files (good old OpenSSL-style =BEGIN CERTIFICATE= things) into an in-memory keystore to configure JSSE. It seems like it would be straightforward, but it turns out not to be in all cases. YMMV. > 3. Once I have the PKCS12, how do I feed it to the embedded Tomcat? If it's a file on the disk, it's easy: just use the path. > Chris, where can I get more information on the latter questions about > getting this certificate to Tomcat once I have it? This mailing list is a good place to start (and likely finish). >> One way to do that would be with e.g. Amazon's key storage service. I'm >> not familiar with that. I know it can store various types of keys; not >> sure about certificates and if we can pull a private key out of it. > > I think your idea of storing this stuff elsewhere is a good first > stepping stone to get to where I want to go. > > Just to get the ball rolling, I could manually run some Let's Encrypt > client, and then store the certificate in S3. Then the first component I > would write would be steps #2 and #3 above. I am already familiar with > the AWS Java library, so I could quickly figure out how to pull the > certificates off S3. But I need your help in finding out how to convert > them to PKCS12 (or whatever; this is new territory for me) on the fly > and feed them to the embedded Tomcat. Go back to my presentation on Let's Encrypt and you'll see how to use openssl to convert to a keystore if that's what you want. Or, better yet, skip that step entirely and use the PEM-encoded DER files that Let's Encrypt already provides to you. You just have to work-out file-permissions issues. >> Honestly, your best bet would probably be to use ELB and just pay for >> it. You only pay for
Re: completely automated (for real) Let's Encrypt on embedded Tomcat
Thank you so much for replying, Chris. Responses below. On 10/5/2020 8:53 AM, Christopher Schultz wrote: Microservices won't work the way you want with Let's Encrypt. You have two options: 1. Hit Let's Encrypt every time you launch a new instance of the microservice to deploy a new certificate 2. Handle the certificate provisioning elsewhere (e.g. ELB) #1 just won't work. LE won't re-issue a certificate more frequently than every 6 weeks or something like that. So that really leaves you with #2. It's good to know about the six-week limit, but you discarded #1 too quickly. Can't the microservice simply store the credentials in S3 or one of a hundred other data stores? (Note that I care less about "microservices" as such at the moment. I just want a turnkey deployment of a single application for now. But the idea is the same.) What you really want is for the orchestrator to provide the certificate and key to the nodes as they come on-line. Look these "orchestrators" are a configuration nightmare at the moment. They end up being worse than my just configuring a CentOS machine from scratch. Plus I have to pay for all those extra services. Read https://leebriggs.co.uk/blog/2019/04/13/the-fargate-illusion.html and try not to shudder. So instead of trying to get LE to work with Tomcat (which does work, but requires some care and feeding), maybe we should try to get Tomcat to load its crypto material from other places. There is already a Java library (https://github.com/shred/acme4j) for talking to Let's Encrypt. It sounds like it does everything I need. I'll need to investigate more, but here are my initial doubts: 1. Does acme4j allow me to verify my certificate behind another port? (e.g. ElasticBeanstalk deploys a JAR behind NGINX port 5000 by default. I'm still reading RFC 8555 to find out if the ACME server has to connect back on a certain port for verification.) 2. Once I have the Let's Encrypt certificate, can I convert to PKCS12 for Tomcat completely in the application without shelling out to openssl or keytool? I'm hoping Bouncy Castle and/or acme4j-util will allow me to do that. 3. Once I have the PKCS12, how do I feed it to the embedded Tomcat? Chris, where can I get more information on the latter questions about getting this certificate to Tomcat once I have it? One way to do that would be with e.g. Amazon's key storage service. I'm not familiar with that. I know it can store various types of keys; not sure about certificates and if we can pull a private key out of it. I think your idea of storing this stuff elsewhere is a good first stepping stone to get to where I want to go. Just to get the ball rolling, I could manually run some Let's Encrypt client, and then store the certificate in S3. Then the first component I would write would be steps #2 and #3 above. I am already familiar with the AWS Java library, so I could quickly figure out how to pull the certificates off S3. But I need your help in finding out how to convert them to PKCS12 (or whatever; this is new territory for me) on the fly and feed them to the embedded Tomcat. Honestly, your best bet would probably be to use ELB and just pay for it. You only pay for data-transfer, so a dormant ELB costs you virtually nothing. Last month I deployed a test application on Elastic Beanstalk on a domain nobody knows about just to see how it worked. The ELB cost me $16 in a month with basically nobody using it! That adds up quickly. I have several little apps I want to toss up. See also my question https://serverfault.com/q/1036276 . Instead of spinning-up an EC2 instance for your service, maybe you should be looking at Lambda instead. You can probably get your costs down more that way than trying to eliminate the ELB. I want to drop either a self-contained JAR file or a Docker image somewhere, and have it immediately start running with SSL support, without my configuring a VM or running scripts. When I have a new version, I want to drop a new JAR or Docker image and have it automatically replace the other one. I don't want to maintain a VM. I have a target price of let's say $5/month for everything (although $3 would be better). Does AWS Lambda give me that? If so, please point me to the guides. Garret
Re: HTTP2: memory filled up fast on increasing the connections to 1000/2000 (Embedded tomcat 9.0.38)
On 05/10/2020 10:56, Arshiya Shariff wrote: > Hi All, > > Thank you so much Mark . > We tested the jars built from latest 9.0.x with 2000 / 5000 users > (connections) from JMeter , We see a very good improvement with the heap > usage Good news. As is the fact that the other errors have been cleared up. > But I see this exception printed multiple times , I am not sure why this > occurs : > Exception in thread "http-nio-x.y.z-1234-exec-213" > java.lang.StackOverflowError > at sun.nio.ch.IOUtil.read(IOUtil.java:240) > at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:440) > at org.apache.tomcat.util.net.NioChannel.read(NioChannel.java:174) > at > org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper$NioOperationState.run(NioEndpoint.java:1468) > at > org.apache.tomcat.util.net.SocketWrapperBase$VectoredIOCompletionHandler.completed(SocketWrapperBase.java:1100) > at > org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper$NioOperationState.run(NioEndpoint.java:1511) > at > org.apache.tomcat.util.net.SocketWrapperBase$VectoredIOCompletionHandler.completed(SocketWrapperBase.java:1100) > at > org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper$NioOperationState.run(NioEndpoint.java:1511) > at > org.apache.tomcat.util.net.SocketWrapperBase$VectoredIOCompletionHandler.completed(SocketWrapperBase.java:1100) > at > org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper$NioOperationState.run(NioEndpoint.java:1511) That looks like an infinite loop reading an incoming frame. New frames are read using a 9 byte buffer for the header and a 16k buffer for the payload (since Tomcat sets this as the max frame size). The loop is occurring because one of those buffers is simultaneously both full and still has more data to read. That should not be possible and I haven't yet been able to figure out how this is happening. How easy is this to reproduce? How often do you see these errors in your test run? Do you have a reliable test case that reproduces this on a clean Tomcat 9.0.x build? If is, can you share the details? Do you have the other end of that stack trace? I'm interested in how the code enters the loop. Thanks, Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: completely automated (for real) Let's Encrypt on embedded Tomcat
Garret, On 10/4/20 14:04, Garret Wilson wrote: > Hi, everyone. I'm back already. (I had intended to leave the list to > focus my efforts elsewhere, but … here I am again.) > > I just realized there is a big SSL problem for small applications, and I > want to fix it. First a little review of where we are. > > Servlet containers are becoming less important and less desirable in > today's world, because we don't want to deploy and maintain some sort of > high-level container infrastructure (in the Java EE container sense, not > the Docker sense) just to deploy an application in it. Modern > distributed micrososervice applications have a bunch of > service/worker/agent application that are identical and redundant. You > spin up as many as you need; if some go down, you (or an orchestrator) > spins up others. Microservices won't work the way you want with Let's Encrypt. You have two options: 1. Hit Let's Encrypt every time you launch a new instance of the microservice to deploy a new certificate 2. Handle the certificate provisioning elsewhere (e.g. ELB) #1 just won't work. LE won't re-issue a certificate more frequently than every 6 weeks or something like that. So that really leaves you with #2. What you really want is for the orchestrator to provide the certificate and key to the nodes as they come on-line. This represents a bit of a security issue, but not any moreso than any node trying to connect to the orchestrator in the first place IMHO. So instead of trying to get LE to work with Tomcat (which does work, but requires some care and feeding), maybe we should try to get Tomcat to load its crypto material from other places. One way to do that would be with e.g. Amazon's key storage service. I'm not familiar with that. I know it can store various types of keys; not sure about certificates and if we can pull a private key out of it. Honestly, your best bet would probably be to use ELB and just pay for it. You only pay for data-transfer, so a dormant ELB costs you virtually nothing. Instead of spinning-up an EC2 instance for your service, maybe you should be looking at Lambda instead. You can probably get your costs down more that way than trying to eliminate the ELB. BTW, if your instance isn't running at all (e.g. the orchestrator has decided that zero nodes are required), what do clients contact to make that initial connection for the orchestrator to spin-up that first instance? > For this reason libraries like Spring Boot allow you to deploy your Java > application as a standalone JAR with embedded Tomcat. The JAR represents > the completely independent application. You just throw it on a node and > it runs and provides a web server or whatever. So we we should be able > to throw a Spring Boot JAR on something like AWS Elastic Beanstalk and > it just runs. I found out it is far from that simple, and SSL is one of > the major problems. > > There seem to be two ways to get SSL support. On something like AWS > Elastic Beanstalk, you deploy a load balancer in front of your EC > instances. Elastic Beanstalk will (using the AWS Route 53 DNS) configure > SSL to the load balancer, spin up EC instances as needed (each running > your standalone JAR), and connect the load balancer to the EC instances, > all in a (sort of) automated fashion. But note that the SSL endpoint is > the load balancer, and the load balancer costs money! Even if you're > just running just a single standalone JAR instance requiring a single EC > instance, that load balancer sits there and drains cash. Significant > cash if you just want to run a little program with SSL support. > > What's the other option to deploy a standalone JAR? Configure an AWS EC > instance (or a VM with another provider), configure certbot, configure > Tomcat, save some files locally on the machine, etc. All this manual > work. I just want to run the standalone JAR! In short, if I have a > standalone program I want to run, I either have to configure and > maintain a VM like I did in the year 2000, or get into the nightmare of > Kubernetes-like orchestration with the endless configurations and/or the > high costs. > > I propose to create a module that integrates with embedded Tomcat that: > > 1. You indicate what domain you're hosting for (as part of the > application configuration or as an environment variable when > deployed, for example). > 2. When your application starts running, it automatically connects to > Let's Encrypt using RFC 8555 (or whatever is needed) and requests a > certificate, based upon the IP address it's running on. > 3. The module exposes the correct HTTP paths and/or connects to a > configured DNS as needed for validation. > 4. The module receives the certificates and caches them in memory or in > a temporary file as needed and provides them to Tomcat; Tomcat now > is serving using SSL/TLS. > 5. If the application dies, who cares? You start up another one. It > automatically does the same thing (on
Re: Unclear sentence in FailedRequestFilter's javadoc
пн, 5 окт. 2020 г. в 14:16, Martin Grigorov : > > Hi, > > What is the meaning of "addresses" at > https://github.com/apache/tomcat/blob/a0fb5f4b42c593fa566878dddfa213e5f3c9c499/java/org/apache/catalina/filters/FailedRequestFilter.java#L40 > ? > > "Parameter parsing does check content type of the request, so there should > not be problems with *addresses* that use > request.getInputStream() and request.getReader()" > > I am not English native speaker, so it might be correct, but it looks like > the wrong word is used here. It looks odd for me as well, although from the commit history it looks that it was my commit that added that sentence [1], 9 years ago. Essentially, the text should be the same as in description of the filter in Configuration Reference [2][3]. It talks about "the servlet protected by this filter". [1] https://svn.apache.org/viewvc?view=revision=1198707 [2] https://svn.apache.org/viewvc?view=revision=1200107 [3] http://tomcat.apache.org/tomcat-9.0-doc/config/filter.html#Failed_Request_Filter It looks as if you are the first person who has read that Javadoc in 9 years. Best regards, Konstantin Kolinko - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Unclear sentence in FailedRequestFilter's javadoc
Hi, What is the meaning of "addresses" at https://github.com/apache/tomcat/blob/a0fb5f4b42c593fa566878dddfa213e5f3c9c499/java/org/apache/catalina/filters/FailedRequestFilter.java#L40 ? "Parameter parsing does check content type of the request, so there should not be problems with *addresses* that use request.getInputStream() and request.getReader()" I am not English native speaker, so it might be correct, but it looks like the wrong word is used here. Martin
RE: HTTP2: memory filled up fast on increasing the connections to 1000/2000 (Embedded tomcat 9.0.38)
Hi All,
Thank you so much Mark .
We tested the jars built from latest 9.0.x with 2000 / 5000 users
(connections) from JMeter , We see a very good improvement with the heap usage .
But I see this exception printed multiple times , I am not sure why this occurs
:
Exception in thread "http-nio-x.y.z-1234-exec-213" java.lang.StackOverflowError
at sun.nio.ch.IOUtil.read(IOUtil.java:240)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:440)
at org.apache.tomcat.util.net.NioChannel.read(NioChannel.java:174)
at
org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper$NioOperationState.run(NioEndpoint.java:1468)
at
org.apache.tomcat.util.net.SocketWrapperBase$VectoredIOCompletionHandler.completed(SocketWrapperBase.java:1100)
at
org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper$NioOperationState.run(NioEndpoint.java:1511)
at
org.apache.tomcat.util.net.SocketWrapperBase$VectoredIOCompletionHandler.completed(SocketWrapperBase.java:1100)
at
org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper$NioOperationState.run(NioEndpoint.java:1511)
at
org.apache.tomcat.util.net.SocketWrapperBase$VectoredIOCompletionHandler.completed(SocketWrapperBase.java:1100)
at
org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper$NioOperationState.run(NioEndpoint.java:1511)
Any help with this please.
Thanks and Regards
Arshiya Shariff
-Original Message-
From: Mark Thomas
Sent: Thursday, October 1, 2020 2:59 PM
To: users@tomcat.apache.org
Subject: Re: HTTP2: memory filled up fast on increasing the connections to
1000/2000 (Embedded tomcat 9.0.38)
On 30/09/2020 18:47, Martin Grigorov wrote:
> On Wed, Sep 30, 2020 at 7:47 PM Mark Thomas wrote:
>> On 30/09/2020 16:17, Mark Thomas wrote:
>>> That is helpful. Looks like you have found a way to reproduce the
>>> buffer issues reported in
>>> https://bz.apache.org/bugzilla/show_bug.cgi?id=64710
>>
>> Can you share the command you used to trigger those errors please.
>>
>
> The Vegeta command I used is:
>
> jq -ncM '{"method": "POST", "url":
> "https://localhost:8080/testbed/plaintext;,
> "body":"payload=Some
> sdgggwwsdgssfshffheeepayload"
> | @base64, header: {"Content-Type":
> ["application/x-www-form-urlencoded"]}}' | vegeta attack -format=json
> -http2 -rate=1000 -max-workers=8 -insecure -duration=2m | vegeta
> encode > /tmp/http2.json; and vegeta report -type=json /tmp/http2.json | jq .
>
> The app is at
> https://protect2.fireeye.com/v1/url?k=c248c6cb-9ce86806-c2488650-866132fe445e-bcef5199a0b9f57e=1=5f78363c-a75a-4bc1-b339-a919d2804f90=https%3A%2F%2Fgithub.com%2Fmartin-g%2Fhttp2-server-perf-tests%2Ftree%2Fmaster%2Fjava%2Ftomcat.
> Just start EmbeddedTomcat#main() with -Dtomcat.http2=true
Definitely timing related as I am unable to reproduce the problem with that
command or some variations.
However, I think I have managed to track down the root cause. The good news is
that the BufferOverflowException is largely harmless. It is a side-effect of
the connection being closed due to an error. My guess is that the error was a
combination of vegeta sending an unexpected reset frame and Tomcat maintaining
state for a very small number of streams in some circumstances.
If you could retest with the latest 9.0.x that would be very helpful.
The memory usage, stream state maintenance and this BufferOverflowException
should all be fixed.
Mark
-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
