RE: Does Tomcat/Java get around the problem of 64K maximum client source ports?
Thanks for all the feedback, André, Christopher, and John. Let me see if I can quickly answer everyone's comments. Since there is a TCB for each connection, and the OS knows which TCBs are associated with which processes, I don't see any problem using the same local port on different sockets. When a packet arrives from a remote server, the stack looks at the full socket details, checks for a matching TCB, and routes the packet to the appropriate process. There's no confusion (except when using tools that don't show process names, like netstat without -p). Using > 64K local source ports seems like a useful capability in high-load environments where tomcat is doing a lot of back-end access (i.e., where JSPs and class files frequently call back-end services). With hashing and indexing, having giant connection tables does not seem like an unrealistic amount of processing load to me. Linux-based stateful firewalls have to keep track of a lot more connections than that, with rule processing and even layer-7 inspection at the same time, on relatively low-powered hardware. FYI, I don't have 1800 tomcat instances on one server. I have about 100 instances on each of 18 servers. That said, I agree that the real focus should probably be on the JDBC driver. I asked the question here because it seemed like a good place to start. Any ideas where I could go to chat with JDBC developers? --Eric > -Original Message- > From: Christopher Schultz > Sent: Friday, March 27, 2020 1:42 PM > To: users@tomcat.apache.org > Subject: Re: Does Tomcat/Java get around the problem of 64K maximum > client source ports? > > -BEGIN PGP SIGNED MESSAGE- > Hash: SHA256 > > André, > > On 3/27/20 11:01, André Warnier (tomcat/perl) wrote: > > On 27.03.2020 14:27, André Warnier (tomcat/perl) wrote: > >> On 26.03.2020 20:42, Eric Robinson wrote: > -Original Message- From: Olaf Kock > Sent: Thursday, March 26, 2020 2:06 PM > To: users@tomcat.apache.org Subject: Re: Does Tomcat/Java get > around the problem of 64K maximum client source ports? > > Hi Eric, > > On 26.03.20 18:58, Eric Robinson wrote: > > Greetings, > > > > Many people say the maximum number of client ports is 64K. > > However, > TCP connections only require unique sockets, which are defined > as... > > > > local_IP:local_port -> remote_ip:remote_port > > > > Theoretically, it is possible for a client process to keep using > > the same local > source port, as long as the connections are to a unique > destinations. For example on a local machine, the following > connections should be possible... > > > > 192.168.5.100:1400 -> 192.168.5.200:3306 192.168.5.100:1400 > > -> 192.168.5.201:3306 192.168.5.100:1400 -> > > 192.168.5.202:3306 192.168.5.100:1400 -> > > 192.168.5.203:3306 > > > > I've seen this demonstrated successfully here: > > > > https://serverfault.com/questions/326819/does-the-tcp-source-port- > have > > > > > - -to-be-unique-per-host > > > > As someone on that page pointed out, while it is possible, it does > > not > commonly occur in practice "because most TCP APIs don't provide a > way to create more than one connection with the same source port, > unless they have different source IP addresses." This leads to the > 64K maximum client port range, but it is really a limitation of the > APIs, not TCP. > > > > So how does tomcat handle things? Is it limited to a maximum 64K > > client > source ports, or is it 64K per destination, as it should be? > > To be honest, I can't remember to have seen a web- or application > server that accepts 64K concurrent connections at all, let alone to > a single client. > > I can't come up with any reason to do so, I'd assume that there's a > DOS attack if I get 100 concurrent incoming connections from a > single IP, and a programming error if the server sets up more than > 1K outgoing connections > > Maybe I'm missing the obvious, or have only administered > meaningless installations, but I fail to see the real world > relevance of this question. > > > >>> > >>> I don't blame you for being puzzled, but this not about tomcat > >>> accepting connections. It's about tomcat acting as the client, where > >>> MySQL is the server. I'm referring to client connections from tomcat > >>> to MySQL. We have about 1800 instances of tomcat running. This > >>> question comes up once in a while when tomcat can't connect to > >>> MySQL. Trust me, it can be an issue. > >>> > >>> --Eric > >>> > >> > >> The question is : is there even any Java API (method) (or even OS > >> API) which allows a client to open a (client, non-LISTEN) socket AND > >> specify the client IP address and/or port ? > >> > >> I mean, if there is none, then the question may be
Re: Practical multipart handling
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256
Mark,
On 3/27/20 14:46, Christopher Schultz wrote:
> Mark,
>
> On 3/27/20 13:35, Mark Thomas wrote:
>> On 27/03/2020 15:52, Christopher Schultz wrote:
>>> On 3/26/20 18:44, Konstantin Kolinko wrote:
>
>>
>
I think that those are available via the standard
request.getParameter(name) API.
>>>
>>> That doesn't work. Mixing request.getParameter() and
>>> request.getParts results in request.getParameter* returning
>>> nulls.
>
>> Then something isn't quite right as that should work. And there
>> are unit tests that cover at least some combinations:
>
>> TestRequest.testBug54984
>
> Now that everything is working, I'll see what I can do to
> reproduce what I was seeing earlier.
>
> Eyeballing that test makes me think that maybe the problem is that
> if you call request.getParts() first, you can't then go back and
> call getParameter("foo").
Nope. I seem to be able to use request.getParameter at any time.
I'll bet I know what it was: I wrote 90% of this code *before*
enabling multipart-processing on this servlet and so nothing was
working at all at first, even when not sending the file.
So it's nice that I don't have to write the code to fetch small String
values from "small" multipart parts. But I wrote it anyway :) so I get
to rip it all out, now, which is fine.
- -chris
-BEGIN PGP SIGNATURE-
Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/
iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAl5+T/MACgkQHPApP6U8
pFi7BA/8DMmtvVnVlBeI9F9er9DEHDgEp7F6jb8eT/lQ1AKG0bZ0dj+BLpY3YLW9
9JExMlt86khWMwcfKgT+VPhlpDRRUvIBBXpcp9AZaB1KvDC97DOUpmxnslUdAo1O
AgHGbkkyiqoeuTKBf66JCGx4P4sdsyNMnXijlAgJDJ99NvcEwq4Ol1RF0UzXQso6
0oE2NY5ul4z+Xtrp8iP2eUPuJgZmzrF78I6/KJ+EOnQ/pYCtoAK+0E/EVmXAsNGw
4ZkUg3RynnJus3RA8GBFl53ol0U5WQQnRfD2kTXhXEqf7Q35eD5b94QeRFBOGH+O
6qlNnr+xzleyEg5eGLX26X5gDBf84FvROjKf602YT5TLicDHR7ZNIlr90zPeXIWC
5jkrgAybVDcil0n8QgD3LojMuh1HueaqHqVE43aXhjTOi/7YoEIzqMr66e/8f1id
gOYXsSTnVl+xEUi/C3tDWPboFNJIRpuOAv61PkKQlw1sgC7S6VCGKt/tfhLr24dh
fpygHMiMe5IOIIY2epvh0SU8wWZhwT2Ne+bRmCw9OGMjA+5wiJDGhFWyeKmawtWI
ZCmweTVdkH4ko0oMRb8UyNDCClPvPRSLlkVtN8NcA4z8U0cfnrRopVvXdnEbZLJW
tBhizazhrbU6sDirCDLe+1FHnEf5XOWOTK02cUqLiUL1D+Lr0SI=
=DPX7
-END PGP SIGNATURE-
-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Practical multipart handling
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256
Mark,
On 3/27/20 13:35, Mark Thomas wrote:
> On 27/03/2020 15:52, Christopher Schultz wrote:
>> On 3/26/20 18:44, Konstantin Kolinko wrote:
>
>
>
>>> I think that those are available via the standard
>>> request.getParameter(name) API.
>>
>> That doesn't work. Mixing request.getParameter() and
>> request.getParts results in request.getParameter* returning
>> nulls.
>
> Then something isn't quite right as that should work. And there are
> unit tests that cover at least some combinations:
>
> TestRequest.testBug54984
Now that everything is working, I'll see what I can do to reproduce
what I was seeing earlier.
Eyeballing that test makes me think that maybe the problem is that if
you call request.getParts() first, you can't then go back and call
getParameter("foo").
- -chris
-BEGIN PGP SIGNATURE-
Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/
iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAl5+Sg8ACgkQHPApP6U8
pFgphxAAgzr5Bsf/ro4ChAh9Ln+RSAr9Mjf1U0Rjt1utoQGBHWfscnZiD9Qhim4D
yTY0lZ8aMxLhDC04F/3Ur5ZgJShutr2OgCHkf/4izXJbsJU+KDzJBh8AySNqZqmC
/UDGL+CNm4ReBcwy4SRqsxJxJdLzX38AqDCGotoDAgl8yifHzfMXP5gS9ARjwMIi
yhArCxAROBAT2ckS5MzOV5cBYru2MT3F/wq5WuvEqPjjC2j4FJO05dDvXZnwdWOm
Pd2qqAx5CmwDIGm6S/c1MnzNi0fb06asgg8DMdIMPblS0OqnyG+aVOJUx3LhEq7d
vvIUczSEPfFM0U8zxJ6WlW0Sx69139D02nWXmyZp+Y2do+fqfeHy2aUdVqekUDXd
8b4gGJQYF2fSozpevtH8kKBDXEAHzIWGu2FTuaRffVY7xIJsAqtAt1tLPPnnj0R6
AzCp4JncFXQ7xeyUh3IkEvC8sx7zC8iQGJX6KRPuwAaghyfHgFUt5pL7mGtS92pL
LfBfISFKeDJCfKhNyXYfFZyPhNjE7zANOSiRBNky4MpKMNVp0kBxu/wRTFbxBt5Q
cxO/RwSrpCk9y+49c6D0L80yw1/ABFIIOLY4YwLynRAfPnCFjHpDhqkgA7s1XuqZ
H3cXpLChsapUEsbF2LBcjUq7fWDP/VXCHBclGLPehp6qfZxaVXA=
=ShRj
-END PGP SIGNATURE-
-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Does Tomcat/Java get around the problem of 64K maximum client source ports?
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 André, On 3/27/20 11:01, André Warnier (tomcat/perl) wrote: > On 27.03.2020 14:27, André Warnier (tomcat/perl) wrote: >> On 26.03.2020 20:42, Eric Robinson wrote: -Original Message- From: Olaf Kock Sent: Thursday, March 26, 2020 2:06 PM To: users@tomcat.apache.org Subject: Re: Does Tomcat/Java get around the problem of 64K maximum client source ports? Hi Eric, On 26.03.20 18:58, Eric Robinson wrote: > Greetings, > > Many people say the maximum number of client ports is 64K. > However, TCP connections only require unique sockets, which are defined as... > > local_IP:local_port -> remote_ip:remote_port > > Theoretically, it is possible for a client process to keep > using the same local source port, as long as the connections are to a unique destinations. For example on a local machine, the following connections should be possible... > > 192.168.5.100:1400 -> 192.168.5.200:3306 192.168.5.100:1400 > -> 192.168.5.201:3306 192.168.5.100:1400 -> > 192.168.5.202:3306 192.168.5.100:1400 -> > 192.168.5.203:3306 > > I've seen this demonstrated successfully here: > > https://serverfault.com/questions/326819/does-the-tcp-source-port- have > > - -to-be-unique-per-host > > As someone on that page pointed out, while it is possible, > it does not commonly occur in practice "because most TCP APIs don't provide a way to create more than one connection with the same source port, unless they have different source IP addresses." This leads to the 64K maximum client port range, but it is really a limitation of the APIs, not TCP. > > So how does tomcat handle things? Is it limited to a > maximum 64K client source ports, or is it 64K per destination, as it should be? To be honest, I can't remember to have seen a web- or application server that accepts 64K concurrent connections at all, let alone to a single client. I can't come up with any reason to do so, I'd assume that there's a DOS attack if I get 100 concurrent incoming connections from a single IP, and a programming error if the server sets up more than 1K outgoing connections Maybe I'm missing the obvious, or have only administered meaningless installations, but I fail to see the real world relevance of this question. >>> >>> I don't blame you for being puzzled, but this not about tomcat >>> accepting connections. It's about tomcat acting as the client, >>> where MySQL is the server. I'm referring to client connections >>> from tomcat to MySQL. We have about 1800 instances of tomcat >>> running. This question comes up once in a while when tomcat >>> can't connect to MySQL. Trust me, it can be an issue. >>> >>> --Eric >>> >> >> The question is : is there even any Java API (method) (or even OS >> API) which allows a client to open a (client, non-LISTEN) socket >> AND specify the client IP address and/or port ? >> >> I mean, if there is none, then the question may be interesting in >> the absolute, but ultimately pointless. >> >> I believe that the IP address of client packets, will be >> determined by which /route/ the OS determines that the target >> server address can be reached (which will determine through which >> network interface the packets "go out", which will determine the >> sender IP address inserted in the packets). I don't think that >> the application-level software (here a java webapp) can determine >> this in advance. And the client port will be decided by the >> OS-level TCP stack, in function of which ones are not yet in use >> (which a java webapp can also not determine in advance). >> >> Example of creating a client socket : Socket echoSocket = new >> Socket(hostName, portNumber); The hostname (or IP address of >> ditto) and port numbers passed as arguments, are the IP:port of >> the server you are connecting /to/, not the ones of the local >> socket. >> > > Addendum : > > https://stackoverflow.com/questions/11129212/tcp-can-two-different-soc kets-share-a-port/11129641 > > > > From which I gather that, although it may be possible (in some > languages or using some API) to open several client connections (to > different target IP/port) using the same local port number, you may > still have other issues when doing this : - the host's TCP/IP stack > has to keep track of all the simultaneous connections, and if the > list gets very large, this may become very resource-intensive Those resources must be available, used or not. You can't have a TCP/IP stack which is documented to support 64k connections but only actually supports 32k or 16k. Also, 64k entries in a connection table isn't large. It can even be indexed in a way that makes it *very* fast to look them up, either by hash or even just port-number(s). Your are more likely t
Re: Does Tomcat/Java get around the problem of 64K maximum client source ports?
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 John, On 3/27/20 13:37, john.e.gr...@wellsfargo.com.INVALID wrote: > A few random thoughts. > > First, this is really a MySQL driver thing, not a Tomcat thing. > If you want to know why the driver does what it does, look at the > driver source. There is actually a Socket constructor that allows > you to specify the local port. I don't know why you would do that, > but it's there. > > Second, it does appear to be possible for there to be multiple > connections using the same source port as long as they go to > different destinations. Just read through the various links that > people have provided. Honestly it blows my mind that that's even > possible, but that just shows how little I know about networking. > How does the client OS route return packets correctly if they have > the same source port? Because the connection is defined by *both* endpoints in a TCP connection. The client knows its own source port and the destination port as well. Imagine two clients: saddr sport daddr sport ::1 1400 x 3306 ::1 1400 y 3306 All is well, there: the OS knows where to route data returning to ::1/1400/y/3306 distinctly from ::1/1400/x/3306. But if you have two clients wanting to connect to the same destination (e.g. x:3306) then they must have distinct source ports. This is why most clients just use a random OS-assigned port. - -chris -BEGIN PGP SIGNATURE- Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/ iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAl5+R5QACgkQHPApP6U8 pFjLsQ/+IdSoJIPFhNfLmQnxEKEQA13Arr9vUE4A+83uh2+tHmo0nwW9VBzWYRLT hEciA++PSBKBDimaeNrjQBmNEmMfLnfWBZfc0TjDCAUEYvvBXslYyw9CySya3ISC d79hCJj55oAiua+eJ7ZN/MyRgeldwF/3CsP6lwpoH/rR9KI8/oKqt4vV64a7BuTW gSIlzKJ5BQCtGZIDZmf7PeL5d48GJnw0jdREFC8Bf8NYjS/uj8Cccy8XH7zPcraP S8krhrqemH8e6UQE3XID+qyatkHXBo5AVC16soQ0+jwDp0oxCsV0Faxk+8KollKL aWti+di9DOQkKSYHq0IgyE82sxxm8FL5vAKABpE/i9864PqwBBv9hIfk4Lyiqirx zuI3kVEEixcgVnv6Wdrmj1oGdhIcV6kHD/cXlTgMRTcGLBqrMFFfj4YdgUUHwOId ZaL1UMlH0sB9NiZtSwxnEFslHbqV6dYZjpWdBnTRjiYWbTsz5/ZLLMLxvg9V+cr2 fCc+/B+Co6q9DnUyipfDfcHfvXDoPFseMmCfXH8I/4DPD8dmAxzM9BoviL9GSnAG T0AVfKtB/HeGeZ67jZ9iIqpNVhCjxRrb1XZmoa4RUTU5J8i1xxBqpnpcYtCQXoJm q7UJpJAVOHg68NQgxh3YK7wemb9nSacWO587rhCNjkIoVKkMDCQ= =K9in -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: Does Tomcat/Java get around the problem of 64K maximum client source ports?
Chris, André, Eric, etc, A few random thoughts. First, this is really a MySQL driver thing, not a Tomcat thing. If you want to know why the driver does what it does, look at the driver source. There is actually a Socket constructor that allows you to specify the local port. I don't know why you would do that, but it's there. Second, it does appear to be possible for there to be multiple connections using the same source port as long as they go to different destinations. Just read through the various links that people have provided. Honestly it blows my mind that that's even possible, but that just shows how little I know about networking. How does the client OS route return packets correctly if they have the same source port? John - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Practical multipart handling
On 27/03/2020 15:52, Christopher Schultz wrote: > On 3/26/20 18:44, Konstantin Kolinko wrote: >> I think that those are available via the standard >> request.getParameter(name) API. > > That doesn't work. Mixing request.getParameter() and request.getParts > results in request.getParameter* returning nulls. Then something isn't quite right as that should work. And there are unit tests that cover at least some combinations: TestRequest.testBug54984 Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: valve execution order
On 23/03/2020 14:23, Jonathan Yom-Tov wrote: > What about valves which are added programmatically within the same engine? > Is it possible to control the order? Not directly. Valves added programmatically are added to the end of the Pipeline. There isn't an insertValve() method so you need to add them in the order you want them to be used. > On Mon, Mar 23, 2020 at 4:14 PM Christopher Schultz < > ch...@christopherschultz.net> wrote: > > Mark, > > On 3/23/20 09:59, Mark Thomas wrote: On 23/03/2020 13:13, Jonathan Yom-Tov wrote: > I'm trying to create a valve which will prevent another valve > from working in case a particular request is made. Is there a way > to control the valve execution order so that I can inject my > valve right before the valve I want to stop? Valves should be added (and hence executed) in the order they appear in the XML. Also, Valves on an Engine always execute before those on a Host which always execute before those on a Context. > > Correct me if I'm wrong, but there is one more caveat: if the > defines its own Authenticator, then whatever Authenticator > Tomcat would have also automatically added is in fact NOT added. > > Where does the context's authentication valve go in that situation? In > the position where the default one would have gone, or in the > 's position according to the XML? As per the order of Valves nested in the Context element. Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Does Tomcat/Java get around the problem of 64K maximum client source ports?
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256
Eric,
On 3/26/20 13:58, Eric Robinson wrote:
> Many people say the maximum number of client ports is 64K.
> However, TCP connections only require unique sockets, which are
> defined as...
>
> local_IP:local_port -> remote_ip:remote_port
>
> Theoretically, it is possible for a client process to keep using
> the same local source port, as long as the connections are to a
> unique destinations. For example on a local machine, the following
> connections should be possible...
>
> 192.168.5.100:1400 -> 192.168.5.200:3306 192.168.5.100:1400 ->
> 192.168.5.201:3306 192.168.5.100:1400 -> 192.168.5.202:3306
> 192.168.5.100:1400 -> 192.168.5.203:3306
I think you must be confusing something, here. The left-hand side of
that list looks like it cannot happen. Each client connection requires
its own source-port, so port 1400 cannot be used by many simultaneous
connections.
It's also strange for me to see client port numbers so low. Typically,
client ports are much higher. It's also very uncommon for the client
to specify a source-port, so the OS's TCP/IP stack usually allocates
an available port, which is why they end up being higher-numbered
ports. Have a look here for a very informative answer:
https://superuser.com/a/1118742/408230
Note that the maximum number of client port is DEFINITELY fixed at
64k, because of the data types used by the C library. This happens
when you bind() a socket to a port:
int bind(int sockfd, const struct sockaddr *addr,
socklen_t addrlen);
struct sockaddr is:
struct sockaddr {
sa_family_t sa_family;
charsa_data[14];
}
and, for the AF_INET socket address family (from ):
struct sockaddr_in {
sa_family_tsin_family; /* address family: AF_INET */
in_port_t sin_port; /* port in network byte order
*/
struct in_addr sin_addr; /* internet address */
};
Continuing down the road, in_port_t (from
/* Type to represent a port. */
typedef uint16_t in_port_t;
So the port number is unsigned 16-bit. you can't squeeze more than 64
ports out of that.
Note that you aren't limited to 64k sockets, as you have intimated:
ports can be re-used.
> I've seen this demonstrated successfully here:
>
> https://serverfault.com/questions/326819/does-the-tcp-source-port-have
- -to-be-unique-per-host
>
> As someone on that page pointed out, while it is possible, it does
> not commonly occur in practice "because most TCP APIs don't
> provide a way to create more than one connection with the same
> source port, unless they have different source IP addresses." This
> leads to the 64K maximum client port range, but it is really a
> limitation of the APIs, not TCP.
>
> So how does tomcat handle things? Is it limited to a maximum 64K
> client source ports, or is it 64K per destination, as it should
> be?
The short answer is that Tomcat doesn't handle this. There are very
few outgoing connections that Tomcat makes, and database connections
aren't on that list. What you are seeing is the JDBC driver's
connection being handled by Java, which ultimately is being handled by
the JRE and the OS working together. I see you are using port 3306
which is MySQL and/or MariaDB. I don't know of any driver which
bothers to set the local port, so it's again strange to me to see port
1400 getting ... any use at all.
- -chris
-BEGIN PGP SIGNATURE-
Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/
iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAl5+KvEACgkQHPApP6U8
pFi5rxAAp8HoqhQFOX6FtR3r+f1wkcdx7eLVc68e1NngB05hXdLTiXtmRy+XRXYy
8XahJkFvqWkKpKKcRopn714g+P44qMcXCtLtgQ3iTRB9dwp5uuYKN5z6QbaAzWfq
vPb8UXESFoxCiXVBuV5Q8H+9utvS1OaST8exkwuAi33dkuh/Es4gFWpmbrKVhExl
s4TNwq2UCTB5d6ybNM8AZzeAiNxkrop9+zi3AcgcVN6KMUS/dPWen3YTchUyTpdc
821gOKVrpy3nvPdCJv2dtoZ7G343D/Ice+ZYaDEe+oooOcPu88siOzna5vVDyUSM
z7o3XsQ/zHIihfDwG5F1z0upWEGPPy5RD9VwOGvlCCDrG3dCHpYrgYfcOm3jA1kh
6Bu1y8vIy2qX7ZWEaHPOyAMpZKsodugpGR6Piov78OsEQWBLW5pD5E1sOBkXa2TO
QSoadpWSjTVlTfDvZvTM4l5KDA+Q8NFmBPvoViejXIQCsshT4Hz4fEnBZTSPzQLf
6bxJrp9+5Y9f2ZFM3Iss4NR6PfCTE5LbWGJZOY+IWzN1p2UUJtEw1nazbWc57kRx
ZLZgwz8D0IiNnooU3NA6rr+hqC+FPYgszGOS3BJUyfUn7PetZbtoOA5ouZn7S5jQ
kq18P3k9z9rc/BKSaSnYdX7NmUavkqfM0Nw+Dwrmds3fdchUVoY=
=DXcE
-END PGP SIGNATURE-
-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Practical multipart handling
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Konstantin, On 3/26/20 18:44, Konstantin Kolinko wrote: > чт, 26 мар. 2020 г. в 18:03, Christopher Schultz > : >> >> All, >> >> I'm developing my first multipart handler since .. I dunno, >> maybe 2005? This is the first time I'll be using the Servlet 3.0 >> multipart handling, of course through Tomcat. Some of these >> questions may have answers which are "implementation-specific", >> so in this case, I would like to know how things will behave in >> Tomcat specifically. Notes of where the spec leaves things up to >> the implementation will be appreciate d. >> >> I'd like to submit a form which has not only a large-ish file >> part, but also some regular fields like . My >> understanding is that I'll have to read those data by calling >> Part.getInputStream(), wrapping the InputStream in an >> InputStreamReader using the right charset, etc. > > I think that those are available via the standard > request.getParameter(name) API. That doesn't work. Mixing request.getParameter() and request.getParts results in request.getParameter* returning nulls. >> Can I rely on the client to send the fields in any particular >> order? I'm not expecting to store the file on the server myself; >> I'd like to process it in a "streaming" fashion and not touch the >> disk if possible. I know that the server may store the file on >> the disk if it decides to. I'm not terribly worried about that. I >> just don't want to have to write the file to the disk TWICE, and >> I need information from those other parameters in order to >> configure the stream-processing. > > Michael already answered this. There is a configurable threshold. > Anything over it will be written to disk as a temporary file. > > The JavaDoc for Part.write() says that it can be implemented as > moving the file. "This method is not guaranteed to succeed if > called more than once" > >> When iterating over the Collection returned from >> HttpServletRequest.getParts(), am I required to process each part >> in order immediately? Or can I store a reference to a Part for >> later? This kind of goes along with the previous question. > > You can store the reference, but your "for later" should be no > longer than until the request processing ends. Oh, of course. I shouldn't have left that detail unaddressed. I just meant that I wanted to process all the parts and then handle the large file, to make sure I had all the "small" fields read. >> When I'm done with a part, must I explicitly call Part.delete()? > > Tomcat deletes the files automatically (I implemented this feature > in Tomcat 7.0.30 - see changelog). In my own web applications I > delete the files explicitly (calling part.delete() in a cycle). I may go ahead and store a reference to the Part which represents the large file and explicitly call Part.delete() on it, just to be sure. Thanks, - -chris -BEGIN PGP SIGNATURE- Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/ iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAl5+IScACgkQHPApP6U8 pFinYxAApRdIDNNNxlkb5HIrPgKlCpQ9B5I5r+ceFD4GeylFZ1NodG3pxCv/E/C1 xxpjHiYv5dS30LIsZzLUe3PEnegWefokDKWLviKak54MFyQqkmmD3iEKypCp4F2y HwXFDKhBRea/D1j/kwXLmfW8FSKM9lZCN23rLmYl2O0YrKqCnaFf9PJqcSBAqBnx 9vfW6RQEdRsALuHMGurdSV9hJDqCZ+l0298bweR3C4DuIpiM5EorNkf4vOrKvOpn 078sxpFtVDrxfwQ7gltagCPmFoNxcTX83h8oYAmb+aso7xeXlCQTIbkHbQV3qR4t fOj1pFc2ZhP9U0wR4obD8Uwhpo8bVDcOsSVXBHGJSGMeCjdiaPmXL80C+fBtAT7i 0xtb92fjZRvyEwN/yWHjmE05dJiMh8sI0rs6cfXi34QtP8AW14VIKNed/BCmYD/F LFw5OtBP0GRC1nENwRHmFIWeqFw48xB+rqSBPMoquA4YRBVekEWlvIeAzCBFmYMb XkUv38Vj73Z9mHD/9VQnXTa0/KJjZhVeLrl77reB6fLdEyBnACGVgJR/HpBwtdV6 QhuUFHkPhl3mLP6weoKLyj8RbiJt11eKQ6XAt9iaaSiehpwzh2Q2BWJ0ckuRqbH1 KoXbFdYBur8Cg2OQVfjArnuX3yFSos1Ew7+MjzEQ3bnCZrgb2o8= =Lekc -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Practical multipart handling
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Michael, On 3/26/20 12:36, Michael Osipov wrote: > Am 2020-03-26 um 16:03 schrieb Christopher Schultz: >> -BEGIN PGP SIGNED MESSAGE- Hash: SHA256 >> >> All, >> >> I'm developing my first multipart handler since .. I dunno, >> maybe 2005? This is the first time I'll be using the Servlet 3.0 >> multipart handling, of course through Tomcat. Some of these >> questions may have answers which are "implementation-specific", >> so in this case, I would like to know how things will behave in >> Tomcat specifically. Notes of where the spec leaves things up to >> the implementation will be appreciate d. >> >> I'd like to submit a form which has not only a large-ish file >> part, but also some regular fields like . My >> understanding is that I'll have to read those data by calling >> Part.getInputStream(), wrapping the InputStream in an >> InputStreamReader using the right charset, etc. >> >> Do I get any help from any existing APIs for doing that, or do I >> have to pull the Content-Type out of the Part-headers, look for a >> "charset" parameter, etc., etc., in order to get the encoding and >> convert to a String value? It seems like a pretty big hole in the >> multipart API not to have something like that already available. >> >> Can I rely on the client to send the fields in any particular >> order? I'm not expecting to store the file on the server myself; >> I'd like to process it in a "streaming" fashion and not touch the >> disk if possible. I know that the server may store the file on >> the disk if it decides to. I'm not terribly worried about that. I >> just don't want to have to write the file to the disk TWICE, and >> I need information from those other parameters in order to >> configure the stream-processing. >> >> When iterating over the Collection returned from >> HttpServletRequest.getParts(), am I required to process each part >> in order immediately? Or can I store a reference to a Part for >> later? This kind of goes along with the previous question. >> >> When I'm done with a part, must I explicitly call Part.delete()? > > A mere coincidence, we are currently designing an API around > multipart/related and I have investigated multipart requests > deeply after 10 years of abstinence. > > W/o going to much in detail: * Tomcat uses Commons File Upload w/ > all its flaws and benefits * Commons FU starts writing to disk as > soon as it hits a threshold * Don't expect an order unless > specified by RFC 7578 RFC 7578 mentions item ordering but only stating that intermediaries may not reorder items. > * Browsers may order, read > https://html.spec.whatwg.org/multipage/forms.html#forms I wasn't able to find anything in there which requires (well...) item-ordering, but my testing with Firefox suggests that the order in the multipart request payload matches the document-order of the form. > * Streaming will be hard/impossible with Commons FU when not > ordered. It reads off the non-repeatable input stream which has to > be segmented on the fly into parts with headers and embedded > payload. Consider that you have payload beyond your huge payload. > You simply cannot access it unless you have consumed the huge one. > In your case, if there are form fields after the huge payload you > need to decide how to process the huge payload you *must* cache > locally. My current code is looping through the parts and obtaining an InputStream from the large-parameter. After the loop, I read the InputStream fully. The previous "parts" are all being read during the loop (as simple String values). > * You must call delete, otherwise the temporary file will left on > disk Hmm. I haven't been calling delete() on any parts and I'm not seeing any files piling up anywhere, neither in Tomcat's "work" directory nor in the "tmpdir". I even tried a multi-GiB file. > I think your best bet is either: > http://commons.apache.org/proper/commons-fileupload/streaming.html > or https://james.apache.org/mime4j/ I've limited my uploads to 1MiB so I think I'd be okay if I had to buffer on-disk. - -chris -BEGIN PGP SIGNATURE- Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/ iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAl5+II4ACgkQHPApP6U8 pFgt5w//edoL5yrkt7kb4mTsZNviKwvXG7Pecvbd+sQX/cUEogXiW/siu6XpZ6Z+ MfjPrXU606mDHHwz95H9CxrNHz5VLOxquaCg5xZinIvfXGpqwlqtbMj/B/vY0344 un0z2sj5wVJEqF9MSXpQgYjdlFPZ29eHuOsffval34PyqVdCQf5Nxu3XwjggRW6K VTWzODxoccHcpHU7uDzL3vTiVfFMeRQQnqybgLl8jupTqwmpL29Aw4QvIvpX0HDg SN46Ovr9a3u8+oVzkSwaKJhyw1CzeT1KaXTdqg5qAEuugsTmnPA/VckReQcXEcvS GSioBNWZFg5p0Yv/PA9EGQxER28+N4rii2O/3H/CyEy48vxUuiizP/5fzufTGD+s kzewtYT7SRtACdVGIYZW9jcoYaQsYlYPTOQfYFH688MSAou2AaLSdiII/ZuLvSwz NipQ2kufsixVbHjtusyC6F3+XJWMEG3DwwaKaM5fo2AMucFd4u3gzA1pl16e/onF LbyXwUNJ7I3tlobQoMQYAw0BmHsQuIQxwLiL/B2L3QZSm9+yZ6qB3crkWJqjBVmH f0HpAr/pClBq7PFYwN2dvvOXr6yzoYkLAt8/Sq/TDR9SXA6vTZWaVXGyFBEWB5sa nQyXGDUq+GmfJgBF2k1PKNZx/PANwY7EajpyUj+8FeIVdfwc75E= =D89D -END PGP SIGNAT
Re: Strange side effect of "antiClickJackingOption" clause in "httpHeaderSecurity"
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 James, On 3/26/20 12:54, James H. H. Lampert wrote: > On 3/24/20 2:25 PM, Christopher Schultz wrote: >> I don't understand exactly how X-Frame-Options (which is what >> the HttpHeaderSecurityFilter is configuring) is being used by >> your application, but I believe X-Frame-Options is essentially >> being replaced by various features of Content-Security-Policy. >> You may want to talk to your engineers about using one of those >> versus the other; you may want to discontinue using the "anti >> click-jacking" features of this filter altogether in favor of >> something else. > > Dear Mr. Schultz (et al): > > Thanks. Our webapp team combined your answer with what we could > glean from the customer's security audit, and has come up with a > solution involving a Content-Security-Policy, using a class he > added to the webapp. I'm not sure, but I think it can be built into > the WAR file. Glad to hear they have a fix. > On our own Tomcat server, we have another webapp that cannot have > clickjack protection via a HTTPHeaderSecurity filter with >> >> antiClickJackingOption >> >> >> SAMEORIGIN >> > > because it's specifically designed to go into a frame, in a page > served from an entirely different server. Is that what the > "ALLOW-FROM" option and the associated "antiClickJackingUri" > parameter are for? I would encourage you to read the Tomcat documentation and then go down the rabbit hole a little bit by reading about the various headers Tomcat is setting. The Tomcat documentation doesn't cover what all those thing do; just what effect the config has on the headers. You will learn better that they do by reading the header documentation yourself than if I try to summarize here for you. Sorry, I know that RTFM isn't the most convenient response, but I think it's the best one for you. - -chris -BEGIN PGP SIGNATURE- Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/ iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAl5+HKoACgkQHPApP6U8 pFjNcxAAgnvS0ki7uf1aFtESZaikCd3LAU4Hl4hRvUFhcSk0vvuSoG+Hqu4snvTM TPUblS2o95wbmlJlBqwPstYEL7mRVHCNUqXoe0dVTHc9vcz9Cgacs7xd3PR7B1nT FEOZnuZSIxTQCOgwlZhx2Lvv/HHp3ja8NlEKs2mxVtb1N1PoxDMhIL2tSFHzTiJf avHgS+wlc2872bum0IFrAIX619toKiVBARRVadXLKB7nXW9ToSkyV0tE78+n1BBI /dPy3HcheoRluFkLTfZEwv6Mg1fPP2t//FW09h4aLNGlc6/C3jXi1b0P8sAyj8xz A18psq0Y3qXw32ymSmEtOJSdEgU9ogGn52nR7aUDtAA6ccUTRJdXT7iMON4w+njb FwBEkWtImbZGAcVuWhxLt1Ji0xJ7/XLRGokiIcc7iBji1VEo9TLk3eSmPaT/pZYz aRg/vGw6PJJAUMoCHG2zDNXK01GMujUi0JCv2Awav4b6fs0YsbkfuF7d28eesH76 227E8AthiVE7YfIZPY++UtVtuvJNls7ii6QpRf5Tdu+AGDO5JDWh+FOgvDnUFzUY J/hHYtFrPD/hxwyaaU/2t8ZAjJkSGzdu7A/eDidTXKdKAsvUBsoOO1w1o0RQTpE8 QU+fLT6tfO4vA6VvOM8OtvOvp+h91WUvoNk+bCj5IxIqib3F+T8= =jgb7 -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Does Tomcat/Java get around the problem of 64K maximum client source ports?
On 27.03.2020 14:27, André Warnier (tomcat/perl) wrote: On 26.03.2020 20:42, Eric Robinson wrote: -Original Message- From: Olaf Kock Sent: Thursday, March 26, 2020 2:06 PM To: users@tomcat.apache.org Subject: Re: Does Tomcat/Java get around the problem of 64K maximum client source ports? Hi Eric, On 26.03.20 18:58, Eric Robinson wrote: Greetings, Many people say the maximum number of client ports is 64K. However, TCP connections only require unique sockets, which are defined as... local_IP:local_port -> remote_ip:remote_port Theoretically, it is possible for a client process to keep using the same local source port, as long as the connections are to a unique destinations. For example on a local machine, the following connections should be possible... 192.168.5.100:1400 -> 192.168.5.200:3306 192.168.5.100:1400 -> 192.168.5.201:3306 192.168.5.100:1400 -> 192.168.5.202:3306 192.168.5.100:1400 -> 192.168.5.203:3306 I've seen this demonstrated successfully here: https://serverfault.com/questions/326819/does-the-tcp-source-port-have -to-be-unique-per-host As someone on that page pointed out, while it is possible, it does not commonly occur in practice "because most TCP APIs don't provide a way to create more than one connection with the same source port, unless they have different source IP addresses." This leads to the 64K maximum client port range, but it is really a limitation of the APIs, not TCP. So how does tomcat handle things? Is it limited to a maximum 64K client source ports, or is it 64K per destination, as it should be? To be honest, I can't remember to have seen a web- or application server that accepts 64K concurrent connections at all, let alone to a single client. I can't come up with any reason to do so, I'd assume that there's a DOS attack if I get 100 concurrent incoming connections from a single IP, and a programming error if the server sets up more than 1K outgoing connections Maybe I'm missing the obvious, or have only administered meaningless installations, but I fail to see the real world relevance of this question. I don't blame you for being puzzled, but this not about tomcat accepting connections. It's about tomcat acting as the client, where MySQL is the server. I'm referring to client connections from tomcat to MySQL. We have about 1800 instances of tomcat running. This question comes up once in a while when tomcat can't connect to MySQL. Trust me, it can be an issue. --Eric The question is : is there even any Java API (method) (or even OS API) which allows a client to open a (client, non-LISTEN) socket AND specify the client IP address and/or port ? I mean, if there is none, then the question may be interesting in the absolute, but ultimately pointless. I believe that the IP address of client packets, will be determined by which /route/ the OS determines that the target server address can be reached (which will determine through which network interface the packets "go out", which will determine the sender IP address inserted in the packets). I don't think that the application-level software (here a java webapp) can determine this in advance. And the client port will be decided by the OS-level TCP stack, in function of which ones are not yet in use (which a java webapp can also not determine in advance). Example of creating a client socket : Socket echoSocket = new Socket(hostName, portNumber); The hostname (or IP address of ditto) and port numbers passed as arguments, are the IP:port of the server you are connecting /to/, not the ones of the local socket. Addendum : https://stackoverflow.com/questions/11129212/tcp-can-two-different-sockets-share-a-port/11129641 From which I gather that, although it may be possible (in some languages or using some API) to open several client connections (to different target IP/port) using the same local port number, you may still have other issues when doing this : - the host's TCP/IP stack has to keep track of all the simultaneous connections, and if the list gets very large, this may become very resource-intensive - there may be a limit at the OS level, to how many sockets are allowed at the same time - to each connection, corresponds some kind of a "file descriptor". There may be limits, at the process level (in this case the JVM running tomcat), or even at the host level, to how many of those may be in use at any one time (So if you were running on a host one JVM for a tomcat instance, it would compete with other processes running on the same host and also using file descriptors) It seems also thus possible that the reason why you are running against a problem not being able to make additional connections to a MySql server, is not necessarily the usage of all 64K port numbers, but one of the other limitations above. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands,
Re: Does Tomcat/Java get around the problem of 64K maximum client source ports?
On 26.03.2020 20:42, Eric Robinson wrote: -Original Message- From: Olaf Kock Sent: Thursday, March 26, 2020 2:06 PM To: users@tomcat.apache.org Subject: Re: Does Tomcat/Java get around the problem of 64K maximum client source ports? Hi Eric, On 26.03.20 18:58, Eric Robinson wrote: Greetings, Many people say the maximum number of client ports is 64K. However, TCP connections only require unique sockets, which are defined as... local_IP:local_port -> remote_ip:remote_port Theoretically, it is possible for a client process to keep using the same local source port, as long as the connections are to a unique destinations. For example on a local machine, the following connections should be possible... 192.168.5.100:1400 -> 192.168.5.200:3306 192.168.5.100:1400 -> 192.168.5.201:3306 192.168.5.100:1400 -> 192.168.5.202:3306 192.168.5.100:1400 -> 192.168.5.203:3306 I've seen this demonstrated successfully here: https://serverfault.com/questions/326819/does-the-tcp-source-port-have -to-be-unique-per-host As someone on that page pointed out, while it is possible, it does not commonly occur in practice "because most TCP APIs don't provide a way to create more than one connection with the same source port, unless they have different source IP addresses." This leads to the 64K maximum client port range, but it is really a limitation of the APIs, not TCP. So how does tomcat handle things? Is it limited to a maximum 64K client source ports, or is it 64K per destination, as it should be? To be honest, I can't remember to have seen a web- or application server that accepts 64K concurrent connections at all, let alone to a single client. I can't come up with any reason to do so, I'd assume that there's a DOS attack if I get 100 concurrent incoming connections from a single IP, and a programming error if the server sets up more than 1K outgoing connections Maybe I'm missing the obvious, or have only administered meaningless installations, but I fail to see the real world relevance of this question. I don't blame you for being puzzled, but this not about tomcat accepting connections. It's about tomcat acting as the client, where MySQL is the server. I'm referring to client connections from tomcat to MySQL. We have about 1800 instances of tomcat running. This question comes up once in a while when tomcat can't connect to MySQL. Trust me, it can be an issue. --Eric The question is : is there even any Java API (method) (or even OS API) which allows a client to open a (client, non-LISTEN) socket AND specify the client IP address and/or port ? I mean, if there is none, then the question may be interesting in the absolute, but ultimately pointless. I believe that the IP address of client packets, will be determined by which /route/ the OS determines that the target server address can be reached (which will determine through which network interface the packets "go out", which will determine the sender IP address inserted in the packets). I don't think that the application-level software (here a java webapp) can determine this in advance. And the client port will be decided by the OS-level TCP stack, in function of which ones are not yet in use (which a java webapp can also not determine in advance). Example of creating a client socket : Socket echoSocket = new Socket(hostName, portNumber); The hostname (or IP address of ditto) and port numbers passed as arguments, are the IP:port of the server you are connecting /to/, not the ones of the local socket. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
