Re: is too quick to respond
Neil, > I think it's a bad idea to design a solution based on the file type. A > webapp can contain all kinds of files. Why is that in this instance when the focus is on deploying to these specific folders as defined by Tomcat? Taken from the Tomcat documentation... - WEB-INF/classes/ - This directory contains any Java class files (and associated resources) required for your application, including both servlet and non-servlet classes, that are not combined into JAR files. If your classes are organized into Java packages, you must reflect this in the directory hierarchy under WEB-INF/classes/. For example, a Java class named com.mycompany.mypackage.MyServlet would need to be stored in a file named WEB-INF/classes/com/mycompany/mypackage/MyServlet.class. - WEB-INF/lib/ - This directory contains JAR files that contain Java class files (and associated resources) required for your application, such as third party class libraries or JDBC drivers. So, while there may be additional resources I would have thought that it would be safe to assume that .war, .jar & .class suffixes contained Java classes / resources, raw (.class) or zipped (.war, .jar). Regarding checking the validity of a .war / .jar, there looks to be a good way of checking validity in the standard java.util.zip.ZipFile class which opens a file for reading and will return an exception if there is a ZIP format error. Of course, if we might expect someone to deploy a file with one of the "reserved" suffixes that does not conform then the file will never pass validation and so either would need to give up after a period of time and let the reload continue or raise an exception reporting that the file did not contain the expected type (i.e. a recognised zip file structure). Given that Tomcat is pretty well defined about what those two folders are for, I would have thought that we should be able to expect that users would confirm to only use the recognised suffixes for the purpose they are intended for and the benefits of a safe deployment outway the outside risk of users abusing the meaning of the suffixes. Of course, only those files with those two suffixes (jar / war) would have the additional validation applied, the rest would simply be subject to the timeout for non activity having expired. John Neil,> I think it's a bad idea to design a solution based on the file type. A webapp can contain all kinds of files.Why is that in this instance when the focus is on deploying to these specific folders as defined by Tomcat? WEB-INF/classes/ - This directory contains any Java class files (and associated resources) required for your application, including both servlet and non-servlet classes, that are not combined into JAR files. If your classes are organized into Java packages, you must reflect this in the directory hierarchy under WEB-INF/classes/. For example, a Java class named com.mycompany.mypackage.MyServlet would need to be stored in a file named WEB-INF/classes/com/mycompany/mypackage/MyServlet.class.WEB-INF/lib/ - This directory contains JAR files that contain Java class files (and associated resources) required for your application, such as third party class libraries or JDBC drivers.So, while there may be additional resources I would have thought that it would be safe to assume that .war, .jar & .class suffixes contained Java classes / resourses, raw (.class) or zipped (.war, .jar).Regarding checking the validity of a .war / .jar, there looks to be a good way of checking validity in the standard java.util.zip.ZipFile class which opens a file for reading and will return an exception if there is a ZIP format error.Of course, if we might expect someone to deploy a file with one of the "reserved" suffixes that does not conform then the file will never pass validation and so either would need to give up after a period of time and let the reload continue or raise an exception reporting that the file did not contain the expected type (i.e. a recognised zip file structure). Given that Tomcat is pretty well defined about what those two folders are for, I would have thought that we should be able to expect that users would confirm to only use the recognised suffixes for the purpose they are intended for and the benefits of a safe deployment outway the outside risk of users abusing the meaning of the suffixes.Of course, only those files with those two suffixes (jar / war) would have the additional validation applied, the rest would simply be subject to the timeout for non activity having expired.John Original message From: Neil Aggarwal Date: 20/02/2022 18:29 (GMT+00:00) To: Tomcat Users List Subject: Re: is too quick to respond I think it's a bad idea to design a solution based on the file type. A webapp can contain all kinds of files. On Sun, Feb 20, 2022, 12:16 PM Simon Matter wrote: > > I may be wrong but I thought .war files are zip files. Wouldn't it be > possible to just wait until the file has a consistent
Re: mod_jk interference with ErrorDocument/Alias on HEAD request
Hi, i created a bugzilla entry (https://bz.apache.org/bugzilla/show_bug.cgi?id=65901) and included a first attempt to patch this issue. Am 12.02.2022 um 14:24 schrieb Stefan Mayr: Hello Tomcat users, this week we were debugging a strange connection issue which I tracked down to an interference between Apache httpd and mod_jk. For the full picture, the infrastructure setup contains 1. a Loadbalancer providing HTTPS, HTTP/1.1 and HTTP/2 for Clients. 2. an Apache httpd 2.4 webserver (HTTP only) with mod_jk 3. a Tomcat mit AJP-Connector We have an application doing many different HEAD requests against an application running in the Tomcat server. The requests contain an Authorization header for Basic authentication. Expected response is a HTTP 200 OK or HTTP 401 if this particular user is not allowed to access that ressource. Because this is a HEAD request there must not be a response body according to RFC 2616. If there is a response body in the response to the HEAD request our loadbalancer does strange things: aborts the connection if the clients uses HTTP/2, SSL errors if using HTTP/1.1. But this is an issue on its own which we might have to solve with the vendor. Now comes the point where I need your help. We have a httpd configuration with mod_jk which generates these invalid response bodies on HEAD requests. I have a gut feeling this could be a bug with mod_jk. For demonstration purpose i created a minimal demo app which only is a WEB-INF/web.xml https://jakarta.ee/xml/ns/jakartaee; xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance; xsi:schemaLocation="https://jakarta.ee/xml/ns/jakartaee https://jakarta.ee/xml/ns/jakartaee/web-app_5_0.xsd; version="5.0"> Login /* manager manager BASIC Then I place a JkMount in my Apache httpd configuration (+ minimal worker.properties): JkMount /demo/* ajp13_worker Testing this with curl works like expected: root@1ae8973f1b6b:~# curl -I -v localhost/demo/ * Trying 127.0.0.1:80... * TCP_NODELAY set * Connected to localhost (127.0.0.1) port 80 (#0) > HEAD /demo/ HTTP/1.1 > Host: localhost > User-Agent: curl/7.68.0 > Accept: */* > * Mark bundle as not supporting multiuse < HTTP/1.1 401 401 HTTP/1.1 401 401 < Date: Sat, 12 Feb 2022 12:57:33 GMT Date: Sat, 12 Feb 2022 12:57:33 GMT < Server: Apache/2.4.41 (Ubuntu) Server: Apache/2.4.41 (Ubuntu) < Cache-Control: private Cache-Control: private < WWW-Authenticate: Basic realm="Authentication required" WWW-Authenticate: Basic realm="Authentication required" < Content-Language: en Content-Language: en < Content-Type: text/html;charset=utf-8 Content-Type: text/html;charset=utf-8 < * Connection #0 to host localhost left intact But our default setup always includes custom error pages: Alias /error/ "/usr/share/apache2/error/" ErrorDocument 401 /error/HTTP_UNAUTHORIZED.html.var If both of those lines are added this results in a response body for the HEAD request. root@1ae8973f1b6b:~# curl -I -v localhost/demo/ * Trying 127.0.0.1:80... * TCP_NODELAY set * Connected to localhost (127.0.0.1) port 80 (#0) > HEAD /demo/ HTTP/1.1 > Host: localhost > User-Agent: curl/7.68.0 > Accept: */* > * Mark bundle as not supporting multiuse < HTTP/1.1 401 401 HTTP/1.1 401 401 < Date: Sat, 12 Feb 2022 12:56:27 GMT Date: Sat, 12 Feb 2022 12:56:27 GMT < Server: Apache/2.4.41 (Ubuntu) Server: Apache/2.4.41 (Ubuntu) < Cache-Control: private Cache-Control: private < WWW-Authenticate: Basic realm="Authentication required" WWW-Authenticate: Basic realm="Authentication required" < Content-Language: en Content-Language: en < Content-Type: text/html;charset=utf-8 Content-Type: text/html;charset=utf-8 < * Excess found: excess = 589 url = /demo/ (zero-length body) * Connection #0 to host localhost left intact Checking with tcpdump on port 8009 we see the expected response without a body from the Tomcat AJP connector. The tcpdump von port 80 reveals httpd is adding the configured ErrorDocument as response body. If we comment out either the Alias or ErrorDocument directive the response is correct again. Doing similar tests with CGI/PHP applications always show the correct response without a response body. This only affects requests which use mod_jk. So far I could reproduce this on SLES12 SP5 and SLES15 SP3 running Apache httpd 2.4.51 and a self compile mod_jk 1.2.46 (same with the included mod_jk 1.2.43) at work. At home the same happens on a stock openSUSE Leap 15.3 with Apache httpd 2.4.51 and mod_jk 1.2.43 as well as on Ubuntu 20.04 with httpd 2.4.41 and mod_jk 1.2.46. I didn't try to compile the latest mod_jk version yet because I didn't spot a relevant point in the changelog. Can anyone confirm this behaviour or
Re: is too quick to respond
Simon,That's my understanding as well, just that they have to have a defined folder structure within the zip but can unzip with any utility program. When you say "consistent content", how would you determine that without actually 'suck and see' which would lead to having to interfere with Tomcats reload algorithms, not something I would be comfortable with. Is there a read-only method (e.g. isValidZip() in a Library that is available to Tomcat) that I could call to check this? I will know the file that has been changed / replaced and so know that it has a .war extension.John Original message From: Simon Matter Date: 20/02/2022 18:15 (GMT+00:00) To: Tomcat Users List Subject: Re: is too quick to respond > Not sure about Tomcat, but what IBM Liberty does is:>> It "will" try to redeploy the war when it detects a file change - and it> does fail naturally since the war isn't complete.>> BUT - it will keep trying since during the upload, the timestamp and file> size automatically keeps changing - so at the end, it will succeed in> deploying the whole war file.I may be wrong but I thought .war files are zip files. Wouldn't it bepossible to just wait until the file has a consistent content and thenextract it?Simon>> I wish they would have just monitored the file size for a configurable> "given" time. And lets say - if the file size or timestamp doesn't change> for -say - 15 seconds, then go ahead and do the deployment, but as what> was mentioned earlier, different OS(s) may handle this differently, but> the JAVA NIO API watchevents point you in the right direction in watching> a file/folder in a loop for a "create" or "modify" or "delete" event to> occur and fire off.>> thanks,> jason>> - Original Message -> From: "chris" > To: "users" > Sent: Sunday, February 20, 2022 9:22:17 AM> Subject: Re: is too quick to respond>> John,>> On 2/20/22 05:50, John Barrow wrote:>> Neil, Thanks for your useful feedback. I am still feeling my way as you can>> probably see from my earlier emails trying to setup a development>> environment. I did actually think of this but didn't put it in scope for a couple of>> reasons. Firstly, the Tomcat documentation for readloadable quotes "Set to true if you want Catalina to monitor classes in>> /WEB-INF/classes/ and /WEB-INF/lib for changes, and automatically>> reload the web application if a change is detected. This feature is>> very useful during application development, but it requires>> significant runtime overhead and is not recommended for use on>> deployed production applications. That's why the default setting for>> this attribute is false. You can use the Manager web application,>> however, to trigger reloads of deployed applications on demand." Therefore, I took it to mean that this flag was geared at development,>> not production which is what I assume when you would deploy a .war>> file. So Tomcat would be listening to specific changes in .classes and>> .jar files that had just been compiled and these are normally small in>> size. But then I suppose that a single .jar file may be so sized that>> Tomcat could react while the file was still being written to the disk.>> The patch you are currently working on should fix this aspect of the> overall problem you are trying to solve.>>> Secondly, I sort of assumed that since the feature was already in>> place and handles changes to single files that this check for>> completeness has already been implemented, but then as I can't get a>> development environment to run, I don't have enough skills to drill>> into the sources without it being interactive to help me explore and>> learn. However, it makes sense that your recommendation is implemented,>> although I was imagining setting the delay to (say) 500ms to ensure>> that whatever IDE had time to complete the copying of all the files as>> that is a small price to pay for automatic refresh. Also by resetting>> the timer after each event it would have to be quite a large upload>> for Tomcat to start reacting. Like you, I am not sure how to formally check that a file has>> completed its copy to the destination. The most common suggestion I>> hear is to try and change its name and then change it back again and>> capture the exception which will be raised if the file is locked. I>> wonder whether attempting to set an attribute (e.g.toggle read-only)>> would have the same effect (i.e. only allow if file wasn't locked) and>> be a little more elegant. I would have to try it.>> Don't do anything like that; it won't work on various environments. For> example, Windows obtains exclusive file-locks for even sometimes> read-only operations. But *NIX does /not/. So you may develop something> that works on Windows but doesn't work at all anywhere else.>> You basically can't check to see if a file is "done uploading"" or> whatever else may be happening. What you *can* do is
Re: is too quick to respond
I think it's a bad idea to design a solution based on the file type. A webapp can contain all kinds of files. On Sun, Feb 20, 2022, 12:16 PM Simon Matter wrote: > > I may be wrong but I thought .war files are zip files. Wouldn't it be > possible to just wait until the file has a consistent content and then > extract it? >
Re: is too quick to respond
> Not sure about Tomcat, but what IBM Liberty does is: > > It "will" try to redeploy the war when it detects a file change - and it > does fail naturally since the war isn't complete. > > BUT - it will keep trying since during the upload, the timestamp and file > size automatically keeps changing - so at the end, it will succeed in > deploying the whole war file. I may be wrong but I thought .war files are zip files. Wouldn't it be possible to just wait until the file has a consistent content and then extract it? Simon > > I wish they would have just monitored the file size for a configurable > "given" time. And lets say - if the file size or timestamp doesn't change > for -say - 15 seconds, then go ahead and do the deployment, but as what > was mentioned earlier, different OS(s) may handle this differently, but > the JAVA NIO API watchevents point you in the right direction in watching > a file/folder in a loop for a "create" or "modify" or "delete" event to > occur and fire off. > >thanks, > jason > > - Original Message - > From: "chris" > To: "users" > Sent: Sunday, February 20, 2022 9:22:17 AM > Subject: Re: is too quick to respond > > John, > > On 2/20/22 05:50, John Barrow wrote: >> Neil, >> >> Thanks for your useful feedback. I am still feeling my way as you can >> probably see from my earlier emails trying to setup a development >> environment. >> >> I did actually think of this but didn't put it in scope for a couple of >> reasons. >> >> Firstly, the Tomcat documentation for readloadable quotes >> >> "Set to true if you want Catalina to monitor classes in >> /WEB-INF/classes/ and /WEB-INF/lib for changes, and automatically >> reload the web application if a change is detected. This feature is >> very useful during application development, but it requires >> significant runtime overhead and is not recommended for use on >> deployed production applications. That's why the default setting for >> this attribute is false. You can use the Manager web application, >> however, to trigger reloads of deployed applications on demand." >> >> Therefore, I took it to mean that this flag was geared at development, >> not production which is what I assume when you would deploy a .war >> file. So Tomcat would be listening to specific changes in .classes and >> .jar files that had just been compiled and these are normally small in >> size. But then I suppose that a single .jar file may be so sized that >> Tomcat could react while the file was still being written to the disk. > > The patch you are currently working on should fix this aspect of the > overall problem you are trying to solve. > >> Secondly, I sort of assumed that since the feature was already in >> place and handles changes to single files that this check for >> completeness has already been implemented, but then as I can't get a >> development environment to run, I don't have enough skills to drill >> into the sources without it being interactive to help me explore and >> learn. >> >> However, it makes sense that your recommendation is implemented, >> although I was imagining setting the delay to (say) 500ms to ensure >> that whatever IDE had time to complete the copying of all the files as >> that is a small price to pay for automatic refresh. Also by resetting >> the timer after each event it would have to be quite a large upload >> for Tomcat to start reacting. >> >> Like you, I am not sure how to formally check that a file has >> completed its copy to the destination. The most common suggestion I >> hear is to try and change its name and then change it back again and >> capture the exception which will be raised if the file is locked. I >> wonder whether attempting to set an attribute (e.g.toggle read-only) >> would have the same effect (i.e. only allow if file wasn't locked) and >> be a little more elegant. I would have to try it. > > Don't do anything like that; it won't work on various environments. For > example, Windows obtains exclusive file-locks for even sometimes > read-only operations. But *NIX does /not/. So you may develop something > that works on Windows but doesn't work at all anywhere else. > > You basically can't check to see if a file is "done uploading"" or > whatever else may be happening. What you *can* do is check to see if any > file in the list-of-files-to-be it *too recent* indicating that a > compile/copy/upload/whatever may still be in progress. > >> I assume that Windows has a way of querying a file lock but not sure >> (a) whether that is exposed via a Java API and (b) whether that would >> apply to Unix as well (as I have only ever used Windows for >> development). >> >>> How does Tomcat test if a file has been updated? > > It's just relative timestamps. Dive into the code Mark suggested and > you'll find it. > >> Again, I don't know this yet (lack of IDE again), but I assumed that >> it would be similar to the method I implemented in the attached source >> code, i.e. Create a listener for
RE: is too quick to respond
Hi all, I have also uploaded the screenshots that I took within NetBeans illustrating some of my issues with getting Tomcat working within Netbeans. As mentioned previously, when I did my test project using Ant, it created a Libraries folder where I could and add JAR/Folder. This structure is not available for the imported Tomcat project after having run ant ide-netbeans. Not that I would expect to have to do this as I would have thought that all the relevant configuration settings would already be set up in the build.xml file, so I am missing a vital step somewhere. Finally, the failed tests zip is also there in case it was stripped out for anyone else that might be interested. John *From: *Christopher Schultz *Sent: *20 February 2022 14:22 *To: *users@tomcat.apache.org *Subject: *Re: is too quick to respond John, On 2/20/22 05:50, John Barrow wrote: > Neil, > > Thanks for your useful feedback. I am still feeling my way as you can > probably see from my earlier emails trying to setup a development > environment. > > I did actually think of this but didn't put it in scope for a couple of reasons. > > Firstly, the Tomcat documentation for readloadable quotes > > "Set to true if you want Catalina to monitor classes in > /WEB-INF/classes/ and /WEB-INF/lib for changes, and automatically > reload the web application if a change is detected. This feature is > very useful during application development, but it requires > significant runtime overhead and is not recommended for use on > deployed production applications. That's why the default setting for > this attribute is false. You can use the Manager web application, > however, to trigger reloads of deployed applications on demand." > > Therefore, I took it to mean that this flag was geared at development, > not production which is what I assume when you would deploy a .war > file. So Tomcat would be listening to specific changes in .classes and > .jar files that had just been compiled and these are normally small in > size. But then I suppose that a single .jar file may be so sized that > Tomcat could react while the file was still being written to the disk. The patch you are currently working on should fix this aspect of the overall problem you are trying to solve. > Secondly, I sort of assumed that since the feature was already in > place and handles changes to single files that this check for > completeness has already been implemented, but then as I can't get a > development environment to run, I don't have enough skills to drill > into the sources without it being interactive to help me explore and > learn. > > However, it makes sense that your recommendation is implemented, > although I was imagining setting the delay to (say) 500ms to ensure > that whatever IDE had time to complete the copying of all the files as > that is a small price to pay for automatic refresh. Also by resetting > the timer after each event it would have to be quite a large upload > for Tomcat to start reacting. > > Like you, I am not sure how to formally check that a file has > completed its copy to the destination. The most common suggestion I > hear is to try and change its name and then change it back again and > capture the exception which will be raised if the file is locked. I > wonder whether attempting to set an attribute (e.g.toggle read-only) > would have the same effect (i.e. only allow if file wasn't locked) and > be a little more elegant. I would have to try it. Don't do anything like that; it won't work on various environments. For example, Windows obtains exclusive file-locks for even sometimes read-only operations. But *NIX does /not/. So you may develop something that works on Windows but doesn't work at all anywhere else. You basically can't check to see if a file is "done uploading"" or whatever else may be happening. What you *can* do is check to see if any file in the list-of-files-to-be it *too recent* indicating that a compile/copy/upload/whatever may still be in progress. > I assume that Windows has a way of querying a file lock but not sure > (a) whether that is exposed via a Java API and (b) whether that would > apply to Unix as well (as I have only ever used Windows for > development). > >> How does Tomcat test if a file has been updated? It's just relative timestamps. Dive into the code Mark suggested and you'll find it. > Again, I don't know this yet (lack of IDE again), but I assumed that > it would be similar to the method I implemented in the attached source > code, i.e. Create a listener for events being triggered on file > changes to either /WEB-INF/classes/ and /WEB-INF/lib, as they are both > hard-coded file paths. I don't think your attachment made it to the list. Maybe you can host it somewhere else and then post a URL to the list? Attachments tend to be stripped. I'm actually surprised your ZIP file made it through. > As an
RE: is too quick to respond
Hi all, I will consolidate the comments from the last three emails to keep the thread from splitting too much. > Christopher: I don't think your attachment made it to the list. Maybe you can > host it somewhere else and then post a URL to the list? Attachments tend to > be stripped. I'm actually surprised your ZIP file made it through. Firstly, I wasn’t aware that text attachments would be stripped. I certainly didn’t get any notification although I did get plenty of notifications when I used Windows Mail to reply (no Plain Text support!) The URL for the sources is below which are held in my DropBox account. Hopefully the link below will allow access. Let me know if there are any issues. https://www.dropbox.com/sh/2ewipogzr48qcxi/AAAf3Rqv6WoRO9hyMC0W7P2za?dl=0 > Christopher: Don't do anything like that; it won't work in various > environments. For example, Windows obtains exclusive file-locks for even > sometimes read-only operations. But *NIX does /not/. So you may develop > something that works on Windows but doesn't work at all anywhere else. That was my understanding. When I was working, I remember there not being a comprehensive solution to this requirement and that Unix was very 'flexible' when handling files in flux, hence my query as I was not sure if something more recent had surfaced that I wasn't aware of. I believe that the 'wait a while' approach is (a) practical and (b) minimises conflicts with the way that Tomcat is operating. It is also flexible in that, if you know that you are refreshing large files then giving (say) a minute for the upload to work before Tomcat reloads is reasonable whereas if developing small Servlets then a few microseconds is probably all that is needed so the administrator can tune to the environment. > Christopher: The patch you are currently working on should fix this aspect of > the overall problem you are trying to solve. Thanks for the vote of confidence - hopefully now that you can see / run the sample app, that confidence will remain! > Christopher: It's just relative timestamps. Dive into the code Mark suggested > and you'll find it. That surprises me. I can't wait to dive in once I can get my IDE working. Is Tomcat polling the folder every 'x' microseconds then? That may explain the caution in the Tomcat documentation in that it puts a strain on the server. You will see from the sources that I have uploaded to DropBox, that I mimicked an event model that seemed to work quite well, however I am not sure which model would be more efficient and less intrusive on Tomcat's operation. > Jason: but the JAVA NIO API watchevents point you in the right direction in > watching a file/folder in a loop for a "create" or "modify" or "delete" event > to occur and fire off. As you should now be able to see from my DropBox source files, that is the approach I took although I wasn't looking to implement that, simply mock-up what I thought that Tomcat was already doing to detect the need for a reload. > I wish they would have just monitored the file size for a configurable > "given" time. And lets say - if the file size or timestamp doesn't change > for -say - 15 seconds, then go ahead and do the deployment. That would be easy enough to add in as a fail-safe and while not perfect as it makes assumptions on external factors, we can at least be sure that if either the timestamp or file size has been amended then it is still in flux whilst not offering any guarantees if they are identical. As I get an event for each file/folder that has been amended, I could then add another loop to check stability of these two attributes (say over 10 ms intervals, or add a further parameter to be user configurable) before starting the final 'waitForQuiet' timer prior to notifying Tomcat to start the reload. It is also a read-only query and so has no side-effects as it appears that users have a need to auto-reload after refreshing a larger .war file. Thoughts? Enjoy your evening. John From: Christopher Schultz Sent: 20 February 2022 14:22 To: users@tomcat.apache.org Subject: Re: is too quick to respond John, On 2/20/22 05:50, John Barrow wrote: > Neil, > > Thanks for your useful feedback. I am still feeling my way as you can > probably see from my earlier emails trying to setup a development > environment. > > I did actually think of this but didn't put it in scope for a couple of > reasons. > > Firstly, the Tomcat documentation for readloadable quotes > > "Set to true if you want Catalina to monitor classes in > /WEB-INF/classes/ and /WEB-INF/lib for changes, and automatically > reload the web application if a change is detected. This feature is > very useful during application development, but it requires > significant runtime overhead and is not recommended for use on > deployed production applications. That's why the default setting for > this attribute is false. You can use the Manager web application, > however, to trigger reloads
RE: is too quick to respond
> as what was mentioned earlier, different OS(s) may handle this > differently, but the > JAVA NIO API watchevents point you in the right direction in watching a > file/folder > in a loop for a "create" or "modify" or "delete" event to occur and fire > off. That is a very good point. Using an event driven system is a much better approach than polling. Thank you, Neil -- Neil Aggarwal, (972) 834-1565, http://www.propfinancing.com We offer 30 year loans on single family houses! - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: is too quick to respond
Not sure about Tomcat, but what IBM Liberty does is: It "will" try to redeploy the war when it detects a file change - and it does fail naturally since the war isn't complete. BUT - it will keep trying since during the upload, the timestamp and file size automatically keeps changing - so at the end, it will succeed in deploying the whole war file. I wish they would have just monitored the file size for a configurable "given" time. And lets say - if the file size or timestamp doesn't change for -say - 15 seconds, then go ahead and do the deployment, but as what was mentioned earlier, different OS(s) may handle this differently, but the JAVA NIO API watchevents point you in the right direction in watching a file/folder in a loop for a "create" or "modify" or "delete" event to occur and fire off. thanks, jason - Original Message - From: "chris" To: "users" Sent: Sunday, February 20, 2022 9:22:17 AM Subject: Re: is too quick to respond John, On 2/20/22 05:50, John Barrow wrote: > Neil, > > Thanks for your useful feedback. I am still feeling my way as you can > probably see from my earlier emails trying to setup a development > environment. > > I did actually think of this but didn't put it in scope for a couple of > reasons. > > Firstly, the Tomcat documentation for readloadable quotes > > "Set to true if you want Catalina to monitor classes in > /WEB-INF/classes/ and /WEB-INF/lib for changes, and automatically > reload the web application if a change is detected. This feature is > very useful during application development, but it requires > significant runtime overhead and is not recommended for use on > deployed production applications. That's why the default setting for > this attribute is false. You can use the Manager web application, > however, to trigger reloads of deployed applications on demand." > > Therefore, I took it to mean that this flag was geared at development, > not production which is what I assume when you would deploy a .war > file. So Tomcat would be listening to specific changes in .classes and > .jar files that had just been compiled and these are normally small in > size. But then I suppose that a single .jar file may be so sized that > Tomcat could react while the file was still being written to the disk. The patch you are currently working on should fix this aspect of the overall problem you are trying to solve. > Secondly, I sort of assumed that since the feature was already in > place and handles changes to single files that this check for > completeness has already been implemented, but then as I can't get a > development environment to run, I don't have enough skills to drill > into the sources without it being interactive to help me explore and > learn. > > However, it makes sense that your recommendation is implemented, > although I was imagining setting the delay to (say) 500ms to ensure > that whatever IDE had time to complete the copying of all the files as > that is a small price to pay for automatic refresh. Also by resetting > the timer after each event it would have to be quite a large upload > for Tomcat to start reacting. > > Like you, I am not sure how to formally check that a file has > completed its copy to the destination. The most common suggestion I > hear is to try and change its name and then change it back again and > capture the exception which will be raised if the file is locked. I > wonder whether attempting to set an attribute (e.g.toggle read-only) > would have the same effect (i.e. only allow if file wasn't locked) and > be a little more elegant. I would have to try it. Don't do anything like that; it won't work on various environments. For example, Windows obtains exclusive file-locks for even sometimes read-only operations. But *NIX does /not/. So you may develop something that works on Windows but doesn't work at all anywhere else. You basically can't check to see if a file is "done uploading"" or whatever else may be happening. What you *can* do is check to see if any file in the list-of-files-to-be it *too recent* indicating that a compile/copy/upload/whatever may still be in progress. > I assume that Windows has a way of querying a file lock but not sure > (a) whether that is exposed via a Java API and (b) whether that would > apply to Unix as well (as I have only ever used Windows for > development). > >> How does Tomcat test if a file has been updated? It's just relative timestamps. Dive into the code Mark suggested and you'll find it. > Again, I don't know this yet (lack of IDE again), but I assumed that > it would be similar to the method I implemented in the attached source > code, i.e. Create a listener for events being triggered on file > changes to either /WEB-INF/classes/ and /WEB-INF/lib, as they are both > hard-coded file paths. I don't think your attachment made it to the list. Maybe you can host it somewhere else and then post a URL to the list? Attachments tend to be
Re: is too quick to respond
John, On 2/20/22 05:50, John Barrow wrote: Neil, Thanks for your useful feedback. I am still feeling my way as you can probably see from my earlier emails trying to setup a development environment. I did actually think of this but didn't put it in scope for a couple of reasons. Firstly, the Tomcat documentation for readloadable quotes "Set to true if you want Catalina to monitor classes in /WEB-INF/classes/ and /WEB-INF/lib for changes, and automatically reload the web application if a change is detected. This feature is very useful during application development, but it requires significant runtime overhead and is not recommended for use on deployed production applications. That's why the default setting for this attribute is false. You can use the Manager web application, however, to trigger reloads of deployed applications on demand." Therefore, I took it to mean that this flag was geared at development, not production which is what I assume when you would deploy a .war file. So Tomcat would be listening to specific changes in .classes and .jar files that had just been compiled and these are normally small in size. But then I suppose that a single .jar file may be so sized that Tomcat could react while the file was still being written to the disk. The patch you are currently working on should fix this aspect of the overall problem you are trying to solve. Secondly, I sort of assumed that since the feature was already in place and handles changes to single files that this check for completeness has already been implemented, but then as I can't get a development environment to run, I don't have enough skills to drill into the sources without it being interactive to help me explore and learn. However, it makes sense that your recommendation is implemented, although I was imagining setting the delay to (say) 500ms to ensure that whatever IDE had time to complete the copying of all the files as that is a small price to pay for automatic refresh. Also by resetting the timer after each event it would have to be quite a large upload for Tomcat to start reacting. Like you, I am not sure how to formally check that a file has completed its copy to the destination. The most common suggestion I hear is to try and change its name and then change it back again and capture the exception which will be raised if the file is locked. I wonder whether attempting to set an attribute (e.g.toggle read-only) would have the same effect (i.e. only allow if file wasn't locked) and be a little more elegant. I would have to try it. Don't do anything like that; it won't work on various environments. For example, Windows obtains exclusive file-locks for even sometimes read-only operations. But *NIX does /not/. So you may develop something that works on Windows but doesn't work at all anywhere else. You basically can't check to see if a file is "done uploading"" or whatever else may be happening. What you *can* do is check to see if any file in the list-of-files-to-be it *too recent* indicating that a compile/copy/upload/whatever may still be in progress. I assume that Windows has a way of querying a file lock but not sure (a) whether that is exposed via a Java API and (b) whether that would apply to Unix as well (as I have only ever used Windows for development). How does Tomcat test if a file has been updated? It's just relative timestamps. Dive into the code Mark suggested and you'll find it. Again, I don't know this yet (lack of IDE again), but I assumed that it would be similar to the method I implemented in the attached source code, i.e. Create a listener for events being triggered on file changes to either /WEB-INF/classes/ and /WEB-INF/lib, as they are both hard-coded file paths. I don't think your attachment made it to the list. Maybe you can host it somewhere else and then post a URL to the list? Attachments tend to be stripped. I'm actually surprised your ZIP file made it through. As an aside, I should have mentioned, for anyone interested in this thread and in case not obvious from the source, but to see the sample source code in action, you need to add, modify, rename or delete files within the specified directory in a File Explorer. -chris On Sun, 20 Feb 2022 at 00:04, Neil Aggarwal wrote: John: If anyone has a moment, can you have a quick look and see if what I am proposing seems acceptable. Thinking about when a large file (Such as a war file) is being uploaded to the server, we don't want Tomcat to reload it until after the file has completed upload and is fully formed. How does Tomcat test if a file has been updated? I assume it uses File.lastModified() or something similar. Does anyone know the detailed nuts and bolts of how that works? Does last modified keep changing as the file is uploading or is it just changed at the start of the upload? Is the behavior the same or different across platforms? These questions should be taken into account when designing
Re: is too quick to respond
Neil, Thanks for your useful feedback. I am still feeling my way as you can probably see from my earlier emails trying to setup a development environment. I did actually think of this but didn't put it in scope for a couple of reasons. Firstly, the Tomcat documentation for readloadable quotes "Set to true if you want Catalina to monitor classes in /WEB-INF/classes/ and /WEB-INF/lib for changes, and automatically reload the web application if a change is detected. This feature is very useful during application development, but it requires significant runtime overhead and is not recommended for use on deployed production applications. That's why the default setting for this attribute is false. You can use the Manager web application, however, to trigger reloads of deployed applications on demand." Therefore, I took it to mean that this flag was geared at development, not production which is what I assume when you would deploy a .war file. So Tomcat would be listening to specific changes in .classes and .jar files that had just been compiled and these are normally small in size. But then I suppose that a single .jar file may be so sized that Tomcat could react while the file was still being written to the disk. Secondly, I sort of assumed that since the feature was already in place and handles changes to single files that this check for completeness has already been implemented, but then as I can't get a development environment to run, I don't have enough skills to drill into the sources without it being interactive to help me explore and learn. However, it makes sense that your recommendation is implemented, although I was imagining setting the delay to (say) 500ms to ensure that whatever IDE had time to complete the copying of all the files as that is a small price to pay for automatic refresh. Also by resetting the timer after each event it would have to be quite a large upload for Tomcat to start reacting. Like you, I am not sure how to formally check that a file has completed its copy to the destination. The most common suggestion I hear is to try and change its name and then change it back again and capture the exception which will be raised if the file is locked. I wonder whether attempting to set an attribute (e.g.toggle read-only) would have the same effect (i.e. only allow if file wasn't locked) and be a little more elegant. I would have to try it. I assume that Windows has a way of querying a file lock but not sure (a) whether that is exposed via a Java API and (b) whether that would apply to Unix as well (as I have only ever used Windows for development). > How does Tomcat test if a file has been updated? Again, I don't know this yet (lack of IDE again), but I assumed that it would be similar to the method I implemented in the attached source code, i.e. Create a listener for events being triggered on file changes to either /WEB-INF/classes/ and /WEB-INF/lib, as they are both hard-coded file paths. As an aside, I should have mentioned, for anyone interested in this thread and in case not obvious from the source, but to see the sample source code in action, you need to add, modify, rename or delete files within the specified directory in a File Explorer. John On Sun, 20 Feb 2022 at 00:04, Neil Aggarwal wrote: > > John: > > > If anyone has a moment, can you have a quick look and see if what I am > > proposing seems acceptable. > > Thinking about when a large file (Such as a war file) is being uploaded > to the server, we don't want Tomcat to reload it until after the file has > completed upload and is fully formed. > > How does Tomcat test if a file has been updated? > I assume it uses File.lastModified() or something similar. > > Does anyone know the detailed nuts and bolts of how that works? > Does last modified keep changing as the file is uploading or is it just > changed at the start of the upload? Is the behavior the same or different > across platforms? > > These questions should be taken into account when designing the solution. > > Thank you, > Neil > > -- > Neil Aggarwal, (972) 834-1565, http://www.propfinancing.com > We offer 30 year loans on single family houses! > > - > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
