subject:" is too quick to respond"


John,

On 2/20/22 12:46, John Barrow wrote:

Firstly, I wasn’t aware that text attachments would be stripped. I
certainly didn’t get any notification although I did get plenty of
notifications when I used Windows Mail to reply (no Plain Text
support!)


??! No Pain Text support from Windows Mail? :/ Well, they don't handle 
MIME properly, so why bother allowing plain text?



Christopher: The patch you are currently working on should fix this aspect of 
the overall problem you are trying to solve.


Thanks for the vote of confidence - hopefully now that you can see /
run the sample app, that confidence will remain!


:)


Christopher: It's just relative timestamps. Dive into the code Mark suggested 
and you'll find it.


That surprises me. I can't wait to dive in once I can get my IDE
working. Is Tomcat polling the folder every 'x' microseconds then?
That may explain the caution in the Tomcat documentation in that it
puts a strain on the server. You will see from the sources that I have
uploaded to DropBox, that I mimicked an event model that seemed to
work quite well, however I am not sure which model would be more
efficient and less intrusive on Tomcat's operation.


While the event-model will indeed work when files change on-disk, Tomcat 
allows files and resources to be loaded from non-disk resources, or 
non-whole-files. It's more complicated than you might think.


The code you will have to modify fits on less than 1 page on your screen 
and it will become obvious what's going on. If you just want to change 
the rules for the timing, you can do that very simply. Don't worry too 
much about the frequency of the checks (they are configured and handled 
separately) or how to reload the files: there is one single method whose 
job it is to answer the question "have any reources been modified".


You will just change the implementation to answer the question "have any 
resources been modified, but not-too-recently?"



As I get an event for each file/folder that has been amended, I could
then add another loop to check stability of these two attributes (say
over 10 ms intervals, or add a further parameter to be user
configurable) before starting the final 'waitForQuiet' timer prior to
notifying Tomcat to start the reload. It is also a read-only query and
so has no side-effects as it appears that users have a need to
auto-reload after refreshing a larger .war file. Thoughts?


The check must be fast, otherwise it will hang-up other processes on the 
server. Don't introduce any new loops or stalls. Just return true or 
false as quickly as possible. The process will be repeated during the 
"next" check so you only have to decide not to reload "too early"?


-chris


From: Christopher Schultz
Sent: 20 February 2022 14:22
To: users@tomcat.apache.org
Subject: Re:  is too quick to respond



John,



On 2/20/22 05:50, John Barrow wrote:


Neil,







Thanks for your useful feedback. I am still feeling my way as you can



probably see from my earlier emails trying to setup a development



environment.







I did actually think of this but didn't put it in scope for a couple of reasons.







Firstly, the Tomcat documentation for readloadable quotes







"Set to true if you want Catalina to monitor classes in



/WEB-INF/classes/ and /WEB-INF/lib for changes, and automatically



reload the web application if a change is detected. This feature is



very useful during application development, but it requires



significant runtime overhead and is not recommended for use on



deployed production applications. That's why the default setting for



this attribute is false. You can use the Manager web application,



however, to trigger reloads of deployed applications on demand."







Therefore, I took it to mean that this flag was geared at development,



not production which is what I assume when you would deploy a .war



file. So Tomcat would be listening to specific changes in .classes and



.jar files that had just been compiled and these are normally small in



size. But then I suppose that a single .jar file may be so sized that



Tomcat could react while the file was still being written to the disk.




The patch you are currently working on should fix this aspect of the

overall problem you are trying to solve.




Secondly, I sort of assumed that since the feature was already in



place and handles changes to single files that this check for



completeness has already been implemented, but then as I can't get a



development environment to run, I don't have enough skills to drill



into the sources without it being interactive to help me explore and



learn.







However, it makes sense that your recommendation is implemented,



although I was imagining setting the delay to (say) 500ms to ensure



that whatever IDE had time to complete the copying o

Re: is too quick to respond


John,

On 2/20/22 13:35, John Barrow wrote:
That's my understanding as well, just that they have to have a 
defined folder structure within the zip but can unzip with any 
utility program. When you say "consistent content", how would you 
determine that without actually 'suck and see' which would lead to 
having to interfere with Tomcats reload algorithms, not something I 
would be comfortable with.


The algorithm just needs a little bit of tweaking IMO.


Is there a read-only method (e.g. isValidZip() in a Library that is
available to Tomcat) that I could call to check this? I will know the
file that has been changed / replaced and so know that it has a .war
extension.


Don't bother peeking into ZIP files, JAR file, etc.

Please take a look at the code that is already there (see Mark's hint
about where to start) and you'll see how it's already done. You don't 
even need to use any Java APIs that deal with files-on-disk *at all*.


-chris

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: is too quick to respond


Jason,

On 2/20/22 10:02, Jason Hall wrote:

Not sure about Tomcat, but what IBM Liberty does is:

It "will" try to redeploy the war when it detects a file change - and
it does fail naturally since the war isn't complete. >
BUT - it will keep trying since during the upload, the timestamp and 
file size automatically keeps changing - so at the end, it will

succeed in deploying the whole war file.
This is fragile. The WAR may actually succeed to deploy but still be 
missing some important resources. CNFE is only one possible way to fail. 
Most applications I know of will fail gracefully if they are missing 
configuration files -- e.g. going into a default state. You definitely 
don't want that for a proper production environment.



I wish they would have just monitored the file size for a
configurable "given" time.  And lets say - if the file size or
timestamp doesn't change for -say - 15 seconds, then go ahead and do
the deployment, but as what was mentioned earlier, different OS(s)
may handle this differently, but the JAVA NIO API watchevents point
you in the right direction in watching a file/folder in a loop for a
"create" or "modify" or "delete" event to occur and fire off.
Tomcat isn't using NIO watch events for a few reasons, so that's not a 
great resource for research in this particular case.


The "timestamp doesn't change for 15 seconds" kind of thing is exactly 
what I was suggesting.


-chris


- Original Message -----
From: "chris" 
To: "users" 
Sent: Sunday, February 20, 2022 9:22:17 AM
Subject: Re:  is too quick to respond

John,

On 2/20/22 05:50, John Barrow wrote:

Neil,

Thanks for your useful feedback. I am still feeling my way as you can
probably see from my earlier emails trying to setup a development
environment.

I did actually think of this but didn't put it in scope for a couple of reasons.

Firstly, the Tomcat documentation for readloadable quotes

"Set to true if you want Catalina to monitor classes in
/WEB-INF/classes/ and /WEB-INF/lib for changes, and automatically
reload the web application if a change is detected. This feature is
very useful during application development, but it requires
significant runtime overhead and is not recommended for use on
deployed production applications. That's why the default setting for
this attribute is false. You can use the Manager web application,
however, to trigger reloads of deployed applications on demand."

Therefore, I took it to mean that this flag was geared at development,
not production which is what I assume when you would deploy a .war
file. So Tomcat would be listening to specific changes in .classes and
.jar files that had just been compiled and these are normally small in
size. But then I suppose that a single .jar file may be so sized that
Tomcat could react while the file was still being written to the disk.


The patch you are currently working on should fix this aspect of the
overall problem you are trying to solve.


Secondly, I sort of assumed that since the feature was already in
place and handles changes to single files that this check for
completeness has already been implemented, but then as I can't get a
development environment to run, I don't have enough skills to drill
into the sources without it being interactive to help me explore and
learn.

However, it makes sense that your recommendation is implemented,
although I was imagining setting the delay to (say) 500ms to ensure
that whatever IDE had time to complete the copying of all the files as
that is a small price to pay for automatic refresh. Also by resetting
the timer after each event it would have to be quite a large upload
for Tomcat to start reacting.

Like you, I am not sure how to formally check that a file has
completed its copy to the destination. The most common suggestion I
hear is to try and change its name and then change it back again and
capture the exception which will be raised if the file is locked. I
wonder whether attempting to set an attribute (e.g.toggle read-only)
would have the same effect (i.e. only allow if file wasn't locked) and
be a little more elegant. I would have to try it.


Don't do anything like that; it won't work on various environments. For
example, Windows obtains exclusive file-locks for even sometimes
read-only operations. But *NIX does /not/. So you may develop something
that works on Windows but doesn't work at all anywhere else.

You basically can't check to see if a file is "done uploading"" or
whatever else may be happening. What you *can* do is check to see if any
file in the list-of-files-to-be it *too recent* indicating that a
compile/copy/upload/whatever may still be in progress.


I assume that Windows has a way of querying a file lock but not sure
(a) whether that is exposed via a Java API and (b) whether that woul

Re: is too quick to respond

John,

On 2/19/22 05:27, John Barrow wrote:

I have re-run the Tomcat tests using 4 cores (all my machine has!) and
time came down to 35m 33s. Not bad.

Definitely better. Ask IT for a better computer ;)

I have written my own ‘grep’ to review the output directory and I
think that the number of failures has come down now using all the
cores so some may have been timing issues

There shouldn't be any timing-issues in the unit tests. Anything you
find will be ... interesting.

You shouldn't have had to write your own grep. Just install "bash" on
Windows (I think even MS offers this, now). It should come with some
basic command-line utilities, grep among them.

The 5 that are reported are

C:\Community\Tomcat\tomcat-main\output\build\logs\TEST-jakarta.servlet.http.TestHttpServletDoHeadValidWrite513.NIO.txt
Tests run: 1152, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 57.275 sec

C:\Community\Tomcat\tomcat-main\output\build\logs\TEST-org.apache.coyote.http2.TestStreamProcessor.NIO.txt
Tests run: 9, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 4.282 sec

C:\Community\Tomcat\tomcat-main\output\build\logs\TEST-org.apache.coyote.http2.TestStreamProcessor.NIO2.txt
Tests run: 9, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 4.008 sec

C:\Community\Tomcat\tomcat-main\output\build\logs\TEST-org.apache.jasper.runtime.TestJspRuntimeLibrary.NIO.txt
Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 3.813 sec

C:\Community\Tomcat\tomcat-main\output\build\logs\TEST-org.apache.jasper.runtime.TestJspRuntimeLibrary.NIO2.txt
Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 4.242 sec

I have attached the 5 txt files as listed above for review (if
needed). The first one may still be a timing issue, but I am not in a
position to judge!

If those 5 failures are all you are seeing, then you are pretty much
good-to-go. None of those things would affect the code you are trying to
write/test.

-chris

On Fri, 18 Feb 2022 at 18:16, Mark Thomas wrote:

On 18/02/2022 18:01, John Barrow wrote:

Do you have a list of the tests which fail? Do you know how to get that list?

Yes, although I am on Windows, not UNIX and so I don't have access to
grep. I did a simple search within logs for 'FAILED' and it came up
with about 20 or so. The list of test results has been cleared now
(probably when I next built Tomcat) and so I will need to run them
again but will try using the threads at the same time and then send
you the results in a separate email.

If you haven't cleared out the results from the logs directory you can do:

ant test-status

Mark

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: is too quick to respond

2022-02-21 Thread Mark Thomas


John,

I'm going back a few emails in this thread but that is because things 
appear to be heading in the wrong direction. This is an attempt to get 
you back on the right track.


On 18/02/2022 15:50, John Barrow wrote:




I use Netbeans 12.6 (Apache loyalty!) on Windows 10 for all my Java
development using JDK 17.


Ack. I use Eclipse so I can't provide NetBeans specific help but I 
should be able to point you in the right direction.



Unfortunately, I am only ‘sort-of’ familiar with Maven and have never
touched Ant so I am flying a bit blind here, but that shouldn’t matter
if it all works as I don't need to touch the build process.

I downloaded the ‘main’ branch from GitHub
(https://github.com/apache/tomcat) and unzipped it into its own
folder, again with Ant (apache-ant-1.9.16).


If you eventual aim is to provide a pull request to address this issue 
then I'd recommend:

- Fork the Tomcat project in GitHub
- Checkout your clone


I have managed to get Ant to build it using instructions in
BUILDING.txt. While it says that Tomcat built successfully, scrolling
up, I got lots of warnings saying ‘can’t find a module etc so not sure
if doing something wrong. I can send the results as an attachment if
that would be useful.


Hmm. You shouldn't see any warnings about modules. A handful of warnings 
about deprecation are expected. If you still see these warnings then 
yes, please provide a copy so we can take a look.


The simplest way to test the build is to try the following:
cd output\build\bin
catalina.bat run

That should start a Tomcat instance in the console. Check 
http://localhost:8080 returns a web page in your browser and then stop 
Tomcat with CTRL-C



I can also get it to build in Netbeans (I first had to set  Project >
Properties > Java Sources to Source Level 17 which may be a bit of a
guess as Netbeans complained about default value of 1.7). The java
files in Netbeans don’t highlight any exceptions, just warnings.


Tomcat 10.1.x (the current main branch) requires Java 11 as a minimum. 
You should be fine setting the source level to anything 11 or above. 
Personally, I'd set it to 11 as that way you avoid the (very low) risk 
of proposing a fix that doesn't work on Java 11.



I appear to have issues with the other two folders
‘C:\Community\Tomcat\tomcat-main\webapps\docs\appdev\sample\src\mypackage’


Tell NetBeans to ignore that folder. It should not be part of the source 
you are trying to compile. It is part of a (rather old) tutorial on 
building a web application.



and ‘C:\Community\Tomcat\tomcat-main\test’ as they don’t seem to be
able to locate the dependencies (HelloWorld is -e.g. import
jakarta.servlet.http.HttpServlet & tests is -e.g. import
org.junit.Assert). In Maven, I would add a dependency to the pom.xml
file, but not sure what to do in Ant as I would expect the
configuration to already be set up.


The problem there is that the JUnit reference in the NetBeans 
configuration wasn't updated when the JUnit version was updated.


I've just fixed that.

This is where using a clone would speed things up as you'd be able to 
fetch the update from Git.


Once you have the updated sources see 
res/ide-support/netbeans/README.txt about running:


ant ide-netbeans-replace


I also ran the ‘ant test’, took about 90 minutes and failed a number
of tests (which I thought was surprising as I would expect sources in
the repository to pass all the tests. I can provide examples if useful
as it may indicate that I have done something wrong.


I wouldn't worry too much about these right now. There are various tests 
that are sensitive to timing, network configuration and/or OS configuration.


What I would recommend is that if you change code, you run the tests 
associated with the class/classes you have changed. That should catch 
most issues.


We can work on getting the full test suite passing but I'd suggest that 
that is not the priority.



I read through the ‘RUNNING.txt’ file, but that assumes a binary to
start, rather than following on from BUILDING Tomcat and so I couldn’t
relate that to what I was trying to achieve.


After a successful build, the "output/build" directory will contain a 
ready to run Tomcat instance. It is both CATALINA_HOME and CATALINA_BASE 
so RUNNING.txt should make sense if you start in that directory.



However, if I launch Tomcat from within Netbeans (F6), I get a dialog
which I don’t understand - probably due to my lack of familiarity with
Ant. Again, I can include a screenshot if that would help.


Looking at res/ide-support/netbeans/README.txt, I don't believe that 
mode of operation is supported. You should be able to run Tomcat from 
the command line with debugging enabled:


catalina.bat jpda run

and then remote debug in NetBeans.


I need to be able to run the locally built version of Tomcat so that,
as a starting point, I can add some System.out.print() messages around
the relevant routines (which I have located – thanks Mark). Of course
I am assu

Re: is too quick to respond

Neil,

> I think it's a bad idea to design a solution based on the file type.  A 
> webapp can contain all kinds of files.

Why is that in this instance when the focus is on deploying to these
specific folders as defined by Tomcat?

Taken from the Tomcat documentation...

- WEB-INF/classes/ - This directory contains any Java class files (and
associated resources) required for your application, including both
servlet and non-servlet classes, that are not combined into JAR files.
If your classes are organized into Java packages, you must reflect
this in the directory hierarchy under WEB-INF/classes/. For example, a
Java class named com.mycompany.mypackage.MyServlet would need to be
stored in a file named
WEB-INF/classes/com/mycompany/mypackage/MyServlet.class.

- WEB-INF/lib/ - This directory contains JAR files that contain Java
class files (and associated resources) required for your application,
such as third party class libraries or JDBC drivers.

So, while there may be additional resources I would have thought that
it would be safe to assume that .war, .jar & .class suffixes contained
Java classes / resources, raw (.class) or zipped (.war, .jar).

Regarding checking the validity of a .war / .jar, there looks to be a
good way of checking validity in the standard  java.util.zip.ZipFile
class which opens a file for reading and will return an exception if
there is a ZIP format error.

Of course, if we might expect someone to deploy a file with one of the
"reserved" suffixes that does not conform then the file will never
pass validation and so either would need to give up after a period of
time and let the reload continue or raise an exception reporting that
the file did not contain the expected type (i.e. a recognised zip file
structure).

Given that Tomcat is pretty well defined about what those two folders
are for, I would have thought that we should be able to expect that
users would confirm to only use the recognised suffixes for the
purpose they are intended for and the benefits of a safe deployment
outway the outside risk of users abusing the meaning of the suffixes.

Of course, only those files with those two suffixes (jar / war) would
have the additional validation applied, the rest would simply be
subject to the timeout for non activity having expired.

John

Neil,> I think it's a bad idea to design a solution based on the file
type.  A webapp can contain all kinds of files.Why is that in this
instance when the focus is on deploying to these specific folders as
defined by Tomcat? WEB-INF/classes/ - This directory contains any Java
class files (and associated resources) required for your application,
including both servlet and non-servlet classes, that are not combined
into JAR files. If your classes are organized into Java packages, you
must reflect this in the directory hierarchy under WEB-INF/classes/.
For example, a Java class named com.mycompany.mypackage.MyServlet
would need to be stored in a file named
WEB-INF/classes/com/mycompany/mypackage/MyServlet.class.WEB-INF/lib/ -
This directory contains JAR files that contain Java class files (and
associated resources) required for your application, such as third
party class libraries or JDBC drivers.So, while there may be
additional resources I would have thought that it would be safe to
assume that .war, .jar & .class suffixes contained Java classes /
resourses, raw (.class) or zipped (.war, .jar).Regarding checking the
validity of a .war / .jar, there looks to be a good way of checking
validity in the standard  java.util.zip.ZipFile class which opens a
file for reading and will return an exception if there is a ZIP format
error.Of course, if we might expect someone to deploy a file with one
of the "reserved" suffixes that does not conform then the file will
never pass validation and so either would need to give up after a
period of time and let the reload continue or raise an exception
reporting that the file did not contain the expected type (i.e. a
recognised zip file structure). Given that Tomcat is pretty well
defined about what those two folders are for, I would have thought
that we should be able to expect that users would confirm to only use
the recognised suffixes for the purpose they are intended for and the
benefits of a safe deployment outway the outside risk of users abusing
the meaning of the suffixes.Of course, only those files with those two
suffixes (jar / war) would have the additional validation applied, the
rest would simply be subject to the timeout for non activity having
expired.John
 Original message 
From: Neil Aggarwal 
Date: 20/02/2022 18:29 (GMT+00:00)
To: Tomcat Users List 
Subject: Re: is too quick to respond

I think it's a bad idea to design a solution based on the file type.  A
webapp can contain all kinds of files.

On Sun, Feb 20, 2022, 12:16 PM Simon Matter  wrote:

>
> I may be wrong but I thought .war files are zip files. Wouldn'

Re: is too quick to respond

Simon,That's my understanding as well, just that they have to have a defined 
folder structure within the zip but can unzip with any utility program. When 
you say "consistent content", how would you determine that without actually 
'suck and see' which would lead to having to interfere with Tomcats reload 
algorithms, not something I would be comfortable with. Is there a read-only 
method (e.g. isValidZip() in a Library that is available to Tomcat) that I 
could call to check this? I will know the file that has been changed / replaced 
and so know that it has a .war extension.John
 Original message From: Simon Matter  
Date: 20/02/2022  18:15  (GMT+00:00) To: Tomcat Users List 
 Subject: Re:  is too quick to respond > Not sure 
about Tomcat, but what IBM Liberty does is:>> It "will" try to redeploy the war 
when it detects a file change - and it> does fail naturally since the war isn't 
complete.>> BUT - it will keep trying since during the upload, the timestamp 
and file> size automatically keeps changing - so at the end, it will succeed 
in> deploying the whole war file.I may be wrong but I thought .war files are 
zip files. Wouldn't it bepossible to just wait until the file has a consistent 
content and thenextract it?Simon>> I wish they would have just monitored the 
file size for a configurable> "given" time.  And lets say - if the file size or 
timestamp doesn't change> for -say - 15 seconds, then go ahead and do the 
deployment, but as what> was mentioned earlier, different OS(s) may handle this 
differently, but> the JAVA NIO API watchevents point you in the right direction 
in watching> a file/folder in a loop for a "create" or "modify" or "delete" 
event to> occur and fire off.>>    thanks,>       jason>> - Original 
Message -> From: "chris" > To: "users" 
> Sent: Sunday, February 20, 2022 9:22:17 AM> Subject: 
Re:  is too quick to respond>> John,>> On 
2/20/22 05:50, John Barrow wrote:>> Neil,>>>> Thanks for your useful feedback. 
I am still feeling my way as you can>> probably see from my earlier emails 
trying to setup a development>> environment.>>>> I did actually think of this 
but didn't put it in scope for a couple of>> reasons.>>>> Firstly, the Tomcat 
documentation for readloadable quotes>>>> "Set to true if you want Catalina to 
monitor classes in>> /WEB-INF/classes/ and /WEB-INF/lib for changes, and 
automatically>> reload the web application if a change is detected. This 
feature is>> very useful during application development, but it requires>> 
significant runtime overhead and is not recommended for use on>> deployed 
production applications. That's why the default setting for>> this attribute is 
false. You can use the Manager web application,>> however, to trigger reloads 
of deployed applications on demand.">>>> Therefore, I took it to mean that this 
flag was geared at development,>> not production which is what I assume when 
you would deploy a .war>> file. So Tomcat would be listening to specific 
changes in .classes and>> .jar files that had just been compiled and these are 
normally small in>> size. But then I suppose that a single .jar file may be so 
sized that>> Tomcat could react while the file was still being written to the 
disk.>> The patch you are currently working on should fix this aspect of the> 
overall problem you are trying to solve.>>> Secondly, I sort of assumed that 
since the feature was already in>> place and handles changes to single files 
that this check for>> completeness has already been implemented, but then as I 
can't get a>> development environment to run, I don't have enough skills to 
drill>> into the sources without it being interactive to help me explore and>> 
learn.>>>> However, it makes sense that your recommendation is implemented,>> 
although I was imagining setting the delay to (say) 500ms to ensure>> that 
whatever IDE had time to complete the copying of all the files as>> that is a 
small price to pay for automatic refresh. Also by resetting>> the timer after 
each event it would have to be quite a large upload>> for Tomcat to start 
reacting.>>>> Like you, I am not sure how to formally check that a file has>> 
completed its copy to the destination. The most common suggestion I>> hear is 
to try and change its name and then change it back again and>> capture the 
exception which will be raised if the file is locked. I>> wonder whether 
attempting to set an attribute (e.g.toggle read-only)>> would have t

Re: is too quick to respond

2022-02-20 Thread Neil Aggarwal

I think it's a bad idea to design a solution based on the file type.  A
webapp can contain all kinds of files.

On Sun, Feb 20, 2022, 12:16 PM Simon Matter  wrote:

>
> I may be wrong but I thought .war files are zip files. Wouldn't it be
> possible to just wait until the file has a consistent content and then
> extract it?
>

Re: is too quick to respond

2022-02-20 Thread Simon Matter

> Not sure about Tomcat, but what IBM Liberty does is:
>
> It "will" try to redeploy the war when it detects a file change - and it
> does fail naturally since the war isn't complete.
>
> BUT - it will keep trying since during the upload, the timestamp and file
> size automatically keeps changing - so at the end, it will succeed in
> deploying the whole war file.

I may be wrong but I thought .war files are zip files. Wouldn't it be
possible to just wait until the file has a consistent content and then
extract it?

Simon

>
> I wish they would have just monitored the file size for a configurable
> "given" time.  And lets say - if the file size or timestamp doesn't change
> for -say - 15 seconds, then go ahead and do the deployment, but as what
> was mentioned earlier, different OS(s) may handle this differently, but
> the JAVA NIO API watchevents point you in the right direction in watching
> a file/folder in a loop for a "create" or "modify" or "delete" event to
> occur and fire off.
>
>thanks,
>   jason
>
> - Original Message -
> From: "chris" 
> To: "users" 
> Sent: Sunday, February 20, 2022 9:22:17 AM
> Subject: Re:  is too quick to respond
>
> John,
>
> On 2/20/22 05:50, John Barrow wrote:
>> Neil,
>>
>> Thanks for your useful feedback. I am still feeling my way as you can
>> probably see from my earlier emails trying to setup a development
>> environment.
>>
>> I did actually think of this but didn't put it in scope for a couple of
>> reasons.
>>
>> Firstly, the Tomcat documentation for readloadable quotes
>>
>> "Set to true if you want Catalina to monitor classes in
>> /WEB-INF/classes/ and /WEB-INF/lib for changes, and automatically
>> reload the web application if a change is detected. This feature is
>> very useful during application development, but it requires
>> significant runtime overhead and is not recommended for use on
>> deployed production applications. That's why the default setting for
>> this attribute is false. You can use the Manager web application,
>> however, to trigger reloads of deployed applications on demand."
>>
>> Therefore, I took it to mean that this flag was geared at development,
>> not production which is what I assume when you would deploy a .war
>> file. So Tomcat would be listening to specific changes in .classes and
>> .jar files that had just been compiled and these are normally small in
>> size. But then I suppose that a single .jar file may be so sized that
>> Tomcat could react while the file was still being written to the disk.
>
> The patch you are currently working on should fix this aspect of the
> overall problem you are trying to solve.
>
>> Secondly, I sort of assumed that since the feature was already in
>> place and handles changes to single files that this check for
>> completeness has already been implemented, but then as I can't get a
>> development environment to run, I don't have enough skills to drill
>> into the sources without it being interactive to help me explore and
>> learn.
>>
>> However, it makes sense that your recommendation is implemented,
>> although I was imagining setting the delay to (say) 500ms to ensure
>> that whatever IDE had time to complete the copying of all the files as
>> that is a small price to pay for automatic refresh. Also by resetting
>> the timer after each event it would have to be quite a large upload
>> for Tomcat to start reacting.
>>
>> Like you, I am not sure how to formally check that a file has
>> completed its copy to the destination. The most common suggestion I
>> hear is to try and change its name and then change it back again and
>> capture the exception which will be raised if the file is locked. I
>> wonder whether attempting to set an attribute (e.g.toggle read-only)
>> would have the same effect (i.e. only allow if file wasn't locked) and
>> be a little more elegant. I would have to try it.
>
> Don't do anything like that; it won't work on various environments. For
> example, Windows obtains exclusive file-locks for even sometimes
> read-only operations. But *NIX does /not/. So you may develop something
> that works on Windows but doesn't work at all anywhere else.
>
> You basically can't check to see if a file is "done uploading"" or
> whatever else may be happening. What you *can* do is check to see if any
> file in the list-of-files-to-be it *too recent* indicating that a
> compile/copy/upload/whatever may still be in p

RE: is too quick to respond

Hi all,



I have also uploaded the screenshots that I took within NetBeans
illustrating some of my issues with getting Tomcat working within Netbeans.



As mentioned previously, when I did my test project using Ant, it created a
Libraries folder where I could  and add JAR/Folder. This
structure is not available for the imported Tomcat project after having run
ant ide-netbeans. Not that I would expect to have to do this as I would
have thought that all the relevant configuration settings would already be
set up in the build.xml file, so I am missing a vital step somewhere.



Finally, the failed tests zip is also there in case it was stripped out for
anyone else that might be interested.


John



*From: *Christopher Schultz 
*Sent: *20 February 2022 14:22
*To: *users@tomcat.apache.org
*Subject: *Re:  is too quick to respond



John,



On 2/20/22 05:50, John Barrow wrote:

> Neil,

>

> Thanks for your useful feedback. I am still feeling my way as you can

> probably see from my earlier emails trying to setup a development

> environment.

>

> I did actually think of this but didn't put it in scope for a couple of
reasons.

>

> Firstly, the Tomcat documentation for readloadable quotes

>

> "Set to true if you want Catalina to monitor classes in

> /WEB-INF/classes/ and /WEB-INF/lib for changes, and automatically

> reload the web application if a change is detected. This feature is

> very useful during application development, but it requires

> significant runtime overhead and is not recommended for use on

> deployed production applications. That's why the default setting for

> this attribute is false. You can use the Manager web application,

> however, to trigger reloads of deployed applications on demand."

>

> Therefore, I took it to mean that this flag was geared at development,

> not production which is what I assume when you would deploy a .war

> file. So Tomcat would be listening to specific changes in .classes and

> .jar files that had just been compiled and these are normally small in

> size. But then I suppose that a single .jar file may be so sized that

> Tomcat could react while the file was still being written to the disk.



The patch you are currently working on should fix this aspect of the

overall problem you are trying to solve.



> Secondly, I sort of assumed that since the feature was already in

> place and handles changes to single files that this check for

> completeness has already been implemented, but then as I can't get a

> development environment to run, I don't have enough skills to drill

> into the sources without it being interactive to help me explore and

> learn.

>

> However, it makes sense that your recommendation is implemented,

> although I was imagining setting the delay to (say) 500ms to ensure

> that whatever IDE had time to complete the copying of all the files as

> that is a small price to pay for automatic refresh. Also by resetting

> the timer after each event it would have to be quite a large upload

> for Tomcat to start reacting.

>

> Like you, I am not sure how to formally check that a file has

> completed its copy to the destination. The most common suggestion I

> hear is to try and change its name and then change it back again and

> capture the exception which will be raised if the file is locked. I

> wonder whether attempting to set an attribute (e.g.toggle read-only)

> would have the same effect (i.e. only allow if file wasn't locked) and

> be a little more elegant. I would have to try it.



Don't do anything like that; it won't work on various environments. For

example, Windows obtains exclusive file-locks for even sometimes

read-only operations. But *NIX does /not/. So you may develop something

that works on Windows but doesn't work at all anywhere else.



You basically can't check to see if a file is "done uploading"" or

whatever else may be happening. What you *can* do is check to see if any

file in the list-of-files-to-be it *too recent* indicating that a

compile/copy/upload/whatever may still be in progress.



> I assume that Windows has a way of querying a file lock but not sure

> (a) whether that is exposed via a Java API and (b) whether that would

> apply to Unix as well (as I have only ever used Windows for

> development).

>

>> How does Tomcat test if a file has been updated?



It's just relative timestamps. Dive into the code Mark suggested and

you'll find it.



> Again, I don't know this yet (lack of IDE again), but I assumed that

> it would be similar to the method I implemented in the attached source

> code, i.e. Create a listener for events being triggered on file

> changes to either /WEB-INF/classes/ and /WEB-INF/lib, as they are both

RE: is too quick to respond

Hi all,

I will consolidate the comments from the last three emails to keep the
thread from splitting too much.

> Christopher: I don't think your attachment made it to the list. Maybe you can 
> host it somewhere else and then post a URL to the list? Attachments tend to 
> be stripped. I'm actually surprised your ZIP file made it through.

Firstly, I wasn’t aware that text attachments would be stripped. I
certainly didn’t get any notification although I did get plenty of
notifications when I used Windows Mail to reply (no Plain Text
support!)

The URL for the sources is below which are held in my DropBox account.
Hopefully the link below will allow access. Let me know if there are
any issues.

https://www.dropbox.com/sh/2ewipogzr48qcxi/AAAf3Rqv6WoRO9hyMC0W7P2za?dl=0

> Christopher: Don't do anything like that; it won't work in various 
> environments. For example, Windows obtains exclusive file-locks for even 
> sometimes read-only operations. But *NIX does /not/. So you may develop 
> something that works on Windows but doesn't work at all anywhere else.

That was my understanding. When I was working, I remember there not
being a comprehensive solution to this requirement and that Unix was
very 'flexible' when handling files in flux, hence my query as I was
not sure if something more recent had surfaced that I wasn't aware of.

I believe that the 'wait a while' approach is (a) practical and (b)
minimises conflicts with the way that Tomcat is operating. It is also
flexible in that, if you know that you are refreshing large files then
giving (say) a minute for the upload to work before Tomcat reloads is
reasonable whereas if developing small Servlets then a few
microseconds is probably all that is needed so the administrator can
tune to the environment.

> Christopher: The patch you are currently working on should fix this aspect of 
> the overall problem you are trying to solve.

Thanks for the vote of confidence - hopefully now that you can see /
run the sample app, that confidence will remain!

> Christopher: It's just relative timestamps. Dive into the code Mark suggested 
> and you'll find it.

That surprises me. I can't wait to dive in once I can get my IDE
working. Is Tomcat polling the folder every 'x' microseconds then?
That may explain the caution in the Tomcat documentation in that it
puts a strain on the server. You will see from the sources that I have
uploaded to DropBox, that I mimicked an event model that seemed to
work quite well, however I am not sure which model would be more
efficient and less intrusive on Tomcat's operation.

> Jason: but the JAVA NIO API watchevents point you in the right direction in 
> watching a file/folder in a loop for a "create" or "modify" or "delete" event 
> to occur and fire off.

As you should now be able to see from my DropBox source files, that is
the approach I took although I wasn't looking to implement that,
simply mock-up what I thought that Tomcat was already doing to detect
the need for a reload.

> I wish they would have just monitored the file size for a configurable 
> "given" time.  And lets say - if the file size or timestamp doesn't change 
> for -say - 15 seconds, then go ahead and do the deployment.

That would be easy enough to add in as a fail-safe and while not
perfect as it makes assumptions on external factors, we can at least
be sure that if either the timestamp or file size has been amended
then it is still in flux whilst not offering any guarantees if they
are identical.

As I get an event for each file/folder that has been amended, I could
then add another loop to check stability of these two attributes (say
over 10 ms intervals, or add a further parameter to be user
configurable) before starting the final 'waitForQuiet' timer prior to
notifying Tomcat to start the reload. It is also a read-only query and
so has no side-effects as it appears that users have a need to
auto-reload after refreshing a larger .war file. Thoughts?

Enjoy your evening.

John

From: Christopher Schultz
Sent: 20 February 2022 14:22
To: users@tomcat.apache.org
Subject: Re:  is too quick to respond

John,

On 2/20/22 05:50, John Barrow wrote:

> Neil,

>

> Thanks for your useful feedback. I am still feeling my way as you can

> probably see from my earlier emails trying to setup a development

> environment.

>

> I did actually think of this but didn't put it in scope for a couple of 
> reasons.

>

> Firstly, the Tomcat documentation for readloadable quotes

>

> "Set to true if you want Catalina to monitor classes in

> /WEB-INF/classes/ and /WEB-INF/lib for changes, and automatically

> reload the web application if a change is detected. This feature is

> very useful during application development, but it

RE: is too quick to respond

2022-02-20 Thread Neil Aggarwal

> as what was mentioned earlier, different OS(s) may handle this
> differently, but the
> JAVA NIO API watchevents point you in the right direction in watching a
> file/folder
> in a loop for a "create" or "modify" or "delete" event to occur and fire
> off.

That is a very good point.  Using an event driven system is a much better
approach
than polling.

Thank you,
  Neil

--
Neil Aggarwal, (972) 834-1565, http://www.propfinancing.com
We offer 30 year loans on single family houses!

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: is too quick to respond

2022-02-20 Thread Jason Hall

Not sure about Tomcat, but what IBM Liberty does is:

It "will" try to redeploy the war when it detects a file change - and it does 
fail naturally since the war isn't complete.

BUT - it will keep trying since during the upload, the timestamp and file size 
automatically keeps changing - so at the end, it will succeed in deploying the 
whole war file.

I wish they would have just monitored the file size for a configurable "given" 
time.  And lets say - if the file size or timestamp doesn't change for -say - 
15 seconds, then go ahead and do the deployment, but as what was mentioned 
earlier, different OS(s) may handle this differently, but the JAVA NIO API 
watchevents point you in the right direction in watching a file/folder in a 
loop for a "create" or "modify" or "delete" event to occur and fire off.

   thanks,
  jason

- Original Message -
From: "chris" 
To: "users" 
Sent: Sunday, February 20, 2022 9:22:17 AM
Subject: Re:  is too quick to respond

John,

On 2/20/22 05:50, John Barrow wrote:
> Neil,
> 
> Thanks for your useful feedback. I am still feeling my way as you can
> probably see from my earlier emails trying to setup a development
> environment.
> 
> I did actually think of this but didn't put it in scope for a couple of 
> reasons.
> 
> Firstly, the Tomcat documentation for readloadable quotes
> 
> "Set to true if you want Catalina to monitor classes in
> /WEB-INF/classes/ and /WEB-INF/lib for changes, and automatically
> reload the web application if a change is detected. This feature is
> very useful during application development, but it requires
> significant runtime overhead and is not recommended for use on
> deployed production applications. That's why the default setting for
> this attribute is false. You can use the Manager web application,
> however, to trigger reloads of deployed applications on demand."
> 
> Therefore, I took it to mean that this flag was geared at development,
> not production which is what I assume when you would deploy a .war
> file. So Tomcat would be listening to specific changes in .classes and
> .jar files that had just been compiled and these are normally small in
> size. But then I suppose that a single .jar file may be so sized that
> Tomcat could react while the file was still being written to the disk.

The patch you are currently working on should fix this aspect of the 
overall problem you are trying to solve.

> Secondly, I sort of assumed that since the feature was already in
> place and handles changes to single files that this check for
> completeness has already been implemented, but then as I can't get a
> development environment to run, I don't have enough skills to drill
> into the sources without it being interactive to help me explore and
> learn.
> 
> However, it makes sense that your recommendation is implemented,
> although I was imagining setting the delay to (say) 500ms to ensure
> that whatever IDE had time to complete the copying of all the files as
> that is a small price to pay for automatic refresh. Also by resetting
> the timer after each event it would have to be quite a large upload
> for Tomcat to start reacting.
> 
> Like you, I am not sure how to formally check that a file has
> completed its copy to the destination. The most common suggestion I
> hear is to try and change its name and then change it back again and
> capture the exception which will be raised if the file is locked. I
> wonder whether attempting to set an attribute (e.g.toggle read-only)
> would have the same effect (i.e. only allow if file wasn't locked) and
> be a little more elegant. I would have to try it.

Don't do anything like that; it won't work on various environments. For 
example, Windows obtains exclusive file-locks for even sometimes 
read-only operations. But *NIX does /not/. So you may develop something 
that works on Windows but doesn't work at all anywhere else.

You basically can't check to see if a file is "done uploading"" or 
whatever else may be happening. What you *can* do is check to see if any 
file in the list-of-files-to-be it *too recent* indicating that a 
compile/copy/upload/whatever may still be in progress.

> I assume that Windows has a way of querying a file lock but not sure
> (a) whether that is exposed via a Java API and (b) whether that would
> apply to Unix as well (as I have only ever used Windows for
> development).
> 
>> How does Tomcat test if a file has been updated?

It's just relative timestamps. Dive into the code Mark suggested and 
you'll find it.

> Again, I don't know this yet (lack of IDE again), but I assumed that
> it would be similar to the method I implemented

Re: is too quick to respond

2022-02-20 Thread Christopher Schultz


John,

On 2/20/22 05:50, John Barrow wrote:

Neil,

Thanks for your useful feedback. I am still feeling my way as you can
probably see from my earlier emails trying to setup a development
environment.

I did actually think of this but didn't put it in scope for a couple of reasons.

Firstly, the Tomcat documentation for readloadable quotes

"Set to true if you want Catalina to monitor classes in
/WEB-INF/classes/ and /WEB-INF/lib for changes, and automatically
reload the web application if a change is detected. This feature is
very useful during application development, but it requires
significant runtime overhead and is not recommended for use on
deployed production applications. That's why the default setting for
this attribute is false. You can use the Manager web application,
however, to trigger reloads of deployed applications on demand."

Therefore, I took it to mean that this flag was geared at development,
not production which is what I assume when you would deploy a .war
file. So Tomcat would be listening to specific changes in .classes and
.jar files that had just been compiled and these are normally small in
size. But then I suppose that a single .jar file may be so sized that
Tomcat could react while the file was still being written to the disk.


The patch you are currently working on should fix this aspect of the 
overall problem you are trying to solve.



Secondly, I sort of assumed that since the feature was already in
place and handles changes to single files that this check for
completeness has already been implemented, but then as I can't get a
development environment to run, I don't have enough skills to drill
into the sources without it being interactive to help me explore and
learn.

However, it makes sense that your recommendation is implemented,
although I was imagining setting the delay to (say) 500ms to ensure
that whatever IDE had time to complete the copying of all the files as
that is a small price to pay for automatic refresh. Also by resetting
the timer after each event it would have to be quite a large upload
for Tomcat to start reacting.

Like you, I am not sure how to formally check that a file has
completed its copy to the destination. The most common suggestion I
hear is to try and change its name and then change it back again and
capture the exception which will be raised if the file is locked. I
wonder whether attempting to set an attribute (e.g.toggle read-only)
would have the same effect (i.e. only allow if file wasn't locked) and
be a little more elegant. I would have to try it.


Don't do anything like that; it won't work on various environments. For 
example, Windows obtains exclusive file-locks for even sometimes 
read-only operations. But *NIX does /not/. So you may develop something 
that works on Windows but doesn't work at all anywhere else.


You basically can't check to see if a file is "done uploading"" or 
whatever else may be happening. What you *can* do is check to see if any 
file in the list-of-files-to-be it *too recent* indicating that a 
compile/copy/upload/whatever may still be in progress.



I assume that Windows has a way of querying a file lock but not sure
(a) whether that is exposed via a Java API and (b) whether that would
apply to Unix as well (as I have only ever used Windows for
development).


How does Tomcat test if a file has been updated?


It's just relative timestamps. Dive into the code Mark suggested and 
you'll find it.



Again, I don't know this yet (lack of IDE again), but I assumed that
it would be similar to the method I implemented in the attached source
code, i.e. Create a listener for events being triggered on file
changes to either /WEB-INF/classes/ and /WEB-INF/lib, as they are both
hard-coded file paths.


I don't think your attachment made it to the list. Maybe you can host it 
somewhere else and then post a URL to the list? Attachments tend to be 
stripped. I'm actually surprised your ZIP file made it through.



As an aside, I should have mentioned, for anyone interested in this
thread and in case not obvious from the source, but to see the sample
source code in action, you need to add, modify, rename or delete files
within the specified directory in a File Explorer.


-chris


On Sun, 20 Feb 2022 at 00:04, Neil Aggarwal  wrote:


John:


If anyone has a moment, can you have a quick look and see if what I am
proposing seems acceptable.


Thinking about when a large file (Such as a war file) is being uploaded
to the server, we don't want Tomcat to reload it until after the file has
completed upload and is fully formed.

How does Tomcat test if a file has been updated?
I assume it uses File.lastModified() or something similar.

Does anyone know the detailed nuts and bolts of how that works?
Does last modified keep changing as the file is uploading or is it just
changed at the start of the upload?  Is the behavior the same or different
across platforms?

These questions should be taken into account when designing th

Re: is too quick to respond

Neil,

Thanks for your useful feedback. I am still feeling my way as you can
probably see from my earlier emails trying to setup a development
environment.

I did actually think of this but didn't put it in scope for a couple of reasons.

Firstly, the Tomcat documentation for readloadable quotes

"Set to true if you want Catalina to monitor classes in
/WEB-INF/classes/ and /WEB-INF/lib for changes, and automatically
reload the web application if a change is detected. This feature is
very useful during application development, but it requires
significant runtime overhead and is not recommended for use on
deployed production applications. That's why the default setting for
this attribute is false. You can use the Manager web application,
however, to trigger reloads of deployed applications on demand."

Therefore, I took it to mean that this flag was geared at development,
not production which is what I assume when you would deploy a .war
file. So Tomcat would be listening to specific changes in .classes and
.jar files that had just been compiled and these are normally small in
size. But then I suppose that a single .jar file may be so sized that
Tomcat could react while the file was still being written to the disk.

Secondly, I sort of assumed that since the feature was already in
place and handles changes to single files that this check for
completeness has already been implemented, but then as I can't get a
development environment to run, I don't have enough skills to drill
into the sources without it being interactive to help me explore and
learn.

However, it makes sense that your recommendation is implemented,
although I was imagining setting the delay to (say) 500ms to ensure
that whatever IDE had time to complete the copying of all the files as
that is a small price to pay for automatic refresh. Also by resetting
the timer after each event it would have to be quite a large upload
for Tomcat to start reacting.

Like you, I am not sure how to formally check that a file has
completed its copy to the destination. The most common suggestion I
hear is to try and change its name and then change it back again and
capture the exception which will be raised if the file is locked. I
wonder whether attempting to set an attribute (e.g.toggle read-only)
would have the same effect (i.e. only allow if file wasn't locked) and
be a little more elegant. I would have to try it.

I assume that Windows has a way of querying a file lock but not sure
(a) whether that is exposed via a Java API and (b) whether that would
apply to Unix as well (as I have only ever used Windows for
development).

> How does Tomcat test if a file has been updated?

Again, I don't know this yet (lack of IDE again), but I assumed that
it would be similar to the method I implemented in the attached source
code, i.e. Create a listener for events being triggered on file
changes to either /WEB-INF/classes/ and /WEB-INF/lib, as they are both
hard-coded file paths.

As an aside, I should have mentioned, for anyone interested in this
thread and in case not obvious from the source, but to see the sample
source code in action, you need to add, modify, rename or delete files
within the specified directory in a File Explorer.

John

On Sun, 20 Feb 2022 at 00:04, Neil Aggarwal  wrote:
>
> John:
>
> > If anyone has a moment, can you have a quick look and see if what I am
> > proposing seems acceptable.
>
> Thinking about when a large file (Such as a war file) is being uploaded
> to the server, we don't want Tomcat to reload it until after the file has
> completed upload and is fully formed.
>
> How does Tomcat test if a file has been updated?
> I assume it uses File.lastModified() or something similar.
>
> Does anyone know the detailed nuts and bolts of how that works?
> Does last modified keep changing as the file is uploading or is it just
> changed at the start of the upload?  Is the behavior the same or different
> across platforms?
>
> These questions should be taken into account when designing the solution.
>
> Thank you,
>   Neil
>
> --
> Neil Aggarwal, (972) 834-1565, http://www.propfinancing.com
> We offer 30 year loans on single family houses!
>
> -
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org
>

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

RE: is too quick to respond

2022-02-19 Thread Neil Aggarwal

John:

> If anyone has a moment, can you have a quick look and see if what I am
> proposing seems acceptable.

Thinking about when a large file (Such as a war file) is being uploaded
to the server, we don't want Tomcat to reload it until after the file has
completed upload and is fully formed.

How does Tomcat test if a file has been updated?
I assume it uses File.lastModified() or something similar.

Does anyone know the detailed nuts and bolts of how that works?
Does last modified keep changing as the file is uploading or is it just
changed at the start of the upload?  Is the behavior the same or different
across platforms?

These questions should be taken into account when designing the solution.

Thank you,
  Neil

--
Neil Aggarwal, (972) 834-1565, http://www.propfinancing.com
We offer 30 year loans on single family houses!

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: is too quick to respond

2022-02-19 Thread John Barrow

Hi,

While we continue the investigation into why I can't get Tomcat to run
within NetBeans, I thought that I would create a mockup of the
functionality that I would look to implement to protect the reloading
of the webapp until the files have been rebuilt. I have mimicked the
expected Tomcat listener for changes to a folder structure but of
course that will already exist within Tomcat and so will only be
looking to react to the existing events that are being triggered. If
anyone has a moment, can you have a quick look and see if what I am
proposing seems acceptable. The source is pretty short. To run this,
once compiled, you just need to create a blank folder (or reference an
existing one) and then amend the three constants, specifically to
reference your folder!

  static final String folderToWatch = "C:\\Community\\MonitorFolder";
  static final int timeToWatchSeconds = 30;
  // How long to listen for changes to files in the folder
  static final int timeForQuietMilliseconds = 5000;
  // How long to wait with no activity before triggering reload (list
directory files in my example)

The only minor observation is that if the period to listen for changes
expires prior to the period of no activity then the 'reload' task
(print the files in the directory) won't be triggered but as that
would (guessing now) only be when Tomcat was shut down then it would
be irrelevant anyway.

Thanks

John

On Fri, 18 Feb 2022 at 17:03, Christopher Schultz
 wrote:
>
> John,
>
> On 2/18/22 10:50, John Barrow wrote:
> >>> Would you be interested in looking at the existing algorithm to see if
> >>> it would be updated in this way?
> >
> >> WebappLoader.backgroundProcess() would be a good place to start.
> >
> > I am sort of interested in looking at this.
>
> Awesome!
>
> > I have only been playing
> > with Java for about 6 months, having been a Pascal developer most of
> > my life (40 odd years). Therefore, I think that I could at least ‘give
> > it a look’, if I can get Tomcat installed locally.
> >
> > However, I appreciate that I could be well out of my depth and there
> > are too many unknowns / barriers in order to be of any use.
> >
> > It is the installation process that I am having issues with. I think
> > it is down to lack of experience / knowledge / familiarity with
> > commercial strength application.
> >
> > I use Netbeans 12.6 (Apache loyalty!) on Windows 10 for all my Java
> > development using JDK 17.
> >
> > Unfortunately, I am only ‘sort-of’ familiar with Maven and have never
> > touched Ant so I am flying a bit blind here, but that shouldn’t matter
> > if it all works as I don't need to touch the build process.
> >
> > I downloaded the ‘main’ branch from GitHub
> > (https://github.com/apache/tomcat) and unzipped it into its own
> > folder, again with Ant (apache-ant-1.9.16).
> >
> > I have managed to get Ant to build it using instructions in
> > BUILDING.txt. While it says that Tomcat built successfully, scrolling
> > up, I got lots of warnings saying ‘can’t find a module etc so not sure
> > if doing something wrong. I can send the results as an attachment if
> > that would be useful.
> >
> > I can also get it to build in Netbeans (I first had to set  Project >
> > Properties > Java Sources to Source Level 17 which may be a bit of a
> > guess as Netbeans complained about default value of 1.7). The java
> > files in Netbeans don’t highlight any exceptions, just warnings.
> >
> > I appear to have issues with the other two folders
> > ‘C:\Community\Tomcat\tomcat-main\webapps\docs\appdev\sample\src\mypackage’
> > and ‘C:\Community\Tomcat\tomcat-main\test’ as they don’t seem to be
> > able to locate the dependencies (HelloWorld is -e.g. import
> > jakarta.servlet.http.HttpServlet & tests is -e.g. import
> > org.junit.Assert). In Maven, I would add a dependency to the pom.xml
> > file, but not sure what to do in Ant as I would expect the
> > configuration to already be set up.
>
> Try this:
>
> 1. Close the project in NetBeans
> 2. On the command-line, go to the directory where your Tomcat git
> checkout is
> 3. Run "ant ide-netbeans"
> 4. Open the project again in Netbeans
>
> See if that improves anything. That ant target configures Netbeans in a
> way that should be "helpful". I don't use Netbeans myself, but the
> equivalent for Eclipse (ide-eclipse) sets up the build-path and lots of
> other goodies.
>
> > I also ran the ‘ant test’, took about 90 minutes and failed a number
> > of tests (which I thought was surprising as I would expect sources in
> > the repository to pass all the tests. I can provide examples if useful
> > as it may indicate that I have done something wrong.
>
> You can speed-up the unit tests by throwing more CPU cores at it; by
> default it uses just one. You can do that by adding this to
> build.properties in the same directory as the build.xml file:
>
> test.threads=n
>
> That n should be something ... reasonable. I have a 6-core hyperthreaded
> CPU (=12 logic CPUs) and setting the number

Re: is too quick to respond

2022-02-19 Thread John Barrow

Hi,

I have re-run the Tomcat tests using 4 cores (all my machine has!) and
time came down to 35m 33s. Not bad.

I have written my own ‘grep’ to review the output directory and I
think that the number of failures has come down now using all the
cores so some may have been timing issues

The 5 that are reported are


C:\Community\Tomcat\tomcat-main\output\build\logs\TEST-jakarta.servlet.http.TestHttpServletDoHeadValidWrite513.NIO.txt
  Tests run: 1152, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 57.275 sec

C:\Community\Tomcat\tomcat-main\output\build\logs\TEST-org.apache.coyote.http2.TestStreamProcessor.NIO.txt
  Tests run: 9, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 4.282 sec

C:\Community\Tomcat\tomcat-main\output\build\logs\TEST-org.apache.coyote.http2.TestStreamProcessor.NIO2.txt
  Tests run: 9, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 4.008 sec

C:\Community\Tomcat\tomcat-main\output\build\logs\TEST-org.apache.jasper.runtime.TestJspRuntimeLibrary.NIO.txt
  Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 3.813 sec

C:\Community\Tomcat\tomcat-main\output\build\logs\TEST-org.apache.jasper.runtime.TestJspRuntimeLibrary.NIO2.txt
  Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 4.242 sec

I have attached the 5 txt files as listed above for review (if
needed).  The first one may still be a timing issue, but I am not in a
position to judge!

John

On Fri, 18 Feb 2022 at 18:16, Mark Thomas  wrote:
>
>
> On 18/02/2022 18:01, John Barrow wrote:
> >> Do you have a list of the tests which fail? Do you know how to get that 
> >> list?
> >
> > Yes, although I am on Windows, not UNIX and so I don't have access to
> > grep. I did a simple search within logs for 'FAILED' and it came up
> > with about 20 or so. The list of test results has been cleared now
> > (probably when I next built Tomcat) and so I will need to run them
> > again but will try using the threads at the same time and then send
> > you the results in a separate email.
>
> If you haven't cleared out the results from the logs directory you can do:
>
> ant test-status
>
> Mark
>
> -
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org
>
<>

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: is too quick to respond

2022-02-19 Thread John Barrow

HI,

As a completely 'left-field' thought and apologies if this next
statement is completely naive, if I started a clean Maven project,
copied across all the sources for Tomcat and then use NetBeans to add
dependencies as required, could that work? I am willing to give it a
try as I don't need to deploy my development version of Tomcat, I just
need to be able to run it within NetBeans.

Once I have finished any changes, I can export the amended sources to
the downloaded copy from github, re-apply the changes (they will
always be minor!) and then use the ant build script from the CMD to
deploy the finished version so I can test it in place of the release
version I am currently using for my other web development.

Which brings me back to a question raised earlier, confirmation of
which source file it is that launches Tomcat to display the CMD like
monitor as there are 15 "main" entry points within the Tomcat sources.
I am assuming that it is the one in Tomcat.java, which interestingly
has the comment

/**
 * Main executable method for use with a Maven packager.
 * @param args the command line arguments
 * @throws Exception if an error occurs
 */
public static void main(String[] args) throws Exception {

indicating that perhaps Tomcat has been built using Maven!

John

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

RE: is too quick to respond

Hi,

As mentioned in my last email, I have built a simple ‘Hello World’ Ant
project in NetBeans, referencing an external library (GSON in my case)
and the project properties looks quite different for the generated Ant
project against the one that was created for Tomcat. The attached
screenshot illustrates the differences (ignore the other unopened
projects!)

The 'Hello world' Ant project (source file attached) was created using
the tools built into NetBeans so I

- Created Java Application with Ant
  - Project name:  HelloWorldAnt
- Wrote the attached class
  - NetBeans flags up that com.google.gson library (jar) not found
-  Projects > HelloWorldAnt > Libraries
   - Locate gson-2.9.0.jar and open it
   - Library is added to the libaries folder within the project (as
per screenshot)
- Run project - compiles successfully and exports myLog to JSON in
c:\Work\UserLogins.json (reproduced below).
- I can also add breakpoints and step through the code during execution.

The file structure for this simple Ant project looks a lot more
complex than the one I am used to for Maven (see below). I am
struggling to compare the two structures (HelloWorldAnt & Tomcat) as
they look so different.

While, I have not included any of the files in the directory structure
listed below, happy to or upload the folder to DropBox. However, I
think the exercise was more to determine that the Tomcat folder
structure isn't as NetBeans would expect it for development to take
place.

Going to re-run the Tomcat tests and see if I can get a list of the
failures in a suitable format.

John

Directory / File structure for created HelloWorldAnt project

C:\Development\Work\HelloWorldAnt\build
C:\Development\Work\HelloWorldAnt\build.xml
C:\Development\Work\HelloWorldAnt\manifest.mf
C:\Development\Work\HelloWorldAnt\nbproject
C:\Development\Work\HelloWorldAnt\src
C:\Development\Work\HelloWorldAnt\test
C:\Development\Work\HelloWorldAnt\build\classes
C:\Development\Work\HelloWorldAnt\build\classes\.netbeans_automatic_build
C:\Development\Work\HelloWorldAnt\build\classes\.netbeans_update_resources
C:\Development\Work\HelloWorldAnt\build\classes\helloworldant
C:\Development\Work\HelloWorldAnt\build\classes\helloworldant\HelloWorldAnt.class
C:\Development\Work\HelloWorldAnt\build\classes\helloworldant\HelloWorldAnt.rs
C:\Development\Work\HelloWorldAnt\build\classes\helloworldant\TransactionLog$LocalDateSerializer.class
C:\Development\Work\HelloWorldAnt\build\classes\helloworldant\TransactionLog$Transaction.class
C:\Development\Work\HelloWorldAnt\build\classes\helloworldant\TransactionLog.class
C:\Development\Work\HelloWorldAnt\nbproject\build-impl.xml
C:\Development\Work\HelloWorldAnt\nbproject\genfiles.properties
C:\Development\Work\HelloWorldAnt\nbproject\private
C:\Development\Work\HelloWorldAnt\nbproject\project.properties
C:\Development\Work\HelloWorldAnt\nbproject\project.xml
C:\Development\Work\HelloWorldAnt\nbproject\private\private.properties
C:\Development\Work\HelloWorldAnt\nbproject\private\private.xml
C:\Development\Work\HelloWorldAnt\src\helloworldant
C:\Development\Work\HelloWorldAnt\src\helloworldant\HelloWorldAnt.java

gson (taken from my Maven .m2\repository) is referenced in
project.properties in the following snippet

excludes=
file.reference.gson-2.9.0.jar=C:\\Users\\John\\.m2\\repository\\com\\google\\code\\gson\\gson\\2.9.0\\gson-2.9.0.jar
includes=**
jar.compress=false
javac.classpath=\
${file.reference.gson-2.9.0.jar}

c:\Work\UserLogins.json file
-
{
  "KP": {
"reference": "KP",
"name": "Kevin Peters",
"loggedIn": "2022-02-18"
  },
  "MS": {
"reference": "MS",
"name": "Mike Smith",
"loggedIn": "2022-02-18"
  },
  "JB": {
    "reference": "JB",
"name": "John Barrow",
"loggedIn": "2022-02-18"
  }
}

From: Mark Thomas
Sent: 18 February 2022 18:16
To: users@tomcat.apache.org
Subject: Re:  is too quick to respond





On 18/02/2022 18:01, John Barrow wrote:

>> Do you have a list of the tests which fail? Do you know how to get that list?

>

> Yes, although I am on Windows, not UNIX and so I don't have access to

> grep. I did a simple search within logs for 'FAILED' and it came up

> with about 20 or so. The list of test results has been cleared now

> (probably when I next built Tomcat) and so I will need to run them

> again but will try using the threads at the same time and then send

> you the results in a separate email.



If you haven't cleared out the results from the logs directory you can do:



ant test-status



Mark



-

To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org

For additional commands, e-mail: users-h...@tomcat.apache.org


-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: is too quick to respond

Mark,I checked the logs folder and it was empty and so I am assuming that Any 
is clearing it each time I build Tomcat.John
 Original message From: Mark Thomas  Date: 
18/02/2022  18:16  (GMT+00:00) To: users@tomcat.apache.org Subject: Re:  is too 
quick to respond On 18/02/2022 18:01, John Barrow wrote:>> Do you have a list 
of the tests which fail? Do you know how to get that list?> > Yes, although I 
am on Windows, not UNIX and so I don't have access to> grep. I did a simple 
search within logs for 'FAILED' and it came up> with about 20 or so. The list 
of test results has been cleared now> (probably when I next built Tomcat) and 
so I will need to run them> again but will try using the threads at the same 
time and then send> you the results in a separate email.If you haven't cleared 
out the results from the logs directory you can do:ant 
test-statusMark-To
 unsubscribe, e-mail: users-unsubscribe@tomcat.apache.orgFor additional 
commands, e-mail: users-h...@tomcat.apache.org

Re: is too quick to respond

2022-02-18 Thread Mark Thomas




On 18/02/2022 18:01, John Barrow wrote:

Do you have a list of the tests which fail? Do you know how to get that list?


Yes, although I am on Windows, not UNIX and so I don't have access to
grep. I did a simple search within logs for 'FAILED' and it came up
with about 20 or so. The list of test results has been cleared now
(probably when I next built Tomcat) and so I will need to run them
again but will try using the threads at the same time and then send
you the results in a separate email.


If you haven't cleared out the results from the logs directory you can do:

ant test-status

Mark

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: is too quick to respond

Hi,

> 3. Run "ant ide-netbeans"

Apologies, I should have made that point clear, I have already run
that otherwise Netbeans wouldn't recognise the project at all - i.e. I
couldn't open it. In that respect, it was helpful, as in I could now
compile and navigate the source files.

> The "deploy" target is probably the one you want, or maybe "compile".

Perhaps I am expecting a different outcome. The 'clean' + 'deploy' is
what happens when I 'Clean and build main project' in Netbeans speak
(checking the Projects properties dialog confirms this, see attached
screenshot). What I was expecting 'Run project' to do, was locate the
source with the "public static void main(final String[] args)" method
in it and run the application, hopefully bringing up the Netbeans
console window in a CMD like window (similar to running the .bat file
%CATALINA_HOME%\bin\startup.bat).

In Maven, this is what happens automatically when I 'Run project', I
don't believe that I ever configured anything to execute a project
from within Netbeans. The only slight variance is if NetBeans finds
multiple main() methods, in which case you get a popup allowing you to
select the appropriate starting point.

So, perhaps, an alternative question would be, how do you go about
making an amendment in Eclipse within the Tomcat sources (e.g. add a
System.out.println() statement to log a field to the Eclipse 'console
window' and then run Tomcat to see the effect. Also how do you run the
Eclipse debugger on the Tomcat application?

> Do you have a list of the tests which fail? Do you know how to get that list?

Yes, although I am on Windows, not UNIX and so I don't have access to
grep. I did a simple search within logs for 'FAILED' and it came up
with about 20 or so. The list of test results has been cleared now
(probably when I next built Tomcat) and so I will need to run them
again but will try using the threads at the same time and then send
you the results in a separate email.

> If you can run the unit tests, then they are compiling and you should be able 
> to build a binary release (ant deploy) without a problem.

I ran the tests from a CMD prompt rather than in NetBeans. NetBeans is
telling me that it can't find the resources. A quick Google seems to
imply that there should be a project.xml as well as a build.xml but
then as I don't know Ant, I am not sure what to expect. I will attempt
to create a simple 'Hello World' project in NetBeans using Ant to see
what I get so I have a simple project to refer to as a base when
trying to relate Tomcat. Without this, I couldn't review or extend the
tests that form part of Tomcat as any attempt to run them within
Netbeans tells me that the packages don't exist.

Is there anyone else in the group that is successfully developing
Tomcat using Netbeans?

On Fri, 18 Feb 2022 at 17:03, Christopher Schultz
 wrote:
>
> John,
>
> On 2/18/22 10:50, John Barrow wrote:
> >>> Would you be interested in looking at the existing algorithm to see if
> >>> it would be updated in this way?
> >
> >> WebappLoader.backgroundProcess() would be a good place to start.
> >
> > I am sort of interested in looking at this.
>
> Awesome!
>
> > I have only been playing
> > with Java for about 6 months, having been a Pascal developer most of
> > my life (40 odd years). Therefore, I think that I could at least ‘give
> > it a look’, if I can get Tomcat installed locally.
> >
> > However, I appreciate that I could be well out of my depth and there
> > are too many unknowns / barriers in order to be of any use.
> >
> > It is the installation process that I am having issues with. I think
> > it is down to lack of experience / knowledge / familiarity with
> > commercial strength application.
> >
> > I use Netbeans 12.6 (Apache loyalty!) on Windows 10 for all my Java
> > development using JDK 17.
> >
> > Unfortunately, I am only ‘sort-of’ familiar with Maven and have never
> > touched Ant so I am flying a bit blind here, but that shouldn’t matter
> > if it all works as I don't need to touch the build process.
> >
> > I downloaded the ‘main’ branch from GitHub
> > (https://github.com/apache/tomcat) and unzipped it into its own
> > folder, again with Ant (apache-ant-1.9.16).
> >
> > I have managed to get Ant to build it using instructions in
> > BUILDING.txt. While it says that Tomcat built successfully, scrolling
> > up, I got lots of warnings saying ‘can’t find a module etc so not sure
> > if doing something wrong. I can send the results as an attachment if
> > that would be useful.
> >
> > I can also get it to build in Netbeans (I first had to set  Project >
> > Properties > Java Sources to Source Level 17 which may be a bit of a
> > guess as Netbeans complained about default value of 1.7). The java
> > files in Netbeans don’t highlight any exceptions, just warnings.
> >
> > I appear to have issues with the other two folders
> > ‘C:\Community\Tomcat\tomcat-main\webapps\docs\appdev\sample\src\mypackage’
> > and ‘C:\Community\Tomcat\tomcat-main\t

Re: is too quick to respond

2022-02-18 Thread Christopher Schultz


John,

On 2/18/22 10:50, John Barrow wrote:

Would you be interested in looking at the existing algorithm to see if
it would be updated in this way?



WebappLoader.backgroundProcess() would be a good place to start.


I am sort of interested in looking at this.


Awesome!


I have only been playing
with Java for about 6 months, having been a Pascal developer most of
my life (40 odd years). Therefore, I think that I could at least ‘give
it a look’, if I can get Tomcat installed locally.

However, I appreciate that I could be well out of my depth and there
are too many unknowns / barriers in order to be of any use.

It is the installation process that I am having issues with. I think
it is down to lack of experience / knowledge / familiarity with
commercial strength application.

I use Netbeans 12.6 (Apache loyalty!) on Windows 10 for all my Java
development using JDK 17.

Unfortunately, I am only ‘sort-of’ familiar with Maven and have never
touched Ant so I am flying a bit blind here, but that shouldn’t matter
if it all works as I don't need to touch the build process.

I downloaded the ‘main’ branch from GitHub
(https://github.com/apache/tomcat) and unzipped it into its own
folder, again with Ant (apache-ant-1.9.16).

I have managed to get Ant to build it using instructions in
BUILDING.txt. While it says that Tomcat built successfully, scrolling
up, I got lots of warnings saying ‘can’t find a module etc so not sure
if doing something wrong. I can send the results as an attachment if
that would be useful.

I can also get it to build in Netbeans (I first had to set  Project >
Properties > Java Sources to Source Level 17 which may be a bit of a
guess as Netbeans complained about default value of 1.7). The java
files in Netbeans don’t highlight any exceptions, just warnings.

I appear to have issues with the other two folders
‘C:\Community\Tomcat\tomcat-main\webapps\docs\appdev\sample\src\mypackage’
and ‘C:\Community\Tomcat\tomcat-main\test’ as they don’t seem to be
able to locate the dependencies (HelloWorld is -e.g. import
jakarta.servlet.http.HttpServlet & tests is -e.g. import
org.junit.Assert). In Maven, I would add a dependency to the pom.xml
file, but not sure what to do in Ant as I would expect the
configuration to already be set up.


Try this:

1. Close the project in NetBeans
2. On the command-line, go to the directory where your Tomcat git 
checkout is

3. Run "ant ide-netbeans"
4. Open the project again in Netbeans

See if that improves anything. That ant target configures Netbeans in a 
way that should be "helpful". I don't use Netbeans myself, but the 
equivalent for Eclipse (ide-eclipse) sets up the build-path and lots of 
other goodies.



I also ran the ‘ant test’, took about 90 minutes and failed a number
of tests (which I thought was surprising as I would expect sources in
the repository to pass all the tests. I can provide examples if useful
as it may indicate that I have done something wrong.


You can speed-up the unit tests by throwing more CPU cores at it; by 
default it uses just one. You can do that by adding this to 
build.properties in the same directory as the build.xml file:


test.threads=n

That n should be something ... reasonable. I have a 6-core hyperthreaded 
CPU (=12 logic CPUs) and setting the number of threads to 4 
significantly reduces the time to run the whole test-suite. I wouldn't 
set this to anything higher than the number of physical cores you 
actually have unless you don't want to use your computer for anything 
else during that test.


As for test failures, some of them are known to fail in certain 
environments. Do you have a list of the tests which fail? Do you know 
how to get that list?


I have an automated build-test-etc script[1] that performs this little 
gem after the unit tests have run:


grep "\(Failures\|Errors\): [^0]" 
"${BASE_SOURCE_DIR}/output/build/logs/"TEST*.txt


That should tell you which tests have failed. Feel free to post them 
here and we'll let you know if any of them are concerning.



I read through the ‘RUNNING.txt’ file, but that assumes a binary to
start, rather than following on from BUILDING Tomcat and so I couldn’t
relate that to what I was trying to achieve.


If you can run the unit tests, then they are compiling and you should be 
able to build a binary release (ant deploy) without a problem.



However, if I launch Tomcat from within Netbeans (F6), I get a dialog
which I don’t understand - probably due to my lack of familiarity with
Ant. Again, I can include a screenshot if that would help.

[Run – Tomcat 8.0]
No build target is associated with Run action.
The free-form project requires a target
Representing the Run action to invoke in
Your Ant script. Select the target to invoke.
[Dropdown with lots of tasks v]


This might be solved by running "ant ide-netbeans". Or maybe not. You 
may have to pick a target. The "deploy" target is probably the one you 
want, or maybe "compile". Running ant from within an IDE only make

RE: is too quick to respond