RE: [cfaussie] CF Temp files
Glad to help, and thanks for the update. I will be curious to know if the files are in fact the result of image processing. It's not necessarily definitely another source, as some cfimage processing involves no creation of temp files. BTW, my last point about looking at the files should help here, to see just what kind of files these are. At least, if they don't have the look of some image filetype, that may help direct your additional investigation. But yes, as for possible error handling hiding the real root cause, note that if there's some global or application-level handling, it would be pretty easy to remove it, even if just temporarily. If you observe how often these files are created, and find they are created really often, then it wouldn't take long to leave the error handling off to find out what's happening. (And note that you don't need to be the user running the page that gets the error, to see the error. It would be logged in the application.log.) No offense intended if you already were aware of the above. I never presume what even experienced people may know. Some never dabble much in error handling, for instance. Also, when I offer answers on lists, I am as often answering for other readers, who may benefit from a little more context. It makes my notes a tad longer (and that frustrates some), but I hope the benefit outweighs the risk. :-) Indeed, thanks for your kind regards on my having surmised things. /charlie From: cfaussie@googlegroups.com [mailto:cfaussie@googlegroups.com] On Behalf Of Brett Payne-Rhodes Sent: Tuesday, December 20, 2011 10:20 PM To: cfaussie@googlegroups.com Subject: Re: [cfaussie] CF Temp files Thanks Charlie, As usual you have surmissed much of what is going on quite accurately. There is no danger of this being a DoS attack, the system in question is effectively closed to the outside world. And the system does have a significant amount of activity in terms of files being uploaded. In addition, all images are 'served' via a CF script using CFIMAGE that will most definitely be another source of these temporary files. And given the volume of work being carried out it does not surprise me that so many temporary files are being created. In terms of the log files, I don't see a lot of activity there, especially with regard to file uploads and image serving but then they are wrapped up in cftry/cfcatch so we are not going to know about it via error logging. Which could suggest the issue is hidden. Maybe the 'delete' permission has been removed without us knowing about it? That would mean the temporary files could not be removed and the try/catch would be hiding it from us seeing it as a problem until the disk fills up. Thanks again Charlie, most hlepful! Cheers, Brett B) -- You received this message because you are subscribed to the Google Groups cfaussie group. To post to this group, send email to cfaussie@googlegroups.com. To unsubscribe from this group, send email to cfaussie+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/cfaussie?hl=en.
[cfaussie] CF Temp files
Hi, A couple of days ago one of our production servers ran out of disk space. I assumed it was the application not cleaning up temporary files but after the initial panic we discovered that the JRun4\servers\cfusion\SERVER-INF\temp\cfusion-war-tmp directory was FULL of files named like neotmp999...999.tmp. I have a good idea where these files are being generated in the application but they are being generated by CF and we have no direct visibility of them. And I assume that up until a few days ago that CF itself was cleaning them up - otherwise the application would have failed months ago. When the disk full error happened there were over 6.000 of these files consuming over 15GB of disk space and all of them were time-stamped in the previous three days and the other server was close to being out of disk space. Today both servers have over 2,000 of these files consuming over 6.5GB of disk so we will be running a manual cleanup very soon. I've searched the usual places but there is nothing I can find that resembles this issue. There is some talk of sandbox security on that directory causing some access issues but nothing about temporary files not being cleaned up. Has anybody seen this before? Or have an idea what might be causing it or how to fix it? Many thanks and Happy Christmas to all! Brett B) -- You received this message because you are subscribed to the Google Groups cfaussie group. To post to this group, send email to cfaussie@googlegroups.com. To unsubscribe from this group, send email to cfaussie+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/cfaussie?hl=en.
RE: [cfaussie] CF Temp files
Did you install anything recently? Like the new CF hotfix perhaps, something must have changed. Regards Dale Fraser http://dale.fraser.id.au http://dale.fraser.id.au http://cfmldocs.com http://cfmldocs.com http://learncf.com http://learncf.com http://flexcf.com http://flexcf.com From: cfaussie@googlegroups.com [mailto:cfaussie@googlegroups.com] On Behalf Of Brett Payne-Rhodes Sent: Wednesday, 21 December 2011 12:23 PM To: cfaussie@googlegroups.com Subject: [cfaussie] CF Temp files Hi, A couple of days ago one of our production servers ran out of disk space. I assumed it was the application not cleaning up temporary files but after the initial panic we discovered that the JRun4\servers\cfusion\SERVER-INF\temp\cfusion-war-tmp directory was FULL of files named like neotmp999...999.tmp. I have a good idea where these files are being generated in the application but they are being generated by CF and we have no direct visibility of them. And I assume that up until a few days ago that CF itself was cleaning them up - otherwise the application would have failed months ago. When the disk full error happened there were over 6.000 of these files consuming over 15GB of disk space and all of them were time-stamped in the previous three days and the other server was close to being out of disk space. Today both servers have over 2,000 of these files consuming over 6.5GB of disk so we will be running a manual cleanup very soon. I've searched the usual places but there is nothing I can find that resembles this issue. There is some talk of sandbox security on that directory causing some access issues but nothing about temporary files not being cleaned up. Has anybody seen this before? Or have an idea what might be causing it or how to fix it? Many thanks and Happy Christmas to all! Brett B) -- You received this message because you are subscribed to the Google Groups cfaussie group. To post to this group, send email to cfaussie@googlegroups.com. To unsubscribe from this group, send email to cfaussie+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/cfaussie?hl=en. -- You received this message because you are subscribed to the Google Groups cfaussie group. To post to this group, send email to cfaussie@googlegroups.com. To unsubscribe from this group, send email to cfaussie+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/cfaussie?hl=en.
Re: [cfaussie] CF Temp files
You're right, it feels like something must have changed and I'm trying to find out from the guys who look after the infrastructure but I would be surprised if there had been a change that we weren't told about. B) On Wed, Dec 21, 2011 at 9:36 AM, Dale Fraser d...@fraser.id.au wrote: Did you install anything recently? ** ** Like the new CF hotfix perhaps, something must have changed. ** ** Regards Dale Fraser ** ** http://dale.fraser.id.au http://cfmldocs.com http://learncf.com http://flexcf.com ** ** *From:* cfaussie@googlegroups.com [mailto:cfaussie@googlegroups.com] *On Behalf Of *Brett Payne-Rhodes *Sent:* Wednesday, 21 December 2011 12:23 PM *To:* cfaussie@googlegroups.com *Subject:* [cfaussie] CF Temp files ** ** Hi, A couple of days ago one of our production servers ran out of disk space. I assumed it was the application not cleaning up temporary files but after the initial panic we discovered that the JRun4\servers\cfusion\SERVER-INF\temp\cfusion-war-tmp directory was FULL of files named like neotmp999...999.tmp. I have a good idea where these files are being generated in the application but they are being generated by CF and we have no direct visibility of them. And I assume that up until a few days ago that CF itself was cleaning them up - otherwise the application would have failed months ago. When the disk full error happened there were over 6.000 of these files consuming over 15GB of disk space and all of them were time-stamped in the previous three days and the other server was close to being out of disk space. Today both servers have over 2,000 of these files consuming over 6.5GB of disk so we will be running a manual cleanup very soon. I've searched the usual places but there is nothing I can find that resembles this issue. There is some talk of sandbox security on that directory causing some access issues but nothing about temporary files not being cleaned up. Has anybody seen this before? Or have an idea what might be causing it or how to fix it? Many thanks and Happy Christmas to all! Brett B) -- You received this message because you are subscribed to the Google Groups cfaussie group. To post to this group, send email to cfaussie@googlegroups.com. To unsubscribe from this group, send email to cfaussie+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/cfaussie?hl=en. -- You received this message because you are subscribed to the Google Groups cfaussie group. To post to this group, send email to cfaussie@googlegroups.com. To unsubscribe from this group, send email to cfaussie+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/cfaussie?hl=en. -- You received this message because you are subscribed to the Google Groups cfaussie group. To post to this group, send email to cfaussie@googlegroups.com. To unsubscribe from this group, send email to cfaussie+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/cfaussie?hl=en.
RE: [cfaussie] CF Temp files
Here's an entirely different possibility, Brett. I'm pretty sure you'll find those files are the result of file uploads to your server (such as is created with an HTML page doing an input type=file that posts to a CFM page). They could be legit file uploads (perhaps with an error causing them to stick around-more on that in a moment), or they could be a form of a denial of service attack. Here are some thoughts to consider. When one does a file upload to a CF server, the file is first stored in CF's temp directory-that location you are observing, with a filename in the form that you are observing. You may say, we don't do any CFFILE ACTION='upload's, but that's not really the key factor. Though many assume that that tag somehow received the upload, it does not. If any HTML page posts a file upload to a CFM page, that upload will be received and put into CF's temp directory. The role of that CFFILE action=upload is merely to move (and typically rename) that file into a desired destination folder. So technically, *any* CFM page could be targeted to receive a file upload. And it doesn't really need to be an HTML page with a user clicking a file input button to upload a file. A hacker (or a CF developer using CFHTTP) could post a request to a CFM page sending along file content in the form post data. To figure out what requests are doing this, you could analyze your web server logs around the time that these files are created to look for URLs that are doing file uploads (look for request-method values = post). You can also configure your web server logs to track the incoming file size (cs-bytes, for client-to-server bytes), and look for large values. As for why the files stick around for you, I suppose that if the page being called were to get an error, it may be possible that CF would not cleanup after itself. For that, you could look in your CF server's application.log to see if there are errors being tracked there around the time these files are created. (Of course, you may also be handling the errors in your code, with try/catch, cferror, or site-wide error handling, which would then prevent an error showing up in the CF logs unless you rethrow them.) Finally, are you able to look at any of the files, to see what the content is? That too may help you start to understand what they're about. Let us know if this is helpful. /charlie arehart char...@carehart.org Providing fast, remote, on-demand troubleshooting services for CF (and CFBuilder) More at http://www.carehart.org/consulting From: cfaussie@googlegroups.com [mailto:cfaussie@googlegroups.com] On Behalf Of Brett Payne-Rhodes Sent: Tuesday, December 20, 2011 8:23 PM To: cfaussie@googlegroups.com Subject: [cfaussie] CF Temp files Hi, A couple of days ago one of our production servers ran out of disk space. I assumed it was the application not cleaning up temporary files but after the initial panic we discovered that the JRun4\servers\cfusion\SERVER-INF\temp\cfusion-war-tmp directory was FULL of files named like neotmp999...999.tmp. I have a good idea where these files are being generated in the application but they are being generated by CF and we have no direct visibility of them. And I assume that up until a few days ago that CF itself was cleaning them up - otherwise the application would have failed months ago. When the disk full error happened there were over 6.000 of these files consuming over 15GB of disk space and all of them were time-stamped in the previous three days and the other server was close to being out of disk space. Today both servers have over 2,000 of these files consuming over 6.5GB of disk so we will be running a manual cleanup very soon. I've searched the usual places but there is nothing I can find that resembles this issue. There is some talk of sandbox security on that directory causing some access issues but nothing about temporary files not being cleaned up. Has anybody seen this before? Or have an idea what might be causing it or how to fix it? Many thanks and Happy Christmas to all! Brett B) -- You received this message because you are subscribed to the Google Groups cfaussie group. To post to this group, send email to cfaussie@googlegroups.com. To unsubscribe from this group, send email to cfaussie+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/cfaussie?hl=en. -- You received this message because you are subscribed to the Google Groups cfaussie group. To post to this group, send email to cfaussie@googlegroups.com. To unsubscribe from this group, send email to cfaussie+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/cfaussie?hl=en.
Re: [cfaussie] CF Temp files
Thanks Charlie, As usual you have surmissed much of what is going on quite accurately. There is no danger of this being a DoS attack, the system in question is effectively closed to the outside world. And the system does have a significant amount of activity in terms of files being uploaded. In addition, all images are 'served' via a CF script using CFIMAGE that will most definitely be another source of these temporary files. And given the volume of work being carried out it does not surprise me that so many temporary files are being created. In terms of the log files, I don't see a lot of activity there, especially with regard to file uploads and image serving but then they are wrapped up in cftry/cfcatch so we are not going to know about it via error logging. Which could suggest the issue is hidden. Maybe the 'delete' permission has been removed without us knowing about it? That would mean the temporary files could not be removed and the try/catch would be hiding it from us seeing it as a problem until the disk fills up. Thanks again Charlie, most hlepful! Cheers, Brett B) On Wed, Dec 21, 2011 at 10:52 AM, charlie arehart charlie_li...@carehart.org wrote: Here’s an entirely different possibility, Brett. I’m pretty sure you’ll find those files are the result of file uploads to your server (such as is created with an HTML page doing an input type=”file” that posts to a CFM page). They could be legit file uploads (perhaps with an error causing them to stick around—more on that in a moment), or they could be a form of a denial of service attack. Here are some thoughts to consider. When one does a file upload to a CF server, the file is first stored in CF’s “temp” directory—that location you are observing, with a filename in the form that you are observing. You may say, “we don’t do any CFFILE ACTION=’upload’s”, but that’s not really the key factor. Though many assume that that tag somehow “received the upload”, it does not. If any HTML page posts a file upload to a CFM page, that upload will be received and put into CF’s temp directory. The role of that CFFILE action=”upload” is merely to move (and typically rename) that file into a desired destination folder. So technically, **any** CFM page could be targeted to receive a file upload. And it doesn’t really need to be an HTML page with a user clicking a file input button to upload a file. A hacker (or a CF developer using CFHTTP) could post a request to a CFM page sending along file content in the form post data. To figure out what requests are doing this, you could analyze your web server logs around the time that these files are created to look for URLs that are doing file uploads (look for request-method values = “post”). You can also configure your web server logs to track the incoming file size (cs-bytes, for client-to-server bytes), and look for large values. As for why the files stick around for you, I suppose that if the page being called were to get an error, it may be possible that CF would not cleanup after itself. For that, you could look in your CF server’s application.log to see if there are errors being tracked there around the time these files are created. (Of course, you may also be handling the errors in your code, with try/catch, cferror, or site-wide error handling, which would then prevent an error showing up in the CF logs unless you rethrow them.) Finally, are you able to look at any of the files, to see what the content is? That too may help you start to understand what they’re about. Let us know if this is helpful. /charlie arehart char...@carehart.org Providing fast, remote, on-demand troubleshooting services for CF (and CFBuilder) More at http://www.carehart.org/consulting ** ** *From:* cfaussie@googlegroups.com [mailto:cfaussie@googlegroups.com] *On Behalf Of *Brett Payne-Rhodes *Sent:* Tuesday, December 20, 2011 8:23 PM *To:* cfaussie@googlegroups.com *Subject:* [cfaussie] CF Temp files ** ** Hi, A couple of days ago one of our production servers ran out of disk space. I assumed it was the application not cleaning up temporary files but after the initial panic we discovered that the JRun4\servers\cfusion\SERVER-INF\temp\cfusion-war-tmp directory was FULL of files named like neotmp999...999.tmp. I have a good idea where these files are being generated in the application but they are being generated by CF and we have no direct visibility of them. And I assume that up until a few days ago that CF itself was cleaning them up - otherwise the application would have failed months ago. When the disk full error happened there were over 6.000 of these files consuming over 15GB of disk space and all of them were time-stamped in the previous three days and the other server was close to being out of disk space. Today both servers have over 2,000 of these files consuming over 6.5GB of disk so we will be running a manual
