Re: [cfaussie] CF Temp files

[Brett Payne-Rhodes]

Thanks Charlie,

As usual you have surmissed much of what is going on quite accurately.

There is no danger of this being a DoS attack, the system in question is
effectively closed to the outside world. And the system does have a
significant amount of activity in terms of files being uploaded. In
addition, all images are 'served' via a CF script using CFIMAGE that will
most definitely be another source of these temporary files. And given the
volume of work being carried out it does not surprise me that so many
temporary files are being created.

In terms of the log files, I don't see a lot of activity there, especially
with regard to file uploads and image serving but then they are wrapped up
in cftry/cfcatch so we are not going to know about it via error logging.
Which could suggest the issue is hidden. Maybe the 'delete' permission has
been removed without us knowing about it? That would mean the temporary
files could not be removed and the try/catch would be hiding it from us
seeing it as a problem until the disk fills up.

 Thanks again Charlie, most hlepful!

Cheers,

Brett
B)


On Wed, Dec 21, 2011 at 10:52 AM, charlie arehart <
charlie_li...@carehart.org> wrote:

> Here’s an entirely different possibility, Brett. I’m pretty sure you’ll
> find those files are the result of file uploads to your server (such as is
> created with an HTML page doing an input type=”file” that posts to a CFM
> page). They could be legit file uploads (perhaps with an error causing them
> to stick around—more on that in a moment), or they could be a form of a
> denial of service attack. Here are some thoughts to consider.
>
> When one does a file upload to a CF server, the file is first stored in
> CF’s “temp” directory—that location you are observing, with a filename in
> the form that you are observing.
>
> You may say, “we don’t do any CFFILE ACTION=’upload’s”, but that’s not
> really the key factor. Though many assume that that tag somehow “received
> the upload”, it does not. If any HTML page posts a file upload to a CFM
> page, that upload will be received and put into CF’s temp directory. The
> role of that CFFILE action=”upload” is merely to move (and typically
> rename) that file into a desired destination folder.
>
> So technically, **any** CFM page could be targeted to receive a file
> upload. And it doesn’t really need to be an HTML page with a user clicking
> a file input button to upload a file. A hacker (or a CF developer using
> CFHTTP) could post a request to a CFM page sending along file content in
> the form post data.
>
> To figure out what requests are doing this, you could analyze your web
> server logs around the time that these files are created  to look for URLs
> that are doing file uploads (look for request-method values = “post”). You
> can also configure your web server logs to track the incoming file size
> (cs-bytes, for client-to-server bytes), and look for large values.
>
> As for why the files stick around for you, I suppose that if the page
> being called were to get an error, it may be possible that CF would not
> cleanup after itself.  For that, you could look in your CF server’s
> application.log to see if there are errors being tracked there around the
> time these files are created. (Of course, you may also be handling the
> errors in your code, with try/catch, cferror, or site-wide error handling,
> which would then prevent an error showing up in the CF logs unless you
> rethrow them.)
>
> Finally, are you able to look at any of the files, to see what the content
> is? That too may help you start to understand what they’re about.
>
> Let us know if this is helpful.
>
> 
>
> /charlie arehart
>
> char...@carehart.org
>
> Providing fast, remote, on-demand troubleshooting services for CF (and
> CFBuilder)
>
> More at http://www.carehart.org/consulting
>
> ** **
>
> *From:* cfaussie@googlegroups.com [mailto:cfaussie@googlegroups.com] *On
> Behalf Of *Brett Payne-Rhodes
> *Sent:* Tuesday, December 20, 2011 8:23 PM
>
> *To:* cfaussie@googlegroups.com
> *Subject:* [cfaussie] CF Temp files
>
> ** **
>
> Hi,
>
>
> A couple of days ago one of our production servers ran out of disk space.
> I assumed it was the application not cleaning up temporary files but after
> the initial panic we discovered that the
> JRun4\servers\cfusion\SERVER-INF\temp\cfusion-war-tmp directory was FULL of
> files named like "neotmp999...999.tmp". I have a good idea where these
> files are being generated in the application but they are being generated
> by CF and we have no direct visibility of them. And I assume that up until
> a few days ago that CF itself was cleaning them up - otherwise the
> application would have failed months ago. When the disk full error happened
> there were over 6.000 of these files consuming over 15GB of disk space and
> all of them were time-stamped in the previous three days and the other
> server was close to being out of disk space. Today both servers have over
> 

RE: [cfaussie] CF Temp files

[charlie arehart]

Here's an entirely different possibility, Brett. I'm pretty sure you'll find
those files are the result of file uploads to your server (such as is
created with an HTML page doing an input type="file" that posts to a CFM
page). They could be legit file uploads (perhaps with an error causing them
to stick around-more on that in a moment), or they could be a form of a
denial of service attack. Here are some thoughts to consider.

When one does a file upload to a CF server, the file is first stored in CF's
"temp" directory-that location you are observing, with a filename in the
form that you are observing. 

You may say, "we don't do any CFFILE ACTION='upload's", but that's not
really the key factor. Though many assume that that tag somehow "received
the upload", it does not. If any HTML page posts a file upload to a CFM
page, that upload will be received and put into CF's temp directory. The
role of that CFFILE action="upload" is merely to move (and typically rename)
that file into a desired destination folder. 

So technically, *any* CFM page could be targeted to receive a file upload.
And it doesn't really need to be an HTML page with a user clicking a file
input button to upload a file. A hacker (or a CF developer using CFHTTP)
could post a request to a CFM page sending along file content in the form
post data. 

To figure out what requests are doing this, you could analyze your web
server logs around the time that these files are created  to look for URLs
that are doing file uploads (look for request-method values = "post"). You
can also configure your web server logs to track the incoming file size
(cs-bytes, for client-to-server bytes), and look for large values.

As for why the files stick around for you, I suppose that if the page being
called were to get an error, it may be possible that CF would not cleanup
after itself.  For that, you could look in your CF server's application.log
to see if there are errors being tracked there around the time these files
are created. (Of course, you may also be handling the errors in your code,
with try/catch, cferror, or site-wide error handling, which would then
prevent an error showing up in the CF logs unless you rethrow them.)

Finally, are you able to look at any of the files, to see what the content
is? That too may help you start to understand what they're about.

Let us know if this is helpful.



/charlie arehart

char...@carehart.org

Providing fast, remote, on-demand troubleshooting services for CF (and
CFBuilder)

More at http://www.carehart.org/consulting

 

From: cfaussie@googlegroups.com [mailto:cfaussie@googlegroups.com] On Behalf
Of Brett Payne-Rhodes
Sent: Tuesday, December 20, 2011 8:23 PM
To: cfaussie@googlegroups.com
Subject: [cfaussie] CF Temp files

 

Hi,

A couple of days ago one of our production servers ran out of disk space. I
assumed it was the application not cleaning up temporary files but after the
initial panic we discovered that the
JRun4\servers\cfusion\SERVER-INF\temp\cfusion-war-tmp directory was FULL of
files named like "neotmp999...999.tmp". I have a good idea where these files
are being generated in the application but they are being generated by CF
and we have no direct visibility of them. And I assume that up until a few
days ago that CF itself was cleaning them up - otherwise the application
would have failed months ago. When the disk full error happened there were
over 6.000 of these files consuming over 15GB of disk space and all of them
were time-stamped in the previous three days and the other server was close
to being out of disk space. Today both servers have over 2,000 of these
files consuming over 6.5GB of disk so we will be running a manual cleanup
very soon.

I've searched the usual places but there is nothing I can find that
resembles this issue. There is some talk of sandbox security on that
directory causing some access issues but nothing about temporary files not
being cleaned up.

Has anybody seen this before? Or have an idea what might be causing it or
how to fix it?

Many thanks and Happy Christmas to all!

Brett
B)

-- 
You received this message because you are subscribed to the Google Groups
"cfaussie" group.
To post to this group, send email to cfaussie@googlegroups.com.
To unsubscribe from this group, send email to
cfaussie+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/cfaussie?hl=en.

-- 
You received this message because you are subscribed to the Google Groups 
"cfaussie" group.
To post to this group, send email to cfaussie@googlegroups.com.
To unsubscribe from this group, send email to 
cfaussie+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/cfaussie?hl=en.

Re: [cfaussie] CF Temp files

[Brett Payne-Rhodes]

You're right, it feels like something must have changed and I'm trying to
find out from the guys who look after the infrastructure but I would be
surprised if there had been a change that we weren't told about.

B)


On Wed, Dec 21, 2011 at 9:36 AM, Dale Fraser  wrote:

> Did you install anything recently?
>
> ** **
>
> Like the new CF hotfix perhaps, something must have changed.
>
> ** **
>
> Regards
>
> Dale Fraser
>
> ** **
>
> http://dale.fraser.id.au
>
> http://cfmldocs.com
>
> http://learncf.com
>
> http://flexcf.com
>
> ** **
>
> *From:* cfaussie@googlegroups.com [mailto:cfaussie@googlegroups.com] *On
> Behalf Of *Brett Payne-Rhodes
> *Sent:* Wednesday, 21 December 2011 12:23 PM
> *To:* cfaussie@googlegroups.com
> *Subject:* [cfaussie] CF Temp files
>
> ** **
>
> Hi,
>
> A couple of days ago one of our production servers ran out of disk space.
> I assumed it was the application not cleaning up temporary files but after
> the initial panic we discovered that the
> JRun4\servers\cfusion\SERVER-INF\temp\cfusion-war-tmp directory was FULL of
> files named like "neotmp999...999.tmp". I have a good idea where these
> files are being generated in the application but they are being generated
> by CF and we have no direct visibility of them. And I assume that up until
> a few days ago that CF itself was cleaning them up - otherwise the
> application would have failed months ago. When the disk full error happened
> there were over 6.000 of these files consuming over 15GB of disk space and
> all of them were time-stamped in the previous three days and the other
> server was close to being out of disk space. Today both servers have over
> 2,000 of these files consuming over 6.5GB of disk so we will be running a
> manual cleanup very soon.
>
> I've searched the usual places but there is nothing I can find that
> resembles this issue. There is some talk of sandbox security on that
> directory causing some access issues but nothing about temporary files not
> being cleaned up.
>
> Has anybody seen this before? Or have an idea what might be causing it or
> how to fix it?
>
> Many thanks and Happy Christmas to all!
>
> Brett
> B)
>
> --
> You received this message because you are subscribed to the Google Groups
> "cfaussie" group.
> To post to this group, send email to cfaussie@googlegroups.com.
> To unsubscribe from this group, send email to
> cfaussie+unsubscr...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/cfaussie?hl=en.
>
> --
> You received this message because you are subscribed to the Google Groups
> "cfaussie" group.
> To post to this group, send email to cfaussie@googlegroups.com.
> To unsubscribe from this group, send email to
> cfaussie+unsubscr...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/cfaussie?hl=en.
>

-- 
You received this message because you are subscribed to the Google Groups 
"cfaussie" group.
To post to this group, send email to cfaussie@googlegroups.com.
To unsubscribe from this group, send email to 
cfaussie+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/cfaussie?hl=en.

RE: [cfaussie] CF Temp files

[Dale Fraser]

Did you install anything recently?

 

Like the new CF hotfix perhaps, something must have changed.

 

Regards

Dale Fraser

 

  http://dale.fraser.id.au

  http://cfmldocs.com

  http://learncf.com

  http://flexcf.com

 

From: cfaussie@googlegroups.com [mailto:cfaussie@googlegroups.com] On Behalf
Of Brett Payne-Rhodes
Sent: Wednesday, 21 December 2011 12:23 PM
To: cfaussie@googlegroups.com
Subject: [cfaussie] CF Temp files

 

Hi,

A couple of days ago one of our production servers ran out of disk space. I
assumed it was the application not cleaning up temporary files but after the
initial panic we discovered that the
JRun4\servers\cfusion\SERVER-INF\temp\cfusion-war-tmp directory was FULL of
files named like "neotmp999...999.tmp". I have a good idea where these files
are being generated in the application but they are being generated by CF
and we have no direct visibility of them. And I assume that up until a few
days ago that CF itself was cleaning them up - otherwise the application
would have failed months ago. When the disk full error happened there were
over 6.000 of these files consuming over 15GB of disk space and all of them
were time-stamped in the previous three days and the other server was close
to being out of disk space. Today both servers have over 2,000 of these
files consuming over 6.5GB of disk so we will be running a manual cleanup
very soon.

I've searched the usual places but there is nothing I can find that
resembles this issue. There is some talk of sandbox security on that
directory causing some access issues but nothing about temporary files not
being cleaned up.

Has anybody seen this before? Or have an idea what might be causing it or
how to fix it?

Many thanks and Happy Christmas to all!

Brett
B)

-- 
You received this message because you are subscribed to the Google Groups
"cfaussie" group.
To post to this group, send email to cfaussie@googlegroups.com.
To unsubscribe from this group, send email to
cfaussie+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/cfaussie?hl=en.

-- 
You received this message because you are subscribed to the Google Groups 
"cfaussie" group.
To post to this group, send email to cfaussie@googlegroups.com.
To unsubscribe from this group, send email to 
cfaussie+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/cfaussie?hl=en.

[cfaussie] CF Temp files

[Brett Payne-Rhodes]

Hi,

A couple of days ago one of our production servers ran out of disk space. I
assumed it was the application not cleaning up temporary files but after
the initial panic we discovered that the
JRun4\servers\cfusion\SERVER-INF\temp\cfusion-war-tmp directory was FULL of
files named like "neotmp999...999.tmp". I have a good idea where these
files are being generated in the application but they are being generated
by CF and we have no direct visibility of them. And I assume that up until
a few days ago that CF itself was cleaning them up - otherwise the
application would have failed months ago. When the disk full error happened
there were over 6.000 of these files consuming over 15GB of disk space and
all of them were time-stamped in the previous three days and the other
server was close to being out of disk space. Today both servers have over
2,000 of these files consuming over 6.5GB of disk so we will be running a
manual cleanup very soon.

I've searched the usual places but there is nothing I can find that
resembles this issue. There is some talk of sandbox security on that
directory causing some access issues but nothing about temporary files not
being cleaned up.

Has anybody seen this before? Or have an idea what might be causing it or
how to fix it?

Many thanks and Happy Christmas to all!

Brett
B)

-- 
You received this message because you are subscribed to the Google Groups 
"cfaussie" group.
To post to this group, send email to cfaussie@googlegroups.com.
To unsubscribe from this group, send email to 
cfaussie+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/cfaussie?hl=en.