[jira] [Commented] (FOP-2211) [PATCH] Fix improve the handling of temporary files using the new URI resource resolvers
[
https://issues.apache.org/jira/browse/FOP-2211?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13632745#comment-13632745
]
Peter Hancock commented on FOP-2211:
Fix committed in revision 1468361
Alexios - Thanks for you help and please feel free to post any ideas or code
for better handling of temp data.
[PATCH] Fix improve the handling of temporary files using the new URI
resource resolvers
--
Key: FOP-2211
URL: https://issues.apache.org/jira/browse/FOP-2211
Project: Fop
Issue Type: Bug
Components: general
Affects Versions: trunk
Reporter: Alexios Giotis
Fix For: trunk
Attachments: fop.patch, fop.patch, tempurisimple.patch, xgc.patch
As written in http://markmail.org/message/zelumstxxsdyvkcz , after the merge
of the Temp_URI_Resolution branch (Sept 2012), the actual pattern of using
temp files has changed from:
{code}
File tmpFile = File.createTempFile();
// Write and read from the file
tmpFile.delete();
{code}
to:
{code}
File tmpFile = new File(System.getProperty(java.io.tmpdir),
counterStartingFrom1AsString);
tmpFile.deleteOnExit();
// Write and read from the file
{code}
This is fine when FOP is executed from the command line (which I guess this
is how most people use it) but it introduces a number of bad side effects for
long running processes that use FOP embedded.
1. Different FOP processes can't be executed in parallel on the same system
because creating the same temp file fails.
2. If the JVM is not normally terminated, the temp files are never deleted
and the next invocation of the JVM fails to run.
3. deleteOnExit() keeps for the life of the JVM an unknown number of temp
files both on disk and a reference in memory.
There should not be a need to implement a custom resource resolver when using
FOP embedded in order to fix those issues. The default implementation should
work at least as good as it worked in FOP 1.1 or earlier.
Attached are 2 patches, one for XGC and one for FOP that should fix and
improve the handling of at least the temporary files.
For reference, [1] lists some reasons for implementing the new URI resource
resolvers.
[1] http://wiki.apache.org/xmlgraphics-fop/URIResolution
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (FOP-2211) [PATCH] Fix improve the handling of temporary files using the new URI resource resolvers
[
https://issues.apache.org/jira/browse/FOP-2211?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13633107#comment-13633107
]
Vincent Hennebert commented on FOP-2211:
To elaborate a bit, I think we now all agree that temporary files shouldn't be
handled by the URI framework. Those two issues are orthogonal.
The suggestion would be to scrape the tmp:// scheme from the URI framework and
to replace it with a system that manages temporary resources. A good use case
would be the PSDocumentHandler that, when optimize-resources is set to true,
stores the output in a temp file in order to re-arrange it in a second pass.
Instead of storing the whole output in one big temp file, it could be split
into several parts each stored in a different file, and then put together. A
new temporary resource manager should allow to easily do that.
Alexios, if you feel like exploring this idea further (you might have started
already?), then you would be most welcome, and I'll try and spare cycles to
review your work.
Thanks,
Vincent
[PATCH] Fix improve the handling of temporary files using the new URI
resource resolvers
--
Key: FOP-2211
URL: https://issues.apache.org/jira/browse/FOP-2211
Project: Fop
Issue Type: Bug
Components: general
Affects Versions: trunk
Reporter: Alexios Giotis
Assignee: Peter Hancock
Fix For: trunk
Attachments: fop.patch, fop.patch, tempurisimple.patch, xgc.patch
As written in http://markmail.org/message/zelumstxxsdyvkcz , after the merge
of the Temp_URI_Resolution branch (Sept 2012), the actual pattern of using
temp files has changed from:
{code}
File tmpFile = File.createTempFile();
// Write and read from the file
tmpFile.delete();
{code}
to:
{code}
File tmpFile = new File(System.getProperty(java.io.tmpdir),
counterStartingFrom1AsString);
tmpFile.deleteOnExit();
// Write and read from the file
{code}
This is fine when FOP is executed from the command line (which I guess this
is how most people use it) but it introduces a number of bad side effects for
long running processes that use FOP embedded.
1. Different FOP processes can't be executed in parallel on the same system
because creating the same temp file fails.
2. If the JVM is not normally terminated, the temp files are never deleted
and the next invocation of the JVM fails to run.
3. deleteOnExit() keeps for the life of the JVM an unknown number of temp
files both on disk and a reference in memory.
There should not be a need to implement a custom resource resolver when using
FOP embedded in order to fix those issues. The default implementation should
work at least as good as it worked in FOP 1.1 or earlier.
Attached are 2 patches, one for XGC and one for FOP that should fix and
improve the handling of at least the temporary files.
For reference, [1] lists some reasons for implementing the new URI resource
resolvers.
[1] http://wiki.apache.org/xmlgraphics-fop/URIResolution
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (FOP-2211) [PATCH] Fix improve the handling of temporary files using the new URI resource resolvers
[
https://issues.apache.org/jira/browse/FOP-2211?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13633353#comment-13633353
]
Alexios Giotis commented on FOP-2211:
-
Peter, thanks for committing this.
Vincent, I have not started working since the patch addresses my runtime issues
and I was not sure if there was an agreement on changing that. After an
official release, it will be much harder to improve the handling of temporary
resources as it's part of the public API. So, I will give it a try next week
and if it looks good, I will post the patches for FOP and XGC.
[PATCH] Fix improve the handling of temporary files using the new URI
resource resolvers
--
Key: FOP-2211
URL: https://issues.apache.org/jira/browse/FOP-2211
Project: Fop
Issue Type: Bug
Components: general
Affects Versions: trunk
Reporter: Alexios Giotis
Assignee: Peter Hancock
Fix For: trunk
Attachments: fop.patch, fop.patch, tempurisimple.patch, xgc.patch
As written in http://markmail.org/message/zelumstxxsdyvkcz , after the merge
of the Temp_URI_Resolution branch (Sept 2012), the actual pattern of using
temp files has changed from:
{code}
File tmpFile = File.createTempFile();
// Write and read from the file
tmpFile.delete();
{code}
to:
{code}
File tmpFile = new File(System.getProperty(java.io.tmpdir),
counterStartingFrom1AsString);
tmpFile.deleteOnExit();
// Write and read from the file
{code}
This is fine when FOP is executed from the command line (which I guess this
is how most people use it) but it introduces a number of bad side effects for
long running processes that use FOP embedded.
1. Different FOP processes can't be executed in parallel on the same system
because creating the same temp file fails.
2. If the JVM is not normally terminated, the temp files are never deleted
and the next invocation of the JVM fails to run.
3. deleteOnExit() keeps for the life of the JVM an unknown number of temp
files both on disk and a reference in memory.
There should not be a need to implement a custom resource resolver when using
FOP embedded in order to fix those issues. The default implementation should
work at least as good as it worked in FOP 1.1 or earlier.
Attached are 2 patches, one for XGC and one for FOP that should fix and
improve the handling of at least the temporary files.
For reference, [1] lists some reasons for implementing the new URI resource
resolvers.
[1] http://wiki.apache.org/xmlgraphics-fop/URIResolution
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (FOP-2211) [PATCH] Fix improve the handling of temporary files using the new URI resource resolvers
[
https://issues.apache.org/jira/browse/FOP-2211?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13628759#comment-13628759
]
Peter Hancock commented on FOP-2211:
Hi Alexios,
Sorry for the late reply. I plan to commit the patch with the omission of the
File.deleteOnExit() in the next few days
[PATCH] Fix improve the handling of temporary files using the new URI
resource resolvers
--
Key: FOP-2211
URL: https://issues.apache.org/jira/browse/FOP-2211
Project: Fop
Issue Type: Bug
Components: general
Affects Versions: trunk
Reporter: Alexios Giotis
Fix For: trunk
Attachments: fop.patch, fop.patch, tempurisimple.patch, xgc.patch
As written in http://markmail.org/message/zelumstxxsdyvkcz , after the merge
of the Temp_URI_Resolution branch (Sept 2012), the actual pattern of using
temp files has changed from:
{code}
File tmpFile = File.createTempFile();
// Write and read from the file
tmpFile.delete();
{code}
to:
{code}
File tmpFile = new File(System.getProperty(java.io.tmpdir),
counterStartingFrom1AsString);
tmpFile.deleteOnExit();
// Write and read from the file
{code}
This is fine when FOP is executed from the command line (which I guess this
is how most people use it) but it introduces a number of bad side effects for
long running processes that use FOP embedded.
1. Different FOP processes can't be executed in parallel on the same system
because creating the same temp file fails.
2. If the JVM is not normally terminated, the temp files are never deleted
and the next invocation of the JVM fails to run.
3. deleteOnExit() keeps for the life of the JVM an unknown number of temp
files both on disk and a reference in memory.
There should not be a need to implement a custom resource resolver when using
FOP embedded in order to fix those issues. The default implementation should
work at least as good as it worked in FOP 1.1 or earlier.
Attached are 2 patches, one for XGC and one for FOP that should fix and
improve the handling of at least the temporary files.
For reference, [1] lists some reasons for implementing the new URI resource
resolvers.
[1] http://wiki.apache.org/xmlgraphics-fop/URIResolution
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (FOP-2211) [PATCH] Fix improve the handling of temporary files using the new URI resource resolvers
[
https://issues.apache.org/jira/browse/FOP-2211?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13625920#comment-13625920
]
Alexios Giotis commented on FOP-2211:
-
Peter (or any other committer), will you commit your (Peter's) patch with the
exception of the deleteOnExit() method ?
Or do you prefer that I create another (bigger) patch that will not be 100%
compatible with the current trunk but will attempt to give an elegant way of
handling temp files ?
[PATCH] Fix improve the handling of temporary files using the new URI
resource resolvers
--
Key: FOP-2211
URL: https://issues.apache.org/jira/browse/FOP-2211
Project: Fop
Issue Type: Bug
Components: general
Affects Versions: trunk
Reporter: Alexios Giotis
Fix For: trunk
Attachments: fop.patch, fop.patch, tempurisimple.patch, xgc.patch
As written in http://markmail.org/message/zelumstxxsdyvkcz , after the merge
of the Temp_URI_Resolution branch (Sept 2012), the actual pattern of using
temp files has changed from:
{code}
File tmpFile = File.createTempFile();
// Write and read from the file
tmpFile.delete();
{code}
to:
{code}
File tmpFile = new File(System.getProperty(java.io.tmpdir),
counterStartingFrom1AsString);
tmpFile.deleteOnExit();
// Write and read from the file
{code}
This is fine when FOP is executed from the command line (which I guess this
is how most people use it) but it introduces a number of bad side effects for
long running processes that use FOP embedded.
1. Different FOP processes can't be executed in parallel on the same system
because creating the same temp file fails.
2. If the JVM is not normally terminated, the temp files are never deleted
and the next invocation of the JVM fails to run.
3. deleteOnExit() keeps for the life of the JVM an unknown number of temp
files both on disk and a reference in memory.
There should not be a need to implement a custom resource resolver when using
FOP embedded in order to fix those issues. The default implementation should
work at least as good as it worked in FOP 1.1 or earlier.
Attached are 2 patches, one for XGC and one for FOP that should fix and
improve the handling of at least the temporary files.
For reference, [1] lists some reasons for implementing the new URI resource
resolvers.
[1] http://wiki.apache.org/xmlgraphics-fop/URIResolution
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (FOP-2211) [PATCH] Fix improve the handling of temporary files using the new URI resource resolvers
[
https://issues.apache.org/jira/browse/FOP-2211?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13600190#comment-13600190
]
Peter Hancock commented on FOP-2211:
Hi Alexios,
The latest patch ensures file is deleted when the resource is closed.
Peter
[PATCH] Fix improve the handling of temporary files using the new URI
resource resolvers
--
Key: FOP-2211
URL: https://issues.apache.org/jira/browse/FOP-2211
Project: Fop
Issue Type: Bug
Components: general
Affects Versions: trunk
Reporter: Alexios Giotis
Fix For: trunk
Attachments: fop.patch, fop.patch, tempurisimple.patch, xgc.patch
As written in http://markmail.org/message/zelumstxxsdyvkcz , after the merge
of the Temp_URI_Resolution branch (Sept 2012), the actual pattern of using
temp files has changed from:
{code}
File tmpFile = File.createTempFile();
// Write and read from the file
tmpFile.delete();
{code}
to:
{code}
File tmpFile = new File(System.getProperty(java.io.tmpdir),
counterStartingFrom1AsString);
tmpFile.deleteOnExit();
// Write and read from the file
{code}
This is fine when FOP is executed from the command line (which I guess this
is how most people use it) but it introduces a number of bad side effects for
long running processes that use FOP embedded.
1. Different FOP processes can't be executed in parallel on the same system
because creating the same temp file fails.
2. If the JVM is not normally terminated, the temp files are never deleted
and the next invocation of the JVM fails to run.
3. deleteOnExit() keeps for the life of the JVM an unknown number of temp
files both on disk and a reference in memory.
There should not be a need to implement a custom resource resolver when using
FOP embedded in order to fix those issues. The default implementation should
work at least as good as it worked in FOP 1.1 or earlier.
Attached are 2 patches, one for XGC and one for FOP that should fix and
improve the handling of at least the temporary files.
For reference, [1] lists some reasons for implementing the new URI resource
resolvers.
[1] http://wiki.apache.org/xmlgraphics-fop/URIResolution
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (FOP-2211) [PATCH] Fix improve the handling of temporary files using the new URI resource resolvers
[
https://issues.apache.org/jira/browse/FOP-2211?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13600242#comment-13600242
]
Alexios Giotis commented on FOP-2211:
-
Hi Peter,
Thanks for the patch. In general I find that the current handling of temp
resources is not optimal but if we don't want to go for bigger changes, this
patch is an improvement of the current situation. Related to the 3rd point I
mentioned in the issue description, would you please delete in your patch the
line tempFile.deleteOnExit(); ?
Normally, you don't need to do this and it is a memory hog. Have you seen the
implementation in java.io.DeleteOnExitHook ? Every file is put in a
LinkedHashSet.
[PATCH] Fix improve the handling of temporary files using the new URI
resource resolvers
--
Key: FOP-2211
URL: https://issues.apache.org/jira/browse/FOP-2211
Project: Fop
Issue Type: Bug
Components: general
Affects Versions: trunk
Reporter: Alexios Giotis
Fix For: trunk
Attachments: fop.patch, fop.patch, tempurisimple.patch, xgc.patch
As written in http://markmail.org/message/zelumstxxsdyvkcz , after the merge
of the Temp_URI_Resolution branch (Sept 2012), the actual pattern of using
temp files has changed from:
{code}
File tmpFile = File.createTempFile();
// Write and read from the file
tmpFile.delete();
{code}
to:
{code}
File tmpFile = new File(System.getProperty(java.io.tmpdir),
counterStartingFrom1AsString);
tmpFile.deleteOnExit();
// Write and read from the file
{code}
This is fine when FOP is executed from the command line (which I guess this
is how most people use it) but it introduces a number of bad side effects for
long running processes that use FOP embedded.
1. Different FOP processes can't be executed in parallel on the same system
because creating the same temp file fails.
2. If the JVM is not normally terminated, the temp files are never deleted
and the next invocation of the JVM fails to run.
3. deleteOnExit() keeps for the life of the JVM an unknown number of temp
files both on disk and a reference in memory.
There should not be a need to implement a custom resource resolver when using
FOP embedded in order to fix those issues. The default implementation should
work at least as good as it worked in FOP 1.1 or earlier.
Attached are 2 patches, one for XGC and one for FOP that should fix and
improve the handling of at least the temporary files.
For reference, [1] lists some reasons for implementing the new URI resource
resolvers.
[1] http://wiki.apache.org/xmlgraphics-fop/URIResolution
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: [jira] [Commented] (FOP-2211) [PATCH] Fix improve the handling of temporary files using the new URI resource resolvers
Hi Peter,
I am not reusing in any way the temp files. Since FOP creates them, I believe
FOP should also delete them. This should happen when temp files are no longer
required and not when the JVM will terminate. This is what FOP does in version
1.1 or before. I could setup my own resource handler and override the close
method of the Resource to delete the temp file, as you suggested. But fixing
this in FOP is cleaner and benefits other people which use FOP embedded in
their Java application.
Is it clear ? Do you agree ?
Thanks,
Alexios
On 7 Mar 2013, at 12:21, Peter Hancock peter.hanc...@gmail.com wrote:
Hi Alexios,
I take it you are you reading from the temp files more than once,
otherwise you could reluctantly hook into the close method of Resource
to do the file deletion. I am interested to know how you are
benefiting from reusing the temp file.
Thanks,
Peter
On Tue, Mar 5, 2013 at 10:55 AM, Alexios Giotis (JIRA) j...@apache.org
wrote:
[
https://issues.apache.org/jira/browse/FOP-2211?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13593313#comment-13593313
]
Alexios Giotis commented on FOP-2211:
-
Hi Simon,
Thank you for the patch. I looked at it and it does not resolve the issue of
keeping on disk many and big files. In a test we did last week, the temp
file was 100GB and for sure we don't wish to keep such files until the JVM
is normally terminated. For us, this is several months or a year.
Also, I don't think that backwards compatibility is an issue here. This is
trunk, there are many changes since 1.1 that do affect users embedding FOP
in their apps and this is not one of them. I am sure it will be easy to
change your code or to create an adapter.
I will try to find some time to submit updated versions of the patches that
take into account the comments above.
[PATCH] Fix improve the handling of temporary files using the new URI
resource resolvers
--
Key: FOP-2211
URL: https://issues.apache.org/jira/browse/FOP-2211
Project: Fop
Issue Type: Bug
Components: general
Affects Versions: trunk
Reporter: Alexios Giotis
Fix For: trunk
Attachments: fop.patch, tempurisimple.patch, xgc.patch
As written in http://markmail.org/message/zelumstxxsdyvkcz , after the
merge of the Temp_URI_Resolution branch (Sept 2012), the actual pattern of
using temp files has changed from:
{code}
File tmpFile = File.createTempFile();
// Write and read from the file
tmpFile.delete();
{code}
to:
{code}
File tmpFile = new File(System.getProperty(java.io.tmpdir),
counterStartingFrom1AsString);
tmpFile.deleteOnExit();
// Write and read from the file
{code}
This is fine when FOP is executed from the command line (which I guess this
is how most people use it) but it introduces a number of bad side effects
for long running processes that use FOP embedded.
1. Different FOP processes can't be executed in parallel on the same system
because creating the same temp file fails.
2. If the JVM is not normally terminated, the temp files are never deleted
and the next invocation of the JVM fails to run.
3. deleteOnExit() keeps for the life of the JVM an unknown number of temp
files both on disk and a reference in memory.
There should not be a need to implement a custom resource resolver when
using FOP embedded in order to fix those issues. The default implementation
should work at least as good as it worked in FOP 1.1 or earlier.
Attached are 2 patches, one for XGC and one for FOP that should fix and
improve the handling of at least the temporary files.
For reference, [1] lists some reasons for implementing the new URI resource
resolvers.
[1] http://wiki.apache.org/xmlgraphics-fop/URIResolution
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: [jira] [Commented] (FOP-2211) [PATCH] Fix improve the handling of temporary files using the new URI resource resolvers
Hi Alexios,
I take it you are you reading from the temp files more than once,
otherwise you could reluctantly hook into the close method of Resource
to do the file deletion. I am interested to know how you are
benefiting from reusing the temp file.
Thanks,
Peter
On Tue, Mar 5, 2013 at 10:55 AM, Alexios Giotis (JIRA) j...@apache.org wrote:
[
https://issues.apache.org/jira/browse/FOP-2211?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13593313#comment-13593313
]
Alexios Giotis commented on FOP-2211:
-
Hi Simon,
Thank you for the patch. I looked at it and it does not resolve the issue of
keeping on disk many and big files. In a test we did last week, the temp file
was 100GB and for sure we don't wish to keep such files until the JVM is
normally terminated. For us, this is several months or a year.
Also, I don't think that backwards compatibility is an issue here. This is
trunk, there are many changes since 1.1 that do affect users embedding FOP in
their apps and this is not one of them. I am sure it will be easy to change
your code or to create an adapter.
I will try to find some time to submit updated versions of the patches that
take into account the comments above.
[PATCH] Fix improve the handling of temporary files using the new URI
resource resolvers
--
Key: FOP-2211
URL: https://issues.apache.org/jira/browse/FOP-2211
Project: Fop
Issue Type: Bug
Components: general
Affects Versions: trunk
Reporter: Alexios Giotis
Fix For: trunk
Attachments: fop.patch, tempurisimple.patch, xgc.patch
As written in http://markmail.org/message/zelumstxxsdyvkcz , after the merge
of the Temp_URI_Resolution branch (Sept 2012), the actual pattern of using
temp files has changed from:
{code}
File tmpFile = File.createTempFile();
// Write and read from the file
tmpFile.delete();
{code}
to:
{code}
File tmpFile = new File(System.getProperty(java.io.tmpdir),
counterStartingFrom1AsString);
tmpFile.deleteOnExit();
// Write and read from the file
{code}
This is fine when FOP is executed from the command line (which I guess this
is how most people use it) but it introduces a number of bad side effects
for long running processes that use FOP embedded.
1. Different FOP processes can't be executed in parallel on the same system
because creating the same temp file fails.
2. If the JVM is not normally terminated, the temp files are never deleted
and the next invocation of the JVM fails to run.
3. deleteOnExit() keeps for the life of the JVM an unknown number of temp
files both on disk and a reference in memory.
There should not be a need to implement a custom resource resolver when
using FOP embedded in order to fix those issues. The default implementation
should work at least as good as it worked in FOP 1.1 or earlier.
Attached are 2 patches, one for XGC and one for FOP that should fix and
improve the handling of at least the temporary files.
For reference, [1] lists some reasons for implementing the new URI resource
resolvers.
[1] http://wiki.apache.org/xmlgraphics-fop/URIResolution
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (FOP-2211) [PATCH] Fix improve the handling of temporary files using the new URI resource resolvers
[
https://issues.apache.org/jira/browse/FOP-2211?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13593313#comment-13593313
]
Alexios Giotis commented on FOP-2211:
-
Hi Simon,
Thank you for the patch. I looked at it and it does not resolve the issue of
keeping on disk many and big files. In a test we did last week, the temp file
was 100GB and for sure we don't wish to keep such files until the JVM is
normally terminated. For us, this is several months or a year.
Also, I don't think that backwards compatibility is an issue here. This is
trunk, there are many changes since 1.1 that do affect users embedding FOP in
their apps and this is not one of them. I am sure it will be easy to change
your code or to create an adapter.
I will try to find some time to submit updated versions of the patches that
take into account the comments above.
[PATCH] Fix improve the handling of temporary files using the new URI
resource resolvers
--
Key: FOP-2211
URL: https://issues.apache.org/jira/browse/FOP-2211
Project: Fop
Issue Type: Bug
Components: general
Affects Versions: trunk
Reporter: Alexios Giotis
Fix For: trunk
Attachments: fop.patch, tempurisimple.patch, xgc.patch
As written in http://markmail.org/message/zelumstxxsdyvkcz , after the merge
of the Temp_URI_Resolution branch (Sept 2012), the actual pattern of using
temp files has changed from:
{code}
File tmpFile = File.createTempFile();
// Write and read from the file
tmpFile.delete();
{code}
to:
{code}
File tmpFile = new File(System.getProperty(java.io.tmpdir),
counterStartingFrom1AsString);
tmpFile.deleteOnExit();
// Write and read from the file
{code}
This is fine when FOP is executed from the command line (which I guess this
is how most people use it) but it introduces a number of bad side effects for
long running processes that use FOP embedded.
1. Different FOP processes can't be executed in parallel on the same system
because creating the same temp file fails.
2. If the JVM is not normally terminated, the temp files are never deleted
and the next invocation of the JVM fails to run.
3. deleteOnExit() keeps for the life of the JVM an unknown number of temp
files both on disk and a reference in memory.
There should not be a need to implement a custom resource resolver when using
FOP embedded in order to fix those issues. The default implementation should
work at least as good as it worked in FOP 1.1 or earlier.
Attached are 2 patches, one for XGC and one for FOP that should fix and
improve the handling of at least the temporary files.
For reference, [1] lists some reasons for implementing the new URI resource
resolvers.
[1] http://wiki.apache.org/xmlgraphics-fop/URIResolution
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (FOP-2211) [PATCH] Fix improve the handling of temporary files using the new URI resource resolvers
[
https://issues.apache.org/jira/browse/FOP-2211?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13583091#comment-13583091
]
Chris Bowditch commented on FOP-2211:
-
Hi Alexios.
A Virtual File System is a really useful tool in a cloud environment as
different components running on different physical machines can talk to each
other and read/write data to shared resources, without having to rely on shared
drives.
It may be unlikely that a temp file written by tenant A can be read by tenant
B, but compliance dictates that we should put safeguards in place to esnure 0%
risk of such an occurance, We have the virtual file system which is partitioned
by tenant so why not use that? Just need an interface in FOP to allow temp
files to be handled in a way other than temp files. Of course that interface
should not prevent temporary data still being handled in temp files, so we will
look to make the necessary revisions to ensure both use cases are possible.
Thanks,
Chris
[PATCH] Fix improve the handling of temporary files using the new URI
resource resolvers
--
Key: FOP-2211
URL: https://issues.apache.org/jira/browse/FOP-2211
Project: Fop
Issue Type: Bug
Components: general
Affects Versions: trunk
Reporter: Alexios Giotis
Fix For: trunk
Attachments: fop.patch, xgc.patch
As written in http://markmail.org/message/zelumstxxsdyvkcz , after the merge
of the Temp_URI_Resolution branch (Sept 2012), the actual pattern of using
temp files has changed from:
{code}
File tmpFile = File.createTempFile();
// Write and read from the file
tmpFile.delete();
{code}
to:
{code}
File tmpFile = new File(System.getProperty(java.io.tmpdir),
counterStartingFrom1AsString);
tmpFile.deleteOnExit();
// Write and read from the file
{code}
This is fine when FOP is executed from the command line (which I guess this
is how most people use it) but it introduces a number of bad side effects for
long running processes that use FOP embedded.
1. Different FOP processes can't be executed in parallel on the same system
because creating the same temp file fails.
2. If the JVM is not normally terminated, the temp files are never deleted
and the next invocation of the JVM fails to run.
3. deleteOnExit() keeps for the life of the JVM an unknown number of temp
files both on disk and a reference in memory.
There should not be a need to implement a custom resource resolver when using
FOP embedded in order to fix those issues. The default implementation should
work at least as good as it worked in FOP 1.1 or earlier.
Attached are 2 patches, one for XGC and one for FOP that should fix and
improve the handling of at least the temporary files.
For reference, [1] lists some reasons for implementing the new URI resource
resolvers.
[1] http://wiki.apache.org/xmlgraphics-fop/URIResolution
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (FOP-2211) [PATCH] Fix improve the handling of temporary files using the new URI resource resolvers
[
https://issues.apache.org/jira/browse/FOP-2211?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13581324#comment-13581324
]
Chris Bowditch commented on FOP-2211:
-
Hi Alex,
I agree that Print Streams can get very large and holding them in memory is not
a good idea. Actually for cloud environments we don't plan to hold such files
in memory, but rather in a virtual file system, one that is ring fenced for
each tenant. Unlike a file system, where every tenant can see the files for
every other tenant on the system, the virtual file system protects against
this. The virtual file system is typically implemented using a Database.
Hopefully this helps clarifies our intentions from a requirements perspective.
I cannot comment on the lower level details.
Thanks,
Chris
[PATCH] Fix improve the handling of temporary files using the new URI
resource resolvers
--
Key: FOP-2211
URL: https://issues.apache.org/jira/browse/FOP-2211
Project: Fop
Issue Type: Bug
Components: general
Affects Versions: trunk
Reporter: Alexios Giotis
Fix For: trunk
Attachments: fop.patch, xgc.patch
As written in http://markmail.org/message/zelumstxxsdyvkcz , after the merge
of the Temp_URI_Resolution branch (Sept 2012), the actual pattern of using
temp files has changed from:
{code}
File tmpFile = File.createTempFile();
// Write and read from the file
tmpFile.delete();
{code}
to:
{code}
File tmpFile = new File(System.getProperty(java.io.tmpdir),
counterStartingFrom1AsString);
tmpFile.deleteOnExit();
// Write and read from the file
{code}
This is fine when FOP is executed from the command line (which I guess this
is how most people use it) but it introduces a number of bad side effects for
long running processes that use FOP embedded.
1. Different FOP processes can't be executed in parallel on the same system
because creating the same temp file fails.
2. If the JVM is not normally terminated, the temp files are never deleted
and the next invocation of the JVM fails to run.
3. deleteOnExit() keeps for the life of the JVM an unknown number of temp
files both on disk and a reference in memory.
There should not be a need to implement a custom resource resolver when using
FOP embedded in order to fix those issues. The default implementation should
work at least as good as it worked in FOP 1.1 or earlier.
Attached are 2 patches, one for XGC and one for FOP that should fix and
improve the handling of at least the temporary files.
For reference, [1] lists some reasons for implementing the new URI resource
resolvers.
[1] http://wiki.apache.org/xmlgraphics-fop/URIResolution
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (FOP-2211) [PATCH] Fix improve the handling of temporary files using the new URI resource resolvers
[
https://issues.apache.org/jira/browse/FOP-2211?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13581402#comment-13581402
]
Alexios Giotis commented on FOP-2211:
-
Hi Chris,
I don't understand the advantages of such a virtual file system or why would
anyone need to isolate the *temp* resources of different tenants that are using
a common filesystem. Are you afraid of collision on filenames ? Or that if the
application crashes, the temp files will not be deleted ?
Anyway, that would need going into more details about the use case and the
relevant cloud infrastructure. I will assume that you really have this need. I
will wait a few more days in case the initial authors (Peter and Mehdi) have
some more input. Then, I will update the 2 patches taking under consideration
the comments so far.
[PATCH] Fix improve the handling of temporary files using the new URI
resource resolvers
--
Key: FOP-2211
URL: https://issues.apache.org/jira/browse/FOP-2211
Project: Fop
Issue Type: Bug
Components: general
Affects Versions: trunk
Reporter: Alexios Giotis
Fix For: trunk
Attachments: fop.patch, xgc.patch
As written in http://markmail.org/message/zelumstxxsdyvkcz , after the merge
of the Temp_URI_Resolution branch (Sept 2012), the actual pattern of using
temp files has changed from:
{code}
File tmpFile = File.createTempFile();
// Write and read from the file
tmpFile.delete();
{code}
to:
{code}
File tmpFile = new File(System.getProperty(java.io.tmpdir),
counterStartingFrom1AsString);
tmpFile.deleteOnExit();
// Write and read from the file
{code}
This is fine when FOP is executed from the command line (which I guess this
is how most people use it) but it introduces a number of bad side effects for
long running processes that use FOP embedded.
1. Different FOP processes can't be executed in parallel on the same system
because creating the same temp file fails.
2. If the JVM is not normally terminated, the temp files are never deleted
and the next invocation of the JVM fails to run.
3. deleteOnExit() keeps for the life of the JVM an unknown number of temp
files both on disk and a reference in memory.
There should not be a need to implement a custom resource resolver when using
FOP embedded in order to fix those issues. The default implementation should
work at least as good as it worked in FOP 1.1 or earlier.
Attached are 2 patches, one for XGC and one for FOP that should fix and
improve the handling of at least the temporary files.
For reference, [1] lists some reasons for implementing the new URI resource
resolvers.
[1] http://wiki.apache.org/xmlgraphics-fop/URIResolution
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (FOP-2211) [PATCH] Fix improve the handling of temporary files using the new URI resource resolvers
[
https://issues.apache.org/jira/browse/FOP-2211?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13578432#comment-13578432
]
simon steiner commented on FOP-2211:
Hi, it was good before that the temp resource creation
(DefaultTempResourceResolver) was separate from the class detecting if the uri
was a temp resource (TempAwareResourceResolver). Our team was using this since
we created our own in memory temp resource class.
These changes mean he user who wants their own temp resource creation method
need to also know about temp uris by calling isTempURI in their class.
[PATCH] Fix improve the handling of temporary files using the new URI
resource resolvers
--
Key: FOP-2211
URL: https://issues.apache.org/jira/browse/FOP-2211
Project: Fop
Issue Type: Bug
Components: general
Affects Versions: trunk
Reporter: Alexios Giotis
Fix For: trunk
Attachments: fop.patch, xgc.patch
As written in http://markmail.org/message/zelumstxxsdyvkcz , after the merge
of the Temp_URI_Resolution branch (Sept 2012), the actual pattern of using
temp files has changed from:
{code}
File tmpFile = File.createTempFile();
// Write and read from the file
tmpFile.delete();
{code}
to:
{code}
File tmpFile = new File(System.getProperty(java.io.tmpdir),
counterStartingFrom1AsString);
tmpFile.deleteOnExit();
// Write and read from the file
{code}
This is fine when FOP is executed from the command line (which I guess this
is how most people use it) but it introduces a number of bad side effects for
long running processes that use FOP embedded.
1. Different FOP processes can't be executed in parallel on the same system
because creating the same temp file fails.
2. If the JVM is not normally terminated, the temp files are never deleted
and the next invocation of the JVM fails to run.
3. deleteOnExit() keeps for the life of the JVM an unknown number of temp
files both on disk and a reference in memory.
There should not be a need to implement a custom resource resolver when using
FOP embedded in order to fix those issues. The default implementation should
work at least as good as it worked in FOP 1.1 or earlier.
Attached are 2 patches, one for XGC and one for FOP that should fix and
improve the handling of at least the temporary files.
For reference, [1] lists some reasons for implementing the new URI resource
resolvers.
[1] http://wiki.apache.org/xmlgraphics-fop/URIResolution
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (FOP-2211) [PATCH] Fix improve the handling of temporary files using the new URI resource resolvers
[
https://issues.apache.org/jira/browse/FOP-2211?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13578494#comment-13578494
]
Alexios Giotis commented on FOP-2211:
-
Hi Simon, since DefaultTempResourceResolver and TempAwareResourceResolver are
both private inner classes of ResourceResolverFactory, I guess your are calling
the public static ResourceResolver createTempAwareResourceResolver(). In
that case you are passing an implementation of TempResourceResolver and
another implementation of ResourceResolver. I will wait for some more input
before creating another patch.
In the meantime, it would help to understand why you need to create your own
temp resource handler. There are currently 3 cases for using temp resources.
1. CachedRenderPagesModel which is used if
(userAgent.isConserveMemoryPolicyEnabled()). Trying to conserve memory by
serialising Java objects in memory sounds a bad idea.
2. AFPStream
Writing an AFP document in memory is again questionable. AFP documents are
typically used for printing and to my experience each AFP file contains
thousands of pages (unless this is a test).
3. PSDocumentHandler
This is used to optimise the postscript resources and actually writes the
postscript output firstly in the temp resource and then in normal output
stream. So, it depends on the size of the output.
Which use case are you using ? Is there a measurable difference in performance ?
My experience is that FOP performance is CPU and memory bound, not disk bound.
What is yours ?
[PATCH] Fix improve the handling of temporary files using the new URI
resource resolvers
--
Key: FOP-2211
URL: https://issues.apache.org/jira/browse/FOP-2211
Project: Fop
Issue Type: Bug
Components: general
Affects Versions: trunk
Reporter: Alexios Giotis
Fix For: trunk
Attachments: fop.patch, xgc.patch
As written in http://markmail.org/message/zelumstxxsdyvkcz , after the merge
of the Temp_URI_Resolution branch (Sept 2012), the actual pattern of using
temp files has changed from:
{code}
File tmpFile = File.createTempFile();
// Write and read from the file
tmpFile.delete();
{code}
to:
{code}
File tmpFile = new File(System.getProperty(java.io.tmpdir),
counterStartingFrom1AsString);
tmpFile.deleteOnExit();
// Write and read from the file
{code}
This is fine when FOP is executed from the command line (which I guess this
is how most people use it) but it introduces a number of bad side effects for
long running processes that use FOP embedded.
1. Different FOP processes can't be executed in parallel on the same system
because creating the same temp file fails.
2. If the JVM is not normally terminated, the temp files are never deleted
and the next invocation of the JVM fails to run.
3. deleteOnExit() keeps for the life of the JVM an unknown number of temp
files both on disk and a reference in memory.
There should not be a need to implement a custom resource resolver when using
FOP embedded in order to fix those issues. The default implementation should
work at least as good as it worked in FOP 1.1 or earlier.
Attached are 2 patches, one for XGC and one for FOP that should fix and
improve the handling of at least the temporary files.
For reference, [1] lists some reasons for implementing the new URI resource
resolvers.
[1] http://wiki.apache.org/xmlgraphics-fop/URIResolution
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (FOP-2211) [PATCH] Fix improve the handling of temporary files using the new URI resource resolvers
[
https://issues.apache.org/jira/browse/FOP-2211?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13577008#comment-13577008
]
Vincent Hennebert commented on FOP-2211:
Hi Alexios,
thanks for your patches. It is true that the current situation is less than
optimal. Maybe temporary resources shouldn't be handled through URIs at all.
However, do we always want to store them in a temp file? Referring to the email
thread you mention above, I think Peter has a point in saying that we may want
to just store them in memory, and reduce the I/O load on the server. Shouldn't
we leave that configurable?
As to the patches: I'm not sure we want to add a delete method to the
ResourceResolver interface. That method applies to temporary resources only and
the scope of that interface is broader than that. I'd rather have that method
on some sub-class implementing that interface.
That said, there seems to be a discrepancy in the code: if I take
PSDocumentHandler as an example, there is a TempResourceURIGenerator that is
used to create tmp URIs, that are then passed on to the userAgent's
ResourceResolver, but nothing indicates that that ResourceResolver can handle
tmp URIs. In fact, that may well not be the case if the user configured the
FopFactory with a custom ResourceResolver.
In light of this, it's probably best to remove the tmp URI facility altogether
and switch back to the createTempFile method for now. I guess we can always
think about a RAM version of temp files later on, if the OS' own caching
facility proves insufficient.
What do people think?
Vincent
[PATCH] Fix improve the handling of temporary files using the new URI
resource resolvers
--
Key: FOP-2211
URL: https://issues.apache.org/jira/browse/FOP-2211
Project: Fop
Issue Type: Bug
Components: general
Affects Versions: trunk
Reporter: Alexis Giotis
Fix For: trunk
Attachments: fop.patch, xgc.patch
As written in http://markmail.org/message/zelumstxxsdyvkcz , after the merge
of the Temp_URI_Resolution branch (Sept 2012), the actual pattern of using
temp files has changed from:
{code}
File tmpFile = File.createTempFile();
// Write and read from the file
tmpFile.delete();
{code}
to:
{code}
File tmpFile = new File(System.getProperty(java.io.tmpdir),
counterStartingFrom1AsString);
tmpFile.deleteOnExit();
// Write and read from the file
{code}
This is fine when FOP is executed from the command line (which I guess this
is how most people use it) but it introduces a number of bad side effects for
long running processes that use FOP embedded.
1. Different FOP processes can't be executed in parallel on the same system
because creating the same temp file fails.
2. If the JVM is not normally terminated, the temp files are never deleted
and the next invocation of the JVM fails to run.
3. deleteOnExit() keeps for the life of the JVM an unknown number of temp
files both on disk and a reference in memory.
There should not be a need to implement a custom resource resolver when using
FOP embedded in order to fix those issues. The default implementation should
work at least as good as it worked in FOP 1.1 or earlier.
Attached are 2 patches, one for XGC and one for FOP that should fix and
improve the handling of at least the temporary files.
For reference, [1] lists some reasons for implementing the new URI resource
resolvers.
[1] http://wiki.apache.org/xmlgraphics-fop/URIResolution
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (FOP-2211) [PATCH] Fix improve the handling of temporary files using the new URI resource resolvers
[
https://issues.apache.org/jira/browse/FOP-2211?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13577086#comment-13577086
]
Alexios Giotis commented on FOP-2211:
-
Hi Vincent,
Thanks for your prompt comment. I don't think that the patch prevents anyone
from plugging another implementation keeping files in memory but I also see
that discrepancy. My opinion is indeed that temporary resources should not be
handled through URIs. As you might noticed in the patch, it makes code more
complex. I had to keep a map from the 'unique' tmp URI to the actual unique
file. A non-URI related class handling them could use the
DeferredFileOutputStream [1] of commons-io (an existing FOP dependency). This
stream has a threshold for switching from memory to disk. If there is a need to
customize something besides that threshold, then we could extract another
interface and have a default implementation using such a stream.
If there is a consensus, I could remove the delete() method from the
ResourceResolver and write a new class for handling temp resources (or an
interface if this needs to be configurable). I think it will be better than
switching back to the createTempFile.
[1]
http://commons.apache.org/io/api-1.3/org/apache/commons/io/output/DeferredFileOutputStream.html
[PATCH] Fix improve the handling of temporary files using the new URI
resource resolvers
--
Key: FOP-2211
URL: https://issues.apache.org/jira/browse/FOP-2211
Project: Fop
Issue Type: Bug
Components: general
Affects Versions: trunk
Reporter: Alexios Giotis
Fix For: trunk
Attachments: fop.patch, xgc.patch
As written in http://markmail.org/message/zelumstxxsdyvkcz , after the merge
of the Temp_URI_Resolution branch (Sept 2012), the actual pattern of using
temp files has changed from:
{code}
File tmpFile = File.createTempFile();
// Write and read from the file
tmpFile.delete();
{code}
to:
{code}
File tmpFile = new File(System.getProperty(java.io.tmpdir),
counterStartingFrom1AsString);
tmpFile.deleteOnExit();
// Write and read from the file
{code}
This is fine when FOP is executed from the command line (which I guess this
is how most people use it) but it introduces a number of bad side effects for
long running processes that use FOP embedded.
1. Different FOP processes can't be executed in parallel on the same system
because creating the same temp file fails.
2. If the JVM is not normally terminated, the temp files are never deleted
and the next invocation of the JVM fails to run.
3. deleteOnExit() keeps for the life of the JVM an unknown number of temp
files both on disk and a reference in memory.
There should not be a need to implement a custom resource resolver when using
FOP embedded in order to fix those issues. The default implementation should
work at least as good as it worked in FOP 1.1 or earlier.
Attached are 2 patches, one for XGC and one for FOP that should fix and
improve the handling of at least the temporary files.
For reference, [1] lists some reasons for implementing the new URI resource
resolvers.
[1] http://wiki.apache.org/xmlgraphics-fop/URIResolution
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
