[jira] [Comment Edited] (FOP-2937) [PATCH]Post PDF generation, Soft reference of PDFObject in PDFReference are not immediately garbage collected leading to excessive memory usage.
[ https://issues.apache.org/jira/browse/FOP-2937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17110890#comment-17110890 ] Piyush Khandelwal edited comment on FOP-2937 at 5/19/20, 6:25 AM: -- [~cbowditch] My suggestion was not to clear the pages - it's to change the SoftReference to WeakReference. If the pages have bookmarks or other links/resources - then for sure the FOP will be holding references to it; and if the reference are held and reachable in program; JVM won't GC them. Also, My test case included bookmarks; forward references - there were no change whatsoever. Also there was no change in the size of the PDF. Please have a look at it once and if you still see there might be some generation issues, please feel free to close it and share the findings. was (Author: piyush.khandelwal): [~cbowditch] My suggestion was not to clear the pages - it's to change the SoftReference to WeakReference. If the pages have bookmarks or other links/resources - then for sure the FOP will be holding references to it; and if the reference are held and reachable in program; JVM won't GC them. Also, My test case included bookmarks; forward references - there were no change whatsoever. Also there was no change in the size of the PDF > [PATCH]Post PDF generation, Soft reference of PDFObject in PDFReference are > not immediately garbage collected leading to excessive memory usage. > > > Key: FOP-2937 > URL: https://issues.apache.org/jira/browse/FOP-2937 > Project: FOP > Issue Type: Improvement >Affects Versions: 2.3, 2.4 >Reporter: Piyush Khandelwal >Priority: Major > Attachments: PDFDictionary.patch, pdfreference.patch > > > PDFReference object holds a SoftReference of PDFObject (PDFPage, PDFLabel, > PDFName etc.). > If we generate a huge PDF ; *I tried with a PDF having around 150 thousand > pages with 12 GB of RAM;* lots of these references linger around waiting for > the garbage collector to collect them. > But GC wont collect them as long as JVM is able to recover enough memory > without throwing out of memory. > Here are few metadata from my testing for further understanding of the issue > - > Stats for generating 1 PDF - > *FO size:* 2.03GB > *Generated PDF No. of Pages:* Around 150 K > RAM: 12 GB > Peak memory that reached while generation - 11.3GB > Residual memory after forced GC: 9 GB > The FO mainly contains tabular data with each pages sequence having max of > 500 rows. > On analyzing the memory dump; found lots of reference for PDFPage, PDFName > etc. > *Question - * Is there any specific reason for using SoftReference in > PDFReference class instead of WeakReference. > Testing by changing SoftReference to WeakReference in PDFReference shows > following improvements without any issue in the generation whatsoever - > Stats for Generating 5 PDF in parallel - > *FO size:* 2.03GB > *Generated PDF No. of Pages:* Around 150 K > RAM: 12 GB > Peak memory that reached while generation - 4GB > Residual memory after forced GC: 300 MB > So, by changing SoftReference to WeakReference, I was able to generate 5 PDF > having 150K pages in parallel with max 4GB Ram; without any generation > issues. > You can clearly see the performance benefits of changing to WeakReference. > But as I dont understand the complete internal details of how FOP works, I > would like to understand if we can target this change and if not what is the > reason behind using SoftReference? -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Reopened] (FOP-2937) [PATCH]Post PDF generation, Soft reference of PDFObject in PDFReference are not immediately garbage collected leading to excessive memory usage.
[ https://issues.apache.org/jira/browse/FOP-2937?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Piyush Khandelwal reopened FOP-2937: > [PATCH]Post PDF generation, Soft reference of PDFObject in PDFReference are > not immediately garbage collected leading to excessive memory usage. > > > Key: FOP-2937 > URL: https://issues.apache.org/jira/browse/FOP-2937 > Project: FOP > Issue Type: Improvement >Affects Versions: 2.3, 2.4 >Reporter: Piyush Khandelwal >Priority: Major > Attachments: PDFDictionary.patch, pdfreference.patch > > > PDFReference object holds a SoftReference of PDFObject (PDFPage, PDFLabel, > PDFName etc.). > If we generate a huge PDF ; *I tried with a PDF having around 150 thousand > pages with 12 GB of RAM;* lots of these references linger around waiting for > the garbage collector to collect them. > But GC wont collect them as long as JVM is able to recover enough memory > without throwing out of memory. > Here are few metadata from my testing for further understanding of the issue > - > Stats for generating 1 PDF - > *FO size:* 2.03GB > *Generated PDF No. of Pages:* Around 150 K > RAM: 12 GB > Peak memory that reached while generation - 11.3GB > Residual memory after forced GC: 9 GB > The FO mainly contains tabular data with each pages sequence having max of > 500 rows. > On analyzing the memory dump; found lots of reference for PDFPage, PDFName > etc. > *Question - * Is there any specific reason for using SoftReference in > PDFReference class instead of WeakReference. > Testing by changing SoftReference to WeakReference in PDFReference shows > following improvements without any issue in the generation whatsoever - > Stats for Generating 5 PDF in parallel - > *FO size:* 2.03GB > *Generated PDF No. of Pages:* Around 150 K > RAM: 12 GB > Peak memory that reached while generation - 4GB > Residual memory after forced GC: 300 MB > So, by changing SoftReference to WeakReference, I was able to generate 5 PDF > having 150K pages in parallel with max 4GB Ram; without any generation > issues. > You can clearly see the performance benefits of changing to WeakReference. > But as I dont understand the complete internal details of how FOP works, I > would like to understand if we can target this change and if not what is the > reason behind using SoftReference? -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FOP-2937) [PATCH]Post PDF generation, Soft reference of PDFObject in PDFReference are not immediately garbage collected leading to excessive memory usage.
[ https://issues.apache.org/jira/browse/FOP-2937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17110890#comment-17110890 ] Piyush Khandelwal commented on FOP-2937: [~cbowditch] My suggestion was not to clear the pages - it's to change the SoftReference to WeakReference. If the pages have bookmarks or other links/resources - then for sure the FOP will be holding references to it; and if the reference are held and reachable in program; JVM won't GC them. Also, My test case included bookmarks; forward references - there were no change whatsoever. Also there was no change in the size of the PDF > [PATCH]Post PDF generation, Soft reference of PDFObject in PDFReference are > not immediately garbage collected leading to excessive memory usage. > > > Key: FOP-2937 > URL: https://issues.apache.org/jira/browse/FOP-2937 > Project: FOP > Issue Type: Improvement >Affects Versions: 2.3, 2.4 >Reporter: Piyush Khandelwal >Priority: Major > Attachments: PDFDictionary.patch, pdfreference.patch > > > PDFReference object holds a SoftReference of PDFObject (PDFPage, PDFLabel, > PDFName etc.). > If we generate a huge PDF ; *I tried with a PDF having around 150 thousand > pages with 12 GB of RAM;* lots of these references linger around waiting for > the garbage collector to collect them. > But GC wont collect them as long as JVM is able to recover enough memory > without throwing out of memory. > Here are few metadata from my testing for further understanding of the issue > - > Stats for generating 1 PDF - > *FO size:* 2.03GB > *Generated PDF No. of Pages:* Around 150 K > RAM: 12 GB > Peak memory that reached while generation - 11.3GB > Residual memory after forced GC: 9 GB > The FO mainly contains tabular data with each pages sequence having max of > 500 rows. > On analyzing the memory dump; found lots of reference for PDFPage, PDFName > etc. > *Question - * Is there any specific reason for using SoftReference in > PDFReference class instead of WeakReference. > Testing by changing SoftReference to WeakReference in PDFReference shows > following improvements without any issue in the generation whatsoever - > Stats for Generating 5 PDF in parallel - > *FO size:* 2.03GB > *Generated PDF No. of Pages:* Around 150 K > RAM: 12 GB > Peak memory that reached while generation - 4GB > Residual memory after forced GC: 300 MB > So, by changing SoftReference to WeakReference, I was able to generate 5 PDF > having 150K pages in parallel with max 4GB Ram; without any generation > issues. > You can clearly see the performance benefits of changing to WeakReference. > But as I dont understand the complete internal details of how FOP works, I > would like to understand if we can target this change and if not what is the > reason behind using SoftReference? -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FOP-2937) [PATCH]Post PDF generation, Soft reference of PDFObject in PDFReference are not immediately garbage collected leading to excessive memory usage.
[ https://issues.apache.org/jira/browse/FOP-2937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17110253#comment-17110253 ] Chris Bowditch commented on FOP-2937: - I think we need to keep these references until entire PDF is generated because bookmarks or other links might refer back to objects on the first page. So clearing them at page 2 onwards would prevent that. We also need to de-duplicate resources and clearing information about earlier pages could also lead to larger file size with duplicated resources. I am therefore closing this suggestion but feel free to disprove my hypothesis and re-open if you can > [PATCH]Post PDF generation, Soft reference of PDFObject in PDFReference are > not immediately garbage collected leading to excessive memory usage. > > > Key: FOP-2937 > URL: https://issues.apache.org/jira/browse/FOP-2937 > Project: FOP > Issue Type: Improvement >Affects Versions: 2.3, 2.4 >Reporter: Piyush Khandelwal >Priority: Major > Attachments: PDFDictionary.patch, pdfreference.patch > > > PDFReference object holds a SoftReference of PDFObject (PDFPage, PDFLabel, > PDFName etc.). > If we generate a huge PDF ; *I tried with a PDF having around 150 thousand > pages with 12 GB of RAM;* lots of these references linger around waiting for > the garbage collector to collect them. > But GC wont collect them as long as JVM is able to recover enough memory > without throwing out of memory. > Here are few metadata from my testing for further understanding of the issue > - > Stats for generating 1 PDF - > *FO size:* 2.03GB > *Generated PDF No. of Pages:* Around 150 K > RAM: 12 GB > Peak memory that reached while generation - 11.3GB > Residual memory after forced GC: 9 GB > The FO mainly contains tabular data with each pages sequence having max of > 500 rows. > On analyzing the memory dump; found lots of reference for PDFPage, PDFName > etc. > *Question - * Is there any specific reason for using SoftReference in > PDFReference class instead of WeakReference. > Testing by changing SoftReference to WeakReference in PDFReference shows > following improvements without any issue in the generation whatsoever - > Stats for Generating 5 PDF in parallel - > *FO size:* 2.03GB > *Generated PDF No. of Pages:* Around 150 K > RAM: 12 GB > Peak memory that reached while generation - 4GB > Residual memory after forced GC: 300 MB > So, by changing SoftReference to WeakReference, I was able to generate 5 PDF > having 150K pages in parallel with max 4GB Ram; without any generation > issues. > You can clearly see the performance benefits of changing to WeakReference. > But as I dont understand the complete internal details of how FOP works, I > would like to understand if we can target this change and if not what is the > reason behind using SoftReference? -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (FOP-2937) [PATCH]Post PDF generation, Soft reference of PDFObject in PDFReference are not immediately garbage collected leading to excessive memory usage.
[ https://issues.apache.org/jira/browse/FOP-2937?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Bowditch closed FOP-2937. --- Resolution: Invalid > [PATCH]Post PDF generation, Soft reference of PDFObject in PDFReference are > not immediately garbage collected leading to excessive memory usage. > > > Key: FOP-2937 > URL: https://issues.apache.org/jira/browse/FOP-2937 > Project: FOP > Issue Type: Improvement >Affects Versions: 2.3, 2.4 >Reporter: Piyush Khandelwal >Priority: Major > Attachments: PDFDictionary.patch, pdfreference.patch > > > PDFReference object holds a SoftReference of PDFObject (PDFPage, PDFLabel, > PDFName etc.). > If we generate a huge PDF ; *I tried with a PDF having around 150 thousand > pages with 12 GB of RAM;* lots of these references linger around waiting for > the garbage collector to collect them. > But GC wont collect them as long as JVM is able to recover enough memory > without throwing out of memory. > Here are few metadata from my testing for further understanding of the issue > - > Stats for generating 1 PDF - > *FO size:* 2.03GB > *Generated PDF No. of Pages:* Around 150 K > RAM: 12 GB > Peak memory that reached while generation - 11.3GB > Residual memory after forced GC: 9 GB > The FO mainly contains tabular data with each pages sequence having max of > 500 rows. > On analyzing the memory dump; found lots of reference for PDFPage, PDFName > etc. > *Question - * Is there any specific reason for using SoftReference in > PDFReference class instead of WeakReference. > Testing by changing SoftReference to WeakReference in PDFReference shows > following improvements without any issue in the generation whatsoever - > Stats for Generating 5 PDF in parallel - > *FO size:* 2.03GB > *Generated PDF No. of Pages:* Around 150 K > RAM: 12 GB > Peak memory that reached while generation - 4GB > Residual memory after forced GC: 300 MB > So, by changing SoftReference to WeakReference, I was able to generate 5 PDF > having 150K pages in parallel with max 4GB Ram; without any generation > issues. > You can clearly see the performance benefits of changing to WeakReference. > But as I dont understand the complete internal details of how FOP works, I > would like to understand if we can target this change and if not what is the > reason behind using SoftReference? -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (FOP-2937) [PATCH]Post PDF generation, Soft reference of PDFObject in PDFReference are not immediately garbage collected leading to excessive memory usage.
[ https://issues.apache.org/jira/browse/FOP-2937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17110107#comment-17110107 ] Piyush Khandelwal edited comment on FOP-2937 at 5/18/20, 10:18 AM: --- Apart from the change that I suggested in the description; I also changed entries HashMap in PDFDictionary class that is extended by PDFPage, PDFPageLabels to WeakHashMap - though this change in itself didn't bring much benefit but when combined with above change; some improvements were seen because of this as well. Adding patch for that as well. was (Author: piyush.khandelwal): Apart from the change that I suggested in the description; I also changed entries HashMap in PDFDictionary class that is extended by PDFPage, PDFPageLabels to WeakHashMap - though this change in itself didn't bring much benefit but in combined to above some improvements was seen because of this well. Adding patch for that as well. > [PATCH]Post PDF generation, Soft reference of PDFObject in PDFReference are > not immediately garbage collected leading to excessive memory usage. > > > Key: FOP-2937 > URL: https://issues.apache.org/jira/browse/FOP-2937 > Project: FOP > Issue Type: Improvement >Affects Versions: 2.3, 2.4 >Reporter: Piyush Khandelwal >Priority: Major > Attachments: PDFDictionary.patch, pdfreference.patch > > > PDFReference object holds a SoftReference of PDFObject (PDFPage, PDFLabel, > PDFName etc.). > If we generate a huge PDF ; *I tried with a PDF having around 150 thousand > pages with 12 GB of RAM;* lots of these references linger around waiting for > the garbage collector to collect them. > But GC wont collect them as long as JVM is able to recover enough memory > without throwing out of memory. > Here are few metadata from my testing for further understanding of the issue > - > Stats for generating 1 PDF - > *FO size:* 2.03GB > *Generated PDF No. of Pages:* Around 150 K > RAM: 12 GB > Peak memory that reached while generation - 11.3GB > Residual memory after forced GC: 9 GB > The FO mainly contains tabular data with each pages sequence having max of > 500 rows. > On analyzing the memory dump; found lots of reference for PDFPage, PDFName > etc. > *Question - * Is there any specific reason for using SoftReference in > PDFReference class instead of WeakReference. > Testing by changing SoftReference to WeakReference in PDFReference shows > following improvements without any issue in the generation whatsoever - > Stats for Generating 5 PDF in parallel - > *FO size:* 2.03GB > *Generated PDF No. of Pages:* Around 150 K > RAM: 12 GB > Peak memory that reached while generation - 4GB > Residual memory after forced GC: 300 MB > So, by changing SoftReference to WeakReference, I was able to generate 5 PDF > having 150K pages in parallel with max 4GB Ram; without any generation > issues. > You can clearly see the performance benefits of changing to WeakReference. > But as I dont understand the complete internal details of how FOP works, I > would like to understand if we can target this change and if not what is the > reason behind using SoftReference? -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FOP-2937) [PATCH]Post PDF generation, Soft reference of PDFObject in PDFReference are not immediately garbage collected leading to excessive memory usage.
[ https://issues.apache.org/jira/browse/FOP-2937?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Piyush Khandelwal updated FOP-2937: --- Attachment: PDFDictionary.patch > [PATCH]Post PDF generation, Soft reference of PDFObject in PDFReference are > not immediately garbage collected leading to excessive memory usage. > > > Key: FOP-2937 > URL: https://issues.apache.org/jira/browse/FOP-2937 > Project: FOP > Issue Type: Improvement >Affects Versions: 2.3, 2.4 >Reporter: Piyush Khandelwal >Priority: Major > Attachments: PDFDictionary.patch, pdfreference.patch > > > PDFReference object holds a SoftReference of PDFObject (PDFPage, PDFLabel, > PDFName etc.). > If we generate a huge PDF ; *I tried with a PDF having around 150 thousand > pages with 12 GB of RAM;* lots of these references linger around waiting for > the garbage collector to collect them. > But GC wont collect them as long as JVM is able to recover enough memory > without throwing out of memory. > Here are few metadata from my testing for further understanding of the issue > - > Stats for generating 1 PDF - > *FO size:* 2.03GB > *Generated PDF No. of Pages:* Around 150 K > RAM: 12 GB > Peak memory that reached while generation - 11.3GB > Residual memory after forced GC: 9 GB > The FO mainly contains tabular data with each pages sequence having max of > 500 rows. > On analyzing the memory dump; found lots of reference for PDFPage, PDFName > etc. > *Question - * Is there any specific reason for using SoftReference in > PDFReference class instead of WeakReference. > Testing by changing SoftReference to WeakReference in PDFReference shows > following improvements without any issue in the generation whatsoever - > Stats for Generating 5 PDF in parallel - > *FO size:* 2.03GB > *Generated PDF No. of Pages:* Around 150 K > RAM: 12 GB > Peak memory that reached while generation - 4GB > Residual memory after forced GC: 300 MB > So, by changing SoftReference to WeakReference, I was able to generate 5 PDF > having 150K pages in parallel with max 4GB Ram; without any generation > issues. > You can clearly see the performance benefits of changing to WeakReference. > But as I dont understand the complete internal details of how FOP works, I > would like to understand if we can target this change and if not what is the > reason behind using SoftReference? -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FOP-2937) [PATCH]Post PDF generation, Soft reference of PDFObject in PDFReference are not immediately garbage collected leading to excessive memory usage.
[ https://issues.apache.org/jira/browse/FOP-2937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17110107#comment-17110107 ] Piyush Khandelwal commented on FOP-2937: Apart from the change that I suggested in the description; I also changed entries HashMap in PDFDictionary class that is extended by PDFPage, PDFPageLabels to WeakHashMap - though this change in itself didn't bring much benefit but in combined to above some improvements was seen because of this well. Adding patch for that as well. > [PATCH]Post PDF generation, Soft reference of PDFObject in PDFReference are > not immediately garbage collected leading to excessive memory usage. > > > Key: FOP-2937 > URL: https://issues.apache.org/jira/browse/FOP-2937 > Project: FOP > Issue Type: Improvement >Affects Versions: 2.3, 2.4 >Reporter: Piyush Khandelwal >Priority: Major > Attachments: pdfreference.patch > > > PDFReference object holds a SoftReference of PDFObject (PDFPage, PDFLabel, > PDFName etc.). > If we generate a huge PDF ; *I tried with a PDF having around 150 thousand > pages with 12 GB of RAM;* lots of these references linger around waiting for > the garbage collector to collect them. > But GC wont collect them as long as JVM is able to recover enough memory > without throwing out of memory. > Here are few metadata from my testing for further understanding of the issue > - > Stats for generating 1 PDF - > *FO size:* 2.03GB > *Generated PDF No. of Pages:* Around 150 K > RAM: 12 GB > Peak memory that reached while generation - 11.3GB > Residual memory after forced GC: 9 GB > The FO mainly contains tabular data with each pages sequence having max of > 500 rows. > On analyzing the memory dump; found lots of reference for PDFPage, PDFName > etc. > *Question - * Is there any specific reason for using SoftReference in > PDFReference class instead of WeakReference. > Testing by changing SoftReference to WeakReference in PDFReference shows > following improvements without any issue in the generation whatsoever - > Stats for Generating 5 PDF in parallel - > *FO size:* 2.03GB > *Generated PDF No. of Pages:* Around 150 K > RAM: 12 GB > Peak memory that reached while generation - 4GB > Residual memory after forced GC: 300 MB > So, by changing SoftReference to WeakReference, I was able to generate 5 PDF > having 150K pages in parallel with max 4GB Ram; without any generation > issues. > You can clearly see the performance benefits of changing to WeakReference. > But as I dont understand the complete internal details of how FOP works, I > would like to understand if we can target this change and if not what is the > reason behind using SoftReference? -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FOP-2937) Post PDF generation, Soft reference of PDFObject in PDFReference are not immediately garbage collected leading to excessive memory usage.
[ https://issues.apache.org/jira/browse/FOP-2937?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Piyush Khandelwal updated FOP-2937: --- Attachment: pdfreference.patch > Post PDF generation, Soft reference of PDFObject in PDFReference are not > immediately garbage collected leading to excessive memory usage. > - > > Key: FOP-2937 > URL: https://issues.apache.org/jira/browse/FOP-2937 > Project: FOP > Issue Type: Improvement >Affects Versions: 2.3, 2.4 >Reporter: Piyush Khandelwal >Priority: Major > Attachments: pdfreference.patch > > > PDFReference object holds a SoftReference of PDFObject (PDFPage, PDFLabel, > PDFName etc.). > If we generate a huge PDF ; *I tried with a PDF having around 150 thousand > pages with 12 GB of RAM;* lots of these references linger around waiting for > the garbage collector to collect them. > But GC wont collect them as long as JVM is able to recover enough memory > without throwing out of memory. > Here are few metadata from my testing for further understanding of the issue > - > Stats for generating 1 PDF - > *FO size:* 2.03GB > *Generated PDF No. of Pages:* Around 150 K > RAM: 12 GB > Peak memory that reached while generation - 11.3GB > Residual memory after forced GC: 9 GB > The FO mainly contains tabular data with each pages sequence having max of > 500 rows. > On analyzing the memory dump; found lots of reference for PDFPage, PDFName > etc. > *Question - * Is there any specific reason for using SoftReference in > PDFReference class instead of WeakReference. > Testing by changing SoftReference to WeakReference in PDFReference shows > following improvements without any issue in the generation whatsoever - > Stats for Generating 5 PDF in parallel - > *FO size:* 2.03GB > *Generated PDF No. of Pages:* Around 150 K > RAM: 12 GB > Peak memory that reached while generation - 4GB > Residual memory after forced GC: 300 MB > So, by changing SoftReference to WeakReference, I was able to generate 5 PDF > having 150K pages in parallel with max 4GB Ram; without any generation > issues. > You can clearly see the performance benefits of changing to WeakReference. > But as I dont understand the complete internal details of how FOP works, I > would like to understand if we can target this change and if not what is the > reason behind using SoftReference? -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FOP-2937) [PATCH]Post PDF generation, Soft reference of PDFObject in PDFReference are not immediately garbage collected leading to excessive memory usage.
[ https://issues.apache.org/jira/browse/FOP-2937?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Piyush Khandelwal updated FOP-2937: --- Summary: [PATCH]Post PDF generation, Soft reference of PDFObject in PDFReference are not immediately garbage collected leading to excessive memory usage. (was: Post PDF generation, Soft reference of PDFObject in PDFReference are not immediately garbage collected leading to excessive memory usage.) > [PATCH]Post PDF generation, Soft reference of PDFObject in PDFReference are > not immediately garbage collected leading to excessive memory usage. > > > Key: FOP-2937 > URL: https://issues.apache.org/jira/browse/FOP-2937 > Project: FOP > Issue Type: Improvement >Affects Versions: 2.3, 2.4 >Reporter: Piyush Khandelwal >Priority: Major > Attachments: pdfreference.patch > > > PDFReference object holds a SoftReference of PDFObject (PDFPage, PDFLabel, > PDFName etc.). > If we generate a huge PDF ; *I tried with a PDF having around 150 thousand > pages with 12 GB of RAM;* lots of these references linger around waiting for > the garbage collector to collect them. > But GC wont collect them as long as JVM is able to recover enough memory > without throwing out of memory. > Here are few metadata from my testing for further understanding of the issue > - > Stats for generating 1 PDF - > *FO size:* 2.03GB > *Generated PDF No. of Pages:* Around 150 K > RAM: 12 GB > Peak memory that reached while generation - 11.3GB > Residual memory after forced GC: 9 GB > The FO mainly contains tabular data with each pages sequence having max of > 500 rows. > On analyzing the memory dump; found lots of reference for PDFPage, PDFName > etc. > *Question - * Is there any specific reason for using SoftReference in > PDFReference class instead of WeakReference. > Testing by changing SoftReference to WeakReference in PDFReference shows > following improvements without any issue in the generation whatsoever - > Stats for Generating 5 PDF in parallel - > *FO size:* 2.03GB > *Generated PDF No. of Pages:* Around 150 K > RAM: 12 GB > Peak memory that reached while generation - 4GB > Residual memory after forced GC: 300 MB > So, by changing SoftReference to WeakReference, I was able to generate 5 PDF > having 150K pages in parallel with max 4GB Ram; without any generation > issues. > You can clearly see the performance benefits of changing to WeakReference. > But as I dont understand the complete internal details of how FOP works, I > would like to understand if we can target this change and if not what is the > reason behind using SoftReference? -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FOP-2938) Incomplete TTF subset files for mutibyte fonts - incomplete mapping between CIDs and codepoints when having two codepoints associated to the same glyph
[ https://issues.apache.org/jira/browse/FOP-2938?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dan Caprioara updated FOP-2938: --- Summary: Incomplete TTF subset files for mutibyte fonts - incomplete mapping between CIDs and codepoints when having two codepoints associated to the same glyph (was: Incomplete TTF subset files - incomplete mapping between CIDs and codepoints when having two codepoints associated to the same glyph) > Incomplete TTF subset files for mutibyte fonts - incomplete mapping between > CIDs and codepoints when having two codepoints associated to the same glyph > --- > > Key: FOP-2938 > URL: https://issues.apache.org/jira/browse/FOP-2938 > Project: FOP > Issue Type: Bug > Components: font/opentype >Affects Versions: 2.5 >Reporter: Dan Caprioara >Priority: Major > Attachments: CIDFonts Subset Architectural Problem.png > > > A generic problem appears for MultiByte CID fonts. > Because of a limitation of the CIDSubset implemenatation, once a glyph has > been used for a code point, next usages of the same glyph for other > codepoints are discarded. This results in incomplete tables in the TTF sub > set file. The structures from CIDSubset class are maps, limiting > expressiveness. > > In the next diagram I used hyphens and soft hyphens, just as an example, any > text with two different characters that map to the same glyph is affected. > See this attached sketch for details: > !CIDFonts Subset Architectural Problem.png! > The solution would be to avoid usage of hash maps, and create some > associative lists (currently the uniqueness of the keys breaks things). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FOP-2938) Incomplete TTF subset files - incomplete mapping between CIDs and codepoints when having two codepoints associated to the same glyph
Dan Caprioara created FOP-2938: -- Summary: Incomplete TTF subset files - incomplete mapping between CIDs and codepoints when having two codepoints associated to the same glyph Key: FOP-2938 URL: https://issues.apache.org/jira/browse/FOP-2938 Project: FOP Issue Type: Bug Components: font/opentype Affects Versions: 2.5 Reporter: Dan Caprioara Attachments: CIDFonts Subset Architectural Problem.png A generic problem appears for MultiByte CID fonts. Because of a limitation of the CIDSubset implemenatation, once a glyph has been used for a code point, next usages of the same glyph for other codepoints are discarded. This results in incomplete tables in the TTF sub set file. The structures from CIDSubset class are maps, limiting expressiveness. In the next diagram I used hyphens and soft hyphens, just as an example, any text with two different characters that map to the same glyph is affected. See this attached sketch for details: !CIDFonts Subset Architectural Problem.png! The solution would be to avoid usage of hash maps, and create some associative lists (currently the uniqueness of the keys breaks things). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FOP-2937) Post PDF generation, Soft reference of PDFObject in PDFReference are not immediately garbage collected leading to excessive memory usage.
[ https://issues.apache.org/jira/browse/FOP-2937?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Piyush Khandelwal updated FOP-2937: --- Description: PDFReference object holds a SoftReference of PDFObject (PDFPage, PDFLabel, PDFName etc.). If we generate a huge PDF ; *I tried with a PDF having around 150 thousand pages with 12 GB of RAM;* lots of these references linger around waiting for the garbage collector to collect them. But GC wont collect them as long as JVM is able to recover enough memory without throwing out of memory. Here are few metadata from my testing for further understanding of the issue - Stats for generating 1 PDF - *FO size:* 2.03GB *Generated PDF No. of Pages:* Around 150 K RAM: 12 GB Peak memory that reached while generation - 11.3GB Residual memory after forced GC: 9 GB The FO mainly contains tabular data with each pages sequence having max of 500 rows. On analyzing the memory dump; found lots of reference for PDFPage, PDFName etc. *Question - * Is there any specific reason for using SoftReference in PDFReference class instead of WeakReference. Testing by changing SoftReference to WeakReference in PDFReference shows following improvements without any issue in the generation whatsoever - Stats for Generating 5 PDF in parallel - *FO size:* 2.03GB *Generated PDF No. of Pages:* Around 150 K RAM: 12 GB Peak memory that reached while generation - 4GB Residual memory after forced GC: 300 MB So, by changing SoftReference to WeakReference, I was able to generate 5 PDF having 150K pages in parallel with max 4GB Ram; without any generation issues. You can clearly see the performance benefits of changing to WeakReference. But as I dont understand the complete internal details of how FOP works, I would like to understand if we can target this change and if not what is the reason behind using SoftReference? was: PDFReference object holds a SoftReference of PDFObject (PDFPage, PDFLabel, PDFName etc.). If we generate a huge PDF ; *I tried with a PDF having around 150 thousand pages with 12 GB of RAM;* lots of these references linger around waiting for the garbage collector to collect them. But GC wont collect them as long as JVM is able to recover enough memory without throwing out of memory. Here are few metadata from my testing for further understanding of the issue - Stats for generating 1 PDF - *FO size:* 2.03GB *Generated PDF No. of Pages:* Around 150 K RAM: 12 GB Peak memory that reached while generation - 11.3GB Residual memory after forced GC: 9 GB The FO mainly contains tabular data with each pages sequence having max of 500 rows. On analyzing the memory dump; found lots of reference for PDFPage, PDFName etc. *Question - * Is there any specific reason for using SoftReference in PDFReference class instead of WeakReference. Testing by changing SoftReference to WeakReference in PDFReference shows following improvements without any issue in the generation whatsoever - Stats for Generating 5 PDF in parallel - *FO size:* 2.03GB *Generated PDF No. of Pages:* Around 150 K RAM: 12 GB Peak memory that reached while generation - 4GB Residual memory after forced GC: 300 MB So, by changing SoftReference to WeakReference, I was able to generate 5 PDF having 150K pages in parallel with max 4GB Ram; without any generation issues. *Question- * Is there any specific reason for using SoftReference in PDFReference class instead of WeakReference? You can clearly see the performance benefits of changing to WeakReference. But as I dont understand the complete internal details of how FOP works, I would like to understand if we can target this change and if not what is the reason behind using SoftReference? > Post PDF generation, Soft reference of PDFObject in PDFReference are not > immediately garbage collected leading to excessive memory usage. > - > > Key: FOP-2937 > URL: https://issues.apache.org/jira/browse/FOP-2937 > Project: FOP > Issue Type: Improvement >Affects Versions: 2.3, 2.4 >Reporter: Piyush Khandelwal >Priority: Major > > PDFReference object holds a SoftReference of PDFObject (PDFPage, PDFLabel, > PDFName etc.). > If we generate a huge PDF ; *I tried with a PDF having around 150 thousand > pages with 12 GB of RAM;* lots of these references linger around waiting for > the garbage collector to collect them. > But GC wont collect them as long as JVM is able to recover enough memory > without throwing out of memory. > Here are few metadata from my testing for further understanding of the issue > - > Stats for generating 1 PDF - > *FO size:* 2.03GB > *Generated PDF No. of Pages:* Around 150 K > RAM: 12 GB > Peak memory that reached while generation
[jira] [Updated] (FOP-2937) Post PDF generation, Soft reference of PDFObject in PDFReference are not immediately garbage collected leading to excessive memory usage.
[ https://issues.apache.org/jira/browse/FOP-2937?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Piyush Khandelwal updated FOP-2937: --- Description: PDFReference object holds a SoftReference of PDFObject (PDFPage, PDFLabel, PDFName etc.). If we generate a huge PDF ; *I tried with a PDF having around 150 thousand pages with 12 GB of RAM;* lots of these references linger around waiting for the garbage collector to collect them. But GC wont collect them as long as JVM is able to recover enough memory without throwing out of memory. Here are few metadata from my testing for further understanding of the issue - Stats for generating 1 PDF - *FO size:* 2.03GB *Generated PDF No. of Pages:* Around 150 K RAM: 12 GB Peak memory that reached while generation - 11.3GB Residual memory after forced GC: 9 GB The FO mainly contains tabular data with each pages sequence having max of 500 rows. On analyzing the memory dump; found lots of reference for PDFPage, PDFName etc. *Question - * Is there any specific reason for using SoftReference in PDFReference class instead of WeakReference. Testing by changing SoftReference to WeakReference in PDFReference shows following improvements without any issue in the generation whatsoever - Stats for Generating 5 PDF in parallel - *FO size:* 2.03GB *Generated PDF No. of Pages:* Around 150 K RAM: 12 GB Peak memory that reached while generation - 4GB Residual memory after forced GC: 300 MB So, by changing SoftReference to WeakReference, I was able to generate 5 PDF having 150K pages in parallel with max 4GB Ram; without any generation issues. *Question- * Is there any specific reason for using SoftReference in PDFReference class instead of WeakReference? You can clearly see the performance benefits of changing to WeakReference. But as I dont understand the complete internal details of how FOP works, I would like to understand if we can target this change and if not what is the reason behind using SoftReference? was: PDFReference object holds a SoftReference of PDFObject (PDFPage, PDFLabel, PDFName etc.). If we generate a huge PDF ; *I tried with a PDF having around 150 thousand pages with 12 GB of RAM;* lots of these references linger around waiting for the garbage collector to collect them. But GC wont collect them as long as JVM is able to recover enough memory without throwing out of memory. Here are few metadata from my testing for further understanding of the issue - Stats for generating 1 PDF - *FO size:* 2.03GB *Generated PDF No. of Pages:* Around 150 K RAM: 12 GB Peak memory that reached while generation - 11.3GB Residual memory after forced GC: 9 GB The FO mainly contains tabular data with each pages sequence having max of 500 rows. On analyzing the memory dump; found lots of reference for PDFPage, PDFName etc. *Question - * Is there any specific reason for using SoftReference in PDFReference class instead of WeakReference. Testing by changing SoftReference to WeakReference in PDFReference shows following improvements without any issue in the generation whatsoever - Stats for Generating 5 PDF in parallel - *FO size:* 2.03GB *Generated PDF No. of Pages:* Around 150 K RAM: 12 GB Peak memory that reached while generation - 4GB Residual memory after forced GC: 300 MB So, by changing SoftReference to WeakReference, I was able to generate 5 PDF having 150K pages in parallel with max 4GB Ram; without any generation issues. *Question- * Is there any specific reason for using SoftReference in PDFReference class instead of WeakReference? You can clearly see the performance benefits of changing to WeakReference. But as I dont understand the complete internal details of FOP works, I would like to understand if we can target this change and if not what is the reason of using SoftReference? > Post PDF generation, Soft reference of PDFObject in PDFReference are not > immediately garbage collected leading to excessive memory usage. > - > > Key: FOP-2937 > URL: https://issues.apache.org/jira/browse/FOP-2937 > Project: FOP > Issue Type: Improvement >Affects Versions: 2.3, 2.4 >Reporter: Piyush Khandelwal >Priority: Major > > PDFReference object holds a SoftReference of PDFObject (PDFPage, PDFLabel, > PDFName etc.). > If we generate a huge PDF ; *I tried with a PDF having around 150 thousand > pages with 12 GB of RAM;* lots of these references linger around waiting for > the garbage collector to collect them. > But GC wont collect them as long as JVM is able to recover enough memory > without throwing out of memory. > Here are few metadata from my testing for further understanding of the issue > - > Stats for generating 1 PDF - > *FO size:*
[jira] [Created] (FOP-2937) Post PDF generation, Soft reference of PDFObject in PDFReference are not immediately garbage collected leading to excessive memory usage.
Piyush Khandelwal created FOP-2937: -- Summary: Post PDF generation, Soft reference of PDFObject in PDFReference are not immediately garbage collected leading to excessive memory usage. Key: FOP-2937 URL: https://issues.apache.org/jira/browse/FOP-2937 Project: FOP Issue Type: Improvement Affects Versions: 2.4, 2.3 Reporter: Piyush Khandelwal PDFReference object holds a SoftReference of PDFObject (PDFPage, PDFLabel, PDFName etc.). If we generate a huge PDF ; *I tried with a PDF having around 150 thousand pages with 12 GB of RAM;* lots of these references linger around waiting for the garbage collector to collect them. But GC wont collect them as long as JVM is able to recover enough memory without throwing out of memory. Here are few metadata from my testing for further understanding of the issue - Stats for generating 1 PDF - *FO size:* 2.03GB *Generated PDF No. of Pages:* Around 150 K RAM: 12 GB Peak memory that reached while generation - 11.3GB Residual memory after forced GC: 9 GB The FO mainly contains tabular data with each pages sequence having max of 500 rows. On analyzing the memory dump; found lots of reference for PDFPage, PDFName etc. *Question - * Is there any specific reason for using SoftReference in PDFReference class instead of WeakReference. Testing by changing SoftReference to WeakReference in PDFReference shows following improvements without any issue in the generation whatsoever - Stats for Generating 5 PDF in parallel - *FO size:* 2.03GB *Generated PDF No. of Pages:* Around 150 K RAM: 12 GB Peak memory that reached while generation - 4GB Residual memory after forced GC: 300 MB So, by changing SoftReference to WeakReference, I was able to generate 5 PDF having 150K pages in parallel with max 4GB Ram; without any generation issues. *Question- * Is there any specific reason for using SoftReference in PDFReference class instead of WeakReference? You can clearly see the performance benefits of changing to WeakReference. But as I dont understand the complete internal details of FOP works, I would like to understand if we can target this change and if not what is the reason of using SoftReference? -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FOP-2880) [PATCH] Hyphenated words are not searchable in readers (when accessibilty is turned on)
[
https://issues.apache.org/jira/browse/FOP-2880?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17110016#comment-17110016
]
Dan Caprioara commented on FOP-2880:
Yes, it is a hack. How can we proceed further to get the glyphlist.xml and
encoding.xml updated?
> [PATCH] Hyphenated words are not searchable in readers (when accessibilty is
> turned on)
> ---
>
> Key: FOP-2880
> URL: https://issues.apache.org/jira/browse/FOP-2880
> Project: FOP
> Issue Type: Bug
> Components: unqualified
>Affects Versions: 2.3
>Reporter: Dan Caprioara
>Priority: Major
> Original Estimate: 1h
> Remaining Estimate: 1h
>
> The hyphenated words are rendered by FOP using the hard hyphen character.
> This contradicts the PDF specification, where in section:
> 14.8.2.2.3 Incidental Artifacts
> clearly states that the SHY soft hyphen U+00AD character should be used.
> The effect is that the hyphenated words are not searchable, and the
> copy/paste feature includes also the hard hyphens, instead of removing them
> and joining the words pieces together.
> Here is a small patch that can be applied on the FOP core project in order to
> fix this - this is more like a proof of concept, the real fix would be to
> change the default hyphenation character * in the FOProppertyMapping and to
> change the font mappings: to remove the replacement of the SHY with the
> HYPHEN, see the org.apache.fop.fonts.CodePointMapping.encStandardEncoding,
> and org.apache.fop.fonts.CodePointMapping.encISOLatin1Encoding
> {code}
>0xad, 0x002D, // hyphen
>0xad, 0x00AD, // hyphen
> {code}
> The patch:
> {code}
> Index: src/main/java/org/apache/fop/fo/properties/CommonHyphenation.java
> ===
> --- src/main/java/org/apache/fop/fo/properties/CommonHyphenation.java
> (revision 191037)
> +++ src/main/java/org/apache/fop/fo/properties/CommonHyphenation.java
> (working copy)
> @@ -15,7 +15,7 @@
> * limitations under the License.
> */
>
> -/* $Id$ */
> +/* $Id: CommonHyphenation.java 1610839 2014-07-15 20:25:58Z vhennebert $ */
>
> package org.apache.fop.fo.properties;
>
> @@ -184,6 +184,16 @@
> */
> public int getHyphIPD(org.apache.fop.fonts.Font font) {
> char hyphChar = getHyphChar(font);
> +
> +if (hyphChar == '\u00ad') {
> + // Bizarre fix, defining the SHY as default hyphenation character
> in the FOPropertyMapping, leads
> + // to hard hyphens not selectable in the PDF reader.
> + //
> + // Mapping also the hard hyphen makes the character selectable!
> + font.mapChar('\u002d');
> +}
> +
> +
> return font.getCharWidth(hyphChar);
> }
>
> Index: src/main/java/org/apache/fop/fo/FOPropertyMapping.java
> ===
> --- src/main/java/org/apache/fop/fo/FOPropertyMapping.java(revision
> 190759)
> +++ src/main/java/org/apache/fop/fo/FOPropertyMapping.java(working copy)
> @@ -1106,7 +1106,10 @@
> // hyphenation-character
> m = new CharacterProperty.Maker(PR_HYPHENATION_CHARACTER);
> m.setInherited(true);
> -m.setDefault("-");
> +//
> +// m.setDefault("-");
> +m.setDefault("\u00ad");
> +
> addPropertyMaker("hyphenation-character", m);
>
> // hyphenation-push-character-count
> Index: src/main/java/org/apache/fop/render/pdf/PDFPainter.java
> ===
> --- src/main/java/org/apache/fop/render/pdf/PDFPainter.java (revision
> 190759)
> +++ src/main/java/org/apache/fop/render/pdf/PDFPainter.java (working copy)
> @@ -420,7 +420,8 @@
> PDFStructElem structElem = (PDFStructElem)
> getContext().getStructureTreeElement();
> languageAvailabilityChecker.checkLanguageAvailability(text);
> MarkedContentInfo mci =
> logicalStructureHandler.addTextContentItem(structElem);
> -String actualText = getContext().isHyphenated() ?
> text.substring(0, text.length() - 1) : null;
> +//String actualText = getContext().isHyphenated() ?
> text.substring(0, text.length() - 1) : null;
> +String actualText = null;
> generator.endTextObject();
> generator.updateColor(state.getTextColor(), true, null);
> generator.beginTextObject(mci.tag, mci.mcid, actualText);
> @@ -490,6 +491,15 @@
> float glyphAdjust = 0;
> if (font.hasCodePoint(orgChar)) {
> ch = font.mapCodePoint(orgChar);
> + if (orgChar ==
>
