[jira] [Updated] (NIFI-6964) Use compression level for xz-lzma2 format of the CompressContent processor
[ https://issues.apache.org/jira/browse/NIFI-6964?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joe Witt updated NIFI-6964: --- Resolution: Fixed Status: Resolved (was: Patch Available) I added docs to clarify memory concern. Tested various scenarios. Updated migration guide. Built on java 8 and 11. Thanks for the contrib! +1 merged to master > Use compression level for xz-lzma2 format of the CompressContent processor > -- > > Key: NIFI-6964 > URL: https://issues.apache.org/jira/browse/NIFI-6964 > Project: Apache NiFi > Issue Type: Improvement > Components: Core Framework >Affects Versions: 1.10.0 >Reporter: John Pierce >Priority: Minor > Labels: compression, migration-guide, xz-lzma2 > Fix For: 1.11.0 > > Original Estimate: 4h > Time Spent: 0.5h > Remaining Estimate: 3.5h > > The CompressContent processor does not use the Compression Level property of > the processor except for when using the GZIP compression format. On the > contrary, the xz-lzma2 compression format defaults to using XZ compression > level 6 for that specific format (I read the CompressContent.java source code > to verify this) – disregarding whatever compression level you set on the > processor itself. > As a side note, the xz compression format supports, amazingly enough, 10 > levels of compression from 0 to 9 – the same as GZIP. The only difference > that I can tell is level 0 of xz is not the lack of compression, but the > lightest compression possible (i.e. still some compression) – whereas GZIP > compression level 0 means just container the content but do not compress. > I have a use case where I must use the xz-lzma2 format (don't ask why) and I > have to send (using the XZ format) already highly-compressed content that is > +*NOT*+ XZ format to begin with. I have in excess of 500 GB of this sort of > already highly compressed content to further compress into the XZ format on a > daily basis. > The attached patch will enhance the CompressContent.java source code enabling > the compression level property to be used in both the GZIP and the XZ-LZMA2 > formats. > Please consider adding this patch to the baseline for this processor. I've > tested it and the results are fantastic because I can crank down the > compression level to 0 for XZ-LZMA2 now and use a lot less CPU. I'm generally > seeing a 66% improvement in elapsed time to process highly compressed content > using XZ format with compression level of 0 versus the hard-coded level 6 of > the baseline code. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (NIFI-6964) Use compression level for xz-lzma2 format of the CompressContent processor
[ https://issues.apache.org/jira/browse/NIFI-6964?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joe Witt updated NIFI-6964: --- Labels: compression migration-guide xz-lzma2 (was: compression xz-lzma2) > Use compression level for xz-lzma2 format of the CompressContent processor > -- > > Key: NIFI-6964 > URL: https://issues.apache.org/jira/browse/NIFI-6964 > Project: Apache NiFi > Issue Type: Improvement > Components: Core Framework >Affects Versions: 1.10.0 >Reporter: John Pierce >Priority: Minor > Labels: compression, migration-guide, xz-lzma2 > Fix For: 1.11.0 > > Original Estimate: 4h > Time Spent: 10m > Remaining Estimate: 3h 50m > > The CompressContent processor does not use the Compression Level property of > the processor except for when using the GZIP compression format. On the > contrary, the xz-lzma2 compression format defaults to using XZ compression > level 6 for that specific format (I read the CompressContent.java source code > to verify this) – disregarding whatever compression level you set on the > processor itself. > As a side note, the xz compression format supports, amazingly enough, 10 > levels of compression from 0 to 9 – the same as GZIP. The only difference > that I can tell is level 0 of xz is not the lack of compression, but the > lightest compression possible (i.e. still some compression) – whereas GZIP > compression level 0 means just container the content but do not compress. > I have a use case where I must use the xz-lzma2 format (don't ask why) and I > have to send (using the XZ format) already highly-compressed content that is > +*NOT*+ XZ format to begin with. I have in excess of 500 GB of this sort of > already highly compressed content to further compress into the XZ format on a > daily basis. > The attached patch will enhance the CompressContent.java source code enabling > the compression level property to be used in both the GZIP and the XZ-LZMA2 > formats. > Please consider adding this patch to the baseline for this processor. I've > tested it and the results are fantastic because I can crank down the > compression level to 0 for XZ-LZMA2 now and use a lot less CPU. I'm generally > seeing a 66% improvement in elapsed time to process highly compressed content > using XZ format with compression level of 0 versus the hard-coded level 6 of > the baseline code. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (NIFI-6964) Use compression level for xz-lzma2 format of the CompressContent processor
[ https://issues.apache.org/jira/browse/NIFI-6964?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Pierce updated NIFI-6964: -- Description: The CompressContent processor does not use the Compression Level property of the processor except for when using the GZIP compression format. On the contrary, the xz-lzma2 compression format defaults to using XZ compression level 6 for that specific format (I read the CompressContent.java source code to verify this) – disregarding whatever compression level you set on the processor itself. As a side note, the xz compression format supports, amazingly enough, 10 levels of compression from 0 to 9 – the same as GZIP. The only difference that I can tell is level 0 of xz is not the lack of compression, but the lightest compression possible (i.e. still some compression) – whereas GZIP compression level 0 means just container the content but do not compress. I have a use case where I must use the xz-lzma2 format (don't ask why) and I have to send (using the XZ format) already highly-compressed content that is +*NOT*+ XZ format to begin with. I have in excess of 500 GB of this sort of already highly compressed content to further compress into the XZ format on a daily basis. The attached patch will enhance the CompressContent.java source code enabling the compression level property to be used in both the GZIP and the XZ-LZMA2 formats. Please consider adding this patch to the baseline for this processor. I've tested it and the results are fantastic because I can crank down the compression level to 0 for XZ-LZMA2 now and use a lot less CPU. I'm generally seeing a 66% improvement in elapsed time to process highly compressed content using XZ format with compression level of 0 versus the hard-coded level 6 of the baseline code. was: The CompressContent processor does not use the Compression Level property of the processor except for when using the GZIP compression format. On the contrary, the xz-lzma2 compression format defaults to using XZ compression level 6 for that specific format (I read the CompressContent.java source code to verify this) – disregarding whatever compression level you set on the processor itself. I have a use case where I must use the xz-lzma2 format (don't ask why) and I have to send (using the XZ format) already highly-compressed content that is +*NOT*+ XZ format to begin with. I have in excess of 500 GB of this sort of already highly compressed content to further compress into the XZ format on a daily basis. The attached patch will enhance the CompressContent.java source code enabling the compression level property to be used in both the GZIP and the XZ-LZMA2 formats. Please consider adding this patch to the baseline for this processor. I've tested it and the results are fantastic because I can crank down the compression level to 0 for XZ-LZMA2 now and use a lot less CPU. I'm generally seeing a 66% improvement in elapsed time to process highly compressed content using XZ format with compression level of 0 versus the hard-coded level 6 of the baseline code. Labels: compression xz-lzma2 (was: xz-lzma2) > Use compression level for xz-lzma2 format of the CompressContent processor > -- > > Key: NIFI-6964 > URL: https://issues.apache.org/jira/browse/NIFI-6964 > Project: Apache NiFi > Issue Type: Improvement > Components: Core Framework >Affects Versions: 1.10.0 >Reporter: John Pierce >Priority: Minor > Labels: compression, xz-lzma2 > Fix For: 1.11.0 > > Original Estimate: 4h > Time Spent: 10m > Remaining Estimate: 3h 50m > > The CompressContent processor does not use the Compression Level property of > the processor except for when using the GZIP compression format. On the > contrary, the xz-lzma2 compression format defaults to using XZ compression > level 6 for that specific format (I read the CompressContent.java source code > to verify this) – disregarding whatever compression level you set on the > processor itself. > As a side note, the xz compression format supports, amazingly enough, 10 > levels of compression from 0 to 9 – the same as GZIP. The only difference > that I can tell is level 0 of xz is not the lack of compression, but the > lightest compression possible (i.e. still some compression) – whereas GZIP > compression level 0 means just container the content but do not compress. > I have a use case where I must use the xz-lzma2 format (don't ask why) and I > have to send (using the XZ format) already highly-compressed content that is > +*NOT*+ XZ format to begin with. I have in excess of 500 GB of this sort of > already highly compressed content to further compress into the XZ format on a > daily basis. > The
[jira] [Updated] (NIFI-6964) Use compression level for xz-lzma2 format of the CompressContent processor
[ https://issues.apache.org/jira/browse/NIFI-6964?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joe Witt updated NIFI-6964: --- Fix Version/s: 1.11.0 > Use compression level for xz-lzma2 format of the CompressContent processor > -- > > Key: NIFI-6964 > URL: https://issues.apache.org/jira/browse/NIFI-6964 > Project: Apache NiFi > Issue Type: Improvement > Components: Core Framework >Affects Versions: 1.10.0 >Reporter: John Pierce >Priority: Minor > Labels: xz-lzma2 > Fix For: 1.11.0 > > Original Estimate: 4h > Time Spent: 10m > Remaining Estimate: 3h 50m > > The CompressContent processor does not use the Compression Level property of > the processor except for when using the GZIP compression format. On the > contrary, the xz-lzma2 compression format defaults to using XZ compression > level 6 for that specific format (I read the CompressContent.java source code > to verify this) – disregarding whatever compression level you set on the > processor itself. > I have a use case where I must use the xz-lzma2 format (don't ask why) and I > have to send (using the XZ format) already highly-compressed content that is > +*NOT*+ XZ format to begin with. I have in excess of 500 GB of this sort of > already highly compressed content to further compress into the XZ format on a > daily basis. > The attached patch will enhance the CompressContent.java source code enabling > the compression level property to be used in both the GZIP and the XZ-LZMA2 > formats. > Please consider adding this patch to the baseline for this processor. I've > tested it and the results are fantastic because I can crank down the > compression level to 0 for XZ-LZMA2 now and use a lot less CPU. I'm generally > seeing a 66% improvement in elapsed time to process highly compressed content > using XZ format with compression level of 0 versus the hard-coded level 6 of > the baseline code. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (NIFI-6964) Use compression level for xz-lzma2 format of the CompressContent processor
[ https://issues.apache.org/jira/browse/NIFI-6964?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Pierce updated NIFI-6964: -- Attachment: (was: CompressContent-enable-xz-compression-levels.patch) > Use compression level for xz-lzma2 format of the CompressContent processor > -- > > Key: NIFI-6964 > URL: https://issues.apache.org/jira/browse/NIFI-6964 > Project: Apache NiFi > Issue Type: Improvement > Components: Core Framework >Affects Versions: 1.10.0 >Reporter: John Pierce >Priority: Minor > Labels: xz-lzma2 > Original Estimate: 4h > Remaining Estimate: 4h > > The CompressContent processor does not use the Compression Level property of > the processor except for when using the GZIP compression format. On the > contrary, the xz-lzma2 compression format defaults to using XZ compression > level 6 for that specific format (I read the CompressContent.java source code > to verify this) – disregarding whatever compression level you set on the > processor itself. > I have a use case where I must use the xz-lzma2 format (don't ask why) and I > have to send (using the XZ format) already highly-compressed content that is > +*NOT*+ XZ format to begin with. I have in excess of 500 GB of this sort of > already highly compressed content to further compress into the XZ format on a > daily basis. > The attached patch will enhance the CompressContent.java source code enabling > the compression level property to be used in both the GZIP and the XZ-LZMA2 > formats. > Please consider adding this patch to the baseline for this processor. I've > tested it and the results are fantastic because I can crank down the > compression level to 0 for XZ-LZMA2 now and use a lot less CPU. I'm generally > seeing a 66% improvement in elapsed time to process highly compressed content > using XZ format with compression level of 0 versus the hard-coded level 6 of > the baseline code. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (NIFI-6964) Use compression level for xz-lzma2 format of the CompressContent processor
[ https://issues.apache.org/jira/browse/NIFI-6964?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Pierce updated NIFI-6964: -- Attachment: (was: CompressContent-enable-xz-compression-levels.patch) > Use compression level for xz-lzma2 format of the CompressContent processor > -- > > Key: NIFI-6964 > URL: https://issues.apache.org/jira/browse/NIFI-6964 > Project: Apache NiFi > Issue Type: Improvement > Components: Core Framework >Affects Versions: 1.10.0 >Reporter: John Pierce >Priority: Minor > Labels: xz-lzma2 > Attachments: CompressContent-enable-xz-compression-levels.patch > > Original Estimate: 4h > Remaining Estimate: 4h > > The CompressContent processor does not use the Compression Level property of > the processor except for when using the GZIP compression format. On the > contrary, the xz-lzma2 compression format defaults to using XZ compression > level 6 for that specific format (I read the CompressContent.java source code > to verify this) – disregarding whatever compression level you set on the > processor itself. > I have a use case where I must use the xz-lzma2 format (don't ask why) and I > have to send (using the XZ format) already highly-compressed content that is > +*NOT*+ XZ format to begin with. I have in excess of 500 GB of this sort of > already highly compressed content to further compress into the XZ format on a > daily basis. > The attached patch will enhance the CompressContent.java source code enabling > the compression level property to be used in both the GZIP and the XZ-LZMA2 > formats. > Please consider adding this patch to the baseline for this processor. I've > tested it and the results are fantastic because I can crank down the > compression level to 0 for XZ-LZMA2 now and use a lot less CPU. I'm generally > seeing a 66% improvement in elapsed time to process highly compressed content > using XZ format with compression level of 0 versus the hard-coded level 6 of > the baseline code. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (NIFI-6964) Use compression level for xz-lzma2 format of the CompressContent processor
[ https://issues.apache.org/jira/browse/NIFI-6964?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Pierce updated NIFI-6964: -- Attachment: CompressContent-enable-xz-compression-levels.patch Status: Patch Available (was: Open) > Use compression level for xz-lzma2 format of the CompressContent processor > -- > > Key: NIFI-6964 > URL: https://issues.apache.org/jira/browse/NIFI-6964 > Project: Apache NiFi > Issue Type: Improvement > Components: Core Framework >Affects Versions: 1.10.0 >Reporter: John Pierce >Priority: Minor > Labels: xz-lzma2 > Attachments: CompressContent-enable-xz-compression-levels.patch, > CompressContent-enable-xz-compression-levels.patch > > Original Estimate: 4h > Remaining Estimate: 4h > > The CompressContent processor does not use the Compression Level property of > the processor except for when using the GZIP compression format. On the > contrary, the xz-lzma2 compression format defaults to using XZ compression > level 6 for that specific format (I read the CompressContent.java source code > to verify this) – disregarding whatever compression level you set on the > processor itself. > I have a use case where I must use the xz-lzma2 format (don't ask why) and I > have to send (using the XZ format) already highly-compressed content that is > +*NOT*+ XZ format to begin with. I have in excess of 500 GB of this sort of > already highly compressed content to further compress into the XZ format on a > daily basis. > The attached patch will enhance the CompressContent.java source code enabling > the compression level property to be used in both the GZIP and the XZ-LZMA2 > formats. > Please consider adding this patch to the baseline for this processor. I've > tested it and the results are fantastic because I can crank down the > compression level to 0 for XZ-LZMA2 now and use a lot less CPU. I'm generally > seeing a 66% improvement in elapsed time to process highly compressed content > using XZ format with compression level of 0 versus the hard-coded level 6 of > the baseline code. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
