[jira] [Updated] (NIFI-5591) Enable compression of Avro in ExecuteSQL
[ https://issues.apache.org/jira/browse/NIFI-5591?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mike Thomsen updated NIFI-5591: --- Resolution: Fixed Fix Version/s: 1.8.0 Status: Resolved (was: Patch Available) > Enable compression of Avro in ExecuteSQL > > > Key: NIFI-5591 > URL: https://issues.apache.org/jira/browse/NIFI-5591 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Affects Versions: 1.7.1 > Environment: macOS, Java 8 >Reporter: Colin Dean >Assignee: Pierre Villard >Priority: Major > Labels: ExecuteSQL, avro, compression > Fix For: 1.8.0 > > > The Avro stream that comes out of the ExecuteSQL processor is uncompressed. > It's possible to rewrite it compressed using a combination of ConvertRecord > processor with AvroReader and AvroRecordSetWriter, but that's a lot of extra > I/O that could be handled transparently at the moment that the Avro data is > created. > For implementation, it looks like ExecuteSQL builds a set of > {{JdbcCommon.AvroConvertionOptions}}[here|https://github.com/apache/nifi/blob/ea9b0db2f620526c8dd0db595cf8b44c3ef835be/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/ExecuteSQL.java#L246]. > That options object would need to gain a compression flag. Then, within > {{JdbcCommon#convertToAvroStream}} > [here|https://github.com/apache/nifi/blob/0dd4a91a6741eec04965a260c8aff38b72b3828d/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/util/JdbcCommon.java#L281], > the {{dataFileWriter}} would get a codec set by {{setCodec}}, with the codec > having been created shortly before. > For example of creating the codec, I looked at how the AvroRecordSetWriter > does it. The {{setCodec()}} is performed > [here|https://github.com/apache/nifi/blob/ea9b0db2f620526c8dd0db595cf8b44c3ef835be/nifi-nar-bundles/nifi-standard-services/nifi-record-serialization-services-bundle/nifi-record-serialization-services/src/main/java/org/apache/nifi/avro/WriteAvroResultWithSchema.java#L44] > after the codec is created by factory option > [here|https://github.com/apache/nifi/blob/ea9b0db2f620526c8dd0db595cf8b44c3ef835be/nifi-nar-bundles/nifi-standard-services/nifi-record-serialization-services-bundle/nifi-record-serialization-services/src/main/java/org/apache/nifi/avro/AvroRecordSetWriter.java#L104] > using a factory method > [here|https://github.com/apache/nifi/blob/ea9b0db2f620526c8dd0db595cf8b44c3ef835be/nifi-nar-bundles/nifi-standard-services/nifi-record-serialization-services-bundle/nifi-record-serialization-services/src/main/java/org/apache/nifi/avro/AvroRecordSetWriter.java#L137]. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (NIFI-5591) Enable compression of Avro in ExecuteSQL
[ https://issues.apache.org/jira/browse/NIFI-5591?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pierre Villard updated NIFI-5591: - Component/s: (was: Core Framework) Extensions > Enable compression of Avro in ExecuteSQL > > > Key: NIFI-5591 > URL: https://issues.apache.org/jira/browse/NIFI-5591 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Affects Versions: 1.7.1 > Environment: macOS, Java 8 >Reporter: Colin Dean >Assignee: Pierre Villard >Priority: Major > Labels: ExecuteSQL, avro, compression > > The Avro stream that comes out of the ExecuteSQL processor is uncompressed. > It's possible to rewrite it compressed using a combination of ConvertRecord > processor with AvroReader and AvroRecordSetWriter, but that's a lot of extra > I/O that could be handled transparently at the moment that the Avro data is > created. > For implementation, it looks like ExecuteSQL builds a set of > {{JdbcCommon.AvroConvertionOptions}}[here|https://github.com/apache/nifi/blob/ea9b0db2f620526c8dd0db595cf8b44c3ef835be/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/ExecuteSQL.java#L246]. > That options object would need to gain a compression flag. Then, within > {{JdbcCommon#convertToAvroStream}} > [here|https://github.com/apache/nifi/blob/0dd4a91a6741eec04965a260c8aff38b72b3828d/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/util/JdbcCommon.java#L281], > the {{dataFileWriter}} would get a codec set by {{setCodec}}, with the codec > having been created shortly before. > For example of creating the codec, I looked at how the AvroRecordSetWriter > does it. The {{setCodec()}} is performed > [here|https://github.com/apache/nifi/blob/ea9b0db2f620526c8dd0db595cf8b44c3ef835be/nifi-nar-bundles/nifi-standard-services/nifi-record-serialization-services-bundle/nifi-record-serialization-services/src/main/java/org/apache/nifi/avro/WriteAvroResultWithSchema.java#L44] > after the codec is created by factory option > [here|https://github.com/apache/nifi/blob/ea9b0db2f620526c8dd0db595cf8b44c3ef835be/nifi-nar-bundles/nifi-standard-services/nifi-record-serialization-services-bundle/nifi-record-serialization-services/src/main/java/org/apache/nifi/avro/AvroRecordSetWriter.java#L104] > using a factory method > [here|https://github.com/apache/nifi/blob/ea9b0db2f620526c8dd0db595cf8b44c3ef835be/nifi-nar-bundles/nifi-standard-services/nifi-record-serialization-services-bundle/nifi-record-serialization-services/src/main/java/org/apache/nifi/avro/AvroRecordSetWriter.java#L137]. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (NIFI-5591) Enable compression of Avro in ExecuteSQL
[ https://issues.apache.org/jira/browse/NIFI-5591?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pierre Villard updated NIFI-5591: - Status: Patch Available (was: Open) > Enable compression of Avro in ExecuteSQL > > > Key: NIFI-5591 > URL: https://issues.apache.org/jira/browse/NIFI-5591 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Affects Versions: 1.7.1 > Environment: macOS, Java 8 >Reporter: Colin Dean >Assignee: Pierre Villard >Priority: Major > Labels: ExecuteSQL, avro, compression > > The Avro stream that comes out of the ExecuteSQL processor is uncompressed. > It's possible to rewrite it compressed using a combination of ConvertRecord > processor with AvroReader and AvroRecordSetWriter, but that's a lot of extra > I/O that could be handled transparently at the moment that the Avro data is > created. > For implementation, it looks like ExecuteSQL builds a set of > {{JdbcCommon.AvroConvertionOptions}}[here|https://github.com/apache/nifi/blob/ea9b0db2f620526c8dd0db595cf8b44c3ef835be/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/ExecuteSQL.java#L246]. > That options object would need to gain a compression flag. Then, within > {{JdbcCommon#convertToAvroStream}} > [here|https://github.com/apache/nifi/blob/0dd4a91a6741eec04965a260c8aff38b72b3828d/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/util/JdbcCommon.java#L281], > the {{dataFileWriter}} would get a codec set by {{setCodec}}, with the codec > having been created shortly before. > For example of creating the codec, I looked at how the AvroRecordSetWriter > does it. The {{setCodec()}} is performed > [here|https://github.com/apache/nifi/blob/ea9b0db2f620526c8dd0db595cf8b44c3ef835be/nifi-nar-bundles/nifi-standard-services/nifi-record-serialization-services-bundle/nifi-record-serialization-services/src/main/java/org/apache/nifi/avro/WriteAvroResultWithSchema.java#L44] > after the codec is created by factory option > [here|https://github.com/apache/nifi/blob/ea9b0db2f620526c8dd0db595cf8b44c3ef835be/nifi-nar-bundles/nifi-standard-services/nifi-record-serialization-services-bundle/nifi-record-serialization-services/src/main/java/org/apache/nifi/avro/AvroRecordSetWriter.java#L104] > using a factory method > [here|https://github.com/apache/nifi/blob/ea9b0db2f620526c8dd0db595cf8b44c3ef835be/nifi-nar-bundles/nifi-standard-services/nifi-record-serialization-services-bundle/nifi-record-serialization-services/src/main/java/org/apache/nifi/avro/AvroRecordSetWriter.java#L137]. -- This message was sent by Atlassian JIRA (v7.6.3#76005)