[jira] [Commented] (NIFI-3029) QueryDatabaseTable supports max fragments property
[ https://issues.apache.org/jira/browse/NIFI-3029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15748681#comment-15748681 ] ASF GitHub Bot commented on NIFI-3029: -- Github user mattyb149 commented on the issue: https://github.com/apache/nifi/pull/1213 +1 LGTM, ran unit tests and tried a NiFi flow with various settings for Max Rows Per Flow File and Max Number of Fragments. Thank you for the contribution! Merging to master > QueryDatabaseTable supports max fragments property > -- > > Key: NIFI-3029 > URL: https://issues.apache.org/jira/browse/NIFI-3029 > Project: Apache NiFi > Issue Type: Bug > Components: Extensions >Affects Versions: 1.1.0 >Reporter: Byunghwa Yun >Priority: Minor > Fix For: 1.2.0 > > > When QueryDatabaseTable ingests huge table that has ten billion data at the > first time, NiFi throws OutOfMemoryError. > Because QueryDatabaseTable creates too many fragments in memory event setting > the MaxRowsPerFlowFile property. > So I suggest QueryDatabaseTable supports max fragments property. > Thank you. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-3029) QueryDatabaseTable supports max fragments property
[ https://issues.apache.org/jira/browse/NIFI-3029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15748685#comment-15748685 ] ASF GitHub Bot commented on NIFI-3029: -- Github user asfgit closed the pull request at: https://github.com/apache/nifi/pull/1213 > QueryDatabaseTable supports max fragments property > -- > > Key: NIFI-3029 > URL: https://issues.apache.org/jira/browse/NIFI-3029 > Project: Apache NiFi > Issue Type: Bug > Components: Extensions >Affects Versions: 1.1.0 >Reporter: Byunghwa Yun >Priority: Minor > Fix For: 1.2.0 > > > When QueryDatabaseTable ingests huge table that has ten billion data at the > first time, NiFi throws OutOfMemoryError. > Because QueryDatabaseTable creates too many fragments in memory event setting > the MaxRowsPerFlowFile property. > So I suggest QueryDatabaseTable supports max fragments property. > Thank you. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-3029) QueryDatabaseTable supports max fragments property
[ https://issues.apache.org/jira/browse/NIFI-3029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15743683#comment-15743683 ] ASF GitHub Bot commented on NIFI-3029: -- Github user combineads commented on the issue: https://github.com/apache/nifi/pull/1213 @mattyb149 I changed this code that you reviewed. I hope to merge master as soon as possible. Thanks for your efforts. > QueryDatabaseTable supports max fragments property > -- > > Key: NIFI-3029 > URL: https://issues.apache.org/jira/browse/NIFI-3029 > Project: Apache NiFi > Issue Type: Bug > Components: Extensions >Affects Versions: 1.1.0 >Reporter: Byunghwa Yun >Priority: Minor > > When QueryDatabaseTable ingests huge table that has ten billion data at the > first time, NiFi throws OutOfMemoryError. > Because QueryDatabaseTable creates too many fragments in memory event setting > the MaxRowsPerFlowFile property. > So I suggest QueryDatabaseTable supports max fragments property. > Thank you. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-3029) QueryDatabaseTable supports max fragments property
[ https://issues.apache.org/jira/browse/NIFI-3029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15727285#comment-15727285 ] ASF GitHub Bot commented on NIFI-3029: -- Github user combineads commented on a diff in the pull request: https://github.com/apache/nifi/pull/1213#discussion_r91208324 --- Diff: nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/QueryDatabaseTable.java --- @@ -179,6 +189,9 @@ public void onTrigger(final ProcessContext context, final ProcessSessionFactory final String maxValueColumnNames = context.getProperty(MAX_VALUE_COLUMN_NAMES).getValue(); final Integer fetchSize = context.getProperty(FETCH_SIZE).asInteger(); final Integer maxRowsPerFlowFile = context.getProperty(MAX_ROWS_PER_FLOW_FILE).asInteger(); +final Integer maxFragments = context.getProperty(MAX_FRAGMENTS).isSet() +? context.getProperty(MAX_FRAGMENTS).asInteger() +: 0; --- End diff -- I changed the default value to zero and modified the description for zero means. Thank you for your the reviews. @mattyb149 > QueryDatabaseTable supports max fragments property > -- > > Key: NIFI-3029 > URL: https://issues.apache.org/jira/browse/NIFI-3029 > Project: Apache NiFi > Issue Type: Bug > Components: Extensions >Affects Versions: 1.1.0 >Reporter: Byunghwa Yun >Priority: Minor > > When QueryDatabaseTable ingests huge table that has ten billion data at the > first time, NiFi throws OutOfMemoryError. > Because QueryDatabaseTable creates too many fragments in memory event setting > the MaxRowsPerFlowFile property. > So I suggest QueryDatabaseTable supports max fragments property. > Thank you. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-3029) QueryDatabaseTable supports max fragments property
[ https://issues.apache.org/jira/browse/NIFI-3029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15725899#comment-15725899 ] ASF GitHub Bot commented on NIFI-3029: -- Github user mattyb149 commented on a diff in the pull request: https://github.com/apache/nifi/pull/1213#discussion_r91108610 --- Diff: nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/QueryDatabaseTable.java --- @@ -179,6 +189,9 @@ public void onTrigger(final ProcessContext context, final ProcessSessionFactory final String maxValueColumnNames = context.getProperty(MAX_VALUE_COLUMN_NAMES).getValue(); final Integer fetchSize = context.getProperty(FETCH_SIZE).asInteger(); final Integer maxRowsPerFlowFile = context.getProperty(MAX_ROWS_PER_FLOW_FILE).asInteger(); +final Integer maxFragments = context.getProperty(MAX_FRAGMENTS).isSet() +? context.getProperty(MAX_FRAGMENTS).asInteger() +: 0; --- End diff -- Is zero a valid number of fragments on its own (versus leaving the property blank)? If so then perhaps the "not set" value should be -1 and the logic below changed to >= 0. Otherwise perhaps the default value for the property should be zero and the description updated to reflect that zero means all fragments. > QueryDatabaseTable supports max fragments property > -- > > Key: NIFI-3029 > URL: https://issues.apache.org/jira/browse/NIFI-3029 > Project: Apache NiFi > Issue Type: Bug > Components: Extensions >Affects Versions: 1.1.0 >Reporter: Byunghwa Yun >Priority: Minor > > When QueryDatabaseTable ingests huge table that has ten billion data at the > first time, NiFi throws OutOfMemoryError. > Because QueryDatabaseTable creates too many fragments in memory event setting > the MaxRowsPerFlowFile property. > So I suggest QueryDatabaseTable supports max fragments property. > Thank you. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (NIFI-3029) QueryDatabaseTable supports max fragments property
[
https://issues.apache.org/jira/browse/NIFI-3029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15725887#comment-15725887
]
ASF GitHub Bot commented on NIFI-3029:
--
Github user mattyb149 commented on a diff in the pull request:
https://github.com/apache/nifi/pull/1213#discussion_r91107893
--- Diff:
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/QueryDatabaseTable.java
---
@@ -120,6 +120,15 @@
.addValidator(StandardValidators.NON_NEGATIVE_INTEGER_VALIDATOR)
.build();
+public static final PropertyDescriptor MAX_FRAGMENTS = new
PropertyDescriptor.Builder()
+.name("Maximum Number of Fragments")
+.description("The maximum number of fragments. If not
specified, there is no maximum. " +
+"This prevents OutOfMemoryError when this processor
ingests huge table.")
+.defaultValue("1000")
--- End diff --
Since this is a new property and it is optional, the default behavior
should be the same as if the property were not there (so the behavior does not
change for flows created before the property was available). I'm thinking the
default value should be empty.
> QueryDatabaseTable supports max fragments property
> --
>
> Key: NIFI-3029
> URL: https://issues.apache.org/jira/browse/NIFI-3029
> Project: Apache NiFi
> Issue Type: Bug
> Components: Extensions
>Affects Versions: 1.1.0
>Reporter: Byunghwa Yun
>Priority: Minor
>
> When QueryDatabaseTable ingests huge table that has ten billion data at the
> first time, NiFi throws OutOfMemoryError.
> Because QueryDatabaseTable creates too many fragments in memory event setting
> the MaxRowsPerFlowFile property.
> So I suggest QueryDatabaseTable supports max fragments property.
> Thank you.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
