[nifi] branch master updated (03473b9 -> 555004c)
This is an automated email from the ASF dual-hosted git repository. pwicks pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/nifi.git. from 03473b9 NIFI-6001: When adding Controller Services from a Versioned Flow, make a second iteration over any newly added Controller Services and update them again. This is done so that any linkage between the Controller Services happens properly, once all services have been created. add 555004c NIFI-6684 - Fixing checkstyle violation (#3788) No new revisions were added by this update. Summary of changes: .../src/main/java/org/apache/nifi/dbcp/hive/Hive3ConnectionPool.java | 1 - 1 file changed, 1 deletion(-)
[nifi] branch master updated (b12a9ad -> f1d35f4)
This is an automated email from the ASF dual-hosted git repository. pwicks pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/nifi.git. from b12a9ad NIFI-6159 - Add BigQuery processor using the Streaming API add f1d35f4 NIFI-6684 Add more property to Hive3ConnectionPool (#3763) No new revisions were added by this update. Summary of changes: .../nifi-hive-bundle/nifi-hive3-processors/pom.xml | 5 + .../apache/nifi/dbcp/hive/Hive3ConnectionPool.java | 129 - .../nifi/dbcp/hive/Hive3ConnectionPoolTest.java| 6 +- .../nifi-dbcp-service-api/pom.xml | 5 + .../java/org/apache/nifi/dbcp/DBCPValidator.java | 54 + .../org/apache/nifi/dbcp/DBCPConnectionPool.java | 42 +-- 6 files changed, 197 insertions(+), 44 deletions(-) create mode 100644 nifi-nar-bundles/nifi-standard-services/nifi-dbcp-service-api/src/main/java/org/apache/nifi/dbcp/DBCPValidator.java
[nifi-site] branch master updated: Update people.hbs
This is an automated email from the ASF dual-hosted git repository. pwicks pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/nifi-site.git The following commit(s) were added to refs/heads/master by this push: new daec96c Update people.hbs daec96c is described below commit daec96cf05073e4cc5cf3e31c7eeddd9a3531469 Author: Peter Wicks AuthorDate: Fri Sep 20 09:51:25 2019 -0600 Update people.hbs Moved myself from committer to PMC. --- src/pages/html/people.hbs | 10 +- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/src/pages/html/people.hbs b/src/pages/html/people.hbs index e0bb7c4..d6e1991 100644 --- a/src/pages/html/people.hbs +++ b/src/pages/html/people.hbs @@ -173,6 +173,11 @@ title: Apache NiFi Team kdoran Kevin Doran + + +pwicks +Peter Wicks + @@ -235,11 +240,6 @@ title: Apache NiFi Team -pwicks -Peter Wicks - - - aichrist Andrew Christianson
[nifi] branch master updated (9ec6314 -> 758035b)
This is an automated email from the ASF dual-hosted git repository. pwicks pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/nifi.git. from 9ec6314 NIFI-6642 - JsonPath support for adding array element add 758035b NIFI-6567 HandleHttpRequest does not shutdown HTTP server in some cir… (#3673) No new revisions were added by this update. Summary of changes: .../nifi/processors/standard/HandleHttpRequest.java| 18 ++ 1 file changed, 18 insertions(+)
[nifi] branch master updated: NIFI-6455 - Fix issue preventing scrolling all the way to the bottom of certain tables. (#3591)
This is an automated email from the ASF dual-hosted git repository.
pwicks pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/nifi.git
The following commit(s) were added to refs/heads/master by this push:
new 7dda04e NIFI-6455 - Fix issue preventing scrolling all the way to the
bottom of certain tables. (#3591)
7dda04e is described below
commit 7dda04e8951228992681538ef3cca4eb99fce026
Author: Rob Fellows
AuthorDate: Thu Jul 25 15:54:13 2019 -0400
NIFI-6455 - Fix issue preventing scrolling all the way to the bottom of
certain tables. (#3591)
This closes #3591.
Signed-off-by: Peter Wicks
---
.../nifi-framework/nifi-web/nifi-web-ui/src/main/frontend/package.json | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git
a/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-ui/src/main/frontend/package.json
b/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-ui/src/main/frontend/package.json
index 9a8ed11..b45e1a1 100644
---
a/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-ui/src/main/frontend/package.json
+++
b/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-ui/src/main/frontend/package.json
@@ -38,7 +38,7 @@
"url-search-params": "0.6.1",
"jsonlint": "1.6.2",
"qtip2": "3.0.3",
-"slickgrid": "2.3.3"
+"slickgrid": "2.3.23"
},
"repository": {
"type": "git",
[nifi] branch master updated: NIFI-6049: Updated MoveHDFS to support FF Attribute for Output Directory (#3433)
This is an automated email from the ASF dual-hosted git repository.
pwicks pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/nifi.git
The following commit(s) were added to refs/heads/master by this push:
new 94c2b1e NIFI-6049: Updated MoveHDFS to support FF Attribute for
Output Directory (#3433)
94c2b1e is described below
commit 94c2b1e76ebab33b6417fb46ce046e59c51fcdca
Author: Sivaprasanna
AuthorDate: Mon Apr 22 21:42:16 2019 +0530
NIFI-6049: Updated MoveHDFS to support FF Attribute for Output Directory
(#3433)
NIFI-6049: Updated MoveHDFS to support FF Attribute for Output Directory
---
.../java/org/apache/nifi/processors/hadoop/MoveHDFS.java | 16 +---
1 file changed, 5 insertions(+), 11 deletions(-)
diff --git
a/nifi-nar-bundles/nifi-hadoop-bundle/nifi-hdfs-processors/src/main/java/org/apache/nifi/processors/hadoop/MoveHDFS.java
b/nifi-nar-bundles/nifi-hadoop-bundle/nifi-hdfs-processors/src/main/java/org/apache/nifi/processors/hadoop/MoveHDFS.java
index a2292f5..95bcdeb 100644
---
a/nifi-nar-bundles/nifi-hadoop-bundle/nifi-hdfs-processors/src/main/java/org/apache/nifi/processors/hadoop/MoveHDFS.java
+++
b/nifi-nar-bundles/nifi-hadoop-bundle/nifi-hdfs-processors/src/main/java/org/apache/nifi/processors/hadoop/MoveHDFS.java
@@ -152,7 +152,7 @@ public class MoveHDFS extends AbstractHadoopProcessor {
.description("The HDFS directory where the files will be moved to")
.required(true)
.addValidator(StandardValidators.ATTRIBUTE_EXPRESSION_LANGUAGE_VALIDATOR)
-
.expressionLanguageSupported(ExpressionLanguageScope.VARIABLE_REGISTRY)
+
.expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
.build();
public static final PropertyDescriptor OPERATION = new
PropertyDescriptor.Builder()
@@ -243,7 +243,7 @@ public class MoveHDFS extends AbstractHadoopProcessor {
final FileSystem hdfs = getFileSystem();
final String filenameValue =
context.getProperty(INPUT_DIRECTORY_OR_FILE).evaluateAttributeExpressions(flowFile).getValue();
-Path inputPath = null;
+Path inputPath;
try {
inputPath = new Path(filenameValue);
if (!hdfs.exists(inputPath)) {
@@ -257,7 +257,7 @@ public class MoveHDFS extends AbstractHadoopProcessor {
return;
}
-List files = new ArrayList();
+List files = new ArrayList<>();
try {
final StopWatch stopWatch = new StopWatch(true);
@@ -348,7 +348,8 @@ public class MoveHDFS extends AbstractHadoopProcessor {
FlowFile flowFile = session.create(parentFlowFile);
try {
final String originalFilename = file.getName();
-final Path configuredRootOutputDirPath =
processorConfig.getOutputDirectory();
+final String outputDirValue =
context.getProperty(OUTPUT_DIRECTORY).evaluateAttributeExpressions(parentFlowFile).getValue();
+final Path configuredRootOutputDirPath = new
Path(outputDirValue);
final Path newFile = new
Path(configuredRootOutputDirPath, originalFilename);
final boolean destinationExists = hdfs.exists(newFile);
// If destination file already exists, resolve that
@@ -502,15 +503,12 @@ public class MoveHDFS extends AbstractHadoopProcessor {
final private String conflictResolution;
final private String operation;
-final private Path outputRootDirPath;
final private Pattern fileFilterPattern;
final private boolean ignoreDottedFiles;
ProcessorConfiguration(final ProcessContext context) {
conflictResolution =
context.getProperty(CONFLICT_RESOLUTION).getValue();
operation = context.getProperty(OPERATION).getValue();
-final String outputDirValue =
context.getProperty(OUTPUT_DIRECTORY).evaluateAttributeExpressions().getValue();
-outputRootDirPath = new Path(outputDirValue);
final String fileFilterRegex =
context.getProperty(FILE_FILTER_REGEX).getValue();
fileFilterPattern = (fileFilterRegex == null) ? null :
Pattern.compile(fileFilterRegex);
ignoreDottedFiles =
context.getProperty(IGNORE_DOTTED_FILES).asBoolean();
@@ -524,10 +522,6 @@ public class MoveHDFS extends AbstractHadoopProcessor {
return conflictResolution;
}
-public Path getOutputDirectory() {
-return outputRootDirPath;
-}
-
protected PathFilter getPathFilter(final Path dir) {
return new PathFilter() {
[nifi] branch master updated: NIFI-6221 Adding 'Download' and 'View' content buttons to List Queue (#3440)
This is an automated email from the ASF dual-hosted git repository.
pwicks pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/nifi.git
The following commit(s) were added to refs/heads/master by this push:
new 512555d NIFI-6221 Adding 'Download' and 'View' content buttons to
List Queue (#3440)
512555d is described below
commit 512555dbb0808d1b4c5b60a6478af0578a6c9abc
Author: aeaversa <36886905+aeave...@users.noreply.github.com>
AuthorDate: Fri Apr 19 18:25:15 2019 -0400
NIFI-6221 Adding 'Download' and 'View' content buttons to List Queue (#3440)
NIFI-6221 Adding 'Download' and 'View' content icon buttons to slick grid
actions column to improve end user navigation.
---
.../main/webapp/js/nf/canvas/nf-queue-listing.js | 67 +++---
1 file changed, 45 insertions(+), 22 deletions(-)
diff --git
a/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-ui/src/main/webapp/js/nf/canvas/nf-queue-listing.js
b/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-ui/src/main/webapp/js/nf/canvas/nf-queue-listing.js
index 79fef78..89131bd 100644
---
a/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-ui/src/main/webapp/js/nf/canvas/nf-queue-listing.js
+++
b/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-ui/src/main/webapp/js/nf/canvas/nf-queue-listing.js
@@ -204,32 +204,48 @@
});
}
-// add an actions column when the user can access provenance
-if (nfCommon.canAccessProvenance()) {
-// function for formatting actions
-var actionsFormatter = function () {
-return '';
-};
+// function for formatting actions column
+var actionsFormatter = function (row,cell,value,columnDef,dataContext)
{
+var formatted = '';
+
+var disabled = (dataContext.size > 0)?false:true;
+formatted += '';
+
+if(nfCommon.isContentViewConfigured()){
+formatted += '';
+}
+
+if(nfCommon.canAccessProvenance()){
+formatted += '';
+}
+
+return formatted;
+};
+
+// add an actions column to the column model
+queueListingColumns.push({
+id: 'actions',
+name: '',
+resizable: false,
+formatter: actionsFormatter,
+sortable: false,
+width: 75,
+maxWidth: 75
+});
-queueListingColumns.push({
-id: 'actions',
-name: '',
-resizable: false,
-formatter: actionsFormatter,
-sortable: false,
-width: 50,
-maxWidth: 50
-});
-}
return queueListingColumns;
};
/**
* Downloads the content for the flowfile currently being viewed.
+ *
+ * @param flowFileSummary|{flowfile} (optional) - the flowfile summary
*/
-var downloadContent = function () {
-var dataUri = $('#flowfile-uri').text() + '/content';
+var downloadContent = function (flowFileSummary) {
+var dataUri =
((nfCommon.isDefinedAndNotNull(flowFileSummary.uri))?flowFileSummary.uri:$('#flowfile-uri').text())+
'/content';
// perform the request once we've received a token
nfCommon.getAccessToken(config.urls.downloadToken).done(function
(downloadToken) {
@@ -241,7 +257,7 @@
}
// conditionally include the cluster node id
-var clusterNodeId = $('#flowfile-cluster-node-id').text();
+var clusterNodeId =
(nfCommon.isDefinedAndNotNull(flowFileSummary.clusterNodeId))?flowFileSummary.clusterNodeId:$('#flowfile-cluster-node-id').text();
if (!nfCommon.isBlank(clusterNodeId)) {
parameters['clusterNodeId'] = clusterNodeId;
}
@@ -262,9 +278,12 @@
/**
* Views the content for the flowfile currently being viewed.
+ *
+ * @param flowFileSummary|{flowfile} (optional) - the flowfile summary
*/
-var viewContent = function () {
-var dataUri = $('#flowfile-uri').text() + '/content';
+var viewContent = function (flowFileSummary) {
+
+var dataUri =
((nfCommon.isDefinedAndNotNull(flowFileSummary.uri))?flowFileSummary.uri:$('#flowfile-uri').text())+
'/content';
// generate tokens as necessary
var getAccessTokens = $.Deferred(function (deferred) {
@@ -301,7 +320,7 @@
var dataUriParameters = {};
// conditionally include the cluster node id
-var clusterNodeId = $('#flowfile-cluster-node-id').text();
+var clusterNodeId =
(nfCommon.isDefinedAndNotNull(flowFileSummary.clusterNodeId))?flowFileSummary.clusterNodeId:$('#flowfile-cluster-node-id').text();
if (!nfCommon.isBlank(clus
[nifi] branch master updated: NIFI-5722 Expose Penalty Remaining Duration (#3091)
This is an automated email from the ASF dual-hosted git repository.
pwicks pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/nifi.git
The following commit(s) were added to refs/heads/master by this push:
new ad1f2fb NIFI-5722 Expose Penalty Remaining Duration (#3091)
ad1f2fb is described below
commit ad1f2fb666a2d98b05c56119c1e1ae8cc5fd2e7d
Author: Peter Wicks
AuthorDate: Fri Feb 1 13:04:51 2019 -0700
NIFI-5722 Expose Penalty Remaining Duration (#3091)
Signed-off-by: Koji Kawamura
---
.../org/apache/nifi/controller/queue/FlowFileSummary.java | 5 +
.../org/apache/nifi/web/api/dto/FlowFileSummaryDTO.java | 15 +++
.../nifi/controller/queue/AbstractFlowFileQueue.java | 8 +++-
.../main/java/org/apache/nifi/web/api/dto/DtoFactory.java | 8
.../src/main/webapp/js/nf/canvas/nf-queue-listing.js | 12 +---
5 files changed, 40 insertions(+), 8 deletions(-)
diff --git
a/nifi-framework-api/src/main/java/org/apache/nifi/controller/queue/FlowFileSummary.java
b/nifi-framework-api/src/main/java/org/apache/nifi/controller/queue/FlowFileSummary.java
index b7207f2..a0dc413 100644
---
a/nifi-framework-api/src/main/java/org/apache/nifi/controller/queue/FlowFileSummary.java
+++
b/nifi-framework-api/src/main/java/org/apache/nifi/controller/queue/FlowFileSummary.java
@@ -56,4 +56,9 @@ public interface FlowFileSummary {
* @return true if the FlowFile is penalized,
false otherwise
*/
boolean isPenalized();
+
+/**
+ * @return the timestamp (in milliseconds since epoch) at which the
FlowFiles Penalty Expires
+ */
+long getPenaltyExpirationMillis();
}
diff --git
a/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-client-dto/src/main/java/org/apache/nifi/web/api/dto/FlowFileSummaryDTO.java
b/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-client-dto/src/main/java/org/apache/nifi/web/api/dto/FlowFileSummaryDTO.java
index 2113231..b6faba3 100644
---
a/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-client-dto/src/main/java/org/apache/nifi/web/api/dto/FlowFileSummaryDTO.java
+++
b/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-client-dto/src/main/java/org/apache/nifi/web/api/dto/FlowFileSummaryDTO.java
@@ -31,6 +31,7 @@ public class FlowFileSummaryDTO {
private Long size;
private Long queuedDuration;
private Long lineageDuration;
+private Long penaltyExpiresIn;
private Boolean isPenalized;
private String clusterNodeId; // include when clustered
@@ -135,6 +136,20 @@ public class FlowFileSummaryDTO {
}
/**
+ * @return when the FlowFile will no longer be penalized
+ */
+@ApiModelProperty(
+value = "How long in milliseconds until the FlowFile penalty
expires."
+)
+public Long getPenaltyExpiresIn() {
+return penaltyExpiresIn;
+}
+
+public void setPenaltyExpiresIn(Long penaltyExpiration) {
+penaltyExpiresIn = penaltyExpiration;
+}
+
+/**
* @return if the FlowFile is penalized
*/
@ApiModelProperty(
diff --git
a/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/queue/AbstractFlowFileQueue.java
b/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/queue/AbstractFlowFileQueue.java
index b986791..ec3c550 100644
---
a/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/queue/AbstractFlowFileQueue.java
+++
b/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/queue/AbstractFlowFileQueue.java
@@ -325,7 +325,7 @@ public abstract class AbstractFlowFileQueue implements
FlowFileQueue {
}
-protected FlowFileSummary summarize(final FlowFile flowFile, final int
position) {
+protected FlowFileSummary summarize(final FlowFileRecord flowFile, final
int position) {
// extract all of the information that we care about into new
variables rather than just
// wrapping the FlowFile object with a FlowFileSummary object. We do
this because we want to
// be able to hold many FlowFileSummary objects in memory and if we
just wrap the FlowFile object,
@@ -337,6 +337,7 @@ public abstract class AbstractFlowFileQueue implements
FlowFileQueue {
final Long lastQueuedTime = flowFile.getLastQueueDate();
final long lineageStart = flowFile.getLineageStartDate();
final boolean penalized = flowFile.isPenalized();
+final long penaltyExpires = flowFile.getPenaltyExpirationMillis();
return new FlowFileSummary() {
@Override
@@ -373,6 +374,11 @@ public abstract class AbstractFlowFileQueue implements
FlowFileQueue {
public boolean isPenalized() {
return
[nifi] branch master updated: NIFI-5790 removed the last test as it's causing a race condition intermittently (#3260)
This is an automated email from the ASF dual-hosted git repository.
pwicks pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/nifi.git
The following commit(s) were added to refs/heads/master by this push:
new cf7ab0c NIFI-5790 removed the last test as it's causing a race
condition intermittently (#3260)
cf7ab0c is described below
commit cf7ab0ce180ecfdeaf1758e896540b58036896b0
Author: SavtechSolutions
AuthorDate: Thu Jan 10 14:13:35 2019 -0500
NIFI-5790 removed the last test as it's causing a race condition
intermittently (#3260)
Signed-off-by: Peter Wicks
---
.../src/test/java/org/apache/nifi/dbcp/DBCPServiceTest.java| 3 ---
1 file changed, 3 deletions(-)
diff --git
a/nifi-nar-bundles/nifi-standard-services/nifi-dbcp-service-bundle/nifi-dbcp-service/src/test/java/org/apache/nifi/dbcp/DBCPServiceTest.java
b/nifi-nar-bundles/nifi-standard-services/nifi-dbcp-service-bundle/nifi-dbcp-service/src/test/java/org/apache/nifi/dbcp/DBCPServiceTest.java
index 4d32b33..2714e2e 100644
---
a/nifi-nar-bundles/nifi-standard-services/nifi-dbcp-service-bundle/nifi-dbcp-service/src/test/java/org/apache/nifi/dbcp/DBCPServiceTest.java
+++
b/nifi-nar-bundles/nifi-standard-services/nifi-dbcp-service-bundle/nifi-dbcp-service/src/test/java/org/apache/nifi/dbcp/DBCPServiceTest.java
@@ -237,9 +237,6 @@ public class DBCPServiceTest {
Assert.assertEquals(4, service.getDataSource().getNumIdle());
Assert.assertEquals(0, service.getDataSource().getNumActive());
-Thread.sleep(500);
-Assert.assertEquals(1, service.getDataSource().getNumIdle());
-
service.getDataSource().close();
}
[nifi] branch master updated: NIFI-5871 ignore UUID attribute when copying flow file attributes (#3203)
This is an automated email from the ASF dual-hosted git repository.
pwicks pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/nifi.git
The following commit(s) were added to refs/heads/master by this push:
new f9f386b NIFI-5871 ignore UUID attribute when copying flow file
attributes (#3203)
f9f386b is described below
commit f9f386b0f01fb7492ea6e9a56ca6cbc3d4578ad3
Author: SavtechSolutions
AuthorDate: Thu Dec 20 12:05:25 2018 -0500
NIFI-5871 ignore UUID attribute when copying flow file attributes (#3203)
NIFI-5871 ignore UUID attribute when copying flow file attributes
Signed-off-by: Peter Wicks
---
.../main/java/org/apache/nifi/util/MockProcessSession.java | 9 -
.../nifi/processors/couchbase/TestPutCouchbaseKey.java | 1 -
.../java/org/apache/nifi/processors/standard/TestWait.java | 12
3 files changed, 8 insertions(+), 14 deletions(-)
diff --git
a/nifi-mock/src/main/java/org/apache/nifi/util/MockProcessSession.java
b/nifi-mock/src/main/java/org/apache/nifi/util/MockProcessSession.java
index 1629e96..f48b919 100644
--- a/nifi-mock/src/main/java/org/apache/nifi/util/MockProcessSession.java
+++ b/nifi-mock/src/main/java/org/apache/nifi/util/MockProcessSession.java
@@ -496,7 +496,14 @@ public class MockProcessSession implements ProcessSession {
final MockFlowFile newFlowFile = new MockFlowFile(mock.getId(),
flowFile);
currentVersions.put(newFlowFile.getId(), newFlowFile);
-newFlowFile.putAttributes(attrs);
+final Map updatedAttributes;
+if (attrs.containsKey(CoreAttributes.UUID.key())) {
+updatedAttributes = new HashMap<>(attrs);
+updatedAttributes.remove(CoreAttributes.UUID.key());
+} else {
+updatedAttributes = attrs;
+}
+newFlowFile.putAttributes(updatedAttributes);
return newFlowFile;
}
diff --git
a/nifi-nar-bundles/nifi-couchbase-bundle/nifi-couchbase-processors/src/test/java/org/apache/nifi/processors/couchbase/TestPutCouchbaseKey.java
b/nifi-nar-bundles/nifi-couchbase-bundle/nifi-couchbase-processors/src/test/java/org/apache/nifi/processors/couchbase/TestPutCouchbaseKey.java
index ce9baa7..c32fff7 100644
---
a/nifi-nar-bundles/nifi-couchbase-bundle/nifi-couchbase-processors/src/test/java/org/apache/nifi/processors/couchbase/TestPutCouchbaseKey.java
+++
b/nifi-nar-bundles/nifi-couchbase-bundle/nifi-couchbase-processors/src/test/java/org/apache/nifi/processors/couchbase/TestPutCouchbaseKey.java
@@ -268,7 +268,6 @@ public class TestPutCouchbaseKey {
ArgumentCaptor capture =
ArgumentCaptor.forClass(RawJsonDocument.class);
verify(bucket, times(1)).upsert(capture.capture(), eq(PersistTo.NONE),
eq(ReplicateTo.NONE));
-assertEquals(uuid, capture.getValue().id());
assertEquals(inFileData, capture.getValue().content());
testRunner.assertTransferCount(REL_SUCCESS, 1);
diff --git
a/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/test/java/org/apache/nifi/processors/standard/TestWait.java
b/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/test/java/org/apache/nifi/processors/standard/TestWait.java
index a4df2f3..5b5b6fc 100644
---
a/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/test/java/org/apache/nifi/processors/standard/TestWait.java
+++
b/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/test/java/org/apache/nifi/processors/standard/TestWait.java
@@ -218,7 +218,6 @@ public class TestWait {
waitAttributes.put("releaseSignalAttribute", "key");
waitAttributes.put("wait.only", "waitValue");
waitAttributes.put("both", "waitValue");
-waitAttributes.put("uuid", UUID.randomUUID().toString());
String flowFileContent = "content";
runner.enqueue(flowFileContent.getBytes("UTF-8"), waitAttributes);
@@ -234,8 +233,6 @@ public class TestWait {
// show a new attribute was copied from the cache
assertEquals("notifyValue",
outputFlowFile.getAttribute("notify.only"));
-// show that uuid was not overwritten
-assertEquals(waitAttributes.get("uuid"),
outputFlowFile.getAttribute("uuid"));
// show that the original attributes are still there
assertEquals("waitValue", outputFlowFile.getAttribute("wait.only"));
@@ -265,7 +262,6 @@ public class TestWait {
waitAttributes.put("releaseSignalAttribute", "key");
waitAttributes.put("wait.only", "waitValue");
waitAttributes.put("both", "waitValue");
-waitAttributes.put("uuid", UUID.randomUUID().toString());
String flowFileContent = &q
nifi git commit: NIFI-5846: Redirect URL is incorrect after logout
Repository: nifi
Updated Branches:
refs/heads/master 30f2f4205 -> 72662f0b2
NIFI-5846: Redirect URL is incorrect after logout
This closes #3185.
Signed-off-by: Peter Wicks
Project: http://git-wip-us.apache.org/repos/asf/nifi/repo
Commit: http://git-wip-us.apache.org/repos/asf/nifi/commit/72662f0b
Tree: http://git-wip-us.apache.org/repos/asf/nifi/tree/72662f0b
Diff: http://git-wip-us.apache.org/repos/asf/nifi/diff/72662f0b
Branch: refs/heads/master
Commit: 72662f0b2c5ec5f088aade94337f7edce665ee7e
Parents: 30f2f42
Author: Kotaro Terada
Authored: Wed Nov 28 14:23:39 2018 +0900
Committer: Peter Wicks
Committed: Thu Nov 29 08:16:39 2018 -0700
--
.../src/main/java/org/apache/nifi/web/filter/LogoutFilter.java | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
--
http://git-wip-us.apache.org/repos/asf/nifi/blob/72662f0b/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-ui/src/main/java/org/apache/nifi/web/filter/LogoutFilter.java
--
diff --git
a/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-ui/src/main/java/org/apache/nifi/web/filter/LogoutFilter.java
b/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-ui/src/main/java/org/apache/nifi/web/filter/LogoutFilter.java
index 9039612..08f9365 100644
---
a/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-ui/src/main/java/org/apache/nifi/web/filter/LogoutFilter.java
+++
b/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-ui/src/main/java/org/apache/nifi/web/filter/LogoutFilter.java
@@ -50,7 +50,7 @@ public class LogoutFilter implements Filter {
final ServletContext apiContext =
servletContext.getContext("/nifi-api");
apiContext.getRequestDispatcher("/access/knox/logout").forward(request,
response);
} else {
-((HttpServletResponse) response).sendRedirect("../login");
+((HttpServletResponse) response).sendRedirect("login");
}
}
nifi git commit: NIFI-5744: Put exception message to attribute while ExecuteSQL fail
Repository: nifi
Updated Branches:
refs/heads/master 7bcf9fcb5 -> 3c7012ffd
NIFI-5744: Put exception message to attribute while ExecuteSQL fail
This closes #3107.
Signed-off-by: Peter Wicks
Project: http://git-wip-us.apache.org/repos/asf/nifi/repo
Commit: http://git-wip-us.apache.org/repos/asf/nifi/commit/3c7012ff
Tree: http://git-wip-us.apache.org/repos/asf/nifi/tree/3c7012ff
Diff: http://git-wip-us.apache.org/repos/asf/nifi/diff/3c7012ff
Branch: refs/heads/master
Commit: 3c7012ffda777b4692e4ea8db042ee1943a7a66a
Parents: 7bcf9fc
Author: yjhyjhyjh0
Authored: Wed Oct 24 21:22:06 2018 +0800
Committer: Peter Wicks
Committed: Tue Nov 20 09:00:46 2018 -0700
--
.../processors/standard/AbstractExecuteSQL.java | 2 +
.../nifi/processors/standard/ExecuteSQL.java| 2 +
.../processors/standard/ExecuteSQLRecord.java | 2 +
.../processors/standard/TestExecuteSQL.java | 7 +++-
.../standard/TestExecuteSQLRecord.java | 43
5 files changed, 55 insertions(+), 1 deletion(-)
--
http://git-wip-us.apache.org/repos/asf/nifi/blob/3c7012ff/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/AbstractExecuteSQL.java
--
diff --git
a/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/AbstractExecuteSQL.java
b/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/AbstractExecuteSQL.java
index d1fabef..76e36fd 100644
---
a/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/AbstractExecuteSQL.java
+++
b/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/AbstractExecuteSQL.java
@@ -61,6 +61,7 @@ public abstract class AbstractExecuteSQL extends
AbstractProcessor {
public static final String RESULT_QUERY_EXECUTION_TIME =
"executesql.query.executiontime";
public static final String RESULT_QUERY_FETCH_TIME =
"executesql.query.fetchtime";
public static final String RESULTSET_INDEX = "executesql.resultset.index";
+public static final String RESULT_ERROR_MESSAGE =
"executesql.error.message";
public static final String FRAGMENT_ID =
FragmentAttributes.FRAGMENT_ID.key();
public static final String FRAGMENT_INDEX =
FragmentAttributes.FRAGMENT_INDEX.key();
@@ -402,6 +403,7 @@ public abstract class AbstractExecuteSQL extends
AbstractProcessor {
new Object[]{selectQuery, e});
context.yield();
}
+
session.putAttribute(fileToProcess,RESULT_ERROR_MESSAGE,e.getMessage());
session.transfer(fileToProcess, REL_FAILURE);
}
}
http://git-wip-us.apache.org/repos/asf/nifi/blob/3c7012ff/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/ExecuteSQL.java
--
diff --git
a/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/ExecuteSQL.java
b/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/ExecuteSQL.java
index 99e0d2a..cfdef29 100644
---
a/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/ExecuteSQL.java
+++
b/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/ExecuteSQL.java
@@ -87,6 +87,8 @@ import static
org.apache.nifi.processors.standard.util.JdbcCommon.USE_AVRO_LOGIC
+ "If 'Max Rows Per Flow File' is set, then this number will
reflect only the fetch time for the rows in the Flow File instead of the entire
result set."),
@WritesAttribute(attribute = "executesql.resultset.index", description
= "Assuming multiple result sets are returned, "
+ "the zero based index of this result set."),
+@WritesAttribute(attribute = "executesql.error.message", description =
"If processing an incoming flow file causes "
++ "an Exception, the Flow File is routed to failure and this
attribute is set to the exception message."),
@WritesAttribute(attribute = "fragment.identifier", description = "If
'Max Rows Per Flow File' is set then all FlowFiles from the same query result
set "
+ "will have the same value for the fragment.identifier
attribute. This can then be used to correlate the results."),
@WritesAttribute(attribute = "fragment.count", description = "If 'Max
Rows Per
[nifi] Git Push Summary
Repository: nifi Updated Branches: refs/heads/NIFI-5744_a [deleted] 9358a60d3
nifi git commit: NIFI-5744: Put exception message to attribute while ExecuteSQL fail
Repository: nifi
Updated Branches:
refs/heads/NIFI-5744_a [created] 9358a60d3
NIFI-5744: Put exception message to attribute while ExecuteSQL fail
This closes #3107.
Signed-off-by: Peter Wicks
Project: http://git-wip-us.apache.org/repos/asf/nifi/repo
Commit: http://git-wip-us.apache.org/repos/asf/nifi/commit/9358a60d
Tree: http://git-wip-us.apache.org/repos/asf/nifi/tree/9358a60d
Diff: http://git-wip-us.apache.org/repos/asf/nifi/diff/9358a60d
Branch: refs/heads/NIFI-5744_a
Commit: 9358a60d33be966233d80b00c07928ccb17c4059
Parents: 7bcf9fc
Author: yjhyjhyjh0
Authored: Wed Oct 24 21:22:06 2018 +0800
Committer: Peter Wicks
Committed: Tue Nov 20 08:56:15 2018 -0700
--
.../processors/standard/AbstractExecuteSQL.java | 2 +
.../nifi/processors/standard/ExecuteSQL.java| 2 +
.../processors/standard/ExecuteSQLRecord.java | 2 +
.../processors/standard/TestExecuteSQL.java | 7 +++-
.../standard/TestExecuteSQLRecord.java | 43
5 files changed, 55 insertions(+), 1 deletion(-)
--
http://git-wip-us.apache.org/repos/asf/nifi/blob/9358a60d/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/AbstractExecuteSQL.java
--
diff --git
a/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/AbstractExecuteSQL.java
b/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/AbstractExecuteSQL.java
index d1fabef..76e36fd 100644
---
a/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/AbstractExecuteSQL.java
+++
b/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/AbstractExecuteSQL.java
@@ -61,6 +61,7 @@ public abstract class AbstractExecuteSQL extends
AbstractProcessor {
public static final String RESULT_QUERY_EXECUTION_TIME =
"executesql.query.executiontime";
public static final String RESULT_QUERY_FETCH_TIME =
"executesql.query.fetchtime";
public static final String RESULTSET_INDEX = "executesql.resultset.index";
+public static final String RESULT_ERROR_MESSAGE =
"executesql.error.message";
public static final String FRAGMENT_ID =
FragmentAttributes.FRAGMENT_ID.key();
public static final String FRAGMENT_INDEX =
FragmentAttributes.FRAGMENT_INDEX.key();
@@ -402,6 +403,7 @@ public abstract class AbstractExecuteSQL extends
AbstractProcessor {
new Object[]{selectQuery, e});
context.yield();
}
+
session.putAttribute(fileToProcess,RESULT_ERROR_MESSAGE,e.getMessage());
session.transfer(fileToProcess, REL_FAILURE);
}
}
http://git-wip-us.apache.org/repos/asf/nifi/blob/9358a60d/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/ExecuteSQL.java
--
diff --git
a/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/ExecuteSQL.java
b/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/ExecuteSQL.java
index 99e0d2a..cfdef29 100644
---
a/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/ExecuteSQL.java
+++
b/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/ExecuteSQL.java
@@ -87,6 +87,8 @@ import static
org.apache.nifi.processors.standard.util.JdbcCommon.USE_AVRO_LOGIC
+ "If 'Max Rows Per Flow File' is set, then this number will
reflect only the fetch time for the rows in the Flow File instead of the entire
result set."),
@WritesAttribute(attribute = "executesql.resultset.index", description
= "Assuming multiple result sets are returned, "
+ "the zero based index of this result set."),
+@WritesAttribute(attribute = "executesql.error.message", description =
"If processing an incoming flow file causes "
++ "an Exception, the Flow File is routed to failure and this
attribute is set to the exception message."),
@WritesAttribute(attribute = "fragment.identifier", description = "If
'Max Rows Per Flow File' is set then all FlowFiles from the same query result
set "
+ "will have the same value for the fragment.identifier
attribute. This can then be used to correlate the results."),
@WritesAttribute(attribute = "fragment.count", description = "If 'Max
nifi git commit: NIFI-5604: Added property to allow empty FlowFile when no SQL generated by GenerateTableFetch
Repository: nifi
Updated Branches:
refs/heads/master 75906226a -> 0207d0813
NIFI-5604: Added property to allow empty FlowFile when no SQL generated by
GenerateTableFetch
co-authored by: Peter Wicks
Signed-off-by: Peter Wicks
This closes #3075.
Project: http://git-wip-us.apache.org/repos/asf/nifi/repo
Commit: http://git-wip-us.apache.org/repos/asf/nifi/commit/0207d081
Tree: http://git-wip-us.apache.org/repos/asf/nifi/tree/0207d081
Diff: http://git-wip-us.apache.org/repos/asf/nifi/diff/0207d081
Branch: refs/heads/master
Commit: 0207d0813ef164ee7227ded61fb10960a4842e2d
Parents: 7590622
Author: Matthew Burgess
Authored: Thu Nov 1 22:09:13 2018 -0400
Committer: Peter Wicks
Committed: Thu Nov 15 13:37:43 2018 -0700
--
.../processors/standard/GenerateTableFetch.java | 104 +--
.../standard/TestGenerateTableFetch.java| 47 -
2 files changed, 118 insertions(+), 33 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/nifi/blob/0207d081/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/GenerateTableFetch.java
--
diff --git
a/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/GenerateTableFetch.java
b/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/GenerateTableFetch.java
index a547393..b4ba9fe 100644
---
a/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/GenerateTableFetch.java
+++
b/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/GenerateTableFetch.java
@@ -142,6 +142,18 @@ public class GenerateTableFetch extends
AbstractDatabaseFetchProcessor {
.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
.build();
+static final PropertyDescriptor OUTPUT_EMPTY_FLOWFILE_ON_ZERO_RESULTS =
new PropertyDescriptor.Builder()
+.name("gen-table-output-flowfile-on-zero-results")
+.displayName("Output Empty FlowFile on Zero Results")
+.description("Depending on the specified properties, an execution
of this processor may not result in any SQL statements generated. When this
property "
++ "is true, an empty flow file will be generated (having
the parent of the incoming flow file if present) and transferred to the
'success' relationship. "
++ "When this property is false, no output flow files will
be generated.")
+.required(true)
+.allowableValues("true", "false")
+.defaultValue("false")
+.addValidator(StandardValidators.BOOLEAN_VALIDATOR)
+.build();
+
public static final Relationship REL_FAILURE = new Relationship.Builder()
.name("failure")
.description("This relationship is only used when SQL query
execution (using an incoming FlowFile) failed. The incoming FlowFile will be
penalized and routed to this relationship. "
@@ -164,6 +176,7 @@ public class GenerateTableFetch extends
AbstractDatabaseFetchProcessor {
pds.add(PARTITION_SIZE);
pds.add(COLUMN_FOR_VALUE_PARTITIONING);
pds.add(WHERE_CLAUSE);
+pds.add(OUTPUT_EMPTY_FLOWFILE_ON_ZERO_RESULTS);
propDescriptors = Collections.unmodifiableList(pds);
}
@@ -247,6 +260,7 @@ public class GenerateTableFetch extends
AbstractDatabaseFetchProcessor {
final String columnForPartitioning =
context.getProperty(COLUMN_FOR_VALUE_PARTITIONING).evaluateAttributeExpressions(fileToProcess).getValue();
final boolean useColumnValsForPaging =
!StringUtils.isEmpty(columnForPartitioning);
final String customWhereClause =
context.getProperty(WHERE_CLAUSE).evaluateAttributeExpressions(fileToProcess).getValue();
+final boolean outputEmptyFlowFileOnZeroResults =
context.getProperty(OUTPUT_EMPTY_FLOWFILE_ON_ZERO_RESULTS).asBoolean();
final StateManager stateManager = context.getStateManager();
final StateMap stateMap;
@@ -435,49 +449,75 @@ public class GenerateTableFetch extends
AbstractDatabaseFetchProcessor {
}
// Generate SQL statements to read "pages" of data
-Long limit = partitionSize == 0 ? null : (long) partitionSize;
final String fragmentIdentifier = UUID.randomUUID().toString();
-for (long i = 0; i < numberOfFetches; i++) {
-// Add a right bounding for the partitioning column if
necessary (only on last partition, meaning we don't need the limit)
-if ((i == numberOfFetches - 1) &&
nifi git commit: NIFI-5780 Add pre and post statements to ExecuteSQL and ExecuteSQLRecord
Repository: nifi
Updated Branches:
refs/heads/master be0949570 -> 75906226a
NIFI-5780 Add pre and post statements to ExecuteSQL and ExecuteSQLRecord
Signed-off-by: Peter Wicks
This closes #3156.
Project: http://git-wip-us.apache.org/repos/asf/nifi/repo
Commit: http://git-wip-us.apache.org/repos/asf/nifi/commit/75906226
Tree: http://git-wip-us.apache.org/repos/asf/nifi/tree/75906226
Diff: http://git-wip-us.apache.org/repos/asf/nifi/diff/75906226
Branch: refs/heads/master
Commit: 75906226a6f265acc7d87414ff477a5b0646b6d8
Parents: be09495
Author: yjhyjhyjh0
Authored: Thu Nov 8 00:25:50 2018 +0800
Committer: Peter Wicks
Committed: Thu Nov 15 13:18:31 2018 -0700
--
.../nifi/processors/hive/SelectHiveQL.java | 4 +-
.../processors/standard/AbstractExecuteSQL.java | 79 +-
.../nifi/processors/standard/ExecuteSQL.java| 2 +
.../processors/standard/ExecuteSQLRecord.java | 2 +
.../processors/standard/TestExecuteSQL.java | 146 +++
.../standard/TestExecuteSQLRecord.java | 132 +
6 files changed, 362 insertions(+), 3 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/nifi/blob/75906226/nifi-nar-bundles/nifi-hive-bundle/nifi-hive-processors/src/main/java/org/apache/nifi/processors/hive/SelectHiveQL.java
--
diff --git
a/nifi-nar-bundles/nifi-hive-bundle/nifi-hive-processors/src/main/java/org/apache/nifi/processors/hive/SelectHiveQL.java
b/nifi-nar-bundles/nifi-hive-bundle/nifi-hive-processors/src/main/java/org/apache/nifi/processors/hive/SelectHiveQL.java
index 5342c09..3b8576b 100644
---
a/nifi-nar-bundles/nifi-hive-bundle/nifi-hive-processors/src/main/java/org/apache/nifi/processors/hive/SelectHiveQL.java
+++
b/nifi-nar-bundles/nifi-hive-bundle/nifi-hive-processors/src/main/java/org/apache/nifi/processors/hive/SelectHiveQL.java
@@ -109,7 +109,7 @@ public class SelectHiveQL extends AbstractHiveQLProcessor {
public static final PropertyDescriptor HIVEQL_PRE_QUERY = new
PropertyDescriptor.Builder()
.name("hive-pre-query")
.displayName("HiveQL Pre-Query")
-.description("HiveQL pre-query to execute. Semicolon-delimited
list of queries. "
+.description("A semicolon-delimited list of queries executed
before the main SQL query is executed. "
+ "Example: 'set tez.queue.name=queue1; set
hive.exec.orc.split.strategy=ETL; set
hive.exec.reducers.bytes.per.reducer=1073741824'. "
+ "Note, the results/outputs of these queries will be
suppressed if successfully executed.")
.required(false)
@@ -129,7 +129,7 @@ public class SelectHiveQL extends AbstractHiveQLProcessor {
public static final PropertyDescriptor HIVEQL_POST_QUERY = new
PropertyDescriptor.Builder()
.name("hive-post-query")
.displayName("HiveQL Post-Query")
-.description("HiveQL post-query to execute. Semicolon-delimited
list of queries. "
+.description("A semicolon-delimited list of queries executed after
the main SQL query is executed. "
+ "Note, the results/outputs of these queries will be
suppressed if successfully executed.")
.required(false)
.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
http://git-wip-us.apache.org/repos/asf/nifi/blob/75906226/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/AbstractExecuteSQL.java
--
diff --git
a/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/AbstractExecuteSQL.java
b/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/AbstractExecuteSQL.java
index bf46549..d1fabef 100644
---
a/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/AbstractExecuteSQL.java
+++
b/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/AbstractExecuteSQL.java
@@ -17,6 +17,7 @@
package org.apache.nifi.processors.standard;
import org.apache.commons.io.IOUtils;
+import org.apache.commons.lang3.tuple.Pair;
import org.apache.nifi.annotation.lifecycle.OnScheduled;
import org.apache.nifi.components.PropertyDescriptor;
import org.apache.nifi.dbcp.DBCPService;
@@ -44,6 +45,7 @@ import java.sql.Statement;
import java.util.ArrayList;
import java.util.Collections;
import java.util.HashMap;
+import java.util.LinkedList;
import java.util.List;
import java.util.Map;
import java.util.Set;
@@ -82,6 +84,17 @@ public abstract class AbstractExecuteSQL extends
[2/2] nifi git commit: NIFI-5718: Added performance-based unit test (Ignored) for LineDemarcator
NIFI-5718: Added performance-based unit test (Ignored) for LineDemarcator
Signed-off-by: Peter Wicks
This closes #3100.
Project: http://git-wip-us.apache.org/repos/asf/nifi/repo
Commit: http://git-wip-us.apache.org/repos/asf/nifi/commit/830f7aa8
Tree: http://git-wip-us.apache.org/repos/asf/nifi/tree/830f7aa8
Diff: http://git-wip-us.apache.org/repos/asf/nifi/diff/830f7aa8
Branch: refs/heads/master
Commit: 830f7aa84d2f61781422daa43648a09c3a08f392
Parents: 564ad0c
Author: Mark Payne
Authored: Fri Nov 9 12:04:43 2018 -0500
Committer: Peter Wicks
Committed: Fri Nov 9 14:27:32 2018 -0700
--
.../nifi/stream/io/util/TestLineDemarcator.java | 28
1 file changed, 28 insertions(+)
--
http://git-wip-us.apache.org/repos/asf/nifi/blob/830f7aa8/nifi-commons/nifi-utils/src/test/java/org/apache/nifi/stream/io/util/TestLineDemarcator.java
--
diff --git
a/nifi-commons/nifi-utils/src/test/java/org/apache/nifi/stream/io/util/TestLineDemarcator.java
b/nifi-commons/nifi-utils/src/test/java/org/apache/nifi/stream/io/util/TestLineDemarcator.java
index 768a60a..85cf85e 100644
---
a/nifi-commons/nifi-utils/src/test/java/org/apache/nifi/stream/io/util/TestLineDemarcator.java
+++
b/nifi-commons/nifi-utils/src/test/java/org/apache/nifi/stream/io/util/TestLineDemarcator.java
@@ -16,10 +16,13 @@
*/
package org.apache.nifi.stream.io.util;
+import org.apache.nifi.stream.io.RepeatingInputStream;
+import org.junit.Ignore;
import org.junit.Test;
import java.io.ByteArrayInputStream;
import java.io.IOException;
+import java.io.InputStream;
import java.io.InputStreamReader;
import java.io.Reader;
import java.nio.charset.StandardCharsets;
@@ -27,6 +30,7 @@ import java.util.ArrayList;
import java.util.Arrays;
import java.util.Collections;
import java.util.List;
+import java.util.concurrent.TimeUnit;
import static org.junit.Assert.assertEquals;
@@ -95,6 +99,30 @@ public class TestLineDemarcator {
assertEquals(Arrays.asList("\n", "The quick brown fox jumped over the
lazy dog."), lines);
}
+@Test
+@Ignore("Intended only for manual testing. While this can take a while to
run, it can be very helpful for manual testing before and after a change to the
class. However, we don't want this to " +
+"run in automated tests because we have no way to compare from one run
to another, so it will only slow down automated tests.")
+public void testPerformance() throws IOException {
+final String lines =
"The\nquick\nbrown\nfox\njumped\nover\nthe\nlazy\ndog.\r\n\n";
+final byte[] bytes = lines.getBytes(StandardCharsets.UTF_8);
+
+for (int i=0; i < 100; i++) {
+final long start = System.nanoTime();
+
+long count = 0;
+try (final InputStream in = new RepeatingInputStream(bytes,
1_000_000);
+ final LineDemarcator demarcator = new LineDemarcator(in,
StandardCharsets.UTF_8, 8192, 8192)) {
+
+while (demarcator.nextLine() != null) {
+count++;
+}
+}
+
+final long millis =
TimeUnit.NANOSECONDS.toMillis(System.nanoTime() - start);
+System.out.println("Took " + millis + " millis to demarcate " +
count + " lines");
+}
+}
+
private List getLines(final String text) throws IOException {
return getLines(text, 8192, 8192);
}
[1/2] nifi git commit: NIFI-5718: Implemented LineDemarcator and removed NLKBufferedReader in order to improve performance
Repository: nifi
Updated Branches:
refs/heads/master 765df6781 -> 830f7aa84
NIFI-5718: Implemented LineDemarcator and removed NLKBufferedReader in order to
improve performance
Project: http://git-wip-us.apache.org/repos/asf/nifi/repo
Commit: http://git-wip-us.apache.org/repos/asf/nifi/commit/564ad0cd
Tree: http://git-wip-us.apache.org/repos/asf/nifi/tree/564ad0cd
Diff: http://git-wip-us.apache.org/repos/asf/nifi/diff/564ad0cd
Branch: refs/heads/master
Commit: 564ad0cd71e56b426e02757c7a0a70e28ccbea12
Parents: 765df67
Author: Mark Payne
Authored: Thu Oct 18 12:05:16 2018 -0400
Committer: Peter Wicks
Committed: Fri Nov 9 14:26:20 2018 -0700
--
.../nifi/stream/io/RepeatingInputStream.java| 103 +
.../stream/io/util/AbstractTextDemarcator.java | 147 +++
.../nifi/stream/io/util/LineDemarcator.java | 116 +++
.../nifi/stream/io/util/TestLineDemarcator.java | 120 +++
.../nifi/processors/standard/ReplaceText.java | 71 +
.../nifi/processors/standard/RouteText.java | 49 +++
.../standard/util/NLKBufferedReader.java| 76 --
7 files changed, 549 insertions(+), 133 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/nifi/blob/564ad0cd/nifi-commons/nifi-utils/src/main/java/org/apache/nifi/stream/io/RepeatingInputStream.java
--
diff --git
a/nifi-commons/nifi-utils/src/main/java/org/apache/nifi/stream/io/RepeatingInputStream.java
b/nifi-commons/nifi-utils/src/main/java/org/apache/nifi/stream/io/RepeatingInputStream.java
new file mode 100644
index 000..f542741
--- /dev/null
+++
b/nifi-commons/nifi-utils/src/main/java/org/apache/nifi/stream/io/RepeatingInputStream.java
@@ -0,0 +1,103 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.stream.io;
+
+import java.io.ByteArrayInputStream;
+import java.io.IOException;
+import java.io.InputStream;
+import java.util.Objects;
+
+public class RepeatingInputStream extends InputStream {
+private final byte[] toRepeat;
+private final int maxIterations;
+
+private InputStream bais;
+private int repeatCount;
+
+
+public RepeatingInputStream(final byte[] toRepeat, final int iterations) {
+if (iterations < 1) {
+throw new IllegalArgumentException();
+}
+if (Objects.requireNonNull(toRepeat).length == 0) {
+throw new IllegalArgumentException();
+}
+
+this.toRepeat = toRepeat;
+this.maxIterations = iterations;
+
+repeat();
+bais = new ByteArrayInputStream(toRepeat);
+repeatCount = 1;
+}
+
+@Override
+public int read() throws IOException {
+final int value = bais.read();
+if (value > -1) {
+return value;
+}
+
+final boolean repeated = repeat();
+if (repeated) {
+return bais.read();
+}
+
+return -1;
+}
+
+@Override
+public int read(final byte[] b, final int off, final int len) throws
IOException {
+final int value = bais.read(b, off, len);
+if (value > -1) {
+return value;
+}
+
+final boolean repeated = repeat();
+if (repeated) {
+return bais.read(b, off, len);
+}
+
+return -1;
+}
+
+@Override
+public int read(final byte[] b) throws IOException {
+final int value = bais.read(b);
+if (value > -1) {
+return value;
+}
+
+final boolean repeated = repeat();
+if (repeated) {
+return bais.read(b);
+}
+
+return -1;
+}
+
+private boolean repeat() {
+if (repeatCount >= maxIterations) {
+return false;
+}
+
+repeatCount++;
+bais = new ByteArrayInputStream(toRepeat);
+
+return true;
+}
+}
http://git-wip-us.apache.org/repos/asf/nifi/blob/564ad0cd/nifi-commons/nifi-utils/src/main/java/org/apache/nifi/stream/io/util/AbstractTextDemarcator.java
nifi git commit: NIFI-5790: Exposes 6 commons-dbcp options in DBCPConnectionPool
Repository: nifi
Updated Branches:
refs/heads/master 63f55d05b -> a628aced6
NIFI-5790: Exposes 6 commons-dbcp options in DBCPConnectionPool
Signed-off-by: Peter Wicks
This Closes #3133
Project: http://git-wip-us.apache.org/repos/asf/nifi/repo
Commit: http://git-wip-us.apache.org/repos/asf/nifi/commit/a628aced
Tree: http://git-wip-us.apache.org/repos/asf/nifi/tree/a628aced
Diff: http://git-wip-us.apache.org/repos/asf/nifi/diff/a628aced
Branch: refs/heads/master
Commit: a628aced6b32b16eb4509094d214266938a915b9
Parents: 63f55d0
Author: Colin Dean
Authored: Mon Nov 5 16:18:21 2018 -0500
Committer: Peter Wicks
Committed: Fri Nov 9 11:29:11 2018 -0700
--
.../apache/nifi/dbcp/DBCPConnectionPool.java| 126 +++-
.../org/apache/nifi/dbcp/DBCPServiceTest.java | 150 +++
2 files changed, 275 insertions(+), 1 deletion(-)
--
http://git-wip-us.apache.org/repos/asf/nifi/blob/a628aced/nifi-nar-bundles/nifi-standard-services/nifi-dbcp-service-bundle/nifi-dbcp-service/src/main/java/org/apache/nifi/dbcp/DBCPConnectionPool.java
--
diff --git
a/nifi-nar-bundles/nifi-standard-services/nifi-dbcp-service-bundle/nifi-dbcp-service/src/main/java/org/apache/nifi/dbcp/DBCPConnectionPool.java
b/nifi-nar-bundles/nifi-standard-services/nifi-dbcp-service-bundle/nifi-dbcp-service/src/main/java/org/apache/nifi/dbcp/DBCPConnectionPool.java
index 01205e2..5228496 100644
---
a/nifi-nar-bundles/nifi-standard-services/nifi-dbcp-service-bundle/nifi-dbcp-service/src/main/java/org/apache/nifi/dbcp/DBCPConnectionPool.java
+++
b/nifi-nar-bundles/nifi-standard-services/nifi-dbcp-service-bundle/nifi-dbcp-service/src/main/java/org/apache/nifi/dbcp/DBCPConnectionPool.java
@@ -17,12 +17,14 @@
package org.apache.nifi.dbcp;
import org.apache.commons.dbcp2.BasicDataSource;
+import org.apache.commons.pool2.impl.GenericObjectPoolConfig;
import org.apache.nifi.annotation.behavior.DynamicProperty;
import org.apache.nifi.annotation.documentation.CapabilityDescription;
import org.apache.nifi.annotation.documentation.Tags;
import org.apache.nifi.annotation.lifecycle.OnDisabled;
import org.apache.nifi.annotation.lifecycle.OnEnabled;
import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.PropertyValue;
import org.apache.nifi.components.ValidationContext;
import org.apache.nifi.components.ValidationResult;
import org.apache.nifi.components.Validator;
@@ -59,6 +61,32 @@ import java.util.regex.Pattern;
+ "Note that no flow file input (attributes, e.g.) is
available for use in Expression Language constructs for these properties.")
public class DBCPConnectionPool extends AbstractControllerService implements
DBCPService {
+/**
+ * Copied from {@link GenericObjectPoolConfig.DEFAULT_MIN_IDLE} in
Commons-DBCP 2.5.0
+ */
+private static final String DEFAULT_MIN_IDLE = "0";
+/**
+ * Copied from {@link GenericObjectPoolConfig.DEFAULT_MAX_IDLE} in
Commons-DBCP 2.5.0
+ */
+private static final String DEFAULT_MAX_IDLE = "8";
+/**
+ * Copied from private variable {@link
BasicDataSource.maxConnLifetimeMillis} in Commons-DBCP 2.5.0
+ */
+private static final String DEFAULT_MAX_CONN_LIFETIME = "-1";
+/**
+ * Copied from {@link
GenericObjectPoolConfig.DEFAULT_TIME_BETWEEN_EVICTION_RUNS_MILLIS} in
Commons-DBCP 2.5.0
+ */
+private static final String DEFAULT_EVICTION_RUN_PERIOD =
String.valueOf(-1L);
+/**
+ * Copied from {@link
GenericObjectPoolConfig.DEFAULT_MIN_EVICTABLE_IDLE_TIME_MILLIS} in Commons-DBCP
2.5.0
+ * and converted from 180L to "180 millis" to "30 mins"
+ */
+private static final String DEFAULT_MIN_EVICTABLE_IDLE_TIME = "30 mins";
+/**
+ * Copied from {@link
GenericObjectPoolConfig.DEFAULT_SOFT_MIN_EVICTABLE_IDLE_TIME_MILLIS} in
Commons-DBCP 2.5.0
+ */
+private static final String DEFAULT_SOFT_MIN_EVICTABLE_IDLE_TIME =
String.valueOf(-1L);
+
private static final Validator CUSTOM_TIME_PERIOD_VALIDATOR = new
Validator() {
private final Pattern TIME_DURATION_PATTERN =
Pattern.compile(FormatUtils.TIME_DURATION_REGEX);
@@ -164,6 +192,77 @@ public class DBCPConnectionPool extends
AbstractControllerService implements DBC
.expressionLanguageSupported(ExpressionLanguageScope.VARIABLE_REGISTRY)
.build();
+public static final PropertyDescriptor MIN_IDLE = new
PropertyDescriptor.Builder()
+.displayName("Minimum Idle Connections")
+.name("dbcp-mim-idle-conns")
+.description("The minimum number of connections that can remain
idle in the pool, without extra ones being " +
+"created, or zero to create none.")
+
nifi git commit: NIFI-5601: Add fragment.* attributes to GenerateTableFetch
Repository: nifi
Updated Branches:
refs/heads/master fdd8cdbb3 -> d8d220ccb
NIFI-5601: Add fragment.* attributes to GenerateTableFetch
Signed-off-by: Peter Wicks
This closes #3074
Project: http://git-wip-us.apache.org/repos/asf/nifi/repo
Commit: http://git-wip-us.apache.org/repos/asf/nifi/commit/d8d220cc
Tree: http://git-wip-us.apache.org/repos/asf/nifi/tree/d8d220cc
Diff: http://git-wip-us.apache.org/repos/asf/nifi/diff/d8d220cc
Branch: refs/heads/master
Commit: d8d220ccb86d1797f56f34649d70a1acff278eb5
Parents: fdd8cdb
Author: Matthew Burgess
Authored: Mon Oct 15 16:07:13 2018 -0400
Committer: patricker
Committed: Thu Nov 1 15:14:29 2018 -0600
--
.../AbstractDatabaseFetchProcessor.java | 4 +++
.../standard/AbstractQueryDatabaseTable.java| 6 +---
.../processors/standard/GenerateTableFetch.java | 33 +++-
.../standard/TestGenerateTableFetch.java| 17 +-
4 files changed, 47 insertions(+), 13 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/nifi/blob/d8d220cc/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/AbstractDatabaseFetchProcessor.java
--
diff --git
a/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/AbstractDatabaseFetchProcessor.java
b/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/AbstractDatabaseFetchProcessor.java
index 3da8a73..a99ca6a 100644
---
a/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/AbstractDatabaseFetchProcessor.java
+++
b/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/AbstractDatabaseFetchProcessor.java
@@ -23,6 +23,7 @@ import org.apache.nifi.components.ValidationResult;
import org.apache.nifi.dbcp.DBCPService;
import org.apache.nifi.expression.ExpressionLanguageScope;
import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.flowfile.attributes.FragmentAttributes;
import org.apache.nifi.processor.AbstractSessionFactoryProcessor;
import org.apache.nifi.processor.ProcessContext;
import org.apache.nifi.processor.Relationship;
@@ -89,6 +90,9 @@ import static java.sql.Types.VARCHAR;
public abstract class AbstractDatabaseFetchProcessor extends
AbstractSessionFactoryProcessor {
public static final String INITIAL_MAX_VALUE_PROP_START =
"initial.maxvalue.";
+public static final String FRAGMENT_ID =
FragmentAttributes.FRAGMENT_ID.key();
+public static final String FRAGMENT_INDEX =
FragmentAttributes.FRAGMENT_INDEX.key();
+public static final String FRAGMENT_COUNT =
FragmentAttributes.FRAGMENT_COUNT.key();
// Relationships
public static final Relationship REL_SUCCESS = new Relationship.Builder()
http://git-wip-us.apache.org/repos/asf/nifi/blob/d8d220cc/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/AbstractQueryDatabaseTable.java
--
diff --git
a/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/AbstractQueryDatabaseTable.java
b/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/AbstractQueryDatabaseTable.java
index 06df6c1..57933b3 100644
---
a/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/AbstractQueryDatabaseTable.java
+++
b/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/AbstractQueryDatabaseTable.java
@@ -27,7 +27,6 @@ import org.apache.nifi.dbcp.DBCPService;
import org.apache.nifi.expression.AttributeExpression;
import org.apache.nifi.expression.ExpressionLanguageScope;
import org.apache.nifi.flowfile.FlowFile;
-import org.apache.nifi.flowfile.attributes.FragmentAttributes;
import org.apache.nifi.logging.ComponentLog;
import org.apache.nifi.processor.ProcessContext;
import org.apache.nifi.processor.ProcessSession;
@@ -66,9 +65,6 @@ public abstract class AbstractQueryDatabaseTable extends
AbstractDatabaseFetchPr
public static final String RESULT_TABLENAME = "tablename";
public static final String RESULT_ROW_COUNT = "querydbtable.row.count";
-public static final String FRAGMENT_ID =
FragmentAttributes.FRAGMENT_ID.key();
-public static final String FRAGMENT_INDEX =
FragmentAttributes.FRAGMENT_INDEX.key();
-
public static final PropertyDescriptor FETCH_SIZE = new
PropertyDescriptor.Builder()
.name("Fetch Size")
