[GitHub] nifi issue #3138: NIFI-5798:Fixed bug in FlattenJson that was escaping text ...
Github user pvillard31 commented on the issue: https://github.com/apache/nifi/pull/3138 +1, merging to master, thanks! ---
[GitHub] nifi pull request #3138: NIFI-5798:Fixed bug in FlattenJson that was escapin...
Github user asfgit closed the pull request at: https://github.com/apache/nifi/pull/3138 ---
[GitHub] nifi issue #3137: NIFI-5797 : EscapedJava for FlattenJson
Github user pvillard31 commented on the issue: https://github.com/apache/nifi/pull/3137 I believe this has been fixed by #3138. If you agree, can you close this PR? ---
[GitHub] nifi issue #3155: NIFI-5784 Edit Admin Guide to remove duplicate content tha...
Github user pvillard31 commented on the issue: https://github.com/apache/nifi/pull/3155 +1, merging to master, thanks! ---
[GitHub] nifi pull request #3155: NIFI-5784 Edit Admin Guide to remove duplicate cont...
Github user asfgit closed the pull request at: https://github.com/apache/nifi/pull/3155 ---
[GitHub] nifi issue #3157: NIFI-5800: Do not recursively call hashCode on child schem...
Github user pvillard31 commented on the issue: https://github.com/apache/nifi/pull/3157 +1, merging to master, thanks! ---
[GitHub] nifi pull request #3157: NIFI-5800: Do not recursively call hashCode on chil...
Github user asfgit closed the pull request at: https://github.com/apache/nifi/pull/3157 ---
[GitHub] nifi issue #3158: NIFI-5802: Add QueryRecord nullable field support
Github user pvillard31 commented on the issue: https://github.com/apache/nifi/pull/3158 +1, merging to master, thanks @ijokarumawak ---
[GitHub] nifi pull request #3158: NIFI-5802: Add QueryRecord nullable field support
Github user asfgit closed the pull request at: https://github.com/apache/nifi/pull/3158 ---
[GitHub] nifi issue #3128: NIFI-5788: Introduce batch size limit in PutDatabaseRecord...
Github user vadimar commented on the issue: https://github.com/apache/nifi/pull/3128 Hi, Can you please review the latest commits? I committed the changes that address all the issues raised by reviewers. Thanks ---
[GitHub] nifi pull request #3159: NIFI-5794 Allow empty string demarcator in Consume/...
GitHub user pvillard31 opened a pull request: https://github.com/apache/nifi/pull/3159 NIFI-5794 Allow empty string demarcator in Consume/PublishKafka proce⦠â¦ssors Thank you for submitting a contribution to Apache NiFi. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [ ] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [ ] Does your PR title start with NIFI- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [ ] Has your PR been rebased against the latest commit within the target branch (typically master)? - [ ] Is your initial contribution a single, squashed commit? ### For code changes: - [ ] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder? - [ ] Have you written or updated unit tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly? - [ ] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly? - [ ] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. You can merge this pull request into a Git repository by running: $ git pull https://github.com/pvillard31/nifi NIFI-5794 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi/pull/3159.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3159 commit 88fbcffdc3fa6aadac14c23af9a5012edaa21e0f Author: Pierre Villard Date: 2018-11-07T09:55:10Z NIFI-5794 Allow empty string demarcator in Consume/PublishKafka processors ---
[GitHub] nifi issue #3159: NIFI-5794 Allow empty string demarcator in Consume/Publish...
Github user pvillard31 commented on the issue: https://github.com/apache/nifi/pull/3159 In the end, I only changed the ConsumeKafka processors since it does not make sense in PublishKafka (one would just have to not set any demarcator to get the same result: all messages in the flow files are sent as one single message in Kafka). The main interest of this PR is to increase efficiency when we consume Avro data from Kafka: there is no need to set a demarcator and we do want to have multiples messages in the same flow file. I've used this [template](https://gist.github.com/pvillard31/57e627d206e7965cfb87cc27c09caf44) to test the change and to confirm the behavior: when pulling Avro data from Kafka with "empty string" as demarcator to have multiple messages in the same flow file, we are able to process the records in the next processor using a ConvertRecord for instance. ---
[GitHub] nifi-minifi-cpp pull request #432: MINIFICPP-648 - add processor and add pro...
Github user phrocker commented on a diff in the pull request:
https://github.com/apache/nifi-minifi-cpp/pull/432#discussion_r231865178
--- Diff: nanofi/include/cxx/CallbackProcessor.h ---
@@ -65,7 +65,7 @@ class CallbackProcessor : public core::Processor {
public:
- void setCallback(void *obj,std::function
ontrigger_callback) {
+ void setCallback(void *obj,std::function
ontrigger_callback) {
--- End diff --
It's been a while since I've looked at that work, but did the python
bindings call this correctly when running python processors?
---
[GitHub] nifi-minifi-cpp pull request #432: MINIFICPP-648 - add processor and add pro...
Github user phrocker commented on a diff in the pull request:
https://github.com/apache/nifi-minifi-cpp/pull/432#discussion_r23188
--- Diff: nanofi/src/api/nanofi.cpp ---
@@ -323,55 +360,45 @@ int transmit_flowfile(flow_file_record *ff,
nifi_instance *instance) {
flow * create_new_flow(nifi_instance * instance) {
auto minifi_instance_ref =
static_cast(instance->instance_ptr);
- flow *new_flow = (flow*) malloc(sizeof(flow));
-
- auto execution_plan = new
ExecutionPlan(minifi_instance_ref->getContentRepository(),
minifi_instance_ref->getNoOpRepository(),
minifi_instance_ref->getNoOpRepository());
-
- new_flow->plan = execution_plan;
-
- return new_flow;
+ return new flow(minifi_instance_ref->getContentRepository(),
minifi_instance_ref->getNoOpRepository(),
minifi_instance_ref->getNoOpRepository());
--- End diff --
https://issues.apache.org/jira/browse/MINIFICPP-645 exists because public
facing objects that are turned from a C API can be malloc'd by users. If a
consumer of our API doesn't use free as designed, we should guarantee that they
only cause memory leaks, not introduce undefined behavior.
---
[GitHub] nifi-minifi-cpp pull request #432: MINIFICPP-648 - add processor and add pro...
Github user phrocker commented on a diff in the pull request:
https://github.com/apache/nifi-minifi-cpp/pull/432#discussion_r231867815
--- Diff: nanofi/src/api/nanofi.cpp ---
@@ -323,55 +360,45 @@ int transmit_flowfile(flow_file_record *ff,
nifi_instance *instance) {
flow * create_new_flow(nifi_instance * instance) {
auto minifi_instance_ref =
static_cast(instance->instance_ptr);
- flow *new_flow = (flow*) malloc(sizeof(flow));
-
- auto execution_plan = new
ExecutionPlan(minifi_instance_ref->getContentRepository(),
minifi_instance_ref->getNoOpRepository(),
minifi_instance_ref->getNoOpRepository());
-
- new_flow->plan = execution_plan;
-
- return new_flow;
+ return new flow(minifi_instance_ref->getContentRepository(),
minifi_instance_ref->getNoOpRepository(),
minifi_instance_ref->getNoOpRepository());
}
flow *create_flow(nifi_instance *instance, const char *first_processor) {
if (nullptr == instance || nullptr == instance->instance_ptr) {
return nullptr;
}
auto minifi_instance_ref =
static_cast(instance->instance_ptr);
- flow *new_flow = (flow*) malloc(sizeof(flow));
-
- auto execution_plan = new
ExecutionPlan(minifi_instance_ref->getContentRepository(),
minifi_instance_ref->getNoOpRepository(),
minifi_instance_ref->getNoOpRepository());
- new_flow->plan = execution_plan;
+ flow *new_flow = new flow(minifi_instance_ref->getContentRepository(),
minifi_instance_ref->getNoOpRepository(),
minifi_instance_ref->getNoOpRepository());
if (first_processor != nullptr && strlen(first_processor) > 0) {
// automatically adds it with success
-execution_plan->addProcessor(first_processor, first_processor);
--- End diff --
Combined C and C++ implies we're moving more toward C++. If that were the
case this API would be drastically different. We should move in the other
direction since C++ functions as a temporary interconnect.
---
[GitHub] nifi-minifi-cpp pull request #432: MINIFICPP-648 - add processor and add pro...
Github user phrocker commented on a diff in the pull request: https://github.com/apache/nifi-minifi-cpp/pull/432#discussion_r231865240 --- Diff: nanofi/include/cxx/Plan.h --- @@ -44,7 +44,7 @@ #include "core/ProcessSession.h" #include "core/ProcessorNode.h" #include "core/reporting/SiteToSiteProvenanceReportingTask.h" -#include "core/cstructs.h" --- End diff -- Should remove the rem'd line. ---
[GitHub] nifi-minifi-cpp pull request #432: MINIFICPP-648 - add processor and add pro...
Github user phrocker commented on a diff in the pull request:
https://github.com/apache/nifi-minifi-cpp/pull/432#discussion_r231867605
--- Diff: nanofi/src/api/nanofi.cpp ---
@@ -323,55 +360,45 @@ int transmit_flowfile(flow_file_record *ff,
nifi_instance *instance) {
flow * create_new_flow(nifi_instance * instance) {
auto minifi_instance_ref =
static_cast(instance->instance_ptr);
- flow *new_flow = (flow*) malloc(sizeof(flow));
-
- auto execution_plan = new
ExecutionPlan(minifi_instance_ref->getContentRepository(),
minifi_instance_ref->getNoOpRepository(),
minifi_instance_ref->getNoOpRepository());
-
- new_flow->plan = execution_plan;
-
- return new_flow;
+ return new flow(minifi_instance_ref->getContentRepository(),
minifi_instance_ref->getNoOpRepository(),
minifi_instance_ref->getNoOpRepository());
}
flow *create_flow(nifi_instance *instance, const char *first_processor) {
if (nullptr == instance || nullptr == instance->instance_ptr) {
return nullptr;
}
auto minifi_instance_ref =
static_cast(instance->instance_ptr);
- flow *new_flow = (flow*) malloc(sizeof(flow));
-
- auto execution_plan = new
ExecutionPlan(minifi_instance_ref->getContentRepository(),
minifi_instance_ref->getNoOpRepository(),
minifi_instance_ref->getNoOpRepository());
- new_flow->plan = execution_plan;
+ flow *new_flow = new flow(minifi_instance_ref->getContentRepository(),
minifi_instance_ref->getNoOpRepository(),
minifi_instance_ref->getNoOpRepository());
--- End diff --
same malloc / new
---
[GitHub] nifi-minifi-cpp pull request #432: MINIFICPP-648 - add processor and add pro...
Github user phrocker commented on a diff in the pull request:
https://github.com/apache/nifi-minifi-cpp/pull/432#discussion_r231865773
--- Diff: nanofi/src/api/nanofi.cpp ---
@@ -89,6 +93,39 @@ nifi_instance *create_instance(const char *url,
nifi_port *port) {
return instance;
}
+standalone_processor *create_processor(const char *name) {
+ static int proc_counter = 0;
+ auto ptr = ExecutionPlan::createProcessor(name, name);
+ if (!ptr) {
+return nullptr;
+ }
+ if (standalone_instance == nullptr) {
+nifi_port port;
--- End diff --
this is an input port identifier.
---
[GitHub] nifi-minifi-cpp pull request #432: MINIFICPP-648 - add processor and add pro...
Github user phrocker commented on a diff in the pull request:
https://github.com/apache/nifi-minifi-cpp/pull/432#discussion_r231868999
--- Diff: nanofi/src/cxx/Plan.cpp ---
@@ -162,6 +153,21 @@ bool
ExecutionPlan::runNextProcessor(std::function current_session =
std::make_shared(context);
process_sessions_.push_back(current_session);
+ if (input_ff != nullptr) {
+auto content_repo =
static_cast*>(input_ff->crp);
+std::shared_ptr claim =
std::make_shared(input_ff->contentLocation,
*content_repo);
+auto stream = (*content_repo)->read(claim);
+std::shared_ptr flowFile =
std::static_pointer_cast(current_session->create());
+auto map = static_cast*>(input_ff->attributes);
+if (map) {
+ for (const auto& attr: *map) {
+flowFile->setAttribute(attr.first, attr.second);
+ }
+}
+current_session->importFrom(*stream, flowFile);
+current_session->transfer(flowFile, core::Relationship("success",
"success"));
+
relationships_[relationships_.size()-1]->put(std::static_pointer_cast(flowFile));
--- End diff --
is there a safeguard to ensure a relationship exists?
---
[GitHub] nifi-minifi-cpp pull request #432: MINIFICPP-648 - add processor and add pro...
Github user phrocker commented on a diff in the pull request: https://github.com/apache/nifi-minifi-cpp/pull/432#discussion_r231864832 --- Diff: nanofi/include/core/cxxstructs.h --- @@ -0,0 +1,41 @@ +/** + * + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#ifndef NIFI_MINIFI_CPP_CXXSTRUCTS_H +#define NIFI_MINIFI_CPP_CXXSTRUCTS_H + +#include "cstructs.h" +#include "cxx/Plan.h" + --- End diff -- A lot of this works seems counter to tickets to move away from C++. These structs were created extending C++ classes. and the use of cxx structs seems unnecessary as a result. ---
[GitHub] nifi-minifi-cpp pull request #432: MINIFICPP-648 - add processor and add pro...
Github user phrocker commented on a diff in the pull request:
https://github.com/apache/nifi-minifi-cpp/pull/432#discussion_r231865428
--- Diff: nanofi/include/cxx/Plan.h ---
@@ -133,8 +133,29 @@ class ExecutionPlan {
next_ff_ = ptr;
}
+ bool hasProcessor() {
+return !processor_queue_.empty();
+ }
+
static std::shared_ptr createProcessor(const
std::string &processor_name, const std::string &name);
+ static std::shared_ptr getPlan(const std::string& uuid) {
+auto it = proc_plan_map_.find(uuid);
+return it != proc_plan_map_.end() ? it->second : nullptr;
+ }
+
+ static void addProcWithPlan(const std::string& uuid,
std::shared_ptr plan) {
--- End diff --
addProcessorWithPlan
---
[GitHub] nifi-minifi-cpp pull request #432: MINIFICPP-648 - add processor and add pro...
Github user arpadboda commented on a diff in the pull request:
https://github.com/apache/nifi-minifi-cpp/pull/432#discussion_r231876679
--- Diff: nanofi/include/cxx/CallbackProcessor.h ---
@@ -65,7 +65,7 @@ class CallbackProcessor : public core::Processor {
public:
- void setCallback(void *obj,std::function
ontrigger_callback) {
+ void setCallback(void *obj,std::function
ontrigger_callback) {
--- End diff --
Yes, tested and works.
---
[GitHub] nifi-minifi-cpp pull request #432: MINIFICPP-648 - add processor and add pro...
Github user arpadboda commented on a diff in the pull request: https://github.com/apache/nifi-minifi-cpp/pull/432#discussion_r231878067 --- Diff: nanofi/include/core/cxxstructs.h --- @@ -0,0 +1,41 @@ +/** + * + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#ifndef NIFI_MINIFI_CPP_CXXSTRUCTS_H +#define NIFI_MINIFI_CPP_CXXSTRUCTS_H + +#include "cstructs.h" +#include "cxx/Plan.h" + --- End diff -- I don't think is moving that way, just simplifies the current structure. There is no change in API, no change in the background C++ classes, just removed the C struct we used to interconnect. In case the user doesn't know anything about the internal layout (the C struct only contains a void*), creating a struct just adds an extra hop in the memory, but no value. With the change the user only sees these structures declared, but not defined and gets pointers. The reason we can do it is that the API user doesn't need to know the size or the internal representation of the object in case only gets and passes pointers. We can change the internal layout as we want without impacting the API anyhow. Even C structs can be added later, that's fine. ---
[GitHub] nifi-minifi-cpp pull request #432: MINIFICPP-648 - add processor and add pro...
Github user arpadboda commented on a diff in the pull request:
https://github.com/apache/nifi-minifi-cpp/pull/432#discussion_r231879717
--- Diff: nanofi/src/api/nanofi.cpp ---
@@ -89,6 +93,39 @@ nifi_instance *create_instance(const char *url,
nifi_port *port) {
return instance;
}
+standalone_processor *create_processor(const char *name) {
+ static int proc_counter = 0;
+ auto ptr = ExecutionPlan::createProcessor(name, name);
+ if (!ptr) {
+return nullptr;
+ }
+ if (standalone_instance == nullptr) {
+nifi_port port;
--- End diff --
Created with dummy value. Do you think having a valid input port is valid
usecase for an instance we don't even expose? (As we use it to "store"
standalone processors)
---
[GitHub] nifi-minifi-cpp pull request #432: MINIFICPP-648 - add processor and add pro...
Github user arpadboda commented on a diff in the pull request: https://github.com/apache/nifi-minifi-cpp/pull/432#discussion_r231880575 --- Diff: nanofi/include/cxx/Plan.h --- @@ -44,7 +44,7 @@ #include "core/ProcessSession.h" #include "core/ProcessorNode.h" #include "core/reporting/SiteToSiteProvenanceReportingTask.h" -#include "core/cstructs.h" --- End diff -- Thanks, done. ---
[GitHub] nifi-minifi-cpp pull request #432: MINIFICPP-648 - add processor and add pro...
Github user arpadboda commented on a diff in the pull request:
https://github.com/apache/nifi-minifi-cpp/pull/432#discussion_r231884707
--- Diff: nanofi/src/api/nanofi.cpp ---
@@ -323,55 +360,45 @@ int transmit_flowfile(flow_file_record *ff,
nifi_instance *instance) {
flow * create_new_flow(nifi_instance * instance) {
auto minifi_instance_ref =
static_cast(instance->instance_ptr);
- flow *new_flow = (flow*) malloc(sizeof(flow));
-
- auto execution_plan = new
ExecutionPlan(minifi_instance_ref->getContentRepository(),
minifi_instance_ref->getNoOpRepository(),
minifi_instance_ref->getNoOpRepository());
-
- new_flow->plan = execution_plan;
-
- return new_flow;
+ return new flow(minifi_instance_ref->getContentRepository(),
minifi_instance_ref->getNoOpRepository(),
minifi_instance_ref->getNoOpRepository());
--- End diff --
Multiple thins to add there:
- Please note that this is not an introduced "new" call, just the one 5
lines above.
- We can change this to use malloc and placement new, in which case it
ensures that free only causes some leak, but no crash.
- I don't think this approach is sustainable in long term. To properly free
structures the users would be required to know the internal layout of the
structure, which is not going to happen and it would lead to very complex free
strategies in case of shared ownerships. Moreover I don't think that the
approach "feel free to free, it's just a leak" is the way forward. One of the
reasons I only left declarations in cstructs.h is that freeing something you
don't even know should make you raise concerns. What I would really push is the
user freeing up all the stuff he allocated and we provide free functions for
every structure that the API may return. Don't free what you don't allocate is
quite clear and well defines responsibilities. To go even further here, letting
the users free things might be a painful barrier in case we plan to change the
internal representation, which they shouldn't be aware of anyway.
In nutshell, combining free with placement new is good enough to avoid
immediate crashes, but we should try to highlight that the structures allocated
by the API should also be freed by that.
---
[GitHub] nifi-minifi-cpp pull request #432: MINIFICPP-648 - add processor and add pro...
Github user arpadboda commented on a diff in the pull request:
https://github.com/apache/nifi-minifi-cpp/pull/432#discussion_r231886702
--- Diff: nanofi/src/cxx/Plan.cpp ---
@@ -162,6 +153,21 @@ bool
ExecutionPlan::runNextProcessor(std::function current_session =
std::make_shared(context);
process_sessions_.push_back(current_session);
+ if (input_ff != nullptr) {
+auto content_repo =
static_cast*>(input_ff->crp);
+std::shared_ptr claim =
std::make_shared(input_ff->contentLocation,
*content_repo);
+auto stream = (*content_repo)->read(claim);
+std::shared_ptr flowFile =
std::static_pointer_cast(current_session->create());
+auto map = static_cast*>(input_ff->attributes);
+if (map) {
+ for (const auto& attr: *map) {
+flowFile->setAttribute(attr.first, attr.second);
+ }
+}
+current_session->importFrom(*stream, flowFile);
+current_session->transfer(flowFile, core::Relationship("success",
"success"));
+
relationships_[relationships_.size()-1]->put(std::static_pointer_cast(flowFile));
--- End diff --
The finalise() call introduces a new relationship anyway, that's the only
one we expect to have in case of standalone processors.
---
[GitHub] nifi-minifi-cpp pull request #432: MINIFICPP-648 - add processor and add pro...
Github user arpadboda commented on a diff in the pull request:
https://github.com/apache/nifi-minifi-cpp/pull/432#discussion_r231886506
--- Diff: nanofi/src/api/nanofi.cpp ---
@@ -323,55 +360,45 @@ int transmit_flowfile(flow_file_record *ff,
nifi_instance *instance) {
flow * create_new_flow(nifi_instance * instance) {
auto minifi_instance_ref =
static_cast(instance->instance_ptr);
- flow *new_flow = (flow*) malloc(sizeof(flow));
-
- auto execution_plan = new
ExecutionPlan(minifi_instance_ref->getContentRepository(),
minifi_instance_ref->getNoOpRepository(),
minifi_instance_ref->getNoOpRepository());
-
- new_flow->plan = execution_plan;
-
- return new_flow;
+ return new flow(minifi_instance_ref->getContentRepository(),
minifi_instance_ref->getNoOpRepository(),
minifi_instance_ref->getNoOpRepository());
}
flow *create_flow(nifi_instance *instance, const char *first_processor) {
if (nullptr == instance || nullptr == instance->instance_ptr) {
return nullptr;
}
auto minifi_instance_ref =
static_cast(instance->instance_ptr);
- flow *new_flow = (flow*) malloc(sizeof(flow));
-
- auto execution_plan = new
ExecutionPlan(minifi_instance_ref->getContentRepository(),
minifi_instance_ref->getNoOpRepository(),
minifi_instance_ref->getNoOpRepository());
- new_flow->plan = execution_plan;
+ flow *new_flow = new flow(minifi_instance_ref->getContentRepository(),
minifi_instance_ref->getNoOpRepository(),
minifi_instance_ref->getNoOpRepository());
if (first_processor != nullptr && strlen(first_processor) > 0) {
// automatically adds it with success
-execution_plan->addProcessor(first_processor, first_processor);
--- End diff --
I guess that only declaring structures via the API leaves us the freedom to
change the internal representation anyhow without breaking it. On the other
side it's now faster than it was.
---
[GitHub] nifi-minifi-cpp pull request #432: MINIFICPP-648 - add processor and add pro...
Github user arpadboda commented on a diff in the pull request:
https://github.com/apache/nifi-minifi-cpp/pull/432#discussion_r231889164
--- Diff: nanofi/include/cxx/Plan.h ---
@@ -133,8 +133,29 @@ class ExecutionPlan {
next_ff_ = ptr;
}
+ bool hasProcessor() {
+return !processor_queue_.empty();
+ }
+
static std::shared_ptr createProcessor(const
std::string &processor_name, const std::string &name);
+ static std::shared_ptr getPlan(const std::string& uuid) {
+auto it = proc_plan_map_.find(uuid);
+return it != proc_plan_map_.end() ? it->second : nullptr;
+ }
+
+ static void addProcWithPlan(const std::string& uuid,
std::shared_ptr plan) {
--- End diff --
Renamed
---
[GitHub] nifi issue #3130: NIFI-5791: Add Apache Daffodil (incubating) bundle
Github user stevedlawrence commented on the issue: https://github.com/apache/nifi/pull/3130 Actually, I should probably add a little more clarification. It is correct to say that Daffodil could be considered a transformation capability, but I'm not sure it could really be seen as an alternative to Jolt specifically. One could definitely write a DFDL schema to describe and parse JSON data, similar to what Jolt can do, but the resulting infoset would just be JSON that is probably more generic and less useful that what Jolt would create. Daffodil does not have the same transformational capabilities that Jolt has (e.g. filtering, sorting, reordering). It really just takes a data format that is not natively supported and transforms it to a logically similar representation in a natively supported format (i.e. XML/JSON). If your data is already in something like XML or JSON, it doesn't really make sense to use Daffodil. So for example, you might take PCAP data, like in the above template, and feed it through Daffodill to transform it to JSON. But there might be extraneous data that Daffodil parsed that you don't need or care about. So you might then feed that JSON through Jolt to to transform it to something that has only the fields you care about, or is reordered/renamed to be something more useful to ones usecase. So this isn't something that could be viewed as an alternative to Jolt, but more like something that could be used in conjunction with Jolt to allow one to transform JSON data that wasn't originally JSON. ---
[GitHub] nifi issue #3130: NIFI-5791: Add Apache Daffodil (incubating) bundle
Github user ottobackwards commented on the issue: https://github.com/apache/nifi/pull/3130 Thanks for taking the time to explain further. I see what you mean. Nifi needs a better way to get across to users when to use what processor in general. ---
[GitHub] nifi issue #3133: NIFI-5790: Exposes 6 commons-dbcp options in DBCPConnectio...
Github user colindean commented on the issue: https://github.com/apache/nifi/pull/3133 Test failure is on the Japanese build. ---
[GitHub] nifi pull request #3156: NIFI-5780 Add pre and post statements to ExecuteSQL...
Github user colindean commented on a diff in the pull request:
https://github.com/apache/nifi/pull/3156#discussion_r231993409
--- Diff:
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/AbstractExecuteSQL.java
---
@@ -82,6 +84,16 @@
.identifiesControllerService(DBCPService.class)
.build();
+public static final PropertyDescriptor SQL_PRE_QUERY = new
PropertyDescriptor.Builder()
+.name("sql-pre-query")
+.displayName("SQL Pre-Query")
+.description("SQL pre-query to execute. Semicolon-delimited
list of queries. "
--- End diff --
"SQL pre-query to execute" is duplicative with the name. Can you explain in
another way its intent? Perhaps something like, "Queries that will be executed
before each intended query is executed."
---
[GitHub] nifi pull request #3156: NIFI-5780 Add pre and post statements to ExecuteSQL...
Github user colindean commented on a diff in the pull request:
https://github.com/apache/nifi/pull/3156#discussion_r231993705
--- Diff:
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/AbstractExecuteSQL.java
---
@@ -94,6 +106,16 @@
.expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
.build();
+public static final PropertyDescriptor SQL_POST_QUERY = new
PropertyDescriptor.Builder()
+.name("sql-post-query")
+.displayName("SQL Post-Query")
+.description("SQL post-query to execute. Semicolon-delimited
list of queries. "
--- End diff --
Likewise, this description is duplicative with the name. "Queries that will
be executed after the intended query."
---
[GitHub] nifi pull request #3156: NIFI-5780 Add pre and post statements to ExecuteSQL...
Github user colindean commented on a diff in the pull request:
https://github.com/apache/nifi/pull/3156#discussion_r231993492
--- Diff:
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/AbstractExecuteSQL.java
---
@@ -82,6 +84,16 @@
.identifiesControllerService(DBCPService.class)
.build();
+public static final PropertyDescriptor SQL_PRE_QUERY = new
PropertyDescriptor.Builder()
+.name("sql-pre-query")
+.displayName("SQL Pre-Query")
+.description("SQL pre-query to execute. Semicolon-delimited
list of queries. "
++ "Note, the results/outputs of these queries will be
suppressed if successfully executed.")
--- End diff --
You can probably omit "Note," here.
---
[jira] [Commented] (NIFI-5780) Add pre and post statements to ExecuteSQL and ExecuteSQLRecord
[
https://issues.apache.org/jira/browse/NIFI-5780?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16680074#comment-16680074
]
ASF GitHub Bot commented on NIFI-5780:
--
Github user colindean commented on a diff in the pull request:
https://github.com/apache/nifi/pull/3156#discussion_r231993705
--- Diff:
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/AbstractExecuteSQL.java
---
@@ -94,6 +106,16 @@
.expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
.build();
+public static final PropertyDescriptor SQL_POST_QUERY = new
PropertyDescriptor.Builder()
+.name("sql-post-query")
+.displayName("SQL Post-Query")
+.description("SQL post-query to execute. Semicolon-delimited
list of queries. "
--- End diff --
Likewise, this description is duplicative with the name. "Queries that will
be executed after the intended query."
> Add pre and post statements to ExecuteSQL and ExecuteSQLRecord
> --
>
> Key: NIFI-5780
> URL: https://issues.apache.org/jira/browse/NIFI-5780
> Project: Apache NiFi
> Issue Type: Improvement
>Affects Versions: 1.8.0
>Reporter: Deon Huang
>Assignee: Deon Huang
>Priority: Minor
>
> Sometimes we might need to set up session relate configuration before or
> after query.
> For example:
> Pre query can be used for session relate setting like our use case Teradata
> Query Banding.
> Same feature (pre query and post query) is added to SelectHiveQL in
> https://issues.apache.org/jira/browse/NIFI-5044
> Planning to add this feature to ExecuteSQL and ExecuteSQLRecord processors.
> If pre or post statement fail, will not produce resultset flowfile.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (NIFI-5780) Add pre and post statements to ExecuteSQL and ExecuteSQLRecord
[
https://issues.apache.org/jira/browse/NIFI-5780?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16680075#comment-16680075
]
ASF GitHub Bot commented on NIFI-5780:
--
Github user colindean commented on a diff in the pull request:
https://github.com/apache/nifi/pull/3156#discussion_r231993492
--- Diff:
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/AbstractExecuteSQL.java
---
@@ -82,6 +84,16 @@
.identifiesControllerService(DBCPService.class)
.build();
+public static final PropertyDescriptor SQL_PRE_QUERY = new
PropertyDescriptor.Builder()
+.name("sql-pre-query")
+.displayName("SQL Pre-Query")
+.description("SQL pre-query to execute. Semicolon-delimited
list of queries. "
++ "Note, the results/outputs of these queries will be
suppressed if successfully executed.")
--- End diff --
You can probably omit "Note," here.
> Add pre and post statements to ExecuteSQL and ExecuteSQLRecord
> --
>
> Key: NIFI-5780
> URL: https://issues.apache.org/jira/browse/NIFI-5780
> Project: Apache NiFi
> Issue Type: Improvement
>Affects Versions: 1.8.0
>Reporter: Deon Huang
>Assignee: Deon Huang
>Priority: Minor
>
> Sometimes we might need to set up session relate configuration before or
> after query.
> For example:
> Pre query can be used for session relate setting like our use case Teradata
> Query Banding.
> Same feature (pre query and post query) is added to SelectHiveQL in
> https://issues.apache.org/jira/browse/NIFI-5044
> Planning to add this feature to ExecuteSQL and ExecuteSQLRecord processors.
> If pre or post statement fail, will not produce resultset flowfile.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (NIFI-5780) Add pre and post statements to ExecuteSQL and ExecuteSQLRecord
[
https://issues.apache.org/jira/browse/NIFI-5780?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16680073#comment-16680073
]
ASF GitHub Bot commented on NIFI-5780:
--
Github user colindean commented on a diff in the pull request:
https://github.com/apache/nifi/pull/3156#discussion_r231993409
--- Diff:
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/AbstractExecuteSQL.java
---
@@ -82,6 +84,16 @@
.identifiesControllerService(DBCPService.class)
.build();
+public static final PropertyDescriptor SQL_PRE_QUERY = new
PropertyDescriptor.Builder()
+.name("sql-pre-query")
+.displayName("SQL Pre-Query")
+.description("SQL pre-query to execute. Semicolon-delimited
list of queries. "
--- End diff --
"SQL pre-query to execute" is duplicative with the name. Can you explain in
another way its intent? Perhaps something like, "Queries that will be executed
before each intended query is executed."
> Add pre and post statements to ExecuteSQL and ExecuteSQLRecord
> --
>
> Key: NIFI-5780
> URL: https://issues.apache.org/jira/browse/NIFI-5780
> Project: Apache NiFi
> Issue Type: Improvement
>Affects Versions: 1.8.0
>Reporter: Deon Huang
>Assignee: Deon Huang
>Priority: Minor
>
> Sometimes we might need to set up session relate configuration before or
> after query.
> For example:
> Pre query can be used for session relate setting like our use case Teradata
> Query Banding.
> Same feature (pre query and post query) is added to SelectHiveQL in
> https://issues.apache.org/jira/browse/NIFI-5044
> Planning to add this feature to ExecuteSQL and ExecuteSQLRecord processors.
> If pre or post statement fail, will not produce resultset flowfile.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (MINIFICPP-648) add processor and add processor with linkage nomenclature is confusing
[
https://issues.apache.org/jira/browse/MINIFICPP-648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16679720#comment-16679720
]
ASF GitHub Bot commented on MINIFICPP-648:
--
Github user arpadboda commented on a diff in the pull request:
https://github.com/apache/nifi-minifi-cpp/pull/432#discussion_r231876679
--- Diff: nanofi/include/cxx/CallbackProcessor.h ---
@@ -65,7 +65,7 @@ class CallbackProcessor : public core::Processor {
public:
- void setCallback(void *obj,std::function
ontrigger_callback) {
+ void setCallback(void *obj,std::function
ontrigger_callback) {
--- End diff --
Yes, tested and works.
> add processor and add processor with linkage nomenclature is confusing
> --
>
> Key: MINIFICPP-648
> URL: https://issues.apache.org/jira/browse/MINIFICPP-648
> Project: NiFi MiNiFi C++
> Issue Type: Improvement
>Reporter: Mr TheSegfault
>Assignee: Arpad Boda
>Priority: Blocker
> Labels: CAPI
> Fix For: 0.6.0
>
>
> add_processor should be changed to always add a processor with linkage
> without compelling documentation as why this exists.. As a result we will
> need to add a create_processor function to create one without adding it to
> the flow ( certain use cases where a flow isn't needed such as invokehttp or
> listenhttp ) this can be moved to 0.7.0 if we tag before recent commits.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (NIFI-5791) Add Apache Daffodil parse/unparse processor
[ https://issues.apache.org/jira/browse/NIFI-5791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16680032#comment-16680032 ] ASF GitHub Bot commented on NIFI-5791: -- Github user ottobackwards commented on the issue: https://github.com/apache/nifi/pull/3130 Thanks for taking the time to explain further. I see what you mean. Nifi needs a better way to get across to users when to use what processor in general. > Add Apache Daffodil parse/unparse processor > --- > > Key: NIFI-5791 > URL: https://issues.apache.org/jira/browse/NIFI-5791 > Project: Apache NiFi > Issue Type: Bug > Components: Extensions >Reporter: Steve Lawrence >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-5798) FlattenJson improperly escapes special characters
[ https://issues.apache.org/jira/browse/NIFI-5798?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16679424#comment-16679424 ] ASF subversion and git services commented on NIFI-5798: --- Commit 3ba0c0ca82b6bf2eb7d855e2b9025a7a9ccb324c in nifi's branch refs/heads/master from [~markap14] [ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=3ba0c0c ] NIFI-5798: Fixed bug in FlattenJson that was escaping text as Java instead of escaping as JSON Signed-off-by: Pierre Villard This closes #3138. > FlattenJson improperly escapes special characters > - > > Key: NIFI-5798 > URL: https://issues.apache.org/jira/browse/NIFI-5798 > Project: Apache NiFi > Issue Type: Bug > Components: Extensions >Reporter: Mark Payne >Assignee: Mark Payne >Priority: Major > Fix For: 1.9.0 > > > FlattenJSON uses a "String Escape Policy" of ESCAPE_JAVA instead of > ESCAPE_JSON. This results in valid JSON characters getting escaped as hex > characters. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (NIFI-5803) ExecuteInfluxDBQuery
sam created NIFI-5803: - Summary: ExecuteInfluxDBQuery Key: NIFI-5803 URL: https://issues.apache.org/jira/browse/NIFI-5803 Project: Apache NiFi Issue Type: Bug Environment: Linux,I think that ExecuteInfluxDBQuery has a bug, because when I put to variable in the field "InfluxDB connection URL" always is empty. Reporter: sam Attachments: ExecuteInfluxDBQuery.PNG -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-5790) DBCPConnectionPool configuration should expose underlying connection idle and eviction configuration
[
https://issues.apache.org/jira/browse/NIFI-5790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16680067#comment-16680067
]
ASF GitHub Bot commented on NIFI-5790:
--
Github user colindean commented on the issue:
https://github.com/apache/nifi/pull/3133
Test failure is on the Japanese build.
> DBCPConnectionPool configuration should expose underlying connection idle and
> eviction configuration
>
>
> Key: NIFI-5790
> URL: https://issues.apache.org/jira/browse/NIFI-5790
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Extensions
>Affects Versions: 1.8.0
>Reporter: Colin Dean
>Priority: Major
> Labels: DBCP, database
>
> While investigating a fix for NIFI-5789, I noticed in the [DBCPConnectionPool
> documentation|https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-dbcp-service-nar/1.8.0/org.apache.nifi.dbcp.DBCPConnectionPool/index.html]
> that NiFi appears _not_ to have controller service configuration options
> associated with [Apache
> Commons-DBCP|https://commons.apache.org/proper/commons-dbcp/configuration.html]
> {{BasicDataSource}} parameters like {{minIdle}} and {{maxIdle}}, which I
> think should be both set to 0 in my particular use case.
> Alternatively, I think I could set {{maxConnLifetimeMillis}} to something
> even in the minutes range and satisfy my use case (a connection need not be
> released _immediately_ but within a reasonable period of time), but this
> option is also not available.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (NIFI-5798) FlattenJson improperly escapes special characters
[ https://issues.apache.org/jira/browse/NIFI-5798?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16679425#comment-16679425 ] ASF GitHub Bot commented on NIFI-5798: -- Github user pvillard31 commented on the issue: https://github.com/apache/nifi/pull/3138 +1, merging to master, thanks! > FlattenJson improperly escapes special characters > - > > Key: NIFI-5798 > URL: https://issues.apache.org/jira/browse/NIFI-5798 > Project: Apache NiFi > Issue Type: Bug > Components: Extensions >Reporter: Mark Payne >Assignee: Mark Payne >Priority: Major > Fix For: 1.9.0 > > > FlattenJSON uses a "String Escape Policy" of ESCAPE_JAVA instead of > ESCAPE_JSON. This results in valid JSON characters getting escaped as hex > characters. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-5797) FlattenJson processor converts special characters to hex
[
https://issues.apache.org/jira/browse/NIFI-5797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16679428#comment-16679428
]
ASF GitHub Bot commented on NIFI-5797:
--
Github user pvillard31 commented on the issue:
https://github.com/apache/nifi/pull/3137
I believe this has been fixed by #3138. If you agree, can you close this PR?
> FlattenJson processor converts special characters to hex
>
>
> Key: NIFI-5797
> URL: https://issues.apache.org/jira/browse/NIFI-5797
> Project: Apache NiFi
> Issue Type: Bug
> Components: Extensions
>Affects Versions: 1.9.0
>Reporter: Ravi Bhardwaj
>Assignee: Mark Payne
>Priority: Major
> Fix For: 1.9.0
>
>
> FlattenJson will convert the special characters in the json to hex format
>
> Example Input:
> {"name": "http://localhost:8080/nifi","full": \{ "name": "José Muñoz"}
> Output:
> {"name":"http://localhost:8080/nifi","full.name":"Jos\u00E9 Mu\u00F1oz"}
> Expected Output:
> {"name":"http://localhost:8080/nifi","full.name":"José Muñoz"}
>
> Possibly regression from NIFI-4962
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (MINIFICPP-648) add processor and add processor with linkage nomenclature is confusing
[ https://issues.apache.org/jira/browse/MINIFICPP-648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16679744#comment-16679744 ] ASF GitHub Bot commented on MINIFICPP-648: -- Github user arpadboda commented on a diff in the pull request: https://github.com/apache/nifi-minifi-cpp/pull/432#discussion_r231880575 --- Diff: nanofi/include/cxx/Plan.h --- @@ -44,7 +44,7 @@ #include "core/ProcessSession.h" #include "core/ProcessorNode.h" #include "core/reporting/SiteToSiteProvenanceReportingTask.h" -#include "core/cstructs.h" --- End diff -- Thanks, done. > add processor and add processor with linkage nomenclature is confusing > -- > > Key: MINIFICPP-648 > URL: https://issues.apache.org/jira/browse/MINIFICPP-648 > Project: NiFi MiNiFi C++ > Issue Type: Improvement >Reporter: Mr TheSegfault >Assignee: Arpad Boda >Priority: Blocker > Labels: CAPI > Fix For: 0.6.0 > > > add_processor should be changed to always add a processor with linkage > without compelling documentation as why this exists.. As a result we will > need to add a create_processor function to create one without adding it to > the flow ( certain use cases where a flow isn't needed such as invokehttp or > listenhttp ) this can be moved to 0.7.0 if we tag before recent commits. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-5798) FlattenJson improperly escapes special characters
[ https://issues.apache.org/jira/browse/NIFI-5798?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16679426#comment-16679426 ] ASF GitHub Bot commented on NIFI-5798: -- Github user asfgit closed the pull request at: https://github.com/apache/nifi/pull/3138 > FlattenJson improperly escapes special characters > - > > Key: NIFI-5798 > URL: https://issues.apache.org/jira/browse/NIFI-5798 > Project: Apache NiFi > Issue Type: Bug > Components: Extensions >Reporter: Mark Payne >Assignee: Mark Payne >Priority: Major > Fix For: 1.9.0 > > > FlattenJSON uses a "String Escape Policy" of ESCAPE_JAVA instead of > ESCAPE_JSON. This results in valid JSON characters getting escaped as hex > characters. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi pull request #3133: NIFI-5790: Exposes 6 commons-dbcp options in DBCPCo...
Github user mattyb149 commented on a diff in the pull request:
https://github.com/apache/nifi/pull/3133#discussion_r232011596
--- Diff:
nifi-nar-bundles/nifi-standard-services/nifi-dbcp-service-bundle/nifi-dbcp-service/src/main/java/org/apache/nifi/dbcp/DBCPConnectionPool.java
---
@@ -164,6 +166,77 @@ public ValidationResult validate(final String subject,
final String input, final
.expressionLanguageSupported(ExpressionLanguageScope.VARIABLE_REGISTRY)
.build();
+public static final PropertyDescriptor MIN_IDLE = new
PropertyDescriptor.Builder()
+.displayName("Minimum Idle Connections")
+.name("dbcp-mim-idle-conns")
+.description("The minimum number of connections that can
remain idle in the pool, without extra ones being " +
+"created, or zero to create none.")
+
.defaultValue(String.valueOf(GenericObjectPoolConfig.DEFAULT_MIN_IDLE))
--- End diff --
I didn't realize the constants were only available from a class in an
"impl" package in DBCP. In that case I take back what I said :( Can we create
our own constants here with a comment that we got the values from DBCP itself?
---
[jira] [Commented] (MINIFICPP-648) add processor and add processor with linkage nomenclature is confusing
[ https://issues.apache.org/jira/browse/MINIFICPP-648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16679687#comment-16679687 ] ASF GitHub Bot commented on MINIFICPP-648: -- Github user phrocker commented on a diff in the pull request: https://github.com/apache/nifi-minifi-cpp/pull/432#discussion_r231865240 --- Diff: nanofi/include/cxx/Plan.h --- @@ -44,7 +44,7 @@ #include "core/ProcessSession.h" #include "core/ProcessorNode.h" #include "core/reporting/SiteToSiteProvenanceReportingTask.h" -#include "core/cstructs.h" --- End diff -- Should remove the rem'd line. > add processor and add processor with linkage nomenclature is confusing > -- > > Key: MINIFICPP-648 > URL: https://issues.apache.org/jira/browse/MINIFICPP-648 > Project: NiFi MiNiFi C++ > Issue Type: Improvement >Reporter: Mr TheSegfault >Assignee: Arpad Boda >Priority: Blocker > Labels: CAPI > Fix For: 0.6.0 > > > add_processor should be changed to always add a processor with linkage > without compelling documentation as why this exists.. As a result we will > need to add a create_processor function to create one without adding it to > the flow ( certain use cases where a flow isn't needed such as invokehttp or > listenhttp ) this can be moved to 0.7.0 if we tag before recent commits. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (NIFI-5804) Data Provenance not showing any results
Jelena Lazarevic created NIFI-5804: -- Summary: Data Provenance not showing any results Key: NIFI-5804 URL: https://issues.apache.org/jira/browse/NIFI-5804 Project: Apache NiFi Issue Type: Bug Affects Versions: 1.8.0 Environment: docker image apache/nifi:latest Reporter: Jelena Lazarevic Fix For: 1.7.1 Attachments: image-2018-11-08-14-51-04-292.png !image-2018-11-08-14-51-04-292.png! -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-5802) QueryRecord ignores field nullability and causes NPE with null input value
[
https://issues.apache.org/jira/browse/NIFI-5802?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16679239#comment-16679239
]
ASF GitHub Bot commented on NIFI-5802:
--
GitHub user ijokarumawak opened a pull request:
https://github.com/apache/nifi/pull/3158
NIFI-5802: Add QueryRecord nullable field support
Thank you for submitting a contribution to Apache NiFi.
In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:
### For all changes:
- [x] Is there a JIRA ticket associated with this PR? Is it referenced
in the commit message?
- [x] Does your PR title start with NIFI- where is the JIRA number
you are trying to resolve? Pay particular attention to the hyphen "-" character.
- [x] Has your PR been rebased against the latest commit within the target
branch (typically master)?
- [x] Is your initial contribution a single, squashed commit?
### For code changes:
- [x] Have you ensured that the full suite of tests is executed via mvn
-Pcontrib-check clean install at the root nifi folder?
- [x] Have you written or updated unit tests to verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies
licensed in a way that is compatible for inclusion under [ASF
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] If applicable, have you updated the LICENSE file, including the main
LICENSE file under nifi-assembly?
- [ ] If applicable, have you updated the NOTICE file, including the main
NOTICE file found under nifi-assembly?
- [ ] If adding new Properties, have you added .displayName in addition to
.name (programmatic access) for each of the new properties?
### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in
which it is rendered?
### Note:
Please ensure that once the PR is submitted, you check travis-ci for build
issues and submit an update to your PR as soon as possible.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/ijokarumawak/nifi nifi-5802
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/nifi/pull/3158.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #3158
commit 8bf29d4c56fb1f507d987c8aea774e5345b286f7
Author: Koji Kawamura
Date: 2018-11-08T03:10:36Z
NIFI-5802: Add QueryRecord nullable field support
> QueryRecord ignores field nullability and causes NPE with null input value
> --
>
> Key: NIFI-5802
> URL: https://issues.apache.org/jira/browse/NIFI-5802
> Project: Apache NiFi
> Issue Type: Bug
> Components: Extensions
>Affects Versions: 1.2.0
>Reporter: Koji Kawamura
>Assignee: Koji Kawamura
>Priority: Major
>
> WIth Avro schema, a field can be defined as nullable by a union type. For
> example:
> {code:java}
> "type": [ "null", "boolean" ]
> {code}
> QueryRecord generates relational data types from the record schema. But it
> currently doesn't utilize the nullable information. And it leads to a
> NullPointerException when the incoming FlowFile has null values for the
> nullable fields.
> A simple 'select * from FLOWFILE' can fail, if the AvroRecordSetWriter is
> used.
> This issue was reported to nifi user ML. See the message for detail.
>
> [https://mail-archives.apache.org/mod_mbox/nifi-users/201811.mbox/%3CCAB10SGVgW2NR76UR1pYJvEZYTO3J1T7t_0q%2BUsQANp%3DcHxS-kw%40mail.gmail.com%3E]
> Here is the full stack trace:
> {code}
> org.apache.nifi.processor.exception.ProcessException: IOException thrown from
> QueryRecord[id=f11017d1-0166-1000-344a-c2f1d9bbd933]: java.io.IOException:
> org.apache.avro.file.DataFileWriter$AppendWriteException:
> java.lang.NullPointerException: null of boolean in field bool_test of
> org.apache.nifi.nifiRecord
> at
> org.apache.nifi.controller.repository.StandardProcessSession.write(StandardProcessSession.java:2667)
> at
> org.apache.nifi.processors.standard.QueryRecord.onTrigger(QueryRecord.java:309)
> at
> org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
> at
> org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1168)
> at
> org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:203)
> at
> org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:117)
> at
> java.util.concurrent.Executors$R
[jira] [Commented] (MINIFICPP-648) add processor and add processor with linkage nomenclature is confusing
[ https://issues.apache.org/jira/browse/MINIFICPP-648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16679686#comment-16679686 ] ASF GitHub Bot commented on MINIFICPP-648: -- Github user phrocker commented on a diff in the pull request: https://github.com/apache/nifi-minifi-cpp/pull/432#discussion_r231864832 --- Diff: nanofi/include/core/cxxstructs.h --- @@ -0,0 +1,41 @@ +/** + * + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#ifndef NIFI_MINIFI_CPP_CXXSTRUCTS_H +#define NIFI_MINIFI_CPP_CXXSTRUCTS_H + +#include "cstructs.h" +#include "cxx/Plan.h" + --- End diff -- A lot of this works seems counter to tickets to move away from C++. These structs were created extending C++ classes. and the use of cxx structs seems unnecessary as a result. > add processor and add processor with linkage nomenclature is confusing > -- > > Key: MINIFICPP-648 > URL: https://issues.apache.org/jira/browse/MINIFICPP-648 > Project: NiFi MiNiFi C++ > Issue Type: Improvement >Reporter: Mr TheSegfault >Assignee: Arpad Boda >Priority: Blocker > Labels: CAPI > Fix For: 0.6.0 > > > add_processor should be changed to always add a processor with linkage > without compelling documentation as why this exists.. As a result we will > need to add a create_processor function to create one without adding it to > the flow ( certain use cases where a flow isn't needed such as invokehttp or > listenhttp ) this can be moved to 0.7.0 if we tag before recent commits. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-5788) Introduce batch size limit in PutDatabaseRecord processor
[ https://issues.apache.org/jira/browse/NIFI-5788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16679466#comment-16679466 ] ASF GitHub Bot commented on NIFI-5788: -- Github user vadimar commented on the issue: https://github.com/apache/nifi/pull/3128 Hi, Can you please review the latest commits? I committed the changes that address all the issues raised by reviewers. Thanks > Introduce batch size limit in PutDatabaseRecord processor > - > > Key: NIFI-5788 > URL: https://issues.apache.org/jira/browse/NIFI-5788 > Project: Apache NiFi > Issue Type: Improvement > Components: Core Framework > Environment: Teradata DB >Reporter: Vadim >Priority: Major > Labels: pull-request-available > > Certain JDBC drivers do not support unlimited batch size in INSERT/UPDATE > prepared SQL statements. Specifically, Teradata JDBC driver > ([https://downloads.teradata.com/download/connectivity/jdbc-driver)] would > fail SQL statement when the batch overflows the internal limits. > Dividing data into smaller chunks before the PutDatabaseRecord is applied can > work around the issue in certain scenarios, but generally, this solution is > not perfect because the SQL statements would be executed in different > transaction contexts and data integrity would not be preserved. > The solution suggests the following: > * introduce a new optional parameter in *PutDatabaseRecord* processor, > *max_batch_size* which defines the maximum batch size in INSERT/UPDATE > statement; the default value zero (INFINITY) preserves the old behavior > * divide the input into batches of the specified size and invoke > PreparedStatement.executeBatch() for each batch > Pull request: [https://github.com/apache/nifi/pull/3128] > > [EDIT] Changed batch_size to max_batch_size. The default value would be zero > (INFINITY) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (NIFI-5802) QueryRecord ignores field nullability and causes NPE with null input value
[
https://issues.apache.org/jira/browse/NIFI-5802?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pierre Villard resolved NIFI-5802.
--
Resolution: Fixed
Fix Version/s: 1.9.0
> QueryRecord ignores field nullability and causes NPE with null input value
> --
>
> Key: NIFI-5802
> URL: https://issues.apache.org/jira/browse/NIFI-5802
> Project: Apache NiFi
> Issue Type: Bug
> Components: Extensions
>Affects Versions: 1.2.0
>Reporter: Koji Kawamura
>Assignee: Koji Kawamura
>Priority: Major
> Fix For: 1.9.0
>
>
> WIth Avro schema, a field can be defined as nullable by a union type. For
> example:
> {code:java}
> "type": [ "null", "boolean" ]
> {code}
> QueryRecord generates relational data types from the record schema. But it
> currently doesn't utilize the nullable information. And it leads to a
> NullPointerException when the incoming FlowFile has null values for the
> nullable fields.
> A simple 'select * from FLOWFILE' can fail, if the AvroRecordSetWriter is
> used.
> This issue was reported to nifi user ML. See the message for detail.
>
> [https://mail-archives.apache.org/mod_mbox/nifi-users/201811.mbox/%3CCAB10SGVgW2NR76UR1pYJvEZYTO3J1T7t_0q%2BUsQANp%3DcHxS-kw%40mail.gmail.com%3E]
> Here is the full stack trace:
> {code}
> org.apache.nifi.processor.exception.ProcessException: IOException thrown from
> QueryRecord[id=f11017d1-0166-1000-344a-c2f1d9bbd933]: java.io.IOException:
> org.apache.avro.file.DataFileWriter$AppendWriteException:
> java.lang.NullPointerException: null of boolean in field bool_test of
> org.apache.nifi.nifiRecord
> at
> org.apache.nifi.controller.repository.StandardProcessSession.write(StandardProcessSession.java:2667)
> at
> org.apache.nifi.processors.standard.QueryRecord.onTrigger(QueryRecord.java:309)
> at
> org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
> at
> org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1168)
> at
> org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:203)
> at
> org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:117)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.io.IOException:
> org.apache.avro.file.DataFileWriter$AppendWriteException:
> java.lang.NullPointerException: null of boolean in field bool_test of
> org.apache.nifi.nifiRecord
> at
> org.apache.nifi.processors.standard.QueryRecord$1.process(QueryRecord.java:327)
> at
> org.apache.nifi.controller.repository.StandardProcessSession.write(StandardProcessSession.java:2648)
> ... 12 common frames omitted
> Caused by: org.apache.avro.file.DataFileWriter$AppendWriteException:
> java.lang.NullPointerException: null of boolean in field bool_test of
> org.apache.nifi.nifiRecord
> at org.apache.avro.file.DataFileWriter.append(DataFileWriter.java:308)
> at
> org.apache.nifi.avro.WriteAvroResultWithSchema.writeRecord(WriteAvroResultWithSchema.java:61)
> at
> org.apache.nifi.serialization.AbstractRecordSetWriter.write(AbstractRecordSetWriter.java:59)
> at
> org.apache.nifi.serialization.AbstractRecordSetWriter.write(AbstractRecordSetWriter.java:52)
> at
> org.apache.nifi.processors.standard.QueryRecord$1.process(QueryRecord.java:324)
> ... 13 common frames omitted
> Caused by: java.lang.NullPointerException: null of boolean in field bool_test
> of org.apache.nifi.nifiRecord
> at
> org.apache.avro.generic.GenericDatumWriter.npe(GenericDatumWriter.java:132)
> at
> org.apache.avro.generic.GenericDatumWriter.writeWithoutConversion(GenericDatumWriter.java:126)
> at
> org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:73)
> at
> org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:60)
> at org.apache.avro.file.DataFileWriter.append(DataFileWriter.java:302)
> ... 17 common frames omitted
> Caused by: java.lang.NullPointerException: null
> at
> org.
[jira] [Commented] (MINIFICPP-648) add processor and add processor with linkage nomenclature is confusing
[
https://issues.apache.org/jira/browse/MINIFICPP-648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16679689#comment-16679689
]
ASF GitHub Bot commented on MINIFICPP-648:
--
Github user phrocker commented on a diff in the pull request:
https://github.com/apache/nifi-minifi-cpp/pull/432#discussion_r23188
--- Diff: nanofi/src/api/nanofi.cpp ---
@@ -323,55 +360,45 @@ int transmit_flowfile(flow_file_record *ff,
nifi_instance *instance) {
flow * create_new_flow(nifi_instance * instance) {
auto minifi_instance_ref =
static_cast(instance->instance_ptr);
- flow *new_flow = (flow*) malloc(sizeof(flow));
-
- auto execution_plan = new
ExecutionPlan(minifi_instance_ref->getContentRepository(),
minifi_instance_ref->getNoOpRepository(),
minifi_instance_ref->getNoOpRepository());
-
- new_flow->plan = execution_plan;
-
- return new_flow;
+ return new flow(minifi_instance_ref->getContentRepository(),
minifi_instance_ref->getNoOpRepository(),
minifi_instance_ref->getNoOpRepository());
--- End diff --
https://issues.apache.org/jira/browse/MINIFICPP-645 exists because public
facing objects that are turned from a C API can be malloc'd by users. If a
consumer of our API doesn't use free as designed, we should guarantee that they
only cause memory leaks, not introduce undefined behavior.
> add processor and add processor with linkage nomenclature is confusing
> --
>
> Key: MINIFICPP-648
> URL: https://issues.apache.org/jira/browse/MINIFICPP-648
> Project: NiFi MiNiFi C++
> Issue Type: Improvement
>Reporter: Mr TheSegfault
>Assignee: Arpad Boda
>Priority: Blocker
> Labels: CAPI
> Fix For: 0.6.0
>
>
> add_processor should be changed to always add a processor with linkage
> without compelling documentation as why this exists.. As a result we will
> need to add a create_processor function to create one without adding it to
> the flow ( certain use cases where a flow isn't needed such as invokehttp or
> listenhttp ) this can be moved to 0.7.0 if we tag before recent commits.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (NIFI-5764) Allow ListSftp connection parameter
[
https://issues.apache.org/jira/browse/NIFI-5764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16679265#comment-16679265
]
Koji Kawamura commented on NIFI-5764:
-
Thanks [~alfredo.deluca]. Excuse me, but I don't have any further comments or
idea to investigate on NiFi side.. I'd suggest investigating more on the SFTP
server, sshd, pam and freeipa side. For example, monitoring your freeipa server
connection can bring new insight to the issue.
https://lists.fedorahosted.org/archives/list/freeipa-us...@lists.fedorahosted.org/thread/7NPSPK3GK3MHBDK22T72NSTMW3525M5H/
> Allow ListSftp connection parameter
> ---
>
> Key: NIFI-5764
> URL: https://issues.apache.org/jira/browse/NIFI-5764
> Project: Apache NiFi
> Issue Type: Wish
> Components: Extensions
>Affects Versions: 1.6.0
>Reporter: dav
>Priority: Critical
> Labels: SFTP, customization, sftp
> Attachments: dumpone
>
>
> ListSftp and other Sftp processors should be able to add parameters
> (like [-B buffer_size] [-b batchfile] [-c cipher]
> [-D sftp_server_path] [-F ssh_config] [-i identity_file] [-l limit]
> [-o ssh_option] [-P port] [-R num_requests] [-S program]
> [-s subsystem | sftp_server] host
> sftp [user@]host[:file ...]
> sftp [user@]host[:dir[/]]
> sftp -b batchfile [user@]host)
> in order to edit the type of connection on Sftp Server.
> For instance, I have this error on nifi:
> 2018-10-29 11:06:09,462 ERROR [Timer-Driven Process Thread-5]
> SimpleProcessLogger.java:254
> ListSFTP[id=766ac418-27ce-335a-5b13-52abe3495592] Failed to perform listing
> on remote host due to java.io.IOException: Failed to obtain connection to
> remote host due to com.jcraft.jsch.JSchException: Auth fail: {}
> java.io.IOException: Failed to obtain connection to remote host due to
> com.jcraft.jsch.JSchException: Auth fail
> at
> org.apache.nifi.processors.standard.util.SFTPTransfer.getChannel(SFTPTransfer.java:468)
> at
> org.apache.nifi.processors.standard.util.SFTPTransfer.getListing(SFTPTransfer.java:192)
> at
> org.apache.nifi.processors.standard.util.SFTPTransfer.getListing(SFTPTransfer.java:156)
> at
> org.apache.nifi.processors.standard.ListFileTransfer.performListing(ListFileTransfer.java:105)
> at
> org.apache.nifi.processor.util.list.AbstractListProcessor.onTrigger(AbstractListProcessor.java:401)
> at
> org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
> at
> org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1147)
> at
> org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:175)
> at
> org.apache.nifi.controller.scheduling.QuartzSchedulingAgent$2.run(QuartzSchedulingAgent.java:140)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: com.jcraft.jsch.JSchException: Auth fail
> at com.jcraft.jsch.Session.connect(Session.java:519)
> at com.jcraft.jsch.Session.connect(Session.java:183)
> at
> org.apache.nifi.processors.standard.util.SFTPTransfer.getChannel(SFTPTransfer.java:448)
> ... 15 common frames omitted
> This can be avoided by connect to Sftp server with this string:
> *sftp -o “controlmaster auto” username@sftp_server*
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (MINIFICPP-641) C API: add support to register failure callback
[ https://issues.apache.org/jira/browse/MINIFICPP-641?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16679634#comment-16679634 ] Arpad Boda commented on MINIFICPP-641: -- Merged in scope of: https://github.com/apache/nifi-minifi-cpp/pull/421 > C API: add support to register failure callback > --- > > Key: MINIFICPP-641 > URL: https://issues.apache.org/jira/browse/MINIFICPP-641 > Project: NiFi MiNiFi C++ > Issue Type: New Feature >Reporter: Arpad Boda >Assignee: Arpad Boda >Priority: Minor > Labels: CAPI > Fix For: 0.6.0 > > > Current implementation of C API provide no option to handle failure > relationships of processors. > Simply adding relationships one-by-one would make it unreasonably complicated > in user's perspective. User should be able to add a simple callback, which - > in the background - creates a processor and connects that all processors in > the given flow with failure relationship. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (NIFI-5800) If RecordSchema has an inner field that references a schema recursively by name, hashCode() throws StackOverflowError
[
https://issues.apache.org/jira/browse/NIFI-5800?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pierre Villard resolved NIFI-5800.
--
Resolution: Fixed
Assignee: Mark Payne
Fix Version/s: 1.9.0
> If RecordSchema has an inner field that references a schema recursively by
> name, hashCode() throws StackOverflowError
> -
>
> Key: NIFI-5800
> URL: https://issues.apache.org/jira/browse/NIFI-5800
> Project: Apache NiFi
> Issue Type: Bug
>Reporter: Mark Payne
>Assignee: Mark Payne
>Priority: Major
> Fix For: 1.9.0
>
>
> If we have a schema where a field is of type RECORD and references an
> outer-level schema, we get StackOverflowError. For example:
> {
> "name": "person",
> "namespace": "nifi",
> "type": "record",
> "fields": [
> { "name": "name", "type": "string" },
> { "name": "mother", "type": "person" }
> ]
> }
> In this case, if we attempt to add this schema to a HashMap, we get the
> following error:
> 2018-11-07 19:09:33,021 ERROR [Timer-Driven Process Thread-38]
> o.a.n.p.k.pubsub.ConsumeKafkaRecord_2_0
> ConsumeKafkaRecord_2_0[id=ef6bd50b-0166-1000--f55a7995] Exception
> while processing data from kafka so will close the lease
> org.apache.nifi.processors.kafka.pubsub.ConsumerPool$SimpleConsumerLease@26706081
> due to java.lang.StackOverflowError: java.lang.StackOverflowError
> java.lang.StackOverflowError: null
> at java.util.AbstractList.hashCode(AbstractList.java:540)
> at
> java.util.Collections$UnmodifiableList.hashCode(Collections.java:1307)
> at
> org.apache.nifi.serialization.SimpleRecordSchema.hashCode(SimpleRecordSchema.java:172)
> at
> org.apache.nifi.serialization.record.type.RecordDataType.hashCode(RecordDataType.java:45)
> at
> org.apache.nifi.serialization.record.type.ArrayDataType.hashCode(ArrayDataType.java:44)
> at
> org.apache.nifi.serialization.record.RecordField.hashCode(RecordField.java:108)
> at java.util.AbstractList.hashCode(AbstractList.java:541)
> at
> java.util.Collections$UnmodifiableList.hashCode(Collections.java:1307)
> at
> org.apache.nifi.serialization.SimpleRecordSchema.hashCode(SimpleRecordSchema.java:172)
> at
> org.apache.nifi.serialization.record.type.RecordDataType.hashCode(RecordDataType.java:45)
> at
> org.apache.nifi.serialization.record.type.ArrayDataType.hashCode(ArrayDataType.java:44)
> at
> org.apache.nifi.serialization.record.RecordField.hashCode(RecordField.java:108)
> at java.util.AbstractList.hashCode(AbstractList.java:541)
> at
> java.util.Collections$UnmodifiableList.hashCode(Collections.java:1307)
> at
> org.apache.nifi.serialization.SimpleRecordSchema.hashCode(SimpleRecordSchema.java:172)
> at
> org.apache.nifi.serialization.record.type.RecordDataType.hashCode(RecordDataType.java:45)
> at
> org.apache.nifi.serialization.record.type.ArrayDataType.hashCode(ArrayDataType.java:44)
> at
> org.apache.nifi.serialization.record.RecordField.hashCode(RecordField.java:108)
> at java.util.AbstractList.hashCode(AbstractList.java:541)
> at
> java.util.Collections$UnmodifiableList.hashCode(Collections.java:1307)
> at
> org.apache.nifi.serialization.SimpleRecordSchema.hashCode(SimpleRecordSchema.java:172)
> at
> org.apache.nifi.serialization.record.type.RecordDataType.hashCode(RecordDataType.java:45)
> at
> org.apache.nifi.serialization.record.type.ArrayDataType.hashCode(ArrayDataType.java:44)
> at
> org.apache.nifi.serialization.record.RecordField.hashCode(RecordField.java:108)
> at java.util.AbstractList.hashCode(AbstractList.java:541)
> at
> java.util.Collections$UnmodifiableList.hashCode(Collections.java:1307)
> at
> org.apache.nifi.serialization.SimpleRecordSchema.hashCode(SimpleRecordSchema.java:172)
> at
> org.apache.nifi.serialization.record.type.RecordDataType.hashCode(RecordDataType.java:45)
> at
> org.apache.nifi.serialization.record.type.ArrayDataType.hashCode(ArrayDataType.java:44)
> at
> org.apache.nifi.serialization.record.RecordField.hashCode(RecordField.java:108)
> at java.util.AbstractList.hashCode(AbstractList.java:541)
> at
> java.util.Collections$UnmodifiableList.hashCode(Collections.java:1307)
> ...
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] nifi pull request #3160: NIFI-5805: Pool the BinaryEncoders used by the Writ...
GitHub user markap14 opened a pull request: https://github.com/apache/nifi/pull/3160 NIFI-5805: Pool the BinaryEncoders used by the WriteAvroResultWithExt⦠â¦ernalSchema writer. Unfortunately, the writer that embeds schemas does not allow for this optimization due to the Avro API Thank you for submitting a contribution to Apache NiFi. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [ ] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [ ] Does your PR title start with NIFI- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [ ] Has your PR been rebased against the latest commit within the target branch (typically master)? - [ ] Is your initial contribution a single, squashed commit? ### For code changes: - [ ] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder? - [ ] Have you written or updated unit tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly? - [ ] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly? - [ ] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. You can merge this pull request into a Git repository by running: $ git pull https://github.com/markap14/nifi NIFI-5805 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi/pull/3160.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3160 commit 0e69e6cacebefc2e6c49a0af436f9a6ac515de06 Author: Mark Payne Date: 2018-11-08T19:54:05Z NIFI-5805: Pool the BinaryEncoders used by the WriteAvroResultWithExternalSchema writer. Unfortunately, the writer that embeds schemas does not allow for this optimization due to the Avro API ---
[GitHub] nifi issue #3160: NIFI-5805: Pool the BinaryEncoders used by the WriteAvroRe...
Github user markap14 commented on the issue: https://github.com/apache/nifi/pull/3160 Performance difference when iterating 10 million times, each time creating a new avro writer and writing a single record, was an improvement from about 58 seconds to about 12 seconds. So approximately 5x improvement in this use case. ---
[jira] [Commented] (NIFI-5802) QueryRecord ignores field nullability and causes NPE with null input value
[
https://issues.apache.org/jira/browse/NIFI-5802?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16679453#comment-16679453
]
ASF subversion and git services commented on NIFI-5802:
---
Commit 78a1cb7c5eb84da5d25d50d0df69970b6d732f0a in nifi's branch
refs/heads/master from [~ijokarumawak]
[ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=78a1cb7 ]
NIFI-5802: Add QueryRecord nullable field support
Signed-off-by: Pierre Villard
This closes #3158.
> QueryRecord ignores field nullability and causes NPE with null input value
> --
>
> Key: NIFI-5802
> URL: https://issues.apache.org/jira/browse/NIFI-5802
> Project: Apache NiFi
> Issue Type: Bug
> Components: Extensions
>Affects Versions: 1.2.0
>Reporter: Koji Kawamura
>Assignee: Koji Kawamura
>Priority: Major
> Fix For: 1.9.0
>
>
> WIth Avro schema, a field can be defined as nullable by a union type. For
> example:
> {code:java}
> "type": [ "null", "boolean" ]
> {code}
> QueryRecord generates relational data types from the record schema. But it
> currently doesn't utilize the nullable information. And it leads to a
> NullPointerException when the incoming FlowFile has null values for the
> nullable fields.
> A simple 'select * from FLOWFILE' can fail, if the AvroRecordSetWriter is
> used.
> This issue was reported to nifi user ML. See the message for detail.
>
> [https://mail-archives.apache.org/mod_mbox/nifi-users/201811.mbox/%3CCAB10SGVgW2NR76UR1pYJvEZYTO3J1T7t_0q%2BUsQANp%3DcHxS-kw%40mail.gmail.com%3E]
> Here is the full stack trace:
> {code}
> org.apache.nifi.processor.exception.ProcessException: IOException thrown from
> QueryRecord[id=f11017d1-0166-1000-344a-c2f1d9bbd933]: java.io.IOException:
> org.apache.avro.file.DataFileWriter$AppendWriteException:
> java.lang.NullPointerException: null of boolean in field bool_test of
> org.apache.nifi.nifiRecord
> at
> org.apache.nifi.controller.repository.StandardProcessSession.write(StandardProcessSession.java:2667)
> at
> org.apache.nifi.processors.standard.QueryRecord.onTrigger(QueryRecord.java:309)
> at
> org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
> at
> org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1168)
> at
> org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:203)
> at
> org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:117)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.io.IOException:
> org.apache.avro.file.DataFileWriter$AppendWriteException:
> java.lang.NullPointerException: null of boolean in field bool_test of
> org.apache.nifi.nifiRecord
> at
> org.apache.nifi.processors.standard.QueryRecord$1.process(QueryRecord.java:327)
> at
> org.apache.nifi.controller.repository.StandardProcessSession.write(StandardProcessSession.java:2648)
> ... 12 common frames omitted
> Caused by: org.apache.avro.file.DataFileWriter$AppendWriteException:
> java.lang.NullPointerException: null of boolean in field bool_test of
> org.apache.nifi.nifiRecord
> at org.apache.avro.file.DataFileWriter.append(DataFileWriter.java:308)
> at
> org.apache.nifi.avro.WriteAvroResultWithSchema.writeRecord(WriteAvroResultWithSchema.java:61)
> at
> org.apache.nifi.serialization.AbstractRecordSetWriter.write(AbstractRecordSetWriter.java:59)
> at
> org.apache.nifi.serialization.AbstractRecordSetWriter.write(AbstractRecordSetWriter.java:52)
> at
> org.apache.nifi.processors.standard.QueryRecord$1.process(QueryRecord.java:324)
> ... 13 common frames omitted
> Caused by: java.lang.NullPointerException: null of boolean in field bool_test
> of org.apache.nifi.nifiRecord
> at
> org.apache.avro.generic.GenericDatumWriter.npe(GenericDatumWriter.java:132)
> at
> org.apache.avro.generic.GenericDatumWriter.writeWithoutConversion(GenericDatumWriter.java:126)
> at
> org.apache.avro.generic.GenericDatumWriter.wri
[jira] [Commented] (MINIFICPP-648) add processor and add processor with linkage nomenclature is confusing
[
https://issues.apache.org/jira/browse/MINIFICPP-648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16679691#comment-16679691
]
ASF GitHub Bot commented on MINIFICPP-648:
--
Github user phrocker commented on a diff in the pull request:
https://github.com/apache/nifi-minifi-cpp/pull/432#discussion_r231867605
--- Diff: nanofi/src/api/nanofi.cpp ---
@@ -323,55 +360,45 @@ int transmit_flowfile(flow_file_record *ff,
nifi_instance *instance) {
flow * create_new_flow(nifi_instance * instance) {
auto minifi_instance_ref =
static_cast(instance->instance_ptr);
- flow *new_flow = (flow*) malloc(sizeof(flow));
-
- auto execution_plan = new
ExecutionPlan(minifi_instance_ref->getContentRepository(),
minifi_instance_ref->getNoOpRepository(),
minifi_instance_ref->getNoOpRepository());
-
- new_flow->plan = execution_plan;
-
- return new_flow;
+ return new flow(minifi_instance_ref->getContentRepository(),
minifi_instance_ref->getNoOpRepository(),
minifi_instance_ref->getNoOpRepository());
}
flow *create_flow(nifi_instance *instance, const char *first_processor) {
if (nullptr == instance || nullptr == instance->instance_ptr) {
return nullptr;
}
auto minifi_instance_ref =
static_cast(instance->instance_ptr);
- flow *new_flow = (flow*) malloc(sizeof(flow));
-
- auto execution_plan = new
ExecutionPlan(minifi_instance_ref->getContentRepository(),
minifi_instance_ref->getNoOpRepository(),
minifi_instance_ref->getNoOpRepository());
- new_flow->plan = execution_plan;
+ flow *new_flow = new flow(minifi_instance_ref->getContentRepository(),
minifi_instance_ref->getNoOpRepository(),
minifi_instance_ref->getNoOpRepository());
--- End diff --
same malloc / new
> add processor and add processor with linkage nomenclature is confusing
> --
>
> Key: MINIFICPP-648
> URL: https://issues.apache.org/jira/browse/MINIFICPP-648
> Project: NiFi MiNiFi C++
> Issue Type: Improvement
>Reporter: Mr TheSegfault
>Assignee: Arpad Boda
>Priority: Blocker
> Labels: CAPI
> Fix For: 0.6.0
>
>
> add_processor should be changed to always add a processor with linkage
> without compelling documentation as why this exists.. As a result we will
> need to add a create_processor function to create one without adding it to
> the flow ( certain use cases where a flow isn't needed such as invokehttp or
> listenhttp ) this can be moved to 0.7.0 if we tag before recent commits.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (NIFI-5805) Avro Record Writer service creates byte buffer for every Writer created
Mark Payne created NIFI-5805: Summary: Avro Record Writer service creates byte buffer for every Writer created Key: NIFI-5805 URL: https://issues.apache.org/jira/browse/NIFI-5805 Project: Apache NiFi Issue Type: Bug Reporter: Mark Payne Assignee: Mark Payne When we use the Avro RecordSet Writer, and do not embed the schema, the Writer uses the Avro BinaryEncoder object to serialize the data. This object can be initialized, but instead we create a new one for each writer. This results in creating a new 64 KB byte[] each time. When we are writing many records to a given FlowFile, this is not a big deal. However, when used in PublishKafkaRecord or similar processors, where a new writer must be created for every Record, this can have a very significant performance impact. An improvement would be to have the user configure the maximum number of BinaryEncoder objects to pool and then use a simple pooling mechanism to reuse these objects. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi pull request #3161: Increase timeouts in unit tests to avoid frequent f...
GitHub user markap14 opened a pull request: https://github.com/apache/nifi/pull/3161 Increase timeouts in unit tests to avoid frequent failures Thank you for submitting a contribution to Apache NiFi. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [ ] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [ ] Does your PR title start with NIFI- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [ ] Has your PR been rebased against the latest commit within the target branch (typically master)? - [ ] Is your initial contribution a single, squashed commit? ### For code changes: - [ ] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder? - [ ] Have you written or updated unit tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly? - [ ] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly? - [ ] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. You can merge this pull request into a Git repository by running: $ git pull https://github.com/markap14/nifi fix-unit-test Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi/pull/3161.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3161 commit 4a25970a9f89c92f60a4012d5466a1a1794fa626 Author: Mark Payne Date: 2018-11-08T20:41:50Z Increase timeouts in unit tests to avoid frequent failures ---
[jira] [Commented] (NIFI-5805) Avro Record Writer service creates byte buffer for every Writer created
[ https://issues.apache.org/jira/browse/NIFI-5805?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16680316#comment-16680316 ] ASF GitHub Bot commented on NIFI-5805: -- GitHub user markap14 opened a pull request: https://github.com/apache/nifi/pull/3160 NIFI-5805: Pool the BinaryEncoders used by the WriteAvroResultWithExt… …ernalSchema writer. Unfortunately, the writer that embeds schemas does not allow for this optimization due to the Avro API Thank you for submitting a contribution to Apache NiFi. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [ ] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [ ] Does your PR title start with NIFI- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [ ] Has your PR been rebased against the latest commit within the target branch (typically master)? - [ ] Is your initial contribution a single, squashed commit? ### For code changes: - [ ] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder? - [ ] Have you written or updated unit tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly? - [ ] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly? - [ ] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. You can merge this pull request into a Git repository by running: $ git pull https://github.com/markap14/nifi NIFI-5805 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi/pull/3160.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3160 commit 0e69e6cacebefc2e6c49a0af436f9a6ac515de06 Author: Mark Payne Date: 2018-11-08T19:54:05Z NIFI-5805: Pool the BinaryEncoders used by the WriteAvroResultWithExternalSchema writer. Unfortunately, the writer that embeds schemas does not allow for this optimization due to the Avro API > Avro Record Writer service creates byte buffer for every Writer created > --- > > Key: NIFI-5805 > URL: https://issues.apache.org/jira/browse/NIFI-5805 > Project: Apache NiFi > Issue Type: Bug >Reporter: Mark Payne >Assignee: Mark Payne >Priority: Major > > When we use the Avro RecordSet Writer, and do not embed the schema, the > Writer uses the Avro BinaryEncoder object to serialize the data. This object > can be initialized, but instead we create a new one for each writer. This > results in creating a new 64 KB byte[] each time. When we are writing many > records to a given FlowFile, this is not a big deal. However, when used in > PublishKafkaRecord or similar processors, where a new writer must be created > for every Record, this can have a very significant performance impact. > An improvement would be to have the user configure the maximum number of > BinaryEncoder objects to pool and then use a simple pooling mechanism to > reuse these objects. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-5805) Avro Record Writer service creates byte buffer for every Writer created
[ https://issues.apache.org/jira/browse/NIFI-5805?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16680317#comment-16680317 ] ASF GitHub Bot commented on NIFI-5805: -- Github user markap14 commented on the issue: https://github.com/apache/nifi/pull/3160 Performance difference when iterating 10 million times, each time creating a new avro writer and writing a single record, was an improvement from about 58 seconds to about 12 seconds. So approximately 5x improvement in this use case. > Avro Record Writer service creates byte buffer for every Writer created > --- > > Key: NIFI-5805 > URL: https://issues.apache.org/jira/browse/NIFI-5805 > Project: Apache NiFi > Issue Type: Bug >Reporter: Mark Payne >Assignee: Mark Payne >Priority: Major > > When we use the Avro RecordSet Writer, and do not embed the schema, the > Writer uses the Avro BinaryEncoder object to serialize the data. This object > can be initialized, but instead we create a new one for each writer. This > results in creating a new 64 KB byte[] each time. When we are writing many > records to a given FlowFile, this is not a big deal. However, when used in > PublishKafkaRecord or similar processors, where a new writer must be created > for every Record, this can have a very significant performance impact. > An improvement would be to have the user configure the maximum number of > BinaryEncoder objects to pool and then use a simple pooling mechanism to > reuse these objects. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (NIFI-5428) Add documentation for S2S and Flow Analyzer tools
[ https://issues.apache.org/jira/browse/NIFI-5428?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Lim resolved NIFI-5428. -- Resolution: Resolved Fix Version/s: 1.9.0 Documented in the new NiFi Toolkit Guide > Add documentation for S2S and Flow Analyzer tools > - > > Key: NIFI-5428 > URL: https://issues.apache.org/jira/browse/NIFI-5428 > Project: Apache NiFi > Issue Type: Improvement > Components: Documentation & Website >Reporter: Andrew Lim >Priority: Minor > Fix For: 1.9.0 > > > I noticed that these two Admin toolkit tools are not documented. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (MINIFICPP-648) add processor and add processor with linkage nomenclature is confusing
[
https://issues.apache.org/jira/browse/MINIFICPP-648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16679694#comment-16679694
]
ASF GitHub Bot commented on MINIFICPP-648:
--
Github user phrocker commented on a diff in the pull request:
https://github.com/apache/nifi-minifi-cpp/pull/432#discussion_r231868999
--- Diff: nanofi/src/cxx/Plan.cpp ---
@@ -162,6 +153,21 @@ bool
ExecutionPlan::runNextProcessor(std::function current_session =
std::make_shared(context);
process_sessions_.push_back(current_session);
+ if (input_ff != nullptr) {
+auto content_repo =
static_cast*>(input_ff->crp);
+std::shared_ptr claim =
std::make_shared(input_ff->contentLocation,
*content_repo);
+auto stream = (*content_repo)->read(claim);
+std::shared_ptr flowFile =
std::static_pointer_cast(current_session->create());
+auto map = static_cast*>(input_ff->attributes);
+if (map) {
+ for (const auto& attr: *map) {
+flowFile->setAttribute(attr.first, attr.second);
+ }
+}
+current_session->importFrom(*stream, flowFile);
+current_session->transfer(flowFile, core::Relationship("success",
"success"));
+
relationships_[relationships_.size()-1]->put(std::static_pointer_cast(flowFile));
--- End diff --
is there a safeguard to ensure a relationship exists?
> add processor and add processor with linkage nomenclature is confusing
> --
>
> Key: MINIFICPP-648
> URL: https://issues.apache.org/jira/browse/MINIFICPP-648
> Project: NiFi MiNiFi C++
> Issue Type: Improvement
>Reporter: Mr TheSegfault
>Assignee: Arpad Boda
>Priority: Blocker
> Labels: CAPI
> Fix For: 0.6.0
>
>
> add_processor should be changed to always add a processor with linkage
> without compelling documentation as why this exists.. As a result we will
> need to add a create_processor function to create one without adding it to
> the flow ( certain use cases where a flow isn't needed such as invokehttp or
> listenhttp ) this can be moved to 0.7.0 if we tag before recent commits.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Closed] (MINIFICPP-641) C API: add support to register failure callback
[ https://issues.apache.org/jira/browse/MINIFICPP-641?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpad Boda closed MINIFICPP-641. > C API: add support to register failure callback > --- > > Key: MINIFICPP-641 > URL: https://issues.apache.org/jira/browse/MINIFICPP-641 > Project: NiFi MiNiFi C++ > Issue Type: New Feature >Reporter: Arpad Boda >Assignee: Arpad Boda >Priority: Minor > Labels: CAPI > Fix For: 0.6.0 > > > Current implementation of C API provide no option to handle failure > relationships of processors. > Simply adding relationships one-by-one would make it unreasonably complicated > in user's perspective. User should be able to add a simple callback, which - > in the background - creates a processor and connects that all processors in > the given flow with failure relationship. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-5794) ConsumeKafka and PublishKafka should allow empty string demarcator
[ https://issues.apache.org/jira/browse/NIFI-5794?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16679648#comment-16679648 ] ASF GitHub Bot commented on NIFI-5794: -- Github user pvillard31 commented on the issue: https://github.com/apache/nifi/pull/3159 In the end, I only changed the ConsumeKafka processors since it does not make sense in PublishKafka (one would just have to not set any demarcator to get the same result: all messages in the flow files are sent as one single message in Kafka). The main interest of this PR is to increase efficiency when we consume Avro data from Kafka: there is no need to set a demarcator and we do want to have multiples messages in the same flow file. I've used this [template](https://gist.github.com/pvillard31/57e627d206e7965cfb87cc27c09caf44) to test the change and to confirm the behavior: when pulling Avro data from Kafka with "empty string" as demarcator to have multiple messages in the same flow file, we are able to process the records in the next processor using a ConvertRecord for instance. > ConsumeKafka and PublishKafka should allow empty string demarcator > -- > > Key: NIFI-5794 > URL: https://issues.apache.org/jira/browse/NIFI-5794 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Reporter: Pierre Villard >Assignee: Pierre Villard >Priority: Major > > ConsumeKafka(_*) and PublishKafka(_*) processors should allow "empty string" > as a message demarcator. This would allow consuming Avro data without the > serialization/de-serialization cost while still allowing the use of Record > processors once the data is in NiFi. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-5784) With the addition of the Toolkit Guide, edit other docs that contain duplicate content
[ https://issues.apache.org/jira/browse/NIFI-5784?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16679433#comment-16679433 ] ASF GitHub Bot commented on NIFI-5784: -- Github user asfgit closed the pull request at: https://github.com/apache/nifi/pull/3155 > With the addition of the Toolkit Guide, edit other docs that contain > duplicate content > -- > > Key: NIFI-5784 > URL: https://issues.apache.org/jira/browse/NIFI-5784 > Project: Apache NiFi > Issue Type: Improvement > Components: Documentation & Website >Reporter: Andrew Lim >Assignee: Andrew Lim >Priority: Minor > Fix For: 1.9.0 > > > The Admin Guide has duplicate content for the following: > * Configuration encryption - > [https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#encrypt-config_tool] > * File manager - > [https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#file-manager] > * Node manager - > [https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#node-manager] > * TLS Toolkit - > [https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#tls_generation_toolkit] > * ZooKeeper migrator - > [https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#zookeeper_migrator] > Will remove and add links to Toolkit Guide as needed. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-5800) If RecordSchema has an inner field that references a schema recursively by name, hashCode() throws StackOverflowError
[
https://issues.apache.org/jira/browse/NIFI-5800?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16679441#comment-16679441
]
ASF GitHub Bot commented on NIFI-5800:
--
Github user asfgit closed the pull request at:
https://github.com/apache/nifi/pull/3157
> If RecordSchema has an inner field that references a schema recursively by
> name, hashCode() throws StackOverflowError
> -
>
> Key: NIFI-5800
> URL: https://issues.apache.org/jira/browse/NIFI-5800
> Project: Apache NiFi
> Issue Type: Bug
>Reporter: Mark Payne
>Priority: Major
> Fix For: 1.9.0
>
>
> If we have a schema where a field is of type RECORD and references an
> outer-level schema, we get StackOverflowError. For example:
> {
> "name": "person",
> "namespace": "nifi",
> "type": "record",
> "fields": [
> { "name": "name", "type": "string" },
> { "name": "mother", "type": "person" }
> ]
> }
> In this case, if we attempt to add this schema to a HashMap, we get the
> following error:
> 2018-11-07 19:09:33,021 ERROR [Timer-Driven Process Thread-38]
> o.a.n.p.k.pubsub.ConsumeKafkaRecord_2_0
> ConsumeKafkaRecord_2_0[id=ef6bd50b-0166-1000--f55a7995] Exception
> while processing data from kafka so will close the lease
> org.apache.nifi.processors.kafka.pubsub.ConsumerPool$SimpleConsumerLease@26706081
> due to java.lang.StackOverflowError: java.lang.StackOverflowError
> java.lang.StackOverflowError: null
> at java.util.AbstractList.hashCode(AbstractList.java:540)
> at
> java.util.Collections$UnmodifiableList.hashCode(Collections.java:1307)
> at
> org.apache.nifi.serialization.SimpleRecordSchema.hashCode(SimpleRecordSchema.java:172)
> at
> org.apache.nifi.serialization.record.type.RecordDataType.hashCode(RecordDataType.java:45)
> at
> org.apache.nifi.serialization.record.type.ArrayDataType.hashCode(ArrayDataType.java:44)
> at
> org.apache.nifi.serialization.record.RecordField.hashCode(RecordField.java:108)
> at java.util.AbstractList.hashCode(AbstractList.java:541)
> at
> java.util.Collections$UnmodifiableList.hashCode(Collections.java:1307)
> at
> org.apache.nifi.serialization.SimpleRecordSchema.hashCode(SimpleRecordSchema.java:172)
> at
> org.apache.nifi.serialization.record.type.RecordDataType.hashCode(RecordDataType.java:45)
> at
> org.apache.nifi.serialization.record.type.ArrayDataType.hashCode(ArrayDataType.java:44)
> at
> org.apache.nifi.serialization.record.RecordField.hashCode(RecordField.java:108)
> at java.util.AbstractList.hashCode(AbstractList.java:541)
> at
> java.util.Collections$UnmodifiableList.hashCode(Collections.java:1307)
> at
> org.apache.nifi.serialization.SimpleRecordSchema.hashCode(SimpleRecordSchema.java:172)
> at
> org.apache.nifi.serialization.record.type.RecordDataType.hashCode(RecordDataType.java:45)
> at
> org.apache.nifi.serialization.record.type.ArrayDataType.hashCode(ArrayDataType.java:44)
> at
> org.apache.nifi.serialization.record.RecordField.hashCode(RecordField.java:108)
> at java.util.AbstractList.hashCode(AbstractList.java:541)
> at
> java.util.Collections$UnmodifiableList.hashCode(Collections.java:1307)
> at
> org.apache.nifi.serialization.SimpleRecordSchema.hashCode(SimpleRecordSchema.java:172)
> at
> org.apache.nifi.serialization.record.type.RecordDataType.hashCode(RecordDataType.java:45)
> at
> org.apache.nifi.serialization.record.type.ArrayDataType.hashCode(ArrayDataType.java:44)
> at
> org.apache.nifi.serialization.record.RecordField.hashCode(RecordField.java:108)
> at java.util.AbstractList.hashCode(AbstractList.java:541)
> at
> java.util.Collections$UnmodifiableList.hashCode(Collections.java:1307)
> at
> org.apache.nifi.serialization.SimpleRecordSchema.hashCode(SimpleRecordSchema.java:172)
> at
> org.apache.nifi.serialization.record.type.RecordDataType.hashCode(RecordDataType.java:45)
> at
> org.apache.nifi.serialization.record.type.ArrayDataType.hashCode(ArrayDataType.java:44)
> at
> org.apache.nifi.serialization.record.RecordField.hashCode(RecordField.java:108)
> at java.util.AbstractList.hashCode(AbstractList.java:541)
> at
> java.util.Collections$UnmodifiableList.hashCode(Collections.java:1307)
> ...
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (MINIFICPP-648) add processor and add processor with linkage nomenclature is confusing
[
https://issues.apache.org/jira/browse/MINIFICPP-648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16679688#comment-16679688
]
ASF GitHub Bot commented on MINIFICPP-648:
--
Github user phrocker commented on a diff in the pull request:
https://github.com/apache/nifi-minifi-cpp/pull/432#discussion_r231865773
--- Diff: nanofi/src/api/nanofi.cpp ---
@@ -89,6 +93,39 @@ nifi_instance *create_instance(const char *url,
nifi_port *port) {
return instance;
}
+standalone_processor *create_processor(const char *name) {
+ static int proc_counter = 0;
+ auto ptr = ExecutionPlan::createProcessor(name, name);
+ if (!ptr) {
+return nullptr;
+ }
+ if (standalone_instance == nullptr) {
+nifi_port port;
--- End diff --
this is an input port identifier.
> add processor and add processor with linkage nomenclature is confusing
> --
>
> Key: MINIFICPP-648
> URL: https://issues.apache.org/jira/browse/MINIFICPP-648
> Project: NiFi MiNiFi C++
> Issue Type: Improvement
>Reporter: Mr TheSegfault
>Assignee: Arpad Boda
>Priority: Blocker
> Labels: CAPI
> Fix For: 0.6.0
>
>
> add_processor should be changed to always add a processor with linkage
> without compelling documentation as why this exists.. As a result we will
> need to add a create_processor function to create one without adding it to
> the flow ( certain use cases where a flow isn't needed such as invokehttp or
> listenhttp ) this can be moved to 0.7.0 if we tag before recent commits.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (NIFI-5802) QueryRecord ignores field nullability and causes NPE with null input value
[
https://issues.apache.org/jira/browse/NIFI-5802?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Koji Kawamura updated NIFI-5802:
Description:
WIth Avro schema, a field can be defined as nullable by a union type. For
example:
{code:java}
"type": [ "null", "boolean" ]
{code}
QueryRecord generates relational data types from the record schema. But it
currently doesn't utilize the nullable information. And it leads to a
NullPointerException when the incoming FlowFile has null values for the
nullable fields.
A simple 'select * from FLOWFILE' can fail, if the AvroRecordSetWriter is used.
This issue was reported to nifi user ML. See the message for detail.
[https://mail-archives.apache.org/mod_mbox/nifi-users/201811.mbox/%3CCAB10SGVgW2NR76UR1pYJvEZYTO3J1T7t_0q%2BUsQANp%3DcHxS-kw%40mail.gmail.com%3E]
Here is the full stack trace:
{code}
org.apache.nifi.processor.exception.ProcessException: IOException thrown from
QueryRecord[id=f11017d1-0166-1000-344a-c2f1d9bbd933]: java.io.IOException:
org.apache.avro.file.DataFileWriter$AppendWriteException:
java.lang.NullPointerException: null of boolean in field bool_test of
org.apache.nifi.nifiRecord
at
org.apache.nifi.controller.repository.StandardProcessSession.write(StandardProcessSession.java:2667)
at
org.apache.nifi.processors.standard.QueryRecord.onTrigger(QueryRecord.java:309)
at
org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
at
org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1168)
at
org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:203)
at
org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:117)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.IOException:
org.apache.avro.file.DataFileWriter$AppendWriteException:
java.lang.NullPointerException: null of boolean in field bool_test of
org.apache.nifi.nifiRecord
at
org.apache.nifi.processors.standard.QueryRecord$1.process(QueryRecord.java:327)
at
org.apache.nifi.controller.repository.StandardProcessSession.write(StandardProcessSession.java:2648)
... 12 common frames omitted
Caused by: org.apache.avro.file.DataFileWriter$AppendWriteException:
java.lang.NullPointerException: null of boolean in field bool_test of
org.apache.nifi.nifiRecord
at org.apache.avro.file.DataFileWriter.append(DataFileWriter.java:308)
at
org.apache.nifi.avro.WriteAvroResultWithSchema.writeRecord(WriteAvroResultWithSchema.java:61)
at
org.apache.nifi.serialization.AbstractRecordSetWriter.write(AbstractRecordSetWriter.java:59)
at
org.apache.nifi.serialization.AbstractRecordSetWriter.write(AbstractRecordSetWriter.java:52)
at
org.apache.nifi.processors.standard.QueryRecord$1.process(QueryRecord.java:324)
... 13 common frames omitted
Caused by: java.lang.NullPointerException: null of boolean in field bool_test
of org.apache.nifi.nifiRecord
at
org.apache.avro.generic.GenericDatumWriter.npe(GenericDatumWriter.java:132)
at
org.apache.avro.generic.GenericDatumWriter.writeWithoutConversion(GenericDatumWriter.java:126)
at
org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:73)
at
org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:60)
at org.apache.avro.file.DataFileWriter.append(DataFileWriter.java:302)
... 17 common frames omitted
Caused by: java.lang.NullPointerException: null
at
org.apache.avro.generic.GenericDatumWriter.writeWithoutConversion(GenericDatumWriter.java:121)
at
org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:73)
at
org.apache.avro.generic.GenericDatumWriter.writeField(GenericDatumWriter.java:153)
at
org.apache.avro.generic.GenericDatumWriter.writeRecord(GenericDatumWriter.java:143)
at
org.apache.avro.generic.GenericDatumWriter.writeWithoutConversion(GenericDatumWriter.java:105)
... 20 common frames omitted
{code}
was:
WIth Avro schema, a field can be defined as nullable by a union type. For
example:
{code:java}
"type": [ "null", "boolean" ]
{code}
QueryRecord generates relational data types from the reco
[jira] [Commented] (MINIFICPP-648) add processor and add processor with linkage nomenclature is confusing
[
https://issues.apache.org/jira/browse/MINIFICPP-648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16679752#comment-16679752
]
ASF GitHub Bot commented on MINIFICPP-648:
--
Github user arpadboda commented on a diff in the pull request:
https://github.com/apache/nifi-minifi-cpp/pull/432#discussion_r231884707
--- Diff: nanofi/src/api/nanofi.cpp ---
@@ -323,55 +360,45 @@ int transmit_flowfile(flow_file_record *ff,
nifi_instance *instance) {
flow * create_new_flow(nifi_instance * instance) {
auto minifi_instance_ref =
static_cast(instance->instance_ptr);
- flow *new_flow = (flow*) malloc(sizeof(flow));
-
- auto execution_plan = new
ExecutionPlan(minifi_instance_ref->getContentRepository(),
minifi_instance_ref->getNoOpRepository(),
minifi_instance_ref->getNoOpRepository());
-
- new_flow->plan = execution_plan;
-
- return new_flow;
+ return new flow(minifi_instance_ref->getContentRepository(),
minifi_instance_ref->getNoOpRepository(),
minifi_instance_ref->getNoOpRepository());
--- End diff --
Multiple thins to add there:
- Please note that this is not an introduced "new" call, just the one 5
lines above.
- We can change this to use malloc and placement new, in which case it
ensures that free only causes some leak, but no crash.
- I don't think this approach is sustainable in long term. To properly free
structures the users would be required to know the internal layout of the
structure, which is not going to happen and it would lead to very complex free
strategies in case of shared ownerships. Moreover I don't think that the
approach "feel free to free, it's just a leak" is the way forward. One of the
reasons I only left declarations in cstructs.h is that freeing something you
don't even know should make you raise concerns. What I would really push is the
user freeing up all the stuff he allocated and we provide free functions for
every structure that the API may return. Don't free what you don't allocate is
quite clear and well defines responsibilities. To go even further here, letting
the users free things might be a painful barrier in case we plan to change the
internal representation, which they shouldn't be aware of anyway.
In nutshell, combining free with placement new is good enough to avoid
immediate crashes, but we should try to highlight that the structures allocated
by the API should also be freed by that.
> add processor and add processor with linkage nomenclature is confusing
> --
>
> Key: MINIFICPP-648
> URL: https://issues.apache.org/jira/browse/MINIFICPP-648
> Project: NiFi MiNiFi C++
> Issue Type: Improvement
>Reporter: Mr TheSegfault
>Assignee: Arpad Boda
>Priority: Blocker
> Labels: CAPI
> Fix For: 0.6.0
>
>
> add_processor should be changed to always add a processor with linkage
> without compelling documentation as why this exists.. As a result we will
> need to add a create_processor function to create one without adding it to
> the flow ( certain use cases where a flow isn't needed such as invokehttp or
> listenhttp ) this can be moved to 0.7.0 if we tag before recent commits.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (NIFI-5784) With the addition of the Toolkit Guide, edit other docs that contain duplicate content
[ https://issues.apache.org/jira/browse/NIFI-5784?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16679431#comment-16679431 ] ASF subversion and git services commented on NIFI-5784: --- Commit 5cf58b42e0dbed7d25f43c189ce6b07a295c0f20 in nifi's branch refs/heads/master from [~andrewmlim] [ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=5cf58b4 ] NIFI-5784 Edit Admin Guide to remove duplicate content that is in new Toolkit Guide Signed-off-by: Pierre Villard This closes #3155. > With the addition of the Toolkit Guide, edit other docs that contain > duplicate content > -- > > Key: NIFI-5784 > URL: https://issues.apache.org/jira/browse/NIFI-5784 > Project: Apache NiFi > Issue Type: Improvement > Components: Documentation & Website >Reporter: Andrew Lim >Assignee: Andrew Lim >Priority: Minor > Fix For: 1.9.0 > > > The Admin Guide has duplicate content for the following: > * Configuration encryption - > [https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#encrypt-config_tool] > * File manager - > [https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#file-manager] > * Node manager - > [https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#node-manager] > * TLS Toolkit - > [https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#tls_generation_toolkit] > * ZooKeeper migrator - > [https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#zookeeper_migrator] > Will remove and add links to Toolkit Guide as needed. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-5780) Add pre and post statements to ExecuteSQL and ExecuteSQLRecord
[
https://issues.apache.org/jira/browse/NIFI-5780?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16679240#comment-16679240
]
ASF GitHub Bot commented on NIFI-5780:
--
Github user yjhyjhyjh0 commented on a diff in the pull request:
https://github.com/apache/nifi/pull/3156#discussion_r231753838
--- Diff:
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/AbstractExecuteSQL.java
---
@@ -82,6 +84,16 @@
.identifiesControllerService(DBCPService.class)
.build();
+public static final PropertyDescriptor SQL_PRE_QUERY = new
PropertyDescriptor.Builder()
+.name("sql-pre-query")
+.displayName("SQL pre-query")
--- End diff --
Hi Peter, thanks for the feedback.
I intentionally type it in 'SQL pre-query' to align with SQL_SELECT_QUERY
property displayName 'SQL select query'.
For example currently,
SQL_PRE_QUERY, display '**SQL pre-query**'
SQL_SELECT_QUERY, display '**SQL select query**'
SQL_POST_QUERY, display '**SQL post-query**'
If modified as you mention (I actually try this version in the first try)
For example,
SQL_PRE_QUERY, display '**SQL Pre-Query**'
SQL_SELECT_QUERY, display '**SQL select query**'
SQL_POST_QUERY, display '**SQL Post-Query**'
Please let me know your though on this alignment, thanks
> Add pre and post statements to ExecuteSQL and ExecuteSQLRecord
> --
>
> Key: NIFI-5780
> URL: https://issues.apache.org/jira/browse/NIFI-5780
> Project: Apache NiFi
> Issue Type: Improvement
>Affects Versions: 1.8.0
>Reporter: Deon Huang
>Assignee: Deon Huang
>Priority: Minor
>
> Sometimes we might need to set up session relate configuration before or
> after query.
> For example:
> Pre query can be used for session relate setting like our use case Teradata
> Query Banding.
> Same feature (pre query and post query) is added to SelectHiveQL in
> https://issues.apache.org/jira/browse/NIFI-5044
> Planning to add this feature to ExecuteSQL and ExecuteSQLRecord processors.
> If pre or post statement fail, will not produce resultset flowfile.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (MINIFICPP-648) add processor and add processor with linkage nomenclature is confusing
[
https://issues.apache.org/jira/browse/MINIFICPP-648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16679757#comment-16679757
]
ASF GitHub Bot commented on MINIFICPP-648:
--
Github user arpadboda commented on a diff in the pull request:
https://github.com/apache/nifi-minifi-cpp/pull/432#discussion_r231886506
--- Diff: nanofi/src/api/nanofi.cpp ---
@@ -323,55 +360,45 @@ int transmit_flowfile(flow_file_record *ff,
nifi_instance *instance) {
flow * create_new_flow(nifi_instance * instance) {
auto minifi_instance_ref =
static_cast(instance->instance_ptr);
- flow *new_flow = (flow*) malloc(sizeof(flow));
-
- auto execution_plan = new
ExecutionPlan(minifi_instance_ref->getContentRepository(),
minifi_instance_ref->getNoOpRepository(),
minifi_instance_ref->getNoOpRepository());
-
- new_flow->plan = execution_plan;
-
- return new_flow;
+ return new flow(minifi_instance_ref->getContentRepository(),
minifi_instance_ref->getNoOpRepository(),
minifi_instance_ref->getNoOpRepository());
}
flow *create_flow(nifi_instance *instance, const char *first_processor) {
if (nullptr == instance || nullptr == instance->instance_ptr) {
return nullptr;
}
auto minifi_instance_ref =
static_cast(instance->instance_ptr);
- flow *new_flow = (flow*) malloc(sizeof(flow));
-
- auto execution_plan = new
ExecutionPlan(minifi_instance_ref->getContentRepository(),
minifi_instance_ref->getNoOpRepository(),
minifi_instance_ref->getNoOpRepository());
- new_flow->plan = execution_plan;
+ flow *new_flow = new flow(minifi_instance_ref->getContentRepository(),
minifi_instance_ref->getNoOpRepository(),
minifi_instance_ref->getNoOpRepository());
if (first_processor != nullptr && strlen(first_processor) > 0) {
// automatically adds it with success
-execution_plan->addProcessor(first_processor, first_processor);
--- End diff --
I guess that only declaring structures via the API leaves us the freedom to
change the internal representation anyhow without breaking it. On the other
side it's now faster than it was.
> add processor and add processor with linkage nomenclature is confusing
> --
>
> Key: MINIFICPP-648
> URL: https://issues.apache.org/jira/browse/MINIFICPP-648
> Project: NiFi MiNiFi C++
> Issue Type: Improvement
>Reporter: Mr TheSegfault
>Assignee: Arpad Boda
>Priority: Blocker
> Labels: CAPI
> Fix For: 0.6.0
>
>
> add_processor should be changed to always add a processor with linkage
> without compelling documentation as why this exists.. As a result we will
> need to add a create_processor function to create one without adding it to
> the flow ( certain use cases where a flow isn't needed such as invokehttp or
> listenhttp ) this can be moved to 0.7.0 if we tag before recent commits.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (NIFI-5802) QueryRecord ignores field nullability and causes NPE with null input value
Koji Kawamura created NIFI-5802:
---
Summary: QueryRecord ignores field nullability and causes NPE with
null input value
Key: NIFI-5802
URL: https://issues.apache.org/jira/browse/NIFI-5802
Project: Apache NiFi
Issue Type: Bug
Components: Extensions
Affects Versions: 1.2.0
Reporter: Koji Kawamura
Assignee: Koji Kawamura
WIth Avro schema, a field can be defined as nullable by a union type. For
example:
{code}
"type": [ "null", "boolean" ]
{code}
QueryRecord generates relational data types from the record schema. But it
currently doesn't utilize the nullable information. And it leads to a
NullPointerException when the incoming FlowFile has null values for the
nullable fields.
A simple 'select * from FLOWFILE' can fail.
This issue was reported to nifi user ML. See the message for detail.
https://mail-archives.apache.org/mod_mbox/nifi-users/201811.mbox/%3CCAB10SGVgW2NR76UR1pYJvEZYTO3J1T7t_0q%2BUsQANp%3DcHxS-kw%40mail.gmail.com%3E
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (NIFI-5780) Add pre and post statements to ExecuteSQL and ExecuteSQLRecord
[
https://issues.apache.org/jira/browse/NIFI-5780?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16679241#comment-16679241
]
ASF GitHub Bot commented on NIFI-5780:
--
Github user yjhyjhyjh0 commented on a diff in the pull request:
https://github.com/apache/nifi/pull/3156#discussion_r231754034
--- Diff:
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/test/java/org/apache/nifi/processors/standard/TestExecuteSQLRecord.java
---
@@ -350,6 +352,141 @@ public void invokeOnTriggerRecords(final Integer
queryTimeout, final String quer
assertEquals(durationTime, fetchTime + executionTime);
}
+@Test
+public void testPreQuery() throws Exception {
+// remove previous test database, if any
+final File dbLocation = new File(DB_LOCATION);
+dbLocation.delete();
+
+// load test data to database
+final Connection con = ((DBCPService)
runner.getControllerService("dbcp")).getConnection();
+Statement stmt = con.createStatement();
+
+try {
+stmt.execute("drop table TEST_NULL_INT");
+} catch (final SQLException sqle) {
+}
+
+stmt.execute("create table TEST_NULL_INT (id integer not null,
val1 integer, val2 integer, constraint my_pk primary key (id))");
+
+runner.setIncomingConnection(true);
+runner.setProperty(ExecuteSQL.SQL_PRE_QUERY, "insert into
TEST_NULL_INT values(1,2,3);insert into TEST_NULL_INT values(4,5,6)");
--- End diff --
That's indeed great use case, I'll update test cases.
Thanks for the sharing the information.
> Add pre and post statements to ExecuteSQL and ExecuteSQLRecord
> --
>
> Key: NIFI-5780
> URL: https://issues.apache.org/jira/browse/NIFI-5780
> Project: Apache NiFi
> Issue Type: Improvement
>Affects Versions: 1.8.0
>Reporter: Deon Huang
>Assignee: Deon Huang
>Priority: Minor
>
> Sometimes we might need to set up session relate configuration before or
> after query.
> For example:
> Pre query can be used for session relate setting like our use case Teradata
> Query Banding.
> Same feature (pre query and post query) is added to SelectHiveQL in
> https://issues.apache.org/jira/browse/NIFI-5044
> Planning to add this feature to ExecuteSQL and ExecuteSQLRecord processors.
> If pre or post statement fail, will not produce resultset flowfile.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (MINIFICPP-641) C API: add support to register failure callback
[ https://issues.apache.org/jira/browse/MINIFICPP-641?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpad Boda updated MINIFICPP-641: - Labels: CAPI (was: ) > C API: add support to register failure callback > --- > > Key: MINIFICPP-641 > URL: https://issues.apache.org/jira/browse/MINIFICPP-641 > Project: NiFi MiNiFi C++ > Issue Type: New Feature >Reporter: Arpad Boda >Assignee: Arpad Boda >Priority: Minor > Labels: CAPI > Fix For: 0.6.0 > > > Current implementation of C API provide no option to handle failure > relationships of processors. > Simply adding relationships one-by-one would make it unreasonably complicated > in user's perspective. User should be able to add a simple callback, which - > in the background - creates a processor and connects that all processors in > the given flow with failure relationship. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-5802) QueryRecord ignores field nullability and causes NPE with null input value
[
https://issues.apache.org/jira/browse/NIFI-5802?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16679456#comment-16679456
]
ASF GitHub Bot commented on NIFI-5802:
--
Github user asfgit closed the pull request at:
https://github.com/apache/nifi/pull/3158
> QueryRecord ignores field nullability and causes NPE with null input value
> --
>
> Key: NIFI-5802
> URL: https://issues.apache.org/jira/browse/NIFI-5802
> Project: Apache NiFi
> Issue Type: Bug
> Components: Extensions
>Affects Versions: 1.2.0
>Reporter: Koji Kawamura
>Assignee: Koji Kawamura
>Priority: Major
> Fix For: 1.9.0
>
>
> WIth Avro schema, a field can be defined as nullable by a union type. For
> example:
> {code:java}
> "type": [ "null", "boolean" ]
> {code}
> QueryRecord generates relational data types from the record schema. But it
> currently doesn't utilize the nullable information. And it leads to a
> NullPointerException when the incoming FlowFile has null values for the
> nullable fields.
> A simple 'select * from FLOWFILE' can fail, if the AvroRecordSetWriter is
> used.
> This issue was reported to nifi user ML. See the message for detail.
>
> [https://mail-archives.apache.org/mod_mbox/nifi-users/201811.mbox/%3CCAB10SGVgW2NR76UR1pYJvEZYTO3J1T7t_0q%2BUsQANp%3DcHxS-kw%40mail.gmail.com%3E]
> Here is the full stack trace:
> {code}
> org.apache.nifi.processor.exception.ProcessException: IOException thrown from
> QueryRecord[id=f11017d1-0166-1000-344a-c2f1d9bbd933]: java.io.IOException:
> org.apache.avro.file.DataFileWriter$AppendWriteException:
> java.lang.NullPointerException: null of boolean in field bool_test of
> org.apache.nifi.nifiRecord
> at
> org.apache.nifi.controller.repository.StandardProcessSession.write(StandardProcessSession.java:2667)
> at
> org.apache.nifi.processors.standard.QueryRecord.onTrigger(QueryRecord.java:309)
> at
> org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
> at
> org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1168)
> at
> org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:203)
> at
> org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:117)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.io.IOException:
> org.apache.avro.file.DataFileWriter$AppendWriteException:
> java.lang.NullPointerException: null of boolean in field bool_test of
> org.apache.nifi.nifiRecord
> at
> org.apache.nifi.processors.standard.QueryRecord$1.process(QueryRecord.java:327)
> at
> org.apache.nifi.controller.repository.StandardProcessSession.write(StandardProcessSession.java:2648)
> ... 12 common frames omitted
> Caused by: org.apache.avro.file.DataFileWriter$AppendWriteException:
> java.lang.NullPointerException: null of boolean in field bool_test of
> org.apache.nifi.nifiRecord
> at org.apache.avro.file.DataFileWriter.append(DataFileWriter.java:308)
> at
> org.apache.nifi.avro.WriteAvroResultWithSchema.writeRecord(WriteAvroResultWithSchema.java:61)
> at
> org.apache.nifi.serialization.AbstractRecordSetWriter.write(AbstractRecordSetWriter.java:59)
> at
> org.apache.nifi.serialization.AbstractRecordSetWriter.write(AbstractRecordSetWriter.java:52)
> at
> org.apache.nifi.processors.standard.QueryRecord$1.process(QueryRecord.java:324)
> ... 13 common frames omitted
> Caused by: java.lang.NullPointerException: null of boolean in field bool_test
> of org.apache.nifi.nifiRecord
> at
> org.apache.avro.generic.GenericDatumWriter.npe(GenericDatumWriter.java:132)
> at
> org.apache.avro.generic.GenericDatumWriter.writeWithoutConversion(GenericDatumWriter.java:126)
> at
> org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:73)
> at
> org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:60)
> at org.apache.avro.file.DataFileWriter.append(DataFileWriter.java:302)
>
[jira] [Commented] (NIFI-5790) DBCPConnectionPool configuration should expose underlying connection idle and eviction configuration
[
https://issues.apache.org/jira/browse/NIFI-5790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16679225#comment-16679225
]
ASF GitHub Bot commented on NIFI-5790:
--
Github user colindean commented on the issue:
https://github.com/apache/nifi/pull/3133
> an Idle connection was one that had been returned to the pool?
That's what I would think, but I couldn't seem to actually trigger it.
Reading through the API docs some more, I didn't think to try checking the
idle connection count _after_ closing a connection. I'll try that tomorrow.
> DBCPConnectionPool configuration should expose underlying connection idle and
> eviction configuration
>
>
> Key: NIFI-5790
> URL: https://issues.apache.org/jira/browse/NIFI-5790
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Extensions
>Affects Versions: 1.8.0
>Reporter: Colin Dean
>Priority: Major
> Labels: DBCP, database
>
> While investigating a fix for NIFI-5789, I noticed in the [DBCPConnectionPool
> documentation|https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-dbcp-service-nar/1.8.0/org.apache.nifi.dbcp.DBCPConnectionPool/index.html]
> that NiFi appears _not_ to have controller service configuration options
> associated with [Apache
> Commons-DBCP|https://commons.apache.org/proper/commons-dbcp/configuration.html]
> {{BasicDataSource}} parameters like {{minIdle}} and {{maxIdle}}, which I
> think should be both set to 0 in my particular use case.
> Alternatively, I think I could set {{maxConnLifetimeMillis}} to something
> even in the minutes range and satisfy my use case (a connection need not be
> released _immediately_ but within a reasonable period of time), but this
> option is also not available.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (MINIFICPP-648) add processor and add processor with linkage nomenclature is confusing
[
https://issues.apache.org/jira/browse/MINIFICPP-648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16679690#comment-16679690
]
ASF GitHub Bot commented on MINIFICPP-648:
--
Github user phrocker commented on a diff in the pull request:
https://github.com/apache/nifi-minifi-cpp/pull/432#discussion_r231865178
--- Diff: nanofi/include/cxx/CallbackProcessor.h ---
@@ -65,7 +65,7 @@ class CallbackProcessor : public core::Processor {
public:
- void setCallback(void *obj,std::function
ontrigger_callback) {
+ void setCallback(void *obj,std::function
ontrigger_callback) {
--- End diff --
It's been a while since I've looked at that work, but did the python
bindings call this correctly when running python processors?
> add processor and add processor with linkage nomenclature is confusing
> --
>
> Key: MINIFICPP-648
> URL: https://issues.apache.org/jira/browse/MINIFICPP-648
> Project: NiFi MiNiFi C++
> Issue Type: Improvement
>Reporter: Mr TheSegfault
>Assignee: Arpad Boda
>Priority: Blocker
> Labels: CAPI
> Fix For: 0.6.0
>
>
> add_processor should be changed to always add a processor with linkage
> without compelling documentation as why this exists.. As a result we will
> need to add a create_processor function to create one without adding it to
> the flow ( certain use cases where a flow isn't needed such as invokehttp or
> listenhttp ) this can be moved to 0.7.0 if we tag before recent commits.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (MINIFICPP-648) add processor and add processor with linkage nomenclature is confusing
[
https://issues.apache.org/jira/browse/MINIFICPP-648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16679741#comment-16679741
]
ASF GitHub Bot commented on MINIFICPP-648:
--
Github user arpadboda commented on a diff in the pull request:
https://github.com/apache/nifi-minifi-cpp/pull/432#discussion_r231879717
--- Diff: nanofi/src/api/nanofi.cpp ---
@@ -89,6 +93,39 @@ nifi_instance *create_instance(const char *url,
nifi_port *port) {
return instance;
}
+standalone_processor *create_processor(const char *name) {
+ static int proc_counter = 0;
+ auto ptr = ExecutionPlan::createProcessor(name, name);
+ if (!ptr) {
+return nullptr;
+ }
+ if (standalone_instance == nullptr) {
+nifi_port port;
--- End diff --
Created with dummy value. Do you think having a valid input port is valid
usecase for an instance we don't even expose? (As we use it to "store"
standalone processors)
> add processor and add processor with linkage nomenclature is confusing
> --
>
> Key: MINIFICPP-648
> URL: https://issues.apache.org/jira/browse/MINIFICPP-648
> Project: NiFi MiNiFi C++
> Issue Type: Improvement
>Reporter: Mr TheSegfault
>Assignee: Arpad Boda
>Priority: Blocker
> Labels: CAPI
> Fix For: 0.6.0
>
>
> add_processor should be changed to always add a processor with linkage
> without compelling documentation as why this exists.. As a result we will
> need to add a create_processor function to create one without adding it to
> the flow ( certain use cases where a flow isn't needed such as invokehttp or
> listenhttp ) this can be moved to 0.7.0 if we tag before recent commits.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (NIFI-5794) ConsumeKafka and PublishKafka should allow empty string demarcator
[ https://issues.apache.org/jira/browse/NIFI-5794?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16679642#comment-16679642 ] ASF GitHub Bot commented on NIFI-5794: -- GitHub user pvillard31 opened a pull request: https://github.com/apache/nifi/pull/3159 NIFI-5794 Allow empty string demarcator in Consume/PublishKafka proce… …ssors Thank you for submitting a contribution to Apache NiFi. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [ ] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [ ] Does your PR title start with NIFI- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [ ] Has your PR been rebased against the latest commit within the target branch (typically master)? - [ ] Is your initial contribution a single, squashed commit? ### For code changes: - [ ] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder? - [ ] Have you written or updated unit tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly? - [ ] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly? - [ ] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. You can merge this pull request into a Git repository by running: $ git pull https://github.com/pvillard31/nifi NIFI-5794 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi/pull/3159.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3159 commit 88fbcffdc3fa6aadac14c23af9a5012edaa21e0f Author: Pierre Villard Date: 2018-11-07T09:55:10Z NIFI-5794 Allow empty string demarcator in Consume/PublishKafka processors > ConsumeKafka and PublishKafka should allow empty string demarcator > -- > > Key: NIFI-5794 > URL: https://issues.apache.org/jira/browse/NIFI-5794 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Reporter: Pierre Villard >Assignee: Pierre Villard >Priority: Major > > ConsumeKafka(_*) and PublishKafka(_*) processors should allow "empty string" > as a message demarcator. This would allow consuming Avro data without the > serialization/de-serialization cost while still allowing the use of Record > processors once the data is in NiFi. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi pull request #3133: NIFI-5790: Exposes 6 commons-dbcp options in DBCPCo...
Github user patricker commented on a diff in the pull request:
https://github.com/apache/nifi/pull/3133#discussion_r232065708
--- Diff:
nifi-nar-bundles/nifi-standard-services/nifi-dbcp-service-bundle/nifi-dbcp-service/src/test/java/org/apache/nifi/dbcp/DBCPServiceTest.java
---
@@ -93,6 +94,155 @@ public void testMaxWait() throws
InitializationException {
runner.assertValid(service);
}
+/**
+ * Checks validity of idle limit and time settings including a default
+ */
+@Test
+public void testIdleConnectionsSettings() throws
InitializationException {
+final TestRunner runner =
TestRunners.newTestRunner(TestProcessor.class);
+final DBCPConnectionPool service = new DBCPConnectionPool();
+runner.addControllerService("test-good1", service);
+
+// remove previous test database, if any
+final File dbLocation = new File(DB_LOCATION);
+dbLocation.delete();
+
+// set embedded Derby database connection url
+runner.setProperty(service, DBCPConnectionPool.DATABASE_URL,
"jdbc:derby:" + DB_LOCATION + ";create=true");
+runner.setProperty(service, DBCPConnectionPool.DB_USER, "tester");
+runner.setProperty(service, DBCPConnectionPool.DB_PASSWORD,
"testerp");
+runner.setProperty(service, DBCPConnectionPool.DB_DRIVERNAME,
"org.apache.derby.jdbc.EmbeddedDriver");
+runner.setProperty(service, DBCPConnectionPool.MAX_WAIT_TIME,
"-1");
+runner.setProperty(service, DBCPConnectionPool.MAX_IDLE, "2");
+runner.setProperty(service, DBCPConnectionPool.MAX_CONN_LIFETIME,
"1 secs");
+runner.setProperty(service,
DBCPConnectionPool.EVICTION_RUN_PERIOD, "1 secs");
+runner.setProperty(service,
DBCPConnectionPool.MIN_EVICTABLE_IDLE_TIME, "1 secs");
+runner.setProperty(service,
DBCPConnectionPool.SOFT_MIN_EVICTABLE_IDLE_TIME, "1 secs");
+
+runner.enableControllerService(service);
+runner.assertValid(service);
+}
+
+@Test
+public void testMinIdleCannotBeNegative() throws
InitializationException {
+final TestRunner runner =
TestRunners.newTestRunner(TestProcessor.class);
+final DBCPConnectionPool service = new DBCPConnectionPool();
+runner.addControllerService("test-good1", service);
+
+// remove previous test database, if any
+final File dbLocation = new File(DB_LOCATION);
+dbLocation.delete();
+
+// set embedded Derby database connection url
+runner.setProperty(service, DBCPConnectionPool.DATABASE_URL,
"jdbc:derby:" + DB_LOCATION + ";create=true");
+runner.setProperty(service, DBCPConnectionPool.DB_USER, "tester");
+runner.setProperty(service, DBCPConnectionPool.DB_PASSWORD,
"testerp");
+runner.setProperty(service, DBCPConnectionPool.DB_DRIVERNAME,
"org.apache.derby.jdbc.EmbeddedDriver");
+runner.setProperty(service, DBCPConnectionPool.MAX_WAIT_TIME,
"-1");
+runner.setProperty(service, DBCPConnectionPool.MIN_IDLE, "-1");
+
+runner.assertNotValid(service);
+}
+
+/**
+ * Checks to ensure that settings have been passed down into the DBCP
+ */
+@Test
+public void testIdleSettingsAreSet() throws InitializationException,
SQLException {
+final TestRunner runner =
TestRunners.newTestRunner(TestProcessor.class);
+final DBCPConnectionPool service = new DBCPConnectionPool();
+runner.addControllerService("test-good1", service);
+
+// remove previous test database, if any
+final File dbLocation = new File(DB_LOCATION);
+dbLocation.delete();
+
+// set embedded Derby database connection url
+runner.setProperty(service, DBCPConnectionPool.DATABASE_URL,
"jdbc:derby:" + DB_LOCATION + ";create=true");
+runner.setProperty(service, DBCPConnectionPool.DB_USER, "tester");
+runner.setProperty(service, DBCPConnectionPool.DB_PASSWORD,
"testerp");
+runner.setProperty(service, DBCPConnectionPool.DB_DRIVERNAME,
"org.apache.derby.jdbc.EmbeddedDriver");
+runner.setProperty(service, DBCPConnectionPool.MAX_WAIT_TIME,
"-1");
+runner.setProperty(service, DBCPConnectionPool.MAX_IDLE, "6");
+runner.setProperty(service, DBCPConnectionPool.MIN_IDLE, "4");
+runner.setProperty(service, DBCPConnectionPool.MAX_CONN_LIFETIME,
"1 secs");
+runner.setProperty(service,
DBCPConnectionPool.EVICTION_RUN_PERIOD, "1 secs");
+runner.setProperty(service,
DBCPConnectionPool.MIN_EVICTABLE_IDLE_TIME, "1 secs");
+runner.setProperty(service,
DBCPConnectionPool.SOFT_MIN_EVICTABLE_IDLE_
[jira] [Resolved] (MINIFICPP-641) C API: add support to register failure callback
[ https://issues.apache.org/jira/browse/MINIFICPP-641?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpad Boda resolved MINIFICPP-641. -- Resolution: Fixed > C API: add support to register failure callback > --- > > Key: MINIFICPP-641 > URL: https://issues.apache.org/jira/browse/MINIFICPP-641 > Project: NiFi MiNiFi C++ > Issue Type: New Feature >Reporter: Arpad Boda >Assignee: Arpad Boda >Priority: Minor > Labels: CAPI > Fix For: 0.6.0 > > > Current implementation of C API provide no option to handle failure > relationships of processors. > Simply adding relationships one-by-one would make it unreasonably complicated > in user's perspective. User should be able to add a simple callback, which - > in the background - creates a processor and connects that all processors in > the given flow with failure relationship. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (MINIFICPP-648) add processor and add processor with linkage nomenclature is confusing
[
https://issues.apache.org/jira/browse/MINIFICPP-648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16679692#comment-16679692
]
ASF GitHub Bot commented on MINIFICPP-648:
--
Github user phrocker commented on a diff in the pull request:
https://github.com/apache/nifi-minifi-cpp/pull/432#discussion_r231867815
--- Diff: nanofi/src/api/nanofi.cpp ---
@@ -323,55 +360,45 @@ int transmit_flowfile(flow_file_record *ff,
nifi_instance *instance) {
flow * create_new_flow(nifi_instance * instance) {
auto minifi_instance_ref =
static_cast(instance->instance_ptr);
- flow *new_flow = (flow*) malloc(sizeof(flow));
-
- auto execution_plan = new
ExecutionPlan(minifi_instance_ref->getContentRepository(),
minifi_instance_ref->getNoOpRepository(),
minifi_instance_ref->getNoOpRepository());
-
- new_flow->plan = execution_plan;
-
- return new_flow;
+ return new flow(minifi_instance_ref->getContentRepository(),
minifi_instance_ref->getNoOpRepository(),
minifi_instance_ref->getNoOpRepository());
}
flow *create_flow(nifi_instance *instance, const char *first_processor) {
if (nullptr == instance || nullptr == instance->instance_ptr) {
return nullptr;
}
auto minifi_instance_ref =
static_cast(instance->instance_ptr);
- flow *new_flow = (flow*) malloc(sizeof(flow));
-
- auto execution_plan = new
ExecutionPlan(minifi_instance_ref->getContentRepository(),
minifi_instance_ref->getNoOpRepository(),
minifi_instance_ref->getNoOpRepository());
- new_flow->plan = execution_plan;
+ flow *new_flow = new flow(minifi_instance_ref->getContentRepository(),
minifi_instance_ref->getNoOpRepository(),
minifi_instance_ref->getNoOpRepository());
if (first_processor != nullptr && strlen(first_processor) > 0) {
// automatically adds it with success
-execution_plan->addProcessor(first_processor, first_processor);
--- End diff --
Combined C and C++ implies we're moving more toward C++. If that were the
case this API would be drastically different. We should move in the other
direction since C++ functions as a temporary interconnect.
> add processor and add processor with linkage nomenclature is confusing
> --
>
> Key: MINIFICPP-648
> URL: https://issues.apache.org/jira/browse/MINIFICPP-648
> Project: NiFi MiNiFi C++
> Issue Type: Improvement
>Reporter: Mr TheSegfault
>Assignee: Arpad Boda
>Priority: Blocker
> Labels: CAPI
> Fix For: 0.6.0
>
>
> add_processor should be changed to always add a processor with linkage
> without compelling documentation as why this exists.. As a result we will
> need to add a create_processor function to create one without adding it to
> the flow ( certain use cases where a flow isn't needed such as invokehttp or
> listenhttp ) this can be moved to 0.7.0 if we tag before recent commits.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (NIFI-5724) Make the autocommit value in the PutSQL processor configurable
[
https://issues.apache.org/jira/browse/NIFI-5724?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16679193#comment-16679193
]
ASF GitHub Bot commented on NIFI-5724:
--
Github user viswaug commented on a diff in the pull request:
https://github.com/apache/nifi/pull/3113#discussion_r231747175
--- Diff:
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/PutSQL.java
---
@@ -134,6 +134,14 @@
.expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
.build();
+static final PropertyDescriptor AUTO_COMMIT = new
PropertyDescriptor.Builder()
+.name("database-session-autocommit")
+.displayName("Database session autocommit value")
+.description("The autocommit mode to set on the database
connection being used.")
+.allowableValues("true", "false")
+.defaultValue("false")
+.build();
--- End diff --
@ijokarumawak that git(hub) part was much easier than i expected it to
be... i checked in my changes... let me know if you need anything tweaked or
changed 🤞
> Make the autocommit value in the PutSQL processor configurable
> --
>
> Key: NIFI-5724
> URL: https://issues.apache.org/jira/browse/NIFI-5724
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Extensions
>Reporter: vish uma
>Priority: Minor
>
> The PutSQL processor currently always sets the autocommit value on the
> database session to false before the SQL statement is run and resets it back
> to the original value after.
> i am not sure if the autocommit value is hardcoded to false for a reason, if
> it is, please let me know.
> This is causing an issue with the snowflake DB where abruptly disconnected
> sessions do not release the locks they have taken.
> i would like to make this autocommit value configurable. I can submit a patch
> for this if there is no objections.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (MINIFICPP-648) add processor and add processor with linkage nomenclature is confusing
[
https://issues.apache.org/jira/browse/MINIFICPP-648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16679693#comment-16679693
]
ASF GitHub Bot commented on MINIFICPP-648:
--
Github user phrocker commented on a diff in the pull request:
https://github.com/apache/nifi-minifi-cpp/pull/432#discussion_r231865428
--- Diff: nanofi/include/cxx/Plan.h ---
@@ -133,8 +133,29 @@ class ExecutionPlan {
next_ff_ = ptr;
}
+ bool hasProcessor() {
+return !processor_queue_.empty();
+ }
+
static std::shared_ptr createProcessor(const
std::string &processor_name, const std::string &name);
+ static std::shared_ptr getPlan(const std::string& uuid) {
+auto it = proc_plan_map_.find(uuid);
+return it != proc_plan_map_.end() ? it->second : nullptr;
+ }
+
+ static void addProcWithPlan(const std::string& uuid,
std::shared_ptr plan) {
--- End diff --
addProcessorWithPlan
> add processor and add processor with linkage nomenclature is confusing
> --
>
> Key: MINIFICPP-648
> URL: https://issues.apache.org/jira/browse/MINIFICPP-648
> Project: NiFi MiNiFi C++
> Issue Type: Improvement
>Reporter: Mr TheSegfault
>Assignee: Arpad Boda
>Priority: Blocker
> Labels: CAPI
> Fix For: 0.6.0
>
>
> add_processor should be changed to always add a processor with linkage
> without compelling documentation as why this exists.. As a result we will
> need to add a create_processor function to create one without adding it to
> the flow ( certain use cases where a flow isn't needed such as invokehttp or
> listenhttp ) this can be moved to 0.7.0 if we tag before recent commits.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (NIFI-5794) ConsumeKafka and PublishKafka should allow empty string demarcator
[ https://issues.apache.org/jira/browse/NIFI-5794?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pierre Villard updated NIFI-5794: - Status: Patch Available (was: Open) > ConsumeKafka and PublishKafka should allow empty string demarcator > -- > > Key: NIFI-5794 > URL: https://issues.apache.org/jira/browse/NIFI-5794 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Reporter: Pierre Villard >Assignee: Pierre Villard >Priority: Major > > ConsumeKafka processors should allow "empty string" as a message demarcator. > This would allow consuming Avro data without the > serialization/de-serialization cost while still allowing the use of Record > processors once the data is in NiFi. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (NIFI-5794) ConsumeKafka and PublishKafka should allow empty string demarcator
[ https://issues.apache.org/jira/browse/NIFI-5794?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pierre Villard updated NIFI-5794: - Description: ConsumeKafka processors should allow "empty string" as a message demarcator. This would allow consuming Avro data without the serialization/de-serialization cost while still allowing the use of Record processors once the data is in NiFi. (was: ConsumeKafka(_*) and PublishKafka(_*) processors should allow "empty string" as a message demarcator. This would allow consuming Avro data without the serialization/de-serialization cost while still allowing the use of Record processors once the data is in NiFi.) > ConsumeKafka and PublishKafka should allow empty string demarcator > -- > > Key: NIFI-5794 > URL: https://issues.apache.org/jira/browse/NIFI-5794 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Reporter: Pierre Villard >Assignee: Pierre Villard >Priority: Major > > ConsumeKafka processors should allow "empty string" as a message demarcator. > This would allow consuming Avro data without the > serialization/de-serialization cost while still allowing the use of Record > processors once the data is in NiFi. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (NIFI-5801) Evaluating Expression Language can in many cases be made much more efficient
[
https://issues.apache.org/jira/browse/NIFI-5801?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mark Payne updated NIFI-5801:
-
Fix Version/s: 1.9.0
Status: Patch Available (was: Open)
> Evaluating Expression Language can in many cases be made much more efficient
>
>
> Key: NIFI-5801
> URL: https://issues.apache.org/jira/browse/NIFI-5801
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework
>Reporter: Mark Payne
>Assignee: Mark Payne
>Priority: Major
> Fix For: 1.9.0
>
>
> When a StandardPropertyValue is obtained and evaluateAttributeExpressions is
> called, it builds the entire Evaluator Tree each time. This was done to
> ensure that Evaluator.evaluate() is called only once. However, the
> requirement to call this only once was introduced as a way to have
> anyMatchingAttribute, anyAttribute, allMatchingAttributes, allAttributes,
> etc. methods work, and these are rarely used. I.e., we introduced semantics
> that significantly slow performance in order to provide functionality that is
> used maybe 1% of the time. Instead, we should optimize for the 99% use case
> and incur a penalty, if necessary, in the 1% use case instead. Profiling the
> ConsumeKafkaRecord processor shows that 80% of the time in that method is
> evaluating Expression Language for `${schema.name}` to determine which schema
> should be used. We can likely make this evaluation just as quick as
> attributeMap.get("schema.name") by pre-building the Evaluators and re-using
> them.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (MINIFICPP-648) add processor and add processor with linkage nomenclature is confusing
[ https://issues.apache.org/jira/browse/MINIFICPP-648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16679733#comment-16679733 ] ASF GitHub Bot commented on MINIFICPP-648: -- Github user arpadboda commented on a diff in the pull request: https://github.com/apache/nifi-minifi-cpp/pull/432#discussion_r231878067 --- Diff: nanofi/include/core/cxxstructs.h --- @@ -0,0 +1,41 @@ +/** + * + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#ifndef NIFI_MINIFI_CPP_CXXSTRUCTS_H +#define NIFI_MINIFI_CPP_CXXSTRUCTS_H + +#include "cstructs.h" +#include "cxx/Plan.h" + --- End diff -- I don't think is moving that way, just simplifies the current structure. There is no change in API, no change in the background C++ classes, just removed the C struct we used to interconnect. In case the user doesn't know anything about the internal layout (the C struct only contains a void*), creating a struct just adds an extra hop in the memory, but no value. With the change the user only sees these structures declared, but not defined and gets pointers. The reason we can do it is that the API user doesn't need to know the size or the internal representation of the object in case only gets and passes pointers. We can change the internal layout as we want without impacting the API anyhow. Even C structs can be added later, that's fine. > add processor and add processor with linkage nomenclature is confusing > -- > > Key: MINIFICPP-648 > URL: https://issues.apache.org/jira/browse/MINIFICPP-648 > Project: NiFi MiNiFi C++ > Issue Type: Improvement >Reporter: Mr TheSegfault >Assignee: Arpad Boda >Priority: Blocker > Labels: CAPI > Fix For: 0.6.0 > > > add_processor should be changed to always add a processor with linkage > without compelling documentation as why this exists.. As a result we will > need to add a create_processor function to create one without adding it to > the flow ( certain use cases where a flow isn't needed such as invokehttp or > listenhttp ) this can be moved to 0.7.0 if we tag before recent commits. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (NIFI-5800) If RecordSchema has an inner field that references a schema recursively by name, hashCode() throws StackOverflowError
[
https://issues.apache.org/jira/browse/NIFI-5800?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pierre Villard updated NIFI-5800:
-
Component/s: Core Framework
> If RecordSchema has an inner field that references a schema recursively by
> name, hashCode() throws StackOverflowError
> -
>
> Key: NIFI-5800
> URL: https://issues.apache.org/jira/browse/NIFI-5800
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework
>Reporter: Mark Payne
>Assignee: Mark Payne
>Priority: Major
> Fix For: 1.9.0
>
>
> If we have a schema where a field is of type RECORD and references an
> outer-level schema, we get StackOverflowError. For example:
> {
> "name": "person",
> "namespace": "nifi",
> "type": "record",
> "fields": [
> { "name": "name", "type": "string" },
> { "name": "mother", "type": "person" }
> ]
> }
> In this case, if we attempt to add this schema to a HashMap, we get the
> following error:
> 2018-11-07 19:09:33,021 ERROR [Timer-Driven Process Thread-38]
> o.a.n.p.k.pubsub.ConsumeKafkaRecord_2_0
> ConsumeKafkaRecord_2_0[id=ef6bd50b-0166-1000--f55a7995] Exception
> while processing data from kafka so will close the lease
> org.apache.nifi.processors.kafka.pubsub.ConsumerPool$SimpleConsumerLease@26706081
> due to java.lang.StackOverflowError: java.lang.StackOverflowError
> java.lang.StackOverflowError: null
> at java.util.AbstractList.hashCode(AbstractList.java:540)
> at
> java.util.Collections$UnmodifiableList.hashCode(Collections.java:1307)
> at
> org.apache.nifi.serialization.SimpleRecordSchema.hashCode(SimpleRecordSchema.java:172)
> at
> org.apache.nifi.serialization.record.type.RecordDataType.hashCode(RecordDataType.java:45)
> at
> org.apache.nifi.serialization.record.type.ArrayDataType.hashCode(ArrayDataType.java:44)
> at
> org.apache.nifi.serialization.record.RecordField.hashCode(RecordField.java:108)
> at java.util.AbstractList.hashCode(AbstractList.java:541)
> at
> java.util.Collections$UnmodifiableList.hashCode(Collections.java:1307)
> at
> org.apache.nifi.serialization.SimpleRecordSchema.hashCode(SimpleRecordSchema.java:172)
> at
> org.apache.nifi.serialization.record.type.RecordDataType.hashCode(RecordDataType.java:45)
> at
> org.apache.nifi.serialization.record.type.ArrayDataType.hashCode(ArrayDataType.java:44)
> at
> org.apache.nifi.serialization.record.RecordField.hashCode(RecordField.java:108)
> at java.util.AbstractList.hashCode(AbstractList.java:541)
> at
> java.util.Collections$UnmodifiableList.hashCode(Collections.java:1307)
> at
> org.apache.nifi.serialization.SimpleRecordSchema.hashCode(SimpleRecordSchema.java:172)
> at
> org.apache.nifi.serialization.record.type.RecordDataType.hashCode(RecordDataType.java:45)
> at
> org.apache.nifi.serialization.record.type.ArrayDataType.hashCode(ArrayDataType.java:44)
> at
> org.apache.nifi.serialization.record.RecordField.hashCode(RecordField.java:108)
> at java.util.AbstractList.hashCode(AbstractList.java:541)
> at
> java.util.Collections$UnmodifiableList.hashCode(Collections.java:1307)
> at
> org.apache.nifi.serialization.SimpleRecordSchema.hashCode(SimpleRecordSchema.java:172)
> at
> org.apache.nifi.serialization.record.type.RecordDataType.hashCode(RecordDataType.java:45)
> at
> org.apache.nifi.serialization.record.type.ArrayDataType.hashCode(ArrayDataType.java:44)
> at
> org.apache.nifi.serialization.record.RecordField.hashCode(RecordField.java:108)
> at java.util.AbstractList.hashCode(AbstractList.java:541)
> at
> java.util.Collections$UnmodifiableList.hashCode(Collections.java:1307)
> at
> org.apache.nifi.serialization.SimpleRecordSchema.hashCode(SimpleRecordSchema.java:172)
> at
> org.apache.nifi.serialization.record.type.RecordDataType.hashCode(RecordDataType.java:45)
> at
> org.apache.nifi.serialization.record.type.ArrayDataType.hashCode(ArrayDataType.java:44)
> at
> org.apache.nifi.serialization.record.RecordField.hashCode(RecordField.java:108)
> at java.util.AbstractList.hashCode(AbstractList.java:541)
> at
> java.util.Collections$UnmodifiableList.hashCode(Collections.java:1307)
> ...
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (NIFI-5784) With the addition of the Toolkit Guide, edit other docs that contain duplicate content
[ https://issues.apache.org/jira/browse/NIFI-5784?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pierre Villard resolved NIFI-5784. -- Resolution: Fixed Fix Version/s: 1.9.0 > With the addition of the Toolkit Guide, edit other docs that contain > duplicate content > -- > > Key: NIFI-5784 > URL: https://issues.apache.org/jira/browse/NIFI-5784 > Project: Apache NiFi > Issue Type: Improvement > Components: Documentation & Website >Reporter: Andrew Lim >Assignee: Andrew Lim >Priority: Minor > Fix For: 1.9.0 > > > The Admin Guide has duplicate content for the following: > * Configuration encryption - > [https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#encrypt-config_tool] > * File manager - > [https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#file-manager] > * Node manager - > [https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#node-manager] > * TLS Toolkit - > [https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#tls_generation_toolkit] > * ZooKeeper migrator - > [https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#zookeeper_migrator] > Will remove and add links to Toolkit Guide as needed. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-5791) Add Apache Daffodil parse/unparse processor
[ https://issues.apache.org/jira/browse/NIFI-5791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16679938#comment-16679938 ] ASF GitHub Bot commented on NIFI-5791: -- Github user stevedlawrence commented on the issue: https://github.com/apache/nifi/pull/3130 Actually, I should probably add a little more clarification. It is correct to say that Daffodil could be considered a transformation capability, but I'm not sure it could really be seen as an alternative to Jolt specifically. One could definitely write a DFDL schema to describe and parse JSON data, similar to what Jolt can do, but the resulting infoset would just be JSON that is probably more generic and less useful that what Jolt would create. Daffodil does not have the same transformational capabilities that Jolt has (e.g. filtering, sorting, reordering). It really just takes a data format that is not natively supported and transforms it to a logically similar representation in a natively supported format (i.e. XML/JSON). If your data is already in something like XML or JSON, it doesn't really make sense to use Daffodil. So for example, you might take PCAP data, like in the above template, and feed it through Daffodill to transform it to JSON. But there might be extraneous data that Daffodil parsed that you don't need or care about. So you might then feed that JSON through Jolt to to transform it to something that has only the fields you care about, or is reordered/renamed to be something more useful to ones usecase. So this isn't something that could be viewed as an alternative to Jolt, but more like something that could be used in conjunction with Jolt to allow one to transform JSON data that wasn't originally JSON. > Add Apache Daffodil parse/unparse processor > --- > > Key: NIFI-5791 > URL: https://issues.apache.org/jira/browse/NIFI-5791 > Project: Apache NiFi > Issue Type: Bug > Components: Extensions >Reporter: Steve Lawrence >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] nifi pull request #3156: NIFI-5780 Add pre and post statements to ExecuteSQL...
Github user patricker commented on a diff in the pull request:
https://github.com/apache/nifi/pull/3156#discussion_r232072719
--- Diff:
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/AbstractExecuteSQL.java
---
@@ -82,6 +84,16 @@
.identifiesControllerService(DBCPService.class)
.build();
+public static final PropertyDescriptor SQL_PRE_QUERY = new
PropertyDescriptor.Builder()
+.name("sql-pre-query")
+.displayName("SQL Pre-Query")
+.description("SQL pre-query to execute. Semicolon-delimited
list of queries. "
--- End diff --
@colindean I understand why it was done this way. This functionality is
being ported from the HIVE processor where the description begins: `HiveQL
pre-query to execute...`. The same for the `Note` comment below.
That doesn't mean it couldn't be clarified, but it is currently internally
consistent.
@yjhyjhyjh0 How about, "A semicolon-delimited list of queries executed
executed before the main SQL query is executed. Results/outputs from these
queries will be suppressed if there are no errors."
---
[GitHub] nifi pull request #3156: NIFI-5780 Add pre and post statements to ExecuteSQL...
Github user patricker commented on a diff in the pull request:
https://github.com/apache/nifi/pull/3156#discussion_r232072895
--- Diff:
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/AbstractExecuteSQL.java
---
@@ -94,6 +106,16 @@
.expressionLanguageSupported(ExpressionLanguageScope.FLOWFILE_ATTRIBUTES)
.build();
+public static final PropertyDescriptor SQL_POST_QUERY = new
PropertyDescriptor.Builder()
+.name("sql-post-query")
+.displayName("SQL Post-Query")
+.description("SQL post-query to execute. Semicolon-delimited
list of queries. "
--- End diff --
"A semicolon-delimited list of queries executed after the main SQL query is
executed. Results/outputs from these queries will be suppressed if there are no
errors."
---
