[GitHub] nifi issue #1917: NIFI-4074: Fix recursive regex search
Github user brosander commented on the issue: https://github.com/apache/nifi/pull/1917 Hey @champagst Thanks for the contribution! I was digging into the ListHDFS implementation and I think the reason it doesn't work on nested files might be that it is only matching against the [file name, not the whole path.](https://github.com/apache/nifi/blob/e85081ed201bc7413e3d2d062c814e23c0ff5467/nifi-nar-bundles/nifi-hadoop-bundle/nifi-hdfs-processors/src/main/java/org/apache/nifi/processors/hadoop/ListHDFS.java#L406) Would adding a property that tells the filter to match [on the whole path](https://docs.oracle.com/javase/8/docs/api/java/nio/file/Path.html#toString--), not just the name address your usecase? That way we could default it to false and prevent existing flows from having their semantics changed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi pull request #1888: NIFI-4015 NIFI-3999 Fix DeleteSQS Issues
Github user brosander commented on a diff in the pull request:
https://github.com/apache/nifi/pull/1888#discussion_r124067647
--- Diff:
nifi-nar-bundles/nifi-aws-bundle/nifi-aws-processors/src/main/java/org/apache/nifi/processors/aws/sqs/DeleteSQS.java
---
@@ -45,7 +46,8 @@
public class DeleteSQS extends AbstractSQSProcessor {
public static final PropertyDescriptor RECEIPT_HANDLE = new
PropertyDescriptor.Builder()
-.name("Receipt Handle")
+.name("sqs-receipt-handle")
--- End diff --
@jvwing
I think changing the name attribute at this point will break existing flows.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi issue #1888: NIFI-4015 NIFI-3999 Fix DeleteSQS Issues
Github user brosander commented on the issue: https://github.com/apache/nifi/pull/1888 Reviewing --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi issue #1919: NIFI-4031: Fix Avro nullable arrays
Github user brosander commented on the issue: https://github.com/apache/nifi/pull/1919 @champagst thanks for the contribution! Would you mind adding tests that exercise this functionality? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi pull request #1390: NIFI-3277 - StandardProcessSession.isSpuriousRouteE...
Github user brosander closed the pull request at: https://github.com/apache/nifi/pull/1390 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi-minifi-cpp pull request #115: MINIFI-324 - More flexible uid generation
Github user brosander commented on a diff in the pull request:
https://github.com/apache/nifi-minifi-cpp/pull/115#discussion_r123745136
--- Diff: libminifi/include/utils/Id.h ---
@@ -0,0 +1,77 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+#ifndef LIBMINIFI_INCLUDE_UTILS_ID_H_
+#define LIBMINIFI_INCLUDE_UTILS_ID_H_
+
+#include
+#include
+#include
+#include
+
+#include "core/logging/Logger.h"
+#include "properties/Properties.h"
+
+#define UNSIGNED_CHAR_MAX 255
+#define UUID_TIME_IMPL 0
+#define UUID_RANDOM_IMPL 1
+#define UUID_DEFAULT_IMPL 2
+#define MINIFI_UID_IMPL 3
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+namespace utils {
+
+class IdGenerator {
+ public:
+ void generate(uuid_t output);
+ void initialize(const std::shared_ptr & properties);
+
+ static std::shared_ptr getIdGenerator() {
+static std::shared_ptr generator =
std::shared_ptr(new IdGenerator());
--- End diff --
I wanted to make the constructor private that seems to require manual
invocation of new
```
error: âorg::apache::nifi::minifi::utils::IdGenerator::IdGenerator()â
is private
IdGenerator();
^
In file included from
/usr/include/x86_64-linux-gnu/c++/5/bits/c++allocator.h:33:0,
from /usr/include/c++/5/bits/allocator.h:46,
from /usr/include/c++/5/bits/stl_tree.h:64,
from /usr/include/c++/5/map:60,
from
/home/bryan/Github/nifi-minifi-cpp/libminifi/include/core/logging/LoggerConfiguration.h:24,
from
/home/bryan/Github/nifi-minifi-cpp/libminifi/src/core/logging/LoggerConfiguration.cpp:21:
/usr/include/c++/5/ext/new_allocator.h:120:4: error: within this context
{ ::new((void *)__p) _Up(std::forward<_Args>(__args)...); }
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi-minifi-cpp pull request #115: MINIFI-324 - More flexible uid generation
Github user brosander commented on a diff in the pull request: https://github.com/apache/nifi-minifi-cpp/pull/115#discussion_r123744302 --- Diff: README.md --- @@ -326,26 +326,44 @@ Additionally, users can utilize the MiNiFi Toolkit Converter (version 0.0.1 - sc ### Http Configuration Listener -Http Configuration Listener will pull flow file configuration from the remote command control server, +Http Configuration Listener will pull flow file configuration from the remote command control server, validate and apply the new flow configuration -in minifi.properties +in minifi.properties nifi.configuration.listener.type=http nifi.configuration.listener.http.url=https://localhost:8080 -nifi.configuration.listener.pull.interval=1 sec +nifi.configuration.listener.pull.interval=1 sec nifi.configuration.listener.client.ca.certificate=./conf/nifi-cert.pem if you want to enable client certificate nifi.configuration.listener.need.ClientAuth=true nifi.configuration.listener.client.certificate=./conf/client.pem nifi.configuration.listener.client.private.key=./conf/client.key nifi.configuration.listener.client.pass.phrase=./conf/password - + +### UID generation + +MiNiFi need to generate many unique identifiers in the course of operations. There are a few different uid implementations available that can be configured in minifi-uid.properties. + +Implementation for uid generation can be selected using the uid.implementation property values: +1. time - use uuid_generate_time +2. random - use uuid_generate_random +3. uuid_default - use uuid_generate (will attempt to use uuid_generate_random and fall back to uuid_generate_time if no high quality randomness is available) --- End diff -- Well, uuid_generate_time is a lot faster than the random version but it leaks information about which node and what time the uuid originate from, I thought that might be relevant info if there are privacy-conscious applications of MiNiFi. https://linux.die.net/man/3/uuid_generate_time --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi-minifi-cpp pull request #115: MINIFI-324 - More flexible uid generation
Github user brosander commented on a diff in the pull request:
https://github.com/apache/nifi-minifi-cpp/pull/115#discussion_r123743755
--- Diff: libminifi/src/provenance/Provenance.cpp ---
@@ -34,9 +36,24 @@ namespace nifi {
namespace minifi {
namespace provenance {
+std::shared_ptr ProvenanceEventRecord::id_generator_ =
utils::IdGenerator::getIdGenerator();
+std::shared_ptr ProvenanceEventRecord::logger_ =
logging::LoggerFactory::getLogger();
+
const char *ProvenanceEventRecord::ProvenanceEventTypeStr[REPLAY + 1] = {
"CREATE", "RECEIVE", "FETCH", "SEND", "DOWNLOAD", "DROP", "EXPIRE", "FORK",
"JOIN", "CLONE", "CONTENT_MODIFIED",
"ATTRIBUTES_MODIFIED", "ROUTE", "ADDINFO", "REPLAY" };
+ProvenanceEventRecord::ProvenanceEventRecord(ProvenanceEventRecord::ProvenanceEventType
event, std::__cxx11::string componentId, std::__cxx11::string componentType) {
--- End diff --
Autocomplete fail, will fix
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi-minifi-cpp pull request #115: MINIFI-324 - More flexible uid generation
GitHub user brosander opened a pull request: https://github.com/apache/nifi-minifi-cpp/pull/115 MINIFI-324 - More flexible uid generation Thank you for submitting a contribution to Apache NiFi - MiNiFi C++. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [X] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [X] Does your PR title start with MINIFI- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [X] Has your PR been rebased against the latest commit within the target branch (typically master)? - [X] Is your initial contribution a single, squashed commit? ### For code changes: - [X] - N/A - If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [X] - N/A - If applicable, have you updated the LICENSE file? - [X] - N/A - If applicable, have you updated the NOTICE file? ### For documentation related changes: - [X] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. You can merge this pull request into a Git repository by running: $ git pull https://github.com/brosander/nifi-minifi-cpp MINIFI-324 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi-minifi-cpp/pull/115.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #115 commit 2ce366c5547aa6608653ce71d98e335ff3a0a9ff Author: Bryan Rosander <brosan...@apache.org> Date: 2017-06-21T18:29:03Z MINIFI-324 - More flexible uid generation --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi issue #1889: NIFI-3696 - Initial commit of ConfigMigration and FileMana...
Github user brosander commented on the issue: https://github.com/apache/nifi/pull/1889 + 1 on condition that we document limitations of downgrade with respect to repos --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi pull request #1889: NIFI-3696 - Initial commit of ConfigMigration and F...
Github user brosander commented on a diff in the pull request: https://github.com/apache/nifi/pull/1889#discussion_r120720022 --- Diff: nifi-docs/src/main/asciidoc/administration-guide.adoc --- @@ -1317,6 +1318,82 @@ cluster if restarted and the flow for the cluster has not changed. If the flow w the removed node should be deleted before restarting the node to allow it to obtain the cluster flow (otherwise an uninheritable flow file exception may occur). +=== File Manager + +The File Manager utility allows system administrators to take a backup of an existing NiFi installation, install a new version of NiFi +in a designated location (while migrating any previous configuration settings) or restore an installation from a previous backup. +File Manager supports NiFi version 1.0.0 and higher and is available in 'file-manager.bat' file for use on Windows machines. + +To show help: + + file-manager.sh -h + +The following are available options: + +* `-o,--operation ` File operation (install | backup | restore) +* `-b,--backupDir ` Backup NiFi Directory (used with backup or restore operation) +* `-c,--nifiCurrentDir ` Current NiFi Installation Directory (used optionally with install or restore operation) +* `-d,--nifiInstallDir ` NiFi Installation Directory (used with install or restore operation) +* `-i,--installFile ` NiFi Install File (used with install operation) +* `-r,--nifiRollbackDir ` NiFi Installation Directory (used with install or restore operation) +* `-t,--bootstrapConf ` Current NiFi Bootstrap Configuration File (used optionally) +* `-m,--moveRepositories` Allow repositories to be moved to new/restored nifi directory from existing installation, if available (used optionally with install or restore operation) +* `-x,--overwriteConfigs` Overwrite existing configuration directory with upgrade changes (used optionally with install or restore operation) +* `-v,--verbose` Verbose messaging (optional) +* `-h,--help` Print help info (optional) + +Example usage on Linux: + + # backup NiFi installation + # option -t may be provided to ensure backup of external boostrap.conf file + ./file-manager.sh + -o backup + âb /tmp/nifi_bak + âc /usr/nifi_old + -v + + # install NiFi using compressed tar file into /usr/nifi directory (should install as /usr/nifi/nifi-1.3.0). --- End diff -- We should notify the user that if they upgrade with -m to also move repositories, start nifi, then try to downgrade, it's likely that the older implementation will not be able to read the newer repos. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi-minifi-cpp issue #103: MINIFI-177 Incorporating Apache RAT to build wit...
Github user brosander commented on the issue: https://github.com/apache/nifi-minifi-cpp/pull/103 +1 LGTM --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi issue #1878: NIFI-4002: Add PutElasticsearchHttpRecord processor
Github user brosander commented on the issue: https://github.com/apache/nifi/pull/1878 +1 LGTM --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi-minifi-cpp pull request #103: MINIFI-177 Incorporating Apache RAT to bu...
Github user brosander commented on a diff in the pull request:
https://github.com/apache/nifi-minifi-cpp/pull/103#discussion_r119398723
--- Diff: cmake/RunApacheRAT.cmake ---
@@ -0,0 +1,44 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# Find the preferred Apache mirror to use for the download by querying the
list of mirrors and filtering out 'preferred'
+execute_process(COMMAND curl -s
https://www.apache.org/dyn/closer.lua/?asjson=1
+COMMAND grep preferred
+COMMAND awk "{print $2}"
+COMMAND tr -d "\""
+TIMEOUT 10
+OUTPUT_STRIP_TRAILING_WHITESPACE
+OUTPUT_VARIABLE MIRROR_URL )
+
+ExternalProject_Add(
+rat-binary
+PREFIX "apache-rat"
+URL
"${MIRROR_URL}creadur/apache-rat-0.12/apache-rat-0.12-bin.tar.gz"
+URL_HASH SHA1=e84dffe8b354871c29f5078b823a726508474a6c
--- End diff --
@apiri can we use sha256 or sha512? if it's not available from the apache
mirror, we could precalculate it since it shouldn't change until we upgrade rat
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi-minifi-cpp pull request #103: MINIFI-177 Incorporating Apache RAT to bu...
Github user brosander commented on a diff in the pull request:
https://github.com/apache/nifi-minifi-cpp/pull/103#discussion_r119398420
--- Diff: cmake/RunApacheRAT.cmake ---
@@ -0,0 +1,44 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# Find the preferred Apache mirror to use for the download by querying the
list of mirrors and filtering out 'preferred'
+execute_process(COMMAND curl -s
https://www.apache.org/dyn/closer.lua/?asjson=1
+COMMAND grep preferred
+COMMAND awk "{print $2}"
+COMMAND tr -d "\""
+TIMEOUT 10
+OUTPUT_STRIP_TRAILING_WHITESPACE
+OUTPUT_VARIABLE MIRROR_URL )
+
+ExternalProject_Add(
--- End diff --
@apiri can we make it so this is only downloaded when we run the rat target?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi-minifi-cpp issue #104: MINIFI-296 - Configurable logging, spdlog 0.13.0
Github user brosander commented on the issue: https://github.com/apache/nifi-minifi-cpp/pull/104 removed atomic_load, atomic_store as they are problematic w/ gcc 4.8 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi-minifi-cpp issue #104: MINIFI-296 - Configurable logging, spdlog 0.13.0
Github user brosander commented on the issue: https://github.com/apache/nifi-minifi-cpp/pull/104 rebased pr, fixed http get integration tests --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi-minifi-cpp pull request #104: MINIFI-296 - Configurable logging, spdlog...
Github user brosander commented on a diff in the pull request:
https://github.com/apache/nifi-minifi-cpp/pull/104#discussion_r118706272
--- Diff: libminifi/test/TestBase.h ---
@@ -21,35 +21,104 @@
#include
#include
#include
+#include
#include "ResourceClaim.h"
#include "catch.hpp"
#include
-#include "core/logging/LogAppenders.h"
#include "core/logging/Logger.h"
#include "core/Core.h"
#include "properties/Configure.h"
+#include "properties/Properties.h"
+#include "core/logging/LoggerConfiguration.h"
+#include "spdlog/sinks/ostream_sink.h"
+#include "spdlog/sinks/dist_sink.h"
class LogTestController {
public:
- LogTestController(const std::string level = "debug") {
-logging::Logger::getLogger()->setLogLevel(level);
+ static LogTestController& getInstance() {
+ static LogTestController instance;
+ return instance;
}
-
- void enableDebug() {
-logging::Logger::getLogger()->setLogLevel("debug");
+
+ template
+ void setDebug() {
+setLevel(spdlog::level::debug);
--- End diff --
Will do another sweep of the codebase for spdlog references and try to
limit them to the Logger, LoggerConfiguration, and LogTestController
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi-minifi-cpp pull request #104: MINIFI-296 - Configurable logging, spdlog...
Github user brosander commented on a diff in the pull request:
https://github.com/apache/nifi-minifi-cpp/pull/104#discussion_r118705983
--- Diff: libminifi/test/TestBase.h ---
@@ -21,35 +21,104 @@
#include
#include
#include
+#include
#include "ResourceClaim.h"
#include "catch.hpp"
#include
-#include "core/logging/LogAppenders.h"
#include "core/logging/Logger.h"
#include "core/Core.h"
#include "properties/Configure.h"
+#include "properties/Properties.h"
+#include "core/logging/LoggerConfiguration.h"
+#include "spdlog/sinks/ostream_sink.h"
+#include "spdlog/sinks/dist_sink.h"
class LogTestController {
public:
- LogTestController(const std::string level = "debug") {
-logging::Logger::getLogger()->setLogLevel(level);
+ static LogTestController& getInstance() {
+ static LogTestController instance;
+ return instance;
}
-
- void enableDebug() {
-logging::Logger::getLogger()->setLogLevel("debug");
+
+ template
+ void setDebug() {
+setLevel(spdlog::level::debug);
--- End diff --
@phrocker the LogTestController class is already coupled to spdlog (for
construction of the sinks, etc) and was where I was trying to isolate spdlog in
context of the tests. I'd removed the parallel level enum logic we had in the
code because it's only needed in tests as far as I can tell.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi-minifi-cpp pull request #104: MINIFI-296 - Configurable logging, spdlog...
Github user brosander commented on a diff in the pull request:
https://github.com/apache/nifi-minifi-cpp/pull/104#discussion_r118704850
--- Diff: libminifi/include/core/logging/LoggerConfiguration.h ---
@@ -0,0 +1,147 @@
+/**
+ * @file LoggerConfiguration.h
+ * Logger class declaration
+ * This is a C++ wrapper for spdlog, a lightweight C++ logging library
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+#ifndef __LOGGER_CONFIGURATION_H__
+#define __LOGGER_CONFIGURATION_H__
+
+#include
+#include
+#include
+#include
+#include "core/logging/Logger.h"
+#include "properties/Properties.h"
+
+#include "spdlog/spdlog.h"
+#include "spdlog/formatter.h"
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+namespace core {
+namespace logging {
+
+namespace internal {
+ /**
+ * This class should not be instantiated directly. Use LoggerFactory to
get an instance.
--- End diff --
@phrocker good call, will fix
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi-minifi-cpp pull request #104: MINIFI-296 - Configurable logging, spdlog...
Github user brosander commented on a diff in the pull request:
https://github.com/apache/nifi-minifi-cpp/pull/104#discussion_r118704917
--- Diff: libminifi/test/TestBase.h ---
@@ -21,35 +21,104 @@
#include
#include
#include
+#include
#include "ResourceClaim.h"
#include "catch.hpp"
#include
-#include "core/logging/LogAppenders.h"
#include "core/logging/Logger.h"
#include "core/Core.h"
#include "properties/Configure.h"
+#include "properties/Properties.h"
+#include "core/logging/LoggerConfiguration.h"
+#include "spdlog/sinks/ostream_sink.h"
+#include "spdlog/sinks/dist_sink.h"
class LogTestController {
public:
- LogTestController(const std::string level = "debug") {
-logging::Logger::getLogger()->setLogLevel(level);
+ static LogTestController& getInstance() {
+ static LogTestController instance;
+ return instance;
}
-
- void enableDebug() {
-logging::Logger::getLogger()->setLogLevel("debug");
+
+ template
+ void setDebug() {
+setLevel(spdlog::level::debug);
+ }
+
+ template
+ void setInfo() {
+setLevel(spdlog::level::info);
+ }
+
+ template
+ void setLevel(spdlog::level::level_enum level) {
+logging::LoggerFactory::getLogger();
+std::string name = core::getClassName();
+modified_loggers.push_back(name);
+setLevel(name, level);
+ }
+
+ bool contains(const std::string ) {
+ return contains(log_output, ending);
+ }
+
+ bool contains(const std::ostringstream , const std::string
) {
+std::string str = stream.str();
+logger_->log_info("Looking for %s in %s.", ending, str);
+return (ending.length() > 0 && str.find(ending) != std::string::npos);
+ }
+
+ void reset() {
+for (auto const & name : modified_loggers) {
+ setLevel(name, spdlog::level::err);
+}
+modified_loggers = std::vector();
+resetStream(log_output);
+ }
+
+ inline bool resetStream(std::ostringstream ) {
+stream.str("");
+stream.clear();
--- End diff --
@phrocker good call, will fix
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi-minifi-cpp pull request #104: MINIFI-296 - Configurable logging, spdlog...
Github user brosander commented on a diff in the pull request:
https://github.com/apache/nifi-minifi-cpp/pull/104#discussion_r118704895
--- Diff: libminifi/include/utils/StringUtils.h ---
@@ -89,6 +89,25 @@ class StringUtils {
s.end());
return s;
}
+
+ static std::vector split(const std::string , const
std::string ) {
+std::vector result;
+int last = 0;
+int next = 0;
+while ((next = str.find(delimiter, last)) != std::string::npos) {
+ result.push_back(str.substr(last, next - last));
+ last = next + delimiter.length();
+}
+result.push_back(str.substr(last, next - last));
+return result;
+ }
+
+ static inline bool starts_with(const std::string , const std::string
) {
--- End diff --
@phrocker good call, will fix
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi-minifi-cpp pull request #104: MINIFI-296 - Configurable logging, spdlog...
Github user brosander commented on a diff in the pull request:
https://github.com/apache/nifi-minifi-cpp/pull/104#discussion_r118704804
--- Diff: libminifi/include/core/controller/ControllerService.h ---
@@ -64,7 +64,7 @@ class ControllerService : public ConfigurableComponent,
public Connectable {
*/
explicit ControllerService()
: Connectable(core::getClassName(), 0),
-ConfigurableComponent(logging::Logger::getLogger()),
+ConfigurableComponent(),
--- End diff --
@phrocker good call, will fix
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi-minifi-cpp pull request #104: MINIFI-296 - Configurable logging, spdlog...
Github user brosander commented on a diff in the pull request:
https://github.com/apache/nifi-minifi-cpp/pull/104#discussion_r118704784
--- Diff: libminifi/include/Site2SitePeer.h ---
@@ -72,8 +73,8 @@ class Site2SitePeer : public
org::apache::nifi::minifi::io::BaseStream {
explicit Site2SitePeer(Site2SitePeer &)
: stream_(ss.stream_.release()),
host_(std::move(ss.host_)),
-port_(std::move(ss.port_)) {
-logger_ = logging::Logger::getLogger();
+port_(std::move(ss.port_)),
+logger_(ss.logger_) {
--- End diff --
@phrocker good call, will fix
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi-minifi-cpp issue #104: MINIFI-296 - Configurable logging, spdlog 0.13.0
Github user brosander commented on the issue: https://github.com/apache/nifi-minifi-cpp/pull/104 @phrocker I've added atomic load and store to the logger and loggerimpl classes and a pattern option to minifi-log.properties along with some alternative patterns and a link to the spdlog wiki --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi-minifi-cpp issue #104: MINIFI-296 - Configurable logging, spdlog 0.13.0
Github user brosander commented on the issue: https://github.com/apache/nifi-minifi-cpp/pull/104 @phrocker For your first point about reinitializing logging while a flow is running, is it worth it to switch the delegate reference on the Logger to an atomic? Second point I think should be fairly easily addressable with a format pattern in the logger config. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi-minifi-cpp pull request #104: MINIFI-296 - Configurable logging, spdlog...
GitHub user brosander opened a pull request: https://github.com/apache/nifi-minifi-cpp/pull/104 MINIFI-296 - Configurable logging, spdlog 0.13.0 Thank you for submitting a contribution to Apache NiFi - MiNiFi C++. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [X] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [X] Does your PR title start with MINIFI- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [X] Has your PR been rebased against the latest commit within the target branch (typically master)? - [X] Is your initial contribution a single, squashed commit? ### For code changes: - [X] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [X] - N/A - If applicable, have you updated the LICENSE file? - [X] - N/A - If applicable, have you updated the NOTICE file? ### For documentation related changes: - [X] - N/A - Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. You can merge this pull request into a Git repository by running: $ git pull https://github.com/brosander/nifi-minifi-cpp MINIFI-296 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi-minifi-cpp/pull/104.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #104 commit fa0304cd48920a1d3b259b9a51a6aff0c79a88e7 Author: Bryan Rosander <brosan...@apache.org> Date: 2017-05-11T19:42:54Z MINIFI-296 - Configurable logging, spdlog 0.13.0 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi-minifi-cpp issue #91: MINIFI-258 - Removing Configure, StreamFactory, T...
Github user brosander commented on the issue: https://github.com/apache/nifi-minifi-cpp/pull/91 Thanks for feedback, changed those back and am rerunning build/tests --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi issue #1753: NIFI-3788 Switched Amazon HTTP client instantiation from u...
Github user brosander commented on the issue: https://github.com/apache/nifi/pull/1753 +1 Merging --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi issue #1753: NIFI-3788 Switched Amazon HTTP client instantiation from u...
Github user brosander commented on the issue: https://github.com/apache/nifi/pull/1753 Reviewing --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi-minifi-cpp pull request #91: MINIFI-258 - Removing Configure, StreamFac...
GitHub user brosander opened a pull request: https://github.com/apache/nifi-minifi-cpp/pull/91 MINIFI-258 - Removing Configure, StreamFactory, TLSContext singletons Thank you for submitting a contribution to Apache NiFi - MiNiFi C++. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [X] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [X] Does your PR title start with MINIFI- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [X] Has your PR been rebased against the latest commit within the target branch (typically master)? - [X] Is your initial contribution a single, squashed commit? ### For code changes: - [X] - N/A - If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [X] - N/A - If applicable, have you updated the LICENSE file? - [X] - N/A - If applicable, have you updated the NOTICE file? ### For documentation related changes: - [X] - N/A - Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. You can merge this pull request into a Git repository by running: $ git pull https://github.com/brosander/nifi-minifi-cpp MINIFI-258 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi-minifi-cpp/pull/91.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #91 commit de55683522115a409029bee89fad64c0a407bcf3 Author: Bryan Rosander <brosan...@apache.org> Date: 2017-05-01T18:49:44Z MINIFI-258 - Removing Configure, StreamFactory, TLSContext singletons --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi-minifi-cpp issue #87: MINIFI-284 Making use of CMAKE_INSTALL_LIBDIR arg...
Github user brosander commented on the issue: https://github.com/apache/nifi-minifi-cpp/pull/87 +1 Merging --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi issue #1719: NIFI-1833 Updates to Azure Storage Processor PR to fix dep...
Github user brosander commented on the issue: https://github.com/apache/nifi/pull/1719 +1 Merged --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi-minifi-cpp issue #84: MINIFI-274 Use external project with existing sou...
Github user brosander commented on the issue: https://github.com/apache/nifi-minifi-cpp/pull/84 +1 works for me on bare metal Ubuntu 16.04, OSx 10.12.4, in docker container on both platforms as well --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi issue #1726: NIFI-3765 - Added a 'status' operation to Node Manager
Github user brosander commented on the issue: https://github.com/apache/nifi/pull/1726 Tested non-secure standalone instance, secure cluster. Works as expected. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi pull request #1719: NIFI-1833 Updates to Azure Storage Processor PR to ...
Github user brosander commented on a diff in the pull request:
https://github.com/apache/nifi/pull/1719#discussion_r114228274
--- Diff:
nifi-nar-bundles/nifi-azure-bundle/nifi-azure-processors/src/main/java/org/apache/nifi/processors/azure/storage/PutAzureBlobStorage.java
---
@@ -106,10 +110,15 @@ public void onTrigger(final ProcessContext context,
final ProcessSession session
final long transferMillis =
TimeUnit.NANOSECONDS.toMillis(System.nanoTime() - startNanos);
session.getProvenanceReporter().send(flowFile,
blob.getSnapshotQualifiedUri().toString(), transferMillis);
-} catch (IllegalArgumentException | URISyntaxException |
StorageException e) {
-getLogger().error("Failed to put Azure blob {}", new
Object[]{blobPath}, e);
-flowFile = session.penalize(flowFile);
-session.transfer(flowFile, REL_FAILURE);
+} catch (IllegalArgumentException | URISyntaxException |
StorageException | ProcessException e) {
+if (e instanceof ProcessException && storedException.get() ==
null) {
+throw (ProcessException) e;
+} else {
+Exception failureException =
Optional.ofNullable(storedException.get()).map(x -> x).orElse(e);
--- End diff --
@jtstorck I don't think you need the .map() here. It looks like orElse()
will already give you the value if the optional is absent.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi pull request #1719: NIFI-1833 Updates to Azure Storage Processor PR to ...
Github user brosander commented on a diff in the pull request:
https://github.com/apache/nifi/pull/1719#discussion_r114228190
--- Diff:
nifi-nar-bundles/nifi-azure-bundle/nifi-azure-processors/src/main/java/org/apache/nifi/processors/azure/storage/FetchAzureBlobStorage.java
---
@@ -103,10 +107,15 @@ public void onTrigger(ProcessContext context,
ProcessSession session) throws Pro
session.transfer(flowFile, REL_SUCCESS);
final long transferMillis =
TimeUnit.NANOSECONDS.toMillis(System.nanoTime() - startNanos);
session.getProvenanceReporter().fetch(flowFile,
blob.getSnapshotQualifiedUri().toString(), transferMillis);
-
-} catch (IllegalArgumentException | URISyntaxException |
StorageException e1) {
-flowFile = session.penalize(flowFile);
-session.transfer(flowFile, REL_FAILURE);
+} catch (IllegalArgumentException | URISyntaxException |
StorageException | ProcessException e) {
+if (e instanceof ProcessException && storedException.get() ==
null) {
+throw (ProcessException) e;
+} else {
+Exception failureException =
Optional.ofNullable(storedException.get()).map(x -> x).orElse(e);
--- End diff --
@jtstorck I don't think you need the .map() here. It looks like
[orElse()](https://docs.oracle.com/javase/8/docs/api/java/util/Optional.html#orElse-T-)
will already give you the value if the optional is absent.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi-minifi-cpp issue #84: MINIFI-274 Use external project with existing sou...
Github user brosander commented on the issue: https://github.com/apache/nifi-minifi-cpp/pull/84 The following docker image repros it for me: https://github.com/brosander/minifi-cpp-tooling/tree/9e5b1031d1772ccbee3ec8844a3decf232185284/Dockerfiles/ubuntu/xenial/build ``` docker build -t minifi-cpp-xenial-build . mkdir build docker run -ti --rm -v "$(pwd)/build:/build" minifi-cpp-xenial-build 84 ``` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi-minifi-cpp issue #84: MINIFI-274 Use external project with existing sou...
Github user brosander commented on the issue: https://github.com/apache/nifi-minifi-cpp/pull/84 Ubuntu 16.04 64 bit (latest LTS release of Ubuntu) I'll craft a dockerfile for easier testing/validation. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi-minifi-cpp issue #84: MINIFI-274 Use external project with existing sou...
Github user brosander commented on the issue: https://github.com/apache/nifi-minifi-cpp/pull/84 ``` TestSiteToSiteVerifySend --- /home/bryan/Github/nifi-minifi-cpp/libminifi/test/unit/Site2SiteTests.cpp:102 ... /home/bryan/Github/nifi-minifi-cpp/libminifi/test/unit/Site2SiteTests.cpp:102: FAILED: due to a fatal error condition: SIGSEGV - Segmentation violation signal ``` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi-minifi-cpp issue #84: MINIFI-274 Use external project with existing sou...
Github user brosander commented on the issue: https://github.com/apache/nifi-minifi-cpp/pull/84 definitely better but I still seem to be getting a segfault on the s2s test --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi-minifi-cpp issue #84: MINIFI-274 Use external project with existing sou...
Github user brosander commented on the issue: https://github.com/apache/nifi-minifi-cpp/pull/84 @apiri I'll take a look, thanks! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi pull request #1719: NIFI-1833 Updates to Azure Storage Processor PR to ...
Github user brosander commented on a diff in the pull request:
https://github.com/apache/nifi/pull/1719#discussion_r114169894
--- Diff:
nifi-nar-bundles/nifi-azure-bundle/nifi-azure-processors/src/main/java/org/apache/nifi/processors/azure/storage/PutAzureBlobStorage.java
---
@@ -0,0 +1,116 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.azure.storage;
+
+import java.io.BufferedInputStream;
+import java.io.IOException;
+import java.io.InputStream;
+import java.net.URISyntaxException;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.concurrent.TimeUnit;
+
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.SeeAlso;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processors.azure.AbstractAzureBlobProcessor;
+import org.apache.nifi.processors.azure.AzureConstants;
+
+import com.microsoft.azure.storage.CloudStorageAccount;
+import com.microsoft.azure.storage.StorageException;
+import com.microsoft.azure.storage.blob.BlobProperties;
+import com.microsoft.azure.storage.blob.CloudBlob;
+import com.microsoft.azure.storage.blob.CloudBlobClient;
+import com.microsoft.azure.storage.blob.CloudBlobContainer;
+
+@Tags({ "azure", "microsoft", "cloud", "storage", "blob" })
+@SeeAlso({ ListAzureBlobStorage.class, FetchAzureBlobStorage.class })
+@CapabilityDescription("Puts content into an Azure Storage Blob")
+@InputRequirement(Requirement.INPUT_REQUIRED)
+@WritesAttributes({ @WritesAttribute(attribute = "azure.container",
description = "The name of the Azure container"),
+@WritesAttribute(attribute = "azure.blobname", description = "The
name of the Azure blob"),
+@WritesAttribute(attribute = "azure.primaryUri", description =
"Primary location for blob content"),
+@WritesAttribute(attribute = "azure.etag", description = "Etag for
the Azure blob"),
+@WritesAttribute(attribute = "azure.length", description = "Length
of the blob"),
+@WritesAttribute(attribute = "azure.timestamp", description = "The
timestamp in Azure for the blob")})
+public class PutAzureBlobStorage extends AbstractAzureBlobProcessor {
+
+public void onTrigger(final ProcessContext context, final
ProcessSession session) throws ProcessException {
+FlowFile flowFile = session.get();
+if (flowFile == null) {
+return;
+}
+
+final long startNanos = System.nanoTime();
+
+String containerName =
context.getProperty(AzureConstants.CONTAINER).evaluateAttributeExpressions(flowFile).getValue();
+
+String blobPath =
context.getProperty(BLOB).evaluateAttributeExpressions(flowFile).getValue();
+
+try {
+CloudStorageAccount storageAccount =
createStorageConnection(context, flowFile);
+CloudBlobClient blobClient =
storageAccount.createCloudBlobClient();
+CloudBlobContainer container =
blobClient.getContainerReference(containerName);
+
+CloudBlob blob = container.getBlockBlobReference(blobPath);
+
+final Map<String, String> attributes = new HashMap<>();
+long length = flowFile.g
[GitHub] nifi pull request #1719: NIFI-1833 Updates to Azure Storage Processor PR to ...
Github user brosander commented on a diff in the pull request:
https://github.com/apache/nifi/pull/1719#discussion_r114169124
--- Diff:
nifi-nar-bundles/nifi-azure-bundle/nifi-azure-processors/src/main/java/org/apache/nifi/processors/azure/storage/ListAzureBlobStorage.java
---
@@ -0,0 +1,180 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.azure.storage;
+
+import java.io.IOException;
+import java.net.URISyntaxException;
+import java.security.InvalidKeyException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.EnumSet;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.Stateful;
+import org.apache.nifi.annotation.behavior.TriggerSerially;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.SeeAlso;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.state.Scope;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.processors.azure.AzureConstants;
+import org.apache.nifi.processors.azure.storage.utils.BlobInfo;
+import org.apache.nifi.processors.azure.storage.utils.BlobInfo.Builder;
+import org.apache.nifi.processor.util.list.AbstractListProcessor;
+
+import com.microsoft.azure.storage.CloudStorageAccount;
+import com.microsoft.azure.storage.StorageException;
+import com.microsoft.azure.storage.StorageUri;
+import com.microsoft.azure.storage.blob.BlobListingDetails;
+import com.microsoft.azure.storage.blob.BlobProperties;
+import com.microsoft.azure.storage.blob.CloudBlob;
+import com.microsoft.azure.storage.blob.CloudBlobClient;
+import com.microsoft.azure.storage.blob.CloudBlobContainer;
+import com.microsoft.azure.storage.blob.CloudBlockBlob;
+import com.microsoft.azure.storage.blob.ListBlobItem;
+
+@TriggerSerially
+@Tags({ "azure", "microsoft", "cloud", "storage", "blob" })
+@SeeAlso({ FetchAzureBlobStorage.class, PutAzureBlobStorage.class })
+@CapabilityDescription("Lists blobs in an Azure Storage container. Listing
details are attached to an empty FlowFile for use with FetchAzureBlobStorage")
+@InputRequirement(Requirement.INPUT_FORBIDDEN)
+@WritesAttributes({ @WritesAttribute(attribute = "azure.container",
description = "The name of the Azure container"),
+@WritesAttribute(attribute = "azure.blobname", description = "The
name of the Azure blob"),
+@WritesAttribute(attribute = "azure.primaryUri", description =
"Primary location for blob content"),
+@WritesAttribute(attribute = "azure.secondaryUri", description =
"Secondary location for blob content"),
+@WritesAttribute(attribute = "azure.etag", description = "Etag for
the Azure blob"),
+@WritesAttribute(attribute = "azure.length", description = "Length
of the blob"),
+@WritesAttribute(attribute = "azure.timestamp", description = "The
timestamp in Azure for the blob"),
+@WritesAttribute(attribute = "mime.type", description = "MimeType
of the content"),
+@WritesAttribute(attribute = "lang", description = "Language code
for the content"),
+@WritesAttribute(attribute = "azure.blobtype", descripti
[GitHub] nifi pull request #1719: NIFI-1833 Updates to Azure Storage Processor PR to ...
Github user brosander commented on a diff in the pull request:
https://github.com/apache/nifi/pull/1719#discussion_r114168730
--- Diff:
nifi-nar-bundles/nifi-azure-bundle/nifi-azure-processors/src/main/java/org/apache/nifi/processors/azure/storage/FetchAzureBlobStorage.java
---
@@ -0,0 +1,112 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.azure.storage;
+
+import java.io.IOException;
+import java.net.URISyntaxException;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.TimeUnit;
+
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.SeeAlso;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processors.azure.AbstractAzureBlobProcessor;
+import org.apache.nifi.processors.azure.AzureConstants;
+
+import com.microsoft.azure.storage.CloudStorageAccount;
+import com.microsoft.azure.storage.StorageException;
+import com.microsoft.azure.storage.blob.CloudBlob;
+import com.microsoft.azure.storage.blob.CloudBlobClient;
+import com.microsoft.azure.storage.blob.CloudBlobContainer;
+
+@Tags({ "azure", "microsoft", "cloud", "storage", "blob" })
+@CapabilityDescription("Retrieves contents of an Azure Storage Blob,
writing the contents to the content of the FlowFile")
+@SeeAlso({ ListAzureBlobStorage.class, PutAzureBlobStorage.class })
+@InputRequirement(Requirement.INPUT_REQUIRED)
+@WritesAttributes({
+@WritesAttribute(attribute = "azure.length", description = "The length
of the blob fetched")
+})
+public class FetchAzureBlobStorage extends AbstractAzureBlobProcessor {
+
+private static final List PROPERTIES = Collections
+.unmodifiableList(Arrays.asList(AzureConstants.ACCOUNT_NAME,
AzureConstants.ACCOUNT_KEY, AzureConstants.CONTAINER, BLOB));
+
+@Override
+protected List getSupportedPropertyDescriptors() {
+return PROPERTIES;
+}
+
+@Override
+public void onTrigger(ProcessContext context, ProcessSession session)
throws ProcessException {
+FlowFile flowFile = session.get();
+if (flowFile == null) {
+return;
+}
+
+final long startNanos = System.nanoTime();
+
+String containerName =
context.getProperty(AzureConstants.CONTAINER).evaluateAttributeExpressions(flowFile).getValue();
+String blobPath =
context.getProperty(BLOB).evaluateAttributeExpressions(flowFile).getValue();
+
+try {
+CloudStorageAccount storageAccount =
createStorageConnection(context, flowFile);
+CloudBlobClient blobClient =
storageAccount.createCloudBlobClient();
+CloudBlobContainer container =
blobClient.getContainerReference(containerName);
+
+final Map<String, String> attributes = new HashMap<>();
+final CloudBlob blob =
container.getBlockBlobReference(blobPath);
+
+// TODO - we may be able do fancier things with ranges and
+// distribution of download over threads, investigate
+flowFile = session.write(flowFile, os -> {
[GitHub] nifi pull request #1719: NIFI-1833 Updates to Azure Storage Processor PR to ...
Github user brosander commented on a diff in the pull request:
https://github.com/apache/nifi/pull/1719#discussion_r114136630
--- Diff:
nifi-nar-bundles/nifi-azure-bundle/nifi-azure-processors/src/main/java/org/apache/nifi/processors/azure/storage/ListAzureBlobStorage.java
---
@@ -0,0 +1,180 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.azure.storage;
+
+import java.io.IOException;
+import java.net.URISyntaxException;
+import java.security.InvalidKeyException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.EnumSet;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+import org.apache.nifi.annotation.behavior.InputRequirement;
+import org.apache.nifi.annotation.behavior.InputRequirement.Requirement;
+import org.apache.nifi.annotation.behavior.Stateful;
+import org.apache.nifi.annotation.behavior.TriggerSerially;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.SeeAlso;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.state.Scope;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.processors.azure.AzureConstants;
+import org.apache.nifi.processors.azure.storage.utils.BlobInfo;
+import org.apache.nifi.processors.azure.storage.utils.BlobInfo.Builder;
+import org.apache.nifi.processor.util.list.AbstractListProcessor;
+
+import com.microsoft.azure.storage.CloudStorageAccount;
+import com.microsoft.azure.storage.StorageException;
+import com.microsoft.azure.storage.StorageUri;
+import com.microsoft.azure.storage.blob.BlobListingDetails;
+import com.microsoft.azure.storage.blob.BlobProperties;
+import com.microsoft.azure.storage.blob.CloudBlob;
+import com.microsoft.azure.storage.blob.CloudBlobClient;
+import com.microsoft.azure.storage.blob.CloudBlobContainer;
+import com.microsoft.azure.storage.blob.CloudBlockBlob;
+import com.microsoft.azure.storage.blob.ListBlobItem;
+
+@TriggerSerially
+@Tags({ "azure", "microsoft", "cloud", "storage", "blob" })
+@SeeAlso({ FetchAzureBlobStorage.class, PutAzureBlobStorage.class })
+@CapabilityDescription("Lists blobs in an Azure Storage container. Listing
details are attached to an empty FlowFile for use with FetchAzureBlobStorage")
+@InputRequirement(Requirement.INPUT_FORBIDDEN)
+@WritesAttributes({ @WritesAttribute(attribute = "azure.container",
description = "The name of the Azure container"),
+@WritesAttribute(attribute = "azure.blobname", description = "The
name of the Azure blob"),
+@WritesAttribute(attribute = "azure.primaryUri", description =
"Primary location for blob content"),
+@WritesAttribute(attribute = "azure.secondaryUri", description =
"Secondary location for blob content"),
+@WritesAttribute(attribute = "azure.etag", description = "Etag for
the Azure blob"),
+@WritesAttribute(attribute = "azure.length", description = "Length
of the blob"),
+@WritesAttribute(attribute = "azure.timestamp", description = "The
timestamp in Azure for the blob"),
+@WritesAttribute(attribute = "mime.type", description = "MimeType
of the content"),
+@WritesAttribute(attribute = "lang", description = "Language code
for the content"),
+@WritesAttribute(attribute = "azure.blobtype", descripti
[GitHub] nifi pull request #1719: NIFI-1833 Updates to Azure Storage Processor PR to ...
Github user brosander commented on a diff in the pull request:
https://github.com/apache/nifi/pull/1719#discussion_r114135508
--- Diff:
nifi-nar-bundles/nifi-azure-bundle/nifi-azure-processors/src/main/java/org/apache/nifi/processors/azure/AbstractAzureBlobProcessor.java
---
@@ -0,0 +1,39 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.azure;
+
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.processor.util.StandardValidators;
+
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+
+public abstract class AbstractAzureBlobProcessor extends
AbstractAzureProcessor {
+
+public static final PropertyDescriptor BLOB = new
PropertyDescriptor.Builder().name("Blob").description("The filename of the
blob").addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
--- End diff --
displayName missing
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi pull request #1719: NIFI-1833 Updates to Azure Storage Processor PR to ...
Github user brosander commented on a diff in the pull request: https://github.com/apache/nifi/pull/1719#discussion_r114135293 --- Diff: nifi-nar-bundles/nifi-azure-bundle/nifi-azure-processors/pom.xml --- @@ -32,11 +32,35 @@ language governing permissions and limitations under the License. --> nifi-utils +org.apache.nifi +nifi-ssl-context-service-api +provided + + +org.apache.avro +avro + + com.microsoft.azure azure-eventhubs 0.9.0 +com.fasterxml.jackson.core +jackson-core +2.8.6 + +
[GitHub] nifi-minifi-cpp issue #84: MINIFI-274 Use external project with existing sou...
Github user brosander commented on the issue: https://github.com/apache/nifi-minifi-cpp/pull/84 +1 merging --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi-minifi-cpp issue #84: MINIFI-274 Use external project with existing sou...
Github user brosander commented on the issue: https://github.com/apache/nifi-minifi-cpp/pull/84 Reviewing --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi-minifi-cpp issue #82: MINIFI-273 Update README to include doxygen depen...
Github user brosander commented on the issue: https://github.com/apache/nifi-minifi-cpp/pull/82 +1 merging --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi-minifi pull request #83: MINIFI-281 - Using RemoteProcessGroupSchemaV2
GitHub user brosander opened a pull request: https://github.com/apache/nifi-minifi/pull/83 MINIFI-281 - Using RemoteProcessGroupSchemaV2 Thank you for submitting a contribution to Apache NiFi - MiNiFi. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [X] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [X] Does your PR title start with MINIFI- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [X] Has your PR been rebased against the latest commit within the target branch (typically master)? - [X] Is your initial contribution a single, squashed commit? ### For code changes: - [X] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi-minifi folder? - [X] Have you written or updated unit tests to verify your changes? - [X] - N/A - If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [X] - N/A - If applicable, have you updated the LICENSE file, including the main LICENSE file under minifi-assembly? - [X] - N/A - If applicable, have you updated the NOTICE file, including the main NOTICE file found under minifi-assembly? ### For documentation related changes: - [X] - N/A - Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. You can merge this pull request into a Git repository by running: $ git pull https://github.com/brosander/nifi-minifi MINIFI-281 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi-minifi/pull/83.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #83 commit 3d6a9fea23c1de8dab0730a3b5de2ff34b7695c2 Author: Bryan Rosander <brosan...@apache.org> Date: 2017-04-28T14:48:31Z MINIFI-281 - Using RemoteProcessGroupSchemaV2 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi-minifi pull request #80: MINIFI-277: Write variable registry property t...
Github user brosander commented on a diff in the pull request:
https://github.com/apache/nifi-minifi/pull/80#discussion_r113939300
--- Diff:
minifi-commons/minifi-commons-schema/src/main/java/org/apache/nifi/minifi/commons/schema/v2/CorePropertiesSchemaV2.java
---
@@ -0,0 +1,95 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.minifi.commons.schema.v2;
+
+import static
org.apache.nifi.minifi.commons.schema.common.CommonPropertyKeys.CORE_PROPS_KEY;
+import static
org.apache.nifi.minifi.commons.schema.common.CommonPropertyKeys.MAX_CONCURRENT_THREADS_KEY;
+
+import java.util.Map;
+
+import org.apache.nifi.minifi.commons.schema.common.BaseSchema;
+import org.apache.nifi.minifi.commons.schema.common.WritableSchema;
+
+/**
+ *
+ */
+public class CorePropertiesSchemaV2 extends BaseSchema implements
WritableSchema {
+
+public static final String FLOW_CONTROLLER_SHUTDOWN_PERIOD_KEY = "flow
controller graceful shutdown period";
+public static final String FLOW_SERVICE_WRITE_DELAY_INTERVAL_KEY =
"flow service write delay interval";
+public static final String ADMINISTRATIVE_YIELD_DURATION_KEY =
"administrative yield duration";
+public static final String BORED_YIELD_DURATION_KEY = "bored yield
duration";
+
+public static final String
DEFAULT_FLOW_CONTROLLER_GRACEFUL_SHUTDOWN_PERIOD = "10 sec";
+public static final String DEFAULT_FLOW_SERVICE_WRITE_DELAY_INTERVAL =
"500 ms";
+public static final String DEFAULT_ADMINISTRATIVE_YIELD_DURATION = "30
sec";
+public static final String DEFAULT_BORED_YIELD_DURATION = "10 millis";
+public static final int DEFAULT_MAX_CONCURRENT_THREADS = 1;
+
+private String flowControllerGracefulShutdownPeriod =
DEFAULT_FLOW_CONTROLLER_GRACEFUL_SHUTDOWN_PERIOD;
+private String flowServiceWriteDelayInterval =
DEFAULT_FLOW_SERVICE_WRITE_DELAY_INTERVAL;
+private String administrativeYieldDuration =
DEFAULT_ADMINISTRATIVE_YIELD_DURATION;
+private String boredYieldDuration = DEFAULT_BORED_YIELD_DURATION;
+private Number maxConcurrentThreads = DEFAULT_MAX_CONCURRENT_THREADS;
+
+public CorePropertiesSchemaV2() {
+}
+
+public CorePropertiesSchemaV2(Map map) {
+flowControllerGracefulShutdownPeriod = getOptionalKeyAsType(map,
FLOW_CONTROLLER_SHUTDOWN_PERIOD_KEY, String.class,
+CORE_PROPS_KEY,
DEFAULT_FLOW_CONTROLLER_GRACEFUL_SHUTDOWN_PERIOD);
+flowServiceWriteDelayInterval = getOptionalKeyAsType(map,
FLOW_SERVICE_WRITE_DELAY_INTERVAL_KEY, String.class,
+CORE_PROPS_KEY, DEFAULT_FLOW_SERVICE_WRITE_DELAY_INTERVAL);
+administrativeYieldDuration = getOptionalKeyAsType(map,
ADMINISTRATIVE_YIELD_DURATION_KEY, String.class,
+CORE_PROPS_KEY, DEFAULT_ADMINISTRATIVE_YIELD_DURATION);
+boredYieldDuration = getOptionalKeyAsType(map,
BORED_YIELD_DURATION_KEY, String.class, CORE_PROPS_KEY,
DEFAULT_BORED_YIELD_DURATION);
+maxConcurrentThreads = getOptionalKeyAsType(map,
MAX_CONCURRENT_THREADS_KEY, Number.class,
+CORE_PROPS_KEY, DEFAULT_MAX_CONCURRENT_THREADS);
+}
+
+@Override
+public Map<String, Object> toMap() {
--- End diff --
Then in ConfigSchemaV2, you'd want something like
```
putIfNotNull(result, CORE_PROPS_KEY, coreProperties.convert());
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi-minifi pull request #80: MINIFI-277: Write variable registry property t...
Github user brosander commented on a diff in the pull request:
https://github.com/apache/nifi-minifi/pull/80#discussion_r113938266
--- Diff:
minifi-commons/minifi-commons-schema/src/main/java/org/apache/nifi/minifi/commons/schema/v2/CorePropertiesSchemaV2.java
---
@@ -0,0 +1,95 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.minifi.commons.schema.v2;
+
+import static
org.apache.nifi.minifi.commons.schema.common.CommonPropertyKeys.CORE_PROPS_KEY;
+import static
org.apache.nifi.minifi.commons.schema.common.CommonPropertyKeys.MAX_CONCURRENT_THREADS_KEY;
+
+import java.util.Map;
+
+import org.apache.nifi.minifi.commons.schema.common.BaseSchema;
+import org.apache.nifi.minifi.commons.schema.common.WritableSchema;
+
+/**
+ *
+ */
+public class CorePropertiesSchemaV2 extends BaseSchema implements
WritableSchema {
+
+public static final String FLOW_CONTROLLER_SHUTDOWN_PERIOD_KEY = "flow
controller graceful shutdown period";
+public static final String FLOW_SERVICE_WRITE_DELAY_INTERVAL_KEY =
"flow service write delay interval";
+public static final String ADMINISTRATIVE_YIELD_DURATION_KEY =
"administrative yield duration";
+public static final String BORED_YIELD_DURATION_KEY = "bored yield
duration";
+
+public static final String
DEFAULT_FLOW_CONTROLLER_GRACEFUL_SHUTDOWN_PERIOD = "10 sec";
+public static final String DEFAULT_FLOW_SERVICE_WRITE_DELAY_INTERVAL =
"500 ms";
+public static final String DEFAULT_ADMINISTRATIVE_YIELD_DURATION = "30
sec";
+public static final String DEFAULT_BORED_YIELD_DURATION = "10 millis";
+public static final int DEFAULT_MAX_CONCURRENT_THREADS = 1;
+
+private String flowControllerGracefulShutdownPeriod =
DEFAULT_FLOW_CONTROLLER_GRACEFUL_SHUTDOWN_PERIOD;
+private String flowServiceWriteDelayInterval =
DEFAULT_FLOW_SERVICE_WRITE_DELAY_INTERVAL;
+private String administrativeYieldDuration =
DEFAULT_ADMINISTRATIVE_YIELD_DURATION;
+private String boredYieldDuration = DEFAULT_BORED_YIELD_DURATION;
+private Number maxConcurrentThreads = DEFAULT_MAX_CONCURRENT_THREADS;
+
+public CorePropertiesSchemaV2() {
+}
+
+public CorePropertiesSchemaV2(Map map) {
+flowControllerGracefulShutdownPeriod = getOptionalKeyAsType(map,
FLOW_CONTROLLER_SHUTDOWN_PERIOD_KEY, String.class,
+CORE_PROPS_KEY,
DEFAULT_FLOW_CONTROLLER_GRACEFUL_SHUTDOWN_PERIOD);
+flowServiceWriteDelayInterval = getOptionalKeyAsType(map,
FLOW_SERVICE_WRITE_DELAY_INTERVAL_KEY, String.class,
+CORE_PROPS_KEY, DEFAULT_FLOW_SERVICE_WRITE_DELAY_INTERVAL);
+administrativeYieldDuration = getOptionalKeyAsType(map,
ADMINISTRATIVE_YIELD_DURATION_KEY, String.class,
+CORE_PROPS_KEY, DEFAULT_ADMINISTRATIVE_YIELD_DURATION);
+boredYieldDuration = getOptionalKeyAsType(map,
BORED_YIELD_DURATION_KEY, String.class, CORE_PROPS_KEY,
DEFAULT_BORED_YIELD_DURATION);
+maxConcurrentThreads = getOptionalKeyAsType(map,
MAX_CONCURRENT_THREADS_KEY, Number.class,
+CORE_PROPS_KEY, DEFAULT_MAX_CONCURRENT_THREADS);
+}
+
+@Override
+public Map<String, Object> toMap() {
--- End diff --
When changing WritableSchema -> ConvertableSchema, this method should be
able to be turned into the convert() method (much like
[RemoteProcessGroupSchemaV2](https://github.com/apache/nifi-minifi/blob/master/minifi-commons/minifi-commons-schema/src/main/java/org/apache/nifi/minifi/commons/schema/v2/RemoteProcessGroupSchemaV2.java#L75))
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi-minifi pull request #80: MINIFI-277: Write variable registry property t...
Github user brosander commented on a diff in the pull request:
https://github.com/apache/nifi-minifi/pull/80#discussion_r113937684
--- Diff:
minifi-commons/minifi-commons-schema/src/main/java/org/apache/nifi/minifi/commons/schema/v2/CorePropertiesSchemaV2.java
---
@@ -0,0 +1,95 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.minifi.commons.schema.v2;
+
+import static
org.apache.nifi.minifi.commons.schema.common.CommonPropertyKeys.CORE_PROPS_KEY;
+import static
org.apache.nifi.minifi.commons.schema.common.CommonPropertyKeys.MAX_CONCURRENT_THREADS_KEY;
+
+import java.util.Map;
+
+import org.apache.nifi.minifi.commons.schema.common.BaseSchema;
+import org.apache.nifi.minifi.commons.schema.common.WritableSchema;
+
+/**
+ *
+ */
+public class CorePropertiesSchemaV2 extends BaseSchema implements
WritableSchema {
+
+public static final String FLOW_CONTROLLER_SHUTDOWN_PERIOD_KEY = "flow
controller graceful shutdown period";
+public static final String FLOW_SERVICE_WRITE_DELAY_INTERVAL_KEY =
"flow service write delay interval";
+public static final String ADMINISTRATIVE_YIELD_DURATION_KEY =
"administrative yield duration";
+public static final String BORED_YIELD_DURATION_KEY = "bored yield
duration";
+
+public static final String
DEFAULT_FLOW_CONTROLLER_GRACEFUL_SHUTDOWN_PERIOD = "10 sec";
+public static final String DEFAULT_FLOW_SERVICE_WRITE_DELAY_INTERVAL =
"500 ms";
+public static final String DEFAULT_ADMINISTRATIVE_YIELD_DURATION = "30
sec";
+public static final String DEFAULT_BORED_YIELD_DURATION = "10 millis";
+public static final int DEFAULT_MAX_CONCURRENT_THREADS = 1;
+
+private String flowControllerGracefulShutdownPeriod =
DEFAULT_FLOW_CONTROLLER_GRACEFUL_SHUTDOWN_PERIOD;
+private String flowServiceWriteDelayInterval =
DEFAULT_FLOW_SERVICE_WRITE_DELAY_INTERVAL;
+private String administrativeYieldDuration =
DEFAULT_ADMINISTRATIVE_YIELD_DURATION;
+private String boredYieldDuration = DEFAULT_BORED_YIELD_DURATION;
+private Number maxConcurrentThreads = DEFAULT_MAX_CONCURRENT_THREADS;
+
+public CorePropertiesSchemaV2() {
+}
+
+public CorePropertiesSchemaV2(Map map) {
+flowControllerGracefulShutdownPeriod = getOptionalKeyAsType(map,
FLOW_CONTROLLER_SHUTDOWN_PERIOD_KEY, String.class,
+CORE_PROPS_KEY,
DEFAULT_FLOW_CONTROLLER_GRACEFUL_SHUTDOWN_PERIOD);
+flowServiceWriteDelayInterval = getOptionalKeyAsType(map,
FLOW_SERVICE_WRITE_DELAY_INTERVAL_KEY, String.class,
+CORE_PROPS_KEY, DEFAULT_FLOW_SERVICE_WRITE_DELAY_INTERVAL);
+administrativeYieldDuration = getOptionalKeyAsType(map,
ADMINISTRATIVE_YIELD_DURATION_KEY, String.class,
+CORE_PROPS_KEY, DEFAULT_ADMINISTRATIVE_YIELD_DURATION);
+boredYieldDuration = getOptionalKeyAsType(map,
BORED_YIELD_DURATION_KEY, String.class, CORE_PROPS_KEY,
DEFAULT_BORED_YIELD_DURATION);
+maxConcurrentThreads = getOptionalKeyAsType(map,
MAX_CONCURRENT_THREADS_KEY, Number.class,
+CORE_PROPS_KEY, DEFAULT_MAX_CONCURRENT_THREADS);
+}
+
+@Override
+public Map<String, Object> toMap() {
+Map<String, Object> result = mapSupplier.get();
+result.put(FLOW_CONTROLLER_SHUTDOWN_PERIOD_KEY,
flowControllerGracefulShutdownPeriod);
+result.put(FLOW_SERVICE_WRITE_DELAY_INTERVAL_KEY,
flowServiceWriteDelayInterval);
+result.put(ADMINISTRATIVE_YIELD_DURATION_KEY,
administrativeYieldDuration);
+result.put(BORED_YIELD_DURATION_KEY, boredYieldDuration);
+result.put(MAX_CONCURRENT_THREADS_KEY, maxConcurrentThreads);
+return result;
+}
+
+public String getFlowControllerGracefulShutdownPeriod() {
--- End diff --
The gett
[GitHub] nifi-minifi pull request #80: MINIFI-277: Write variable registry property t...
Github user brosander commented on a diff in the pull request:
https://github.com/apache/nifi-minifi/pull/80#discussion_r113937508
--- Diff:
minifi-commons/minifi-commons-schema/src/main/java/org/apache/nifi/minifi/commons/schema/v2/CorePropertiesSchemaV2.java
---
@@ -0,0 +1,95 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.minifi.commons.schema.v2;
+
+import static
org.apache.nifi.minifi.commons.schema.common.CommonPropertyKeys.CORE_PROPS_KEY;
+import static
org.apache.nifi.minifi.commons.schema.common.CommonPropertyKeys.MAX_CONCURRENT_THREADS_KEY;
+
+import java.util.Map;
+
+import org.apache.nifi.minifi.commons.schema.common.BaseSchema;
+import org.apache.nifi.minifi.commons.schema.common.WritableSchema;
+
+/**
+ *
+ */
+public class CorePropertiesSchemaV2 extends BaseSchema implements
WritableSchema {
+
+public static final String FLOW_CONTROLLER_SHUTDOWN_PERIOD_KEY = "flow
controller graceful shutdown period";
+public static final String FLOW_SERVICE_WRITE_DELAY_INTERVAL_KEY =
"flow service write delay interval";
+public static final String ADMINISTRATIVE_YIELD_DURATION_KEY =
"administrative yield duration";
+public static final String BORED_YIELD_DURATION_KEY = "bored yield
duration";
+
+public static final String
DEFAULT_FLOW_CONTROLLER_GRACEFUL_SHUTDOWN_PERIOD = "10 sec";
+public static final String DEFAULT_FLOW_SERVICE_WRITE_DELAY_INTERVAL =
"500 ms";
+public static final String DEFAULT_ADMINISTRATIVE_YIELD_DURATION = "30
sec";
+public static final String DEFAULT_BORED_YIELD_DURATION = "10 millis";
+public static final int DEFAULT_MAX_CONCURRENT_THREADS = 1;
--- End diff --
The above constants can probably be deleted and statically imported from
CorePropertiesSchema
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi-minifi pull request #80: MINIFI-277: Write variable registry property t...
Github user brosander commented on a diff in the pull request:
https://github.com/apache/nifi-minifi/pull/80#discussion_r113937276
--- Diff:
minifi-commons/minifi-commons-schema/src/main/java/org/apache/nifi/minifi/commons/schema/v2/CorePropertiesSchemaV2.java
---
@@ -0,0 +1,95 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.minifi.commons.schema.v2;
+
+import static
org.apache.nifi.minifi.commons.schema.common.CommonPropertyKeys.CORE_PROPS_KEY;
+import static
org.apache.nifi.minifi.commons.schema.common.CommonPropertyKeys.MAX_CONCURRENT_THREADS_KEY;
+
+import java.util.Map;
+
+import org.apache.nifi.minifi.commons.schema.common.BaseSchema;
+import org.apache.nifi.minifi.commons.schema.common.WritableSchema;
+
+/**
+ *
+ */
+public class CorePropertiesSchemaV2 extends BaseSchema implements
WritableSchema {
--- End diff --
Can this implement ConvertableSchema instead of
WritableSchema?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi pull request #1711: NIFI-3755 - Restoring Hive exception handling behav...
GitHub user brosander opened a pull request: https://github.com/apache/nifi/pull/1711 NIFI-3755 - Restoring Hive exception handling behavior Thank you for submitting a contribution to Apache NiFi. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [X] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [X] Does your PR title start with NIFI- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [X] Has your PR been rebased against the latest commit within the target branch (typically master)? - [X] Is your initial contribution a single, squashed commit? ### For code changes: - [X] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder? - [X] Have you written or updated unit tests to verify your changes? - [X] - N/A - If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [X] - N/A - If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly? - [X] - N/A - If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly? - [X] - N/A - If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties? ### For documentation related changes: - [X] - N/A - Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. You can merge this pull request into a Git repository by running: $ git pull https://github.com/brosander/nifi NIFI-3755 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi/pull/1711.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1711 commit db9f7dbedff812ae5fd692f7c6b8050aac339a22 Author: Bryan Rosander <brosan...@apache.org> Date: 2017-04-27T19:37:27Z NIFI-3755 - Restoring Hive exception handling behavior --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi issue #1697: NIFI-3695 - added proxy dn flag to support providing autho...
Github user brosander commented on the issue: https://github.com/apache/nifi/pull/1697 +1 looks good to me, thoughts @mcgilman ? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi pull request #1698: NIFI-3744 - PutHiveStreaming cleanup null fixes
GitHub user brosander opened a pull request: https://github.com/apache/nifi/pull/1698 NIFI-3744 - PutHiveStreaming cleanup null fixes Thank you for submitting a contribution to Apache NiFi. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [X] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [X] Does your PR title start with NIFI- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [X] Has your PR been rebased against the latest commit within the target branch (typically master)? - [X] Is your initial contribution a single, squashed commit? ### For code changes: - [X] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder? - [X] Have you written or updated unit tests to verify your changes? - [X] - N/A - If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [X] - N/A - If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly? - [X] - N/A - If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly? - [X] - N/A - If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties? ### For documentation related changes: - [X] - N/A - Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. You can merge this pull request into a Git repository by running: $ git pull https://github.com/brosander/nifi NIFI-3744 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi/pull/1698.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1698 commit 29b37d78c199679d4bd0eb8b879bd1d53e0285e1 Author: Bryan Rosander <brosan...@apache.org> Date: 2017-04-26T14:56:40Z NIFI-3744 - PutHiveStreaming cleanup null fixes --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi issue #1693: NIFI-3737 - Ran optipng on all png files
Github user brosander commented on the issue: https://github.com/apache/nifi/pull/1693 @andrewmlim those images work fine for me on OSx Sierra (10.12.4) and on Ubuntu Linux. I asked @mattyb149 to try them and he said they worked for him on Sierra and not El Capitan. I think we may have found an already resolved bug in the OSx png rendering functionality. Still backed images out so that we don't cause confusion or problems for El Capitan users. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi pull request #1693: NIFI-3737 - Ran optipng on all png files
GitHub user brosander opened a pull request: https://github.com/apache/nifi/pull/1693 NIFI-3737 - Ran optipng on all png files Thank you for submitting a contribution to Apache NiFi. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [X] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [X] Does your PR title start with NIFI- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [X] Has your PR been rebased against the latest commit within the target branch (typically master)? - [X] Is your initial contribution a single, squashed commit? ### For code changes: - [X] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder? - [X] - N/A - Have you written or updated unit tests to verify your changes? - [X] - N/A - If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [X] - N/A - If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly? - [X] - N/A - If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly? - [X] - N/A - If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties? ### For documentation related changes: - [X] - N/A - Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. You can merge this pull request into a Git repository by running: $ git pull https://github.com/brosander/nifi NIFI-3737 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi/pull/1693.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1693 commit 599a25015cb9287b3e79852c44e4759702a4c52c Author: Bryan Rosander <brosan...@apache.org> Date: 2017-04-25T05:03:22Z NIFI-3737 - Ran optipng on all png files --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi issue #1669: NIFI-3695 - created the nifi admin toolkit which includes ...
Github user brosander commented on the issue: https://github.com/apache/nifi/pull/1669 +1 looks good to me, seems to be working, @alopresto did @YolandaMDavis resolve your concerns? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi-minifi pull request #79: MINIFI-272 - Delegating C2 Provider, Caching C...
GitHub user brosander opened a pull request: https://github.com/apache/nifi-minifi/pull/79 MINIFI-272 - Delegating C2 Provider, Caching ConfigService, Tests Thank you for submitting a contribution to Apache NiFi - MiNiFi. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [X] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [X] Does your PR title start with MINIFI- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [X] Has your PR been rebased against the latest commit within the target branch (typically master)? - [X] Is your initial contribution a single, squashed commit? ### For code changes: - [X] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi-minifi folder? - [X] Have you written or updated unit tests to verify your changes? - [X] - N/A - If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [X] - N/A - If applicable, have you updated the LICENSE file, including the main LICENSE file under minifi-assembly? - [X] - N/A - If applicable, have you updated the NOTICE file, including the main NOTICE file found under minifi-assembly? ### For documentation related changes: - [X] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. You can merge this pull request into a Git repository by running: $ git pull https://github.com/brosander/nifi-minifi MINIFI-272 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi-minifi/pull/79.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #79 commit ebd0a8ee06cfdec0330d1ee07bc248ae8d71f1fa Author: Bryan Rosander <brosan...@apache.org> Date: 2017-04-18T19:25:08Z MINIFI-272 - Delegating C2 Provider, Caching ConfigService, Tests --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi issue #1669: NIFI-3695 - created the nifi admin toolkit which includes ...
Github user brosander commented on the issue: https://github.com/apache/nifi/pull/1669 Hey @YolandaMDavis this looks like a really good addition to the nifi toolkit! Being able to send bulletins and manage nodes via the command line will likely be a really nice-to-have feature for admins. Put feedback above. Please let me know if you need more info. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi issue #1669: NIFI-3695 - created the nifi admin toolkit which includes ...
Github user brosander commented on the issue: https://github.com/apache/nifi/pull/1669 We may want to add some usage information to the documentation as well. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi issue #1669: NIFI-3695 - created the nifi admin toolkit which includes ...
Github user brosander commented on the issue: https://github.com/apache/nifi/pull/1669 Would it be difficult to create the other binary resources in test setup? If not it might be preferable to having zips and tars in source control. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi issue #1669: NIFI-3695 - created the nifi admin toolkit which includes ...
Github user brosander commented on the issue: https://github.com/apache/nifi/pull/1669 For the keystore, truststores in test resources, is it possible we could use the tls-toolkit to generate those on demand? https://github.com/apache/nifi-minifi/blob/master/minifi-c2/minifi-c2-integration-tests/src/test/java/org/apache/nifi/minifi/c2/integration/test/AbstractTestSecure.java#L71 That should keep them from expiring or needing to be changed later. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi pull request #1669: NIFI-3695 - created the nifi admin toolkit which in...
Github user brosander commented on a diff in the pull request:
https://github.com/apache/nifi/pull/1669#discussion_r111808071
--- Diff:
nifi-toolkit/nifi-toolkit-admin/src/main/groovy/org/apache/nifi/toolkit/admin/client/NiFiClientUtil.groovy
---
@@ -0,0 +1,156 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.toolkit.admin.client
+
+import com.google.common.collect.Lists
+import com.sun.jersey.api.client.Client
+import com.sun.jersey.api.client.ClientResponse
+import com.sun.jersey.api.client.WebResource
+import org.apache.nifi.util.NiFiProperties
+import org.apache.nifi.util.StringUtils
+import org.apache.nifi.web.api.dto.NodeDTO
+import org.apache.nifi.web.api.entity.ClusterEntity
+import org.slf4j.Logger
+import org.slf4j.LoggerFactory
+
+public class NiFiClientUtil {
+
+private static final Logger logger =
LoggerFactory.getLogger(NiFiClientUtil.class)
+private final static String GET_CLUSTER_ENDPOINT
="/nifi-api/controller/cluster"
+
+public static Boolean isCluster(final NiFiProperties niFiProperties){
+String clusterNode =
niFiProperties.getProperty(NiFiProperties.CLUSTER_IS_NODE)
+return Boolean.valueOf(clusterNode)
+}
+
+public static String getUrl(NiFiProperties niFiProperties, String
endpoint){
+
+final StringBuilder urlBuilder = new StringBuilder();
+
+
if(!StringUtils.isEmpty(niFiProperties.getProperty(NiFiProperties.WEB_HTTPS_PORT))){
+urlBuilder.append("https://;)
+
urlBuilder.append(StringUtils.isEmpty(niFiProperties.getProperty(NiFiProperties.WEB_HTTPS_HOST))
? "localhost": niFiProperties.getProperty(NiFiProperties.WEB_HTTPS_HOST))
+urlBuilder.append(":")
+
urlBuilder.append(StringUtils.isEmpty(niFiProperties.getProperty(NiFiProperties.WEB_HTTPS_PORT))
? "8081" : niFiProperties.getProperty(NiFiProperties.WEB_HTTPS_PORT))
+}else{
+urlBuilder.append("http://;)
+
urlBuilder.append(StringUtils.isEmpty(niFiProperties.getProperty(NiFiProperties.WEB_HTTP_HOST))
? "localhost": niFiProperties.getProperty(NiFiProperties.WEB_HTTP_HOST))
+urlBuilder.append(":")
+
urlBuilder.append(StringUtils.isEmpty(niFiProperties.getProperty(NiFiProperties.WEB_HTTPS_PORT))
? "8080": niFiProperties.getProperty(NiFiProperties.WEB_HTTPS_PORT))
+}
+
+if(!StringUtils.isEmpty(endpoint)) {
+urlBuilder.append(endpoint)
+}
+
+urlBuilder.toString()
+}
+
+public static String getUrl(NiFiProperties niFiProperties, NodeDTO
nodeDTO, String endpoint){
+
+final StringBuilder urlBuilder = new StringBuilder();
+
if(!StringUtils.isEmpty(niFiProperties.getProperty(NiFiProperties.WEB_HTTPS_PORT))){
+urlBuilder.append("https://;)
+
+}else{
+urlBuilder.append("http://;)
+}
+urlBuilder.append(nodeDTO.address)
+urlBuilder.append(":")
+urlBuilder.append(nodeDTO.apiPort)
+
+if(!StringUtils.isEmpty(endpoint)) {
+urlBuilder.append(endpoint)
+}
+
+urlBuilder.toString()
+}
+
+public static ClusterEntity getCluster(final Client client,
NiFiProperties niFiProperties, List activeUrls){
+
+if(activeUrls.isEmpty()){
+final String url = getUrl(niFiProperties,null)
+activeUrls.add(url)
+}
+
+for(String activeUrl: activeUrls) {
+
+try {
+
+String url = activeUrl + GET_CLUSTER_ENDPOINT
+final WebResource webResource = client.resource(url)
+final ClientResponse response =
webResource.type("application/json"
[GitHub] nifi pull request #1669: NIFI-3695 - created the nifi admin toolkit which in...
Github user brosander commented on a diff in the pull request: https://github.com/apache/nifi/pull/1669#discussion_r111807253 --- Diff: nifi-toolkit/nifi-toolkit-admin/pom.xml --- @@ -0,0 +1,189 @@ + + +http://maven.apache.org/POM/4.0.0; + xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance; + xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd;> + +org.apache.nifi +nifi-toolkit +1.2.0-SNAPSHOT + + +4.0.0 + +nifi-toolkit-admin + + + +commons-cli +commons-cli + + +com.google.guava +guava + + +org.apache.nifi +nifi-toolkit-tls + + +com.sun.jersey +jersey-client + + +com.fasterxml.jackson.core +jackson-databind + + +org.apache.nifi +nifi-client-dto +1.2.0-SNAPSHOT + + +org.apache.nifi +nifi-properties + + +org.apache.nifi +nifi-properties-loader + + +ch.qos.logback +logback-classic + + + + +org.codehaus.jackson +jackson-mapper-asl + + +com.fasterxml.jackson.jaxrs +jackson-jaxrs-json-provider +RELEASE + + +com.sun.jersey +jersey-bundle +RELEASE + + +com.sun.jersey +jersey-json +RELEASE + + +org.apache.commons +commons-compress + + + +com.github.stefanbirkner +system-rules +1.16.0 +test + + +org.spockframework +spock-core +test + + +cglib +cglib-nodep +test + + + + + + +org.apache.maven.plugins +maven-compiler-plugin + + + +compile +testCompile + + + groovy-eclipse-compiler + + + + +1.8 +1.8 + + + +org.codehaus.groovy +groovy-eclipse-compiler +2.9.2-01 + + +org.codehaus.groovy +groovy-eclipse-batch +2.4.3-01 + + + + +org.codehaus.mojo +build-helper-maven-plugin +1.5 + + +add-source +generate-sources + +add-source + + + +src/main/groovy + + + + +add-test-source +generate-test-sources + +add-test-source + + + +src/test/groovy + + + + + + +org.apache.rat +apache-rat-plugin + + + src/test/resources/conf/nifi.properties --- End diff -- Not a huge deal, but the properties and xml files could have the license headers so they wouldn't have to be i
[GitHub] nifi pull request #1669: NIFI-3695 - created the nifi admin toolkit which in...
Github user brosander commented on a diff in the pull request: https://github.com/apache/nifi/pull/1669#discussion_r111807027 --- Diff: nifi-toolkit/nifi-toolkit-admin/pom.xml --- @@ -0,0 +1,189 @@ + + +http://maven.apache.org/POM/4.0.0; + xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance; + xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd;> + +org.apache.nifi +nifi-toolkit +1.2.0-SNAPSHOT + + +4.0.0 + +nifi-toolkit-admin + + + +commons-cli +commons-cli + + +com.google.guava +guava + + +org.apache.nifi +nifi-toolkit-tls + + +com.sun.jersey +jersey-client + + +com.fasterxml.jackson.core +jackson-databind + + +org.apache.nifi +nifi-client-dto +1.2.0-SNAPSHOT --- End diff -- can we move the version info up into parent pom for nifi-client-dto? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi pull request #1669: NIFI-3695 - created the nifi admin toolkit which in...
Github user brosander commented on a diff in the pull request:
https://github.com/apache/nifi/pull/1669#discussion_r111806902
--- Diff:
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/test/groovy/org/apache/nifi/web/StandardNiFiServiceFacadeSpec.groovy
---
@@ -49,6 +53,7 @@ class StandardNiFiServiceFacadeSpec extends Specification
{
SecurityContextHolder.getContext().setAuthentication(null);
}
--- End diff --
Is there an issue with these ignored tests?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi pull request #1669: NIFI-3695 - created the nifi admin toolkit which in...
Github user brosander commented on a diff in the pull request:
https://github.com/apache/nifi/pull/1669#discussion_r111806379
--- Diff:
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-cluster/src/main/java/org/apache/nifi/cluster/manager/BulletinMerger.java
---
@@ -71,7 +74,12 @@ public int compare(BulletinEntity o1, BulletinEntity o2)
{
}
}
-Collections.sort(bulletinEntities, (BulletinEntity o1,
BulletinEntity o2) -> {
+final List entities = Lists.newArrayList();
+
+final Map<String,List> groupingEntities =
bulletinEntities.stream().collect(Collectors.groupingBy(b ->
b.getBulletin().getMessage()));
+groupingEntities.keySet().stream().forEach(m ->
entities.add(groupingEntities.get(m).get(0)));
--- End diff --
Would streaming values make this operation a little easier?
Something like:
```
groupingEntities.values().stream().map(e ->
e.get(0)).forEach(entities::add);
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi pull request #1641: NIFI-3666 - TlsToolkit using .equals() comparison f...
GitHub user brosander opened a pull request: https://github.com/apache/nifi/pull/1641 NIFI-3666 - TlsToolkit using .equals() comparison for parent and root Thank you for submitting a contribution to Apache NiFi. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [X] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [X] Does your PR title start with NIFI- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [X] Has your PR been rebased against the latest commit within the target branch (typically master)? - [X] Is your initial contribution a single, squashed commit? ### For code changes: - [X] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder? - [X] Have you written or updated unit tests to verify your changes? - [X] - N/A - If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [X] - N/A - If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly? - [X] - N/A - If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly? - [X] - N/A - If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties? ### For documentation related changes: - [X] - N/A - Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. You can merge this pull request into a Git repository by running: $ git pull https://github.com/brosander/nifi NIFI-3666-tls-toolkit Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi/pull/1641.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1641 commit 862e8034353cd5453b15f8cf578c4463da9c2526 Author: Bryan Rosander <brosan...@apache.org> Date: 2017-03-31T17:16:57Z NIFI-3666 - TlsToolkit using .equals() comparison for parent and root --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi-minifi pull request #78: MINIFI-251 - Added explicit Java runtime argum...
Github user brosander commented on a diff in the pull request: https://github.com/apache/nifi-minifi/pull/78#discussion_r108285965 --- Diff: minifi-nar-bundles/minifi-framework-bundle/minifi-framework/minifi-resources/src/main/resources/conf/bootstrap.conf --- @@ -85,9 +85,13 @@ java.arg.4=-Djava.net.preferIPv4Stack=true java.arg.5=-Dsun.net.http.allowRestrictedHeaders=true java.arg.6=-Djava.protocol.handler.pkgs=sun.net.www.protocol +# Sets the provider of SecureRandom to /dev/urandom to prevent blocking on VMs +java.arg.7=-Djava.security.egd=file:/dev/urandom --- End diff -- @trixpan have you tested that this works in Java 8? I've used a slightly modified version found [on stack overflow](http://stackoverflow.com/questions/137212/how-to-solve-performance-problem-with-java-securerandom#answer-2325109): ``` -Djava.security.egd=file:/dev/./urandom ``` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi-minifi pull request #76: MINIFI-242 - Initial integration tests focused...
GitHub user brosander opened a pull request: https://github.com/apache/nifi-minifi/pull/76 MINIFI-242 - Initial integration tests focused around backwards compa⦠â¦tibility Thank you for submitting a contribution to Apache NiFi - MiNiFi. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [ ] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [ ] Does your PR title start with MINIFI- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [ ] Has your PR been rebased against the latest commit within the target branch (typically master)? - [ ] Is your initial contribution a single, squashed commit? ### For code changes: - [ ] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi-minifi folder? - [ ] Have you written or updated unit tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the LICENSE file, including the main LICENSE file under minifi-assembly? - [ ] If applicable, have you updated the NOTICE file, including the main NOTICE file found under minifi-assembly? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. You can merge this pull request into a Git repository by running: $ git pull https://github.com/brosander/nifi-minifi MINIFI-242 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi-minifi/pull/76.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #76 commit 3893d61255837fb93ec9eadca376ac5c480a5b99 Author: Bryan Rosander <brosan...@apache.org> Date: 2017-03-21T16:37:05Z MINIFI-242 - Initial integration tests focused around backwards compatibility --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi-minifi pull request #74: MINIFI-240 - Docker image for MiNiFi Java
Github user brosander commented on a diff in the pull request: https://github.com/apache/nifi-minifi/pull/74#discussion_r106938593 --- Diff: minifi-docker/dockerhub/Dockerfile --- @@ -0,0 +1,46 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +# + +FROM openjdk:8 +MAINTAINER Apache MiNiFi <d...@nifi.apache.org> + +ARG UID=1000 +ARG GID=50 +ARG MINIFI_VERSION=0.2.0 + +ENV MINIFI_BASE_DIR /opt/minifi +ENV MINIFI_HOME $MINIFI_BASE_DIR/minifi-$MINIFI_VERSION +ENV MINIFI_BINARY_URL https://archive.apache.org/dist/minifi/$MINIFI_VERSION/minifi-$MINIFI_VERSION-bin.tar.gz --- End diff -- Need to fix this url path be https://archive.apache.org/dist/nifi/minifi/ --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi-minifi pull request #75: MINIFI-238 - MiNiFi Initial Command and Contro...
GitHub user brosander opened a pull request: https://github.com/apache/nifi-minifi/pull/75 MINIFI-238 - MiNiFi Initial Command and Control Server Implementation Thank you for submitting a contribution to Apache NiFi - MiNiFi. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [X] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [X] Does your PR title start with MINIFI- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [X] Has your PR been rebased against the latest commit within the target branch (typically master)? - [X] Is your initial contribution a single, squashed commit? ### For code changes: - [X] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi-minifi folder? - [X] Have you written or updated unit tests to verify your changes? - [X] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [X] If applicable, have you updated the LICENSE file, including the main LICENSE file under minifi-assembly? - [X] If applicable, have you updated the NOTICE file, including the main NOTICE file found under minifi-assembly? ### For documentation related changes: - [X] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. You can merge this pull request into a Git repository by running: $ git pull https://github.com/brosander/nifi-minifi MINIFI-238 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi-minifi/pull/75.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #75 commit f1d2c21070321bf9abbb97f9885344a20b546d8b Author: Bryan Rosander <brosan...@apache.org> Date: 2017-03-01T22:06:11Z MINIFI-238 - MiNiFi Initial Command and Control Server Implementation --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi-minifi pull request #74: MINIFI-240 - Docker image for MiNiFi Java
GitHub user brosander opened a pull request: https://github.com/apache/nifi-minifi/pull/74 MINIFI-240 - Docker image for MiNiFi Java Thank you for submitting a contribution to Apache NiFi - MiNiFi. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [X] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [X] Does your PR title start with MINIFI- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [X] Has your PR been rebased against the latest commit within the target branch (typically master)? - [X] Is your initial contribution a single, squashed commit? ### For code changes: - [X] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi-minifi folder? - [X] - N/A - Have you written or updated unit tests to verify your changes? - [X] - N/A - If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [X] - N/A - If applicable, have you updated the LICENSE file, including the main LICENSE file under minifi-assembly? - [X] - N/A - If applicable, have you updated the NOTICE file, including the main NOTICE file found under minifi-assembly? ### For documentation related changes: - [X] - N/A - Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. You can merge this pull request into a Git repository by running: $ git pull https://github.com/brosander/nifi-minifi MINIFI-240 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi-minifi/pull/74.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #74 commit efc5c669702b930bca49bf07da6e679f95adc990 Author: Bryan Rosander <brosan...@apache.org> Date: 2017-03-15T17:21:58Z MINIFI-240 - Docker image for MiNiFi Java --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi-minifi-cpp issue #63: MINIFI-217: Implement namespaces.
Github user brosander commented on the issue: https://github.com/apache/nifi-minifi-cpp/pull/63 @phrocker From a Java dev perspective, the namespaces look a little shallower than the Java package names. I think that's probably justifiable though to prevent a ton of boilerplate, given that the number of classes is much lower in cpp-land. Big fan of namespacing to avoid name clashes and just generally keep different ideas separate from each other. Think this looks like a great step forward :smile: --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi pull request #1578: NIFI-3574 - PutHiveStreaming UGI fixes
Github user brosander commented on a diff in the pull request:
https://github.com/apache/nifi/pull/1578#discussion_r105418460
--- Diff:
nifi-nar-bundles/nifi-hive-bundle/nifi-hive-processors/src/main/java/org/apache/nifi/util/hive/HiveConfigurator.java
---
@@ -74,7 +74,8 @@ public HiveConf getConfigurationFromFiles(final String
configFiles) {
public void preload(Configuration configuration) {
try {
-FileSystem.get(configuration);
+FileSystem.get(configuration).close();
--- End diff --
Hey @mattyb149
The two clusters I was testing against (secure and unsecure) were both
configured for HA so I did test preloading with HA inadvertently :smile:
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi issue #1578: NIFI-3574 - PutHiveStreaming UGI fixes
Github user brosander commented on the issue: https://github.com/apache/nifi/pull/1578 contrib-check build working on linux, from root nifi folder, on OSX I've been having issues with kafka tests so I just ran the changed modules and those are checking out --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi pull request #1578: NIFI-3574 - PutHiveStreaming UGI fixes
GitHub user brosander opened a pull request: https://github.com/apache/nifi/pull/1578 NIFI-3574 - PutHiveStreaming UGI fixes Thank you for submitting a contribution to Apache NiFi. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [X] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [X] Does your PR title start with NIFI- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [X] Has your PR been rebased against the latest commit within the target branch (typically master)? - [X] Is your initial contribution a single, squashed commit? ### For code changes: - [ ] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder? - [X] Have you written or updated unit tests to verify your changes? - [X] - N/A - If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [X] - N/A - If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly? - [X] - N/A - If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly? - [X] - N/A - If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties? ### For documentation related changes: - [X] - N/A - Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. You can merge this pull request into a Git repository by running: $ git pull https://github.com/brosander/nifi NIFI-3574 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi/pull/1578.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1578 commit 095a235075d2676d870b63dcba5b35507d9f45e7 Author: Bryan Rosander <brosan...@apache.org> Date: 2017-03-08T22:41:59Z NIFI-3574 - PutHiveStreaming UGI fixes --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi issue #1530: NIFI-3490 added SAN option for TLS toolkit in standalone m...
Github user brosander commented on the issue: https://github.com/apache/nifi/pull/1530 I agree that changing flags is likely to cause confusion although since we haven't had a release yet, I would hope it would be minimal impact. Consistency across modes of operation will help reduce surprise which is always a good thing. Do we expect this to be something most users specify or is it a more advanced usecase? If it's more advanced and less common then a short version of the argument may not be necessary. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi pull request #1544: NIFI-3530 - Ensuring configuration and ugi informat...
GitHub user brosander reopened a pull request: https://github.com/apache/nifi/pull/1544 NIFI-3530 - Ensuring configuration and ugi information is passed down⦠⦠to Hive in PutHiveStreaming Thank you for submitting a contribution to Apache NiFi. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [X] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [X] Does your PR title start with NIFI- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [X] Has your PR been rebased against the latest commit within the target branch (typically master)? - [X] Is your initial contribution a single, squashed commit? ### For code changes: - [X] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder? - [X] Have you written or updated unit tests to verify your changes? - [X] - N/A - If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [X] - N/A - If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly? - [X] - N/A - If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly? - [X] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties? ### For documentation related changes: - [X] - N/A - Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. You can merge this pull request into a Git repository by running: $ git pull https://github.com/brosander/nifi NIFI-3530 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi/pull/1544.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1544 commit e476c43a5bde14e15f317b6e5eb371206c22ce06 Author: Bryan Rosander <brosan...@apache.org> Date: 2017-02-24T21:09:42Z NIFI-3530 - Ensuring configuration and ugi information is passed down to Hive in PutHiveStreaming --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi pull request #1544: NIFI-3530 - Ensuring configuration and ugi informat...
GitHub user brosander opened a pull request: https://github.com/apache/nifi/pull/1544 NIFI-3530 - Ensuring configuration and ugi information is passed down⦠⦠to Hive in PutHiveStreaming Thank you for submitting a contribution to Apache NiFi. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [X] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [X] Does your PR title start with NIFI- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [X] Has your PR been rebased against the latest commit within the target branch (typically master)? - [X] Is your initial contribution a single, squashed commit? ### For code changes: - [X] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder? - [X] Have you written or updated unit tests to verify your changes? - [X] - N/A - If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [X] - N/A - If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly? - [X] - N/A - If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly? - [X] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties? ### For documentation related changes: - [X] - N/A - Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. You can merge this pull request into a Git repository by running: $ git pull https://github.com/brosander/nifi NIFI-3530 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi/pull/1544.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1544 commit e476c43a5bde14e15f317b6e5eb371206c22ce06 Author: Bryan Rosander <brosan...@apache.org> Date: 2017-02-24T21:09:42Z NIFI-3530 - Ensuring configuration and ugi information is passed down to Hive in PutHiveStreaming --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi pull request #1544: NIFI-3530 - Ensuring configuration and ugi informat...
Github user brosander closed the pull request at: https://github.com/apache/nifi/pull/1544 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi-minifi pull request #72: MINIFI-216 - Allowing override of nifi.propert...
Github user brosander commented on a diff in the pull request: https://github.com/apache/nifi-minifi/pull/72#discussion_r102292357 --- Diff: minifi-bootstrap/src/main/java/org/apache/nifi/minifi/bootstrap/util/ConfigTransformer.java --- @@ -704,7 +691,7 @@ protected static void addTextElement(final Element element, final String name, f } public static final String PROPERTIES_FILE_APACHE_2_0_LICENSE = -"# Licensed to the Apache Software Foundation (ASF) under one or more\n" + +" Licensed to the Apache Software Foundation (ASF) under one or more\n" + --- End diff -- @phrocker due to the change in the way the props file is written out. [The store method](https://github.com/apache/nifi-minifi/pull/72/files#diff-a022622bcf128ab992d247f2a8694a0eR263) will prepend a # to the comment string. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi-minifi pull request #72: MINIFI-216 - Allowing override of nifi.propert...
GitHub user brosander opened a pull request: https://github.com/apache/nifi-minifi/pull/72 MINIFI-216 - Allowing override of nifi.properties values Thank you for submitting a contribution to Apache NiFi - MiNiFi. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [X] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [X] Does your PR title start with MINIFI- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [X] Has your PR been rebased against the latest commit within the target branch (typically master)? - [X] Is your initial contribution a single, squashed commit? ### For code changes: - [X] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi-minifi folder? - [X] Have you written or updated unit tests to verify your changes? - [X] - N/A - If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [X] - N/A - If applicable, have you updated the LICENSE file, including the main LICENSE file under minifi-assembly? - [X] - N/A - If applicable, have you updated the NOTICE file, including the main NOTICE file found under minifi-assembly? ### For documentation related changes: - [X] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. You can merge this pull request into a Git repository by running: $ git pull https://github.com/brosander/nifi-minifi MINIFI-216 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/nifi-minifi/pull/72.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #72 commit 397d52f99ae3e7ef0b252336c11e164bfd66ec71 Author: Bryan Rosander <brosan...@apache.org> Date: 2017-02-21T16:15:09Z MINIFI-216 - Allowing override of nifi.properties values --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi issue #1508: NIFI-3355 Allows NiFi to bind to specific network interfac...
Github user brosander commented on the issue: https://github.com/apache/nifi/pull/1508 +1 merging --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi issue #1508: NIFI-3355 Allows NiFi to bind to specific network interfac...
Github user brosander commented on the issue: https://github.com/apache/nifi/pull/1508 Reviewing --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi pull request #1364: NIFI-1856 ExecuteStreamCommand Needs to Consume Sta...
Github user brosander commented on a diff in the pull request:
https://github.com/apache/nifi/pull/1364#discussion_r101328875
--- Diff:
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/ExecuteStreamCommand.java
---
@@ -342,16 +393,45 @@ public void onTrigger(ProcessContext context, final
ProcessSession session) thro
final InputStream pis = process.getInputStream();
final InputStream pes = process.getErrorStream();
final BufferedInputStream bis = new
BufferedInputStream(pis);
+final BufferedInputStream bes = new
BufferedInputStream(pes);
final BufferedReader bufferedReader = new
BufferedReader(new InputStreamReader(pes))) {
int exitCode = -1;
final BufferedOutputStream bos = new BufferedOutputStream(pos);
+final StringBuilder strBldr = new StringBuilder();
FlowFile outputFlowFile = putToAttribute ? inputFlowFile :
session.create(inputFlowFile);
+FlowFile errorFlowFile=null;
+FlowWriterThread flowWriter = null;
+if (REDIRECT_ERROR_ERROR_STREAM.equals(redirectErrorStream)) {
+ errorFlowFile = session.create(inputFlowFile);
+flowWriter = new FlowWriterThread(errorFlowFile, session,
bes);
+this.executor.submit(flowWriter);
+}
+//if redirect error stream is set to log, then write the error
to nifi component log.
+if (REDIRECT_ERROR_LOG.equals(redirectErrorStream)) {
+this.executor.submit(new Runnable() {
+@Override
+public void run() {
+ try (final BufferedReader reader = new
BufferedReader(new InputStreamReader(pes))) {
+String line;
+while ((line = bufferedReader.readLine()) !=
null) {
+ logger.warn("ExecuteStreamCommand
:"+line);
+ strBldr.append(line).append("\n");
--- End diff --
I think this probably warrants its own jira as it was existing behavior,
I'm going to write it up separately
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi pull request #1364: NIFI-1856 ExecuteStreamCommand Needs to Consume Sta...
Github user brosander commented on a diff in the pull request:
https://github.com/apache/nifi/pull/1364#discussion_r101160112
--- Diff:
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/ExecuteStreamCommand.java
---
@@ -342,16 +393,45 @@ public void onTrigger(ProcessContext context, final
ProcessSession session) thro
final InputStream pis = process.getInputStream();
final InputStream pes = process.getErrorStream();
final BufferedInputStream bis = new
BufferedInputStream(pis);
+final BufferedInputStream bes = new
BufferedInputStream(pes);
final BufferedReader bufferedReader = new
BufferedReader(new InputStreamReader(pes))) {
int exitCode = -1;
final BufferedOutputStream bos = new BufferedOutputStream(pos);
+final StringBuilder strBldr = new StringBuilder();
FlowFile outputFlowFile = putToAttribute ? inputFlowFile :
session.create(inputFlowFile);
+FlowFile errorFlowFile=null;
+FlowWriterThread flowWriter = null;
+if (REDIRECT_ERROR_ERROR_STREAM.equals(redirectErrorStream)) {
+ errorFlowFile = session.create(inputFlowFile);
+flowWriter = new FlowWriterThread(errorFlowFile, session,
bes);
+this.executor.submit(flowWriter);
+}
+//if redirect error stream is set to log, then write the error
to nifi component log.
+if (REDIRECT_ERROR_LOG.equals(redirectErrorStream)) {
+this.executor.submit(new Runnable() {
+@Override
+public void run() {
+ try (final BufferedReader reader = new
BufferedReader(new InputStreamReader(pes))) {
+String line;
+while ((line = bufferedReader.readLine()) !=
null) {
+ logger.warn("ExecuteStreamCommand
:"+line);
+ strBldr.append(line).append("\n");
--- End diff --
It looks like we're only going to use the first 4000 characters of strBldr,
we might want to preemptively throw away anything before that so we don't have
to store all the output in memory
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi issue #1364: NIFI-1856 ExecuteStreamCommand Needs to Consume Standard E...
Github user brosander commented on the issue: https://github.com/apache/nifi/pull/1364 Reviewing --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi pull request #1364: NIFI-1856 ExecuteStreamCommand Needs to Consume Sta...
Github user brosander commented on a diff in the pull request:
https://github.com/apache/nifi/pull/1364#discussion_r101158900
--- Diff:
nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/ExecuteStreamCommand.java
---
@@ -287,6 +311,32 @@ protected PropertyDescriptor
getSupportedDynamicPropertyDescriptor(final String
.build();
}
+@OnScheduled
+public void setupExecutor(final ProcessContext context) {
+executor =
Executors.newFixedThreadPool(context.getMaxConcurrentTasks() * 2, new
ThreadFactory() {
+private final ThreadFactory defaultFactory =
Executors.defaultThreadFactory();
+
+@Override
+public Thread newThread(final Runnable r) {
+final Thread t = defaultFactory.newThread(r);
+t.setName("ExecuteStreamCommand " + getIdentifier() + "
Task");
+return t;
+}
+});
+}
+
+@OnUnscheduled
--- End diff --
@rkarthik29
OnUnscheduled runs before all the executing threads are complete, I think
it would be safer to use OnStopped to shutdown the executor
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi issue #1199: NIFI-3017: Exclude org.json dependency from Hive bundle
Github user brosander commented on the issue: https://github.com/apache/nifi/pull/1199 +1, merged --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi issue #972: NIFI-2711 - Making top-level nifi-assemblies directory with...
Github user brosander commented on the issue: https://github.com/apache/nifi/pull/972 Seems like lack of demand for this change, probably should raise visibility on impact via a discuss thread before messing with assemblies anyway --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi pull request #972: NIFI-2711 - Making top-level nifi-assemblies directo...
Github user brosander closed the pull request at: https://github.com/apache/nifi/pull/972 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi issue #1199: NIFI-3017: Exclude org.json dependency from Hive bundle
Github user brosander commented on the issue: https://github.com/apache/nifi/pull/1199 Reviewing... --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] nifi issue #1446: NIFI-3389 - Using long string type for attribute name and ...
Github user brosander commented on the issue: https://github.com/apache/nifi/pull/1446 @markap14 I was thinking this might be a time when verbosity makes it less likely that someone would inadvertently change old behavior I can pull the logic out into methods but anyone wanting to further change the v2 schema would have to be careful not to also impact the v1 schema. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
