nifi git commit: NIFI-4030 Populating default values on GenericRecord from Avro schema if not present in RecordSchema

2017-06-07 Thread markap14 
Repository: nifi
Updated Branches:
  refs/heads/master 45e035686 -> b0c942877


NIFI-4030 Populating default values on GenericRecord from Avro schema if not 
present in RecordSchema

This closes #1896.


Project: http://git-wip-us.apache.org/repos/asf/nifi/repo
Commit: http://git-wip-us.apache.org/repos/asf/nifi/commit/b0c94287
Tree: http://git-wip-us.apache.org/repos/asf/nifi/tree/b0c94287
Diff: http://git-wip-us.apache.org/repos/asf/nifi/diff/b0c94287

Branch: refs/heads/master
Commit: b0c942877682d7aaebe4bec003fae3bff9be263e
Parents: 45e0356
Author: Bryan Bende 
Authored: Wed Jun 7 13:00:13 2017 -0400
Committer: Mark Payne 
Committed: Wed Jun 7 13:59:40 2017 -0400

--
 .../src/main/java/org/apache/nifi/avro/AvroTypeUtil.java| 9 +
 1 file changed, 9 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/nifi/blob/b0c94287/nifi-nar-bundles/nifi-extension-utils/nifi-record-utils/nifi-avro-record-utils/src/main/java/org/apache/nifi/avro/AvroTypeUtil.java
--
diff --git 
a/nifi-nar-bundles/nifi-extension-utils/nifi-record-utils/nifi-avro-record-utils/src/main/java/org/apache/nifi/avro/AvroTypeUtil.java
 
b/nifi-nar-bundles/nifi-extension-utils/nifi-record-utils/nifi-avro-record-utils/src/main/java/org/apache/nifi/avro/AvroTypeUtil.java
index 1417e67..76438c5 100644
--- 
a/nifi-nar-bundles/nifi-extension-utils/nifi-record-utils/nifi-avro-record-utils/src/main/java/org/apache/nifi/avro/AvroTypeUtil.java
+++ 
b/nifi-nar-bundles/nifi-extension-utils/nifi-record-utils/nifi-avro-record-utils/src/main/java/org/apache/nifi/avro/AvroTypeUtil.java
@@ -273,6 +273,15 @@ public class AvroTypeUtil {
 rec.put(fieldName, converted);
 }
 
+// see if the Avro schema has any fields that aren't in the 
RecordSchema, and if those fields have a default
+// value then we want to populate it in the GenericRecord being 
produced
+for (final Field field : avroSchema.getFields()) {
+final Optional recordField = 
recordSchema.getField(field.name());
+if (!recordField.isPresent() && rec.get(field.name()) == null && 
field.defaultVal() != null) {
+rec.put(field.name(), field.defaultVal());
+}
+}
+
 return rec;
 }

[jira] [Commented] (MINIFI-262) Configuration Listener in C++ client to support C2 operations

2017-06-07 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/MINIFI-262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16041310#comment-16041310
 ] 

ASF GitHub Bot commented on MINIFI-262:
---

Github user asfgit closed the pull request at:

https://github.com/apache/nifi-minifi-cpp/pull/112


> Configuration Listener in C++ client to support C2 operations
> -
>
> Key: MINIFI-262
> URL: https://issues.apache.org/jira/browse/MINIFI-262
> Project: Apache NiFi MiNiFi
>  Issue Type: New Feature
>  Components: C++, Command and Control
>Reporter: marco polo
>Assignee: bqiu
>
> Create the concept of configuration listener support a variety of 
> implementations that will restart operations for the MiNiFi C++ agent. 
> The listener will implement a listener that will restart the MiNiFi process. 
> MiNiFi-263 will include the work to implement the configuration listener 
> interface. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (MINIFI-262) Configuration Listener in C++ client to support C2 operations

2017-06-07 Thread ASF subversion and git services (JIRA) 

[ 
https://issues.apache.org/jira/browse/MINIFI-262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16041309#comment-16041309
 ] 

ASF subversion and git services commented on MINIFI-262:


Commit 1a1716dc675886689217860a299f7a8d36850adf in nifi-minifi-cpp's branch 
refs/heads/master from [~bqiu]
[ https://git-wip-us.apache.org/repos/asf?p=nifi-minifi-cpp.git;h=1a1716d ]

MINIFI-262: Configuration Listener

This closes #112.

Signed-off-by: Marc Parisi 


> Configuration Listener in C++ client to support C2 operations
> -
>
> Key: MINIFI-262
> URL: https://issues.apache.org/jira/browse/MINIFI-262
> Project: Apache NiFi MiNiFi
>  Issue Type: New Feature
>  Components: C++, Command and Control
>Reporter: marco polo
>Assignee: bqiu
>
> Create the concept of configuration listener support a variety of 
> implementations that will restart operations for the MiNiFi C++ agent. 
> The listener will implement a listener that will restart the MiNiFi process. 
> MiNiFi-263 will include the work to implement the configuration listener 
> interface. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

nifi-minifi-cpp git commit: MINIFI-262: Configuration Listener

2017-06-07 Thread phrocker 
Repository: nifi-minifi-cpp
Updated Branches:
  refs/heads/master 77a20dbe3 -> 1a1716dc6


MINIFI-262: Configuration Listener

This closes #112.

Signed-off-by: Marc Parisi 


Project: http://git-wip-us.apache.org/repos/asf/nifi-minifi-cpp/repo
Commit: http://git-wip-us.apache.org/repos/asf/nifi-minifi-cpp/commit/1a1716dc
Tree: http://git-wip-us.apache.org/repos/asf/nifi-minifi-cpp/tree/1a1716dc
Diff: http://git-wip-us.apache.org/repos/asf/nifi-minifi-cpp/diff/1a1716dc

Branch: refs/heads/master
Commit: 1a1716dc675886689217860a299f7a8d36850adf
Parents: 77a20db
Author: Bin Qiu 
Authored: Wed Jun 7 09:26:11 2017 -0700
Committer: Marc Parisi 
Committed: Wed Jun 7 13:55:28 2017 -0400

--
 README.md   |  18 ++
 cmake/BuildTests.cmake  |   3 +
 libminifi/include/ConfigurationListener.h   | 122 ++
 libminifi/include/FlowControlProtocol.h |   4 -
 libminifi/include/FlowController.h  |  39 -
 libminifi/include/HttpConfigurationListener.h   |  82 +
 libminifi/include/core/FlowConfiguration.h  |   9 +-
 libminifi/include/core/ProcessGroup.h   |   9 +-
 .../core/repository/FlowFileRepository.h|   2 +-
 libminifi/include/core/yaml/YamlConfiguration.h |  16 ++
 libminifi/include/processors/InvokeHTTP.h   |  58 +--
 libminifi/include/properties/Configure.h|  11 ++
 libminifi/include/utils/HTTPUtils.h |  97 +++
 libminifi/src/ConfigurationListener.cpp | 131 +++
 libminifi/src/Configure.cpp |  34 +++-
 libminifi/src/FlowController.cpp|  37 +++-
 libminifi/src/HttpConfigurationListener.cpp | 167 +++
 libminifi/src/core/FlowConfiguration.cpp|   6 +-
 libminifi/src/core/ProcessGroup.cpp |   4 +-
 libminifi/src/core/yaml/YamlConfiguration.cpp   |  14 +-
 libminifi/src/processors/InvokeHTTP.cpp |  16 +-
 .../HttpConfigurationListenerTest.cpp   | 144 
 22 files changed, 938 insertions(+), 85 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/nifi-minifi-cpp/blob/1a1716dc/README.md
--
diff --git a/README.md b/README.md
index acaf8dd..0fb25f3 100644
--- a/README.md
+++ b/README.md
@@ -323,6 +323,24 @@ Additionally, users can utilize the MiNiFi Toolkit 
Converter (version 0.0.1 - sc
   host: localhost
   port uuid: 471deef6-2a6e-4a7d-912a-81cc17e3a204
   batch size: 100
+
+### Http Configuration Listener
+
+Http Configuration Listener will pull flow file configuration from the 
remote command control server, 
+validate and apply the new flow configuration
+
+in minifi.properties 
+
+nifi.configuration.listener.type=http
+nifi.configuration.listener.http.url=https://localhost:8080
+nifi.configuration.listener.pull.interval=1 sec 
+nifi.configuration.listener.client.ca.certificate=./conf/nifi-cert.pem
+
+if you want to enable client certificate
+nifi.configuration.listener.need.ClientAuth=true
+nifi.configuration.listener.client.certificate=./conf/client.pem
+nifi.configuration.listener.client.private.key=./conf/client.key
+nifi.configuration.listener.client.pass.phrase=./conf/password
   
 ### Controller Services
  If you need to reference a controller service in your config.yml file, use 
the following template. In the example, below, ControllerServiceClass is the 
name of the class defining the controller Service. ControllerService1 

http://git-wip-us.apache.org/repos/asf/nifi-minifi-cpp/blob/1a1716dc/cmake/BuildTests.cmake
--
diff --git a/cmake/BuildTests.cmake b/cmake/BuildTests.cmake
index bad01d3..aedae10 100644
--- a/cmake/BuildTests.cmake
+++ b/cmake/BuildTests.cmake
@@ -36,6 +36,7 @@ function(createTests testName)
 target_include_directories(${testName} PRIVATE BEFORE 
"thirdparty/spdlog-0.13.0/include")
 target_include_directories(${testName} PRIVATE BEFORE 
"thirdparty/yaml-cpp-yaml-cpp-0.5.3/include")
 target_include_directories(${testName} PRIVATE BEFORE 
"thirdparty/jsoncpp/include")
+target_include_directories(${testName} PRIVATE BEFORE 
"thirdparty/civetweb-1.9.1/include")
 target_include_directories(${testName} PRIVATE BEFORE 
${LEVELDB_INCLUDE_DIRS})
 target_include_directories(${testName} PRIVATE BEFORE "include")
 target_include_directories(${testName} PRIVATE BEFORE "libminifi/include/")
@@ -87,6 +88,8 @@ add_test(NAME ControllerServiceIntegrationTests COMMAND 
ControllerServiceIntegra
 
 add_test(NAME HttpGetIntegrationTest COMMAND HttpGetIntegrationTest 
"${TEST_RESOURCES}/TestHTTPGet.yml"

[nifi-minifi-cpp] Git Push Summary

2017-06-07 Thread phrocker 
Repository: nifi-minifi-cpp
Updated Branches:
  refs/remotes/apache/master [created] 77a20dbe3

[jira] [Commented] (MINIFI-262) Configuration Listener in C++ client to support C2 operations

2017-06-07 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/MINIFI-262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16041244#comment-16041244
 ] 

ASF GitHub Bot commented on MINIFI-262:
---

Github user phrocker commented on the issue:

https://github.com/apache/nifi-minifi-cpp/pull/112
  
+1 approved per evaluation in #107 


> Configuration Listener in C++ client to support C2 operations
> -
>
> Key: MINIFI-262
> URL: https://issues.apache.org/jira/browse/MINIFI-262
> Project: Apache NiFi MiNiFi
>  Issue Type: New Feature
>  Components: C++, Command and Control
>Reporter: marco polo
>Assignee: bqiu
>
> Create the concept of configuration listener support a variety of 
> implementations that will restart operations for the MiNiFi C++ agent. 
> The listener will implement a listener that will restart the MiNiFi process. 
> MiNiFi-263 will include the work to implement the configuration listener 
> interface. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (MINIFI-262) Configuration Listener in C++ client to support C2 operations

2017-06-07 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/MINIFI-262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16041183#comment-16041183
 ] 

ASF GitHub Bot commented on MINIFI-262:
---

Github user benqiu2016 closed the pull request at:

https://github.com/apache/nifi-minifi-cpp/pull/107


> Configuration Listener in C++ client to support C2 operations
> -
>
> Key: MINIFI-262
> URL: https://issues.apache.org/jira/browse/MINIFI-262
> Project: Apache NiFi MiNiFi
>  Issue Type: New Feature
>  Components: C++, Command and Control
>Reporter: marco polo
>Assignee: bqiu
>
> Create the concept of configuration listener support a variety of 
> implementations that will restart operations for the MiNiFi C++ agent. 
> The listener will implement a listener that will restart the MiNiFi process. 
> MiNiFi-263 will include the work to implement the configuration listener 
> interface. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (MINIFI-262) Configuration Listener in C++ client to support C2 operations

2017-06-07 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/MINIFI-262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16041182#comment-16041182
 ] 

ASF GitHub Bot commented on MINIFI-262:
---

Github user benqiu2016 commented on the issue:

https://github.com/apache/nifi-minifi-cpp/pull/107
  
@phrocker 
https://github.com/apache/nifi-minifi-cpp/pull/112 for a single commit
close this one for now



> Configuration Listener in C++ client to support C2 operations
> -
>
> Key: MINIFI-262
> URL: https://issues.apache.org/jira/browse/MINIFI-262
> Project: Apache NiFi MiNiFi
>  Issue Type: New Feature
>  Components: C++, Command and Control
>Reporter: marco polo
>Assignee: bqiu
>
> Create the concept of configuration listener support a variety of 
> implementations that will restart operations for the MiNiFi C++ agent. 
> The listener will implement a listener that will restart the MiNiFi process. 
> MiNiFi-263 will include the work to implement the configuration listener 
> interface. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (MINIFI-262) Configuration Listener in C++ client to support C2 operations

2017-06-07 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/MINIFI-262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16041178#comment-16041178
 ] 

ASF GitHub Bot commented on MINIFI-262:
---

GitHub user benqiu2016 opened a pull request:

https://github.com/apache/nifi-minifi-cpp/pull/112

MINIFI-262: Configuration Listener

Thank you for submitting a contribution to Apache NiFi - MiNiFi C++.

In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:

### For all changes:
- [ ] Is there a JIRA ticket associated with this PR? Is it referenced
 in the commit message?

- [ ] Does your PR title start with MINIFI- where  is the JIRA 
number you are trying to resolve? Pay particular attention to the hyphen "-" 
character.

- [ ] Has your PR been rebased against the latest commit within the target 
branch (typically master)?

- [ ] Is your initial contribution a single, squashed commit?

### For code changes:
- [ ] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] If applicable, have you updated the LICENSE file?
- [ ] If applicable, have you updated the NOTICE file?

### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in 
which it is rendered?

### Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/benqiu2016/nifi-minifi-cpp 
configuration_listener_merge

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi-minifi-cpp/pull/112.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #112


commit 1924ce57bf42dc4c88b924aa05f14524a67aa4c4
Author: Bin Qiu 
Date:   2017-06-07T16:26:11Z

MINIFI-262: Configuration Listener




> Configuration Listener in C++ client to support C2 operations
> -
>
> Key: MINIFI-262
> URL: https://issues.apache.org/jira/browse/MINIFI-262
> Project: Apache NiFi MiNiFi
>  Issue Type: New Feature
>  Components: C++, Command and Control
>Reporter: marco polo
>Assignee: bqiu
>
> Create the concept of configuration listener support a variety of 
> implementations that will restart operations for the MiNiFi C++ agent. 
> The listener will implement a listener that will restart the MiNiFi process. 
> MiNiFi-263 will include the work to implement the configuration listener 
> interface. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

nifi git commit: NIFI-4029: Allow null Avro default values in HortonworksSchemaRegistry

2017-06-07 Thread bbende 
Repository: nifi
Updated Branches:
  refs/heads/master c86190c51 -> 45e035686


NIFI-4029: Allow null Avro default values in HortonworksSchemaRegistry

This closes #1894.

Signed-off-by: Bryan Bende 


Project: http://git-wip-us.apache.org/repos/asf/nifi/repo
Commit: http://git-wip-us.apache.org/repos/asf/nifi/commit/45e03568
Tree: http://git-wip-us.apache.org/repos/asf/nifi/tree/45e03568
Diff: http://git-wip-us.apache.org/repos/asf/nifi/diff/45e03568

Branch: refs/heads/master
Commit: 45e035686f0a51c29629f6f6dfa1d26496e21997
Parents: c86190c
Author: Steve Champagne 
Authored: Wed Jun 7 12:36:28 2017 +
Committer: Bryan Bende 
Committed: Wed Jun 7 12:03:53 2017 -0400

--
 .../java/org/apache/nifi/avro/AvroTypeUtil.java |   6 +-
 .../nifi-hwx-schema-registry-service/pom.xml|   4 +
 .../hortonworks/HortonworksSchemaRegistry.java  | 122 +--
 3 files changed, 13 insertions(+), 119 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/nifi/blob/45e03568/nifi-nar-bundles/nifi-extension-utils/nifi-record-utils/nifi-avro-record-utils/src/main/java/org/apache/nifi/avro/AvroTypeUtil.java
--
diff --git 
a/nifi-nar-bundles/nifi-extension-utils/nifi-record-utils/nifi-avro-record-utils/src/main/java/org/apache/nifi/avro/AvroTypeUtil.java
 
b/nifi-nar-bundles/nifi-extension-utils/nifi-record-utils/nifi-avro-record-utils/src/main/java/org/apache/nifi/avro/AvroTypeUtil.java
index 52c55fc..1417e67 100644
--- 
a/nifi-nar-bundles/nifi-extension-utils/nifi-record-utils/nifi-avro-record-utils/src/main/java/org/apache/nifi/avro/AvroTypeUtil.java
+++ 
b/nifi-nar-bundles/nifi-extension-utils/nifi-record-utils/nifi-avro-record-utils/src/main/java/org/apache/nifi/avro/AvroTypeUtil.java
@@ -152,7 +152,11 @@ public class AvroTypeUtil {
 final Schema fieldSchema = field.schema();
 final DataType fieldType = determineDataType(fieldSchema);
 
-recordFields.add(new RecordField(fieldName, fieldType, 
field.defaultVal(), field.aliases()));
+if (field.defaultVal() == JsonProperties.NULL_VALUE) {
+recordFields.add(new RecordField(fieldName, fieldType, 
field.aliases()));
+} else {
+recordFields.add(new RecordField(fieldName, fieldType, 
field.defaultVal(), field.aliases()));
+}
 }
 
 final RecordSchema recordSchema = new 
SimpleRecordSchema(recordFields, avroSchema.toString(), AVRO_SCHEMA_FORMAT, 
SchemaIdentifier.EMPTY);

http://git-wip-us.apache.org/repos/asf/nifi/blob/45e03568/nifi-nar-bundles/nifi-standard-services/nifi-hwx-schema-registry-bundle/nifi-hwx-schema-registry-service/pom.xml
--
diff --git 
a/nifi-nar-bundles/nifi-standard-services/nifi-hwx-schema-registry-bundle/nifi-hwx-schema-registry-service/pom.xml
 
b/nifi-nar-bundles/nifi-standard-services/nifi-hwx-schema-registry-bundle/nifi-hwx-schema-registry-service/pom.xml
index 79dbc84..38e175c 100644
--- 
a/nifi-nar-bundles/nifi-standard-services/nifi-hwx-schema-registry-bundle/nifi-hwx-schema-registry-service/pom.xml
+++ 
b/nifi-nar-bundles/nifi-standard-services/nifi-hwx-schema-registry-bundle/nifi-hwx-schema-registry-service/pom.xml
@@ -43,6 +43,10 @@ limitations under the License.

 
 org.apache.nifi
+nifi-avro-record-utils
+
+
+org.apache.nifi
 nifi-schema-registry-service-api
 
 

http://git-wip-us.apache.org/repos/asf/nifi/blob/45e03568/nifi-nar-bundles/nifi-standard-services/nifi-hwx-schema-registry-bundle/nifi-hwx-schema-registry-service/src/main/java/org/apache/nifi/schemaregistry/hortonworks/HortonworksSchemaRegistry.java
--
diff --git 
a/nifi-nar-bundles/nifi-standard-services/nifi-hwx-schema-registry-bundle/nifi-hwx-schema-registry-service/src/main/java/org/apache/nifi/schemaregistry/hortonworks/HortonworksSchemaRegistry.java
 
b/nifi-nar-bundles/nifi-standard-services/nifi-hwx-schema-registry-bundle/nifi-hwx-schema-registry-service/src/main/java/org/apache/nifi/schemaregistry/hortonworks/HortonworksSchemaRegistry.java
index d2289a2..3027e5f 100644
--- 
a/nifi-nar-bundles/nifi-standard-services/nifi-hwx-schema-registry-bundle/nifi-hwx-schema-registry-service/src/main/java/org/apache/nifi/schemaregistry/hortonworks/HortonworksSchemaRegistry.java
+++ 
b/nifi-nar-bundles/nifi-standard-services/nifi-hwx-schema-registry-bundle/nifi-hwx-schema-registry-service/src/main/java/org/apache/nifi/schemaregistry/hortonworks/HortonworksSchemaRegistry.java
@@ -25,12

[jira] [Commented] (MINIFI-262) Configuration Listener in C++ client to support C2 operations

2017-06-07 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/MINIFI-262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16040755#comment-16040755
 ] 

ASF GitHub Bot commented on MINIFI-262:
---

Github user phrocker commented on the issue:

https://github.com/apache/nifi-minifi-cpp/pull/107
  
+1 

@benqiu2016 Thanks for the update. Would you mind squashing your commits 
with a message you would like to include before we merge it?


> Configuration Listener in C++ client to support C2 operations
> -
>
> Key: MINIFI-262
> URL: https://issues.apache.org/jira/browse/MINIFI-262
> Project: Apache NiFi MiNiFi
>  Issue Type: New Feature
>  Components: C++, Command and Control
>Reporter: marco polo
>Assignee: bqiu
>
> Create the concept of configuration listener support a variety of 
> implementations that will restart operations for the MiNiFi C++ agent. 
> The listener will implement a listener that will restart the MiNiFi process. 
> MiNiFi-263 will include the work to implement the configuration listener 
> interface. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)