Re: [NIFI-CPP] Problem with UpdateAttribute processor

[Iyán Méndez Veiga]

I attach here another log file with TRACE level instead of INFO. Hope it 
helps.

Best regards,
Iyán

-- 
Iyán Méndez Veiga | Physicist
GPG: 0x422E3694311E5AC1
Web: https://iyanmv.com

♫♪.ılılıll|̲̅̅●̲̅̅|̲̅̅=̲̅̅|̲̅̅●̲̅̅|llılılı.♫♪[2018-04-19 01:36:12.732] [org::apache::nifi::minifi::core::logging::LoggerConfiguration] [debug] org::apache::nifi::minifi::utils::IdGenerator logger got sinks from namespace root and level trace from namespace org::apache::nifi::minifi
[2018-04-19 01:36:12.732] [org::apache::nifi::minifi::core::logging::LoggerConfiguration] [debug] org::apache::nifi::minifi::core::FlowFile logger got sinks from namespace root and level trace from namespace org::apache::nifi::minifi
[2018-04-19 01:36:12.732] [org::apache::nifi::minifi::core::logging::LoggerConfiguration] [debug] org::apache::nifi::minifi::provenance::ProvenanceEventRecord logger got sinks from namespace root and level trace from namespace org::apache::nifi::minifi
[2018-04-19 01:36:12.732] [org::apache::nifi::minifi::core::logging::LoggerConfiguration] [debug] org::apache::nifi::minifi::FlowFileRecord logger got sinks from namespace root and level trace from namespace org::apache::nifi::minifi
[2018-04-19 01:36:12.732] [org::apache::nifi::minifi::core::logging::LoggerConfiguration] [debug] main logger got sinks from namespace root and level info from namespace root
[2018-04-19 01:36:12.732] [org::apache::nifi::minifi::core::logging::LoggerConfiguration] [debug] org::apache::nifi::minifi::Properties logger got sinks from namespace root and level trace from namespace org::apache::nifi::minifi
[2018-04-19 01:36:12.732] [org::apache::nifi::minifi::core::logging::LoggerConfiguration] [debug] Set following pattern on loggers: [%Y-%m-%d %H:%M:%S.%e] [%n] [%l] %v
[2018-04-19 01:36:12.732] [org::apache::nifi::minifi::Properties] [info] Using configuration file located at /home/minifi/nifi-minifi-cpp-0.4.0/conf/minifi-uid.properties
[2018-04-19 01:36:12.732] [org::apache::nifi::minifi::utils::IdGenerator] [debug] Using uuid_generate_time implementation for uids.
[2018-04-19 01:36:12.732] [main] [info] MINIFI_HOME=/home/minifi/nifi-minifi-cpp-0.4.0
[2018-04-19 01:36:12.732] [org::apache::nifi::minifi::Properties] [info] Using configuration file located at /home/minifi/nifi-minifi-cpp-0.4.0/conf/minifi.properties
[2018-04-19 01:36:12.732] [org::apache::nifi::minifi::core::logging::LoggerConfiguration] [debug] org::apache::nifi::minifi::core::Connectable logger got sinks from namespace root and level trace from namespace org::apache::nifi::minifi
[2018-04-19 01:36:12.732] [org::apache::nifi::minifi::core::logging::LoggerConfiguration] [debug] org::apache::nifi::minifi::core::Repository logger got sinks from namespace root and level trace from namespace org::apache::nifi::minifi
[2018-04-19 01:36:12.732] [org::apache::nifi::minifi::core::logging::LoggerConfiguration] [debug] org::apache::nifi::minifi::provenance::ProvenanceRepository logger got sinks from namespace root and level trace from namespace org::apache::nifi::minifi
[2018-04-19 01:36:12.732] [org::apache::nifi::minifi::provenance::ProvenanceRepository] [debug] NiFi Provenance Repository Directory ./provenance_repository
[2018-04-19 01:36:12.732] [org::apache::nifi::minifi::provenance::ProvenanceRepository] [debug] NiFi Provenance Max Partition Bytes 1048576
[2018-04-19 01:36:12.732] [org::apache::nifi::minifi::provenance::ProvenanceRepository] [debug] NiFi Provenance Max Storage Time: [6] ms
[2018-04-19 01:36:13.524] [org::apache::nifi::minifi::provenance::ProvenanceRepository] [debug] NiFi Provenance Repository database open ./provenance_repository success
[2018-04-19 01:36:13.524] [org::apache::nifi::minifi::core::logging::LoggerConfiguration] [debug] org::apache::nifi::minifi::core::repository::FlowFileRepository logger got sinks from namespace root and level trace from namespace org::apache::nifi::minifi
[2018-04-19 01:36:13.524] [org::apache::nifi::minifi::core::repository::FlowFileRepository] [debug] NiFi FlowFile Repository Directory ./flowfile_repository
[2018-04-19 01:36:13.524] [org::apache::nifi::minifi::core::repository::FlowFileRepository] [debug] NiFi FlowFile Max Partition Bytes 10485760
[2018-04-19 01:36:13.524] [org::apache::nifi::minifi::core::repository::FlowFileRepository] [debug] NiFi FlowFile Max Storage Time: [60] ms
[2018-04-19 01:36:13.650] [org::apache::nifi::minifi::core::repository::FlowFileRepository] [debug] NiFi FlowFile Repository database open ./flowfile_repository success
[2018-04-19 01:36:13.650] [org::apache::nifi::minifi::core::logging::LoggerConfiguration] [debug] org::apache::nifi::minifi::core::repository::FileSystemRepository logger got sinks from namespace root and level trace from namespace org::apache::nifi::minifi
[2018-04-19 01:36:13.650] [org::apache::nifi::minifi::core::logging::LoggerConfiguration] [debug] org::apache::nifi::minifi::core::FlowConfiguration logger got sinks from namespace root and level trace from namespace 

[NIFI-CPP] Problem with UpdateAttribute processor

[Iyán Méndez Veiga]

Hi,

After few days trying NiFi and MiNiFi C++ I (finally) have been able to 
compile minifi-cpp in a rasp3 with tensorflow and usb camera support.

Now I am having a silly issue. I cannot use UpdateAttribute processor 
successfully. This is the error I get:

[org::apache::nifi::minifi::core::FlowConfiguration] [error] No Processor 
defined for UpdateAttribute

I attached the relevant files in the email.

What I am trying to do is replicate what Andy explains in this article:
https://community.hortonworks.com/articles/174520/minifi-c-iot-cat-sensor.html

Of course, mini is a silly example. But using this config file:
https://gist.github.com/achristianson/1dea217e5fcbc88b87e526d919dad2c0#file-config-yml-L93

I get the same error in the step where tf.type attributes are added to the 
DataFlows.

Hope you can help me.

Best regards,
Iyán

-- 
Iyán Méndez Veiga | Physicist
GPG: 0x422E3694311E5AC1
Web: https://iyanmv.com

♫♪.ılılıll|̲̅̅●̲̅̅|̲̅̅=̲̅̅|̲̅̅●̲̅̅|llılılı.♫♪

config.yml
Description: application/yaml
[2018-04-19 01:18:32.787] [org::apache::nifi::minifi::Properties] [info] Using configuration file located at /home/minifi/nifi-minifi-cpp-0.4.0/conf/minifi-uid.properties
[2018-04-19 01:18:32.787] [main] [info] MINIFI_HOME=/home/minifi/nifi-minifi-cpp-0.4.0
[2018-04-19 01:18:32.787] [org::apache::nifi::minifi::Properties] [info] Using configuration file located at /home/minifi/nifi-minifi-cpp-0.4.0/conf/minifi.properties
[2018-04-19 01:18:33.101] [org::apache::nifi::minifi::FlowController] [info] FlowController NiFi Configuration file /home/minifi/nifi-minifi-cpp-0.4.0/conf/config.yml
[2018-04-19 01:18:33.101] [org::apache::nifi::minifi::FlowController] [info] FlowController content directory /home/minifi/nifi-minifi-cpp-0.4.0/content_repository
[2018-04-19 01:18:33.101] [main] [info] Loading FlowController
[2018-04-19 01:18:33.101] [org::apache::nifi::minifi::FlowController] [info] Load Flow Controller from file /home/minifi/nifi-minifi-cpp-0.4.0/conf/config.yml
[2018-04-19 01:18:33.102] [org::apache::nifi::minifi::core::FlowConfiguration] [error] No Processor defined for UpdateAttribute
[2018-04-19 01:18:33.102] [org::apache::nifi::minifi::core::YamlConfiguration] [error] Could not create a processor add_attribute with id d008c4fe-435e-11e8-a30e-3ca82a2045cc
[2018-04-19 01:18:33.102] [main] [error] Failed to load configuration due to exception: Could not create processor add_attribute
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements.  See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License.  You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

#More verbose pattern by default
#Format details at https://github.com/gabime/spdlog/wiki/3.-Custom-formatting
spdlog.pattern=[%Y-%m-%d %H:%M:%S.%e] [%n] [%l] %v

#Old format
#spdlog.pattern=[%Y-%m-%d %H:%M:%S.%e] [minifi log] [%l] %v

#More compact format example
#spdlog.pattern=[%D %H:%M:%S.%e] [%L] %v

appender.rolling=rollingappender
appender.rolling.file_name=minifi-app.log
appender.rolling.max_files=3
appender.rolling.max_file_size=5242880

#Other possible appenders
#appender.stdout=stdout
#appender.stderr=stderr
#appender.null=null

logger.root=INFO,rolling

#Logging configurable by namespace
logger.org::apache::nifi::minifi=INFO,rolling

#Logging configurable by class fully qualified name
#logger.org::apache::nifi::minifi::core::logging::LoggerConfiguration=DEBUG
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements.  See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License.  You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

# Implementation for uid generation.
# Valid values:
# time - use uuid_generate_time
# random - use uuid_generate_random
# uuid_default - use uuid_generate (will attempt to use uuid_generate_random 
and fall back to 

RE: Re: Use AWS:KMS SSE with PutS3Object?

[Simon Tack]

James,

OK, thank you very much for the reply and the Jira link.

Simon

From: James Wing [mailto:jvw...@gmail.com]
Sent: Wednesday, April 18, 2018 2:30 PM
To: users@nifi.apache.org
Subject: [External] Re: Use AWS:KMS SSE with PutS3Object?

Simon,

I'm afraid NiFi does not yet support server-side encryption with a specific KMS 
key ID, only the more basic SSE.  This has certainly come up before, you are 
definitely not alone in wishing for this feature.  There is a JIRA:

NIFI-4256 Add support for all AWS S3 Encryption Options
https://issues.apache.org/jira/browse/NIFI-4256


On Wed, Apr 18, 2018 at 10:37 AM, Simon Tack 
> wrote:
Hi.

I am trying to use the PutS3Object Processor in NiFi 1.0.0 to write a file to 
S3.  I am getting a 403 Access Denied error.  My bucket requires SSE encryption 
with a certain key.  I would expect that I need to specify the ARN to that key 
for PutS3Object to work, but I can’t seem to tell how or where I would specify 
it.  Is what I am trying to do possible with the NiFi 1.0.0 PutS3Object 
processor?  If not, is it possible with any newer version of NiFi?

I can upload files to the S3 bucket fine using the AWS CLI tools, but I have to 
specify --sse as aws:kms and -sse-kms-key-id, i.e.:

aws s3 cp   --sse aws:kms --sse-kms-key-id 

Thanks,
Simon

Re: Use AWS:KMS SSE with PutS3Object?

[James Wing]

Simon,

I'm afraid NiFi does not yet support server-side encryption with a specific
KMS key ID, only the more basic SSE.  This has certainly come up before,
you are definitely not alone in wishing for this feature.  There is a JIRA:

NIFI-4256 Add support for all AWS S3 Encryption Options
https://issues.apache.org/jira/browse/NIFI-4256


On Wed, Apr 18, 2018 at 10:37 AM, Simon Tack 
wrote:

> Hi.
>
>
>
> I am trying to use the PutS3Object Processor in NiFi 1.0.0 to write a file
> to S3.  I am getting a 403 Access Denied error.  My bucket requires SSE
> encryption with a certain key.  I would expect that I need to specify the
> ARN to that key for PutS3Object to work, but I can’t seem to tell how or
> where I would specify it.  Is what I am trying to do possible with the NiFi
> 1.0.0 PutS3Object processor?  If not, is it possible with any newer version
> of NiFi?
>
>
>
> I can upload files to the S3 bucket fine using the AWS CLI tools, but I
> have to specify --sse as aws:kms and -sse-kms-key-id, i.e.:
>
>
>
> aws s3 cp   --sse aws:kms --sse-kms-key-id 
>
>
>
> Thanks,
>
> Simon
>

Use AWS:KMS SSE with PutS3Object?

[Simon Tack]

Hi.

I am trying to use the PutS3Object Processor in NiFi 1.0.0 to write a file to 
S3.  I am getting a 403 Access Denied error.  My bucket requires SSE encryption 
with a certain key.  I would expect that I need to specify the ARN to that key 
for PutS3Object to work, but I can't seem to tell how or where I would specify 
it.  Is what I am trying to do possible with the NiFi 1.0.0 PutS3Object 
processor?  If not, is it possible with any newer version of NiFi?

I can upload files to the S3 bucket fine using the AWS CLI tools, but I have to 
specify --sse as aws:kms and -sse-kms-key-id, i.e.:

aws s3 cp   --sse aws:kms --sse-kms-key-id 

Thanks,
Simon

Re: Nifi driven by external configuration

[Charlie Meyer]

To solve this problem, I have written code that sets configuration
properties in the variable registry on each nifi instance, then each flow
that is provisioned has its configuration reference those variables. This
way we are able to deploy nifi instances for different environments, etc
but maintain common logic.

On Wed, Apr 18, 2018 at 6:39 AM, Carlos Manuel Fernandes (DSI) <
carlos.antonio.fernan...@cgd.pt> wrote:

> Nifi simple principles of flow, attributes of flow ,  processes and
>  properties of  processors are very powerful and permit a lot of great
> combinations.
>
>
>
> The problem i see,  is the tendency to repeat flows to make the  same
> thing, because some properties are hardwire (properties without expression
> language)  and doesn’t exist a processor to read External configuration to
> map to attributes , which can be properties  in processors.
>
>
>
> In my Use Cases  I try to address this problem making services with this
> skeleton :
>
> HandleHttpRequest (receive some Key)  -> GetExternalConfig (Using  Key) ->
> Others Nifi Procs with  properties based on attributes read from external
> config -> HandleHttpResponse
>
> GetExternalConfig is a custom script processor, which transform a sql
> Query (all the columns of first row)  in attributes of the current Flow.
>
>
>
> In this arrangement , in the zone of “Other Nifi Procs” ,  if properties
> of processors don’t permit expression language is not possible to use the
> configuration already read, which is annoying.
>
> Till now, I had problems with SplitText (Line Split Count*) *and
> CSVRecordSetWriter (value separator, time format, etc), but the problem is
> general.
>
>
>
> If this make sense for more people in the community , we can think to:
>
> 1-  Create a Nifi Processor to read external Configuration from
> relational Databases  to Attributes. (probably make sense, read from
> another non relational sources)
>
> 2-  By default all the processors properties must admit expression
> language (probably some technical ones don’t make sense)
>
>
>
> If not, I continue  with my custom GetExternalConfig and blaming when I
> find a property without expression language J
>
>
>
> Thanks
>
>
>
> Carlos
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>

Nifi driven by external configuration

[Carlos Manuel Fernandes (DSI)]

Nifi simple principles of flow, attributes of flow ,  processes and  properties 
of  processors are very powerful and permit a lot of great combinations.

The problem i see,  is the tendency to repeat flows to make the  same thing, 
because some properties are hardwire (properties without expression language)  
and doesn't exist a processor to read External configuration to map to 
attributes , which can be properties  in processors.

In my Use Cases  I try to address this problem making services with this 
skeleton :
HandleHttpRequest (receive some Key)  -> GetExternalConfig (Using  Key) -> 
Others Nifi Procs with  properties based on attributes read from external 
config -> HandleHttpResponse
GetExternalConfig is a custom script processor, which transform a sql Query 
(all the columns of first row)  in attributes of the current Flow.

In this arrangement , in the zone of "Other Nifi Procs" ,  if properties of 
processors don't permit expression language is not possible to use the 
configuration already read, which is annoying.
Till now, I had problems with SplitText (Line Split Count) and 
CSVRecordSetWriter (value separator, time format, etc), but the problem is 
general.

If this make sense for more people in the community , we can think to:

1-  Create a Nifi Processor to read external Configuration from relational 
Databases  to Attributes. (probably make sense, read from another non 
relational sources)

2-  By default all the processors properties must admit expression language 
(probably some technical ones don't make sense)

If not, I continue  with my custom GetExternalConfig and blaming when I find a 
property without expression language :)

Thanks

Carlos

Re: Clustering Questions

[Pierre Villard]

Hi Jon,

Just as a note for your unrelated question:
I opened NIFI-4026 few months ago but didn't have time to work on it so far.

[1] https://issues.apache.org/jira/browse/NIFI-4026



2018-04-17 20:34 GMT+02:00 Jon Logan :

> Thanks Joe, just a few follow-up questions:
>
> re:durability -- is this something that people have just been accepting as
> a risk and hoping for the best? Or is this something people build their
> applications around -- ie. using durability outside of the Nifi system
> boundary and push it into a database, etc?
>
> re:heterogenous -- you can join nodes of differing hardware specs, but it
> seems like you will end up causing your lighter-weight nodes to explode as
> there's no way to configure how many tasks and how much to have processing
> "in-flight" on the node different than the other nodes? ie. if I know my
> large nodes can handle 3 of a cpu-intensive task, that's going to cause
> issues for smaller nodes. This is an even bigger problem for differing
> memory sizes.
>
> And an unrelated question to the previous -- is there a way to skew or
> influence how a RPG distributes its tasks? Say, you wanted to do a group-by
> type distribution?
>
>
> Thanks again!
> Jon
>
>
> On Fri, Apr 13, 2018 at 2:17 PM, Joe Witt  wrote:
>
>> Jon,
>>
>> Node Failure:
>>  You have to care about two things generally speaking.  First is the
>> flow execution and second is data in-flight
>>  For flow execution nifi clustering will take care of re-assigning the
>> primary node and cluster coordinator as needed.
>>  For data we do not at present offer distributed data durability.  The
>> current model is predicated on using reliable storage such as RAID,
>> EBS, etc..
>>   There is a very clear and awesome looking K8S based path though that
>> will make this work really nicely with persistent volumes and elastic
>> scaling.  No clear timeline but discussions/JIRA/contributions i hope
>> to start or participate in soon.
>>
>> How scalable is the NiFi scaling model:
>>   Usually NiFi clusters are a few nodes to maybe 10-20 or so.  Some
>> have been larger but generally if you're needing that much flow
>> management then often it makes more sense to have clusters dedicated
>> along various domains of expertise anyway.  So say 3-10 nodes with
>> each handling 100,000 events per second around say 100MB per second
>> (conservatively) and you can see why a single fairly small cluster can
>> handle pretty massive volumes.
>>
>> RPGs feeding back:
>> - This caused issues previously but I believe in recent releases has
>> improved significantly.
>>
>> UI Actions Causing issues:
>> There have been reports similar to this especially for some of the
>> really massive flows we've seen in terms of number of components and
>> concurrent users.  These JIRAs when sorted will help a lot [1], [2],
>> [3].
>>
>> Heterogenous cluster nodes:
>> - This should work quite well actually and is a major reason why NiFi
>> and the S2S protocol supports/honors backpressure.  Nodes that can
>> take on more work take on more work and nodes that cannot pushback.
>> You also want to ensure you're using good and scalable protocols to
>> source data into the cluster.  If you find you're using a lot of
>> protocols requiring you to make many data sourcing steps run 'primary
>> node only' then that will require that primary node to do more work
>> than others and I have seen uneven behavior in such cases.  Yes, you
>> can then route using S2S/RPG which we recommend but still...try to
>> design away from 'primary node only' when possible.
>>
>>
>> Thanks
>> Joe
>>
>>
>> [1] https://issues.apache.org/jira/browse/NIFI-950
>> [2] https://issues.apache.org/jira/browse/NIFI-5064
>> [3] https://issues.apache.org/jira/browse/NIFI-5066
>>
>> On Fri, Apr 13, 2018 at 5:49 PM, Jon Logan  wrote:
>> > All, I had a few general questions regarding Clustering, and was
>> looking for
>> > any sort of advice or best-practices information --
>> >
>> > - documentation discusses failure handling primarily from a NiFi crash
>> > scenario, but I don't recall seeing any information on entire
>> node-failure
>> > scenarios. Is there a way that this is supposed to be handled?
>> > - at what point should we expect pain in scaling? I am particularly
>> > concerned about the all-to-all relationship that seems to exist if you
>> > connect a cluster RPG to itself, as all nodes need to distribute all
>> data to
>> > all other nodes. We have been also been having some issues when things
>> are
>> > not as responsive as NiFi would like -- namely, the UI seems to get very
>> > upset and crash
>> > - do UI actions (incl read-only) require delegation to all nodes
>> underneath?
>> > I suspect this is the case as otherwise you wouldn't be able to
>> determine
>> > queue sizes?
>> > - is there a way to have a cluster with heterogeneous node sizes?
>> >
>> >
>> > Thanks in advance!
>>
>
>