Aldrin,
To simplify it, the situation is analogous to a deployment of temperature
sensors. Each sensor has a unique ID that is assigned by us at deployment
time and each sensor periodically adds a new row to a database table that
is stored on the sensor. Each sensor uses the same database schema so if
you combined all the rows you couldn't tell which rows originated from
which sensor. In NiFi, I need to do different things based on where the
data originated and I need to associate the sensor's ID with its data.
(Such as inserting the data into DynamoDB with the sensor ID as the Hash
key and a timestamp as the Range key.) The goal is to use the same MiNiFi
configuration for all devices.
I can easily use the ExecuteSQL processor to grab the new rows. But I need
some way to attach an attribute to the data that identifies where it
originated. That was what led to the initial question in this thread. The
Variable Registry along with the UpdateAttribute processor appears to
satisfy that need cleaner than a custom processor.
I hope that explains the situation a bit!
Thanks,
Jeff
On Tue, Apr 25, 2017 at 11:17 AM, Aldrin Piri <aldrinp...@gmail.com> wrote:
> Jeff,
>
> Could you expand upon what a device id is in your case? Something
> intrinsic to the device? The agent? Are these generated and assigned
> during provisioning? How are you making use of these when the data
> arrives at its desired destination?
>
> What you are expressing is certainly a common need. Would welcome any
> perspective on what your deployment looks like such that we can frame uses
> people are rolling out to guide assumptions that get made during our
> development and design processes.
>
> Thanks for diving in and exploring!
> --Aldrin
>
>
> On Tue, Apr 25, 2017 at 11:05 AM, Andre <andre-li...@fucs.org> wrote:
>
>> Jeff,
>>
>> That would be next suggestion. :-)
>>
>> Cheers
>>
>> On Wed, Apr 26, 2017 at 1:04 AM, Jeff Zemerick <jzemer...@apache.org>
>> wrote:
>>
>>> It is possible. I will take a look to see if the hostname is sufficient
>>> for the device ID.
>>>
>>> I just learned about the Variable Registry. It seems if I use the
>>> Variable Registry to store the device ID it would be available to the
>>> UpdateAttribute processor. Is that correct?
>>>
>>> Thanks,
>>> Jeff
>>>
>>>
>>> On Tue, Apr 25, 2017 at 10:48 AM, Andre <andre-li...@fucs.org> wrote:
>>>
>>>> Jeff,
>>>>
>>>> Would if be feasible for you use UpdateAttribute (which I believe is
>>>> part of MiNiFi core processors) and use the ${hostname(true)} Expression
>>>> language function?
>>>>
>>>> More about it can be found here:
>>>>
>>>> https://nifi.apache.org/docs/nifi-docs/html/expression-langu
>>>> age-guide.html#hostname
>>>>
>>>> Cheers
>>>>
>>>> On Wed, Apr 26, 2017 at 12:39 AM, Jeff Zemerick <jzemer...@apache.org>
>>>> wrote:
>>>>
>>>>> When processing data in NiFi that was received via MiNiFi edge devices
>>>>> I need to be able to identify the source of the data. All of the data on
>>>>> the edge devices will be pulled from a database and will not contain any
>>>>> data that self-identifies the source. My attempt to solve this was to
>>>>> write
>>>>> a processor that reads a configuration file on the edge device to get its
>>>>> device ID and put that ID as an attribute in the flowfile. This appears to
>>>>> work, but, I was wondering if there is a more recommended approach?
>>>>>
>>>>> Thanks,
>>>>> Jeff
>>>>>
>>>>
>>>>
>>>
>>
>
