Re: using avro flume source in nifi

[Joe Witt]

Chris,

I dont believe there are many in the community here familiar enough
with Flume so that might make getting help here a bit tougher.  Are
you wanting NiFi to act as a TCP socket listener on a given port
receiving Avro objects?

Thanks

On Thu, Aug 3, 2017 at 11:22 PM, Chris Herssens
 wrote:
> Hello All,
>
> I want to configure a flume source processor using avro as flume source.
> When I start the ExecuteFlumeSource, I don't get any errors, but I can't
> connect to the source port.
>
> telnet localhost 5151 gives connection refused.
>
> The configuration is as follows
>
> type = avro
>
> agent name = a1
>
> source name = batchSource
>
> Flume configuration
>
>a1.sources.batchSource.type = avro
>
>a1.sources.batchSource.bind = 0.0.0.0
>
>a1.sources.batchSource.port=5151
>
>
> Using netcat as flume source works.

Re: Interested in adding two new features to GetMongo

[Joe Witt]

Team,

Is there anyone else familiar with Mongo that could discuss this with Mike?

Thanks
Joe

On Fri, Aug 4, 2017 at 8:56 AM, Mike Thomsen  wrote:
> 1. Add the ability to run aggregations to GetMongo.
> 2. Add the ability to get the query from a flowfile.
>
> I know you can't do #2 right now, but was wondering if there was an
> existing way to invoke the aggregation pipeline that I'm missing before I
> dive into that.
>
> Thanks,
>
> Mike

Using GregorianCalendar in apache nifi

[sally]

Is it  possible to import  java.util.GregorianCallendar in groovy script(in
executescript processor)?and  if it isn't possible  what should i use
instead of it?



--
View this message in context: 
http://apache-nifi-developer-list.39713.n7.nabble.com/Using-GregorianCalendar-in-apache-nifi-tp16629.html
Sent from the Apache NiFi Developer List mailing list archive at Nabble.com.

Re: contributor to NIFI JIRA

[Mark Payne]

Sandish,

I have added you as a contributor. Thanks and welcome!

-Mark


> On Aug 10, 2017, at 10:33 AM, Sandish Kumar HN  wrote:
> 
> Hi All,
> 
> 
> Can any one add me as a contributor to NIFI JIRA? I'm Interested in working
> on few issue's
> 
> -- 
> 
> Thanks,
> Regards,
> SandishKumar HN

Re: NiFi RecordPath Question

[Mark Payne]

Hi David,

Just wanted to chime in that what Andy describes below should indeed work to 
provide an 'AND'
type of functionality. If you have problems getting this to work, please let us 
know!

Thanks
-Mark


On Aug 9, 2017, at 9:52 PM, Andy LoPresto 
> wrote:

Hi David,

This is absolutely the correct place to ask these questions. There is also a 
mailing list more focused on the user experience and less of the development 
discussion at us...@nifi.apache.org which may 
help as well.

Mark Payne contributed most of the record processing code, and has written 
about it here [1] and here [2], and Bryan Bende has also written a tutorial [3] 
on using it. I am far from a record path expert, but have you tried simply 
combining the predicates like so? (I don’t have a 1.2 instance available right 
now to test with).

 /loc2[*][isEmpty(./rlp)][./src = 'network']/acc

If this doesn’t work, hopefully Mark or another user can chime in with a 
suggestion, or if it is not currently possible, you can open a Jira ticket 
requesting this feature.

[1] https://blogs.apache.org/nifi/entry/record-oriented-data-with-nifi
[2] https://blogs.apache.org/nifi/entry/real-time-sql-on-event
[3] 
http://bryanbende.com/development/2017/06/20/apache-nifi-records-and-schema-registries

Andy LoPresto
alopre...@apache.org
alopresto.apa...@gmail.com
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69

On Aug 9, 2017, at 3:51 PM, David Nahoopii 
> wrote:

https://nifi.apache.org/docs/nifi-docs/html/record-path-guide.html#filters


Apologies in advance - I'm sure this email alias isn't used for questions, but 
I did see it at the top of the page in the link above.
Is it possible to do any AND condition with using record path?



Given the below json object, I would like to use RecordPath in an UpdateRecord 
processor to get the value of /loc2[*][isEmpty(./rlp) AND ./src = 
'network']/acc  (meaning, src = 'network' and rlp is empty/does not exist).  
Using nifi 1.2, is there no AND condition to use?  Has one been added in 1.3 or 
later?


Thanks and apologies in advance. - Dave


{

   "loc2": [
   {
   "acc": 92,
   "src": "gps",
   "ll": {
   "lat": 41.83173031,
   "lon": -88.09725264
   },
   "sol": 1498861615000,
   "brn": 307.7,
   "alt": 191,
   "spd": 2.48reddi
   },
   {
   "acc": 18.088,
   "src": "network",
   "ll": {
   "lat": 41.8317428,
   "lon": -88.096802
   },
   "sol": 1498865950344,
   "alt": 193.9
   },
   {
   "acc": 18.088,
   "src": "network",
   "ll": {
   "lat": 41.8317428,
   "lon": -88.096802
   },
   "sol": 1498865950344,
   "alt": 193.9,
   "rlp": "passive"
   },

   {
   "acc": 20,
   "src": "fused",
   "ll": {
   "lat": 41.8317428,
   "lon": -88.096802
   },
   "sol": 1498867975640,
   "alt": 0,
   "spd": 0
   }
   ],


}

contributor to NIFI JIRA

[Sandish Kumar HN]

Hi All,


Can any one add me as a contributor to NIFI JIRA? I'm Interested in working
on few issue's

-- 

Thanks,
Regards,
SandishKumar HN

Re: get controller service's configuration

[Bryan Bende]

The way controller services are setup you have the following...

- DBCPService interface (provides getConnection()) extends
ControllerService interface (empty interface to indicate it is a CS)
- DBCPConnectionPool extends AbstractControllerService implements DBCPService
- Processor XYZ depends on DBCPService interface

The DBCPService interface is the common point between the processor
and the implementations. The processor XYZ classpath only knows about
the DBCPService interface, it doesn't know anything about the classes
that implement it... there could actually be several implementations
in different NARs, but it is up to the framework to provide access to
these.

Since the processor only depends on the interface, which in this case
only exposes getConnection(), you can't really assume the service has
certain properties because DBCPConnectionPool.DB_DRIVERNAME is
specific to the DBCPConnectionPool implementation... another
implementation may not have that property, or may call it something
different. The interface would have to provide getDriverName() so that
each implementation could provide that.

-Bryan


On Thu, Aug 10, 2017 at 4:33 AM, 尹文才  wrote:
> Thanks Andy, I've tried your approach, in my case the controller service is
> a DBCPConnectionPool and when I tried to get driver class name property
> through context.getProperty(DBCPConnectionPool.DB_DRIVERNAME).getValue(),
> but I the value is null. The AbstractControllerService class does have a
> method getConfigurationContext() to get configuration context, but the
> method is protected. So I still didn't find a feasible way to get the
> controller service's properties.
>
> Regards,
> Ben
>
> 2017-08-10 12:18 GMT+08:00 Andy LoPresto :
>
>> You can get the current property values of a controller service from the
>> processor by using the ProcessContext object. For example, in GetHTTP [1],
>> in the @OnScheduled method, you could do:
>>
>> context.getControllerServiceLookup().getControllerService("my-
>> controller-service-id”);
>>
>> context.getProperty("controller-service-property-name");
>> context.getProperty(SomeControllerService.CONSTANT_PROPERTY_DESCRIPTOR);
>>
>> I forget if context.getProperty() will give the controller service
>> properties as well as the processor properties. If it doesn’t, you can cast
>> the retrieved ControllerService into AbstractControllerService or the
>> concrete class and access available properties directly from the
>> encapsulated ConfigurationContext.
>>
>> [1] https://github.com/apache/nifi/blob/master/nifi-nar-
>> bundles/nifi-standard-bundle/nifi-standard-processors/src/
>> main/java/org/apache/nifi/processors/standard/GetHTTP.java#L295
>>
>> Andy LoPresto
>> alopre...@apache.org
>> *alopresto.apa...@gmail.com *
>> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
>>
>> On Aug 9, 2017, at 6:57 PM, 尹文才  wrote:
>>
>> Thanks Koji, I checked the link you provided and I think getting a
>> DataSource is no different than getting the DBCP service(they could just
>> get the connection). Actually I was trying to get the configured driver
>> class to check the database type.
>>
>> Regards,
>> Ben
>>
>> 2017-08-10 9:29 GMT+08:00 Koji Kawamura :
>>
>> Hi Ben,
>>
>> I'm not aware of ways to obtain configurations of a controller from a
>> processor. Those should be encapsulated inside a controller service.
>> If you'd like to create DataSource instance instead of just obtaining
>> a connection, this discussion might be helpful:
>> https://github.com/apache/nifi/pull/1417
>>
>> Although I would not recommend, if you really need to obtain all
>> configurations, you can do so by calling NiFi REST API from your
>> processor.
>>
>> Thanks,
>> Koji
>>
>> On Thu, Aug 10, 2017 at 10:09 AM, 尹文才  wrote:
>>
>> Hi guys, I have a customized processor with a DBCP controller service as
>>
>> a
>>
>> property. I could get the DBCP controller service in my code, but does
>> anyone know how to obtain all the configurations of the DBCP controller
>> service in java code(e.g. Database Connection URL, Database Driver
>> Location, etc) Thanks.
>>
>> Regards,
>> Ben
>>
>>
>>
>>

Re: get controller service's configuration

[尹文才]

Thanks Andy, I've tried your approach, in my case the controller service is
a DBCPConnectionPool and when I tried to get driver class name property
through context.getProperty(DBCPConnectionPool.DB_DRIVERNAME).getValue(),
but I the value is null. The AbstractControllerService class does have a
method getConfigurationContext() to get configuration context, but the
method is protected. So I still didn't find a feasible way to get the
controller service's properties.

Regards,
Ben

2017-08-10 12:18 GMT+08:00 Andy LoPresto :

> You can get the current property values of a controller service from the
> processor by using the ProcessContext object. For example, in GetHTTP [1],
> in the @OnScheduled method, you could do:
>
> context.getControllerServiceLookup().getControllerService("my-
> controller-service-id”);
>
> context.getProperty("controller-service-property-name");
> context.getProperty(SomeControllerService.CONSTANT_PROPERTY_DESCRIPTOR);
>
> I forget if context.getProperty() will give the controller service
> properties as well as the processor properties. If it doesn’t, you can cast
> the retrieved ControllerService into AbstractControllerService or the
> concrete class and access available properties directly from the
> encapsulated ConfigurationContext.
>
> [1] https://github.com/apache/nifi/blob/master/nifi-nar-
> bundles/nifi-standard-bundle/nifi-standard-processors/src/
> main/java/org/apache/nifi/processors/standard/GetHTTP.java#L295
>
> Andy LoPresto
> alopre...@apache.org
> *alopresto.apa...@gmail.com *
> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
>
> On Aug 9, 2017, at 6:57 PM, 尹文才  wrote:
>
> Thanks Koji, I checked the link you provided and I think getting a
> DataSource is no different than getting the DBCP service(they could just
> get the connection). Actually I was trying to get the configured driver
> class to check the database type.
>
> Regards,
> Ben
>
> 2017-08-10 9:29 GMT+08:00 Koji Kawamura :
>
> Hi Ben,
>
> I'm not aware of ways to obtain configurations of a controller from a
> processor. Those should be encapsulated inside a controller service.
> If you'd like to create DataSource instance instead of just obtaining
> a connection, this discussion might be helpful:
> https://github.com/apache/nifi/pull/1417
>
> Although I would not recommend, if you really need to obtain all
> configurations, you can do so by calling NiFi REST API from your
> processor.
>
> Thanks,
> Koji
>
> On Thu, Aug 10, 2017 at 10:09 AM, 尹文才  wrote:
>
> Hi guys, I have a customized processor with a DBCP controller service as
>
> a
>
> property. I could get the DBCP controller service in my code, but does
> anyone know how to obtain all the configurations of the DBCP controller
> service in java code(e.g. Database Connection URL, Database Driver
> Location, etc) Thanks.
>
> Regards,
> Ben
>
>
>
>