Re: Processors on the fly for many sensor devices

2016-10-18 Thread Andrew Psaltis
'd be happy to test and provide feedback. > We have a use-case now with Nifi that we're in the process of implementing. > > If you have sources or binaries feel free to let me know. > > Regards, > Davy > > On Fri, Sep 30, 2016 at 9:24 PM, Andrew Psaltis > w

Re: Processors on the fly for many sensor devices

2016-10-31 Thread Andrew Psaltis
> received from the tcp stream. I've seen something similar in the ListenTCP > processor. > > Sorry to hear about your loss. My condolences. > > Regards, > Davy > > > > On Tue, Oct 18, 2016 at 10:16 AM, Andrew Psaltis > wrote: > >> Davey, >> So

Re: Processors on the fly for many sensor devices

2016-11-01 Thread Andrew Psaltis
undle before submitting the P/R. Let me know if you want me to add in the processing for the demarcator splitting. Thanks, Andrew On Mon, Oct 31, 2016 at 4:37 PM, Andrew Psaltis wrote: > Hi Davy, > Thanks for the P/R - I merged that in just a little bit ago. > > I think the feature y

Re: error with flow.xml.gz

2016-11-29 Thread Andrew Psaltis
>) and > delete this email from your system. Any views expressed in this message are > those of the individual sender, except where the sender specifically states > them to be the views of Two Degrees Mobile Limited. We do not guarantee > that this material is free from viruses or any other

Re: Processors on the fly for many sensor devices

2016-12-14 Thread Andrew Psaltis
sor-devices-tp47p427.html > Sent from the Apache NiFi Users List mailing list archive at Nabble.com. > -- Thanks, Andrew Subscribe to my book: Streaming Data <http://manning.com/psaltis> <https://www.linkedin.com/pub/andrew-psaltis/1/17b/306> twiiter: @itmdata <http://twitter.com/intent/user?screen_name=itmdata>

Re: Processors on the fly for many sensor devices

2016-12-16 Thread Andrew Psaltis
Apache NiFi Users List mailing list archive at Nabble.com. > -- Thanks, Andrew Subscribe to my book: Streaming Data <http://manning.com/psaltis> <https://www.linkedin.com/pub/andrew-psaltis/1/17b/306> twiiter: @itmdata <http://twitter.com/intent/user?screen_name=itmdata>

Re: How to wait for multiple processors' flowfile?

2017-05-09 Thread Andrew Psaltis
files from A and B. > > So my question is is there a way for C to wait for both flowfiles from A > and B > to arrive and then start the work in C? Thanks. > -- Thanks, Andrew Subscribe to my book: Streaming Data <http://manning.com/psaltis> <https://www.linkedin.com/pub

Re: Spark Streaming with Nifi

2017-06-04 Thread Andrew Psaltis
ontroller > service ? or any better approach for running spark streaming application > with Nifi ? > > Thanks and Regards, > Shashi > > > -- Thanks, Andrew Subscribe to my book: Streaming Data <http://manning.com/psaltis> <https://www.linkedin.com/pub/andrew-psaltis/1/17b/306> twiiter: @itmdata <http://twitter.com/intent/user?screen_name=itmdata>

Re: Spark Streaming with Nifi

2017-06-04 Thread Andrew Psaltis
logical ? > > Thanks > Shashi > > > On Sun, Jun 4, 2017 at 3:10 PM, Andrew Psaltis > wrote: > >> Hi Shashi, >> I'm sure there is a way to make this work. However, my first question is >> why you would want to? By design a Spark Streaming application should

Re: Re: Any possibility to run ExecuteProcess like crontab

2017-07-13 Thread Andrew Psaltis
t; Thanks, > > Hollis > > > *From:* Andrew Psaltis > *Date:* 2017-07-12 19:29 > *To:* users > *Subject:* Re: Any possibility to run ExecuteProcess like crontab > > Hollis, > You should be able to accomplish what you want by using the Cron > scheduling under the s

Re: Output Port: Port X is invalid because Output Connection for port X is not defined

2017-09-04 Thread Andrew Psaltis
a Process Group and I am getting > error that - Port X is invalid because Output Connection for port X is not > defined. > > What configuration am I missing? > Thanks!! > -- Thanks, Andrew Subscribe to my book: Streaming Data <http://manning.com/psaltis> <https://www.l

Re: Output Port: Port X is invalid because Output Connection for port X is not defined

2017-09-04 Thread Andrew Psaltis
mayank rathi wrote: > Hello Andres, > > Yes. I am getting same error message. I tried on 1.3 and 1.4 version. > > Thanks for prompt response. > > Regards > Mayank > > On Mon, Sep 4, 2017 at 3:53 PM, Andrew Psaltis > wrote: > >> Hi Mayank, >> J

Re: Nifi kafka ser/deser problem

2017-09-05 Thread Andrew Psaltis
> > As emails may be altered, Orange is not liable for messages that have > been modified, changed or falsified. > > Thank you. > > > > ____________ > _ > > Ce message et ses pieces jointes peuvent contenir des informations > confidentielles ou privilegiees et ne doivent donc > pas etre diffuses, exploites ou copies sans autorisation. Si vous avez > recu ce message par erreur, veuillez le signaler > a l'expediteur et le detruire ainsi que les pieces jointes. Les messages > electroniques etant susceptibles d'alteration, > Orange decline toute responsabilite si ce message a ete altere, deforme ou > falsifie. Merci. > > This message and its attachments may contain confidential or privileged > information that may be protected by law; > they should not be distributed, used or copied without authorisation. > If you have received this email in error, please notify the sender and > delete this message and its attachments. > As emails may be altered, Orange is not liable for messages that have been > modified, changed or falsified. > Thank you. > > -- Thanks, Andrew Subscribe to my book: Streaming Data <http://manning.com/psaltis> <https://www.linkedin.com/pub/andrew-psaltis/1/17b/306> twiiter: @itmdata <http://twitter.com/intent/user?screen_name=itmdata>

Help Wanted

2016-05-03 Thread Andrew Psaltis
Oleg, I would love to help -- couple of quick questions: The GH PR's are ~60 as you indicated, but the How To Contribute guide (Code review process -- https://cwiki.apache.org/confluence/display/NIFI/Contributor+Guide#ContributorGuide-CodeReviewProcess ) shows a JIRA list with patches available.

Re: Help Wanted

2016-05-03 Thread Andrew Psaltis
us that doesn’t need testing. > > Thanks again > Cheers > Oleg > > On May 3, 2016, at 11:07 AM, Andrew Psaltis > wrote: > > Oleg, > I would love to help -- couple of quick questions: > > The GH PR's are ~60 as you indicated, but the How To Contribute guide

Re: Help Wanted

2016-05-03 Thread Andrew Psaltis
;> This has been an ongoing discussion and i’ll let other’s to contribute >>> to this. Basically we support both. That said, personally (and it appears >>> to be embraced by the rest of the community) PR is the preference >>> specifically due to the inline review/comment

Re: Help Wanted

2016-05-03 Thread Andrew Psaltis
n ongoing discussion and i’ll let other’s to contribute >>>>> to this. Basically we support both. That said, personally (and it appears >>>>> to be embraced by the rest of the community) PR is the preference >>>>> specifically due to the inline review/comme

Re: Logstash/ Filebeat/ Lumberjack -> Nifi

2016-05-07 Thread Andrew Psaltis
ited. Registered in England & Wales 04365896. >> Registered Address: SecureData House, Hermitage Court, Hermitage Lane, >> Maidstone, Kent, ME16 9NT >> > > > > ***This email originated outside SecureData*** > > Click here > <https://www.mailcontrol.com/sr/CdEQiWndhxLGX2PQPOmvUsrLibhXE7+SpVooqDfjfmrv9UcAoCvw58JRjsQQpswieUDNxz32L0IKghm6!2a+jw==> > to report this email as spam. > -- Thanks, Andrew Subscribe to my book: Streaming Data <http://manning.com/psaltis> <https://www.linkedin.com/pub/andrew-psaltis/1/17b/306> twiiter: @itmdata <http://twitter.com/intent/user?screen_name=itmdata>

Re: Build error for the custom processor against 0.7.0-SNAPSHOT

2016-05-10 Thread Andrew Psaltis
OR] > > [ERROR] For more information about the errors and possible solutions, > please read the following articles: > > [ERROR] [Help 1] > http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException > > > > Thanks > > Kumiko > -- Thanks, Andrew Subscribe to my book: Streaming Data <http://manning.com/psaltis> <https://www.linkedin.com/pub/andrew-psaltis/1/17b/306> twiiter: @itmdata <http://twitter.com/intent/user?screen_name=itmdata>

Re: Build error for the custom processor against 0.7.0-SNAPSHOT

2016-05-10 Thread Andrew Psaltis
es, > I locally build the 0.7.0-SNAPSHOT. Was the 0.7.0-SNAPSHOT deleted > recently from Maven repository? This was working fine for me until last > Friday. > > > > Thanks > > Kumiko > > > > *From:* Andrew Psaltis [mailto:psaltis.and...@gmail.com]

Re: GetKafka exception

2016-05-26 Thread Andrew Psaltis
[kafka_2.10-0.8.2.2.jar:na] >> Caused by: java.nio.channels.ClosedChannelException: null >> at kafka.network.BlockingChannel.send(BlockingChannel.scala:100) >> ~[kafka_2.10-0.8.2.2.jar:na] >> at kafka.producer.SyncProducer.liftedTree1$1(SyncProducer.scala:73) >> ~[kafka_2.10-0.8.2.2.jar:na] >> at >> kafka.producer.SyncProducer.kafka$producer$SyncProducer$$doSend(SyncProducer.scala:72) >> ~[kafka_2.10-0.8.2.2.jar:na] >> at kafka.producer.SyncProducer.send(SyncProducer.scala:113) >> ~[kafka_2.10-0.8.2.2.jar:na] >> at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:58) >> ~[kafka_2.10-0.8.2.2.jar:na] >> ... 3 common frames omitted >> >> > -- Thanks, Andrew Subscribe to my book: Streaming Data <http://manning.com/psaltis> <https://www.linkedin.com/pub/andrew-psaltis/1/17b/306> twiiter: @itmdata <http://twitter.com/intent/user?screen_name=itmdata>

Re: How to control NiFi logging

2016-06-05 Thread Andrew Psaltis
u -s * > 28 bin > 756 conf > 8412docs > 427468 lib > 64 LICENSE > 4331244 logs > 44 NOTICE > 8 README > 144 state > 503232 work > > > Thanks! > Stephane > -- Thanks, Andrew Subscribe to my book: Streaming Data <htt

Re: How to control NiFi logging

2016-06-05 Thread Andrew Psaltis
Initializer.configureByResource(ContextInitializer.java:77) >> at at >> ch.qos.logback.classic.util.ContextInitializer.autoConfig(ContextInitializer.java:152) >> at at >> org.slf4j.impl.StaticLoggerBinder.init(StaticLoggerBinder.java:85) >>

Re: How to control NiFi logging

2016-06-06 Thread Andrew Psaltis
Hi Pat, It is all standard logback, described here: http://logback.qos.ch/ On Mon, Jun 6, 2016 at 12:18 PM, Pat Trainor wrote: > Andrew, > Is any of this not standard log4j, and nifi-specific? > On Jun 6, 2016 2:34 AM, "Andrew Psaltis" wrote: > >> You are correc

Re: putazureeventhub error

2016-08-09 Thread Andrew Psaltis
ee if there is any more information? > > > > Thanks > > -Mark > > > > On Aug 9, 2016, at 10:58 AM, Smith, Aaron wrote: > > > > I am trying to use the put azure event hub processor in nifi .7 and am > getting the below error: > > > > PutAzureEventHub[id

Re: putazureeventhub error

2016-08-09 Thread Andrew Psaltis
dProcessorNode.access$ > 100(StandardProcessorNode.java:90) ~[na:na] > > at org.apache.nifi.controller.StandardProcessorNode$1.run( > StandardProcessorNode.java:1230) ~[na:na] > > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) >

Re: putazureeventhub error

2016-08-09 Thread Andrew Psaltis
lutions Engineer > EG WW HQ Presales > > > asm...@hpe.com > T +1 832 502 8829 > > Houston, Texas USA > > [image: > http://graphics8.nytimes.com/images/2015/06/03/technology/03bits-hp/03bits-hp-master315.png] > > > > *From:* Andrew Psaltis [mailto:psalti

Re: [ANNOUNCE] Apache NiFi 1.0.0-Beta release

2016-08-09 Thread Andrew Psaltis
lease note highlights can be found here:https://cwiki.apache.org/ > confluence/display/NIFI/Release+Notes#ReleaseNotes-Version1.0.0-Beta > Thank youThe Apache NiFi team > -- Thanks, Andrew Subscribe to my book: Streaming Data <http://manning.com/psaltis> <https://www.linkedin.c

Re: putazureeventhub error

2016-08-11 Thread Andrew Psaltis
asm...@hpe.com > T +1 832 502 8829 > > Houston, Texas USA > > [image: > http://graphics8.nytimes.com/images/2015/06/03/technology/03bits-hp/03bits-hp-master315.png] > > > > *From:* Andrew Psaltis [mailto:psaltis.and...@gmail.com] > *Sent:* Tuesday, August 9, 2016 12:4

Re: putazureeventhub error

2016-08-11 Thread Andrew Psaltis
Hi Aaron, Attached is a simple flow that publishes events to an eventhub I have -- if you want to try and publish to it, hit me up privately and I can share the info for the hub. Thanks, Andrew On Thu, Aug 11, 2016 at 9:11 AM, Andrew Psaltis wrote: > Hey Aaron, > Sorry for the slow respo

Re: Processors on the fly for many sensor devices

2016-09-28 Thread Andrew Psaltis
t; We have an inventory of those sensors (exposed via a REST service >> endpoint), >> > containing the sensor tcp information like ip and port) >> > >> > Is there an easy way to create these ListenTCP processors on the fly >> based >> > on a REST endpoint or some other external configuration ? How would that >> > work ? >> > >> > Thx. >> > > -- Thanks, Andrew Subscribe to my book: Streaming Data <http://manning.com/psaltis> <https://www.linkedin.com/pub/andrew-psaltis/1/17b/306> twiiter: @itmdata <http://twitter.com/intent/user?screen_name=itmdata>

Re: Processors on the fly for many sensor devices

2016-09-28 Thread Andrew Psaltis
ly we have ListenTCP which is waiting for incoming connections, and > PutTCP which makes a connection, but only writes data. > > On Wed, Sep 28, 2016 at 5:02 PM, Andrew Psaltis > wrote: > >> Davy, >> It sounds like you need a GetTCP type of processor that connects

Re: Processors on the fly for many sensor devices

2016-09-30 Thread Andrew Psaltis
t we want. > Unfortunately it is not compatible with the current Nifi version. > Do you have a binary available that would work with nifi 1.0.0 ? > > Thx. > > On Wed, Sep 28, 2016 at 11:39 PM, Andrew Psaltis > wrote: > >> Davy, >> The processor I have been worki

Re: Row counts - date lake

2018-06-10 Thread Andrew Psaltis
Hi Faisal, There are various ways this can be handled. But this is going to depend on, how are you receiving data from Oracle via Golden Gate. Are you using the HBase Handler, the HDFS Handler, a Flat File, Kafka, or via another means? Thanks, Andrew On Mon, Jun 11, 2018 at 9:43 AM Faisal Durrani

Re: Row counts - date lake

2018-06-11 Thread Andrew Psaltis
the golden gate transactions from Kafka which is received > in Nifi through consume kafka processor . Our data flow then reduces the > golden gate json message and sends the data to the target table in Hbase > using the PutHbase Json processor. > > Thanks, > Faisal > > On Mon,

Re: Consumekafka error I

2018-06-14 Thread Andrew Psaltis
Hi Faisal, How many partitions are there for that TEST_KAFKA_TOPIC topic? On Thu, Jun 14, 2018 at 9:06 PM Faisal Durrani wrote: > Hi Mark, The heap size is set to 4 gb and the time driven threas count is > set to 32 since its a 8 core node( we have a cluster of 4 nifi nodes) . > Despite of all

Re: Consumekafka error I

2018-06-20 Thread Andrew Psaltis
nsumed in the past >> so i think there is some thing also wrong with the commit configuration. >> There is some other additional property that guess i need to setup on the >> broker side which will make it scalable. But i'm unable to find it. Kindly >> let me kno

Re: Consumekafka error I

2018-06-21 Thread Andrew Psaltis
Hi Faisal, Some observations and next steps. 1. Brian has a great point of using the more appropriate consumer that will leverage the latest API. 2. This still certainly feels like a resource issue -- testing a single Kafka broker first and not growing the cluster seems odd especially