Re: Filtering GetTwitter with both terms and locations

2015-10-21 Thread Bryan Bende
>From looking at the processor code it looks like it adds both the terms and locations to the filter endpoint and should be able to filter on both. The processor leverages the Hosebird Client [1] so it could be possible that library is not working as expected. Is there a specific example of terms

Re: how to putkafka to multiple partions

2015-10-21 Thread Bryan Bende
Hello, This might related to: https://issues.apache.org/jira/browse/NIFI-1020 Does anyone know if the upgrade of the Kafka client for the next NiFi release addresses this issue? -Bryan On Mon, Oct 19, 2015 at 4:01 AM, 彭光裕 wrote: > hi, > > > > I have a test topic

Re: StoreInKiteDataset cannot recognize hive dataset

2015-11-09 Thread Bryan Bende
Hello, I'm not that familiar with Kite, but is it possible that you need to create the Kite dataset using the Kite CLI before StoreInKiteDataset tries to write data to it? It looks like that is how the test cases for this processor work:

Re: Managing flows

2015-11-11 Thread Bryan Bende
In addition to what Mark said, there is also the option of templates [1]. Templates let you export a portion, or all of your flow, and then import it again later. When you export a template it will not export any properties that are marked as sensitive properties, so it is safe to share with

Re: Post to REST service?

2015-11-11 Thread Bryan Bende
Hello, You should be able to use expression language in the URL value, you could reference any attribute by doing the following: ${attributeName}. So your URL could be http://myhost/${id} -Bryan On Wed, Nov 11, 2015 at 8:03 AM, Darren Govoni wrote: > Hi, > I am trying

Re: Why does PutFile create directories for you but PutHDFS does not?

2015-11-12 Thread Bryan Bende
I think PutHDFS always creates them so it isn't an option through the properties. -Bryan On Thu, Nov 12, 2015 at 8:19 AM, Mark Payne wrote: > Mark, > > My guess is that it was an oversight. I don't believe it was intentional > to leave it out of PutHDFS. > > Thanks >

Re: template repo for learning

2015-11-03 Thread Bryan Bende
query. > On Nov 3, 2015 9:41 AM, "Bryan Bende" <bbe...@gmail.com> wrote: > >> Christopher, >> >> In terms of templates, the best resources we have right now are: >> >> https://cwiki.apache.org/confluence/display/NIFI/Example+Dataflow+Templates >

Re: Filtering GetTwitter with both terms and locations

2015-11-01 Thread Bryan Bende
Hi David, After re-reading the Twitter API documentation [1], it says: "The track, follow, and locations fields should be considered to be combined with an OR operator. track=foo=1234 returns Tweets matching “foo” OR created by user 1234." -Bryan [1]

Re: Processor to use

2015-11-05 Thread Bryan Bende
Hello, Can you share the data format coming from Kafka? json, avro, something else? The reason I ask is because there are many different processors that can modify FlowFiles in different ways, but a lot depends on the type of data involved. For general text-based operations we have... -

Re: JSON / Avro issues

2015-11-05 Thread Bryan Bende
Jeff, Are you using the 0.3.0 release? I think this is the issue you ran into which is resolved for the next release: https://issues.apache.org/jira/browse/NIFI-944 With regards to ConvertJSONtoAvro, I believe it one json document per line with a new line at the end of each line (your second

Re: Processor to use

2015-11-05 Thread Bryan Bende
nst a lookup file ? > > > From: Bryan Bende <bbe...@gmail.com> > Reply-To: "users@nifi.apache.org" <users@nifi.apache.org> > Date: Thursday, November 5, 2015 at 4:07 PM > To: "users@nifi.apache.org" <users@nifi.apache.org> > Subject: Re: Pr

Re: Is it just me or do others think it should be a bit harder to delete your components?

2015-10-31 Thread Bryan Bende
We previously had a very long thread about this topic [1] which I think resulted in the idea of "undo" [2]. [1] https://mail-archives.apache.org/mod_mbox/nifi-dev/201508.mbox/%3ccaljk9a6nn5y7ep7-dw5yeb-33m7zjvvw+h2nj4auccmlpeh...@mail.gmail.com%3E [2]

Re: Referencing Source File Attributes with MergeContent Processor

2015-10-14 Thread Bryan Bende
I'm not sure if this makes sense with out seeing the full flow, but can you construct the Json for each image before MergeContent by using ReplaceText, this way the id will already be taken care of before merging? On Wednesday, October 14, 2015, Chris Mangold wrote: > I have

Re: Need help in nifi- flume processor

2015-10-08 Thread Bryan Bende
flume/PollableSource." > > Can you please help me in this regard. Any step/configuration I missed. > > Thanks and Regards, > Parul > > > On Wed, Oct 7, 2015 at 6:57 PM, Bryan Bende <bbe...@gmail.com> wrote: > >> Hello, >> >> The NiFi Flume processors are

Re: Nifi & Spark receiver performance configuration

2015-10-08 Thread Bryan Bende
Hello, When you say you were unhappy with the performance, can you give some more information about what was not performing well? Was the NiFi Spark Receiver not pulling messages in fast enough and they were queuing up in NiFi? Was NiFi not producing messages as fast as you expected? What kind

Re: output port

2015-10-19 Thread Bryan Bende
Hello, Just to clarify, so you are seeing the messages reach the output port and then get removed from the queue? And on the spark side the NiFi Spark receiver never receives anything? Or it receives message, but they have no content? -Bryan On Monday, October 19, 2015, Rama Krishna Manne

Re: output port

2015-10-20 Thread Bryan Bende
ed > from the queue > > yes > > And on the spark side the NiFi Spark receiver never receives anything? Or > it receives message, but they have no content? > > It receives the data but no content to do computation > > On Mon, Oct 19, 2015 at 2:14 PM, Bryan Bende <bbe.

Re: Need help in nifi- flume processor

2015-10-12 Thread Bryan Bende
t;>> 2015-10-10 02:30:46,029 ERROR [Timer-Driven Process Thread-9] >>> o.a.n.processors.flume.ExecuteFlumeSink >>> ExecuteFlumeSink[id=2d08dfe7-4fd1-4a10-9d25-0b007a2c41bf] >>> ExecuteFlumeSink[id=2d08dfe7-4fd1-4a10-9d25-0b007a2c41bf] failed to process >>> d

Re: Need help in nifi- flume processor

2015-10-07 Thread Bryan Bende
Hello, The NiFi Flume processors are for running Flume sources and sinks with in NiFi. They don't communicate with an external Flume process. In your example you would need an ExecuteFlumeSource configured to run the netcat source, connected to a ExecuteFlumeSink configured with the logger.

Re: Need help in nifi- flume processor

2015-10-13 Thread Bryan Bende
processor > so that this new.json can be converted in to a flat document of key/value > pairs, or an array of flat documents. > > Any help/guidance would be really useful. > > Thanks and Regards, > Parul > > On Mon, Oct 12, 2015 at 10:36 PM, Bryan Bende <bbe...

Re: Solr Cloud support

2015-09-03 Thread Bryan Bende
vention. > > > > Another approach is to have cluster master in NiFi talk to ZK and decide > > which shards to query. Divide these shards among slave nodes. > > My understanding is NiFi cluster master is not indented for such purpose. > > I'm not sure if this even possible.

Re: MergeContent Question

2015-10-01 Thread Bryan Bende
Hi Chris, In the MergeContent case, instead of putting \n in the file, try putting a new line by hitting return. I remembering doing this once and I had created the empty file with vi and added a new line, and then I actually got 2 new lines in my output because I guess vi has a new line

Re: Array into MongoDB

2015-09-24 Thread Bryan Bende
? If so it would be interesting to see what happens if you bump that up. On Thu, Sep 24, 2015 at 5:06 PM, Bryan Bende <bbe...@gmail.com> wrote: > Adam, > > Based on that message I suspect that MongoDB does not support sending in > an array of documents since it looks like i

Re: Generate flowfiles from flowfile content

2015-09-23 Thread Bryan Bende
Sorry I missed Joe's email while sending mine... I can put together a template showing this. On Wednesday, September 23, 2015, Bryan Bende <bbe...@gmail.com> wrote: > David, > > Take a look at ExtractText, it is for pulling FlowFile content into > attributes. I think tha

Re: Nifi cluster features - Questions

2016-01-09 Thread Bryan Bende
to try this setup to see the results, appreciate your help. > > Thanks, > -Chakri > > From: Bryan Bende <bbe...@gmail.com > <javascript:_e(%7B%7D,'cvml','bbe...@gmail.com');>> > Reply-To: "users@nifi.apache.org > <javascript:_e(%7B%7D,'cvml','users@nifi.apa

Re: Nifi cluster features - Questions

2016-01-08 Thread Bryan Bende
Hi Chakri, I believe the DistributeLoad processor is more for load balancing when sending to downstream systems. For example, if you had two HTTP endpoints, you could have the first relationship from DistributeLoad going to a PostHTTP that posts to endpoint #1, and the second relationship going

Re: Use Case Validation

2015-12-23 Thread Bryan Bende
Hi Dan, This is definitely a use case that NiFi can handle. A possible architecture for your scenario would be something like the following... - Run NiFi instances on the machines where you need to collect logs, these would not be clustered, just stand-alone instances. - These would pick up your

Re: Use Case Validation

2015-12-23 Thread Bryan Bende
Dan, A stand-alone instance is the default behavior. If you extract a NiFi distribution and run "bin/nifi.sh start", without changing any of the clustering related properties, then you get a single instance running on port 8080 by default. My thought behind sending them via site-to-site is to

Re: Use Case Validation

2015-12-23 Thread Bryan Bende
Dan, Thanks for the feedback, and we are glad to hear the documentation is helpful. Please feel free to let us know anything else that is not clear, or could be improved. -Bryan On Wed, Dec 23, 2015 at 2:06 PM, Dan wrote: > Got it; I think I noticed a 404 error

Re: How to save the original JSON to Solr - PutSolr

2015-12-23 Thread Bryan Bende
Bob, The field mappings with the json paths is actually something provided by Solr. The PutSolr processor is just passing all those parameters over, and Solr is doing the extraction, so we are limited here by what Solr provides. I'm not aware of a way to tell Solr to select the whole document

Re: queued files

2015-11-19 Thread Bryan Bende
Charlie, The behavior you described usually means that the processor encountered an unexpected error which was thrown back to the framework which rolls back the processing of that flow file and leaves it in the queue, as opposed to an error it expected where it would usually route to a failure

Re: Data Ingestion forLarge Source Files and Masking

2016-01-12 Thread Bryan Bende
Obaid, I can't say for sure how much this would improve performance, but you might want to wrap the OutputStream with BufferedOutputStream or BufferedWriter. Would be curious to here if that helps. A similar scenario from the standard processors is ReplaceText, here is one example where it uses

Re: "Processor requires an upstream connection" for FetchS3Object?

2016-01-12 Thread Bryan Bende
Russell/Corey, In 0.4.0 there is a new way for processors to indicate what they expect as far as input, it can be required, allowed, or forbidden. This prevents scenarios like ExecuteSQL which at one point required an input FlowFile, but the processor could be running and started with out an

Re: How does GetSFTP treat files being loaded?

2016-06-03 Thread Bryan Bende
Adding to what Oleg said... GetSFTP has a property called Ignore Dotted Files which defaults to true and tells GetSFTP to ignore filenames that begin with dots which can help in this scenario if the uploader can upload to temporary file starting with a dot and rename when done. -Bryan On Fri,

Re: Custom processor is failing for concurrency

2016-06-03 Thread Bryan Bende
t > documentation says? Have you looked at the source to see what’s going on > there? If its open please link and we can tale a look. > > Cheers > Oleg > > On Jun 3, 2016, at 4:58 PM, Kumiko Yada <kumiko.y...@ds-iq.com> wrote: > > Here is the code, https://g

Re: Kerberos / NiFi 0.5.1 / ListHDFS

2016-06-08 Thread Bryan Bende
Hi Michael, Can you confirm that the ListHDFS that is showing the problem is pointing at a core-site.xml that has the value of "hadoop.security.authentication" set to "kerberos" ? -Bryan On Wed, Jun 8, 2016 at 12:22 PM, Michael Dyer wrote: > I'm looking for any

Re: Custom processor is failing for concurrency

2016-06-09 Thread Bryan Bende
eems > to be doing some iteration where I presume the remove is called. Perhaps it > is not a thread safe class after all. What does Microsoft documentation > says? Have you looked at the source to see what’s going on there? If its > open please link and we can tale a look. > > > > C

Re: Custom processor is failing for concurrency

2016-06-09 Thread Bryan Bende
e threads > via UI might be the same thing. > > Thanks > Kumiko > ------ > *From:* Bryan Bende <bbe...@gmail.com> > *Sent:* Thursday, June 9, 2016 11:26:10 AM > *To:* users@nifi.apache.org > *Cc:* Kevin Verhoeven; Ki Kang > > *Subje

Re: Dependency for SSL

2016-05-25 Thread Bryan Bende
Hello, This Wiki page shows how to setup the dependencies to use the SSLContextService from a custom processor bundle: https://cwiki.apache.org/confluence/display/NIFI/Maven+Projects+for+Extensions#MavenProjectsforExtensions-LinkingProcessorsandControllerServices Thanks, Bryan On Wed, May

Re: Replace Text

2016-06-14 Thread Bryan Bende
What is the error you are getting from TransformXML? On Tue, Jun 14, 2016 at 9:38 AM, Anuj Handa wrote: > anybody has any thoughts on UTF 8 Flow files with XMLtransforemation and > other processors ? > > Anuj > > On Mon, Jun 13, 2016 at 4:45 PM, Anuj Handa

Re: NoClassDefFoundError: org/slf4j/LoggerFactory

2016-06-13 Thread Bryan Bende
Hi Donald, I know this does not directly address the conflict in dependencies, but I wanted to mention that it is not required to inherit from nifi-nar-bundles. The Maven archetype does that by default, but you can certainly remove it, there are some instructions on how to do so here:

Re: Writing files to MapR File system using putHDFS

2016-06-13 Thread Bryan Bende
I'm not sure if this would make a difference, but typically the configuration resources would be the full paths to core-site.xml and hdfs-site.xml. Wondering if using those instead of yarn-site.xml changes anything. On Monday, June 13, 2016, Ravi Papisetti (rpapiset) wrote:

Re: How to configure site-to-site communication between nodes in one cluster.

2016-06-02 Thread Bryan Bende
-> Send it to validation/transformation Hadoop Job -> get results back to > Nifi -> do other things. What do you think of this approach? > > 2016-06-01 21:24 GMT+03:00 Bryan Bende <bbe...@gmail.com>: > >> NiFi is definitely suitable for processing large

Re: Spark or custom processor?

2016-06-02 Thread Bryan Bende
Conrad, I would think that you could do this all in NiFi. How do the log files come into NiFi? TailFile, ListenUDP/ListenTCP, List+FetchFile? -Bryan On Thu, Jun 2, 2016 at 6:41 AM, Conrad Crampton wrote: > Hi, > > Any advice on ‘best’ architectural approach

Re: How to access the counter and provenance Info

2016-06-02 Thread Bryan Bende
Hi Kumiko, I think you can only increment counters and report provenance events from processors, but not query counters or query provenance. If you are in a custom processor, could have an AtomicInteger counter that increments on every onTrigger and reset when 24 hours has passed? Thanks,

Re: Spark or custom processor?

2016-06-02 Thread Bryan Bende
url splitting/ > enriching etc. – probably could have done with ExecuteScript processor in > hindsight) so am comfortable with going this route if that is deemed more > appropriate. > > Thanks > Conrad > > *From: *Bryan Bende <bbe...@gmail.com> > *Reply-To: *"user

Re: Spark or custom processor?

2016-06-02 Thread Bryan Bende
more >> >> questions, as always, don't hesitate to shoot another email! >> >> >> >> Thanks >> >> -Mark >> >> >> >> >> >> On Jun 2, 2016, at 9:28 AM, Conrad Crampton <conrad.cramp...@secdata.com> >> wrote:

Re: How to configure site-to-site communication between nodes in one cluster.

2016-06-01 Thread Bryan Bende
not suitable for Nifi. > > Is Nifi a suitable tool for processing large files or I should not do > actual processing work outside the Nifi flow? > > 2016-06-01 17:28 GMT+03:00 Bryan Bende <bbe...@gmail.com>: > >> Hello, >> >> This post [1] has a description of how to r

Re: Which processor to use to cleanly convert xml to json?

2016-06-01 Thread Bryan Bende
Hi Keith, There is currently no built in processor that directly transforms XML to JSON. TransformXML leverages XSLT to transform and XML document into some other format. In that post, the XSLT happens to transform into JSON, but it looks like maybe it only handles top-level elements and not

Re: How to configure site-to-site communication between nodes in one cluster.

2016-06-01 Thread Bryan Bende
Hello, This post [1] has a description of how to redistribute data with in the same cluster. You are correct that it involves a RPG pointing back to the same cluster. One thing to keep in mind is that typically we do this with a List + Fetch pattern, where the List operation produces lightweight

Re: IDE-specific setup

2016-06-21 Thread Bryan Bende
I personally use IntelliJ and it generally does pretty well at automatically importing everything. The process to import an existing Maven project is described here: https://www.jetbrains.com/help/idea/2016.1/importing-project-from-maven-model.html In Step #2 you would select the root directory

Re: Scheduling using CRON driven on Windows OS

2016-06-16 Thread Bryan Bende
Also, the user guide has a description of the scheduling strategies which described the cron format: https://nifi.apache.org/docs/nifi-docs/html/user-guide.html#scheduling-tab On Thu, Jun 16, 2016 at 1:17 PM, Pierre Villard wrote: > Hi Keith, > > This is the

Re: Merge ListenSyslog events

2016-01-08 Thread Bryan Bende
Hello, Glad to hear you are getting started using ListenSyslog! You are definitely running into something that we should consider supporting. The current implementation treats each new-line as the message delimiter and places each message on to a queue. When the processor is triggered, it grabs

Re: Nifi cluster features - Questions

2016-01-10 Thread Bryan Bende
0 > > nifi.remote.input.secure=false > > > > When I try drop remote process group (with http://:8080/nifi), > I see > > error as follows for two nodes. > > > > [:8080] - Remote instance is not allowed for Site to Site > > communication > > [:8080] - Remote ins

Re: Merge ListenSyslog events

2016-01-14 Thread Bryan Bende
e. I'm not sure of the process for a > feature request but I'll try to find some times to create a pull request or > a patch for this. > > > Best regards, > Louis-Etienne > > On 8 January 2016 at 12:15, Bryan Bende <bbe...@gmail.com> wrote: > >> Hello, >> &

Re: MergeContent: Correlation Attribute Name syntax for matching syslog events

2016-02-07 Thread Bryan Bende
I believe what Joe was referring to with RouteText was that it can take a regular expression with a capture group, and output a FlowFile per unique value of the capturing group. So if the incoming data is a FlowFile with a bunch of syslog messages and you provide a regex that captures hostname, it

Re: Send parameters while using getmongo processor

2016-02-04 Thread Bryan Bende
ery parameter? > > Regards, > > Sudeep > > On Thu, Feb 4, 2016 at 9:11 PM, sudeep mishra <sudeepshekh...@gmail.com> > wrote: > >> Thanks for the feedback Bryan. >> >> Yes I need a processor similar to what you described. >> >> On Thu, Feb 4,

Re: Send parameters while using getmongo processor

2016-02-04 Thread Bryan Bende
Hi Sudeep, >From looking at the GetMongo processor, I don't think this can be done today. That processor is meant to be a source processor that extracts data from Mongo using a fixed query. It seems to me like we would need a FetchMongo processor with a Query field that supported expression

Re: Log4j/logback parser via syslog

2016-02-12 Thread Bryan Bende
Hello, Currently there are no built in processors to parse log formats, but have you taken a look at the ExtractText processor [1]? If you can come up with a regular expression for whatever you are trying to extract, then you should be able to use ExtractText. Other options... You could write

Re: Log4j/logback parser via syslog

2016-02-12 Thread Bryan Bende
at 10:48 AM, Madhukar Thota <madhukar.th...@gmail.com> wrote: > Thanks Bryan. I will look into ExtractText processor. > > Do you know what scripting languages are supported with new processors? > > -Madhu > > On Fri, Feb 12, 2016 at 9:27 AM, Bryan Bende <bbe..

Re: Log4j/logback parser via syslog

2016-02-12 Thread Bryan Bende
t; > -Madhu > > > On Fri, Feb 12, 2016 at 11:33 AM, Madhukar Thota <madhukar.th...@gmail.com> > wrote: > > Thanks Bryan. Looking forward for the release. > > > > > > > > > > > >On Fri, Feb 12, 2016 at 10:55 AM, Bryan B

Re: Add date in the flow file attribute

2016-02-03 Thread Bryan Bende
Just adding to what Juan said... The PutMongo processor sends the content of a FlowFile to Mongo, so if you use AttributesToJson -> PutMongo, with AttributesToJson Destination set to flowfile-content, then you would be sending the attributes to Mongo. -Bryan On Wed, Feb 3, 2016 at 9:22 AM, Juan

Re: GetMail processor

2016-02-22 Thread Bryan Bende
or is available somewhere or > planned J . I have seen PutEmail but not the dual processor in the help > > > > The goal is to automatically and regularly processes incoming mail , > transforms the content and index the transformed content with solr > > phil > > Best r

Re: Connecting Spark to Nifi 0.4.0

2016-02-22 Thread Bryan Bende
Hi Kyle, It seems like the stack trace is suggesting that Spark is trying to download dependencies from the like that references Executor.updateDependencies: https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/executor/Executor.scala#L391 Any chance you are behind

Re: Newbie looking for docs subdirectory in NAR

2016-01-20 Thread Bryan Bende
Hi Russ, If you have the traditional bundle with a jar project for your processors and a nar project that packages everything, then the additionalDetails.html goes in the jar project under src/main/resources/docs followed by a directory with the fully qualified class name of your processor. As

Re: splitText output appears to be getting dropped

2016-02-19 Thread Bryan Bende
Hello, MergeContent has properties for header, demarcator, and footer, and also has a strategy property which specifies whether these values come from a file or inline text. If you do inline text and specify a demarcator of a new line (shift + enter in the demarcator value) then binary

Re: Connecting Spark to Nifi 0.4.0

2016-02-20 Thread Bryan Bende
Just wanted to point out that the stack trace doesn't actually show the error coming from code in the NiFi Site-To-Site client, so I wonder if it is something else related to Spark. Seems similar to this error, but not sure:

Re: Nifi 0.50 and GetKafka Issues

2016-02-21 Thread Bryan Bende
I know this does not address the larger problem, but in this specific case, would the 0.4.1 Kafka NAR still work in 0.5.x? If the NAR doesn't depend on any other NARs I would think it would still work, and could be a work around for those that need to stay on Kafka 0.8.2. On Sunday, February 21,

Re: Nifi 'as a service'?

2016-02-17 Thread Bryan Bende
Keaton, You can definitely build a REST service in NiFi! I would take a look at HandleHttpRequest and HandleHttpResponse. HandleHttpRequest would be the entry point of your service, the FlowFiles coming out of this processor would represent the request being made, you can then perform whatever

Re: Large dataset on hbase

2016-04-12 Thread Bryan Bende
Hi Prabhu, How did you end up converting your CSV into JSON? PutHBaseJSON creates a single row from a JSON document. In your example above, using n1 as the rowId, it would create a row with columns n2 - n22. Are you seeing columns missing, or are you missing whole rows from your original CSV?

Re: Large dataset on hbase

2016-04-12 Thread Bryan Bende
2 store as columns in hbase. > > some of rows (n1's) are stored inside the table but remaining are read > well but not stored. > > Thanks, > Prabhu Mahendran > > On Tue, Apr 12, 2016 at 1:58 PM, Bryan Bende <bbe...@gmail.com> wrote: > >> Hi Prabhu, >>

Re: ListFile - FetchFile Cluster behavior

2016-03-24 Thread Bryan Bende
Hi Lee, The List+Fetch model in a cluster is one of the trickier configurations to set up. This article has a good description with a diagram under the "pulling section" that shows ListHDFS+FetchHDFS, but should be the same for ListFile+FetchFile:

Re: String conversion to Int, float double

2016-03-24 Thread Bryan Bende
I think the problem is that all attributes are actually Strings internally, even after calling toNumber() that is only temporary while the expression language is executing. So by the time it gets to AttributesToJson it doesn't have any information about the type of each attribute and they all end

Re: ExecuteSQL and NiFi 0.5.1 - Error org.apache.avro.SchemaParseException: Empty name

2016-03-05 Thread Bryan Bende
I think this a legitimate bug that was introduced in 0.5.0. I created this ticket: https://issues.apache.org/jira/browse/NIFI-1596 For those interested, I think the line of code causing the problem is this:

Re: PutMongo Processor

2016-03-05 Thread Bryan Bende
Uwe, Personally I don't have that much experience with MongoDB, but the additional functionality you described sounds like something we would want to support. Looking through JIRA I only see one ticket related to MongoDB to add SSL support [1] so I think it would be great to create a new JIRA to

Re: Advance usage documentation not displayed

2016-03-05 Thread Bryan Bende
Russell, Just want to confirm what you are seeing... so when you bring up the usage for your processor, you see the normal documentation, but you don't see an "Additional Details..." link at the top of the page? One example I know of is the PutSolrContentStream processor:

Re: No controller service types found that are applicable for this property

2016-03-31 Thread Bryan Bende
I suspect this is a dependency problem with the way the NAR was built. How did you create the new project structure for your copied PutHBaseJSON? You would need the same dependencies that nifi-hbase-bundle has... A provided dependency on the hbase client service API in your processors pom:

Re: Create row keys for HBase from Json messages

2016-03-21 Thread Bryan Bende
Hong, Glad to hear you are getting started with NiFi! What do your property names look like on EvaluatJsonPath? Typically if you wanted to extract the effective timestamp, event id, and applicant id from your example json, then you would add properties to EvaluateJsonPath like the following:

Re: Apache NiFi/Hive - store merged tweets in HDFS, create table in hive

2016-04-21 Thread Bryan Bende
Hello, I believe this example shows an approach to do it (it includes Hive even though the title is Solr/banana): https://community.hortonworks.com/articles/1282/sample-hdfnifi-flow-to-push-tweets-into-solrbanana.html The short version is that it extracts several attributes from each tweet using

Re: Apache NiFi/Hive - store merged tweets in HDFS, create table in hive

2016-04-21 Thread Bryan Bende
Also, this blog has a picture of what I described with MergeContent: https://blogs.apache.org/nifi/entry/indexing_tweets_with_nifi_and -Bryan On Thu, Apr 21, 2016 at 4:37 PM, Bryan Bende <bbe...@gmail.com> wrote: > Hi Igor, > > I don't know that much about Hive so I can't

Re: Question on setting up nifi flow

2016-04-28 Thread Bryan Bende
Hi Susheel, In addition to what Pierre mentioned, if you are interested in an example of using HandleHttpRequest/Response, there is a template in this repository: https://github.com/hortonworks-gallery/nifi-templates The template is HttpExecuteLsCommand.xml and shows how to build a web service

Re: Spark & NiFi question

2016-05-20 Thread Bryan Bende
Hi Conrad, Sorry this has been so challenging to setup. After trying it out myself, I believe the problem you ran into when you didn't set the System properties is actually a legit bug in the SiteToSiteClient... I wrote it up in this JIRA [1], but the short answer is that it never uses those

Re: Spark & NiFi question

2016-05-23 Thread Bryan Bende
Conrad, I think the error message is mis-leading a little bit, it says... "Unable to communicate with yarn-cm1.mis-cds.local:9870 because it requires Secure Site-to-Site communications, but this instance is not configured for secure communications" That statement is saying that your NiFi

Re: Spark & NiFi question

2016-05-19 Thread Bryan Bende
Hi Conrad, I think there are a couple of things at play here... One is that the SSL properties need to be set on the SiteToSiteClientBuilder, rather than through system properties. There should be methods to set the keystore and other values. In a secured NiFi instance, the certificate you are

Re: Spark & NiFi question

2016-05-23 Thread Bryan Bende
Conrad, Unfortunately I think this is a result of the issue you discovered with the SSLContext not getting created from the properties on the SiteToSiteClientBuilder... Whats happening is the spark side is hitting this: if (siteToSiteSecure) { if (sslContext == null) { throw new

Re: Build a CSV file using MergeContent processor

2016-05-12 Thread Bryan Bende
When using the "text" strategy you may have to do shift+enter in the demarcator field to create a new line. On Thu, May 12, 2016 at 3:52 PM, Joe Witt wrote: > Igor, > > I believe it will encode whatever you give it in UTF-8 and place those > bytes in. For absolute control

Re: Json Split

2016-05-17 Thread Bryan Bende
Hello, I think this would probably be better handled by SplitText with a line count of 1. SplitJson would be more for splitting an array of JSON documents, or a field that is an array. -Bryan On Tue, May 17, 2016 at 12:15 PM, Madhukar Thota wrote: > I have a

Re: Json Split

2016-05-17 Thread Bryan Bende
17, 2016 at 1:29 PM, Andrew Grande <agra...@hortonworks.com> >> wrote: >> >>> Try SplitText with a header line count of 1. It should skip it and give >>> the 2nd line as a result. >>> >>> Andrew >>> >>> From: Madhukar Thota <

Re: Json Split

2016-05-17 Thread Bryan Bende
wrote: >>> >>>> Try SplitText with a header line count of 1. It should skip it and give >>>> the 2nd line as a result. >>>> >>>> Andrew >>>> >>>> From: Madhukar Thota <madhukar.th...@gmail.com> >>&

Re: How to extract mutiple json properties/fields into processor properties?

2016-05-11 Thread Bryan Bende
Hi Keith, If you have a single JSON document and want to extract data from it into attributes, then the processor you would be interested in is EvaluateJSONPath. You add user defined properties to the processor, where the name will be the id of the resulting attribute, and the value will be the

Re: Help Wanted

2016-05-03 Thread Bryan Bende
They are treated with same priority, but as Oleg mentioned, the PRs do make it easier for collaborative review and has the built in integration with Travis, although currently some issues to get it consistently working. On Tue, May 3, 2016 at 11:26 AM, Suneel Marthi wrote: >

Re: Help Wanted

2016-05-03 Thread Bryan Bende
wProcess > page to remove the ambiguity. > > > On Tue, May 3, 2016 at 11:27 AM, Bryan Bende <bbe...@gmail.com> wrote: > >> They are treated with same priority, but as Oleg mentioned, the PRs do >> make it easier for collaborative review and has the built in integra

Re: Failed to receive data from Hbase due to java net connection exception :Connection Refused in Nifi

2016-05-06 Thread Bryan Bende
provenance_repository/123674.prov in 698 milliseconds > 2016-05-06 20:03:16,584 INFO [Provenance Repository Rollover Thread-1] > o.a.n.p.PersistentProvenanceRepository Successfully Rolled over Provenance > Event file containing 2770 records > 2016-05-06 20:03:33,473 INFO [pool-18-thread-1] >

Re: Cannot Authenticate the Azure Datalake Store

2016-05-06 Thread Bryan Bende
Hello, It seems like maybe a wrong version of a library is being used, or maybe a JAR is missing that needs to be included in your NAR. Can you share what dependencies you have in the pom.xml of your processors project? You can check under

Re: Failed to receive data from Hbase due to java net connection exception :Connection Refused in Nifi

2016-05-06 Thread Bryan Bende
fo : > /usr/local/hadoop/hadoop-2.6.0/etc/hadoop/core-site.xml,/usr/local/hadoop/hbase-1.1.2/conf/hbase-site.xml > ZooKeeper QuorumInfo: No value set > ZooKeeper Client PortInfo : No value set > ZooKeeper ZNode ParentInfo : /hbase > HBase Client RetriesInfo : 1 >

Re: Failed to receive data from Hbase due to java net connection exception :Connection Refused in Nifi

2016-05-06 Thread Bryan Bende
or in GetHbase Processor. I update > Hbase-Site.xml and remove remaining properties like " Zookeeper Quorum > Info,Zookeeper Client Port Info . still i got the same error in GetHbase > processor. > > On Thu, May 5, 2016 at 9:07 PM, Bryan Bende <bbe...@gmail.com> wrote: &g

Re: Multi data able joining

2016-05-04 Thread Bryan Bende
Hello, I'm not sure if this will work depending how large the tables are, but since you were already able to move to the data into three separate tables... Could you then do an ExecuteSQL processor that used a select query that joined the three tables together, so the results coming out of the

Re: Passing update processor to PutSolrContentStream

2016-05-04 Thread Bryan Bende
Hi Alex, PutSolrContentStream would likely need the following configuration based on what you described: Solr Type = Cloud Solr Location = Your ZK Collection = feed Content Stream Path = /update Content Type = application/json Add two user defined properties: commit = true update.chain =

Re: Failed to receive data from Hbase due to java net connection exception :Connection Refused in Nifi

2016-05-05 Thread Bryan Bende
Hello, For HBaseClient, are you sure your ZooKeeper being used by HBase is running on localhost:2181? Typically you don't really need to set the three ZooKeeper properties, and you can instead just set the hbase-site.xml in the config resources. For example, my Hadoop Configuration Resources is

Re: Failed to receive data from Hbase due to java net connection exception :Connection Refused in Nifi

2016-05-05 Thread Bryan Bende
at 11:30 AM, Bryan Bende <bbe...@gmail.com> wrote: > Hello, > > For HBaseClient, are you sure your ZooKeeper being used by HBase is > running on localhost:2181? > > Typically you don't really need to set the three ZooKeeper properties, and > you can instead just set the h

  1   2   3   4   5   6   7   >