ListS3 duration

Hello, I'm trying to ingest AWS CloudTrail logs with NiFi. I think I configured ListS3 correctly, but it has been running for hours & hours without showing anything (except for the # of tasks). How long does it take before I should see _any_ output/state/something in the ListS3 processor?

Re: ListS3 duration

te) Are you sure there is data to retrieve more recent that what is > currently in the processor's state? > > Pierre > > 2017-07-20 18:16 GMT+02:00 Laurens Vets <laur...@daemon.be>: > > I'm running 1.3.0 at the moment... I'm tempted to go back to 1.2.0 as I

Re: ListS3 duration

n > like that on buckets with many files, but it was fixed in version 1.1.0 IIRC. > > Hope that helps, > Adam > > On Thu, Jul 20, 2017 at 10:05 AM, Laurens Vets <laur...@daemon.be> wrote: > >> Hello, >> >> I'm trying to ingest AWS CloudTrail logs with NiFi.

Re: ListS3 duration

> On Thu, Jul 20, 2017 at 4:56 PM Laurens Vets <laur...@daemon.be> wrote: > >> Please see inline for my answers and some additional information. >> >>> It sounds like you are doing the right troubleshooting steps. A few >>> more ideas off the top of my h

Re: ListS3 duration

> Sounds like you had some success using the prefix > "AWSLogs//CloudTrail/ap-northeast-1/2017/07/03/", is that right? If > so, when you right-click and choose "View State", is the timestamp and latest > key name stored? > > Try changing the prefix to >

Re: ListS3 duration

t I get. I used 1.3.0 the last time I looked > at Cloudtrail data. > > On Thu, Jul 20, 2017 at 4:56 PM Laurens Vets <laur...@daemon.be> wrote: > >> Please see inline for my answers and some additional information. >> >>> It sounds like you are doing the

*Kafka processors

Hi list, Simple question really, which Kafka processor should I use to stream items into Kafka, more specifically, Kafka as it is used in Apache Metron 0.4.0. The sources are very small (1KB) JSON messages.

Re: ListS3 duration

-07-20 09:31, Laurens Vets wrote: > There's no state currently, ie state is empty. > > I would think that when there's no state, ListS3 would start from the > beginning? > > FYI, the only items I've filled in in the ListS3 processor are: > > - Bucket: Our bu

Re: ListS3 duration

isted objects are skipped. Thanks Joe On Thu, Jul 20, 2017 at 2:44 PM, Laurens Vets <laur...@daemon.be> wrote: I enabled DEBUG logging and I see the following: 2017-07-20 11:39:08,670 DEBUG [StandardProcessScheduler Thread-1] org.apache.nifi.processors.aws.s3.ListS3 ListS3[id=6119854

Re: ListS3 question

dealing with subsets of the bucket > contents. > > Regards, > Joe S > > On Tue, Aug 8, 2017 at 11:22 AM, Laurens Vets <laur...@daemon.be> wrote: > >> Hi list, >> >> Does the ListS3 processor keep state of multiple directories in a bucket? >> >> F

ListS3 question

Hi list, Does the ListS3 processor keep state of multiple directories in a bucket? For instance, suppose I have a directory "logs" with subdirectories "host1", "host2" & "host3". Each directory contains logfiles which are added dailty. Will ListS3 keep state correctly here for all 3

S3 Cloudtrail log ingestion to Kafka via NiFi

Hello, Has anyone been able to ingest S3 Cloudtrail logs into Kafka with NiFi? I got as far ListS3 -> FetchS3Object -> Gunzip, but I'm stuck here. It seems I'm not actually unzipping the logs, but references to the S3 objects? Any help would be appreciated.

Simple check to see whether a file is compressed or not?

Is there a simple way to see whether a file is compressed or not? In my current flow, if an object is not compressed it should bypass the CompressContent processor, otherwise it needs to be decompressed.

Multiple or statements

Hi list, Is it possible to have multiple 'or' statements in the query language? What I want to do is route on a JSON attribute if it doesn't start with 'alpha', 'beta' or 'gamma'. Is it possible to add a 3rd condition to the following somehow: ${field:startsWith('alpha'):or(

Re: Multiple or statements

On Aug 24, 2017, at 11:06 AM, Laurens Vets <laur...@daemon.be> wrote: Hi list, Is it possible to have multiple 'or' statements in the query language? What I want to do is route on a JSON attribute if it doesn't start with 'alpha', 'beta' or 'gamma'. Is it possible to add a 3rd con

Re: Multiple or statements

rtsWith('gamma')} )} Or, alternatively, you could use a regex: ${field:matches( '(alpha|beta|gamma).*' )} Thanks -Mark On Aug 24, 2017, at 11:06 AM, Laurens Vets <laur...@daemon.be> wrote: Hi list, Is it possible to have multiple 'or' statements in the query language? What I want to do is route

Best way to drop data based on its content

Hello list, I'm using NiFi to ship JSON formatted data around. However, I want NiFi to drop certain data when some field matches. Let's say we have a field called 'username' and we don't want to further process any data which contains "username":"laurens". What I currently do is get the

Re: Best way to drop data based on its content

James > > On Mon, Apr 23, 2018 at 2:37 PM, Laurens Vets <laur...@daemon.be> wrote: > >> Hello list, >> >> I'm using NiFi to ship JSON formatted data around. However, I want NiFi to >> drop certain data when some field matches. Let's say we have a field calle

Re: AWS CloudWatch

> Laurens, > > 12 hours seems quite long cycle for the availability of CloudTrail logs! Are > the logs low volume? > > I assume you are using ListS3 -> FetchS3 to currently get the logs from S3? > Am I correct? > > Cheers > > On Sat, Mar 24, 2018 at 5

AWS CloudWatch

Hi list, Has anyone tried to setup NiFi to get real-time CloudWatch logs somehow? I can export CloudWatch logs to S3, but it might take up to 12 hours for them to become available. I suspect the only other option is to go through AWS Kinesis Firehose to stream to S3 and have NiFi pick up the