Re: ListS3 emitted record twice

Neil, I'm not aware of this problem for ListS3. I do not suggest there are no issues, rather that many users might not notice or have come to accept some variance in the accuracy of ListS3. If you can persuade ListS3 to do it again, that would be great :). We did recently hear a report of

RE: [EXT] CDC like updates on Nifi

Alberto, You probably just need to try out the options and see what works best (Avro or ORC, etc…). With the Avro option, you wouldn’t need to change the type of your main HIVE table, keep that as ORC. Only the staging table would use Avro. Then call Hive QL to merge the data from your

Re: Hyphenated Tables and Columns names

Matt, Not sure if it's related. I'm trying to use a timestamp column as Maximum-value Column, but it keeps looping. I have set Use Avro Logical Types = true on my QueryDatabase processor. The original columns values are like this: _Time-Stamp -- 2017-12-01

Re: [EXT] CDC like updates on Nifi

On Tue, Dec 5, 2017 at 11:55 PM, Peter Wicks (pwicks) wrote: > Alberto, > Hello Peter, Thanks for your answer. > > > Since it sounds like you have control over the structure of the tables, > this should be doable. > > > > If you have a changelog table for each table this

Re: Hyphenated Tables and Columns names

Matt, Perfect! Enabled and working now. Thank you! Cheers, Alberto On Wed, Dec 6, 2017 at 3:54 PM, Matt Burgess wrote: > Alberto, > > What version of NiFi are you using? As of version 1.1.0, > QueryDatabaseTable has a "Normalize Table/Column Names" property that > you

Re: Hyphenated Tables and Columns names

Alberto, What version of NiFi are you using? As of version 1.1.0, QueryDatabaseTable has a "Normalize Table/Column Names" property that you can set to true, and it will replace all Avro-illegal characters with underscores. Regards, Matt On Wed, Dec 6, 2017 at 12:06 PM, Alberto Bengoa

Hyphenated Tables and Columns names

Hey Folks, I'm facing an odd situation with Nifi and Tables / Columns that have hyphens on names (traceback below). I found on Avro Spec [1] that hyphens are not allowed, which makes sense to have this error. There is any way to deal with this situation on Nifi instead of changing table/columns

Re: NiFi copying windows .part files

File names are ending with .part (.part is getting suffixed to filename, not prefixed). In case of files from linux file system, “.” (dot) getting prefixed while transfer is in progress. Thanks, Ravi Papisetti On 06/12/17, 1:47 AM, "Joe Witt" wrote: Imagine a

Re: unable to start InvokeHTTP processor in secure Nifi 1.4.0 cluster....

Joe, If I can find some time I would love too, not sure when that may happen. If someone beats me to the punch, it won't hurt my feelings, but if the JIRA is still open when I get some time, I may take a stab at it. Thanks, Josh On Wed, Dec 6, 2017 at 9:19 AM, Joe Witt

Re: unable to start InvokeHTTP processor in secure Nifi 1.4.0 cluster....

Josh - great find and response! Thanks. Any chance you'd like to make a PR for it? On Wed, Dec 6, 2017 at 9:15 AM, dan young wrote: > Heya Josh, > > Awesome! This seemed to get me past at least starting the InvokeHTTP. I > will try the flow out later this morning. Thank

Re: ValidateRecord1.4.0 vs ConvertJsonToAvro1.4.0 regarding required field in nested object

Hey Martin, Thanks for flagging this and for providing the template! It makes it really easy to reproduce. It looks like there is in fact a bug. Whenever the Schema is created from the AvroSchemaRegistry, it is marking all fields except for the top-level fields as as 'nullable'. I created a

Re: unable to start InvokeHTTP processor in secure Nifi 1.4.0 cluster....

Heya Josh, Awesome! This seemed to get me past at least starting the InvokeHTTP. I will try the flow out later this morning. Thank you for the follow-up! Regards, Dano On Tue, Dec 5, 2017 at 10:39 PM Josh Anderton wrote: > Hi Dan/Joe, > > I have encountered the

Re: ValidateRecord1.4.0 vs ConvertJsonToAvro1.4.0 regarding required field in nested object

I don't really understand what you're asking for... In attachment you have nifi template, avro template is: { "name": "aRecord", "type": "record", "namespace": "a", "fields": [ { "name": "a", "type": { "name": "bRecord", "type":"record", "fields":

ValidateRecord1.4.0 vs ConvertJsonToAvro1.4.0 regarding required field in nested object

Hi, I have JSON like: { "a": { "b": "1" } } and corresponding avro schema (written for the sake of this e-mail, need not to be 100% accurate) { "name": "aRecord", "type": "record", "namespace": "a", "fields": [ { "name": "a", "type": { "name": "bRecord",