Re: Review PR 2020 - Add PutKudu Processor for Kudu

[Ricky Saltzer]

Hi Cam -

Thanks for the PR, I'll see if I can take a look this afternoon.

Ricky

On Thu, Jul 27, 2017 at 2:08 PM, Cam Q. Mach <camm...@inspur.com> wrote:

> Can some body help to review this PR?
>
>
>
> Thanks,
>
> Cam
>
>
>
> *From:* Cam Q. Mach
> *Sent:* Friday, July 21, 2017 4:38 PM
> *To:* dev@nifi.apache.org
> *Cc:* William Li <william...@inspur.com>
> *Subject:* RE: Review PR 2020 - Add PutKudu Processor for Kudu
>
>
>
> Anyone interested in reviewing this PR?
>
>
>
> Thanks,
>
> Cam Mach
>
> Inspur USA
>
>
>
> *From:* Cam Q. Mach
> *Sent:* Thursday, July 20, 2017 9:58 AM
> *To:* dev@nifi.apache.org
> *Subject:* Review PR 2020 - Add PutKudu Processor for Kudu
>
>
>
> Hello NiFi Dev,
>
>
>
> Would you please help to review this Pull Request?
>
> https://github.com/apache/nifi/pull/2020
>
>
>
> Jira: https://issues.apache.org/jira/browse/NIFI-3973
>
>
>
> Thanks,
>
> Cam Mach
>
> Inspur USA
>



-- 
Ricky Saltzer
http://www.cloudera.com

RE: [DISCUSS] Proposal for an Apache NiFi sub-project - NiFi Registry

[Ricky Saltzer]

I'm a big +1 to this proposal. It would solve a huge burden that is keeping
NARs up to date in environments where there's alot of teams that share NARs
but have separate NiFi deployments and repositories.

On Feb 8, 2017 7:09 PM, "Peter Wicks (pwicks)"  wrote:

> I think a lot of us are facing the same challenges, and this sounds like a
> step in the right direction.
> I had actually started to dig into a Flow Configuration plugin that would
> use Git branches to copy/sync flows between instances/environments, and
> keep them versioned; hadn't gotten very far.
>
> -Original Message-
> From: Jeremy Dyer [mailto:jdy...@gmail.com]
> Sent: Wednesday, February 08, 2017 3:54 PM
> To: dev@nifi.apache.org
> Subject: Re: [DISCUSS] Proposal for an Apache NiFi sub-project - NiFi
> Registry
>
> Bryan - I think this is a fantastic idea. I would also think this would be
> a good place to add a "device registry" as well. It makes much more sense
> in my mind to have these efforts in sub projects outside of the nifi/minifi
> core.
>
> On Wed, Feb 8, 2017 at 4:50 PM, Bryan Bende  wrote:
>
> > NiFi Community,
> >
> > I'd like to initiate a discussion around creating a sub-project of
> > NiFi to encompass the registry capabilities outlined in several of the
> > feature proposals on the Wiki [1]. A possible name for this
> > sub-project is simply "NiFi Registry".
> >
> > Currently there are two feature proposals that call for NiFi to
> > interact with an external registry:
> >
> > Configuration Management of Flows [2]  - This feature proposal calls
> > for a flow registry where versioned flows can be published and
> > consumed, allowing flows to be easily migrated between environments .
> >
> > Extension Registry [3] - This feature proposal calls for a place to
> > publish NARs containing extensions, allowing NiFi to decouple itself
> > from including all of the NARs in the main distribution, and allowing
> > better discovery of available extensions.
> >
> > The idea would be to create a NiFi Registry sub-project, with
> > sub-modules for the various registries. These registries could then be
> > packaged and distributed as a single artifact and run as a
> > complimentary application to NiFi and MiNiFi. NiFi would not require
> > the registry application, however, a given NiFi could be configured to
> > know about one or more flow registries, or one or more extension
> > registries.
> >
> > Creating a sub-project would allow the registry code to evolve
> > independently of NiFi and be released on it's own timeline. In
> > addition, it would make tracking issues/work much clearer through a
> > separate JIRA.
> >
> > Please discuss and provide and thoughts or feedback.
> >
> > Thanks,
> >
> > Bryan
> >
> > [1] https://cwiki.apache.org/confluence/display/NIFI/NiFi+
> > Feature+Proposals
> > [2] https://cwiki.apache.org/confluence/display/NIFI/
> > Configuration+Management+of+Flows
> > [3] https://cwiki.apache.org/confluence/display/NIFI/
> > Extension+Repositories+%28aka+Extension+Registry%29+for+
> > Dynamically-loaded+Extensions
> >
>

Re: Unable to List Queue with Policy

[Ricky Saltzer]

Joe -

You're a life saver!

Cheers!
Ricky

On Tue, Dec 6, 2016 at 2:21 PM, Joe Gresock <jgres...@gmail.com> wrote:

> Hi Ricky,
>
> I ran into this recently: you have to give your proxying server access to
> View the Data.
>
> On Tue, Dec 6, 2016 at 7:15 PM, Ricky Saltzer <ri...@cloudera.com> wrote:
>
> > Hi -
> >
> > I am struggling with allowing myself to list any connection queue. I have
> > given myself every permission possible for the entire data flow, but even
> > still, I am told I do not have permission to list a data queue. In
> > addition, there is seemingly zero records in the provenance reporter,
> even
> > though I can see the provenance repository is filling up.
> >
> > From what I understand, the main policy I need to grant is "View The
> Data".
> > Is there something else I am missing?
> >
> > Thanks,
> > Ricky
> >
>
>
>
> --
> I know what it is to be in need, and I know what it is to have plenty.  I
> have learned the secret of being content in any and every situation,
> whether well fed or hungry, whether living in plenty or in want.  I can do
> all this through him who gives me strength.*-Philippians 4:12-13*
>



-- 
Ricky Saltzer
http://www.cloudera.com

Unable to List Queue with Policy

[Ricky Saltzer]

Hi -

I am struggling with allowing myself to list any connection queue. I have
given myself every permission possible for the entire data flow, but even
still, I am told I do not have permission to list a data queue. In
addition, there is seemingly zero records in the provenance reporter, even
though I can see the provenance repository is filling up.

>From what I understand, the main policy I need to grant is "View The Data".
Is there something else I am missing?

Thanks,
Ricky

Re: Secure Cluster Mode Issues

[Ricky Saltzer]

Andy -

I'm using version 1.1.0 (official binary). I'm using kerberos
authentication, and able to log in using my internal Kerberos principal. To
be clear, the only difference between blanking out the *keyPasswd *and
*keystorePasswd* was that I was allowed to access the UI without manually
importing the certificate, but instead agreeing to proceed even though I
know the certificate was untrusted.

[image: Inline image 1]
[image: Inline image 2]

Ricky

On Wed, Nov 30, 2016 at 7:43 PM, Andy LoPresto <alopre...@apache.org> wrote:

> Ricky,
>
> Removing the redundant key password property shouldn’t have an impact
> (although you may be running a legacy version before NIFI- [1] and
> NIFI-2466 [2] were fixed). Can you look at the top right of your NiFi UI
> and see what user is accessing the system? It should look like the
> screenshot I have attached. This, and the contents of logs/nifi-user.log,
> will indicate the authenticated user. That should help you figure out how
> the authentication is occurring (client certificate, LDAP, or Kerberos). If
> you still cannot determine it, you can update conf/logback.xml and change
> the logging level for the following loggers from INFO to DEBUG:
>
>  additivity="false">
> 
> 
>  additivity="false">
> 
> 
>  additivity="false">
> 
> 
>
>
> I only ask for this information because your results do not make sense and
> I fear that they will not be reproducible for the rest of your team when
> you try to deploy the system and let them access NiFi and I would hope we
> can provide the best experience from the beginning.
>
> [1] https://issues.apache.org/jira/browse/NIFI-
> [2] https://issues.apache.org/jira/browse/NIFI-2466
>
>
> Andy LoPresto
> alopre...@apache.org
> *alopresto.apa...@gmail.com <alopresto.apa...@gmail.com>*
> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
>
> On Nov 30, 2016, at 11:12 AM, Ricky Saltzer <ri...@cloudera.com> wrote:
>
> Hey Andy -
>
> I think I may have figured out the problem. Although the keystorePasswd and
> keyPasswd are the same, after completely removing the value for
> nifi.security.keyPasswd,and restarting NiFi...I'm able to access the web UI
> without manually importing the certificate.
>
> On Tue, Nov 29, 2016 at 2:19 PM, Andy LoPresto <alopre...@apache.org>
> wrote:
>
> Ricky,
>
> When using HTTPS in non-cluster mode, NiFi still requires user
> authentication — this can be either client certificate (perhaps you already
> had one loaded?), LDAP, or Kerberos. If you are able to access the NiFi UI
> over HTTPS without presenting some authentication, something is seriously
> broken. The warning in the browser is because the CA certificate that
> signed the server certificate (the one being presented *to* the browser
> by the application) is not trusted in the browser’s pre-installed trust
> chain. If, for example, that CA cert had been imported to the browser ahead
> of time, or if it was signed by a publicly known entity like DigiCert,
> Verisign, Comodo, etc., you would not receive a warning.
>
> For small teams, client certificates can be manageable, but if you want to
> allow multiple users to connect with minimal identity management, I
> recommend setting up an LDAP server (OpenLDAP, Microsoft ActiveDirectory,
> Apache Directory Studio, etc.) and administering users there. Then the
> users will just enter a username and password into a login field on their
> first connection to NiFi and be authenticated.
>
>
> Andy LoPresto
> alopre...@apache.org
> *alopresto.apa...@gmail.com <alopresto.apa...@gmail.com>*
> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
>
> On Nov 29, 2016, at 11:07 AM, Ricky Saltzer <ri...@cloudera.com> wrote:
>
> Hey Andy -
>
> Thanks for the reply, I used the openssl command you provided and indeed
> the return code was *OK*. Before proceeding with the recommendation of
>
> importing the key into my OSX keychain, I would like to understand why this
> required. When using HTTPS mode in non-clustered mode, it does not require
> clients to have a special key or cert imported to their machine. Instead,
> the client is given a warning in the browser, and it's up to them to
> proceed. This UI will serve as an endpoint to several users, and I would
> really like to avoid the cumbersomeness of having members of multiple teams
> follow instructions for importing keys just so they can access a web UI.
>
> On Tue, Nov 29, 2016 at 1:55 PM, Andy LoPresto <alopre...@apache.org>
> wrote:
>
> Hi Ricky,
>
> The ERR_CONNECTION_CLOSED is likely because you are not s

Re: Secure Cluster Mode Issues

[Ricky Saltzer]

Hey Andy -

I think I may have figured out the problem. Although the keystorePasswd and
keyPasswd are the same, after completely removing the value for
nifi.security.keyPasswd,and restarting NiFi...I'm able to access the web UI
without manually importing the certificate.

On Tue, Nov 29, 2016 at 2:19 PM, Andy LoPresto <alopre...@apache.org> wrote:

> Ricky,
>
> When using HTTPS in non-cluster mode, NiFi still requires user
> authentication — this can be either client certificate (perhaps you already
> had one loaded?), LDAP, or Kerberos. If you are able to access the NiFi UI
> over HTTPS without presenting some authentication, something is seriously
> broken. The warning in the browser is because the CA certificate that
> signed the server certificate (the one being presented *to* the browser
> by the application) is not trusted in the browser’s pre-installed trust
> chain. If, for example, that CA cert had been imported to the browser ahead
> of time, or if it was signed by a publicly known entity like DigiCert,
> Verisign, Comodo, etc., you would not receive a warning.
>
> For small teams, client certificates can be manageable, but if you want to
> allow multiple users to connect with minimal identity management, I
> recommend setting up an LDAP server (OpenLDAP, Microsoft ActiveDirectory,
> Apache Directory Studio, etc.) and administering users there. Then the
> users will just enter a username and password into a login field on their
> first connection to NiFi and be authenticated.
>
>
> Andy LoPresto
> alopre...@apache.org
> *alopresto.apa...@gmail.com <alopresto.apa...@gmail.com>*
> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
>
> On Nov 29, 2016, at 11:07 AM, Ricky Saltzer <ri...@cloudera.com> wrote:
>
> Hey Andy -
>
> Thanks for the reply, I used the openssl command you provided and indeed
> the return code was *OK*. Before proceeding with the recommendation of
>
> importing the key into my OSX keychain, I would like to understand why this
> required. When using HTTPS mode in non-clustered mode, it does not require
> clients to have a special key or cert imported to their machine. Instead,
> the client is given a warning in the browser, and it's up to them to
> proceed. This UI will serve as an endpoint to several users, and I would
> really like to avoid the cumbersomeness of having members of multiple teams
> follow instructions for importing keys just so they can access a web UI.
>
> On Tue, Nov 29, 2016 at 1:55 PM, Andy LoPresto <alopre...@apache.org>
> wrote:
>
> Hi Ricky,
>
> The ERR_CONNECTION_CLOSED is likely because you are not sending a client
> certificate on the HTTP request. By default, a secured cluster requires
> client certificate authentication unless LDAP or Kerberos are configured as
> identity providers [1]. The TLS Toolkit provides a quick way to generate a
> valid client certificate which you can load into your browser in order to
> access the site.
>
> First, verify the cluster is running and accepting incoming connections
> (we’re going to cheat here just to be quick about it; disclaimer that this
> is not the RIGHT way to do this):
>
> In the directory where you ran the toolkit, you noted there was a
> “nifi-cert.pem” and “nifi-cert.key” file. The pem file is the PEM-encoded
> public certificate of the NiFi CA cert that was generated by the toolkit,
> and the key file is the PEM-encoded private key. Because this is the same
> certificate that signed the NiFi server key loaded in the keystore, it is
> also loaded into the truststore. That means the server will accept an
> incoming connection with any certificate signed by the CA cert.
> Coincidentally, the CA cert is self-signed, so…
>
> $ openssl s_client -connect  -debug -state -cert nifi-cert.pem
> -key nifi-key.key -CAfile nifi-cert.pem
>
> That command will attempt to negotiate a TLS connection to your server by
> presenting the CA cert and key as the client. Again, not semantically
> correct, but  technically will work. You’ll get a long output, but it
> should end in a section like this:
>
> ---
> New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES256-SHA384
> Server public key is 2048 bit
> Secure Renegotiation IS supported
> Compression: NONE
> Expansion: NONE
> No ALPN negotiated
> SSL-Session:
>Protocol  : TLSv1.2
>Cipher: ECDHE-RSA-AES256-SHA384
>Session-ID: 583DCD...9E828C
>Session-ID-ctx:
>Master-Key: 5477C0...A51E85
>Key-Arg   : None
>PSK identity: None
>PSK identity hint: None
>SRP username: None
>Start Time: 1480445265
>Timeout   : 300 (sec)
>Verify return code: 0 (ok)
> ---
>
> The important part is the last line — you want the

Re: Secure Cluster Mode Issues

[Ricky Saltzer]

Hey Andy -

Thanks for the reply, I used the openssl command you provided and indeed
the return code was *OK*. Before proceeding with the recommendation of
importing the key into my OSX keychain, I would like to understand why this
required. When using HTTPS mode in non-clustered mode, it does not require
clients to have a special key or cert imported to their machine. Instead,
the client is given a warning in the browser, and it's up to them to
proceed. This UI will serve as an endpoint to several users, and I would
really like to avoid the cumbersomeness of having members of multiple teams
follow instructions for importing keys just so they can access a web UI.

On Tue, Nov 29, 2016 at 1:55 PM, Andy LoPresto <alopre...@apache.org> wrote:

> Hi Ricky,
>
> The ERR_CONNECTION_CLOSED is likely because you are not sending a client
> certificate on the HTTP request. By default, a secured cluster requires
> client certificate authentication unless LDAP or Kerberos are configured as
> identity providers [1]. The TLS Toolkit provides a quick way to generate a
> valid client certificate which you can load into your browser in order to
> access the site.
>
> First, verify the cluster is running and accepting incoming connections
> (we’re going to cheat here just to be quick about it; disclaimer that this
> is not the RIGHT way to do this):
>
> In the directory where you ran the toolkit, you noted there was a
> “nifi-cert.pem” and “nifi-cert.key” file. The pem file is the PEM-encoded
> public certificate of the NiFi CA cert that was generated by the toolkit,
> and the key file is the PEM-encoded private key. Because this is the same
> certificate that signed the NiFi server key loaded in the keystore, it is
> also loaded into the truststore. That means the server will accept an
> incoming connection with any certificate signed by the CA cert.
> Coincidentally, the CA cert is self-signed, so…
>
> $ openssl s_client -connect  -debug -state -cert nifi-cert.pem
> -key nifi-key.key -CAfile nifi-cert.pem
>
> That command will attempt to negotiate a TLS connection to your server by
> presenting the CA cert and key as the client. Again, not semantically
> correct, but  technically will work. You’ll get a long output, but it
> should end in a section like this:
>
> ---
> New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES256-SHA384
> Server public key is 2048 bit
> Secure Renegotiation IS supported
> Compression: NONE
> Expansion: NONE
> No ALPN negotiated
> SSL-Session:
> Protocol  : TLSv1.2
> Cipher: ECDHE-RSA-AES256-SHA384
> Session-ID: 583DCD...9E828C
> Session-ID-ctx:
> Master-Key: 5477C0...A51E85
> Key-Arg   : None
> PSK identity: None
> PSK identity hint: None
> SRP username: None
> Start Time: 1480445265
> Timeout   : 300 (sec)
> Verify return code: 0 (ok)
> ---
>
> The important part is the last line — you want the *Verify return code* to
> be 0 for success. Once you have verified this, run the TLS toolkit again to
> generate a valid client certificate:
>
> $ ./bin/tls-toolkit.sh standalone -C 'CN=Ricky Saltzer, OU=Apache NiFi'
> -B thisIsABadPassword
>
> This will generate a PKCS12 keystore (*.p12) containing your public
> certificate AND the private key, as well as an additional file (.password)
> containing the password you provided for -B.
>
> hw12203:...lkit/nifi-toolkit-assembly/target/nifi-toolkit-1.1.0-bin/nifi-toolkit-1.1.0
> (master) alopresto
>  146s @ 10:50:14 $ ./bin/tls-toolkit.sh standalone -C 'CN=Ricky Saltzer,
> OU=Apache NiFi' -B password
> 2016/11/29 10:50:20 INFO [main] org.apache.nifi.toolkit.tls.standalone.
> TlsToolkitStandaloneCommandLine: No nifiPropertiesFile specified, using
> embedded one.
> 2016/11/29 10:50:21 INFO [main] 
> org.apache.nifi.toolkit.tls.standalone.TlsToolkitStandalone:
> Running standalone certificate generation with output directory
> ../nifi-toolkit-1.1.0
> 2016/11/29 10:50:21 INFO [main] 
> org.apache.nifi.toolkit.tls.standalone.TlsToolkitStandalone:
> Using existing CA certificate ../nifi-toolkit-1.1.0/nifi-cert.pem and key
> ../nifi-toolkit-1.1.0/nifi-key.key
> 2016/11/29 10:50:21 INFO [main] 
> org.apache.nifi.toolkit.tls.standalone.TlsToolkitStandalone:
> No hostnames specified, not generating any host certificates or
> configuration.
> 2016/11/29 10:50:21 INFO [main] 
> org.apache.nifi.toolkit.tls.standalone.TlsToolkitStandalone:
> Generating new client certificate ../nifi-toolkit-1.1.0/CN=
> Ricky_Saltzer_OU=Apache_NiFi.p12
> 2016/11/29 10:50:21 INFO [main] 
> org.apache.nifi.toolkit.tls.standalone.TlsToolkitStandalone:
> Successfully generated client certificate ../nifi-toolkit-1.1.0/CN=
> Ricky_Saltzer_OU=Apache_NiFi.p12
> 2016/11/29 10:50:21 

Re: Secure Cluster Mode Issues

[Ricky Saltzer]

Hi Andy -

Thanks again for the help, sorry I've been a little preoccupied with some
other tasks. I took your advice and used the tls-toolkit and things are
looking a lot better. I generated a keystore/truststore for both of my
nodes. It also generated a pem and key file. I was able to get NiFi to
start in clustered mode (horray)! The only issue I'm seeing now is that I
can't actually connect to the web UI using HTTPS, as it immediately throws
a "ERR_CONNECTION_CLOSED" message in my browser.

Judging by the logs, NiFi is seemingly running. A leader election took
place and the node is heart beating to itself. Am I missing a step where
I'm supposed to be doing something with the pem/key files?

Thanks again,
Ricky

On Tue, Nov 8, 2016 at 8:22 PM, Andy LoPresto <alopre...@apache.org> wrote:

> Hi Ricky,
>
> Sorry for the delay in my response. I see a couple things which could be
> causing an issue:
>
> 1. You do not specify a hostname for the NiFi node (nifi.web.https.host=)
> which you should do. I have not set up a one-node cluster before, but my
> suspicion is that this field needs to be populated, and the value needs to
> match the CN for the issued server certificate. This value does not need to
> be unique (i.e. it could be nifi.cloudera.com or localhost and you could
> run multiple instances on the same machine, identified by the same
> certificate and different ports), but I think it has to be populated.
> 1.a. While you have nifi.security.needClientAuth=false set in both
> cluster and standalone mode, I am not sure if this allows for the
> truststore to be missing completely. I would have to explore this further.
> There is a current Jira for clarifying cluster, UI/API, and S2S TLS
> configurations [1]. You can try using the default JRE truststore, located
> at “$JAVA_HOME/jre/lib/security/cacerts” with password “changeit”.
> 2. The certificate you have put into the keystore appears to be a
> certificate identifying you (Ricky the person) rather than a server entity.
> You should create a certificate identifying the server itself, so the CN is
> the FQDN as mentioned above. Then import into (or generate directly inside)
> the keystore. You can also use the provided NiFi TLS Toolkit to automate
> this entire process [2].
> Certificate chain
> 0 s:/C=CA/ST=CA/L=Palo Alto/O=Cloudera/OU=EDH Team, Cloudera/CN=Ricky
> Saltzer
> i:/C=CA/ST=CA/L=Palo Alto/O=Cloudera/OU=EDH Team, Cloudera/CN=Ricky 
> Saltzercompare
> to an example connection to google.com:443:
>
> Certificate chain
>  0 s:/C=US/ST=California/L=Mountain View/O=Google Inc/*CN=*.google.com
> <http://google.com>*
>i:/C=US/O=Google Inc/CN=Google Internet Authority G2
>  1 s:/C=US/O=Google Inc/CN=Google Internet Authority G2
>i:/C=US/O=GeoTrust Inc./CN=GeoTrust Global CA
>  2 s:/C=US/O=GeoTrust Inc./CN=GeoTrust Global CA
>i:/C=US/O=Equifax/OU=Equifax Secure Certificate Authority
>
> 3. It appears the certificate you are using is expired. If you look at the
> output of the OpenSSL command you ran, you can see that the result code was
> *10*, not *0* as would be in the case of a successful connection (I
> understand that it looks successful because it negotiates a key and
> encrypts the channel).
> New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES256-SHA384
> Server public key is 2048 bit
> Secure Renegotiation IS supported
> Compression: NONE
> Expansion: NONE
> SSL-Session:
> Protocol : TLSv1.2
> Cipher : ECDHE-RSA-AES256-SHA384
> Session-ID: 581F5A5...41153B
> Session-ID-ctx:
> Master-Key: CEEED2...944C30
> Key-Arg : None
> PSK identity: None
> PSK identity hint: None
> SRP username: None
> Start Time: 1478449748
> Timeout : 300 (sec)
> Verify return code: *10* *(certificate has expired)*
> ---
> HTTP/1.1 400 No URI
> Content-Length: 0
> Connection: close
> Server: Jetty(9.3.9.v20160517)
> I rebuilt your certificate from the Base64-encoded version in the output,
> and it appears it expired on October 25, 2016. You can verify this by
> copying the section between "-BEGIN CERTIFICATE-“ and "-END
> CERTIFICATE-“ (including these markers) into a text file and naming it
> “ricky.pem” and then running the following command:
>
> hw12203:/Users/alopresto/Workspace/scratch (master) alopresto
>  11s @ 17:09:30 $ more ricky.pem
> -BEGIN CERTIFICATE-
> MIIDizCCAnOgAwIBAgIEHPLwBzANBgkqhkiG9w0BAQsFADB2MQswCQYDVQQGEwJD
> ...
> 45GrxLz7n/ZXZk8q9aXcrSGaj+HHCyrO0/9NYtMNP2V5wpYcBRiHO9f3sHKgnzU=
> -END CERTIFICATE-
> hw12203:/Users/alopresto/Workspace/scratch (master) alopresto
>  5s @ 17:09:35 $ openssl x509 -in ricky.pem -text -noout
> Certificate:
> Data:
> Version: 3 (0x2)
> Serial Number: 485683207 (0x1cf2f007)
&g

Re: [VOTE] Release Apache NiFi 1.1.0 (RC2)

[Ricky Saltzer]

+1

Ran verifications, and a sample data flow.

On Mon, Nov 28, 2016 at 12:25 PM, Joe Percivall <
joeperciv...@yahoo.com.invalid> wrote:

> +1 (binding) Release this as nifi-1.1.0
>
> - - - - - - Joseph Percivalllinkedin.com/in/Percivalle:
> joeperciv...@yahoo.com
>
>
> On Monday, November 28, 2016, 10:25:53 AM EST, Bryan Bende <
> bbe...@gmail.com> wrote:+1 (binding) Release this package as nifi-1.1.0
>
> - Verified everything in release helper
> - Ran the resulting binary in standalone mode secure and unsecure and
> verified a couple of test flows
>
> The original email said the git commit was
> f61e42c65e1c2387591ee368c2148cd5bda646bd
> but the commit that actually sets the pom versions to 1.1.0 is the next
> commit
> https://github.com/apache/nifi/commit/5536f690a81418955442d52687695f
> 65f0a44cd0
>
>
>
>
> On Mon, Nov 28, 2016 at 6:44 AM, Scott Aslan <scottyas...@gmail.com>
> wrote:
>
> > +1 (non-binding)
> >
> > - Verified signatures, checksums, LICENSE, NOTICE, and README
> > - Built using a full clean build with contrib check successfully on OSX,
> > with Oracle JDK 1.8.0_77-b03, mvn 3.3.9
> > - Run example data flows on OS X with Chrome Version 54.0.2840.98
> (64-bit)
> > on
> > unsecured instance
> > - Run example data flows on OS X with Chrome Version 54.0.2840.98
> (64-bit)
> > on
> > secured instance (using an old 1.0 secured config)
> >
> >
> > Successfully tested:
> > * Back pressure UI.
> > * New restricted-components permission.
> > * Status Color in the UI
> > * Canvas component fill color
> > * User centric policies view
> >
> > On Mon, Nov 28, 2016 at 9:15 AM, Matt Gilman <matt.c.gil...@gmail.com>
> > wrote:
> >
> > > +1 Release this package as nifi-1.1.0
> > >
> > > Ran through numerous configurations including standalone and clustered
> in
> > > both secure and unsecured mode and all is working as excepted.
> > >
> > > Matt
> > >
> > > On Sat, Nov 26, 2016 at 1:11 AM, Joe Witt <joew...@apache.org> wrote:
> > >
> > > > Hello Apache NiFi Community,
> > > >
> > > > I am pleased to be calling this vote for the source release of Apache
> > > NiFi,
> > > > nifi-1.1.0.
> > > >
> > > > The source release zip and convenience binaries, including signatures
> > > > and digests can be found at:
> > > >  https://dist.apache.org/repos/dist/dev/nifi/1.0.0-rc2/
> > > >
> > > > The Git tag is nifi-1.1.0-RC2
> > > > The Git commit hash is f61e42c65e1c2387591ee368c2148cd5bda646bd
> > > > * https://git-wip-us.apache.org/repos/asf?p=nifi.git;a=commit;h=
> > > > f61e42c65e1c2387591ee368c2148cd5bda646bd
> > > > * https://github.com/apache/nifi/commit/
> f61e42c65e1c2387591ee368c2148c
> > > > d5bda646bd
> > > >
> > > > Checksums of nifi-1.1.0-source-release.zip:
> > > > MD5: 371fb856d9c3947603239ea98f171f6f
> > > > SHA1: 532c2e14e915dfa522254745bbc068aa6620babb
> > > > SHA256: dd1d0569f209fd7f179b85a50fe4bf81b3d850c79b13d32cad88982a8234
> > a719
> > > >
> > > > Release artifacts are signed with the following key:
> > > >  https://people.apache.org/keys/committer/joewitt
> > > >
> > > > KEYS file available here:
> > > >  https://dist.apache.org/repos/dist/release/nifi/KEYS
> > > >
> > > > 268 issues were closed/resolved for this release:
> > > >  https://issues.apache.org/jira/secure/ReleaseNote.jspa?
> > > > projectId=12316020=12337875
> > > >
> > > >
> > > > Release note highlights can be found here:
> > > >  https://cwiki.apache.org/confluence/display/NIFI/
> > > > Release+Notes#ReleaseNotes-Version1.1.0
> > > >
> > > > The vote will be open for 72 hours.
> > > > Please download the release candidate and evaluate the necessary
> items
> > > > including checking hashes, signatures, build from source, and test.
> > Then
> > > > please vote:
> > > >
> > > > [ ] +1 Release this package as nifi-1.1.0
> > > > [ ] +0 no opinion
> > > > [ ] -1 Do not release this package because...
> > > >
> > > > Thanks!
> > > >
> > >
> >
> >
> >
> > --
> > *Scott Aslan = new WebDeveloper(*
> > *{"location": {"city": "Saint Cloud","state": "FL",
> >"zip": "34771"},"contact": {"mobile":
> > "(321)-591-0870","email": "scottyas...@gmail.com
> > <scottyas...@gmail.com>","linkedin":
> > "http://www.linkedin.com/in/scottaslan
> > <http://www.linkedin.com/in/scottaslan>","skype": "astechdev"
> > }});*
> >
>



-- 
Ricky Saltzer
http://www.cloudera.com

Re: Secure Cluster Mode Issues

[Ricky Saltzer]

Hey guys -

I went ahead and uploaded the boostrap log. I took a look at it and it
seems to be the same error [1]

[1]:
https://gist.githubusercontent.com/rickysaltzer/b156594f92066873f80928eddf84e7bb/raw/4d0e018038b332f4fdf14644910dfa9e70c57e49/gistfile1.txt

On Fri, Nov 4, 2016 at 2:14 PM, Mark Payne <marka...@hotmail.com> wrote:

> Hey Ricky,
>
> When you enable debug logging for SSL, it writes to StdErr (or StdOut?) so
> it will end up in your logs/nifi-bootstrap.log instead of nifi-app.log.
> Can you give that a look?
>
> Thanks
> -Mark
>
> > On Nov 4, 2016, at 2:07 PM, Ricky Saltzer <ri...@cloudera.com> wrote:
> >
> > Hey Andy -
> >
> > Thanks for the response. I'm currently just trying to get one node in
> > clustered mode before adding a second. The keystore is stored locally and
> > I've confirmed it's readable, as it was able to start once I took it out
> of
> > clustered mode. I added that line to the bootstrap.conf, but I don't
> > believe any additional logging was produced in regards to troubleshooting
> > this problem. Just in case, I've attached the entire log [1].
> >
> > [1]:
> > https://gist.githubusercontent.com/rickysaltzer/
> ed454d87d2207d5acab401a473d4be57/raw/425c42da762fc5cc997153d48b09f0
> fedabc88bb/gistfile1.txt <https://gist.githubusercontent.com/rickysaltzer/
> ed454d87d2207d5acab401a473d4be57/raw/425c42da762fc5cc997153d48b09f0
> fedabc88bb/gistfile1.txt>
> >
> > On Wed, Nov 2, 2016 at 7:08 PM, Andy LoPresto <alopre...@apache.org
> <mailto:alopre...@apache.org>> wrote:
> >
> >> Hi Ricky,
> >>
> >> Sorry to hear you are having this issue. Is the keystore available on
> all
> >> nodes of the cluster? It appears from the log message that the keystore
> is
> >> not found during startup. To further debug, you can add the following
> line
> >> in bootstrap.conf to provide additional logging:
> >>
> >> java.arg.15=-Djavax.net.debug=ssl,handshake
> >>
> >> Andy LoPresto
> >> alopre...@apache.org <mailto:alopre...@apache.org>
> >> *alopresto.apa...@gmail.com <mailto:alopresto.apa...@gmail.com> <
> alopresto.apa...@gmail.com <mailto:alopresto.apa...@gmail.com>>*
> >> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
> >>
> >> On Nov 2, 2016, at 2:25 PM, Ricky Saltzer <ri...@cloudera.com> wrote:
> >>
> >> Hey all -
> >>
> >> I'm using NiFi 1.0 and I'm having an issue using secure mode with a
> local
> >> key store while in clustered mode. If I set the node in clustered mode,
> and
> >> also provide a valid keystore, I receive a KeyStoreException [1]. If I
> set
> >> the configuration to not use clustered mode, NiFi will start up fine
> with
> >> the provided key store. Am I supposed to be storing this key store in
> >> Zookeeper somewhere?
> >>
> >>
> >> [1]
> >>
> >>
> >> Caused by: java.security.KeyStoreException:  not found
> >>
> >>
> >>   at java.security.KeyStore.getInstance(KeyStore.java:839)
> >> ~[na:1.8.0_11]
> >>
> >>   at
> >> org.apache.nifi.io.socket.SSLContextFactory.(
> >> SSLContextFactory.java:61)
> >> ~[nifi-socket-utils-1.0.0.jar:1.0.0]
> >>
> >>   at
> >> org.apache.nifi.cluster.protocol.spring.ServerSocketConfigurationFacto
> >> ryBean.getObject(ServerSocketConfigurationFactoryBean.java:45)
> >> ~[nifi-framework-cluster-protocol-1.0.0.jar:1.0.0]
> >>
> >>   at
> >> org.apache.nifi.cluster.protocol.spring.ServerSocketConfigurationFacto
> >> ryBean.getObject(ServerSocketConfigurationFactoryBean.java:30)
> >> ~[nifi-framework-cluster-protocol-1.0.0.jar:1.0.0]
> >>
> >>   at
> >> org.springframework.beans.factory.support.FactoryBeanRegistrySupport.
> >> doGetObjectFromFactoryBean(FactoryBeanRegistrySupport.java:168)
> >> ~[spring-beans-4.2.4.RELEASE.jar:4.2.4.RELEASE]
> >>
> >>   ... 69 common frames omitted
> >>
> >> Caused by: java.security.NoSuchAlgorithmException:  KeyStore not
> available
> >>
> >>   at sun.security.jca.GetInstance.getInstance(GetInstance.java:159)
> >> ~[na:1.8.0_11]
> >>
> >>   at java.security.Security.getImpl(Security.java:695)
> ~[na:1.8.0_11]
> >>
> >>   at java.security.KeyStore.getInstance(KeyStore.java:836)
> >> ~[na:1.8.0_11]
> >>
> >>   ... 73 common frames omitted
> >>
> >>
> >>
> >
> >
> > --
> > Ricky Saltzer
> > http://www.cloudera.com <http://www.cloudera.com/>
>



-- 
Ricky Saltzer
http://www.cloudera.com

Re: Secure Cluster Mode Issues

[Ricky Saltzer]

Hey Andy -

Thanks for the response. I'm currently just trying to get one node in
clustered mode before adding a second. The keystore is stored locally and
I've confirmed it's readable, as it was able to start once I took it out of
clustered mode. I added that line to the bootstrap.conf, but I don't
believe any additional logging was produced in regards to troubleshooting
this problem. Just in case, I've attached the entire log [1].

[1]:
https://gist.githubusercontent.com/rickysaltzer/ed454d87d2207d5acab401a473d4be57/raw/425c42da762fc5cc997153d48b09f0fedabc88bb/gistfile1.txt

On Wed, Nov 2, 2016 at 7:08 PM, Andy LoPresto <alopre...@apache.org> wrote:

> Hi Ricky,
>
> Sorry to hear you are having this issue. Is the keystore available on all
> nodes of the cluster? It appears from the log message that the keystore is
> not found during startup. To further debug, you can add the following line
> in bootstrap.conf to provide additional logging:
>
> java.arg.15=-Djavax.net.debug=ssl,handshake
>
> Andy LoPresto
> alopre...@apache.org
> *alopresto.apa...@gmail.com <alopresto.apa...@gmail.com>*
> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
>
> On Nov 2, 2016, at 2:25 PM, Ricky Saltzer <ri...@cloudera.com> wrote:
>
> Hey all -
>
> I'm using NiFi 1.0 and I'm having an issue using secure mode with a local
> key store while in clustered mode. If I set the node in clustered mode, and
> also provide a valid keystore, I receive a KeyStoreException [1]. If I set
> the configuration to not use clustered mode, NiFi will start up fine with
> the provided key store. Am I supposed to be storing this key store in
> Zookeeper somewhere?
>
>
> [1]
>
>
> Caused by: java.security.KeyStoreException:  not found
>
>
>at java.security.KeyStore.getInstance(KeyStore.java:839)
> ~[na:1.8.0_11]
>
>at
> org.apache.nifi.io.socket.SSLContextFactory.(
> SSLContextFactory.java:61)
> ~[nifi-socket-utils-1.0.0.jar:1.0.0]
>
>at
> org.apache.nifi.cluster.protocol.spring.ServerSocketConfigurationFacto
> ryBean.getObject(ServerSocketConfigurationFactoryBean.java:45)
> ~[nifi-framework-cluster-protocol-1.0.0.jar:1.0.0]
>
>at
> org.apache.nifi.cluster.protocol.spring.ServerSocketConfigurationFacto
> ryBean.getObject(ServerSocketConfigurationFactoryBean.java:30)
> ~[nifi-framework-cluster-protocol-1.0.0.jar:1.0.0]
>
>at
> org.springframework.beans.factory.support.FactoryBeanRegistrySupport.
> doGetObjectFromFactoryBean(FactoryBeanRegistrySupport.java:168)
> ~[spring-beans-4.2.4.RELEASE.jar:4.2.4.RELEASE]
>
>... 69 common frames omitted
>
> Caused by: java.security.NoSuchAlgorithmException:  KeyStore not available
>
>at sun.security.jca.GetInstance.getInstance(GetInstance.java:159)
> ~[na:1.8.0_11]
>
>    at java.security.Security.getImpl(Security.java:695) ~[na:1.8.0_11]
>
>at java.security.KeyStore.getInstance(KeyStore.java:836)
> ~[na:1.8.0_11]
>
>... 73 common frames omitted
>
>
>


-- 
Ricky Saltzer
http://www.cloudera.com

Secure Cluster Mode Issues

[Ricky Saltzer]

Hey all -

I'm using NiFi 1.0 and I'm having an issue using secure mode with a local
key store while in clustered mode. If I set the node in clustered mode, and
also provide a valid keystore, I receive a KeyStoreException [1]. If I set
the configuration to not use clustered mode, NiFi will start up fine with
the provided key store. Am I supposed to be storing this key store in
Zookeeper somewhere?


[1]


Caused by: java.security.KeyStoreException:  not found


at java.security.KeyStore.getInstance(KeyStore.java:839)
~[na:1.8.0_11]

at
org.apache.nifi.io.socket.SSLContextFactory.(SSLContextFactory.java:61)
~[nifi-socket-utils-1.0.0.jar:1.0.0]

at
org.apache.nifi.cluster.protocol.spring.ServerSocketConfigurationFactoryBean.getObject(ServerSocketConfigurationFactoryBean.java:45)
~[nifi-framework-cluster-protocol-1.0.0.jar:1.0.0]

at
org.apache.nifi.cluster.protocol.spring.ServerSocketConfigurationFactoryBean.getObject(ServerSocketConfigurationFactoryBean.java:30)
~[nifi-framework-cluster-protocol-1.0.0.jar:1.0.0]

at
org.springframework.beans.factory.support.FactoryBeanRegistrySupport.doGetObjectFromFactoryBean(FactoryBeanRegistrySupport.java:168)
~[spring-beans-4.2.4.RELEASE.jar:4.2.4.RELEASE]

... 69 common frames omitted

Caused by: java.security.NoSuchAlgorithmException:  KeyStore not available

at sun.security.jca.GetInstance.getInstance(GetInstance.java:159)
~[na:1.8.0_11]

at java.security.Security.getImpl(Security.java:695) ~[na:1.8.0_11]

at java.security.KeyStore.getInstance(KeyStore.java:836)
~[na:1.8.0_11]

... 73 common frames omitted

Re: Content Repository Cleanup Schedule

[Ricky Saltzer]

Hey Mark -

Yeah, agreed. I'm moving some of the 15+ day old files out just because
this is kind of an emergency. Yeah, that's not exactly "normal", but I have
a new pipeline that batches up errors and the FlowFile is basically 0-bytes
with attribute information regarding the error so they can be retried at a
later time.

There's definitely some very large files in that content repo, the sizes
vary from KB to several GB.

$ find . -size +1G | wc -l
163

On Wed, Jun 15, 2016 at 5:13 PM, Mark Payne <marka...@hotmail.com> wrote:

> Deleting the old files could certainly cause some problems.
>
> The weird thing is that it shows that you have 10,000+ FlowFiles, each of
> which is 0 bytes.
> Is that normal for your flow?
>
> Could you try running the following against your content repo:
>
> find . -size +1M
>
> find . | wc -l
>
> Curious how many files there are and how many are "large" files.
>
>
>
> > On Jun 15, 2016, at 5:02 PM, Ricky Saltzer <ri...@cloudera.com> wrote:
> >
> > Is it safe to manually remove some of the older files in the repository
> to
> > avoid our disk from filling up?
> >
> > On Wed, Jun 15, 2016 at 4:55 PM, Ricky Saltzer <ri...@cloudera.com>
> wrote:
> >
> >> Just a reminder, I just today noticed the "archive.enabled" option was
> >> false and changed it to true.
> >>
> >> $ find . -type f -ls | grep archive | wc -l
> >> 0
> >>
> >>
> >>
> >> On Wed, Jun 15, 2016 at 4:53 PM, Mark Payne <marka...@hotmail.com>
> wrote:
> >>
> >>> OK, thanks. It doesn't appear that it believes there is anything to
> >>> reclaim.
> >>>
> >>> Can you try going to your content repository and running:
> >>>
> >>> find . -type f -ls | grep archive
> >>>
> >>> Curious as to how much data it has archived.
> >>>
> >>>> On Jun 15, 2016, at 4:48 PM, Ricky Saltzer <ri...@cloudera.com>
> wrote:
> >>>>
> >>>> Oh sorry! Trying again
> >>>>
> >>>> [1]
> >>>>
> >>>
> https://gist.githubusercontent.com/rickysaltzer/b00196a3881c052df9b38b418722cd02/raw/279a1bc8c60530426732eb7b653de1f3f74574e2/gistfile1.txt
> >>>>
> >>>>
> >>>> On Wed, Jun 15, 2016 at 4:38 PM, Ricky Saltzer <ri...@cloudera.com>
> >>> wrote:
> >>>>
> >>>>> I should also mention, I just realized that our worker nodes are on
> >>> 0.5.1,
> >>>>> and for some reason I missed updating the master from 0.4.0. I'm sure
> >>> that
> >>>>> is not helping.
> >>>>>
> >>>>> On Wed, Jun 15, 2016 at 4:36 PM, Ricky Saltzer <ri...@cloudera.com>
> >>> wrote:
> >>>>>
> >>>>>> Looks like the threads are parked and waiting [1]
> >>>>>>
> >>>>>> [1]
> >>>>>>
> >>>
> http://github.mtv.cloudera.com/gist/ricky/7a5d89f2eeba58e2206d/raw/0e2b446ca049a8b5f27298c700ac709772d2847c/gistfile1.txt
> >>>>>>
> >>>>>> On Wed, Jun 15, 2016 at 4:33 PM, Joe Witt <joe.w...@gmail.com>
> wrote:
> >>>>>>
> >>>>>>> thanks Ricky - then please take a look at mark's note as that is
> >>>>>>> probably more relevant to your case.
> >>>>>>>
> >>>>>>> On Wed, Jun 15, 2016 at 4:32 PM, Ricky Saltzer <ri...@cloudera.com
> >
> >>>>>>> wrote:
> >>>>>>>> Hey Joe -
> >>>>>>>>
> >>>>>>>> The NiFi web UI currently reads as:
> >>>>>>>>
> >>>>>>>> Active threads: 3
> >>>>>>>> Queued: 10,173 / 0 bytes
> >>>>>>>> Connected nodes: 2 / 2
> >>>>>>>> Stats last refreshed: 13:31:28 PDT
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> On Wed, Jun 15, 2016 at 4:29 PM, Joe Witt <joe.w...@gmail.com>
> >>> wrote:
> >>>>>>>>
> >>>>>>>>> And the data remains?  If so that is an interesting data point I
> >>>>>>>>> think.  So to mark's point how much data do you have queued up
> >>>>>>>>> actively in the flow then on that nodes?  Number of obje

Re: Content Repository Cleanup Schedule

[Ricky Saltzer]

Is it safe to manually remove some of the older files in the repository to
avoid our disk from filling up?

On Wed, Jun 15, 2016 at 4:55 PM, Ricky Saltzer <ri...@cloudera.com> wrote:

> Just a reminder, I just today noticed the "archive.enabled" option was
> false and changed it to true.
>
> $ find . -type f -ls | grep archive | wc -l
> 0
>
>
>
> On Wed, Jun 15, 2016 at 4:53 PM, Mark Payne <marka...@hotmail.com> wrote:
>
>> OK, thanks. It doesn't appear that it believes there is anything to
>> reclaim.
>>
>> Can you try going to your content repository and running:
>>
>> find . -type f -ls | grep archive
>>
>> Curious as to how much data it has archived.
>>
>> > On Jun 15, 2016, at 4:48 PM, Ricky Saltzer <ri...@cloudera.com> wrote:
>> >
>> > Oh sorry! Trying again
>> >
>> > [1]
>> >
>> https://gist.githubusercontent.com/rickysaltzer/b00196a3881c052df9b38b418722cd02/raw/279a1bc8c60530426732eb7b653de1f3f74574e2/gistfile1.txt
>> >
>> >
>> > On Wed, Jun 15, 2016 at 4:38 PM, Ricky Saltzer <ri...@cloudera.com>
>> wrote:
>> >
>> >> I should also mention, I just realized that our worker nodes are on
>> 0.5.1,
>> >> and for some reason I missed updating the master from 0.4.0. I'm sure
>> that
>> >> is not helping.
>> >>
>> >> On Wed, Jun 15, 2016 at 4:36 PM, Ricky Saltzer <ri...@cloudera.com>
>> wrote:
>> >>
>> >>> Looks like the threads are parked and waiting [1]
>> >>>
>> >>> [1]
>> >>>
>> http://github.mtv.cloudera.com/gist/ricky/7a5d89f2eeba58e2206d/raw/0e2b446ca049a8b5f27298c700ac709772d2847c/gistfile1.txt
>> >>>
>> >>> On Wed, Jun 15, 2016 at 4:33 PM, Joe Witt <joe.w...@gmail.com> wrote:
>> >>>
>> >>>> thanks Ricky - then please take a look at mark's note as that is
>> >>>> probably more relevant to your case.
>> >>>>
>> >>>> On Wed, Jun 15, 2016 at 4:32 PM, Ricky Saltzer <ri...@cloudera.com>
>> >>>> wrote:
>> >>>>> Hey Joe -
>> >>>>>
>> >>>>> The NiFi web UI currently reads as:
>> >>>>>
>> >>>>> Active threads: 3
>> >>>>> Queued: 10,173 / 0 bytes
>> >>>>> Connected nodes: 2 / 2
>> >>>>> Stats last refreshed: 13:31:28 PDT
>> >>>>>
>> >>>>>
>> >>>>> On Wed, Jun 15, 2016 at 4:29 PM, Joe Witt <joe.w...@gmail.com>
>> wrote:
>> >>>>>
>> >>>>>> And the data remains?  If so that is an interesting data point I
>> >>>>>> think.  So to mark's point how much data do you have queued up
>> >>>>>> actively in the flow then on that nodes?  Number of objects you
>> >>>>>> mention is 3273 files corresponding to 825GB in the content
>> >>>>>> repository.  Does NiFi see those 825GB worth of data as being in
>> the
>> >>>>>> flow/queued up?  And then if that is the case are we talking about
>> a
>> >>>>>> roughly 1TB repo and so the reported value seems correct and this
>> is
>> >>>>>> simply a case of queueing near to the limit your system can hold?
>> >>>>>>
>> >>>>>> On Wed, Jun 15, 2016 at 4:24 PM, Ricky Saltzer <ri...@cloudera.com
>> >
>> >>>> wrote:
>> >>>>>>> I have two nodes in clustered mode. I have the other node that
>> isn't
>> >>>>>>> filling up as my primary. I've actually already restarted nifi on
>> >>>> the
>> >>>>>> node
>> >>>>>>> which has the large repository a few times.
>> >>>>>>>
>> >>>>>>> On Wed, Jun 15, 2016 at 4:22 PM, Joe Witt <joe.w...@gmail.com>
>> >>>> wrote:
>> >>>>>>>
>> >>>>>>>> Ricky,
>> >>>>>>>>
>> >>>>>>>> If you restart nifi and then find that it cleans those things up
>> I
>> >>>>>>>> believe then it is related to the defects corrected in the
>> 0.5/0.6
>> >>>>>>>> timeframe.
>> >>>>>>>>
>> >>>

Re: Content Repository Cleanup Schedule

[Ricky Saltzer]

Just a reminder, I just today noticed the "archive.enabled" option was
false and changed it to true.

$ find . -type f -ls | grep archive | wc -l
0



On Wed, Jun 15, 2016 at 4:53 PM, Mark Payne <marka...@hotmail.com> wrote:

> OK, thanks. It doesn't appear that it believes there is anything to
> reclaim.
>
> Can you try going to your content repository and running:
>
> find . -type f -ls | grep archive
>
> Curious as to how much data it has archived.
>
> > On Jun 15, 2016, at 4:48 PM, Ricky Saltzer <ri...@cloudera.com> wrote:
> >
> > Oh sorry! Trying again
> >
> > [1]
> >
> https://gist.githubusercontent.com/rickysaltzer/b00196a3881c052df9b38b418722cd02/raw/279a1bc8c60530426732eb7b653de1f3f74574e2/gistfile1.txt
> >
> >
> > On Wed, Jun 15, 2016 at 4:38 PM, Ricky Saltzer <ri...@cloudera.com>
> wrote:
> >
> >> I should also mention, I just realized that our worker nodes are on
> 0.5.1,
> >> and for some reason I missed updating the master from 0.4.0. I'm sure
> that
> >> is not helping.
> >>
> >> On Wed, Jun 15, 2016 at 4:36 PM, Ricky Saltzer <ri...@cloudera.com>
> wrote:
> >>
> >>> Looks like the threads are parked and waiting [1]
> >>>
> >>> [1]
> >>>
> http://github.mtv.cloudera.com/gist/ricky/7a5d89f2eeba58e2206d/raw/0e2b446ca049a8b5f27298c700ac709772d2847c/gistfile1.txt
> >>>
> >>> On Wed, Jun 15, 2016 at 4:33 PM, Joe Witt <joe.w...@gmail.com> wrote:
> >>>
> >>>> thanks Ricky - then please take a look at mark's note as that is
> >>>> probably more relevant to your case.
> >>>>
> >>>> On Wed, Jun 15, 2016 at 4:32 PM, Ricky Saltzer <ri...@cloudera.com>
> >>>> wrote:
> >>>>> Hey Joe -
> >>>>>
> >>>>> The NiFi web UI currently reads as:
> >>>>>
> >>>>> Active threads: 3
> >>>>> Queued: 10,173 / 0 bytes
> >>>>> Connected nodes: 2 / 2
> >>>>> Stats last refreshed: 13:31:28 PDT
> >>>>>
> >>>>>
> >>>>> On Wed, Jun 15, 2016 at 4:29 PM, Joe Witt <joe.w...@gmail.com>
> wrote:
> >>>>>
> >>>>>> And the data remains?  If so that is an interesting data point I
> >>>>>> think.  So to mark's point how much data do you have queued up
> >>>>>> actively in the flow then on that nodes?  Number of objects you
> >>>>>> mention is 3273 files corresponding to 825GB in the content
> >>>>>> repository.  Does NiFi see those 825GB worth of data as being in the
> >>>>>> flow/queued up?  And then if that is the case are we talking about a
> >>>>>> roughly 1TB repo and so the reported value seems correct and this is
> >>>>>> simply a case of queueing near to the limit your system can hold?
> >>>>>>
> >>>>>> On Wed, Jun 15, 2016 at 4:24 PM, Ricky Saltzer <ri...@cloudera.com>
> >>>> wrote:
> >>>>>>> I have two nodes in clustered mode. I have the other node that
> isn't
> >>>>>>> filling up as my primary. I've actually already restarted nifi on
> >>>> the
> >>>>>> node
> >>>>>>> which has the large repository a few times.
> >>>>>>>
> >>>>>>> On Wed, Jun 15, 2016 at 4:22 PM, Joe Witt <joe.w...@gmail.com>
> >>>> wrote:
> >>>>>>>
> >>>>>>>> Ricky,
> >>>>>>>>
> >>>>>>>> If you restart nifi and then find that it cleans those things up I
> >>>>>>>> believe then it is related to the defects corrected in the 0.5/0.6
> >>>>>>>> timeframe.
> >>>>>>>>
> >>>>>>>> Is restarting an option for you at this time.  You agree mark?
> >>>>>>>>
> >>>>>>>> Thanks
> >>>>>>>> Joe
> >>>>>>>>
> >>>>>>>> On Wed, Jun 15, 2016 at 4:21 PM, Ricky Saltzer <
> ri...@cloudera.com
> >>>>>
> >>>>>> wrote:
> >>>>>>>>> Hey Mark -
> >>>>>>>>>
> >>>>>>>>> Thanks for the quick reply! This is our production system so it

Re: Content Repository Cleanup Schedule

[Ricky Saltzer]

Oh sorry! Trying again

[1]
https://gist.githubusercontent.com/rickysaltzer/b00196a3881c052df9b38b418722cd02/raw/279a1bc8c60530426732eb7b653de1f3f74574e2/gistfile1.txt


On Wed, Jun 15, 2016 at 4:38 PM, Ricky Saltzer <ri...@cloudera.com> wrote:

> I should also mention, I just realized that our worker nodes are on 0.5.1,
> and for some reason I missed updating the master from 0.4.0. I'm sure that
> is not helping.
>
> On Wed, Jun 15, 2016 at 4:36 PM, Ricky Saltzer <ri...@cloudera.com> wrote:
>
>> Looks like the threads are parked and waiting [1]
>>
>> [1]
>> http://github.mtv.cloudera.com/gist/ricky/7a5d89f2eeba58e2206d/raw/0e2b446ca049a8b5f27298c700ac709772d2847c/gistfile1.txt
>>
>> On Wed, Jun 15, 2016 at 4:33 PM, Joe Witt <joe.w...@gmail.com> wrote:
>>
>>> thanks Ricky - then please take a look at mark's note as that is
>>> probably more relevant to your case.
>>>
>>> On Wed, Jun 15, 2016 at 4:32 PM, Ricky Saltzer <ri...@cloudera.com>
>>> wrote:
>>> > Hey Joe -
>>> >
>>> > The NiFi web UI currently reads as:
>>> >
>>> > Active threads: 3
>>> > Queued: 10,173 / 0 bytes
>>> > Connected nodes: 2 / 2
>>> > Stats last refreshed: 13:31:28 PDT
>>> >
>>> >
>>> > On Wed, Jun 15, 2016 at 4:29 PM, Joe Witt <joe.w...@gmail.com> wrote:
>>> >
>>> >> And the data remains?  If so that is an interesting data point I
>>> >> think.  So to mark's point how much data do you have queued up
>>> >> actively in the flow then on that nodes?  Number of objects you
>>> >> mention is 3273 files corresponding to 825GB in the content
>>> >> repository.  Does NiFi see those 825GB worth of data as being in the
>>> >> flow/queued up?  And then if that is the case are we talking about a
>>> >> roughly 1TB repo and so the reported value seems correct and this is
>>> >> simply a case of queueing near to the limit your system can hold?
>>> >>
>>> >> On Wed, Jun 15, 2016 at 4:24 PM, Ricky Saltzer <ri...@cloudera.com>
>>> wrote:
>>> >> > I have two nodes in clustered mode. I have the other node that isn't
>>> >> > filling up as my primary. I've actually already restarted nifi on
>>> the
>>> >> node
>>> >> > which has the large repository a few times.
>>> >> >
>>> >> > On Wed, Jun 15, 2016 at 4:22 PM, Joe Witt <joe.w...@gmail.com>
>>> wrote:
>>> >> >
>>> >> >> Ricky,
>>> >> >>
>>> >> >> If you restart nifi and then find that it cleans those things up I
>>> >> >> believe then it is related to the defects corrected in the 0.5/0.6
>>> >> >> timeframe.
>>> >> >>
>>> >> >> Is restarting an option for you at this time.  You agree mark?
>>> >> >>
>>> >> >> Thanks
>>> >> >> Joe
>>> >> >>
>>> >> >> On Wed, Jun 15, 2016 at 4:21 PM, Ricky Saltzer <ri...@cloudera.com
>>> >
>>> >> wrote:
>>> >> >> > Hey Mark -
>>> >> >> >
>>> >> >> > Thanks for the quick reply! This is our production system so it's
>>> >> >> > unfortunately running 0.4.0. There are currently 3273 files,
>>> with some
>>> >> >> > files dating back to May 18th. The content repository itself is
>>> 825G.
>>> >> >> >
>>> >> >> > Ricky
>>> >> >> >
>>> >> >> > On Wed, Jun 15, 2016 at 4:17 PM, Mark Payne <
>>> marka...@hotmail.com>
>>> >> >> wrote:
>>> >> >> >
>>> >> >> >> Hey Ricky
>>> >> >> >>
>>> >> >> >> The reclaim process is pretty much continuous. What version of
>>> NiFi
>>> >> are
>>> >> >> >> you running?
>>> >> >> >> I know there was an issue with this a while back that caused it
>>> not
>>> >> to
>>> >> >> >> cleanup properly.
>>> >> >> >>
>>> >> >> >> Also, how much data & how many FlowFiles do you have queued up
>>> in
>>> >> your
>>> >> >> >> flow?
>>> >> >> >> Data won't be archived or reclaimed if in the flow.
>>> >> >> >>
>>> >> >> >> Thanks
>>> >> >> >> -Mark
>>> >> >> >>
>>> >> >> >>
>>> >> >> >>
>>> >> >> >> > On Jun 15, 2016, at 4:04 PM, Ricky Saltzer <
>>> ri...@cloudera.com>
>>> >> >> wrote:
>>> >> >> >> >
>>> >> >> >> > Hey guys -
>>> >> >> >> >
>>> >> >> >> > I recently discovered I didn't have my "archive.enabled"
>>> option
>>> >> set to
>>> >> >> >> true
>>> >> >> >> > after my disk filled up to 95%. I enabled it and then set the
>>> >> >> retention
>>> >> >> >> > period to 12 hours and 50% (default values). However, after
>>> >> restarting
>>> >> >> >> > NiFi, I am not seeing any disk space reclaimed.
>>> >> >> >> >
>>> >> >> >> > I'm curious, is the reclaiming process periodic or continuous?
>>> >> >> >> >
>>> >> >> >> > ---
>>> >> >> >> > ricky
>>> >> >> >>
>>> >> >> >>
>>> >> >> >
>>> >> >> >
>>> >> >> > --
>>> >> >> > Ricky Saltzer
>>> >> >> > http://www.cloudera.com
>>> >> >>
>>> >> >
>>> >> >
>>> >> >
>>> >> > --
>>> >> > Ricky Saltzer
>>> >> > http://www.cloudera.com
>>> >>
>>> >
>>> >
>>> >
>>> > --
>>> > Ricky Saltzer
>>> > http://www.cloudera.com
>>>
>>
>>
>>
>> --
>> Ricky Saltzer
>> http://www.cloudera.com
>>
>>
>
>
> --
> Ricky Saltzer
> http://www.cloudera.com
>
>


-- 
Ricky Saltzer
http://www.cloudera.com

Re: Content Repository Cleanup Schedule

[Ricky Saltzer]

I should also mention, I just realized that our worker nodes are on 0.5.1,
and for some reason I missed updating the master from 0.4.0. I'm sure that
is not helping.

On Wed, Jun 15, 2016 at 4:36 PM, Ricky Saltzer <ri...@cloudera.com> wrote:

> Looks like the threads are parked and waiting [1]
>
> [1]
> http://github.mtv.cloudera.com/gist/ricky/7a5d89f2eeba58e2206d/raw/0e2b446ca049a8b5f27298c700ac709772d2847c/gistfile1.txt
>
> On Wed, Jun 15, 2016 at 4:33 PM, Joe Witt <joe.w...@gmail.com> wrote:
>
>> thanks Ricky - then please take a look at mark's note as that is
>> probably more relevant to your case.
>>
>> On Wed, Jun 15, 2016 at 4:32 PM, Ricky Saltzer <ri...@cloudera.com>
>> wrote:
>> > Hey Joe -
>> >
>> > The NiFi web UI currently reads as:
>> >
>> > Active threads: 3
>> > Queued: 10,173 / 0 bytes
>> > Connected nodes: 2 / 2
>> > Stats last refreshed: 13:31:28 PDT
>> >
>> >
>> > On Wed, Jun 15, 2016 at 4:29 PM, Joe Witt <joe.w...@gmail.com> wrote:
>> >
>> >> And the data remains?  If so that is an interesting data point I
>> >> think.  So to mark's point how much data do you have queued up
>> >> actively in the flow then on that nodes?  Number of objects you
>> >> mention is 3273 files corresponding to 825GB in the content
>> >> repository.  Does NiFi see those 825GB worth of data as being in the
>> >> flow/queued up?  And then if that is the case are we talking about a
>> >> roughly 1TB repo and so the reported value seems correct and this is
>> >> simply a case of queueing near to the limit your system can hold?
>> >>
>> >> On Wed, Jun 15, 2016 at 4:24 PM, Ricky Saltzer <ri...@cloudera.com>
>> wrote:
>> >> > I have two nodes in clustered mode. I have the other node that isn't
>> >> > filling up as my primary. I've actually already restarted nifi on the
>> >> node
>> >> > which has the large repository a few times.
>> >> >
>> >> > On Wed, Jun 15, 2016 at 4:22 PM, Joe Witt <joe.w...@gmail.com>
>> wrote:
>> >> >
>> >> >> Ricky,
>> >> >>
>> >> >> If you restart nifi and then find that it cleans those things up I
>> >> >> believe then it is related to the defects corrected in the 0.5/0.6
>> >> >> timeframe.
>> >> >>
>> >> >> Is restarting an option for you at this time.  You agree mark?
>> >> >>
>> >> >> Thanks
>> >> >> Joe
>> >> >>
>> >> >> On Wed, Jun 15, 2016 at 4:21 PM, Ricky Saltzer <ri...@cloudera.com>
>> >> wrote:
>> >> >> > Hey Mark -
>> >> >> >
>> >> >> > Thanks for the quick reply! This is our production system so it's
>> >> >> > unfortunately running 0.4.0. There are currently 3273 files, with
>> some
>> >> >> > files dating back to May 18th. The content repository itself is
>> 825G.
>> >> >> >
>> >> >> > Ricky
>> >> >> >
>> >> >> > On Wed, Jun 15, 2016 at 4:17 PM, Mark Payne <marka...@hotmail.com
>> >
>> >> >> wrote:
>> >> >> >
>> >> >> >> Hey Ricky
>> >> >> >>
>> >> >> >> The reclaim process is pretty much continuous. What version of
>> NiFi
>> >> are
>> >> >> >> you running?
>> >> >> >> I know there was an issue with this a while back that caused it
>> not
>> >> to
>> >> >> >> cleanup properly.
>> >> >> >>
>> >> >> >> Also, how much data & how many FlowFiles do you have queued up in
>> >> your
>> >> >> >> flow?
>> >> >> >> Data won't be archived or reclaimed if in the flow.
>> >> >> >>
>> >> >> >> Thanks
>> >> >> >> -Mark
>> >> >> >>
>> >> >> >>
>> >> >> >>
>> >> >> >> > On Jun 15, 2016, at 4:04 PM, Ricky Saltzer <ri...@cloudera.com
>> >
>> >> >> wrote:
>> >> >> >> >
>> >> >> >> > Hey guys -
>> >> >> >> >
>> >> >> >> > I recently discovered I didn't have my "archive.enabled" option
>> >> set to
>> >> >> >> true
>> >> >> >> > after my disk filled up to 95%. I enabled it and then set the
>> >> >> retention
>> >> >> >> > period to 12 hours and 50% (default values). However, after
>> >> restarting
>> >> >> >> > NiFi, I am not seeing any disk space reclaimed.
>> >> >> >> >
>> >> >> >> > I'm curious, is the reclaiming process periodic or continuous?
>> >> >> >> >
>> >> >> >> > ---
>> >> >> >> > ricky
>> >> >> >>
>> >> >> >>
>> >> >> >
>> >> >> >
>> >> >> > --
>> >> >> > Ricky Saltzer
>> >> >> > http://www.cloudera.com
>> >> >>
>> >> >
>> >> >
>> >> >
>> >> > --
>> >> > Ricky Saltzer
>> >> > http://www.cloudera.com
>> >>
>> >
>> >
>> >
>> > --
>> > Ricky Saltzer
>> > http://www.cloudera.com
>>
>
>
>
> --
> Ricky Saltzer
> http://www.cloudera.com
>
>


-- 
Ricky Saltzer
http://www.cloudera.com

Re: Content Repository Cleanup Schedule

[Ricky Saltzer]

Looks like the threads are parked and waiting [1]

[1]
http://github.mtv.cloudera.com/gist/ricky/7a5d89f2eeba58e2206d/raw/0e2b446ca049a8b5f27298c700ac709772d2847c/gistfile1.txt

On Wed, Jun 15, 2016 at 4:33 PM, Joe Witt <joe.w...@gmail.com> wrote:

> thanks Ricky - then please take a look at mark's note as that is
> probably more relevant to your case.
>
> On Wed, Jun 15, 2016 at 4:32 PM, Ricky Saltzer <ri...@cloudera.com> wrote:
> > Hey Joe -
> >
> > The NiFi web UI currently reads as:
> >
> > Active threads: 3
> > Queued: 10,173 / 0 bytes
> > Connected nodes: 2 / 2
> > Stats last refreshed: 13:31:28 PDT
> >
> >
> > On Wed, Jun 15, 2016 at 4:29 PM, Joe Witt <joe.w...@gmail.com> wrote:
> >
> >> And the data remains?  If so that is an interesting data point I
> >> think.  So to mark's point how much data do you have queued up
> >> actively in the flow then on that nodes?  Number of objects you
> >> mention is 3273 files corresponding to 825GB in the content
> >> repository.  Does NiFi see those 825GB worth of data as being in the
> >> flow/queued up?  And then if that is the case are we talking about a
> >> roughly 1TB repo and so the reported value seems correct and this is
> >> simply a case of queueing near to the limit your system can hold?
> >>
> >> On Wed, Jun 15, 2016 at 4:24 PM, Ricky Saltzer <ri...@cloudera.com>
> wrote:
> >> > I have two nodes in clustered mode. I have the other node that isn't
> >> > filling up as my primary. I've actually already restarted nifi on the
> >> node
> >> > which has the large repository a few times.
> >> >
> >> > On Wed, Jun 15, 2016 at 4:22 PM, Joe Witt <joe.w...@gmail.com> wrote:
> >> >
> >> >> Ricky,
> >> >>
> >> >> If you restart nifi and then find that it cleans those things up I
> >> >> believe then it is related to the defects corrected in the 0.5/0.6
> >> >> timeframe.
> >> >>
> >> >> Is restarting an option for you at this time.  You agree mark?
> >> >>
> >> >> Thanks
> >> >> Joe
> >> >>
> >> >> On Wed, Jun 15, 2016 at 4:21 PM, Ricky Saltzer <ri...@cloudera.com>
> >> wrote:
> >> >> > Hey Mark -
> >> >> >
> >> >> > Thanks for the quick reply! This is our production system so it's
> >> >> > unfortunately running 0.4.0. There are currently 3273 files, with
> some
> >> >> > files dating back to May 18th. The content repository itself is
> 825G.
> >> >> >
> >> >> > Ricky
> >> >> >
> >> >> > On Wed, Jun 15, 2016 at 4:17 PM, Mark Payne <marka...@hotmail.com>
> >> >> wrote:
> >> >> >
> >> >> >> Hey Ricky
> >> >> >>
> >> >> >> The reclaim process is pretty much continuous. What version of
> NiFi
> >> are
> >> >> >> you running?
> >> >> >> I know there was an issue with this a while back that caused it
> not
> >> to
> >> >> >> cleanup properly.
> >> >> >>
> >> >> >> Also, how much data & how many FlowFiles do you have queued up in
> >> your
> >> >> >> flow?
> >> >> >> Data won't be archived or reclaimed if in the flow.
> >> >> >>
> >> >> >> Thanks
> >> >> >> -Mark
> >> >> >>
> >> >> >>
> >> >> >>
> >> >> >> > On Jun 15, 2016, at 4:04 PM, Ricky Saltzer <ri...@cloudera.com>
> >> >> wrote:
> >> >> >> >
> >> >> >> > Hey guys -
> >> >> >> >
> >> >> >> > I recently discovered I didn't have my "archive.enabled" option
> >> set to
> >> >> >> true
> >> >> >> > after my disk filled up to 95%. I enabled it and then set the
> >> >> retention
> >> >> >> > period to 12 hours and 50% (default values). However, after
> >> restarting
> >> >> >> > NiFi, I am not seeing any disk space reclaimed.
> >> >> >> >
> >> >> >> > I'm curious, is the reclaiming process periodic or continuous?
> >> >> >> >
> >> >> >> > ---
> >> >> >> > ricky
> >> >> >>
> >> >> >>
> >> >> >
> >> >> >
> >> >> > --
> >> >> > Ricky Saltzer
> >> >> > http://www.cloudera.com
> >> >>
> >> >
> >> >
> >> >
> >> > --
> >> > Ricky Saltzer
> >> > http://www.cloudera.com
> >>
> >
> >
> >
> > --
> > Ricky Saltzer
> > http://www.cloudera.com
>



-- 
Ricky Saltzer
http://www.cloudera.com

Re: Content Repository Cleanup Schedule

[Ricky Saltzer]

Hey Joe -

The NiFi web UI currently reads as:

Active threads: 3
Queued: 10,173 / 0 bytes
Connected nodes: 2 / 2
Stats last refreshed: 13:31:28 PDT


On Wed, Jun 15, 2016 at 4:29 PM, Joe Witt <joe.w...@gmail.com> wrote:

> And the data remains?  If so that is an interesting data point I
> think.  So to mark's point how much data do you have queued up
> actively in the flow then on that nodes?  Number of objects you
> mention is 3273 files corresponding to 825GB in the content
> repository.  Does NiFi see those 825GB worth of data as being in the
> flow/queued up?  And then if that is the case are we talking about a
> roughly 1TB repo and so the reported value seems correct and this is
> simply a case of queueing near to the limit your system can hold?
>
> On Wed, Jun 15, 2016 at 4:24 PM, Ricky Saltzer <ri...@cloudera.com> wrote:
> > I have two nodes in clustered mode. I have the other node that isn't
> > filling up as my primary. I've actually already restarted nifi on the
> node
> > which has the large repository a few times.
> >
> > On Wed, Jun 15, 2016 at 4:22 PM, Joe Witt <joe.w...@gmail.com> wrote:
> >
> >> Ricky,
> >>
> >> If you restart nifi and then find that it cleans those things up I
> >> believe then it is related to the defects corrected in the 0.5/0.6
> >> timeframe.
> >>
> >> Is restarting an option for you at this time.  You agree mark?
> >>
> >> Thanks
> >> Joe
> >>
> >> On Wed, Jun 15, 2016 at 4:21 PM, Ricky Saltzer <ri...@cloudera.com>
> wrote:
> >> > Hey Mark -
> >> >
> >> > Thanks for the quick reply! This is our production system so it's
> >> > unfortunately running 0.4.0. There are currently 3273 files, with some
> >> > files dating back to May 18th. The content repository itself is 825G.
> >> >
> >> > Ricky
> >> >
> >> > On Wed, Jun 15, 2016 at 4:17 PM, Mark Payne <marka...@hotmail.com>
> >> wrote:
> >> >
> >> >> Hey Ricky
> >> >>
> >> >> The reclaim process is pretty much continuous. What version of NiFi
> are
> >> >> you running?
> >> >> I know there was an issue with this a while back that caused it not
> to
> >> >> cleanup properly.
> >> >>
> >> >> Also, how much data & how many FlowFiles do you have queued up in
> your
> >> >> flow?
> >> >> Data won't be archived or reclaimed if in the flow.
> >> >>
> >> >> Thanks
> >> >> -Mark
> >> >>
> >> >>
> >> >>
> >> >> > On Jun 15, 2016, at 4:04 PM, Ricky Saltzer <ri...@cloudera.com>
> >> wrote:
> >> >> >
> >> >> > Hey guys -
> >> >> >
> >> >> > I recently discovered I didn't have my "archive.enabled" option
> set to
> >> >> true
> >> >> > after my disk filled up to 95%. I enabled it and then set the
> >> retention
> >> >> > period to 12 hours and 50% (default values). However, after
> restarting
> >> >> > NiFi, I am not seeing any disk space reclaimed.
> >> >> >
> >> >> > I'm curious, is the reclaiming process periodic or continuous?
> >> >> >
> >> >> > ---
> >> >> > ricky
> >> >>
> >> >>
> >> >
> >> >
> >> > --
> >> > Ricky Saltzer
> >> > http://www.cloudera.com
> >>
> >
> >
> >
> > --
> > Ricky Saltzer
> > http://www.cloudera.com
>



-- 
Ricky Saltzer
http://www.cloudera.com

Re: Content Repository Cleanup Schedule

[Ricky Saltzer]

I have two nodes in clustered mode. I have the other node that isn't
filling up as my primary. I've actually already restarted nifi on the node
which has the large repository a few times.

On Wed, Jun 15, 2016 at 4:22 PM, Joe Witt <joe.w...@gmail.com> wrote:

> Ricky,
>
> If you restart nifi and then find that it cleans those things up I
> believe then it is related to the defects corrected in the 0.5/0.6
> timeframe.
>
> Is restarting an option for you at this time.  You agree mark?
>
> Thanks
> Joe
>
> On Wed, Jun 15, 2016 at 4:21 PM, Ricky Saltzer <ri...@cloudera.com> wrote:
> > Hey Mark -
> >
> > Thanks for the quick reply! This is our production system so it's
> > unfortunately running 0.4.0. There are currently 3273 files, with some
> > files dating back to May 18th. The content repository itself is 825G.
> >
> > Ricky
> >
> > On Wed, Jun 15, 2016 at 4:17 PM, Mark Payne <marka...@hotmail.com>
> wrote:
> >
> >> Hey Ricky
> >>
> >> The reclaim process is pretty much continuous. What version of NiFi are
> >> you running?
> >> I know there was an issue with this a while back that caused it not to
> >> cleanup properly.
> >>
> >> Also, how much data & how many FlowFiles do you have queued up in your
> >> flow?
> >> Data won't be archived or reclaimed if in the flow.
> >>
> >> Thanks
> >> -Mark
> >>
> >>
> >>
> >> > On Jun 15, 2016, at 4:04 PM, Ricky Saltzer <ri...@cloudera.com>
> wrote:
> >> >
> >> > Hey guys -
> >> >
> >> > I recently discovered I didn't have my "archive.enabled" option set to
> >> true
> >> > after my disk filled up to 95%. I enabled it and then set the
> retention
> >> > period to 12 hours and 50% (default values). However, after restarting
> >> > NiFi, I am not seeing any disk space reclaimed.
> >> >
> >> > I'm curious, is the reclaiming process periodic or continuous?
> >> >
> >> > ---
> >> > ricky
> >>
> >>
> >
> >
> > --
> > Ricky Saltzer
> > http://www.cloudera.com
>



-- 
Ricky Saltzer
http://www.cloudera.com

Re: Content Repository Cleanup Schedule

[Ricky Saltzer]

Hey Mark -

Thanks for the quick reply! This is our production system so it's
unfortunately running 0.4.0. There are currently 3273 files, with some
files dating back to May 18th. The content repository itself is 825G.

Ricky

On Wed, Jun 15, 2016 at 4:17 PM, Mark Payne <marka...@hotmail.com> wrote:

> Hey Ricky
>
> The reclaim process is pretty much continuous. What version of NiFi are
> you running?
> I know there was an issue with this a while back that caused it not to
> cleanup properly.
>
> Also, how much data & how many FlowFiles do you have queued up in your
> flow?
> Data won't be archived or reclaimed if in the flow.
>
> Thanks
> -Mark
>
>
>
> > On Jun 15, 2016, at 4:04 PM, Ricky Saltzer <ri...@cloudera.com> wrote:
> >
> > Hey guys -
> >
> > I recently discovered I didn't have my "archive.enabled" option set to
> true
> > after my disk filled up to 95%. I enabled it and then set the retention
> > period to 12 hours and 50% (default values). However, after restarting
> > NiFi, I am not seeing any disk space reclaimed.
> >
> > I'm curious, is the reclaiming process periodic or continuous?
> >
> > ---
> > ricky
>
>


-- 
Ricky Saltzer
http://www.cloudera.com

Content Repository Cleanup Schedule

[Ricky Saltzer]

Hey guys -

I recently discovered I didn't have my "archive.enabled" option set to true
after my disk filled up to 95%. I enabled it and then set the retention
period to 12 hours and 50% (default values). However, after restarting
NiFi, I am not seeing any disk space reclaimed.

I'm curious, is the reclaiming process periodic or continuous?

---
ricky

Re: StoreInKiteDataset cant validate URI

[Ricky Saltzer]

That usually means that the dataset URI is malformed in some way. Can you
confirm that there's no white spaces, or special characters that might be
messing up the path? I've not had this issue writing to HDFS paths.


On Tue, May 17, 2016 at 10:40 AM, pradeepbill <pradeep.b...@gmail.com>
wrote:

> hi there, I am using StoreInKiteDataset , and when I give the  dataset URI
> as
> "dataset:hdfs:/user/xxx/dataset" path, it says "is invalid because dataset
> URI is invalid:unknown dataset URI", not sure why its saying invalid.
>
>
> I used Kite command
>
> >./kite-dataset create dataset:hdfs:/user/xxx/dataset --schema
> hdfs:/user/xxx/schemas/yy.avsc  --format parquet
>
> PLease help
>
> Thanks
> Pradeep
>
>
>
> --
> View this message in context:
> http://apache-nifi-developer-list.39713.n7.nabble.com/StoreInKiteDataset-cant-validate-URI-tp10409.html
> Sent from the Apache NiFi Developer List mailing list archive at
> Nabble.com.
>



-- 
Ricky Saltzer
http://www.cloudera.com

Re: Trouble with the LDAP Authentication Provider

[Ricky Saltzer]

Ah I believe I've figured it out. It appears I was getting confused by the
difference between authority-providers and the login-identity-providers as
they have identical stanzas. The solution was to add the provider to the
login-identity-providers.xml.

Thanks so much for your sample configs, Andy!

On Sun, May 15, 2016 at 7:43 PM, Andy LoPresto <alopre...@apache.org> wrote:

> Hi Ricky,
>
> I checked out nifi-0.6.1 and built on my system, then deployed with a
> Kerberos configuration and a KDC running in Vagrant and everything worked
> fine. Was able to run kinit on the command line of the client machine, and
> then opening Safari established a session using my Kerberos principal
> immediately. I looked at your app log, and it appears it might be a file
> permission/existence issue. I admit the error could be more helpful — it’s
> unclear as to whether it’s an IO problem or an XML problem or a Spring
> problem. Can you please verify that the authority-providers.xml file exists
> in the correct location, has the correct access permissions, and is
> well-formed XML? I’ve published my nifi.properties [1],
> authority-providers.xml [2], authorized-users.xml [3], and
> login-identity-provider.xml [4] files as gists as well for comparison.
>
> In the nifi.properties, note lines 142 & 143, as they define the
> references to the authority and login identity providers, and lines 187 &
> 189, as they define the Kerberos properties.
>
> From your nifi-app.log:
>
> 2016-05-12 14:14:04,468 ERROR [main] o.s.web.context.ContextLoader Context
> initialization failed
> org.springframework.beans.factory.BeanCreationException: Error creating
> bean with name 'niFiWebApiSecurityConfiguration': Injection of autowired
> dependencies failed; nested exception is
> org.springframework.beans.factory.BeanCreationException: Could not autowire
> method: public void
> org.apache.nifi.web.NiFiWebApiSecurityConfiguration.setUserDetailsService(org.springframework.security.core.userdetails.AuthenticationUserDetailsService);
> nested exception is
> org.springframework.beans.factory.BeanCreationException: Error creating
> bean with name 'userDetailsService' defined in class path resource
> [nifi-web-security-context.xml]: Cannot resolve reference to bean
> 'userService' while setting bean property 'userService'; nested exception
> is org.springframework.beans.factory.BeanCreationException: Error creating
> bean with name 'userService' defined in class path resource
> [nifi-administration-context.xml]: Cannot resolve reference to bean
> 'userTransactionBuilder' while setting bean property 'transactionBuilder';
> nested exception is
> org.springframework.beans.factory.BeanCreationException: Error creating
> bean with name 'userTransactionBuilder' defined in class path resource
> [nifi-administration-context.xml]: Cannot resolve reference to bean
> 'authorityProvider' while setting bean property 'authorityProvider'; nested
> exception is org.springframework.beans.factory.BeanCreationException: Error
> creating bean with name 'authorityProvider': FactoryBean threw exception on
> object creation; *nested exception is java.lang.Exception: Unable to load
> the authority provider configuration file at:
> /private/tmp/nifi-0.6.1/./conf/authority-providers.xml*
>
> [1] https://gist.github.com/alopresto/dfad48f55780fee3d0d62b7a0169f2d7
> [2] https://gist.github.com/alopresto/b3bd36676ff72351e641df6869bc1b84
> [3] https://gist.github.com/alopresto/e6bca539876fe4324f49e4996f41c91a
> [4] https://gist.github.com/alopresto/06938e4d0ccdf2168fe0fc6158780a56
>
> Andy LoPresto
> alopre...@apache.org
> *alopresto.apa...@gmail.com <alopresto.apa...@gmail.com>*
> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
>
> On May 13, 2016, at 4:05 PM, Ricky Saltzer <ri...@cloudera.com> wrote:
>
> Right on! I appreciate you helping out. Have a good weekend!
>
> On Fri, May 13, 2016 at 3:59 PM, Andy LoPresto <alopre...@apache.org>
> wrote:
>
> Thanks Ricky. I’ll set up a demo environment with 0.6.1 and LDAP/Kerberos
> authentication
> locally and see if I can reproduce. Probably get back to you Monday?
>
> Andy LoPresto
> alopre...@apache.org
> *alopresto.apa...@gmail.com <alopresto.apa...@gmail.com>*
> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
>
> On May 13, 2016, at 1:47 PM, Ricky Saltzer <ri...@cloudera.com> wrote:
>
> Hey Andy -
>
> The full log file, nifi.properties, and authority-providers in the
> following gists. Obviously I've replaced some values in the
> authority-providers with fake data for security reasons.
>
> *Log:*
>
>
> https://gist.githubusercontent.com/rickysaltzer/a645f18a4b3d8bacd16d57

Re: Trouble with the LDAP Authentication Provider

[Ricky Saltzer]

Right on! I appreciate you helping out. Have a good weekend!

On Fri, May 13, 2016 at 3:59 PM, Andy LoPresto <alopre...@apache.org> wrote:

> Thanks Ricky. I’ll set up a demo environment with 0.6.1 and LDAP/Kerberos 
> authentication
> locally and see if I can reproduce. Probably get back to you Monday?
>
> Andy LoPresto
> alopre...@apache.org
> *alopresto.apa...@gmail.com <alopresto.apa...@gmail.com>*
> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
>
> On May 13, 2016, at 1:47 PM, Ricky Saltzer <ri...@cloudera.com> wrote:
>
> Hey Andy -
>
> The full log file, nifi.properties, and authority-providers in the
> following gists. Obviously I've replaced some values in the
> authority-providers with fake data for security reasons.
>
> *Log:*
>
> https://gist.githubusercontent.com/rickysaltzer/a645f18a4b3d8bacd16d57cd093f8997/raw/08f78789b66a4d7094629699af7f408870b2c0da/gistfile1.txt
>
> *Authority: *
>
> https://gist.githubusercontent.com/rickysaltzer/b6db60311ea9e3abb94ac183e1c02a59/raw/a75b348ea9515acf0d7bbe0a936972c9b6cb38fe/gistfile1.txt
>
> *Properties:*
>
> https://gist.githubusercontent.com/rickysaltzer/3b29f430d0d1b6361a7ff097e8fcea6a/raw/28bb328fc01ed5256b41bfb324341c083f6fa354/gistfile1.txt
>
> On Fri, May 13, 2016 at 10:55 AM, Andy LoPresto <alopre...@apache.org>
> wrote:
>
> Hi Ricky,
>
> Can you provide the contents of logs/nifi-app.log as well to see if there
> is anything relevant to this exception? The code where this is failing
> attempts to deserialize the XML into one of a number of classes
> implementing the AuthorityProvider interface via the factory. Are you sure
> the XML is valid and complete, and that the provider identifier is also
> specified in nifi.properties?
>
> Andy LoPresto
> alopre...@apache.org
> *alopresto.apa...@gmail.com <alopresto.apa...@gmail.com>*
>
> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
>
> On May 12, 2016, at 2:26 PM, Ricky Saltzer <ri...@cloudera.com> wrote:
>
> Using the following provider on 0.6.1, I'm faced with a ClassCastException.
> It might also be worth noting that I face the same exception when
> attempting to us the KerberosProvider option.
>
> *Provider:*
> 
>   ldap-provider
>   org.apache.nifi.ldap.LdapProvider
>   SIMPLE
>
>   dethklok\toki
>   bananasticker
>
>   
>   
>   
>   
>   
>   
>   
>   
>   
>
>   FOLLOW
>   10 secs
>   10 secs
>
>   ldap://ldap.metalocalypse.com
>   CN=Users,DC=metalocalypse,DC=local
>   foo
>
>   12 hours
> 
>
> *Exception:*
> Caused by: java.lang.ClassCastException: class
> org.apache.nifi.ldap.LdapProvider
>   at java.lang.Class.asSubclass(Class.java:3208) ~[na:1.7.0_79]
>   at
>
>
> org.apache.nifi.authorization.AuthorityProviderFactoryBean.createAuthorityProvider(AuthorityProviderFactoryBean.java:173)
> ~[na:na]
>   at
>
>
> org.apache.nifi.authorization.AuthorityProviderFactoryBean.getObject(AuthorityProviderFactoryBean.java:111)
> ~[na:na]
>   at
>
>
> org.springframework.beans.factory.support.FactoryBeanRegistrySupport.doGetObjectFromFactoryBean(FactoryBeanRegistrySupport.java:168)
> ~[na:na]
>   ... 75 common frames omitted
>
>
>
>
>
> --
> Ricky Saltzer
> http://www.cloudera.com
>
>
>


-- 
Ricky Saltzer
http://www.cloudera.com

Re: Trouble with the LDAP Authentication Provider

[Ricky Saltzer]

Hey Andy -

The full log file, nifi.properties, and authority-providers in the
following gists. Obviously I've replaced some values in the
authority-providers with fake data for security reasons.

*Log:*
https://gist.githubusercontent.com/rickysaltzer/a645f18a4b3d8bacd16d57cd093f8997/raw/08f78789b66a4d7094629699af7f408870b2c0da/gistfile1.txt

*Authority: *
https://gist.githubusercontent.com/rickysaltzer/b6db60311ea9e3abb94ac183e1c02a59/raw/a75b348ea9515acf0d7bbe0a936972c9b6cb38fe/gistfile1.txt

*Properties:*
https://gist.githubusercontent.com/rickysaltzer/3b29f430d0d1b6361a7ff097e8fcea6a/raw/28bb328fc01ed5256b41bfb324341c083f6fa354/gistfile1.txt

On Fri, May 13, 2016 at 10:55 AM, Andy LoPresto <alopre...@apache.org>
wrote:

> Hi Ricky,
>
> Can you provide the contents of logs/nifi-app.log as well to see if there
> is anything relevant to this exception? The code where this is failing
> attempts to deserialize the XML into one of a number of classes
> implementing the AuthorityProvider interface via the factory. Are you sure
> the XML is valid and complete, and that the provider identifier is also
> specified in nifi.properties?
>
> Andy LoPresto
> alopre...@apache.org
> *alopresto.apa...@gmail.com <alopresto.apa...@gmail.com>*
> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
>
> On May 12, 2016, at 2:26 PM, Ricky Saltzer <ri...@cloudera.com> wrote:
>
> Using the following provider on 0.6.1, I'm faced with a ClassCastException.
> It might also be worth noting that I face the same exception when
> attempting to us the KerberosProvider option.
>
> *Provider:*
> 
>ldap-provider
>org.apache.nifi.ldap.LdapProvider
>SIMPLE
>
>dethklok\toki
>bananasticker
>
>
>
>
>
>
>
>
>
>
>
>FOLLOW
>10 secs
>10 secs
>
>ldap://ldap.metalocalypse.com
>CN=Users,DC=metalocalypse,DC=local
>foo
>
>12 hours
> 
>
> *Exception:*
> Caused by: java.lang.ClassCastException: class
> org.apache.nifi.ldap.LdapProvider
>at java.lang.Class.asSubclass(Class.java:3208) ~[na:1.7.0_79]
>at
>
> org.apache.nifi.authorization.AuthorityProviderFactoryBean.createAuthorityProvider(AuthorityProviderFactoryBean.java:173)
> ~[na:na]
>at
>
> org.apache.nifi.authorization.AuthorityProviderFactoryBean.getObject(AuthorityProviderFactoryBean.java:111)
> ~[na:na]
>    at
>
> org.springframework.beans.factory.support.FactoryBeanRegistrySupport.doGetObjectFromFactoryBean(FactoryBeanRegistrySupport.java:168)
> ~[na:na]
>... 75 common frames omitted
>
>
>


-- 
Ricky Saltzer
http://www.cloudera.com

Trouble with the LDAP Authentication Provider

[Ricky Saltzer]

Using the following provider on 0.6.1, I'm faced with a ClassCastException.
It might also be worth noting that I face the same exception when
attempting to us the KerberosProvider option.

*Provider:*

ldap-provider
org.apache.nifi.ldap.LdapProvider
SIMPLE

dethklok\toki
bananasticker











FOLLOW
10 secs
10 secs

ldap://ldap.metalocalypse.com
CN=Users,DC=metalocalypse,DC=local
foo

12 hours


*Exception:*
Caused by: java.lang.ClassCastException: class
org.apache.nifi.ldap.LdapProvider
at java.lang.Class.asSubclass(Class.java:3208) ~[na:1.7.0_79]
at
org.apache.nifi.authorization.AuthorityProviderFactoryBean.createAuthorityProvider(AuthorityProviderFactoryBean.java:173)
~[na:na]
at
org.apache.nifi.authorization.AuthorityProviderFactoryBean.getObject(AuthorityProviderFactoryBean.java:111)
~[na:na]
at
org.springframework.beans.factory.support.FactoryBeanRegistrySupport.doGetObjectFromFactoryBean(FactoryBeanRegistrySupport.java:168)
~[na:na]
... 75 common frames omitted

Re: parquet format

[Ricky Saltzer]

great, I'd love to here if you're successful with this or not.

On Wed, May 11, 2016 at 1:10 PM, pradeepbill <pradeep.b...@gmail.com> wrote:

> Thanks for the explanation Ricky, I think I got the idea, will proceed with
> this approach and see how it works.
>
>
>
> --
> View this message in context:
> http://apache-nifi-developer-list.39713.n7.nabble.com/parquet-format-tp10145p10186.html
> Sent from the Apache NiFi Developer List mailing list archive at
> Nabble.com.
>



-- 
Ricky Saltzer
http://www.cloudera.com

Re: [ANNOUNCE] New Apache NiFi Committer James Wing

[Ricky Saltzer]

Congratulations! Welcome, James!

On Wed, May 11, 2016 at 1:34 PM, Andrew Psaltis <psaltis.and...@gmail.com>
wrote:

> Congrats James!!
>
> On Wed, May 11, 2016 at 4:30 PM, Pierre Villard <
> pierre.villard...@gmail.com
> > wrote:
>
> > Congrats James!
> >
> >
> > 2016-05-11 22:15 GMT+02:00 dan bress <danbr...@gmail.com>:
> >
> > > Welcome James!
> > >
> > > On Wed, May 11, 2016 at 1:15 PM Mark Payne <marka...@hotmail.com>
> wrote:
> > >
> > > > Welcome, James! Glad to have you aboard!
> > > >
> > > > > On May 11, 2016, at 4:10 PM, Joe Witt <joew...@apache.org> wrote:
> > > > >
> > > > > On behalf of the Apache NiFi PMC, I am pleased to announce that
> James
> > > > > Wing has accepted the PMC's inviation to become a committer on the
> > > > > Apache NiFi project.  James has been an excellent contributor to a
> > > > > number of code submissions, peer reviews, mailing list discussions
> > and
> > > > > decision making and it is greatly appreciated.  We look forward to
> > his
> > > > > continued involvement in the project.
> > > > >
> > > > > Welcome and congratulations!
> > > >
> > > >
> > >
> >
>
>
>
> --
> Thanks,
> Andrew
>
> Subscribe to my book: Streaming Data <http://manning.com/psaltis>
> <https://www.linkedin.com/pub/andrew-psaltis/1/17b/306>
> twiiter: @itmdata <http://twitter.com/intent/user?screen_name=itmdata>
>



-- 
Ricky Saltzer
http://www.cloudera.com

Re: parquet format

[Ricky Saltzer]

If I'm not mistaken you should be able to do this using the Kite processor.
If your Kite dataset in HDFS is parquet, then Kite should automatically
write it using Parquet.

On Wed, May 11, 2016 at 11:30 AM, Joe Witt <joe.w...@gmail.com> wrote:

> also responded to the same question on SO:
>
> http://stackoverflow.com/questions/37149331/apache-nifi-hdfs-parquet-format/37170672#37170672
>
> On Wed, May 11, 2016 at 12:32 PM, Bryan Bende <bbe...@gmail.com> wrote:
> > Hi Pradeep,
> >
> > Currently there isn't anything in NiFi that produces parquet format,
> > although it has been mentioned before.
> >
> > The HDFS processors just write the bytes of the FlowFile to HDFS, so we
> > would need an a processor before that that was able to produce parquet.
> >
> > If you have experience with parquet and are interested in contributing,
> let
> > us know, we'd be happy to help guide the process.
> >
> > -Bryan
> >
> > On Wed, May 11, 2016 at 9:09 AM, pradeepbill <pradeep.b...@gmail.com>
> wrote:
> >
> >> Hi there,how can I write file to HDFS in parquet format ?.Please advice
> >>
> >> Thanks
> >> Pradeep
> >>
> >>
> >>
> >> --
> >> View this message in context:
> >>
> http://apache-nifi-developer-list.39713.n7.nabble.com/parquet-format-tp10145.html
> >> Sent from the Apache NiFi Developer List mailing list archive at
> >> Nabble.com.
> >>
>



-- 
Ricky Saltzer
http://www.cloudera.com

Re: [discuss] PropertyDescriptor name and displayName attributes

[Ricky Saltzer]

An idea I had is to modify the API and use a setter that would require both
the name and the display name. That way and you deprecate the other two
over time.

Example:
setName(String name, String DisplayName)

This would require both values to be filled out, and since it's overloaded
you could introduce non intrusively.

I understand this might not fit with the Java builder pattern best practice.

On Tue, May 3, 2016 at 12:09 PM, Andy LoPresto  wrote:

> As a result of some conversations and some varying feedback on PRs, I’d
like
> to discuss with the community an issue I see with PropertyDescriptor name
> and displayName attributes. I’ll describe the scenarios that cause issues
> and my proposed solution, and then solicit responses and other
perspectives.

Andy,

I'd be +1 on this as well. I think the power of this approach will
become more clear over time, and some of the automation will make it
possible to more widely enforce.

What do you think about a mixed mode where config reading code can
fetch the property value with either the name or display name as the
key, defaulting to the name if it is present? A sample read of
flow.xml.gz might look like this:

* Processor asks for value of MY_CONFIG_PROPERTY
* Configuration code looks for key "my-config-property", returns if present
* Configuration code looks for key "My Config Property", returns if present
* On finding no valid key, configuration returns blank/default/null
value (whatever is done today)

On configuration write, the new attributes could be written as the
normalized name (instead of display name), to allow processors that
have made the switch to start using the normalized name field and
start taking advantage of the new features around it (e.g.
internationalization). Perhaps a disadvantage to this approach is that
it auto-upgrades an existing flow.xml.gz, making it difficult to
downgrade from (for example) 0.7 back to 0.6.

A strategy like this (or something similar) might help speed adoption
by making the process a bit smoother for existing flow.xml.gz files
and easier to upgrade processors incrementally. At 1.0, display names
as configuration keys could be dropped, and the upgrade path for users
on old 0.x releases would be to upgrade to the latest 0.x before
making the jump to 1.0.

Cheers,
Adam

Re: [ANNOUNCE] New Apache NiFi Committer Oleg Zhurakousky

[Ricky Saltzer]

Big congrats!
On Mar 15, 2016 8:42 PM, "Michael Moser"  wrote:

> Yes, welcome Oleg!  Your hard work is very much appreciated by everyone.
>
> -- Mike
>
>
> On Tue, Mar 15, 2016 at 8:22 PM, Matt Burgess  wrote:
>
> > Congratulations! Well deserved.
> >
> > > On Mar 15, 2016, at 8:17 PM, Tony Kurc  wrote:
> > >
> > > On behalf of the Apache NiFI PMC, I am very pleased to announce that
> Oleg
> > > Zhurakousky has accepted the PMC's invitation to become a committer on
> > the
> > > Apache NiFi project. We greatly appreciate all of Oleg's hard work and
> > > generous contributions to the project. We look forward to his continued
> > > involvement in the project.
> > >
> > > Some highlights of Oleg's contributions include identifying and
> > mitigating
> > > some challenging framework bugs, amqp support in 0.5.x, and some pretty
> > > exciting spring support slated for 0.6.x. As I have noticed, many of
> you
> > > have already had the opportunity to interact with Oleg on the mailing
> > > lists, the jiras, and pull request comments. I, for one, look forward
> to
> > > the continued community support!
> > >
> > > Welcome and congratulations!
> > >
> > > Tony
> >
>

Re: [ANNOUNCE] New Apache NiFi Committer Matt Burgess

[Ricky Saltzer]

Congratulations!

On Fri, Mar 4, 2016 at 2:11 PM, Edmon Begoli <ebeg...@gmail.com> wrote:

> Congrats, Matt. I have seen Matt's previous open source code contributions,
> and I can attest that he is an expert programmer.
>
> Edmon
>
> On Fri, Mar 4, 2016 at 2:02 PM, Joe Witt <joe.w...@gmail.com> wrote:
>
> > Congrats Matthew and very much welcome your contributions to the
> > project in terms of reviews, code, and also blogs to help drive
> > awareness and feedback of those capabilities.
> >
> > Thanks
> > Joe
> >
> > On Fri, Mar 4, 2016 at 2:00 PM, Tony Kurc <tk...@apache.org> wrote:
> > > On behalf of the Apache NiFI PMC, I am very pleased to announce that
> Matt
> > > Burgess has accepted the PMC's invitation to become a committer on the
> > > Apache NiFi project. We greatly appreciate all of Matt's hard work and
> > > generous contributions to the project. We look forward to his continued
> > > involvement in the project.
> > >
> > > Matt has taken lead on some of our longer standing feature requests as
> > well
> > > as representing the community well in the blogosphere and twitterverse.
> > > We're delighted to have him on board as a committer now!
> > >
> > > Tony
> >
>



-- 
Ricky Saltzer
http://www.cloudera.com

Re: [VOTE] Release Apache NiFi 0.5.0 (RC3)

[Ricky Saltzer]

+1 (non-binding) - tested out some basic data flows as well.

On Tue, Feb 16, 2016 at 9:10 AM, Matt Gilman <matt.c.gil...@gmail.com>
wrote:

> +1 (binding)
>
> Release this package as nifi-0.5.0
>
> Matt
>
> On Fri, Feb 12, 2016 at 6:27 PM, Tony Kurc <trk...@gmail.com> wrote:
>
> > Hello
> > I am pleased to be calling this vote for the source release of Apache
> NiFi
> > nifi-0.5.0.
> >
> > The source zip, including signatures, digests, etc. can be found at:
> > https://repository.apache.org/content/repositories/orgapachenifi-1073
> >
> >
> > The Git tag is nifi-0.5.0-RC3
> > The Git commit ID is 8309dba80bbc0e6c0dfe5b8f93750d424f5b9a90
> >
> >
> https://git-wip-us.apache.org/repos/asf?p=nifi.git;a=commit;h=8309dba80bbc0e6c0dfe5b8f93750d424f5b9a90
> >
> > This release candidate is a branch off of master at
> > cae5b109c09be80df77f2d767220b82850178ed5.
> >
> > Checksums of nifi-0.5.0-source-release.zip:
> > MD5: 5d20a5bb2fcdb3d96b2c62949db1c780
> > SHA1: e1c541d27373b94a58c8ede4eae7f6a7eb51c5a0
> >
> > Release artifacts are signed with the following key:
> > https://people.apache.org/keys/committer/tkurc.asc
> >
> > KEYS file available here:
> > https://dist.apache.org/repos/dist/dev/nifi/KEYS
> >
> > 106 issues were closed/resolved for this release:
> >
> >
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020=12334158
> >
> > Release note highlights can be found here:
> >
> >
> https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version0.5.0
> >
> > Migration guidance can be found here:
> > https://cwiki.apache.org/confluence/display/NIFI/Migration+Guidance
> >
> > The vote will be open for 96 hours (24 extra hours due to late Friday
> > release candidate)
> > Please download the release candidate and evaluate the necessary items
> > including checking hashes, signatures, build from source, and test.  The
> > please vote:
> >
> > [ ] +1 Release this package as nifi-0.5.0
> > [ ] +0 no opinion
> > [ ] -1 Do not release this package because because...
> >
>



-- 
Ricky Saltzer
http://www.cloudera.com

Re: Are we thinking about Penalization all wrong?

[Ricky Saltzer]

t;> - Developers often forget to call penalize()
> >>> - Developer has to determine whether or not to penalize when building a
> >> processor. It is based on what the developer will
> >>> think may make sense, but in reality DFM's sometimes want to penalize
> >> things when the processor doesn't behave that way.
> >>>
> >>> I'm wondering if it doesn't make sense to remove the concept of
> >> penalization all together from Processors and instead
> >>> move the Penalty Duration so that it's a setting on the Connection. I
> >> think this would clear up the confusion and give the DFM
> >>> more control over when/how long to penalize. Could set to the default
> to
> >> 30 seconds for self-looping connections and no penalization
> >>> for other connections.
> >>>
> >>> Any thoughts?
> >>>
> >>> Thanks
> >>> -Mark
> >>
> >>
>
>


-- 
Ricky Saltzer
http://www.cloudera.com

Re: Are we thinking about Penalization all wrong?

[Ricky Saltzer]

That's a good point, Mark. I also agree that it's better to give the user
control whenever possible. I imagine the RouteOnAttribute pattern to
eventually "give up" on a FlowFile will be a common pattern, and so so we
should account for that, rather than forcing the user into knowing this
pattern.

On Thu, Jan 28, 2016 at 2:11 PM, Mark Payne <marka...@hotmail.com> wrote:

>
> The retry idea concerns me a bit. If we were to have a method like:
>
> penalizeOrTransfer(FlowFile flowFile, int numberOfTries, Relationship
> relationship)
>
> I think that leaves out some info - even if a FlowFile is
> penalized, it must be penalized and sent somewhere. So there would have to
> be
> a relationship to send it to if penalized and another to send it to if not
> penalizing.
> This also I think puts more onus on the developer to understand how it
> would be
> used - I believe the user should be making decisions about how many times
> to
> penalize, not the developer.
>
> > On Jan 28, 2016, at 2:03 PM, Bryan Bende <bbe...@gmail.com> wrote:
> >
> > Regarding throwing an exception... I believe if you are extending
> > AbstractProcessor and an exception is thrown out of onTrigger() then the
> > session is rolled back and any flow files that were accessed are
> penalized,
> > which results in leaving them in the incoming connection to the processor
> > and not being retried until the penalty duration passes. This seems
> similar
> > to what Michael described, although it is not stopping the processor from
> > processing other incoming  flow files.
> >
> > Ricky's retry idea sounds interesting... I think a lot of people handle
> > this today by creating a retry loop using UpdateAttribute and
> > RouteOnAttribute [1].
> >
> > [1]
> >
> https://cwiki.apache.org/confluence/download/attachments/57904847/Retry_Count_Loop.xml?version=1=1433271239000=v2
> >
> >
> > On Thu, Jan 28, 2016 at 1:24 PM, Ricky Saltzer <ri...@cloudera.com>
> wrote:
> >
> >> Is there currently a way to know how many times a FlowFile has been
> >> penalized? Do we have use cases where we want to penalize a FlowFile *n
> >> *number
> >> of times before sending it down an alternate relationship? I could
> imagine
> >> an API like penalizeOrTransfer(FlowFile flowFile, int numberOfTries,
> >> Relationship relationship). For example, someone might want to process a
> >> FlowFile three times before giving up on it.
> >>
> >> On Thu, Jan 28, 2016 at 12:47 PM, Michael de Courci <
> >> mdecou...@googlemail.com> wrote:
> >>
> >>> Matt thanks for your reply
> >>>
> >>> I guess what I am saying in that case - if there is an error in a
> >>> FlowFile, then the processor that detects this cannot proceed so
> instead
> >> of
> >>> calling an action to penalize the FlowFile it raises an exception
> >>> OutOFServiceException or ProcessorException.
> >>> You could have an exception cause PeanilisedFlowFileException for this
> >>> case.
> >>>
> >>> But within the processor other error causes may arise for an
> >>> OutOFServiceException
> >>>
> >>> The point is that if the processor threw this exception then there can
> be
> >>> a duration configuration - a time limit to keep this processor out of
> >>> service and the connection to it and possibly any processors leading
> upto
> >>> it - Naturally this will need to be indicated on the DFM - this will
> free
> >>> resources and make the flow well behaved.
> >>>
> >>> Environmental failures will simply be a different category/cause of
> error
> >>> that can be wrapped/captured also with a more general one
> >>>
> >>> With Kind Regards
> >>> Michael de Courci
> >>> mdecou...@gmail.com
> >>>
> >>>
> >>>
> >>>
> >>>> On 28 Jan 2016, at 17:16, Matt Gilman <matt.c.gil...@gmail.com>
> wrote:
> >>>>
> >>>> Just to recap/level set...
> >>>>
> >>>> The distinct between yielding and penalization is important.
> >> Penalization
> >>>> is an action taken on a FlowFile because the FlowFile cannot be
> >> processed
> >>>> right now (like a naming conflict for instance). The Processor is
> >>>> indicating that it cannot process that specific FlowFile at the moment
> >>> but
> >>>> may be able to process 

Re: State Management

[Ricky Saltzer]

+1 for making the znode a configurable option. Especially if we can nest
it, such as "/nifi//production" and "/nifi//development".
On Dec 31, 2015 3:49 PM, "Mark Payne" <marka...@hotmail.com> wrote:

> At this point I'm thinking that it would just be a configurable value,
> defaulting to /nifi
> This way, admins can easily configure it and that way they could view
> whats in there, etc.
> out-of-band of NiFi. Though I'm all ears if there's a better way of doing
> this.
>
>
> > On Dec 31, 2015, at 3:44 PM, Ricky Saltzer <ri...@cloudera.com> wrote:
> >
> > Overall this looks great, and will prove very useful as we try to scale
> > nifi out.
> >
> > When in clustered mode, will we have control over which znode the nifi
> > cluster persists to? Or - do we want this unique to the cluster (e.g uuid
> > of the flow)?
> > On Dec 31, 2015 12:35 PM, "Mark Payne" <marka...@hotmail.com> wrote:
> >
> >> All,
> >>
> >> I have spent a good amount of time in the past few weeks working on the
> >> State Management Feature Proposal, described at [1].
> >> The main JIRA for this is found at [2].
> >>
> >> I have updated the Feature Proposal with more implementation details
> >> describing the path that I have taken.
> >> If anyone has any interest in reviewing the ideas and providing
> feedback,
> >> please do so, as this is something that
> >> we want to ensure that we get right!
> >>
> >> Also of note, for those interested and located in the MD/DC/VA area, I
> >> will be talking a bit about this feature at the next
> >> Maryland Apache NiFi meetup on Jan. 7.
> >>
> >> Thanks
> >> -Mark
> >>
> >>
> >> [1] https://cwiki.apache.org/confluence/display/NIFI/State+Management <
> >> https://cwiki.apache.org/confluence/display/NIFI/State+Management>
> >> [2] https://issues.apache.org/jira/browse/NIFI-259 <
> >> https://issues.apache.org/jira/browse/NIFI-259>
> >>
> >>
>
>

Re: State Management

[Ricky Saltzer]

Overall this looks great, and will prove very useful as we try to scale
nifi out.

When in clustered mode, will we have control over which znode the nifi
cluster persists to? Or - do we want this unique to the cluster (e.g uuid
of the flow)?
On Dec 31, 2015 12:35 PM, "Mark Payne"  wrote:

> All,
>
> I have spent a good amount of time in the past few weeks working on the
> State Management Feature Proposal, described at [1].
> The main JIRA for this is found at [2].
>
> I have updated the Feature Proposal with more implementation details
> describing the path that I have taken.
> If anyone has any interest in reviewing the ideas and providing feedback,
> please do so, as this is something that
> we want to ensure that we get right!
>
> Also of note, for those interested and located in the MD/DC/VA area, I
> will be talking a bit about this feature at the next
> Maryland Apache NiFi meetup on Jan. 7.
>
> Thanks
> -Mark
>
>
> [1] https://cwiki.apache.org/confluence/display/NIFI/State+Management <
> https://cwiki.apache.org/confluence/display/NIFI/State+Management>
> [2] https://issues.apache.org/jira/browse/NIFI-259 <
> https://issues.apache.org/jira/browse/NIFI-259>
>
>

Re: Input Forbidden Requirement

[Ricky Saltzer]

Aha, that makes sense. Thanks for the explanation! I do agree that it could
be useful to have control over when those types of processors execute.

Re: [VOTE] Release Apache NiFi 0.4.0 (rc2)

[Ricky Saltzer]

+1

build works
test works
hashes/keys check out
tested a couple simple workflows

On Tue, Dec 8, 2015 at 4:57 PM, Joe Percivall <
joeperciv...@yahoo.com.invalid> wrote:

> Ran through the helper: validated keys, built binaries on Windows and Mac,
> ran some templates.
> Everything worked as expected.
> +1 Release this package as Apache NiFi 0.4.0. - - - - - - Joseph
> Percivalllinkedin.com/in/Percivalle: joeperciv...@yahoo.com
>
>
>
> On Tuesday, December 8, 2015 3:29 PM, Joe Witt <joew...@apache.org>
> wrote:
>
>
>  Hello NiFi Community,
>
> I am pleased to be calling this vote for the source release of Apache
> NiFi 0.4.0.
>
> The source zip, including signatures, digests, and associated
> convenience binaries can be found at
>   https://dist.apache.org/repos/dist/dev/nifi/nifi-0.4.0/
>
> The staged maven artifacts of the build can be found at
>   https://repository.apache.org/content/repositories/orgapachenifi-1065
>
> The Git tag is nifi-0.4.0-RC2
> The Git commit ID is b66c029090f395c0cbc001fd483e86895b133e46
>
> https://git-wip-us.apache.org/repos/asf?p=nifi.git;a=commit;h=b66c029090f395c0cbc001fd483e86895b133e46
>
> Checksums of NiFi 0.4.0 Source Release
> MD5: da733f8fdb520a0346dcda59940b2c12
> SHA1: 82fffbc5f8d7e4724bbe2f794bdde39396dae745
>
> Release artifacts are signed with the following key
>   https://people.apache.org/keys/committer/joewitt.asc
>
> KEYS file available here
>   https://dist.apache.org/repos/dist/release/nifi/KEYS
>
> 161 issues were closed/resolved for this release
>
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020=12333070
>
> Release note highlights
>
> https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version0.4.0
>
> Migration/Upgrade guidance
>   https://cwiki.apache.org/confluence/display/NIFI/Migration+Guidance
>   https://cwiki.apache.org/confluence/display/NIFI/Upgrading+NiFi
>
> The vote will be open for 72 hours.
> Please download the release candidate and evaluate the necessary items
> including checking hashes, signatures, build from source, and test.
>
> Then please vote:
>
> [ ] +1 Release this package as Apache NiFi 0.4.0
> [ ] +0 no opinion
> [ ] -1 Do not release this package because...
>
>
>
>



-- 
Ricky Saltzer
http://www.cloudera.com

Re: [ANNOUNCE] New Apache NiFi PMC Member (and Committer) Sean Busbey

[Ricky Saltzer]

Congrats, Busbey!!
On Nov 25, 2015 1:14 PM, "Tony Kurc"  wrote:

> On behalf of the Apache NiFI PMC, I am very pleased to announce that Sean
> Busbey has accepted the PMC's invitation to become a PMC Member and
> Committer on the Apache NiFi project. We greatly appreciate all of Sean's
> hard work and generous contributions to the project.
>
> In addition to his contributions to NiFi, Sean is what I would describe as
> "prolific" in the Apache community, a PMC member on other projects, notably
> Apache Yetus. We look forward to his continued technical work and the
> interesting perspective someone with a breadth of experience brings to the
> NiFi communitty.
>
> Welcome and congratulations!
> Tony
>

Re: [ANNOUNCE] New Apache NiFi Committer Toivo Adams

[Ricky Saltzer]

Congrats, Toivo!! Great to see more committers coming on board.

On Thu, Oct 29, 2015 at 4:05 PM, Joey Echeverria <joe...@gmail.com> wrote:

> Congratulations Toivo!
>
> I'm so excited to see the PMC bringing on new committers!
>
> -Joey
>
> On Thu, Oct 29, 2015 at 10:16 AM, Mark Payne <marka...@apache.org> wrote:
>
> > All,
> >
> > On behalf of the Apache NiFI PMC, I am very pleased to announce that
> Toivo
> > Adams has accepted the PMC's invitation to become a committer on the
> Apache
> > NiFi project. We greatly appreciate all of Toivo's hard work and generous
> > contributions to the project. We look forward to his continued
> involvement
> > in the project.
> >
> > Welcome, Toivo, and congratulations!
> >
>



-- 
Ricky Saltzer
http://www.cloudera.com

Re: NiFi 0.3 : Query regarding HDFS processor

[Ricky Saltzer]

Bryan is correct in that you must be able to connect to the NameNode in
order to perform an HDFS write request. Although the write does not go
*through* the NameNode, you must first consult with it in order to obtain a
DataNode write path. Depending on which distribution of Hadoop you are
using, you will need to obtain the client configuration files
(core-site.xml, hdfs-site.xml) and place them in a readable directory on
your NiFi server. Then just modify your PutHDFS processor and supply the
client configurations (e.g. /path/to/core-site.xml,/path/to/hdfs-site.xml).
That should be all you need to do.


ricky

On Thu, Oct 22, 2015 at 10:30 AM, Bryan Bende <bbe...@gmail.com> wrote:

> Hello,
>
> There is a property on PutHDFS where you can specify the Hadoop
> configuration files which tell the processor about your HDFS installation:
>
> Hadoop Configuration Resources - A file or comma separated list of files
> which contains the Hadoop file system configuration. Without this, Hadoop
> will search the classpath for a 'core-site.xml' and 'hdfs-site.xml' file or
> will revert to a default configuration.
>
> I don't think it is possible to bypass the name node since the name node
> tracks where all the files are in HDFS.
>
> -Bryan
>
>
> On Thu, Oct 22, 2015 at 2:35 AM, <ramkishan.be...@bt.com> wrote:
>
> > Hi Team,
> >
> > I have been exploring NiFi for couple of days now.
> >
> > NiFi is running on a machine which is not a prt of Hadoop cluster. I want
> > to put files into HDFS from an external source. How do I configure the
> > Hadoop cluster host details in the NiFi?
> >
> > I can get the file from remote source using GetSFTP processor and write
> it
> > into the Hadoop edge node using PutSFTP, followed by PutHDFS.
> > But I would like to know if there is any way to directly write the
> > FlowFile to HDFS using PutHDFS without writing to Hadoop name node.
> >
> > Could you please help me to identify the same?
> >
> > Regards,
> > Ramkishan Betta,
> > Consultant, BT e-serv India,
> > Bengaluru - INDIA
> >
> >
>



-- 
Ricky Saltzer
http://www.cloudera.com

Re: [ANNOUNCE] New Apache NiFi Committer Ricky Saltzer

[Ricky Saltzer]

I am very excited for the opportunity and can't wait to help drive this
project forward with the rest of you!

Ricky

On Wed, Oct 21, 2015 at 3:05 PM, dan bress <danbr...@gmail.com> wrote:

> Welcome Ricky!  Thanks for your contributions and I look forward to you
> pushing Apache NiFi forward!
>
> On Wed, Oct 21, 2015 at 3:04 PM Tony Kurc <trk...@gmail.com> wrote:
>
> > NiFi Community!
> >
> > Great news!
> >
> > On behalf of the Apache NiFI PMC, I am very pleased to announce that
> Ricky
> > Saltzer has accepted the PMC's invitation to become a committer on the
> > Apache NiFi project. We greatly appreciate all of Ricky's hard work and
> > generous contributions to the project. We look forward to his continued
> > involvement in the project.
> >
> > Welcome Ricky, and congratulations!
> >
> > Tony
> >
>



-- 
Ricky Saltzer
http://www.cloudera.com

Re: PutHDFS Configuration Issue

[Ricky Saltzer]

Hey Domenic -

Since it looks like you're going to be using Kerberos, you should probably
be aware of NIFI-997 <https://issues.apache.org/jira/browse/NIFI-997>,
which I recently posted a patch to fix. Please let us know if any other
issues come up.

Ricky

On Wed, Sep 30, 2015 at 2:01 PM, Bryan Bende <bbe...@gmail.com> wrote:

> Glad that first issue was resolved! I am by no means a kerberos expert, but
> having set this up once before, the setup should be something like the
> following:
>
> nifi.kerberos.krb5.file=/etc/krb5.conf  (or wherever your conf file is)
>
>  This is the file that would have your realms defined. Then on PutHDFS your
> properties would be something like:
>
> Kerberos Principal = myprinicpal@MYREALM
> Kerberos Keytab = /etc/security/keytabs/myprinciapl.keytab
>
> MYREALM would have to be defined in krb5.conf.
>
> -Bryan
>
>
> On Wed, Sep 30, 2015 at 1:04 PM, DomenicPuzio <
> domenic.pu...@capitalone.com>
> wrote:
>
> > Thank you so much for the quick reply! Restarting NiFi resolved the
> issue!
> > I
> > had not thought of that.
> >
> > However, a new issue was raised that I am currently working through:
> "Can't
> > find Kerberos realm". Do you have any idea where this should be set and
> > what
> > value it is looking for?
> >
> > I really appreciate the help!
> >
> >
> >
> > --
> > View this message in context:
> >
> http://apache-nifi-developer-list.39713.n7.nabble.com/PutHDFS-Configuration-Issue-tp2998p3003.html
> > Sent from the Apache NiFi Developer List mailing list archive at
> > Nabble.com.
> >
>



-- 
Ricky Saltzer
http://www.cloudera.com

Re: apache nifi oscon briefing

[Ricky Saltzer]

+1!
On Aug 21, 2015 9:04 AM, Ryan Ward ryan.wa...@gmail.com wrote:

 +1 great demo!

 On Fri, Aug 21, 2015 at 9:00 AM, Mark Payne marka...@hotmail.com wrote:

  Definitely a +1 from me. This is a really great talk and well worth the
  time for those who haven't had a chance to watch it.
 
 
  
   From: cflow...@onyxpoint.com
   Date: Fri, 21 Aug 2015 07:28:37 -0400
   Subject: Re: apache nifi oscon briefing
   To: dev@nifi.apache.org
  
   +1
  
   Sent from my iPhone
  
   On Aug 21, 2015, at 6:53 AM, estrada.a...@gmail.com 
  estrada.a...@gmail.com wrote:
  
   +1
  
   Sent from my iPhone
  
   On Aug 21, 2015, at 5:49 AM, Jennifer Barnabee 
  jennifer.barna...@gmail.com wrote:
  
   Fine by me. It would be a good addition.
  
   Sent from my iPhone
  
   On Aug 20, 2015, at 9:04 PM, Joe Witt joe.w...@gmail.com wrote:
  
   Team,
  
   Does anyone object to me adding this briefing to our apache nifi
  webcasts page?
  
   https://www.youtube.com/watch?v=sQCgtCoZyFQ
  
   It is the talk I gave at OSCON this July on Apache NiFi and its
   relationship to messaging and dataflow.
  
   After a couple days if all seems fine I'll go ahead and update the
   site to point to it.
  
   Thanks
   Joe
 
 

Re: [DISCUSS] Feature proposal: First-class Avro Support

[Ricky Saltzer]

Thanks for the write up, the proposal looks great. I imagine moving towards
a standardization on using Avro for record centric FlowFiles will make
writing future processors a lot easier, too. All of the proposed
requirements seem doable without too much trouble.

On Thu, Aug 13, 2015 at 10:26 AM, Mark Payne marka...@hotmail.com wrote:

 Bryan,

 The wiki looks good to me. A lot of stuff going on there, but I think it's
 all good. Would love to see
 a lot more support for Avro!

 Thanks
 -Mark

 
  Date: Wed, 12 Aug 2015 21:09:24 -0400
  Subject: [DISCUSS] Feature proposal: First-class Avro Support
  From: bbe...@gmail.com
  To: dev@nifi.apache.org
 
  All,
 
  Given how popular Avro has become, I'm very interested in making progress
  on providing first-class support with in NiFi. I took a stab at filling
 in
  some of the requirements on the Feature Proposal Wiki page [1] and wanted
  to get feedback from everyone to see if these ideas are headed in the
 right
  direction.
 
  Are there any major features missing from that list? any other
  recommendations?
 
  I'm also proposing that we create a new Avro bundle to capture the
  functionality that is decided upon, and we can consider whether any of
 the
  existing Avro-specific functionality in the Kite bundle could eventually
  move to the Avro bundle. If anyone feels strongly about this, or has an
  alternative recommendation, let us know.
 
  [1]
 
 https://cwiki.apache.org/confluence/display/NIFI/First-class+Avro+Support
 
  Thanks,
 
  Bryan





-- 
Ricky Saltzer
http://www.cloudera.com

Re: [DISCUSS] Feature proposal: Read-only mode as default

[Ricky Saltzer]

+1 for read-only by default. It would be nice to have some easy way to tell
if you're in edit/view mode, perhaps the canvas be black/white during view
and color during edit? or, something along those lines.

On Tue, Aug 11, 2015 at 9:57 AM, Michael Moser moser...@gmail.com wrote:

 undo seems to be the stretch goal that I think that would solve most
 concerns of unintended modifications to a graph.  +1

 Meanwhile, I'd like to caution against confirmation dialogs.  Extra clicks
 quickly annoy users while they work.  I suggest no dialog when deleting a
 single queue or processor, or when moving 'objects' around.  Perhaps bring
 a confirmation dialog into play only when deleting more than 1 'object'.

 Personally I really like the idea of a read-only mode toggle, even if it
 was not persisted as a user preference and was only remembered during the
 current browser 'session'.

 -- Mike

 On Tue, Aug 11, 2015 at 9:11 AM, Rob Moran rmo...@gmail.com wrote:

  The consensus view looks good to me. I believe preserving the current
 model
  as Joe describes it is a smart approach.
 
  An undo action and restrained use of confirmation dialogs are minimal and
  should not significantly impede experienced operators. More often than
 not,
  I'd bet a user would expect similar functionality.
 
  As is evident by the views expressed around read-only/locking, it will be
  very difficult to please a majority of users with different user modes
 and
  UI states.
 
  On Tue, Aug 11, 2015 at 8:29 AM Joe Witt joe.w...@gmail.com wrote:
 
   To summarize where we're at ...
  
   Proposed approaches (summary):
  
   - Establish a default read-only view whereby an operator can enable
   edit mode.  Use confirmation dialogs for deletes.
  
   - Keep the current model but add support for ‘undo’.
  
   - Let the user choose whether to lock their view or not as user
  preference.
  
   - For delete add more protections to make accidents less likely and
   for movement provide an explicit ‘move action’.
  
   The idea of locking seems to have some strong views on both sides and
   both sides have even been argued by the same people (i now count
   myself among that group).
  
   It looks like a consensus view there though is:
  
   - Try to make panning the view of the flow and moving components on
   the flow two specific/discrete actions to avoid accidental movement.
  
   - Add support for undo
  
   - Provide sufficient dialog/protection for delete cases.
  
   This preserves the model whereby we generally trust that the user will
   do the right thing and we’ll do more to help them and that when they
   don’t they will learn and have help to promptly restore a good state.
   How do folks feel about that?
  
   Thanks
   Joe
  
   On Tue, Aug 11, 2015 at 5:11 AM, Alex Moundalexis al...@cloudera.com
   wrote:
Counterpoint, accidents happen; I prefer to enable users to learn
 from
mistakes and exercise more care next time. You can't remove every
  mildly
sharp edge without impacting experienced operators; resist the urge
 at
  a
   few
comments to dumb down the tool.
   
If some protection is added to the UI, suggest an option for expert
   mode
that retains original functionality... that way experienced operators
   aren't
affected.
   
Alex Moundalexis
Senior Solutions Architect
Cloudera Partner Engineering
   
Sent from a mobile device; please excuse brevity.
   
On Aug 7, 2015 10:31 PM, Joe Witt joe.w...@gmail.com wrote:
   
Team,
   
We've been hearing from users of nifi that it is too easy to
accidentally move something on the flow or delete large portions of
the flow.  This is the case when panning the screen for example and
accidentally having a processor selected instead.
   
What is worth consideration then is the notion of making the flow
'read-only' by default.  In this case the user would need to
explicitly 'enable edit mode'.  We would then also support a
confirmation dialog or similar construct whenever deleting
 components
on the flow.
   
Anyone have a strong objection to this concept?  If so, do you have
 an
alternative in mind that would help avoid accidental movement?
   
Thanks
Joe
  
  --
  Rob
 




-- 
Ricky Saltzer
http://www.cloudera.com

Re: [VOTE] Release Apache NiFi 0.2.1

[Ricky Saltzer]

+1 (non-binding)

Hashes, signatures all good
Builds, tests run
Verified new account functionality
On Jul 24, 2015 8:48 AM, Matt Gilman matt.c.gil...@gmail.com wrote:

 +1 Release this package as nifi-0.2.1 (binding)

 - Checked signatures and hashes
 - LICENSE, NOTICE, and README were good
 - Build with contrib-check passes
 - Verified new users are prompted to request account


 On Thu, Jul 23, 2015 at 6:49 PM, Joe Witt joe.w...@gmail.com wrote:

  Hello
 
  I am pleased to be calling this vote for the source release of Apache
  NiFi nifi-0.2.1.
 
  The source zip, including signatures, digests, etc. can be found at:
  https://repository.apache.org/content/repositories/orgapachenifi-1058
 
  The Git tag is nifi-0.2.1-RC1
  The Git commit ID is 08cd40ddcbee21b7a7d9ff5980264a2b4ee5f1f3
 
 
 
 https://git-wip-us.apache.org/repos/asf?p=nifi.git;a=commit;h=08cd40ddcbee21b7a7d9ff5980264a2b4ee5f1f3
 
  Checksums of nifi-0.2.1-source-release.zip:
  MD5: e87b9c20660cea6045f8b16a99278aa8
  SHA1: 2dcfd583a3e16c39787dc3872362ace6e35d1514
 
  Release artifacts are signed with the following key:
  https://people.apache.org/keys/committer/joewitt.asc
 
  Should the release vote succeed convenience binaries will be made
  available at the proper dist location and NiFi downloads page.
 
  KEYS file available here:
  https://dist.apache.org/repos/dist/release/nifi/KEYS
 
  2 issues were closed/resolved for this release:
 
 
 https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020version=12333089
 
  The vote will be open for 72 hours. (18:45EST 26 July 2015)
  Please download the release candidate and evaluate the necessary items
  including checking hashes, signatures, build from source, and test.
  The please vote:
 
  [ ] +1 Release this package as nifi-0.2.1
  [ ] +0 no opinion
  [ ] -1 Do not release this package because because...