[jira] [Commented] (NIFI-2193) Command Line Keystore and Truststore utility

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15398707#comment-15398707
 ] 

ASF GitHub Bot commented on NIFI-2193:
--

Github user alopresto commented on a diff in the pull request:

https://github.com/apache/nifi/pull/695#discussion_r72741316
  
--- Diff: 
nifi-commons/nifi-security-utils/src/test/groovy/org/apache/nifi/security/util/CertificateUtilsTest.groovy
 ---
@@ -116,53 +132,7 @@ class CertificateUtilsTest extends GroovyTestCase {
 private
 static X509Certificate generateCertificate(String dn) throws 
IOException, NoSuchAlgorithmException, CertificateException, 
NoSuchProviderException, SignatureException, InvalidKeyException, 
OperatorCreationException {
 KeyPair keyPair = generateKeyPair();
-return generateCertificate(dn, keyPair);
-}
-
-/**
- * Generates a signed certificate with a specific keypair.
- *
- * @param dn the DN
- * @param keyPair the public key will be included in the certificate 
and the the private key is used to sign the certificate
- * @return the certificate
- * @throws IOException
- * @throws NoSuchAlgorithmException
- * @throws CertificateException
- * @throws NoSuchProviderException
- * @throws SignatureException
- * @throws InvalidKeyException
- * @throws OperatorCreationException
- */
-private
-static X509Certificate generateCertificate(String dn, KeyPair keyPair) 
throws IOException, NoSuchAlgorithmException, CertificateException, 
NoSuchProviderException, SignatureException, InvalidKeyException, 
OperatorCreationException {
-PrivateKey privateKey = keyPair.getPrivate();
-ContentSigner sigGen = new 
JcaContentSignerBuilder(SIGNATURE_ALGORITHM).setProvider(PROVIDER).build(privateKey);
-SubjectPublicKeyInfo subPubKeyInfo = 
SubjectPublicKeyInfo.getInstance(keyPair.getPublic().getEncoded());
-Date startDate = new Date(YESTERDAY);
-Date endDate = new Date(ONE_YEAR_FROM_NOW);
-
-X509v3CertificateBuilder certBuilder = new 
X509v3CertificateBuilder(
-new X500Name(dn),
-BigInteger.valueOf(System.currentTimeMillis()),
-startDate, endDate,
-new X500Name(dn),
-subPubKeyInfo);
-
-// Set certificate extensions
-// (1) digitalSignature extension
-certBuilder.addExtension(X509Extension.keyUsage, true,
-new KeyUsage(KeyUsage.digitalSignature | 
KeyUsage.keyEncipherment | KeyUsage.dataEncipherment | KeyUsage.keyAgreement));
-
-// (2) extendedKeyUsage extension
-Vector ekUsages = new Vector<>();
-ekUsages.add(KeyPurposeId.id_kp_clientAuth);
-ekUsages.add(KeyPurposeId.id_kp_serverAuth);
-certBuilder.addExtension(X509Extension.extendedKeyUsage, false, 
new ExtendedKeyUsage(ekUsages));
-
-// Sign the certificate
-X509CertificateHolder certificateHolder = 
certBuilder.build(sigGen);
-return new JcaX509CertificateConverter().setProvider(PROVIDER)
-.getCertificate(certificateHolder);
+return CertificateUtils.generateSelfSignedX509Certificate(keyPair, 
dn, SIGNATURE_ALGORITHM, 365);
--- End diff --

@brosander and I discussed the need for certificate migration, especially 
for the CA, and handling the trust chain amongst the nodes. I think if this 
value (throughout the tool) is increased for now, the additional use cases and 
logic to handle key/cert rollover can be addressed in a `x.1.x` release. Not 
ideal, but it is not an easy problem to tackle so close to the current release 
deadline. 


> Command Line Keystore and Truststore utility
> 
>
> Key: NIFI-2193
> URL: https://issues.apache.org/jira/browse/NIFI-2193
> Project: Apache NiFi
>  Issue Type: New Feature
>Reporter: Bryan Rosander
>Assignee: Bryan Rosander
>
> In order to facilitate secure setup of NiFi, it would be useful to have a 
> command line utility capable of generating the required keystores, 
> truststore, and relevant configuration files.
> It should be able to generate keystores for each NiFi node, a truststore that 
> they all use, and relevant passwords and configuration files for using the 
> keystores and truststore.
> Additionally, in order to support distributed deployment, a web based 
> certificate authority with corresponding client will allow for each NiFi 
> instance to generate its own keypair and then request signing by the CA.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] nifi pull request #695: NIFI-2193 - Command line SSL config utility as well ...

[alopresto]

Github user alopresto commented on a diff in the pull request:

https://github.com/apache/nifi/pull/695#discussion_r72741196
  
--- Diff: 
nifi-toolkit/nifi-toolkit-tls/src/main/java/org/apache/nifi/toolkit/tls/service/TlsCertificateAuthorityClient.java
 ---
@@ -0,0 +1,158 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.toolkit.tls.service;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.nifi.toolkit.tls.TlsToolkitMain;
+import org.apache.nifi.toolkit.tls.configuration.TlsClientConfig;
+import org.apache.nifi.toolkit.tls.configuration.TlsConfig;
+import org.apache.nifi.toolkit.tls.util.InputStreamFactory;
+import org.apache.nifi.toolkit.tls.util.OutputStreamFactory;
+import org.apache.nifi.toolkit.tls.util.PasswordUtil;
+import org.apache.nifi.toolkit.tls.util.TlsHelper;
+import org.apache.nifi.util.StringUtils;
+
+import java.io.File;
+import java.io.FileInputStream;
+import java.io.FileOutputStream;
+import java.io.IOException;
+import java.io.OutputStream;
+import java.io.OutputStreamWriter;
+import java.io.Writer;
+import java.security.KeyPair;
+import java.security.KeyStore;
+import java.security.NoSuchAlgorithmException;
+import java.security.SecureRandom;
+import java.security.cert.X509Certificate;
+
+/**
+ * Client that will generate a CSR and submit to a CA, writing out the 
results to a keystore and truststore along with a config file if successful
+ */
+public class TlsCertificateAuthorityClient {
+private final File configFile;
+private final TlsHelper tlsHelper;
+private final PasswordUtil passwordUtil;
+private final TlsClientConfig tlsClientConfig;
+private final OutputStreamFactory outputStreamFactory;
+private final ObjectMapper objectMapper;
+private final TlsCertificateSigningRequestPerformer 
tlsCertificateSigningRequestPerformer;
+
+public TlsCertificateAuthorityClient(File configFile) throws 
IOException, NoSuchAlgorithmException {
+this(configFile, FileInputStream::new, FileOutputStream::new);
+}
+
+public TlsCertificateAuthorityClient(File configFile, 
InputStreamFactory inputStreamFactory, OutputStreamFactory outputStreamFactory)
+throws IOException, NoSuchAlgorithmException {
+this(configFile, outputStreamFactory, new 
ObjectMapper().readValue(inputStreamFactory.create(configFile), 
TlsClientConfig.class));
+}
+
+public TlsCertificateAuthorityClient(File configFile, 
OutputStreamFactory outputStreamFactory, TlsClientConfig tlsClientConfig)
+throws NoSuchAlgorithmException {
+this.configFile = configFile;
+this.objectMapper = new ObjectMapper();
+this.tlsClientConfig = tlsClientConfig;
+this.tlsHelper = tlsClientConfig.createTlsHelper();
+this.passwordUtil = new PasswordUtil(new SecureRandom());
+this.outputStreamFactory = outputStreamFactory;
+this.tlsCertificateSigningRequestPerformer = 
tlsClientConfig.createCertificateSigningRequestPerformer();
+}
+
+public static void main(String[] args) throws Exception {
+TlsHelper.addBouncyCastleProvider();
+if (args.length != 1 || StringUtils.isEmpty(args[0])) {
+throw new Exception("Expected config file as only argument");
+}
+TlsCertificateAuthorityClient tlsCertificateAuthorityClient = new 
TlsCertificateAuthorityClient(new File(args[0]));
+if (tlsCertificateAuthorityClient.needsRun()) {
+
tlsCertificateAuthorityClient.generateCertificateAndGetItSigned();
+}
+}
+
+public boolean needsRun() {
+return !(new File(tlsClientConfig.getKeyStore()).exists() && new 
File(tlsClientConfig.getTrustStore()).exists());
+}
+
+public void generateCertificateAndGetItSigned() throws Exception {
+

[jira] [Commented] (NIFI-2193) Command Line Keystore and Truststore utility

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15398704#comment-15398704
 ] 

ASF GitHub Bot commented on NIFI-2193:
--

Github user alopresto commented on a diff in the pull request:

https://github.com/apache/nifi/pull/695#discussion_r72741196
  
--- Diff: 
nifi-toolkit/nifi-toolkit-tls/src/main/java/org/apache/nifi/toolkit/tls/service/TlsCertificateAuthorityClient.java
 ---
@@ -0,0 +1,158 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.toolkit.tls.service;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.nifi.toolkit.tls.TlsToolkitMain;
+import org.apache.nifi.toolkit.tls.configuration.TlsClientConfig;
+import org.apache.nifi.toolkit.tls.configuration.TlsConfig;
+import org.apache.nifi.toolkit.tls.util.InputStreamFactory;
+import org.apache.nifi.toolkit.tls.util.OutputStreamFactory;
+import org.apache.nifi.toolkit.tls.util.PasswordUtil;
+import org.apache.nifi.toolkit.tls.util.TlsHelper;
+import org.apache.nifi.util.StringUtils;
+
+import java.io.File;
+import java.io.FileInputStream;
+import java.io.FileOutputStream;
+import java.io.IOException;
+import java.io.OutputStream;
+import java.io.OutputStreamWriter;
+import java.io.Writer;
+import java.security.KeyPair;
+import java.security.KeyStore;
+import java.security.NoSuchAlgorithmException;
+import java.security.SecureRandom;
+import java.security.cert.X509Certificate;
+
+/**
+ * Client that will generate a CSR and submit to a CA, writing out the 
results to a keystore and truststore along with a config file if successful
+ */
+public class TlsCertificateAuthorityClient {
+private final File configFile;
+private final TlsHelper tlsHelper;
+private final PasswordUtil passwordUtil;
+private final TlsClientConfig tlsClientConfig;
+private final OutputStreamFactory outputStreamFactory;
+private final ObjectMapper objectMapper;
+private final TlsCertificateSigningRequestPerformer 
tlsCertificateSigningRequestPerformer;
+
+public TlsCertificateAuthorityClient(File configFile) throws 
IOException, NoSuchAlgorithmException {
+this(configFile, FileInputStream::new, FileOutputStream::new);
+}
+
+public TlsCertificateAuthorityClient(File configFile, 
InputStreamFactory inputStreamFactory, OutputStreamFactory outputStreamFactory)
+throws IOException, NoSuchAlgorithmException {
+this(configFile, outputStreamFactory, new 
ObjectMapper().readValue(inputStreamFactory.create(configFile), 
TlsClientConfig.class));
+}
+
+public TlsCertificateAuthorityClient(File configFile, 
OutputStreamFactory outputStreamFactory, TlsClientConfig tlsClientConfig)
+throws NoSuchAlgorithmException {
+this.configFile = configFile;
+this.objectMapper = new ObjectMapper();
+this.tlsClientConfig = tlsClientConfig;
+this.tlsHelper = tlsClientConfig.createTlsHelper();
+this.passwordUtil = new PasswordUtil(new SecureRandom());
+this.outputStreamFactory = outputStreamFactory;
+this.tlsCertificateSigningRequestPerformer = 
tlsClientConfig.createCertificateSigningRequestPerformer();
+}
+
+public static void main(String[] args) throws Exception {
+TlsHelper.addBouncyCastleProvider();
+if (args.length != 1 || StringUtils.isEmpty(args[0])) {
+throw new Exception("Expected config file as only argument");
+}
+TlsCertificateAuthorityClient tlsCertificateAuthorityClient = new 
TlsCertificateAuthorityClient(new File(args[0]));
+if (tlsCertificateAuthorityClient.needsRun()) {
+
tlsCertificateAuthorityClient.generateCertificateAndGetItSigned();
+}
+}
+
+public boolean needsRun() {
+return !(new 

[jira] [Commented] (NIFI-2193) Command Line Keystore and Truststore utility

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15398703#comment-15398703
 ] 

ASF GitHub Bot commented on NIFI-2193:
--

Github user alopresto commented on a diff in the pull request:

https://github.com/apache/nifi/pull/695#discussion_r72741115
  
--- Diff: 
nifi-toolkit/nifi-toolkit-tls/src/main/java/org/apache/nifi/toolkit/tls/service/TlsCertificateSigningRequestPerformer.java
 ---
@@ -0,0 +1,144 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.toolkit.tls.service;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.commons.io.IOUtils;
+import org.apache.commons.io.input.BoundedInputStream;
+import org.apache.http.HttpHost;
+import org.apache.http.client.methods.CloseableHttpResponse;
+import org.apache.http.client.methods.HttpPost;
+import org.apache.http.conn.ssl.TrustSelfSignedStrategy;
+import org.apache.http.entity.ByteArrayEntity;
+import org.apache.http.impl.client.CloseableHttpClient;
+import org.apache.http.impl.client.HttpClientBuilder;
+import org.apache.http.ssl.SSLContextBuilder;
+import org.apache.nifi.toolkit.tls.configuration.TlsClientConfig;
+import org.apache.nifi.toolkit.tls.util.TlsHelper;
+import org.bouncycastle.pkcs.jcajce.JcaPKCS10CertificationRequest;
+import org.eclipse.jetty.server.Response;
+
+import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+import java.security.KeyPair;
+import java.security.NoSuchAlgorithmException;
+import java.security.cert.X509Certificate;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.function.Supplier;
+
+public class TlsCertificateSigningRequestPerformer {
+public static final String RECEIVED_RESPONSE_CODE = "Received response 
code ";
+public static final String EXPECTED_ONE_CERTIFICATE = "Expected one 
certificate";
+public static final String EXPECTED_RESPONSE_TO_CONTAIN_HMAC = 
"Expected response to contain hmac";
+public static final String 
UNEXPECTED_HMAC_RECEIVED_POSSIBLE_MAN_IN_THE_MIDDLE = "Unexpected hmac 
received, possible man in the middle";
+public static final String EXPECTED_RESPONSE_TO_CONTAIN_CERTIFICATE = 
"Expected response to contain certificate";
+private final Supplier httpClientBuilderSupplier;
+private final String caHostname;
+private final String dn;
+private final String token;
+private final int port;
+private final TlsHelper tlsHelper;
+
+public TlsCertificateSigningRequestPerformer(TlsClientConfig 
tlsClientConfig) throws NoSuchAlgorithmException {
+this(HttpClientBuilder::create, tlsClientConfig.getCaHostname(), 
tlsClientConfig.getDn(), tlsClientConfig.getToken(), tlsClientConfig.getPort(), 
tlsClientConfig.createTlsHelper());
+}
+
+public 
TlsCertificateSigningRequestPerformer(Supplier 
httpClientBuilderSupplier, TlsClientConfig tlsClientConfig) throws 
NoSuchAlgorithmException {
+this(httpClientBuilderSupplier, tlsClientConfig.getCaHostname(), 
tlsClientConfig.getDn(), tlsClientConfig.getToken(), tlsClientConfig.getPort(), 
tlsClientConfig.createTlsHelper());
+}
+
+public 
TlsCertificateSigningRequestPerformer(Supplier 
httpClientBuilderSupplier, String caHostname, String dn, String token, int 
port, TlsHelper tlsHelper) {
+this.httpClientBuilderSupplier = httpClientBuilderSupplier;
+this.caHostname = caHostname;
+this.dn = dn;
+this.token = token;
+this.port = port;
+this.tlsHelper = tlsHelper;
+}
+
+public static String getDn(String hostname) {
+return "CN=" + hostname + ",OU=NIFI";
+}
+
+/**
+ * Submits a CSR to the Certificate authority, checks the resulting 
hmac, and returns the chain if everything succeeds
+ *
+ * @param objectMapper for 

[GitHub] nifi pull request #695: NIFI-2193 - Command line SSL config utility as well ...

[alopresto]

Github user alopresto commented on a diff in the pull request:

https://github.com/apache/nifi/pull/695#discussion_r72741115
  
--- Diff: 
nifi-toolkit/nifi-toolkit-tls/src/main/java/org/apache/nifi/toolkit/tls/service/TlsCertificateSigningRequestPerformer.java
 ---
@@ -0,0 +1,144 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.toolkit.tls.service;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.commons.io.IOUtils;
+import org.apache.commons.io.input.BoundedInputStream;
+import org.apache.http.HttpHost;
+import org.apache.http.client.methods.CloseableHttpResponse;
+import org.apache.http.client.methods.HttpPost;
+import org.apache.http.conn.ssl.TrustSelfSignedStrategy;
+import org.apache.http.entity.ByteArrayEntity;
+import org.apache.http.impl.client.CloseableHttpClient;
+import org.apache.http.impl.client.HttpClientBuilder;
+import org.apache.http.ssl.SSLContextBuilder;
+import org.apache.nifi.toolkit.tls.configuration.TlsClientConfig;
+import org.apache.nifi.toolkit.tls.util.TlsHelper;
+import org.bouncycastle.pkcs.jcajce.JcaPKCS10CertificationRequest;
+import org.eclipse.jetty.server.Response;
+
+import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+import java.security.KeyPair;
+import java.security.NoSuchAlgorithmException;
+import java.security.cert.X509Certificate;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.function.Supplier;
+
+public class TlsCertificateSigningRequestPerformer {
+public static final String RECEIVED_RESPONSE_CODE = "Received response 
code ";
+public static final String EXPECTED_ONE_CERTIFICATE = "Expected one 
certificate";
+public static final String EXPECTED_RESPONSE_TO_CONTAIN_HMAC = 
"Expected response to contain hmac";
+public static final String 
UNEXPECTED_HMAC_RECEIVED_POSSIBLE_MAN_IN_THE_MIDDLE = "Unexpected hmac 
received, possible man in the middle";
+public static final String EXPECTED_RESPONSE_TO_CONTAIN_CERTIFICATE = 
"Expected response to contain certificate";
+private final Supplier httpClientBuilderSupplier;
+private final String caHostname;
+private final String dn;
+private final String token;
+private final int port;
+private final TlsHelper tlsHelper;
+
+public TlsCertificateSigningRequestPerformer(TlsClientConfig 
tlsClientConfig) throws NoSuchAlgorithmException {
+this(HttpClientBuilder::create, tlsClientConfig.getCaHostname(), 
tlsClientConfig.getDn(), tlsClientConfig.getToken(), tlsClientConfig.getPort(), 
tlsClientConfig.createTlsHelper());
+}
+
+public 
TlsCertificateSigningRequestPerformer(Supplier 
httpClientBuilderSupplier, TlsClientConfig tlsClientConfig) throws 
NoSuchAlgorithmException {
+this(httpClientBuilderSupplier, tlsClientConfig.getCaHostname(), 
tlsClientConfig.getDn(), tlsClientConfig.getToken(), tlsClientConfig.getPort(), 
tlsClientConfig.createTlsHelper());
+}
+
+public 
TlsCertificateSigningRequestPerformer(Supplier 
httpClientBuilderSupplier, String caHostname, String dn, String token, int 
port, TlsHelper tlsHelper) {
+this.httpClientBuilderSupplier = httpClientBuilderSupplier;
+this.caHostname = caHostname;
+this.dn = dn;
+this.token = token;
+this.port = port;
+this.tlsHelper = tlsHelper;
+}
+
+public static String getDn(String hostname) {
+return "CN=" + hostname + ",OU=NIFI";
+}
+
+/**
+ * Submits a CSR to the Certificate authority, checks the resulting 
hmac, and returns the chain if everything succeeds
+ *
+ * @param objectMapper for serialization
+ * @param keyPair  the keypair to generate the csr for
+ * @throws IOException if there is a problem during the process
+ * @returnd the resulting certificate chain
+ */
+public 

[jira] [Commented] (NIFI-2193) Command Line Keystore and Truststore utility

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15398695#comment-15398695
 ] 

ASF GitHub Bot commented on NIFI-2193:
--

Github user alopresto commented on a diff in the pull request:

https://github.com/apache/nifi/pull/695#discussion_r72740653
  
--- Diff: 
nifi-toolkit/nifi-toolkit-tls/src/main/java/org/apache/nifi/toolkit/tls/service/TlsCertificateAuthorityClientSocketFactory.java
 ---
@@ -0,0 +1,77 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.toolkit.tls.service;
+
+import org.apache.http.HttpHost;
+import org.apache.http.conn.ssl.SSLConnectionSocketFactory;
+import org.apache.http.protocol.HttpContext;
+import org.bouncycastle.asn1.x500.style.BCStyle;
+import org.bouncycastle.asn1.x500.style.IETFUtils;
+import org.bouncycastle.cert.jcajce.JcaX509CertificateHolder;
+
+import javax.net.ssl.SSLContext;
+import javax.net.ssl.SSLSocket;
+import java.io.IOException;
+import java.net.InetSocketAddress;
+import java.net.Socket;
+import java.security.cert.X509Certificate;
+import java.util.List;
+
+/**
+ * Socket Factory validates that it is talking to a RootCa claiming to 
have the given hostname.  It adds the certificate
+ * to a list for later validation against the payload's hmac
+ */
+public class TlsCertificateAuthorityClientSocketFactory extends 
SSLConnectionSocketFactory {
+private final String caHostname;
+private final List certificates;
+
+public TlsCertificateAuthorityClientSocketFactory(SSLContext 
sslContext, String caHostname, List certificates) {
--- End diff --

Because we are only using this for direct communication between NiFi nodes 
and NiFi CAs, we don't have to worry about legacy compatibility, so we can 
restrict the TLS protocol version to `TLSv1.2` to enforce a strong protocol 
with strong and fast cipher suites. 


> Command Line Keystore and Truststore utility
> 
>
> Key: NIFI-2193
> URL: https://issues.apache.org/jira/browse/NIFI-2193
> Project: Apache NiFi
>  Issue Type: New Feature
>Reporter: Bryan Rosander
>Assignee: Bryan Rosander
>
> In order to facilitate secure setup of NiFi, it would be useful to have a 
> command line utility capable of generating the required keystores, 
> truststore, and relevant configuration files.
> It should be able to generate keystores for each NiFi node, a truststore that 
> they all use, and relevant passwords and configuration files for using the 
> keystores and truststore.
> Additionally, in order to support distributed deployment, a web based 
> certificate authority with corresponding client will allow for each NiFi 
> instance to generate its own keypair and then request signing by the CA.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-2193) Command Line Keystore and Truststore utility

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15398692#comment-15398692
 ] 

ASF GitHub Bot commented on NIFI-2193:
--

Github user alopresto commented on a diff in the pull request:

https://github.com/apache/nifi/pull/695#discussion_r72740552
  
--- Diff: 
nifi-toolkit/nifi-toolkit-tls/src/main/java/org/apache/nifi/toolkit/tls/service/TlsCertificateAuthorityClientSocketFactory.java
 ---
@@ -0,0 +1,77 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.toolkit.tls.service;
+
+import org.apache.http.HttpHost;
+import org.apache.http.conn.ssl.SSLConnectionSocketFactory;
+import org.apache.http.protocol.HttpContext;
+import org.bouncycastle.asn1.x500.style.BCStyle;
+import org.bouncycastle.asn1.x500.style.IETFUtils;
+import org.bouncycastle.cert.jcajce.JcaX509CertificateHolder;
+
+import javax.net.ssl.SSLContext;
+import javax.net.ssl.SSLSocket;
+import java.io.IOException;
+import java.net.InetSocketAddress;
+import java.net.Socket;
+import java.security.cert.X509Certificate;
+import java.util.List;
+
+/**
+ * Socket Factory validates that it is talking to a RootCa claiming to 
have the given hostname.  It adds the certificate
+ * to a list for later validation against the payload's hmac
+ */
+public class TlsCertificateAuthorityClientSocketFactory extends 
SSLConnectionSocketFactory {
+private final String caHostname;
+private final List certificates;
+
+public TlsCertificateAuthorityClientSocketFactory(SSLContext 
sslContext, String caHostname, List certificates) {
+super(sslContext);
+this.caHostname = caHostname;
+this.certificates = certificates;
+}
+
+@Override
+public synchronized Socket connectSocket(int connectTimeout, Socket 
socket, HttpHost host, InetSocketAddress remoteAddress,
+ InetSocketAddress 
localAddress, HttpContext context) throws IOException {
+Socket result = super.connectSocket(connectTimeout, socket, host, 
remoteAddress, localAddress, context);
+if (!SSLSocket.class.isInstance(result)) {
+throw new IOException("Expected tls socket");
+}
+SSLSocket sslSocket = (SSLSocket) result;
+java.security.cert.Certificate[] peerCertificateChain = 
sslSocket.getSession().getPeerCertificates();
+if (peerCertificateChain.length != 1) {
+throw new IOException("Expected root ca cert");
+}
+if (!X509Certificate.class.isInstance(peerCertificateChain[0])) {
+throw new IOException("Expected root ca cert in X509 format");
+}
+String cn;
+try {
+X509Certificate certificate = (X509Certificate) 
peerCertificateChain[0];
+cn = IETFUtils.valueToString(new 
JcaX509CertificateHolder(certificate).getSubject().getRDNs(BCStyle.CN)[0].getFirst().getValue());
--- End diff --

I'd recommend looking at `CertificateUtils.extractPeerDNFromSSLSocket()` 
for this operation as it handles a bit more validation. 


> Command Line Keystore and Truststore utility
> 
>
> Key: NIFI-2193
> URL: https://issues.apache.org/jira/browse/NIFI-2193
> Project: Apache NiFi
>  Issue Type: New Feature
>Reporter: Bryan Rosander
>Assignee: Bryan Rosander
>
> In order to facilitate secure setup of NiFi, it would be useful to have a 
> command line utility capable of generating the required keystores, 
> truststore, and relevant configuration files.
> It should be able to generate keystores for each NiFi node, a truststore that 
> they all use, and relevant passwords and configuration files for using the 
> keystores and truststore.
> Additionally, in order to support distributed deployment, a web based 
> 

[GitHub] nifi pull request #695: NIFI-2193 - Command line SSL config utility as well ...

[alopresto]

Github user alopresto commented on a diff in the pull request:

https://github.com/apache/nifi/pull/695#discussion_r72740552
  
--- Diff: 
nifi-toolkit/nifi-toolkit-tls/src/main/java/org/apache/nifi/toolkit/tls/service/TlsCertificateAuthorityClientSocketFactory.java
 ---
@@ -0,0 +1,77 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.toolkit.tls.service;
+
+import org.apache.http.HttpHost;
+import org.apache.http.conn.ssl.SSLConnectionSocketFactory;
+import org.apache.http.protocol.HttpContext;
+import org.bouncycastle.asn1.x500.style.BCStyle;
+import org.bouncycastle.asn1.x500.style.IETFUtils;
+import org.bouncycastle.cert.jcajce.JcaX509CertificateHolder;
+
+import javax.net.ssl.SSLContext;
+import javax.net.ssl.SSLSocket;
+import java.io.IOException;
+import java.net.InetSocketAddress;
+import java.net.Socket;
+import java.security.cert.X509Certificate;
+import java.util.List;
+
+/**
+ * Socket Factory validates that it is talking to a RootCa claiming to 
have the given hostname.  It adds the certificate
+ * to a list for later validation against the payload's hmac
+ */
+public class TlsCertificateAuthorityClientSocketFactory extends 
SSLConnectionSocketFactory {
+private final String caHostname;
+private final List certificates;
+
+public TlsCertificateAuthorityClientSocketFactory(SSLContext 
sslContext, String caHostname, List certificates) {
+super(sslContext);
+this.caHostname = caHostname;
+this.certificates = certificates;
+}
+
+@Override
+public synchronized Socket connectSocket(int connectTimeout, Socket 
socket, HttpHost host, InetSocketAddress remoteAddress,
+ InetSocketAddress 
localAddress, HttpContext context) throws IOException {
+Socket result = super.connectSocket(connectTimeout, socket, host, 
remoteAddress, localAddress, context);
+if (!SSLSocket.class.isInstance(result)) {
+throw new IOException("Expected tls socket");
+}
+SSLSocket sslSocket = (SSLSocket) result;
+java.security.cert.Certificate[] peerCertificateChain = 
sslSocket.getSession().getPeerCertificates();
+if (peerCertificateChain.length != 1) {
+throw new IOException("Expected root ca cert");
+}
+if (!X509Certificate.class.isInstance(peerCertificateChain[0])) {
+throw new IOException("Expected root ca cert in X509 format");
+}
+String cn;
+try {
+X509Certificate certificate = (X509Certificate) 
peerCertificateChain[0];
+cn = IETFUtils.valueToString(new 
JcaX509CertificateHolder(certificate).getSubject().getRDNs(BCStyle.CN)[0].getFirst().getValue());
--- End diff --

I'd recommend looking at `CertificateUtils.extractPeerDNFromSSLSocket()` 
for this operation as it handles a bit more validation. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Updated] (NIFI-2208) Support Custom Properties in Expression Language - 1.x baseline

[Yolanda M. Davis (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-2208?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yolanda M. Davis updated NIFI-2208:
---
Description: 
NIFI-1974 addressed this for the 0.x baseline but the PR does not apply cleanly 
to the 1.x baseline. Creating a separate JIRA for 1.x so that we can close out 
NIFI-1974 since 0.7.0 is ready to be released.

In addition to the merge this should also include a fix to ensure that variable 
registry is initialized on startup that variables from the registry are applied 
during EL compilation based on the following order of precedence:

1)  Flow File Attribute
2)  Processor provided variables
3)  User Defined Variables (via custom properties)
4)  JVM System Properties
5)  OS Environment Variables

Finally the following processor properties should be enabled to support 
expression language:

Put HDFS/Get HDFS/List HDFS
- Directory property

ConsumeJMS/PublishJMS
- Destination Name property

MS Connection Factory Provider
-MQ ConnectionFactory Implementation (fqn classname)
-MQ client library path
-Broker URI 

DBCP Connection Pool: 
-Database Connection URL
-Database Driver Class Name
-DB Driver jar url
-DB username 
-DB password

ConvertCSVToAvro
-add EL support for the following property
-csv charset
-and below...


  was:
NIFI-1974 addressed this for the 0.x baseline but the PR does not apply cleanly 
to the 1.x baseline. Creating a separate JIRA for 1.x so that we can close out 
NIFI-1974 since 0.7.0 is ready to be released.

In addition to the merge this should also include a fix to ensure that 
variables are discovered and applied in the registry based on the following 
order of precedence:

1)  Flow File Attribute
2)  Processor provided variables
3)  User Defined Variables (via custom properties)
4)  JVM System Properties
5)  OS Environment Variables

Finally specific the following processor's properties should be enabled to 
support expression language:

Put HDFS/Get HDFS/List HDFS
- Directory property

ConsumeJMS/PublishJMS
- Destination Name property

MS Connection Factory Provider
-MQ ConnectionFactory Implementation (fqn classname)
-MQ client library path
-Broker URI 

DBCP Connection Pool: 
-Database Connection URL
-Database Driver Class Name
-DB Driver jar url
-DB username 
-DB password

ConvertCSVToAvro
-add EL support for the following property
-csv charset
-and below...



> Support Custom Properties in Expression Language - 1.x baseline
> ---
>
> Key: NIFI-2208
> URL: https://issues.apache.org/jira/browse/NIFI-2208
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Reporter: Mark Payne
>Assignee: Yolanda M. Davis
>Priority: Blocker
> Fix For: 1.0.0
>
>
> NIFI-1974 addressed this for the 0.x baseline but the PR does not apply 
> cleanly to the 1.x baseline. Creating a separate JIRA for 1.x so that we can 
> close out NIFI-1974 since 0.7.0 is ready to be released.
> In addition to the merge this should also include a fix to ensure that 
> variable registry is initialized on startup that variables from the registry 
> are applied during EL compilation based on the following order of precedence:
> 1)  Flow File Attribute
> 2)  Processor provided variables
> 3)  User Defined Variables (via custom properties)
> 4)  JVM System Properties
> 5)  OS Environment Variables
> Finally the following processor properties should be enabled to support 
> expression language:
> Put HDFS/Get HDFS/List HDFS
> - Directory property
> ConsumeJMS/PublishJMS
> - Destination Name property
> MS Connection Factory Provider
> -MQ ConnectionFactory Implementation (fqn classname)
> -MQ client library path
> -Broker URI 
> DBCP Connection Pool: 
> -Database Connection URL
> -Database Driver Class Name
> -DB Driver jar url
> -DB username 
> -DB password
> ConvertCSVToAvro
> -add EL support for the following property
> -csv charset
> -and below...



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (NIFI-2208) Support Custom Properties in Expression Language - 1.x baseline

[Yolanda M. Davis (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-2208?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yolanda M. Davis updated NIFI-2208:
---
Description: 
NIFI-1974 addressed this for the 0.x baseline but the PR does not apply cleanly 
to the 1.x baseline. Creating a separate JIRA for 1.x so that we can close out 
NIFI-1974 since 0.7.0 is ready to be released.

In addition to the merge this should also include a fix to ensure that 
variables are discovered and applied in the registry based on the following 
order of precedence:

1)  Flow File Attribute
2)  Processor provided variables
3)  User Defined Variables (via custom properties)
4)  JVM System Properties
5)  OS Environment Variables

Finally specific the following processor's properties should be enabled to 
support expression language:

Put HDFS/Get HDFS/List HDFS
- Directory property

ConsumeJMS/PublishJMS
- Destination Name property

MS Connection Factory Provider
-MQ ConnectionFactory Implementation (fqn classname)
-MQ client library path
-Broker URI 

DBCP Connection Pool: 
-Database Connection URL
-Database Driver Class Name
-DB Driver jar url
-DB username 
-DB password

ConvertCSVToAvro
-add EL support for the following property
-csv charset
-and below...


  was:NIFI-1974 addressed this for the 0.x baseline but the PR does not apply 
cleanly to the 1.x baseline. Creating a separate JIRA for 1.x so that we can 
close out NIFI-1974 since 0.7.0 is ready to be released.


> Support Custom Properties in Expression Language - 1.x baseline
> ---
>
> Key: NIFI-2208
> URL: https://issues.apache.org/jira/browse/NIFI-2208
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Reporter: Mark Payne
>Assignee: Yolanda M. Davis
>Priority: Blocker
> Fix For: 1.0.0
>
>
> NIFI-1974 addressed this for the 0.x baseline but the PR does not apply 
> cleanly to the 1.x baseline. Creating a separate JIRA for 1.x so that we can 
> close out NIFI-1974 since 0.7.0 is ready to be released.
> In addition to the merge this should also include a fix to ensure that 
> variables are discovered and applied in the registry based on the following 
> order of precedence:
> 1)  Flow File Attribute
> 2)  Processor provided variables
> 3)  User Defined Variables (via custom properties)
> 4)  JVM System Properties
> 5)  OS Environment Variables
> Finally specific the following processor's properties should be enabled to 
> support expression language:
> Put HDFS/Get HDFS/List HDFS
> - Directory property
> ConsumeJMS/PublishJMS
> - Destination Name property
> MS Connection Factory Provider
> -MQ ConnectionFactory Implementation (fqn classname)
> -MQ client library path
> -Broker URI 
> DBCP Connection Pool: 
> -Database Connection URL
> -Database Driver Class Name
> -DB Driver jar url
> -DB username 
> -DB password
> ConvertCSVToAvro
> -add EL support for the following property
> -csv charset
> -and below...



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (NIFI-1969) Update Documentation for new Authorization Properties

[Joseph Witt (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-1969?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joseph Witt resolved NIFI-1969.
---
Resolution: Duplicate

> Update Documentation for new Authorization Properties
> -
>
> Key: NIFI-1969
> URL: https://issues.apache.org/jira/browse/NIFI-1969
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Documentation & Website
>Affects Versions: 1.0.0
>Reporter: Mark Payne
>Assignee: Andrew Lim
>Priority: Critical
>  Labels: migration
> Fix For: 1.0.0
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-2420) Add a section for Multi-tenant Authorization to the Admin Guide

[Andrew Lim (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15398625#comment-15398625
 ] 

Andrew Lim commented on NIFI-2420:
--

As part of this JIRA, will also include an update to the Admin guide with the 
new authorization system properties:

-nifi.authorizer.configuration.file
-nifi.security.user.authorizer

> Add a section for Multi-tenant Authorization to the Admin Guide
> ---
>
> Key: NIFI-2420
> URL: https://issues.apache.org/jira/browse/NIFI-2420
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Documentation & Website
>Affects Versions: 1.0.0
>Reporter: Andrew Lim
>Assignee: Joseph Witt
> Fix For: 1.0.0
>
> Attachments: Multi-tenantAuthManage.pdf, Multi-tenantAuthSetup.pdf
>
>
> Multi-tenant Authorization is a new feature in 1.0.0. A section covering 
> setup and management should be included in the documentation.
> Attaching my first attempt at the doc in case anyone has any thoughts.
> I plan to submit a PR soon depending on feedback and after more 
> editing/review on my part.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-1969) Update Documentation for new Authorization Properties

[Andrew Lim (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-1969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15398621#comment-15398621
 ] 

Andrew Lim commented on NIFI-1969:
--

[~joewitt], I agree this JIRA can be closed as a duplicate.  I was aware of 
this ticket but thought of it as a minor update to add the 1.0 authorizer 
properties (nifi.authorizer.configuration.file, nifi.security.user.authorizer) 
to the System Properties list.   I think it makes sense to add these properties 
with NIFI-2420 as part of a larger update to Admin for Multi-Tenant 
authorization.

> Update Documentation for new Authorization Properties
> -
>
> Key: NIFI-1969
> URL: https://issues.apache.org/jira/browse/NIFI-1969
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Documentation & Website
>Affects Versions: 1.0.0
>Reporter: Mark Payne
>Assignee: Andrew Lim
>Priority: Critical
>  Labels: migration
> Fix For: 1.0.0
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-2395) PersistentProvenanceRepository Deadlocks caused by a blocked journal merge

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2395?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15398619#comment-15398619
 ] 

ASF GitHub Bot commented on NIFI-2395:
--

Github user joewitt commented on a diff in the pull request:

https://github.com/apache/nifi/pull/734#discussion_r72735096
  
--- Diff: 
nifi-nar-bundles/nifi-provenance-repository-bundle/nifi-persistent-provenance-repository/src/main/java/org/apache/nifi/provenance/PersistentProvenanceRepository.java
 ---
@@ -1668,24 +1669,32 @@ public Thread newThread(final Runnable r) {
 }
 });
 
+final AtomicInteger indexingFailureCount = new 
AtomicInteger(0);
 try {
 for (int i = 0; i < 
configuration.getIndexThreadPoolSize(); i++) {
 final Callable callable = new 
Callable() {
 @Override
 public Object call() throws IOException {
 while (!eventQueue.isEmpty() || 
!finishedAdding.get()) {
-final 
Tuple tuple;
 try {
-tuple = eventQueue.poll(10, 
TimeUnit.MILLISECONDS);
-} catch (final 
InterruptedException ie) {
-continue;
+final 
Tuple tuple;
+try {
+tuple = 
eventQueue.poll(10, TimeUnit.MILLISECONDS);
+} catch (final 
InterruptedException ie) {
+continue;
--- End diff --

@markap14  we want a "Thread.currentThread().interrupt();" here as well yes?


> PersistentProvenanceRepository Deadlocks caused by a blocked journal merge
> --
>
> Key: NIFI-2395
> URL: https://issues.apache.org/jira/browse/NIFI-2395
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Affects Versions: 0.6.0, 0.7.0
>Reporter: Brian Davis
>Assignee: Joseph Witt
>Priority: Blocker
> Fix For: 1.0.0
>
>
> I have a nifi instance that I have been running for about a week and has 
> deadlocked at least 3 times during this time.  When I say deadlock the whole 
> nifi instance stops doing any progress on flowfiles.  I looked at the stack 
> trace and there are a lot of threads stuck doing tasks in the 
> PersistentProvenanceRepository.  Looking at the code I think this is what is 
> happening:
> There is a ReadWriteLock that all the reads are waiting for a write.  The 
> write is in the loop:
> {code}
> while (journalFileCount > journalCountThreshold || repoSize > 
> sizeThreshold) {
> // if a shutdown happens while we are in this loop, kill 
> the rollover thread and break
> if (this.closed.get()) {
> if (future != null) {
> future.cancel(true);
> }
> break;
> }
> if (repoSize > sizeThreshold) {
> logger.debug("Provenance Repository has exceeded its 
> size threshold; will trigger purging of oldest events");
> purgeOldEvents();
> journalFileCount = getJournalCount();
> repoSize = getSize(getLogFiles(), 0L);
> continue;
> } else {
> // if we are constrained by the number of journal 
> files rather than the size of the repo,
> // then we will just sleep a bit because another 
> thread is already actively merging the journals,
> // due to the runnable that we scheduled above
> try {
> Thread.sleep(100L);
> } catch (final InterruptedException ie) {
> }
> }
> logger.debug("Provenance Repository is still behind. 
> Keeping flow slowed down "
> + "to accommodate. Currently, there are {} 
> journal files ({} bytes) and "
> + "threshold for blocking is {} ({} bytes)", 
> journalFileCount, repoSize, journalCountThreshold, sizeThreshold);
> journalFileCount = getJournalCount();
>  

[GitHub] nifi pull request #734: NIFI-2395: Ensure that if we fail to index provenanc...

[joewitt]

Github user joewitt commented on a diff in the pull request:

https://github.com/apache/nifi/pull/734#discussion_r72735096
  
--- Diff: 
nifi-nar-bundles/nifi-provenance-repository-bundle/nifi-persistent-provenance-repository/src/main/java/org/apache/nifi/provenance/PersistentProvenanceRepository.java
 ---
@@ -1668,24 +1669,32 @@ public Thread newThread(final Runnable r) {
 }
 });
 
+final AtomicInteger indexingFailureCount = new 
AtomicInteger(0);
 try {
 for (int i = 0; i < 
configuration.getIndexThreadPoolSize(); i++) {
 final Callable callable = new 
Callable() {
 @Override
 public Object call() throws IOException {
 while (!eventQueue.isEmpty() || 
!finishedAdding.get()) {
-final 
Tuple tuple;
 try {
-tuple = eventQueue.poll(10, 
TimeUnit.MILLISECONDS);
-} catch (final 
InterruptedException ie) {
-continue;
+final 
Tuple tuple;
+try {
+tuple = 
eventQueue.poll(10, TimeUnit.MILLISECONDS);
+} catch (final 
InterruptedException ie) {
+continue;
--- End diff --

@markap14  we want a "Thread.currentThread().interrupt();" here as well yes?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Updated] (NIFI-2395) PersistentProvenanceRepository Deadlocks caused by a blocked journal merge

[Joseph Witt (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-2395?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joseph Witt updated NIFI-2395:
--
Fix Version/s: (was: 0.8.0)

> PersistentProvenanceRepository Deadlocks caused by a blocked journal merge
> --
>
> Key: NIFI-2395
> URL: https://issues.apache.org/jira/browse/NIFI-2395
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Affects Versions: 0.6.0, 0.7.0
>Reporter: Brian Davis
>Assignee: Joseph Witt
>Priority: Blocker
> Fix For: 1.0.0
>
>
> I have a nifi instance that I have been running for about a week and has 
> deadlocked at least 3 times during this time.  When I say deadlock the whole 
> nifi instance stops doing any progress on flowfiles.  I looked at the stack 
> trace and there are a lot of threads stuck doing tasks in the 
> PersistentProvenanceRepository.  Looking at the code I think this is what is 
> happening:
> There is a ReadWriteLock that all the reads are waiting for a write.  The 
> write is in the loop:
> {code}
> while (journalFileCount > journalCountThreshold || repoSize > 
> sizeThreshold) {
> // if a shutdown happens while we are in this loop, kill 
> the rollover thread and break
> if (this.closed.get()) {
> if (future != null) {
> future.cancel(true);
> }
> break;
> }
> if (repoSize > sizeThreshold) {
> logger.debug("Provenance Repository has exceeded its 
> size threshold; will trigger purging of oldest events");
> purgeOldEvents();
> journalFileCount = getJournalCount();
> repoSize = getSize(getLogFiles(), 0L);
> continue;
> } else {
> // if we are constrained by the number of journal 
> files rather than the size of the repo,
> // then we will just sleep a bit because another 
> thread is already actively merging the journals,
> // due to the runnable that we scheduled above
> try {
> Thread.sleep(100L);
> } catch (final InterruptedException ie) {
> }
> }
> logger.debug("Provenance Repository is still behind. 
> Keeping flow slowed down "
> + "to accommodate. Currently, there are {} 
> journal files ({} bytes) and "
> + "threshold for blocking is {} ({} bytes)", 
> journalFileCount, repoSize, journalCountThreshold, sizeThreshold);
> journalFileCount = getJournalCount();
> repoSize = getSize(getLogFiles(), 0L);
> }
> logger.info("Provenance Repository has now caught up with 
> rolling over journal files. Current number of "
> + "journal files to be rolled over is {}", 
> journalFileCount);
> }
> {code}
> My nifi is at the sleep indefinitely.  The reason my nifi cannot move forward 
> is because of the thread doing the merge is stopped.  The thread doing the 
> merge is at:
> {code}
> accepted = eventQueue.offer(new Tuple<>(record, blockIndex), 10, 
> TimeUnit.MILLISECONDS);
> {code}
> so the queue is full.  
> What I believe happened is that the callables created here:
> {code}
> final Callable callable = new 
> Callable() {
> @Override
> public Object call() throws IOException {
> while (!eventQueue.isEmpty() || 
> !finishedAdding.get()) {
> final 
> Tuple tuple;
> try {
> tuple = eventQueue.poll(10, 
> TimeUnit.MILLISECONDS);
> } catch (final InterruptedException 
> ie) {
> continue;
> }
> if (tuple == null) {
> continue;
> }
> indexingAction.index(tuple.getKey(), 
> indexWriter, tuple.getValue());
> }
> return null;
> }
> {code}
> finish before the offer adds its first event because I do not see any 

[jira] [Created] (NIFI-2429) Persistent provenance repo should continue despite indexing failures 0.x branch improvement

[Joseph Witt (JIRA)]

Joseph Witt created NIFI-2429:
-

 Summary: Persistent provenance repo should continue despite 
indexing failures 0.x branch improvement
 Key: NIFI-2429
 URL: https://issues.apache.org/jira/browse/NIFI-2429
 Project: Apache NiFi
  Issue Type: Sub-task
  Components: Core Framework
Affects Versions: 0.7.0
Reporter: Joseph Witt
Assignee: Mark Payne
Priority: Critical
 Fix For: 0.8.0


[~markap14] created this subtask for just the 0.x line so we can get the 1.x 
ticket addressed.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-2337) IllegalMonitorStateException when disconnecting node

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2337?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15398613#comment-15398613
 ] 

ASF GitHub Bot commented on NIFI-2337:
--

Github user asfgit closed the pull request at:

https://github.com/apache/nifi/pull/731


> IllegalMonitorStateException when disconnecting node
> 
>
> Key: NIFI-2337
> URL: https://issues.apache.org/jira/browse/NIFI-2337
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Affects Versions: 1.0.0
>Reporter: Mark Payne
>Assignee: Joseph Witt
> Fix For: 1.0.0
>
>
> When a node is disconnected from the cluster, it often logs a stacktrace 
> similar to:
> 2016-07-20 20:20:47,814 ERROR [Leader Election Notification Thread-1] 
> o.a.c.f.recipes.leader.LeaderSelector The leader threw an exception
> java.lang.IllegalMonitorStateException: You do not own the lock: 
> /leaders/Cluster Coordinator
> at 
> org.apache.curator.framework.recipes.locks.InterProcessMutex.release(InterProcessMutex.java:140)
>  ~[curator-recipes-2.10.0.jar:na]
> at 
> org.apache.curator.framework.recipes.leader.LeaderSelector.doWork(LeaderSelector.java:425)
>  [curator-recipes-2.10.0.jar:na]
> at 
> org.apache.curator.framework.recipes.leader.LeaderSelector.doWorkLoop(LeaderSelector.java:441)
>  [curator-recipes-2.10.0.jar:na]
> at 
> org.apache.curator.framework.recipes.leader.LeaderSelector.access$100(LeaderSelector.java:64)
>  [curator-recipes-2.10.0.jar:na]
> at 
> org.apache.curator.framework.recipes.leader.LeaderSelector$2.call(LeaderSelector.java:245)
>  [curator-recipes-2.10.0.jar:na]
> at 
> org.apache.curator.framework.recipes.leader.LeaderSelector$2.call(LeaderSelector.java:239)
>  [curator-recipes-2.10.0.jar:na]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266) 
> [na:1.8.0_74]
> at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) 
> [na:1.8.0_74]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266) 
> [na:1.8.0_74]
> at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) 
> [na:1.8.0_74]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266) 
> [na:1.8.0_74]
> at 
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
>  [na:1.8.0_74]
> at 
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
>  [na:1.8.0_74]
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>  [na:1.8.0_74]
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>  [na:1.8.0_74]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_74]
> This error, I believe, is harmless but is alarming and should be addressed.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (NIFI-2337) IllegalMonitorStateException when disconnecting node

[Joseph Witt (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-2337?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joseph Witt updated NIFI-2337:
--
Resolution: Fixed
Status: Resolved  (was: Patch Available)

+1 merged to master.  Confirmed logback update in build to shut off logger for 
this class.

> IllegalMonitorStateException when disconnecting node
> 
>
> Key: NIFI-2337
> URL: https://issues.apache.org/jira/browse/NIFI-2337
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Affects Versions: 1.0.0
>Reporter: Mark Payne
>Assignee: Joseph Witt
> Fix For: 1.0.0
>
>
> When a node is disconnected from the cluster, it often logs a stacktrace 
> similar to:
> 2016-07-20 20:20:47,814 ERROR [Leader Election Notification Thread-1] 
> o.a.c.f.recipes.leader.LeaderSelector The leader threw an exception
> java.lang.IllegalMonitorStateException: You do not own the lock: 
> /leaders/Cluster Coordinator
> at 
> org.apache.curator.framework.recipes.locks.InterProcessMutex.release(InterProcessMutex.java:140)
>  ~[curator-recipes-2.10.0.jar:na]
> at 
> org.apache.curator.framework.recipes.leader.LeaderSelector.doWork(LeaderSelector.java:425)
>  [curator-recipes-2.10.0.jar:na]
> at 
> org.apache.curator.framework.recipes.leader.LeaderSelector.doWorkLoop(LeaderSelector.java:441)
>  [curator-recipes-2.10.0.jar:na]
> at 
> org.apache.curator.framework.recipes.leader.LeaderSelector.access$100(LeaderSelector.java:64)
>  [curator-recipes-2.10.0.jar:na]
> at 
> org.apache.curator.framework.recipes.leader.LeaderSelector$2.call(LeaderSelector.java:245)
>  [curator-recipes-2.10.0.jar:na]
> at 
> org.apache.curator.framework.recipes.leader.LeaderSelector$2.call(LeaderSelector.java:239)
>  [curator-recipes-2.10.0.jar:na]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266) 
> [na:1.8.0_74]
> at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) 
> [na:1.8.0_74]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266) 
> [na:1.8.0_74]
> at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) 
> [na:1.8.0_74]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266) 
> [na:1.8.0_74]
> at 
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
>  [na:1.8.0_74]
> at 
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
>  [na:1.8.0_74]
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>  [na:1.8.0_74]
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>  [na:1.8.0_74]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_74]
> This error, I believe, is harmless but is alarming and should be addressed.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-2337) IllegalMonitorStateException when disconnecting node

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2337?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15398612#comment-15398612
 ] 

ASF subversion and git services commented on NIFI-2337:
---

Commit 3a153d3a486bf201d2bc755f12701e9b7230d5e0 in nifi's branch 
refs/heads/master from [~markap14]
[ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=3a153d3 ]

NIFI-2337 This closes #731. Turn off logging for 
org.apache.curator.framework.recipes.leader.LeaderSelector since it logs only a 
single error, and that error is a but in Curator


> IllegalMonitorStateException when disconnecting node
> 
>
> Key: NIFI-2337
> URL: https://issues.apache.org/jira/browse/NIFI-2337
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Affects Versions: 1.0.0
>Reporter: Mark Payne
>Assignee: Joseph Witt
> Fix For: 1.0.0
>
>
> When a node is disconnected from the cluster, it often logs a stacktrace 
> similar to:
> 2016-07-20 20:20:47,814 ERROR [Leader Election Notification Thread-1] 
> o.a.c.f.recipes.leader.LeaderSelector The leader threw an exception
> java.lang.IllegalMonitorStateException: You do not own the lock: 
> /leaders/Cluster Coordinator
> at 
> org.apache.curator.framework.recipes.locks.InterProcessMutex.release(InterProcessMutex.java:140)
>  ~[curator-recipes-2.10.0.jar:na]
> at 
> org.apache.curator.framework.recipes.leader.LeaderSelector.doWork(LeaderSelector.java:425)
>  [curator-recipes-2.10.0.jar:na]
> at 
> org.apache.curator.framework.recipes.leader.LeaderSelector.doWorkLoop(LeaderSelector.java:441)
>  [curator-recipes-2.10.0.jar:na]
> at 
> org.apache.curator.framework.recipes.leader.LeaderSelector.access$100(LeaderSelector.java:64)
>  [curator-recipes-2.10.0.jar:na]
> at 
> org.apache.curator.framework.recipes.leader.LeaderSelector$2.call(LeaderSelector.java:245)
>  [curator-recipes-2.10.0.jar:na]
> at 
> org.apache.curator.framework.recipes.leader.LeaderSelector$2.call(LeaderSelector.java:239)
>  [curator-recipes-2.10.0.jar:na]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266) 
> [na:1.8.0_74]
> at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) 
> [na:1.8.0_74]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266) 
> [na:1.8.0_74]
> at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) 
> [na:1.8.0_74]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266) 
> [na:1.8.0_74]
> at 
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
>  [na:1.8.0_74]
> at 
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
>  [na:1.8.0_74]
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>  [na:1.8.0_74]
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>  [na:1.8.0_74]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_74]
> This error, I believe, is harmless but is alarming and should be addressed.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] nifi pull request #731: NIFI-2337: Turn off logging for org.apache.curator.f...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/nifi/pull/731


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Resolved] (NIFI-2386) HTTP Site-to-Site fails if port is not specified explicitly

[Joseph Witt (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-2386?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joseph Witt resolved NIFI-2386.
---
Resolution: Fixed

+1 merged to master.
Full clean build w/contrib check good.  Verified functionality for http and 
checked https but did not run that.  Ran test with HTTP.

> HTTP Site-to-Site fails if port is not specified explicitly
> ---
>
> Key: NIFI-2386
> URL: https://issues.apache.org/jira/browse/NIFI-2386
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Affects Versions: 1.0.0
>Reporter: Koji Kawamura
>Assignee: Joseph Witt
> Fix For: 1.0.0
>
>
> It fails if a specified URL doesn't have a port number explicitly. It should 
> use 80 for http and 443 for https if port is not defined in a given URL.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] nifi pull request #716: NIFI-2386: HTTP Site-to-Site fails without port no

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/nifi/pull/716


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (NIFI-2386) HTTP Site-to-Site fails if port is not specified explicitly

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2386?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15398605#comment-15398605
 ] 

ASF GitHub Bot commented on NIFI-2386:
--

Github user asfgit closed the pull request at:

https://github.com/apache/nifi/pull/716


> HTTP Site-to-Site fails if port is not specified explicitly
> ---
>
> Key: NIFI-2386
> URL: https://issues.apache.org/jira/browse/NIFI-2386
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Affects Versions: 1.0.0
>Reporter: Koji Kawamura
>Assignee: Joseph Witt
> Fix For: 1.0.0
>
>
> It fails if a specified URL doesn't have a port number explicitly. It should 
> use 80 for http and 443 for https if port is not defined in a given URL.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-2386) HTTP Site-to-Site fails if port is not specified explicitly

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2386?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15398604#comment-15398604
 ] 

ASF subversion and git services commented on NIFI-2386:
---

Commit b3968678471dec1eafd1f8d90eab86fd7035a4e2 in nifi's branch 
refs/heads/master from [~ijokarumawak]
[ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=b396867 ]

NIFI-2386 This closes #716. Site-to-Site fails without port no

It fails if a given URL doesn't have port in it.
This fixes its behavior with default http 80 and https 443 port.


> HTTP Site-to-Site fails if port is not specified explicitly
> ---
>
> Key: NIFI-2386
> URL: https://issues.apache.org/jira/browse/NIFI-2386
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Affects Versions: 1.0.0
>Reporter: Koji Kawamura
>Assignee: Joseph Witt
> Fix For: 1.0.0
>
>
> It fails if a specified URL doesn't have a port number explicitly. It should 
> use 80 for http and 443 for https if port is not defined in a given URL.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (NIFI-2321) Update Getting Started documentation for 1.0 changes

[Joseph Witt (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-2321?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joseph Witt resolved NIFI-2321.
---
Resolution: Fixed

+1 merged to master.  Thank you

> Update Getting Started documentation for 1.0 changes
> 
>
> Key: NIFI-2321
> URL: https://issues.apache.org/jira/browse/NIFI-2321
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Documentation & Website
>Affects Versions: 1.0.0
>Reporter: Andrew Lim
>Assignee: Joseph Witt
> Fix For: 1.0.0
>
>
> Getting Started guide needs to be updated for the changes in the 1.0 UI.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-2028) Site-to-Site Transit URI is inconsistent

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15398583#comment-15398583
 ] 

ASF GitHub Bot commented on NIFI-2028:
--

Github user ijokarumawak commented on a diff in the pull request:

https://github.com/apache/nifi/pull/529#discussion_r72733243
  
--- Diff: 
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-site-to-site/src/main/java/org/apache/nifi/remote/StandardRemoteGroupPort.java
 ---
@@ -366,6 +371,14 @@ private int receiveFlowFiles(final Transaction 
transaction, final ProcessContext
 return flowFilesReceived.size();
 }
 
+private String createTransitUri(Transaction transaction, String 
sourceFlowFileIdentifier) {
+if (transaction instanceof CustomTransitUriTransaction) {
--- End diff --

@markap14 Thanks for the comment, I agree with that. I felt that I should 
have kept Peer completely immutable, so I might bypass the createTransitUri 
call to CommunicationsSession. Will update the PR soon.


> Site-to-Site Transit URI is inconsistent
> 
>
> Key: NIFI-2028
> URL: https://issues.apache.org/jira/browse/NIFI-2028
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core UI
>Affects Versions: 1.0.0
>Reporter: Koji Kawamura
>Assignee: Koji Kawamura
> Fix For: 1.0.0
>
> Attachments: send-to-the-same-box.png
>
>
> Site-to-Site client and server create provenance event at both end, and those 
> have Transit URI in it to record how flow files are transferred. However, the 
> URI formats are inconsistent among RAW vs HTTP.
> Test result as follows:
> - These port numbers are configurable in nifi.properties
> - 3080: Web API port (nifi.web.http.port)
> - 3081: Site-to-Site RAW Socket port (nifi.remote.input.socket.port)
> *Before Fix*
> PUSH - RAW
> |Client - SEND: | nifi://localhost:3081/flow-file-uuid |
> |Server - RECEIVE: | nifi://localhost:3081/flow-file-uuid|
> PULL - RAW
> |Client - RECEIVE:| nifi://localhost:3081flow-file-uuid|
> |Server - SEND:   | nifi://localhost:3081/flow-file-uuid|
> PUSH - HTTP
> |Client - SEND:   | http://127.0.0.1:3080/nifi-api/flow-file-uuid|
> |Server - RECEIVE:| nifi://127.0.0.1:57390|
> PULL - HTTP
> |Client - RECEIVE:| http://127.0.0.1:3080/flow-file-uuid|
> |Server - SEND:   | nifi://127.0.0.1:57673|
> *Issues*
> - PULL - RAW, Client - RECEIVE: lacks '/' in between port and flow-file uuid
> - RAW uses server's host and port on both end (by transit url prefix), HTTP 
> should follow this rule
> - HTTP transit uri looks like REST endpoint but it is not a real endpoint. It 
> should be an actual endpoint URI
> - RAW uses hostname, while HTTP uses IP address
> *After Fix*
> PUSH - RAW
> |Client - SEND: | nifi://localhost:3081/flow-file-uuid |
> |Server - RECEIVE: | nifi://localhost:3081/flow-file-uuid|
> PULL - RAW
> |Client - RECEIVE: | nifi://localhost:3081/flow-file-uuid|
> |Server - SEND:   | nifi://localhost:3081/flow-file-uuid|
> PUSH - HTTP
> |Client - SEND:   | 
> http://localhost:3080/nifi-api/data-transfer/input-ports/$port-id/transactions/$tx-id/flow-files|
> |Server - RECEIVE:| 
> http://localhost:3080/nifi-api/data-transfer/input-ports/$port-id/transactions/$tx-id/flow-files|
> PULL - HTTP
> |Client - RECEIVE: | 
> http://localhost:3080/nifi-api/data-transfer/output-ports/$port-id/transactions/$tx-id/flow-files|
> |Server - SEND:   | 
> http://localhost:3080/nifi-api/data-transfer/output-ports/$port-id/transactions/$tx-id/flow-files|



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] nifi pull request #529: NIFI-2028: Fixed Site-to-Site Transit URI

[ijokarumawak]

Github user ijokarumawak commented on a diff in the pull request:

https://github.com/apache/nifi/pull/529#discussion_r72733243
  
--- Diff: 
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-site-to-site/src/main/java/org/apache/nifi/remote/StandardRemoteGroupPort.java
 ---
@@ -366,6 +371,14 @@ private int receiveFlowFiles(final Transaction 
transaction, final ProcessContext
 return flowFilesReceived.size();
 }
 
+private String createTransitUri(Transaction transaction, String 
sourceFlowFileIdentifier) {
+if (transaction instanceof CustomTransitUriTransaction) {
--- End diff --

@markap14 Thanks for the comment, I agree with that. I felt that I should 
have kept Peer completely immutable, so I might bypass the createTransitUri 
call to CommunicationsSession. Will update the PR soon.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (NIFI-2321) Update Getting Started documentation for 1.0 changes

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15398580#comment-15398580
 ] 

ASF GitHub Bot commented on NIFI-2321:
--

Github user asfgit closed the pull request at:

https://github.com/apache/nifi/pull/707


> Update Getting Started documentation for 1.0 changes
> 
>
> Key: NIFI-2321
> URL: https://issues.apache.org/jira/browse/NIFI-2321
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Documentation & Website
>Affects Versions: 1.0.0
>Reporter: Andrew Lim
>Assignee: Joseph Witt
> Fix For: 1.0.0
>
>
> Getting Started guide needs to be updated for the changes in the 1.0 UI.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-2321) Update Getting Started documentation for 1.0 changes

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15398579#comment-15398579
 ] 

ASF subversion and git services commented on NIFI-2321:
---

Commit 986f951ae82c1f0bac5f3658b8a749fbd5756646 in nifi's branch 
refs/heads/master from [~andrewmlim]
[ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=986f951 ]

NIFI-2321 This closes #707. Update Getting Started doc for 1.0 UI changes: made 
text edits/improvements and added new screenshots


> Update Getting Started documentation for 1.0 changes
> 
>
> Key: NIFI-2321
> URL: https://issues.apache.org/jira/browse/NIFI-2321
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Documentation & Website
>Affects Versions: 1.0.0
>Reporter: Andrew Lim
>Assignee: Joseph Witt
> Fix For: 1.0.0
>
>
> Getting Started guide needs to be updated for the changes in the 1.0 UI.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-2428) Hitting escape from stats pane causes the pane to go away but the canvas is still glossed over

[Joseph Percivall (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2428?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15398574#comment-15398574
 ] 

Joseph Percivall commented on NIFI-2428:


This is also relevant when the stats panel is brought up from the summary pane. 
If escape is hit (closing the stats panel) the summary panel can *mostly* 
longer be clicked. There is no glossed over effect and the X to close the 
summary pane can be clicked.

> Hitting escape from stats pane causes the pane to go away but the canvas is 
> still glossed over
> --
>
> Key: NIFI-2428
> URL: https://issues.apache.org/jira/browse/NIFI-2428
> Project: Apache NiFi
>  Issue Type: Sub-task
>  Components: Core UI
>Reporter: Joseph Percivall
>Priority: Minor
>
> Component on the graph
> go to the stats of the component
> hit escape
> observe the window go away but the background is still glossed over and 
> unclickable
> have to refresh window to regain usability



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-2304) Cluster Coordinator reported incorrectly

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15398562#comment-15398562
 ] 

ASF GitHub Bot commented on NIFI-2304:
--

Github user ijokarumawak commented on a diff in the pull request:

https://github.com/apache/nifi/pull/688#discussion_r72731867
  
--- Diff: 
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-cluster/src/main/java/org/apache/nifi/cluster/coordination/heartbeat/ClusterProtocolHeartbeatMonitor.java
 ---
@@ -91,12 +91,8 @@ public ClusterProtocolHeartbeatMonitor(final 
ClusterCoordinator clusterCoordinat
 this.clusterNodesPath = 
zkClientConfig.resolvePath("cluster/nodes");
 
 String hostname = 
properties.getProperty(NiFiProperties.CLUSTER_NODE_ADDRESS);
-if (hostname == null) {
-try {
-hostname = InetAddress.getLocalHost().getHostName();
-} catch (UnknownHostException e) {
-throw new RuntimeException("Unable to determine local 
hostname and the '" + NiFiProperties.CLUSTER_NODE_ADDRESS + "' property is not 
set");
-}
+if (hostname == null || hostname.trim().isEmpty()) {
+hostname = "localhost";
--- End diff --

Thanks for the explanation. I found that [docs says 
so](https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#clustering).

I have read `nifi.remote.input.socket.host` description before, and 
wondered what's the reason of these difference:
```
By default, it is the value from InetAddress.getLocalHost().getHostName(). 
On UNIX-like operating systems, this is typically the output from the hostname 
command.
```

I understand that as to keep how it was done in 0.x.


> Cluster Coordinator reported incorrectly
> 
>
> Key: NIFI-2304
> URL: https://issues.apache.org/jira/browse/NIFI-2304
> Project: Apache NiFi
>  Issue Type: Bug
>Affects Versions: 1.0.0
>Reporter: Bryan Bende
>Assignee: Mark Payne
>Priority: Minor
> Fix For: 1.0.0
>
>
> I created a two node cluster locally, running embedded ZK on node1.. both 
> nodes start up, but when trying to access the UI on the second node, it says 
> no coordinator exists. 
> Looking in the logs the coordinator is being reported as ":8889" where 8889 
> is the node protocol port. I left the node host blank assuming it would 
> default to localhost, but looks like it isn't. When I filled in the node host 
> everything worked as expected.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-1876) Clustering - Merge all responses based on authorization

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-1876?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15398561#comment-15398561
 ] 

ASF GitHub Bot commented on NIFI-1876:
--

Github user JPercivall commented on a diff in the pull request:

https://github.com/apache/nifi/pull/694#discussion_r72731831
  
--- Diff: 
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/main/java/org/apache/nifi/web/api/dto/EntityFactory.java
 ---
@@ -77,6 +77,8 @@ public ProcessorEntity createProcessorEntity(final 
ProcessorDTO dto, final Revis
 if (dto != null) {
 entity.setPermissions(permissions);
 entity.setStatus(status);
+status.setCanRead(permissions.getCanRead());
--- End diff --

Yup, hit it when I was trying to delete a connection:

2016-07-28 21:41:44,415 INFO [NiFi Web Server-231] 
org.apache.nifi.web.filter.RequestLogger Attempting request for (anonymous) 
DELETE 
http://localhost:8080/nifi-api/connections/344c598c-0156-1000--53115355 
(source ip: 127.0.0.1)
2016-07-28 21:41:44,429 ERROR [NiFi Web Server-231] 
o.a.nifi.web.api.config.ThrowableMapper An unexpected error has occurred: 
java.lang.NullPointerException. Returning Internal Server Error response.
java.lang.NullPointerException: null
at 
org.apache.nifi.web.api.dto.EntityFactory.createConnectionEntity(EntityFactory.java:251)
 ~[classes/:na]
at 
org.apache.nifi.web.StandardNiFiServiceFacade.deleteConnection(StandardNiFiServiceFacade.java:858)
 ~[classes/:1.0.0-SNAPSHOT]
at 
org.apache.nifi.web.StandardNiFiServiceFacade$$FastClassBySpringCGLIB$$358780e0.invoke()
 ~[classes/:1.0.0-SNAPSHOT]



> Clustering - Merge all responses based on authorization
> ---
>
> Key: NIFI-1876
> URL: https://issues.apache.org/jira/browse/NIFI-1876
> Project: Apache NiFi
>  Issue Type: Sub-task
>  Components: Core Framework
>Reporter: Matt Gilman
>Assignee: Jeff Storck
> Fix For: 1.0.0
>
>
> Each node in a cluster may have a different view of the authorization access 
> policies simply to in the timing of updates. Because of this, all requests 
> need to be merged accordingly.
> Requests are directed at a specific resource. These would result in some 403 
> responses.
> Some requests are contain a filtered view of a number of resources. These 
> would need to be updated to return the most restrictive set of responses.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] nifi pull request #694: NIFI-1876 Implements merging of responses to success...

[JPercivall]

Github user JPercivall commented on a diff in the pull request:

https://github.com/apache/nifi/pull/694#discussion_r72731831
  
--- Diff: 
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/main/java/org/apache/nifi/web/api/dto/EntityFactory.java
 ---
@@ -77,6 +77,8 @@ public ProcessorEntity createProcessorEntity(final 
ProcessorDTO dto, final Revis
 if (dto != null) {
 entity.setPermissions(permissions);
 entity.setStatus(status);
+status.setCanRead(permissions.getCanRead());
--- End diff --

Yup, hit it when I was trying to delete a connection:

2016-07-28 21:41:44,415 INFO [NiFi Web Server-231] 
org.apache.nifi.web.filter.RequestLogger Attempting request for (anonymous) 
DELETE 
http://localhost:8080/nifi-api/connections/344c598c-0156-1000--53115355 
(source ip: 127.0.0.1)
2016-07-28 21:41:44,429 ERROR [NiFi Web Server-231] 
o.a.nifi.web.api.config.ThrowableMapper An unexpected error has occurred: 
java.lang.NullPointerException. Returning Internal Server Error response.
java.lang.NullPointerException: null
at 
org.apache.nifi.web.api.dto.EntityFactory.createConnectionEntity(EntityFactory.java:251)
 ~[classes/:na]
at 
org.apache.nifi.web.StandardNiFiServiceFacade.deleteConnection(StandardNiFiServiceFacade.java:858)
 ~[classes/:1.0.0-SNAPSHOT]
at 
org.apache.nifi.web.StandardNiFiServiceFacade$$FastClassBySpringCGLIB$$358780e0.invoke()
 ~[classes/:1.0.0-SNAPSHOT]



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request #688: NIFI-2304: If node address is not set, default to lo...

[ijokarumawak]

Github user ijokarumawak commented on a diff in the pull request:

https://github.com/apache/nifi/pull/688#discussion_r72731867
  
--- Diff: 
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-cluster/src/main/java/org/apache/nifi/cluster/coordination/heartbeat/ClusterProtocolHeartbeatMonitor.java
 ---
@@ -91,12 +91,8 @@ public ClusterProtocolHeartbeatMonitor(final 
ClusterCoordinator clusterCoordinat
 this.clusterNodesPath = 
zkClientConfig.resolvePath("cluster/nodes");
 
 String hostname = 
properties.getProperty(NiFiProperties.CLUSTER_NODE_ADDRESS);
-if (hostname == null) {
-try {
-hostname = InetAddress.getLocalHost().getHostName();
-} catch (UnknownHostException e) {
-throw new RuntimeException("Unable to determine local 
hostname and the '" + NiFiProperties.CLUSTER_NODE_ADDRESS + "' property is not 
set");
-}
+if (hostname == null || hostname.trim().isEmpty()) {
+hostname = "localhost";
--- End diff --

Thanks for the explanation. I found that [docs says 
so](https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#clustering).

I have read `nifi.remote.input.socket.host` description before, and 
wondered what's the reason of these difference:
```
By default, it is the value from InetAddress.getLocalHost().getHostName(). 
On UNIX-like operating systems, this is typically the output from the hostname 
command.
```

I understand that as to keep how it was done in 0.x.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (NIFI-1831) Allow encrypted passwords in configuration files

[Andy LoPresto (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-1831?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15398556#comment-15398556
 ] 

Andy LoPresto commented on NIFI-1831:
-

I have completed the architecture and the first implementation (AES/GCM 
128/256-bit encryption) which provides transparent encryption/decryption of the 
sensitive {{nifi.properties}} values to the rest of the application. I had to 
pause to review [NIFI-2193] and will now resume writing the command-line 
utility to allow deployers to write these encrypted values into the properties 
files. 

> Allow encrypted passwords in configuration files
> 
>
> Key: NIFI-1831
> URL: https://issues.apache.org/jira/browse/NIFI-1831
> Project: Apache NiFi
>  Issue Type: New Feature
>  Components: Configuration, Core Framework
>Affects Versions: 0.6.1
>Reporter: Andy LoPresto
>Assignee: Andy LoPresto
>Priority: Critical
>  Labels: configuration, encryption, password, security
> Fix For: 1.0.0
>
>   Original Estimate: 504h
>  Remaining Estimate: 504h
>
> Storing passwords in plaintext in configuration files is not a security best 
> practice. While file access can be restricted through OS permissions, these 
> configuration files can be accidentally checked into source control, shared 
> or deployed to multiple instances, etc. 
> NiFi should allow a deployer to provide an encrypted password in the 
> configuration file to minimize exposure of the passwords. On application 
> start-up, NiFi should decrypt the passwords in memory. NiFi should also 
> include a utility to encrypt the raw passwords (and optionally populate the 
> configuration files and provide additional metadata in the configuration 
> files). 
> I am aware this simply shifts the responsibility/delegation of trust from the 
> passwords in the properties file to a new location on the same system, but 
> mitigating the visibility of the raw passwords in the properties file can be 
> one step in a defense in depth approach and is often mandated by security 
> policies within organizations using NiFi. 
> The key used for encryption should not be hard-coded into the application 
> source code, nor should it be universally consistent. The key could be 
> determined by reading static information from the deployed system and feeding 
> it to a key derivation function based on a cryptographically-secure hash 
> function, such as PBKDF2, bcrypt, or scrypt. However, this does introduce 
> upgrade, system migration, and portability issues. These challenges will have 
> to be kept in consideration when determining the key derivation process. 
> Manual key entry is a possibility, and then the master key would only be 
> present in memory, but this prevents automatic reboot on loss of power or 
> other recovery scenario. 
> This must be backward-compatible to allow systems with plaintext passwords to 
> continue operating. Options for achieving this are to only attempt to decrypt 
> passwords when a sibling property is present, or to match a specific format. 
> For these examples, I have used the following default values:
> {code}
> password: thisIsABadPassword
> key: 0123456789ABCDEFFEDCBA98765432100123456789ABCDEFFEDCBA9876543210
> iv:  0123456789ABCDEFFEDCBA9876543210
> algorithm: AES/CBC 256-bit
> {code}
> **Note: These values should not be used in production systems -- the key and 
> IV are common test values, and an AEAD cipher is preferable to provide cipher 
> text integrity assurances, however OpenSSL does not support the use of AEAD 
> ciphers for command-line encryption at this time**
> Example 1: *here the sibling property indicates the password is encrypted and 
> with which implementation; the absence of the property would default to a raw 
> password*
> {code}
> hw12203:/Users/alopresto/Workspace/scratch/encrypted-passwords (master) 
> alopresto
>  0s @ 16:25:56 $ echo "thisIsABadPassword" > password.txt
> hw12203:/Users/alopresto/Workspace/scratch/encrypted-passwords (master) 
> alopresto
>  0s @ 16:26:47 $ ossl aes-256-cbc -e -nosalt -p -K 
> 0123456789ABCDEFFEDCBA98765432100123456789ABCDEFFEDCBA9876543210 -iv 
> 0123456789ABCDEFFEDCBA9876543210 -a -in password.txt -out password.enc
> key=0123456789ABCDEFFEDCBA98765432100123456789ABCDEFFEDCBA9876543210
> iv =0123456789ABCDEFFEDCBA9876543210
> hw12203:/Users/alopresto/Workspace/scratch/encrypted-passwords (master) 
> alopresto
>  0s @ 16:27:09 $ xxd password.enc
> 000: 5643 5856 6146 6250 4158 364f 5743 7646  VCXVaFbPAX6OWCvF
> 010: 6963 6b76 4a63 7744 3854 6b67 3731 4c76  ickvJcwD8Tkg71Lv
> 020: 4d38 6d32 7952 4776 5739 413d 0a M8m2yRGvW9A=.
> hw12203:/Users/alopresto/Workspace/scratch/encrypted-passwords (master) 
> alopresto
>  0s @ 16:27:16 $ more password.enc
> 

[jira] [Comment Edited] (NIFI-1831) Allow encrypted passwords in configuration files

[Andy LoPresto (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-1831?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15398556#comment-15398556
 ] 

Andy LoPresto edited comment on NIFI-1831 at 7/29/16 1:34 AM:
--

I have completed the architecture and the first implementation (AES/GCM 
128/256-bit encryption) which provides transparent encryption/decryption of the 
sensitive {{nifi.properties}} values to the rest of the application. I had to 
pause to review [NIFI-2193] and will now resume writing the command-line 
utility to allow deployers to write these encrypted values into the properties 
files. 

Current progress can be seen 
[https://github.com/apache/nifi/compare/master...alopresto:NIFI-1831?expand=1](here).
 


was (Author: alopresto):
I have completed the architecture and the first implementation (AES/GCM 
128/256-bit encryption) which provides transparent encryption/decryption of the 
sensitive {{nifi.properties}} values to the rest of the application. I had to 
pause to review [NIFI-2193] and will now resume writing the command-line 
utility to allow deployers to write these encrypted values into the properties 
files. 

> Allow encrypted passwords in configuration files
> 
>
> Key: NIFI-1831
> URL: https://issues.apache.org/jira/browse/NIFI-1831
> Project: Apache NiFi
>  Issue Type: New Feature
>  Components: Configuration, Core Framework
>Affects Versions: 0.6.1
>Reporter: Andy LoPresto
>Assignee: Andy LoPresto
>Priority: Critical
>  Labels: configuration, encryption, password, security
> Fix For: 1.0.0
>
>   Original Estimate: 504h
>  Remaining Estimate: 504h
>
> Storing passwords in plaintext in configuration files is not a security best 
> practice. While file access can be restricted through OS permissions, these 
> configuration files can be accidentally checked into source control, shared 
> or deployed to multiple instances, etc. 
> NiFi should allow a deployer to provide an encrypted password in the 
> configuration file to minimize exposure of the passwords. On application 
> start-up, NiFi should decrypt the passwords in memory. NiFi should also 
> include a utility to encrypt the raw passwords (and optionally populate the 
> configuration files and provide additional metadata in the configuration 
> files). 
> I am aware this simply shifts the responsibility/delegation of trust from the 
> passwords in the properties file to a new location on the same system, but 
> mitigating the visibility of the raw passwords in the properties file can be 
> one step in a defense in depth approach and is often mandated by security 
> policies within organizations using NiFi. 
> The key used for encryption should not be hard-coded into the application 
> source code, nor should it be universally consistent. The key could be 
> determined by reading static information from the deployed system and feeding 
> it to a key derivation function based on a cryptographically-secure hash 
> function, such as PBKDF2, bcrypt, or scrypt. However, this does introduce 
> upgrade, system migration, and portability issues. These challenges will have 
> to be kept in consideration when determining the key derivation process. 
> Manual key entry is a possibility, and then the master key would only be 
> present in memory, but this prevents automatic reboot on loss of power or 
> other recovery scenario. 
> This must be backward-compatible to allow systems with plaintext passwords to 
> continue operating. Options for achieving this are to only attempt to decrypt 
> passwords when a sibling property is present, or to match a specific format. 
> For these examples, I have used the following default values:
> {code}
> password: thisIsABadPassword
> key: 0123456789ABCDEFFEDCBA98765432100123456789ABCDEFFEDCBA9876543210
> iv:  0123456789ABCDEFFEDCBA9876543210
> algorithm: AES/CBC 256-bit
> {code}
> **Note: These values should not be used in production systems -- the key and 
> IV are common test values, and an AEAD cipher is preferable to provide cipher 
> text integrity assurances, however OpenSSL does not support the use of AEAD 
> ciphers for command-line encryption at this time**
> Example 1: *here the sibling property indicates the password is encrypted and 
> with which implementation; the absence of the property would default to a raw 
> password*
> {code}
> hw12203:/Users/alopresto/Workspace/scratch/encrypted-passwords (master) 
> alopresto
>  0s @ 16:25:56 $ echo "thisIsABadPassword" > password.txt
> hw12203:/Users/alopresto/Workspace/scratch/encrypted-passwords (master) 
> alopresto
>  0s @ 16:26:47 $ ossl aes-256-cbc -e -nosalt -p -K 
> 0123456789ABCDEFFEDCBA98765432100123456789ABCDEFFEDCBA9876543210 -iv 
> 0123456789ABCDEFFEDCBA9876543210 -a -in password.txt -out 

[jira] [Assigned] (NIFI-2321) Update Getting Started documentation for 1.0 changes

[Joseph Witt (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-2321?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joseph Witt reassigned NIFI-2321:
-

Assignee: Joseph Witt  (was: Andrew Lim)

> Update Getting Started documentation for 1.0 changes
> 
>
> Key: NIFI-2321
> URL: https://issues.apache.org/jira/browse/NIFI-2321
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Documentation & Website
>Affects Versions: 1.0.0
>Reporter: Andrew Lim
>Assignee: Joseph Witt
> Fix For: 1.0.0
>
>
> Getting Started guide needs to be updated for the changes in the 1.0 UI.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-1148) Add processor to GetEmail supporting IMAP and POP3

[Joseph Witt (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-1148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15398548#comment-15398548
 ] 

Joseph Witt commented on NIFI-1148:
---

[~ozhurakousky] [~trixpan] looks like the discussion has stalled a bit.  Is 
there consensus on a path forward?  Perhaps the current approach is good to 
start with and then can be used to gather more information on what should 
happen next.

> Add processor to GetEmail supporting IMAP and POP3
> --
>
> Key: NIFI-1148
> URL: https://issues.apache.org/jira/browse/NIFI-1148
> Project: Apache NiFi
>  Issue Type: New Feature
>  Components: Extensions
>Reporter: Joseph Witt
>Assignee: Oleg Zhurakousky
> Fix For: 1.0.0
>
>
> It is fairly common that users want to be able to acquire data via email.  
> This means both IMAP and POP3.  POP3 is easier as it is a sort of fire/forget 
> model whereas IMAP involves more state handling.  But in any event both modes 
> are important to support.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (NIFI-2386) HTTP Site-to-Site fails if port is not specified explicitly

[Joseph Witt (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-2386?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joseph Witt reassigned NIFI-2386:
-

Assignee: Joseph Witt  (was: Koji Kawamura)

> HTTP Site-to-Site fails if port is not specified explicitly
> ---
>
> Key: NIFI-2386
> URL: https://issues.apache.org/jira/browse/NIFI-2386
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Affects Versions: 1.0.0
>Reporter: Koji Kawamura
>Assignee: Joseph Witt
> Fix For: 1.0.0
>
>
> It fails if a specified URL doesn't have a port number explicitly. It should 
> use 80 for http and 443 for https if port is not defined in a given URL.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-2366) After creating a template, it errors out when trying to add it back

[Joseph Witt (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2366?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15398544#comment-15398544
 ] 

Joseph Witt commented on NIFI-2366:
---

[~ozhurakousky] is this PR still for review only?  Does that mean don't merge?

> After creating a template, it errors out when trying to add it back
> ---
>
> Key: NIFI-2366
> URL: https://issues.apache.org/jira/browse/NIFI-2366
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Affects Versions: 1.0.0
>Reporter: Haimo Liu
>Assignee: Oleg Zhurakousky
>Priority: Blocker
> Fix For: 1.0.0
>
> Attachments: minifi_test.xml
>
>
> NIFI build on Jul.22nd. 
> Template: GenerateFlowFile -> RPG with a input port
> after the template is created, cannot add it back to the same canvas, due to 
> the following error: "The flow snippet contains a Connection that references 
> a component that is not included."
> after further investigation, realized that the component ID of the input port 
> referenced in the template is not accurate.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] nifi issue #688: NIFI-2304: If node address is not set, default to localhost...

[ijokarumawak]

Github user ijokarumawak commented on the issue:

https://github.com/apache/nifi/pull/688
  
@markap14 Thanks for the updates!
I tested the latest PR, set `nifi.cluster.node.address` as blank in 
nifi.properties, confirmed that e node NiFi cluster can start up successfully.

However, for `nifi.cluster.node.protocol.port`, there's another usage of 
that configuration, and I got following error message with blank property value:

```
Caused by: 
org.springframework.beans.factory.UnsatisfiedDependencyException: Error 
creating bean with name 'protocolListener' defined in class path resource 
[nifi-cluster-protocol-context.xml]: Unsatisfied dependency expressed through 
constructor argument with index 1 of type [int]: Could not convert constructor 
argument value of type [null] to required type [int]: Failed to convert value 
of type [null] to required type [int]; nested exception is 
java.lang.IllegalArgumentException: Cannot convert value of type [null] to 
required type [int]: PropertyEditor 
[org.springframework.beans.propertyeditors.CustomNumberEditor] returned 
inappropriate value of type [null]
at 
org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:724)
 ~[spring-beans-4.2.4.RELEASE.jar:4.2.4.RELEASE]
at 
org.springframework.beans.factory.support.ConstructorResolver.autowireConstructor(ConstructorResolver.java:185)
 ~[spring-beans-4.2.4.RELEASE.jar:4.2.4.RELEASE]
at 
org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.autowireConstructor(AbstractAutowireCapableBeanFactory.java:1143)
 ~[spring-beans-4.2.4.RELEASE.jar:4.2.4.RELEASE]
at 
org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1046)
 ~[spring-beans-4.2.4.RELEASE.jar:4.2.4.RELEASE]
at 
org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:510)
 ~[spring-beans-4.2.4.RELEASE.jar:4.2.4.RELEASE]
at 
org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:482)
 ~[spring-beans-4.2.4.RELEASE.jar:4.2.4.RELEASE]
at 
org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:306)
 ~[spring-beans-4.2.4.RELEASE.jar:4.2.4.RELEASE]
at 
org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)
 ~[spring-beans-4.2.4.RELEASE.jar:4.2.4.RELEASE]
at 
org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:302)
 ~[spring-beans-4.2.4.RELEASE.jar:4.2.4.RELEASE]
at 
org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:197)
 ~[spring-beans-4.2.4.RELEASE.jar:4.2.4.RELEASE]
at 
org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference(BeanDefinitionValueResolver.java:351)
 ~[spring-beans-4.2.4.RELEASE.jar:4.2.4.RELEASE]
... 52 common frames omitted
```


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (NIFI-1373) Annotate all classes to indicate their API stability and intended audience

[Joseph Witt (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-1373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15398541#comment-15398541
 ] 

Joseph Witt commented on NIFI-1373:
---

The apache yetus community is making progress on this 
https://issues.apache.org/jira/browse/YETUS-406

For now might end up deferring this ticket to 1.1 and instead more clearly 
articulating that the only public api is 'nifi-api' and our REST API and 
anything else is subject to change.

> Annotate all classes to indicate their API stability and intended audience
> --
>
> Key: NIFI-1373
> URL: https://issues.apache.org/jira/browse/NIFI-1373
> Project: Apache NiFi
>  Issue Type: Task
>  Components: Core Framework, Core UI, Extensions
>Reporter: Joseph Witt
>Assignee: Joseph Witt
> Fix For: 1.0.0
>
>
> Sean Busbey and Tony Kurc have pointed out the need to document the intended 
> audience and stability of the APIs.  This is very necessary as we need to be 
> able to make changes and be fluid where intended and extremely strict to 
> compatibility where intended.  Presently with things being unmarked it can be 
> ambiguous for those not deeply familiar with the api and codebase.
> By leveraging the annotations from Apache Yetus we can resolve this 
> discrepancy across the codebase:  
> http://yetus.apache.org/documentation/0.1.0/audience-annotations-apidocs/
> To Do:
> 1) Go through entire codebase and tag classes appropriate with Yetus 
> annotations.
> 2) Update our versioning guidance to reflect that our major/minor/incremental 
> model only applies to those classes which are tagged as stable/etc..  We 
> should be able to safely maneuver and update internal APIs without any risks 
> to compatibility.  Through discussions things should become more stable and 
> if we find folks wanting to extend/alter things which we didn't have listed 
> as stable then those are great things to start discussion on.  Right now 
> we're too hamstrung by thinking we have to hold onto everything.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-2337) IllegalMonitorStateException when disconnecting node

[Joseph Witt (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2337?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15398535#comment-15398535
 ] 

Joseph Witt commented on NIFI-2337:
---

+1.  Will merge to master in a bit.  Very cool that you also patched the issue 
for curator and that they already accepted and merged it.  We can remove this 
line then later when we pull in that update.  Slick.

> IllegalMonitorStateException when disconnecting node
> 
>
> Key: NIFI-2337
> URL: https://issues.apache.org/jira/browse/NIFI-2337
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Affects Versions: 1.0.0
>Reporter: Mark Payne
>Assignee: Joseph Witt
> Fix For: 1.0.0
>
>
> When a node is disconnected from the cluster, it often logs a stacktrace 
> similar to:
> 2016-07-20 20:20:47,814 ERROR [Leader Election Notification Thread-1] 
> o.a.c.f.recipes.leader.LeaderSelector The leader threw an exception
> java.lang.IllegalMonitorStateException: You do not own the lock: 
> /leaders/Cluster Coordinator
> at 
> org.apache.curator.framework.recipes.locks.InterProcessMutex.release(InterProcessMutex.java:140)
>  ~[curator-recipes-2.10.0.jar:na]
> at 
> org.apache.curator.framework.recipes.leader.LeaderSelector.doWork(LeaderSelector.java:425)
>  [curator-recipes-2.10.0.jar:na]
> at 
> org.apache.curator.framework.recipes.leader.LeaderSelector.doWorkLoop(LeaderSelector.java:441)
>  [curator-recipes-2.10.0.jar:na]
> at 
> org.apache.curator.framework.recipes.leader.LeaderSelector.access$100(LeaderSelector.java:64)
>  [curator-recipes-2.10.0.jar:na]
> at 
> org.apache.curator.framework.recipes.leader.LeaderSelector$2.call(LeaderSelector.java:245)
>  [curator-recipes-2.10.0.jar:na]
> at 
> org.apache.curator.framework.recipes.leader.LeaderSelector$2.call(LeaderSelector.java:239)
>  [curator-recipes-2.10.0.jar:na]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266) 
> [na:1.8.0_74]
> at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) 
> [na:1.8.0_74]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266) 
> [na:1.8.0_74]
> at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) 
> [na:1.8.0_74]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266) 
> [na:1.8.0_74]
> at 
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
>  [na:1.8.0_74]
> at 
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
>  [na:1.8.0_74]
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>  [na:1.8.0_74]
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>  [na:1.8.0_74]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_74]
> This error, I believe, is harmless but is alarming and should be addressed.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-1969) Update Documentation for new Authorization Properties

[Joseph Witt (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-1969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15398532#comment-15398532
 ] 

Joseph Witt commented on NIFI-1969:
---

[~andrewmlim] how does this JIRA differ from NIFI-2420?  Could we close this as 
a duplicate or superseded by that?

> Update Documentation for new Authorization Properties
> -
>
> Key: NIFI-1969
> URL: https://issues.apache.org/jira/browse/NIFI-1969
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Documentation & Website
>Affects Versions: 1.0.0
>Reporter: Mark Payne
>Assignee: Andrew Lim
>Priority: Critical
>  Labels: migration
> Fix For: 1.0.0
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (NIFI-1214) Mock Framework should allow order-independent assumptions on FlowFiles

[Joseph Witt (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-1214?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joseph Witt updated NIFI-1214:
--
Fix Version/s: (was: 1.0.0)

> Mock Framework should allow order-independent assumptions on FlowFiles
> --
>
> Key: NIFI-1214
> URL: https://issues.apache.org/jira/browse/NIFI-1214
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Tools and Build
>Reporter: Mark Payne
>Assignee: Toivo Adams
>
> A common pattern in unit testing is to iterate over all FlowFiles that are 
> output to a Relationship and verify that each FlowFile matches one criteria 
> or another and that all criteria are met. For example, the following code 
> snippet from TestRouteText  verifies that two FlowFiles were output and that 
> Criteria A was met by one of them and Criteria B was met by the other:
> {code}
> final List list = 
> runner.getFlowFilesForRelationship("o");
> boolean found1 = false;
> boolean found2 = false;
> for (final MockFlowFile mff : list) {
> if (mff.getAttribute(RouteText.GROUP_ATTRIBUTE_KEY).equals("1")) {
> mff.assertContentEquals("1,hello\n1,good-bye");
> found1 = true;
> } else {
> mff.assertAttributeEquals(RouteText.GROUP_ATTRIBUTE_KEY, "2");
> mff.assertContentEquals("2,world\n");
> found2 = true;
> }
> }
> assertTrue(found1);
> assertTrue(found2);
> {code}
> This is very verbose, and error-prone. It could be done much more concisely 
> if we have a method like:
> {code}
> TestRunner.assertAllConditionsMet( Relationship relationship, 
> FlowFileVerifier... verifier );
> {code}
> Where FlowFileVerifier is able to verify some condition on a FlowFile. This 
> method would then be responsible for ensuring that each FlowFile that was 
> routed to 'relationship' meets one of the criteria specified by a verifier, 
> and that all of the verifiers were met. For example:
> {code}
> runner.assertAllConditionsMet( "o", 
> { mff -> mff.isAttributeEqual(RouteText.GROUP_ATTRIBUTE_KEY, "1") && 
> mff.isContentEqual("1,hello\n1,good-bye") },
> { mff -> mff.isAttributeEqual(RouteText.GROUP_ATTRIBUTE_KEY, "2") && 
> mff.isContentEqual("2,world\n") }
> );
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-2391) Add toString() implementation to ComponentDTO

[Joseph Witt (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15398528#comment-15398528
 ] 

Joseph Witt commented on NIFI-2391:
---

[~ozhurakousky]can you please drop the fix version for this until there is a PR 
and review progress?

> Add toString() implementation to ComponentDTO
> -
>
> Key: NIFI-2391
> URL: https://issues.apache.org/jira/browse/NIFI-2391
> Project: Apache NiFi
>  Issue Type: Sub-task
>  Components: Core Framework
>Reporter: Oleg Zhurakousky
>Assignee: Oleg Zhurakousky
> Fix For: 1.0.0
>
>
> While working on deterministic templates and some of the follow up issues 
> after it makes it increasingly difficult to debug without more informative 
> toString implementation. 
> For now I'll go with "[Simple Class Name]:[Component ID] scheme. For example:
> {code}
> ProcessorDTO:59595eb1-7296-4db1-bb06-012705f5a698
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (NIFI-2395) PersistentProvenanceRepository Deadlocks caused by a blocked journal merge

[Joseph Witt (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-2395?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joseph Witt reassigned NIFI-2395:
-

Assignee: Joseph Witt  (was: Mark Payne)

> PersistentProvenanceRepository Deadlocks caused by a blocked journal merge
> --
>
> Key: NIFI-2395
> URL: https://issues.apache.org/jira/browse/NIFI-2395
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Affects Versions: 0.6.0, 0.7.0
>Reporter: Brian Davis
>Assignee: Joseph Witt
>Priority: Blocker
> Fix For: 1.0.0, 0.8.0
>
>
> I have a nifi instance that I have been running for about a week and has 
> deadlocked at least 3 times during this time.  When I say deadlock the whole 
> nifi instance stops doing any progress on flowfiles.  I looked at the stack 
> trace and there are a lot of threads stuck doing tasks in the 
> PersistentProvenanceRepository.  Looking at the code I think this is what is 
> happening:
> There is a ReadWriteLock that all the reads are waiting for a write.  The 
> write is in the loop:
> {code}
> while (journalFileCount > journalCountThreshold || repoSize > 
> sizeThreshold) {
> // if a shutdown happens while we are in this loop, kill 
> the rollover thread and break
> if (this.closed.get()) {
> if (future != null) {
> future.cancel(true);
> }
> break;
> }
> if (repoSize > sizeThreshold) {
> logger.debug("Provenance Repository has exceeded its 
> size threshold; will trigger purging of oldest events");
> purgeOldEvents();
> journalFileCount = getJournalCount();
> repoSize = getSize(getLogFiles(), 0L);
> continue;
> } else {
> // if we are constrained by the number of journal 
> files rather than the size of the repo,
> // then we will just sleep a bit because another 
> thread is already actively merging the journals,
> // due to the runnable that we scheduled above
> try {
> Thread.sleep(100L);
> } catch (final InterruptedException ie) {
> }
> }
> logger.debug("Provenance Repository is still behind. 
> Keeping flow slowed down "
> + "to accommodate. Currently, there are {} 
> journal files ({} bytes) and "
> + "threshold for blocking is {} ({} bytes)", 
> journalFileCount, repoSize, journalCountThreshold, sizeThreshold);
> journalFileCount = getJournalCount();
> repoSize = getSize(getLogFiles(), 0L);
> }
> logger.info("Provenance Repository has now caught up with 
> rolling over journal files. Current number of "
> + "journal files to be rolled over is {}", 
> journalFileCount);
> }
> {code}
> My nifi is at the sleep indefinitely.  The reason my nifi cannot move forward 
> is because of the thread doing the merge is stopped.  The thread doing the 
> merge is at:
> {code}
> accepted = eventQueue.offer(new Tuple<>(record, blockIndex), 10, 
> TimeUnit.MILLISECONDS);
> {code}
> so the queue is full.  
> What I believe happened is that the callables created here:
> {code}
> final Callable callable = new 
> Callable() {
> @Override
> public Object call() throws IOException {
> while (!eventQueue.isEmpty() || 
> !finishedAdding.get()) {
> final 
> Tuple tuple;
> try {
> tuple = eventQueue.poll(10, 
> TimeUnit.MILLISECONDS);
> } catch (final InterruptedException 
> ie) {
> continue;
> }
> if (tuple == null) {
> continue;
> }
> indexingAction.index(tuple.getKey(), 
> indexWriter, tuple.getValue());
> }
> return null;
> }
> {code}
> finish before the offer adds its first event 

[jira] [Commented] (NIFI-2371) Cluster Page padding

[Joseph Witt (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2371?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15398527#comment-15398527
 ] 

Joseph Witt commented on NIFI-2371:
---

reviewing

> Cluster Page padding 
> -
>
> Key: NIFI-2371
> URL: https://issues.apache.org/jira/browse/NIFI-2371
> Project: Apache NiFi
>  Issue Type: Sub-task
>  Components: Core UI
>Reporter: Scott Aslan
>Assignee: Joseph Witt
> Fix For: 1.0.0
>
>
> open the cluster shell and click to pop it out...looks like there is no 
> padding



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (NIFI-2371) Cluster Page padding

[Joseph Witt (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-2371?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joseph Witt reassigned NIFI-2371:
-

Assignee: Joseph Witt  (was: Scott Aslan)

> Cluster Page padding 
> -
>
> Key: NIFI-2371
> URL: https://issues.apache.org/jira/browse/NIFI-2371
> Project: Apache NiFi
>  Issue Type: Sub-task
>  Components: Core UI
>Reporter: Scott Aslan
>Assignee: Joseph Witt
> Fix For: 1.0.0
>
>
> open the cluster shell and click to pop it out...looks like there is no 
> padding



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (NIFI-2428) Hitting escape from stats pane causes the pane to go away but the canvas is still glossed over

[Joseph Percivall (JIRA)]

Joseph Percivall created NIFI-2428:
--

 Summary: Hitting escape from stats pane causes the pane to go away 
but the canvas is still glossed over
 Key: NIFI-2428
 URL: https://issues.apache.org/jira/browse/NIFI-2428
 Project: Apache NiFi
  Issue Type: Sub-task
Reporter: Joseph Percivall
Priority: Minor


Component on the graph
go to the stats of the component
hit escape
observe the window go away but the background is still glossed over and 
unclickable

have to refresh window to regain usability



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-2420) Add a section for Multi-tenant Authorization to the Admin Guide

[Joseph Witt (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15398522#comment-15398522
 ] 

Joseph Witt commented on NIFI-2420:
---

reviewing but initial pass suggests this is an awesome effort!

> Add a section for Multi-tenant Authorization to the Admin Guide
> ---
>
> Key: NIFI-2420
> URL: https://issues.apache.org/jira/browse/NIFI-2420
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Documentation & Website
>Affects Versions: 1.0.0
>Reporter: Andrew Lim
>Assignee: Joseph Witt
> Fix For: 1.0.0
>
> Attachments: Multi-tenantAuthManage.pdf, Multi-tenantAuthSetup.pdf
>
>
> Multi-tenant Authorization is a new feature in 1.0.0. A section covering 
> setup and management should be included in the documentation.
> Attaching my first attempt at the doc in case anyone has any thoughts.
> I plan to submit a PR soon depending on feedback and after more 
> editing/review on my part.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (NIFI-2420) Add a section for Multi-tenant Authorization to the Admin Guide

[Joseph Witt (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-2420?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joseph Witt reassigned NIFI-2420:
-

Assignee: Joseph Witt

> Add a section for Multi-tenant Authorization to the Admin Guide
> ---
>
> Key: NIFI-2420
> URL: https://issues.apache.org/jira/browse/NIFI-2420
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Documentation & Website
>Affects Versions: 1.0.0
>Reporter: Andrew Lim
>Assignee: Joseph Witt
> Fix For: 1.0.0
>
> Attachments: Multi-tenantAuthManage.pdf, Multi-tenantAuthSetup.pdf
>
>
> Multi-tenant Authorization is a new feature in 1.0.0. A section covering 
> setup and management should be included in the documentation.
> Attaching my first attempt at the doc in case anyone has any thoughts.
> I plan to submit a PR soon depending on feedback and after more 
> editing/review on my part.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (NIFI-2390) Decouple users/user groups and policies

[Joseph Witt (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-2390?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joseph Witt updated NIFI-2390:
--
Resolution: Fixed
Status: Resolved  (was: Patch Available)

[~mcgilman] and [~bbende] closing as it appears merged to master.  If not 
accurate please advise.

> Decouple users/user groups and policies
> ---
>
> Key: NIFI-2390
> URL: https://issues.apache.org/jira/browse/NIFI-2390
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Extensions
>Reporter: Matt Gilman
>Assignee: Bryan Bende
> Fix For: 1.0.0
>
>
> Currently, users, user groups, and policies are all persisted in the same 
> file. Because the policies are associated with a particular NiFi instance it 
> makes this file not portable. However, the users and user groups 
> configuration is completely independent of a particular instance and should 
> be persisted in a separate file to promote portable of this configuration 
> between NiFi instances.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] nifi pull request #694: NIFI-1876 Implements merging of responses to success...

[JPercivall]

Github user JPercivall commented on a diff in the pull request:

https://github.com/apache/nifi/pull/694#discussion_r72728226
  
--- Diff: 
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/main/java/org/apache/nifi/web/api/dto/EntityFactory.java
 ---
@@ -77,6 +77,8 @@ public ProcessorEntity createProcessorEntity(final 
ProcessorDTO dto, final Revis
 if (dto != null) {
 entity.setPermissions(permissions);
 entity.setStatus(status);
+status.setCanRead(permissions.getCanRead());
--- End diff --

I believe the potential for a NPE was introduce here. Permissions doesn't 
get checked for null until line 85.

This also applies to the other changes in this file.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (NIFI-2193) Command Line Keystore and Truststore utility

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15398458#comment-15398458
 ] 

ASF GitHub Bot commented on NIFI-2193:
--

Github user alopresto commented on a diff in the pull request:

https://github.com/apache/nifi/pull/695#discussion_r72723924
  
--- Diff: 
nifi-toolkit/nifi-toolkit-tls/src/main/java/org/apache/nifi/toolkit/tls/util/PasswordUtil.java
 ---
@@ -0,0 +1,34 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.toolkit.tls.util;
+
+import java.math.BigInteger;
+import java.security.SecureRandom;
+
+public class PasswordUtil {
+private final SecureRandom secureRandom;
+
+public PasswordUtil(SecureRandom secureRandom) {
--- End diff --

Is the `SecureRandom` initialized externally for a reason? Is this just to 
make testing easier?


> Command Line Keystore and Truststore utility
> 
>
> Key: NIFI-2193
> URL: https://issues.apache.org/jira/browse/NIFI-2193
> Project: Apache NiFi
>  Issue Type: New Feature
>Reporter: Bryan Rosander
>Assignee: Bryan Rosander
>
> In order to facilitate secure setup of NiFi, it would be useful to have a 
> command line utility capable of generating the required keystores, 
> truststore, and relevant configuration files.
> It should be able to generate keystores for each NiFi node, a truststore that 
> they all use, and relevant passwords and configuration files for using the 
> keystores and truststore.
> Additionally, in order to support distributed deployment, a web based 
> certificate authority with corresponding client will allow for each NiFi 
> instance to generate its own keypair and then request signing by the CA.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] nifi pull request #695: NIFI-2193 - Command line SSL config utility as well ...

[alopresto]

Github user alopresto commented on a diff in the pull request:

https://github.com/apache/nifi/pull/695#discussion_r72723924
  
--- Diff: 
nifi-toolkit/nifi-toolkit-tls/src/main/java/org/apache/nifi/toolkit/tls/util/PasswordUtil.java
 ---
@@ -0,0 +1,34 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.toolkit.tls.util;
+
+import java.math.BigInteger;
+import java.security.SecureRandom;
+
+public class PasswordUtil {
+private final SecureRandom secureRandom;
+
+public PasswordUtil(SecureRandom secureRandom) {
--- End diff --

Is the `SecureRandom` initialized externally for a reason? Is this just to 
make testing easier?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (NIFI-2193) Command Line Keystore and Truststore utility

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15398456#comment-15398456
 ] 

ASF GitHub Bot commented on NIFI-2193:
--

Github user alopresto commented on a diff in the pull request:

https://github.com/apache/nifi/pull/695#discussion_r72723879
  
--- Diff: 
nifi-toolkit/nifi-toolkit-tls/src/main/java/org/apache/nifi/toolkit/tls/util/PasswordUtil.java
 ---
@@ -0,0 +1,34 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.toolkit.tls.util;
+
+import java.math.BigInteger;
+import java.security.SecureRandom;
+
+public class PasswordUtil {
+private final SecureRandom secureRandom;
+
+public PasswordUtil(SecureRandom secureRandom) {
+this.secureRandom = secureRandom;
+}
+
+public String generatePassword() {
+// [see 
http://stackoverflow.com/questions/41107/how-to-generate-a-random-alpha-numeric-string#answer-41156]
+return new BigInteger(1024, secureRandom).toString(36);
--- End diff --

I appreciate the *strong* entropy here, but 1024 bits in base 36  
198 characters. Something that provides more than 128 bits is sufficient here; 
256 will be more than enough for a number of years. I might have commented 
off-hand about the base earlier, but base 32 makes sense because it uses a 
"round" number of bits  ( `2^5 = 32` ) to fully express its alphabet, while 
base 36 (`0-9a-f`) needs 6 ( `2^6 = 64` ) but wastes a number of those bits. 


> Command Line Keystore and Truststore utility
> 
>
> Key: NIFI-2193
> URL: https://issues.apache.org/jira/browse/NIFI-2193
> Project: Apache NiFi
>  Issue Type: New Feature
>Reporter: Bryan Rosander
>Assignee: Bryan Rosander
>
> In order to facilitate secure setup of NiFi, it would be useful to have a 
> command line utility capable of generating the required keystores, 
> truststore, and relevant configuration files.
> It should be able to generate keystores for each NiFi node, a truststore that 
> they all use, and relevant passwords and configuration files for using the 
> keystores and truststore.
> Additionally, in order to support distributed deployment, a web based 
> certificate authority with corresponding client will allow for each NiFi 
> instance to generate its own keypair and then request signing by the CA.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] nifi pull request #695: NIFI-2193 - Command line SSL config utility as well ...

[alopresto]

Github user alopresto commented on a diff in the pull request:

https://github.com/apache/nifi/pull/695#discussion_r72723879
  
--- Diff: 
nifi-toolkit/nifi-toolkit-tls/src/main/java/org/apache/nifi/toolkit/tls/util/PasswordUtil.java
 ---
@@ -0,0 +1,34 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.toolkit.tls.util;
+
+import java.math.BigInteger;
+import java.security.SecureRandom;
+
+public class PasswordUtil {
+private final SecureRandom secureRandom;
+
+public PasswordUtil(SecureRandom secureRandom) {
+this.secureRandom = secureRandom;
+}
+
+public String generatePassword() {
+// [see 
http://stackoverflow.com/questions/41107/how-to-generate-a-random-alpha-numeric-string#answer-41156]
+return new BigInteger(1024, secureRandom).toString(36);
--- End diff --

I appreciate the *strong* entropy here, but 1024 bits in base 36  
198 characters. Something that provides more than 128 bits is sufficient here; 
256 will be more than enough for a number of years. I might have commented 
off-hand about the base earlier, but base 32 makes sense because it uses a 
"round" number of bits  ( `2^5 = 32` ) to fully express its alphabet, while 
base 36 (`0-9a-f`) needs 6 ( `2^6 = 64` ) but wastes a number of those bits. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi issue #740: Message on history page when clustered

[mcgilman]

Github user mcgilman commented on the issue:

https://github.com/apache/nifi/pull/740
  
The message is only shown when the node is part of a cluster. I think we 
shouldn't overcomplicate things. Inform the user that they are viewing the 
history of the current node and suggest they can browse to another node to view 
it's history. I believe this is a clear, simple, and accurate description of 
what is being display and what other options the user has.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi issue #740: Message on history page when clustered

[mcgilman]

Github user mcgilman commented on the issue:

https://github.com/apache/nifi/pull/740
  
Do you have another suggestion for the message? What if we create another 
JIRA for further explanation in the user guide? Tough to explain the scenario 
in a short sentence or two.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (NIFI-2193) Command Line Keystore and Truststore utility

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15398344#comment-15398344
 ] 

ASF GitHub Bot commented on NIFI-2193:
--

Github user alopresto commented on a diff in the pull request:

https://github.com/apache/nifi/pull/695#discussion_r72715741
  
--- Diff: 
nifi-toolkit/nifi-toolkit-tls/src/main/java/org/apache/nifi/toolkit/tls/service/TlsCertificateSigningRequestPerformer.java
 ---
@@ -0,0 +1,144 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.toolkit.tls.service;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.commons.io.IOUtils;
+import org.apache.commons.io.input.BoundedInputStream;
+import org.apache.http.HttpHost;
+import org.apache.http.client.methods.CloseableHttpResponse;
+import org.apache.http.client.methods.HttpPost;
+import org.apache.http.conn.ssl.TrustSelfSignedStrategy;
+import org.apache.http.entity.ByteArrayEntity;
+import org.apache.http.impl.client.CloseableHttpClient;
+import org.apache.http.impl.client.HttpClientBuilder;
+import org.apache.http.ssl.SSLContextBuilder;
+import org.apache.nifi.toolkit.tls.configuration.TlsClientConfig;
+import org.apache.nifi.toolkit.tls.util.TlsHelper;
+import org.bouncycastle.pkcs.jcajce.JcaPKCS10CertificationRequest;
+import org.eclipse.jetty.server.Response;
+
+import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+import java.security.KeyPair;
+import java.security.NoSuchAlgorithmException;
+import java.security.cert.X509Certificate;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.function.Supplier;
+
+public class TlsCertificateSigningRequestPerformer {
+public static final String RECEIVED_RESPONSE_CODE = "Received response 
code ";
+public static final String EXPECTED_ONE_CERTIFICATE = "Expected one 
certificate";
+public static final String EXPECTED_RESPONSE_TO_CONTAIN_HMAC = 
"Expected response to contain hmac";
+public static final String 
UNEXPECTED_HMAC_RECEIVED_POSSIBLE_MAN_IN_THE_MIDDLE = "Unexpected hmac 
received, possible man in the middle";
+public static final String EXPECTED_RESPONSE_TO_CONTAIN_CERTIFICATE = 
"Expected response to contain certificate";
+private final Supplier httpClientBuilderSupplier;
+private final String caHostname;
+private final String dn;
+private final String token;
+private final int port;
+private final TlsHelper tlsHelper;
+
+public TlsCertificateSigningRequestPerformer(TlsClientConfig 
tlsClientConfig) throws NoSuchAlgorithmException {
+this(HttpClientBuilder::create, tlsClientConfig.getCaHostname(), 
tlsClientConfig.getDn(), tlsClientConfig.getToken(), tlsClientConfig.getPort(), 
tlsClientConfig.createTlsHelper());
+}
+
+public 
TlsCertificateSigningRequestPerformer(Supplier 
httpClientBuilderSupplier, TlsClientConfig tlsClientConfig) throws 
NoSuchAlgorithmException {
+this(httpClientBuilderSupplier, tlsClientConfig.getCaHostname(), 
tlsClientConfig.getDn(), tlsClientConfig.getToken(), tlsClientConfig.getPort(), 
tlsClientConfig.createTlsHelper());
+}
+
+public 
TlsCertificateSigningRequestPerformer(Supplier 
httpClientBuilderSupplier, String caHostname, String dn, String token, int 
port, TlsHelper tlsHelper) {
+this.httpClientBuilderSupplier = httpClientBuilderSupplier;
+this.caHostname = caHostname;
+this.dn = dn;
+this.token = token;
+this.port = port;
+this.tlsHelper = tlsHelper;
+}
+
+public static String getDn(String hostname) {
+return "CN=" + hostname + ",OU=NIFI";
+}
+
+/**
+ * Submits a CSR to the Certificate authority, checks the resulting 
hmac, and returns the chain if everything succeeds
+ *
+ * @param objectMapper for 

[GitHub] nifi pull request #695: NIFI-2193 - Command line SSL config utility as well ...

[alopresto]

Github user alopresto commented on a diff in the pull request:

https://github.com/apache/nifi/pull/695#discussion_r72715741
  
--- Diff: 
nifi-toolkit/nifi-toolkit-tls/src/main/java/org/apache/nifi/toolkit/tls/service/TlsCertificateSigningRequestPerformer.java
 ---
@@ -0,0 +1,144 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.toolkit.tls.service;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.commons.io.IOUtils;
+import org.apache.commons.io.input.BoundedInputStream;
+import org.apache.http.HttpHost;
+import org.apache.http.client.methods.CloseableHttpResponse;
+import org.apache.http.client.methods.HttpPost;
+import org.apache.http.conn.ssl.TrustSelfSignedStrategy;
+import org.apache.http.entity.ByteArrayEntity;
+import org.apache.http.impl.client.CloseableHttpClient;
+import org.apache.http.impl.client.HttpClientBuilder;
+import org.apache.http.ssl.SSLContextBuilder;
+import org.apache.nifi.toolkit.tls.configuration.TlsClientConfig;
+import org.apache.nifi.toolkit.tls.util.TlsHelper;
+import org.bouncycastle.pkcs.jcajce.JcaPKCS10CertificationRequest;
+import org.eclipse.jetty.server.Response;
+
+import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+import java.security.KeyPair;
+import java.security.NoSuchAlgorithmException;
+import java.security.cert.X509Certificate;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.function.Supplier;
+
+public class TlsCertificateSigningRequestPerformer {
+public static final String RECEIVED_RESPONSE_CODE = "Received response 
code ";
+public static final String EXPECTED_ONE_CERTIFICATE = "Expected one 
certificate";
+public static final String EXPECTED_RESPONSE_TO_CONTAIN_HMAC = 
"Expected response to contain hmac";
+public static final String 
UNEXPECTED_HMAC_RECEIVED_POSSIBLE_MAN_IN_THE_MIDDLE = "Unexpected hmac 
received, possible man in the middle";
+public static final String EXPECTED_RESPONSE_TO_CONTAIN_CERTIFICATE = 
"Expected response to contain certificate";
+private final Supplier httpClientBuilderSupplier;
+private final String caHostname;
+private final String dn;
+private final String token;
+private final int port;
+private final TlsHelper tlsHelper;
+
+public TlsCertificateSigningRequestPerformer(TlsClientConfig 
tlsClientConfig) throws NoSuchAlgorithmException {
+this(HttpClientBuilder::create, tlsClientConfig.getCaHostname(), 
tlsClientConfig.getDn(), tlsClientConfig.getToken(), tlsClientConfig.getPort(), 
tlsClientConfig.createTlsHelper());
+}
+
+public 
TlsCertificateSigningRequestPerformer(Supplier 
httpClientBuilderSupplier, TlsClientConfig tlsClientConfig) throws 
NoSuchAlgorithmException {
+this(httpClientBuilderSupplier, tlsClientConfig.getCaHostname(), 
tlsClientConfig.getDn(), tlsClientConfig.getToken(), tlsClientConfig.getPort(), 
tlsClientConfig.createTlsHelper());
+}
+
+public 
TlsCertificateSigningRequestPerformer(Supplier 
httpClientBuilderSupplier, String caHostname, String dn, String token, int 
port, TlsHelper tlsHelper) {
+this.httpClientBuilderSupplier = httpClientBuilderSupplier;
+this.caHostname = caHostname;
+this.dn = dn;
+this.token = token;
+this.port = port;
+this.tlsHelper = tlsHelper;
+}
+
+public static String getDn(String hostname) {
+return "CN=" + hostname + ",OU=NIFI";
+}
+
+/**
+ * Submits a CSR to the Certificate authority, checks the resulting 
hmac, and returns the chain if everything succeeds
+ *
+ * @param objectMapper for serialization
+ * @param keyPair  the keypair to generate the csr for
+ * @throws IOException if there is a problem during the process
+ * @returnd the resulting certificate chain
+ */
+public 

[GitHub] nifi issue #740: Message on history page when clustered

[JPercivall]

Github user JPercivall commented on the issue:

https://github.com/apache/nifi/pull/740
  
Yup, that was my confusion and I think the target user of this message 
(people not really familiar with the in's/out's of clustering) will get hung up 
on that as well. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (NIFI-1876) Clustering - Merge all responses based on authorization

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-1876?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15398334#comment-15398334
 ] 

ASF GitHub Bot commented on NIFI-1876:
--

Github user jtstorck commented on the issue:

https://github.com/apache/nifi/pull/694
  
Rebased on master and resolved conflicts.  Ready for review again.


> Clustering - Merge all responses based on authorization
> ---
>
> Key: NIFI-1876
> URL: https://issues.apache.org/jira/browse/NIFI-1876
> Project: Apache NiFi
>  Issue Type: Sub-task
>  Components: Core Framework
>Reporter: Matt Gilman
>Assignee: Jeff Storck
> Fix For: 1.0.0
>
>
> Each node in a cluster may have a different view of the authorization access 
> policies simply to in the timing of updates. Because of this, all requests 
> need to be merged accordingly.
> Requests are directed at a specific resource. These would result in some 403 
> responses.
> Some requests are contain a filtered view of a number of resources. These 
> would need to be updated to return the most restrictive set of responses.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-2193) Command Line Keystore and Truststore utility

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15398333#comment-15398333
 ] 

ASF GitHub Bot commented on NIFI-2193:
--

Github user alopresto commented on a diff in the pull request:

https://github.com/apache/nifi/pull/695#discussion_r72715250
  
--- Diff: 
nifi-toolkit/nifi-toolkit-tls/src/main/java/org/apache/nifi/toolkit/tls/util/TlsHelper.java
 ---
@@ -0,0 +1,177 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.toolkit.tls.util;
+
+import org.apache.nifi.security.util.CertificateUtils;
+import org.apache.nifi.toolkit.tls.commandLine.TlsToolkitCommandLine;
+import org.apache.nifi.toolkit.tls.configuration.TlsHelperConfig;
+import org.bouncycastle.cert.X509CertificateHolder;
+import org.bouncycastle.cert.crmf.CRMFException;
+import org.bouncycastle.cert.jcajce.JcaX509CertificateConverter;
+import org.bouncycastle.cert.jcajce.JcaX509ExtensionUtils;
+import org.bouncycastle.eac.EACException;
+import org.bouncycastle.jce.provider.BouncyCastleProvider;
+import org.bouncycastle.openssl.PEMParser;
+import org.bouncycastle.openssl.jcajce.JcaMiscPEMGenerator;
+import org.bouncycastle.operator.OperatorCreationException;
+import org.bouncycastle.operator.jcajce.JcaContentSignerBuilder;
+import org.bouncycastle.pkcs.PKCS10CertificationRequest;
+import org.bouncycastle.pkcs.jcajce.JcaPKCS10CertificationRequest;
+import org.bouncycastle.pkcs.jcajce.JcaPKCS10CertificationRequestBuilder;
+import org.bouncycastle.util.io.pem.PemWriter;
+
+import javax.crypto.Mac;
+import javax.crypto.spec.SecretKeySpec;
+import javax.security.auth.x500.X500Principal;
+import java.io.IOException;
+import java.io.StringReader;
+import java.io.StringWriter;
+import java.nio.charset.StandardCharsets;
+import java.security.GeneralSecurityException;
+import java.security.InvalidKeyException;
+import java.security.KeyPair;
+import java.security.KeyPairGenerator;
+import java.security.KeyStore;
+import java.security.KeyStoreException;
+import java.security.MessageDigest;
+import java.security.NoSuchAlgorithmException;
+import java.security.NoSuchProviderException;
+import java.security.PublicKey;
+import java.security.Security;
+import java.security.SignatureException;
+import java.security.cert.Certificate;
+import java.security.cert.CertificateException;
+import java.security.cert.X509Certificate;
+import java.security.spec.InvalidKeySpecException;
+
+public class TlsHelper {
+public static final String PROVIDER = 
BouncyCastleProvider.PROVIDER_NAME;
+public static final String PKCS12 = "PKCS12";
+private final KeyPairGenerator keyPairGenerator;
+private final int days;
+private final String signingAlgorithm;
+
+public TlsHelper(TlsHelperConfig tlsHelperConfig) throws 
NoSuchAlgorithmException {
+this(tlsHelperConfig.getDays(), tlsHelperConfig.getKeySize(), 
tlsHelperConfig.getKeyPairAlgorithm(), tlsHelperConfig.getSigningAlgorithm());
+}
+
+public TlsHelper(TlsToolkitCommandLine tlsToolkitCommandLine) throws 
NoSuchAlgorithmException {
+this(tlsToolkitCommandLine.getTlsHelperConfig());
+}
+
+public TlsHelper(int days, int keySize, String keyPairAlgorithm, 
String signingAlgorithm) throws NoSuchAlgorithmException {
+this(createKeyPairGenerator(keyPairAlgorithm, keySize), days, 
signingAlgorithm);
+}
+
+protected TlsHelper(KeyPairGenerator keyPairGenerator, int days, 
String signingAlgorithm) {
+this.keyPairGenerator = keyPairGenerator;
+this.days = days;
+this.signingAlgorithm = signingAlgorithm;
+}
+
+public static void addBouncyCastleProvider() {
+Security.addProvider(new BouncyCastleProvider());
+}
+
+private static KeyPairGenerator 

[GitHub] nifi pull request #695: NIFI-2193 - Command line SSL config utility as well ...

[alopresto]

Github user alopresto commented on a diff in the pull request:

https://github.com/apache/nifi/pull/695#discussion_r72715250
  
--- Diff: 
nifi-toolkit/nifi-toolkit-tls/src/main/java/org/apache/nifi/toolkit/tls/util/TlsHelper.java
 ---
@@ -0,0 +1,177 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.toolkit.tls.util;
+
+import org.apache.nifi.security.util.CertificateUtils;
+import org.apache.nifi.toolkit.tls.commandLine.TlsToolkitCommandLine;
+import org.apache.nifi.toolkit.tls.configuration.TlsHelperConfig;
+import org.bouncycastle.cert.X509CertificateHolder;
+import org.bouncycastle.cert.crmf.CRMFException;
+import org.bouncycastle.cert.jcajce.JcaX509CertificateConverter;
+import org.bouncycastle.cert.jcajce.JcaX509ExtensionUtils;
+import org.bouncycastle.eac.EACException;
+import org.bouncycastle.jce.provider.BouncyCastleProvider;
+import org.bouncycastle.openssl.PEMParser;
+import org.bouncycastle.openssl.jcajce.JcaMiscPEMGenerator;
+import org.bouncycastle.operator.OperatorCreationException;
+import org.bouncycastle.operator.jcajce.JcaContentSignerBuilder;
+import org.bouncycastle.pkcs.PKCS10CertificationRequest;
+import org.bouncycastle.pkcs.jcajce.JcaPKCS10CertificationRequest;
+import org.bouncycastle.pkcs.jcajce.JcaPKCS10CertificationRequestBuilder;
+import org.bouncycastle.util.io.pem.PemWriter;
+
+import javax.crypto.Mac;
+import javax.crypto.spec.SecretKeySpec;
+import javax.security.auth.x500.X500Principal;
+import java.io.IOException;
+import java.io.StringReader;
+import java.io.StringWriter;
+import java.nio.charset.StandardCharsets;
+import java.security.GeneralSecurityException;
+import java.security.InvalidKeyException;
+import java.security.KeyPair;
+import java.security.KeyPairGenerator;
+import java.security.KeyStore;
+import java.security.KeyStoreException;
+import java.security.MessageDigest;
+import java.security.NoSuchAlgorithmException;
+import java.security.NoSuchProviderException;
+import java.security.PublicKey;
+import java.security.Security;
+import java.security.SignatureException;
+import java.security.cert.Certificate;
+import java.security.cert.CertificateException;
+import java.security.cert.X509Certificate;
+import java.security.spec.InvalidKeySpecException;
+
+public class TlsHelper {
+public static final String PROVIDER = 
BouncyCastleProvider.PROVIDER_NAME;
+public static final String PKCS12 = "PKCS12";
+private final KeyPairGenerator keyPairGenerator;
+private final int days;
+private final String signingAlgorithm;
+
+public TlsHelper(TlsHelperConfig tlsHelperConfig) throws 
NoSuchAlgorithmException {
+this(tlsHelperConfig.getDays(), tlsHelperConfig.getKeySize(), 
tlsHelperConfig.getKeyPairAlgorithm(), tlsHelperConfig.getSigningAlgorithm());
+}
+
+public TlsHelper(TlsToolkitCommandLine tlsToolkitCommandLine) throws 
NoSuchAlgorithmException {
+this(tlsToolkitCommandLine.getTlsHelperConfig());
+}
+
+public TlsHelper(int days, int keySize, String keyPairAlgorithm, 
String signingAlgorithm) throws NoSuchAlgorithmException {
+this(createKeyPairGenerator(keyPairAlgorithm, keySize), days, 
signingAlgorithm);
+}
+
+protected TlsHelper(KeyPairGenerator keyPairGenerator, int days, 
String signingAlgorithm) {
+this.keyPairGenerator = keyPairGenerator;
+this.days = days;
+this.signingAlgorithm = signingAlgorithm;
+}
+
+public static void addBouncyCastleProvider() {
+Security.addProvider(new BouncyCastleProvider());
+}
+
+private static KeyPairGenerator createKeyPairGenerator(String 
algorithm, int keySize) throws NoSuchAlgorithmException {
+KeyPairGenerator instance = 
KeyPairGenerator.getInstance(algorithm);
+instance.initialize(keySize);
+return instance;
+}
  

[GitHub] nifi issue #740: Message on history page when clustered

[mcgilman]

Github user mcgilman commented on the issue:

https://github.com/apache/nifi/pull/740
  
Each node processed each action regardless if your browser was pointed to 
it. That's how the flows are kept in sync. The potential confusion here is with 
actions that happened to a node when it was not part of the cluster.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi pull request #695: NIFI-2193 - Command line SSL config utility as well ...

[alopresto]

Github user alopresto commented on a diff in the pull request:

https://github.com/apache/nifi/pull/695#discussion_r72713766
  
--- Diff: 
nifi-toolkit/nifi-toolkit-tls/src/main/java/org/apache/nifi/toolkit/tls/service/TlsCertificateAuthorityServiceHandler.java
 ---
@@ -0,0 +1,97 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.toolkit.tls.service;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.commons.io.input.BoundedReader;
+import org.apache.nifi.toolkit.tls.util.TlsHelper;
+import org.bouncycastle.pkcs.jcajce.JcaPKCS10CertificationRequest;
+import org.eclipse.jetty.server.Request;
+import org.eclipse.jetty.server.Response;
+import org.eclipse.jetty.server.handler.AbstractHandler;
+
+import javax.servlet.ServletException;
+import javax.servlet.http.HttpServletRequest;
+import javax.servlet.http.HttpServletResponse;
+import java.io.IOException;
+import java.security.KeyPair;
+import java.security.cert.X509Certificate;
+
+/**
+ * Jetty service handler that validates the hmac of a CSR and issues a 
certificate if it checks out
+ */
+public class TlsCertificateAuthorityServiceHandler extends AbstractHandler 
{
+public static final String CSR_FIELD_MUST_BE_SET = "csr field must be 
set";
+public static final String HMAC_FIELD_MUST_BE_SET = "hmac field must 
be set";
+public static final String FORBIDDEN = "forbidden";
+private final TlsHelper tlsHelper;
+private final String token;
+private final X509Certificate caCert;
+private final KeyPair keyPair;
+private final ObjectMapper objectMapper;
+
+public TlsCertificateAuthorityServiceHandler(TlsHelper tlsHelper, 
String token, X509Certificate caCert, KeyPair keyPair, ObjectMapper 
objectMapper) {
+this.tlsHelper = tlsHelper;
+this.token = token;
+this.caCert = caCert;
+this.keyPair = keyPair;
+this.objectMapper = objectMapper;
+}
+
+@Override
+public void handle(String target, Request baseRequest, 
HttpServletRequest request, HttpServletResponse response) throws IOException, 
ServletException {
+try {
+TlsCertificateAuthorityRequest tlsCertificateAuthorityRequest 
= objectMapper.readValue(new BoundedReader(request.getReader(), 1024 * 1024), 
TlsCertificateAuthorityRequest.class);
+
+if (!tlsCertificateAuthorityRequest.hasCsr()) {
+writeResponse(objectMapper, response, new 
TlsCertificateAuthorityResponse(CSR_FIELD_MUST_BE_SET), 
Response.SC_BAD_REQUEST);
+return;
+}
+
+if (!tlsCertificateAuthorityRequest.hasHmac()) {
--- End diff --

I think we should move the HMAC presence & validation to be the first steps 
in the logic here. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (NIFI-2193) Command Line Keystore and Truststore utility

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15398319#comment-15398319
 ] 

ASF GitHub Bot commented on NIFI-2193:
--

Github user alopresto commented on a diff in the pull request:

https://github.com/apache/nifi/pull/695#discussion_r72714290
  
--- Diff: 
nifi-toolkit/nifi-toolkit-tls/src/main/java/org/apache/nifi/toolkit/tls/service/TlsCertificateSigningRequestPerformer.java
 ---
@@ -0,0 +1,144 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.toolkit.tls.service;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.commons.io.IOUtils;
+import org.apache.commons.io.input.BoundedInputStream;
+import org.apache.http.HttpHost;
+import org.apache.http.client.methods.CloseableHttpResponse;
+import org.apache.http.client.methods.HttpPost;
+import org.apache.http.conn.ssl.TrustSelfSignedStrategy;
+import org.apache.http.entity.ByteArrayEntity;
+import org.apache.http.impl.client.CloseableHttpClient;
+import org.apache.http.impl.client.HttpClientBuilder;
+import org.apache.http.ssl.SSLContextBuilder;
+import org.apache.nifi.toolkit.tls.configuration.TlsClientConfig;
+import org.apache.nifi.toolkit.tls.util.TlsHelper;
+import org.bouncycastle.pkcs.jcajce.JcaPKCS10CertificationRequest;
+import org.eclipse.jetty.server.Response;
+
+import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+import java.security.KeyPair;
+import java.security.NoSuchAlgorithmException;
+import java.security.cert.X509Certificate;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.function.Supplier;
+
+public class TlsCertificateSigningRequestPerformer {
+public static final String RECEIVED_RESPONSE_CODE = "Received response 
code ";
+public static final String EXPECTED_ONE_CERTIFICATE = "Expected one 
certificate";
+public static final String EXPECTED_RESPONSE_TO_CONTAIN_HMAC = 
"Expected response to contain hmac";
+public static final String 
UNEXPECTED_HMAC_RECEIVED_POSSIBLE_MAN_IN_THE_MIDDLE = "Unexpected hmac 
received, possible man in the middle";
+public static final String EXPECTED_RESPONSE_TO_CONTAIN_CERTIFICATE = 
"Expected response to contain certificate";
+private final Supplier httpClientBuilderSupplier;
+private final String caHostname;
+private final String dn;
+private final String token;
+private final int port;
+private final TlsHelper tlsHelper;
+
+public TlsCertificateSigningRequestPerformer(TlsClientConfig 
tlsClientConfig) throws NoSuchAlgorithmException {
+this(HttpClientBuilder::create, tlsClientConfig.getCaHostname(), 
tlsClientConfig.getDn(), tlsClientConfig.getToken(), tlsClientConfig.getPort(), 
tlsClientConfig.createTlsHelper());
+}
+
+public 
TlsCertificateSigningRequestPerformer(Supplier 
httpClientBuilderSupplier, TlsClientConfig tlsClientConfig) throws 
NoSuchAlgorithmException {
--- End diff --

I don't see any external invocations of the two constructors that pass the 
`Supplier` argument. I'm all for dependency injection, but 
is this something that needs to be provided externally?


> Command Line Keystore and Truststore utility
> 
>
> Key: NIFI-2193
> URL: https://issues.apache.org/jira/browse/NIFI-2193
> Project: Apache NiFi
>  Issue Type: New Feature
>Reporter: Bryan Rosander
>Assignee: Bryan Rosander
>
> In order to facilitate secure setup of NiFi, it would be useful to have a 
> command line utility capable of generating the required keystores, 
> truststore, and relevant configuration files.
> It should be able to generate keystores for each NiFi node, a truststore that 
> they all use, and relevant passwords and configuration files for using the 
> keystores and truststore.
> Additionally, in 

[GitHub] nifi pull request #695: NIFI-2193 - Command line SSL config utility as well ...

[alopresto]

Github user alopresto commented on a diff in the pull request:

https://github.com/apache/nifi/pull/695#discussion_r72714290
  
--- Diff: 
nifi-toolkit/nifi-toolkit-tls/src/main/java/org/apache/nifi/toolkit/tls/service/TlsCertificateSigningRequestPerformer.java
 ---
@@ -0,0 +1,144 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.toolkit.tls.service;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.commons.io.IOUtils;
+import org.apache.commons.io.input.BoundedInputStream;
+import org.apache.http.HttpHost;
+import org.apache.http.client.methods.CloseableHttpResponse;
+import org.apache.http.client.methods.HttpPost;
+import org.apache.http.conn.ssl.TrustSelfSignedStrategy;
+import org.apache.http.entity.ByteArrayEntity;
+import org.apache.http.impl.client.CloseableHttpClient;
+import org.apache.http.impl.client.HttpClientBuilder;
+import org.apache.http.ssl.SSLContextBuilder;
+import org.apache.nifi.toolkit.tls.configuration.TlsClientConfig;
+import org.apache.nifi.toolkit.tls.util.TlsHelper;
+import org.bouncycastle.pkcs.jcajce.JcaPKCS10CertificationRequest;
+import org.eclipse.jetty.server.Response;
+
+import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+import java.security.KeyPair;
+import java.security.NoSuchAlgorithmException;
+import java.security.cert.X509Certificate;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.function.Supplier;
+
+public class TlsCertificateSigningRequestPerformer {
+public static final String RECEIVED_RESPONSE_CODE = "Received response 
code ";
+public static final String EXPECTED_ONE_CERTIFICATE = "Expected one 
certificate";
+public static final String EXPECTED_RESPONSE_TO_CONTAIN_HMAC = 
"Expected response to contain hmac";
+public static final String 
UNEXPECTED_HMAC_RECEIVED_POSSIBLE_MAN_IN_THE_MIDDLE = "Unexpected hmac 
received, possible man in the middle";
+public static final String EXPECTED_RESPONSE_TO_CONTAIN_CERTIFICATE = 
"Expected response to contain certificate";
+private final Supplier httpClientBuilderSupplier;
+private final String caHostname;
+private final String dn;
+private final String token;
+private final int port;
+private final TlsHelper tlsHelper;
+
+public TlsCertificateSigningRequestPerformer(TlsClientConfig 
tlsClientConfig) throws NoSuchAlgorithmException {
+this(HttpClientBuilder::create, tlsClientConfig.getCaHostname(), 
tlsClientConfig.getDn(), tlsClientConfig.getToken(), tlsClientConfig.getPort(), 
tlsClientConfig.createTlsHelper());
+}
+
+public 
TlsCertificateSigningRequestPerformer(Supplier 
httpClientBuilderSupplier, TlsClientConfig tlsClientConfig) throws 
NoSuchAlgorithmException {
--- End diff --

I don't see any external invocations of the two constructors that pass the 
`Supplier` argument. I'm all for dependency injection, but 
is this something that needs to be provided externally?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (NIFI-2193) Command Line Keystore and Truststore utility

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15398305#comment-15398305
 ] 

ASF GitHub Bot commented on NIFI-2193:
--

Github user alopresto commented on the issue:

https://github.com/apache/nifi/pull/695
  
Had a discussion with @brosander and I believe he is re-organizing the 
structure of the classes. I will continue adding comments on this iteration but 
I expect a new push is coming soon. 


> Command Line Keystore and Truststore utility
> 
>
> Key: NIFI-2193
> URL: https://issues.apache.org/jira/browse/NIFI-2193
> Project: Apache NiFi
>  Issue Type: New Feature
>Reporter: Bryan Rosander
>Assignee: Bryan Rosander
>
> In order to facilitate secure setup of NiFi, it would be useful to have a 
> command line utility capable of generating the required keystores, 
> truststore, and relevant configuration files.
> It should be able to generate keystores for each NiFi node, a truststore that 
> they all use, and relevant passwords and configuration files for using the 
> keystores and truststore.
> Additionally, in order to support distributed deployment, a web based 
> certificate authority with corresponding client will allow for each NiFi 
> instance to generate its own keypair and then request signing by the CA.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] nifi issue #740: Message on history page when clustered

[JPercivall]

Github user JPercivall commented on the issue:

https://github.com/apache/nifi/pull/740
  
Sorry more specifically, I have a 3 node cluster and made one change on 
each (while all were connected), I didn't expect to see all three of the 
changes in the history of each node. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi issue #695: NIFI-2193 - Command line SSL config utility as well as cert...

[alopresto]

Github user alopresto commented on the issue:

https://github.com/apache/nifi/pull/695
  
Had a discussion with @brosander and I believe he is re-organizing the 
structure of the classes. I will continue adding comments on this iteration but 
I expect a new push is coming soon. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi issue #740: Message on history page when clustered

[mcgilman]

Github user mcgilman commented on the issue:

https://github.com/apache/nifi/pull/740
  
Your first impression of that message is accurate. We are only showing 
actions from the node the browser is pointing to. Meaning, if that node has 
other history before it was clustered it would be available. Each node is only 
showing the history it knows about. If every node was only part of its current 
cluster then all nodes would report the same history.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] nifi issue #740: Message on history page when clustered

[JPercivall]

Github user JPercivall commented on the issue:

https://github.com/apache/nifi/pull/740
  
I was a bit confused by the message at first, I took it to mean "this 
history only shows changes that were made on this node" and was confused when I 
saw changes from every node on the history of each node. I'm not sure the most 
succinct/correct way to phrase it but maybe something more like "this history 
shows all changes made to the cluster while this node was connected"?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (NIFI-2423) Add a sensitive property to store the trust store password in the consume/publish kafka processors

[Joseph Witt (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15398255#comment-15398255
 ] 

Joseph Witt commented on NIFI-2423:
---

please double and triple check that any possible kafka property we'd want to 
support that needs sensitive protection gets accounted for.  That is the least 
we can do here.  It would be better to have all TLS related properties exposed 
frankly but mvp in my view is any possible property that needs to be consider 
sensitive.

> Add a sensitive property to store the trust store password in the 
> consume/publish kafka processors
> --
>
> Key: NIFI-2423
> URL: https://issues.apache.org/jira/browse/NIFI-2423
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Extensions
>Affects Versions: 0.7.0
>Reporter: Arpit Gupta
>Priority: Critical
>
> When user configures kafka processors to interact with SSL enabled protocol 
> they need to set additional configs.
> ssl.truststore.location and ssl.truststore.password. 
> Today the trust store password is clear text. We should make that a default 
> property and make it sensitive.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (NIFI-2303) Update lineage styles

[Scott Aslan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-2303?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Scott Aslan reassigned NIFI-2303:
-

Assignee: Scott Aslan

> Update lineage styles
> -
>
> Key: NIFI-2303
> URL: https://issues.apache.org/jira/browse/NIFI-2303
> Project: Apache NiFi
>  Issue Type: Sub-task
>  Components: Core UI
>Affects Versions: 1.0.0
>Reporter: Rob Moran
>Assignee: Scott Aslan
> Attachments: nifi-2303_reference-for-description.png
>
>
> * Selected event: #ba554a / rgb(186, 85, 74)
> * Other events: #aabbc3 / rgb(170,187,195)
> * Data provence icon: change to icon-provenance; color:#ad9897 / rgb(173, 
> 152, 151); change background/fill color to white (#fff / rgb(255, 255, 255)
> * Event type label: change font to Roboto, font-size to 11px
> * Path link "selected" strokes: change color to #ba554a / rgb(186, 85, 74)
> * Close icon: use fa-long-arrow-left; change tooltip to "Go back to event 
> list"
> * Update timestamp (below timeline slider control) to 13px Roboto Medium 
> #775351
> * Update slider to angular material slider? If necessary, the current style 
> is okay for now 
> * Change container border color to #e5ebed
> * Change container background color to #f9fafb
> ​Context menus should be updated to reflect current styling. For icons please 
> use:
> * View details: fa-info-circle
> * Find parents: fa-binoculars
> * Expand: fa-plus-square-o
> * Collapse: ​fa-minus-square-o



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] nifi pull request #736: Cleanup endpoints/UI when not running with a configu...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/nifi/pull/736


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (NIFI-2122) Users, Groups, Policies API handling of unsecure request

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15398231#comment-15398231
 ] 

ASF subversion and git services commented on NIFI-2122:
---

Commit da238b16efacb46cb78847027de07b197f3f88d0 in nifi's branch 
refs/heads/master from [~mcgilman]
[ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=da238b1 ]

NIFI-2122: - Merging responses for the current user and the flow configuration. 
- Returning whether NiFi is configured with a policy based authorizer in the 
flow configuration. - Only showing the users and policy icons when configured 
with a policy based authorizer. - Failing faster when invoking the users or 
policies endpoint when not configured with a configurable authorizer.

This closes #736

Signed-off-by: jpercivall 


> Users, Groups, Policies API handling of unsecure request
> 
>
> Key: NIFI-2122
> URL: https://issues.apache.org/jira/browse/NIFI-2122
> Project: Apache NiFi
>  Issue Type: Sub-task
>  Components: Core Framework
>Reporter: Matt Gilman
>Assignee: Matt Gilman
>Priority: Minor
> Fix For: 1.0.0
>
>
> When the users, groups, or policies endpoint receives an unsecure request we 
> should fail fast likely with an IllegalStateException as using the users, 
> groups, and policies endpoints require HTTPS. Only HTTP the user is always 
> authenticated as an anonymous user.
> Also update visibility of controls in the UI as appropriate for these 
> conditions as well.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (NIFI-2427) Labels shouldn't support "View the data" or "Modify the data" policies

[Joseph Percivall (JIRA)]

Joseph Percivall created NIFI-2427:
--

 Summary: Labels shouldn't support "View the data" or "Modify the 
data" policies
 Key: NIFI-2427
 URL: https://issues.apache.org/jira/browse/NIFI-2427
 Project: Apache NiFi
  Issue Type: Bug
Reporter: Joseph Percivall
Priority: Minor
 Fix For: 1.0.0


Currently Labels support the same properties as other configurable components 
which includes "View the data" and "Modify the data". Since labels never 
interact with data, these labels should be disabled for labels.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] nifi issue #736: Cleanup endpoints/UI when not running with a configurable a...

[JPercivall]

Github user JPercivall commented on the issue:

https://github.com/apache/nifi/pull/736
  
+1

Visually verified code and did a contrib check build. In a standalone 
instance and 3 node secure cluster verified that the policies didn't/did show 
up as expected. Thanks @mcgilman, I will merge it in.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (NIFI-2390) Decouple users/user groups and policies

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15398162#comment-15398162
 ] 

ASF GitHub Bot commented on NIFI-2390:
--

Github user asfgit closed the pull request at:

https://github.com/apache/nifi/pull/735


> Decouple users/user groups and policies
> ---
>
> Key: NIFI-2390
> URL: https://issues.apache.org/jira/browse/NIFI-2390
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Extensions
>Reporter: Matt Gilman
>Assignee: Bryan Bende
> Fix For: 1.0.0
>
>
> Currently, users, user groups, and policies are all persisted in the same 
> file. Because the policies are associated with a particular NiFi instance it 
> makes this file not portable. However, the users and user groups 
> configuration is completely independent of a particular instance and should 
> be persisted in a separate file to promote portable of this configuration 
> between NiFi instances.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] nifi issue #735: NIFI-2390 Separating of users and groups from authorization...

[mcgilman]

Github user mcgilman commented on the issue:

https://github.com/apache/nifi/pull/735
  
Looks good @bbende! This has been merged to master.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (NIFI-2390) Decouple users/user groups and policies

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15398163#comment-15398163
 ] 

ASF GitHub Bot commented on NIFI-2390:
--

Github user mcgilman commented on the issue:

https://github.com/apache/nifi/pull/735
  
Looks good @bbende! This has been merged to master.


> Decouple users/user groups and policies
> ---
>
> Key: NIFI-2390
> URL: https://issues.apache.org/jira/browse/NIFI-2390
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Extensions
>Reporter: Matt Gilman
>Assignee: Bryan Bende
> Fix For: 1.0.0
>
>
> Currently, users, user groups, and policies are all persisted in the same 
> file. Because the policies are associated with a particular NiFi instance it 
> makes this file not portable. However, the users and user groups 
> configuration is completely independent of a particular instance and should 
> be persisted in a separate file to promote portable of this configuration 
> between NiFi instances.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-2390) Decouple users/user groups and policies

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15398161#comment-15398161
 ] 

ASF subversion and git services commented on NIFI-2390:
---

Commit 5e4ba04589686963de1fc4218dd1365197c31c02 in nifi's branch 
refs/heads/master from [~bbende]
[ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=5e4ba04 ]

NIFI-2390 Separating of users and groups from authorizations.xml into separate 
file. This closes #735


> Decouple users/user groups and policies
> ---
>
> Key: NIFI-2390
> URL: https://issues.apache.org/jira/browse/NIFI-2390
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Extensions
>Reporter: Matt Gilman
>Assignee: Bryan Bende
> Fix For: 1.0.0
>
>
> Currently, users, user groups, and policies are all persisted in the same 
> file. Because the policies are associated with a particular NiFi instance it 
> makes this file not portable. However, the users and user groups 
> configuration is completely independent of a particular instance and should 
> be persisted in a separate file to promote portable of this configuration 
> between NiFi instances.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] nifi pull request #735: NIFI-2390 Separating of users and groups from author...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/nifi/pull/735


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Assigned] (NIFI-2322) ConsumeKafka gets stuck (cannot be stopped)

[Oleg Zhurakousky (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-2322?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Oleg Zhurakousky reassigned NIFI-2322:
--

Assignee: Oleg Zhurakousky

> ConsumeKafka gets stuck (cannot be stopped)
> ---
>
> Key: NIFI-2322
> URL: https://issues.apache.org/jira/browse/NIFI-2322
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Extensions
>Affects Versions: 1.0.0
>Reporter: Haimo Liu
>Assignee: Oleg Zhurakousky
> Fix For: 1.0.0
>
> Attachments: Screen Shot 2016-07-19 at 2.14.32 PM.png, Screen Shot 
> 2016-07-19 at 2.14.48 PM.png
>
>
> If kafka broker path is invalid or inaccurate, ConsumeKafka processor can get 
> stuck (concurrent tasks cannot be stopped in a clustered mode). Please refer 
> to the images in the attachment. It appears that a configurable timeout 
> property would potentially solve the problem.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (NIFI-2426) New service/task button visibility in Controller Settings is not correct

[Matt Gilman (JIRA)]

Matt Gilman created NIFI-2426:
-

 Summary: New service/task button visibility in Controller Settings 
is not correct
 Key: NIFI-2426
 URL: https://issues.apache.org/jira/browse/NIFI-2426
 Project: Apache NiFi
  Issue Type: Bug
  Components: Core UI
Reporter: Matt Gilman
 Fix For: 1.0.0


The new controller service / reporting task button is not visible in the 
correct states when viewing the Controller Settings.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-2322) ConsumeKafka gets stuck (cannot be stopped)

[Oleg Zhurakousky (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15398135#comment-15398135
 ] 

Oleg Zhurakousky commented on NIFI-2322:


At this point there seem to be nothing I can do other then performing a custom 
check if connection is possible to the specified server:
{code}
Socket client = new Socket();
client.connect(new InetSocketAddress("host", 9092), 1);
{code}
Tried every consumer property related to timeout
{code}
session.timeout.ms
 request.timeout.ms
 fetch.max.wait.ms
 reconnect.backoff.ms
 retry.backoff.ms
{code}
. . . with no joy. KafkaConsumer blocks on poll(timeout) even when such timeout 
is set to 0. Yet Kafka javadoc states:
{code}
/**
 * Fetch data for the topics or partitions specified using one of the 
subscribe/assign APIs. It is an error to not have
 * subscribed to any topics or partitions before polling for data.
 * 
 * On each poll, consumer will try to use the last consumed offset as the 
starting offset and fetch sequentially. The last
 * consumed offset can be manually set through {@link #seek(TopicPartition, 
long)} or automatically set as the last committed
 * offset for the subscribed list of partitions
 *
 *
 * @param timeout The time, in milliseconds, spent waiting in poll if data 
is not available. If 0, returns
 *immediately with any records that are available now. Must not 
be negative.
 * . . . .
 */
@Override
public ConsumerRecords poll(long timeout) {
. . .
{code}


> ConsumeKafka gets stuck (cannot be stopped)
> ---
>
> Key: NIFI-2322
> URL: https://issues.apache.org/jira/browse/NIFI-2322
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Extensions
>Affects Versions: 1.0.0
>Reporter: Haimo Liu
> Fix For: 1.0.0
>
> Attachments: Screen Shot 2016-07-19 at 2.14.32 PM.png, Screen Shot 
> 2016-07-19 at 2.14.48 PM.png
>
>
> If kafka broker path is invalid or inaccurate, ConsumeKafka processor can get 
> stuck (concurrent tasks cannot be stopped in a clustered mode). Please refer 
> to the images in the attachment. It appears that a configurable timeout 
> property would potentially solve the problem.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (NIFI-2425) Processors have invalid reference to controller service from template

[Bryan Bende (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-2425?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bryan Bende updated NIFI-2425:
--
Description: 
After importing and instantiating a template that existed before 1.0.0, the 
template imported without errors, and the controller service was created in the 
root group, but the two processors referencing it are invalid and show the UUID 
of the controller service rather than the name. This can easily be corrected by 
going into each processor's properties and changing the controller service 
selection, but should have been correct on import.

Template is attached.


  was:After importing and instantiating a template that existed before 1.0.0, 
the template imported without errors, and the controller service was created in 
the root group, but the two processors referencing it are invalid and show the 
UUID of the controller service rather than the name. This can easily be 
corrected by going into each processor's properties and changing the controller 
service selection, but should have been correct on import.


> Processors have invalid reference to controller service from template
> -
>
> Key: NIFI-2425
> URL: https://issues.apache.org/jira/browse/NIFI-2425
> Project: Apache NiFi
>  Issue Type: Bug
>Affects Versions: 1.0.0
>Reporter: Bryan Bende
>Priority: Minor
> Fix For: 1.0.0
>
> Attachments: HiveProcessors.xml
>
>
> After importing and instantiating a template that existed before 1.0.0, the 
> template imported without errors, and the controller service was created in 
> the root group, but the two processors referencing it are invalid and show 
> the UUID of the controller service rather than the name. This can easily be 
> corrected by going into each processor's properties and changing the 
> controller service selection, but should have been correct on import.
> Template is attached.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] nifi issue #735: NIFI-2390 Separating of users and groups from authorization...

[mcgilman]

Github user mcgilman commented on the issue:

https://github.com/apache/nifi/pull/735
  
Reviewing...


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (NIFI-2302) Add note to History to indicate it represents only the current Node

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15398040#comment-15398040
 ] 

ASF GitHub Bot commented on NIFI-2302:
--

GitHub user mcgilman opened a pull request:

https://github.com/apache/nifi/pull/740

Message on history page when clustered

NIFI-2302:
- Showing a message on the history page when in a cluster to indicate that 
only the actions from the current node are displayed.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/mcgilman/nifi NIFI-2302

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/740.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #740


commit bdda93c626c505634c2c52097425c2ed7b76f001
Author: Matt Gilman 
Date:   2016-07-28T19:07:16Z

NIFI-2302:
- Showing a message on the history page when in a cluster to indicate that 
only the actions from the current node are displayed.




> Add note to History to indicate it represents only the current Node
> ---
>
> Key: NIFI-2302
> URL: https://issues.apache.org/jira/browse/NIFI-2302
> Project: Apache NiFi
>  Issue Type: Improvement
>  Components: Core UI
>Reporter: Matt Gilman
>Assignee: Matt Gilman
> Fix For: 1.0.0
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] nifi pull request #706: NIFI-1868: Add PutHiveStreaming processor

[bbende]

Github user bbende commented on a diff in the pull request:

https://github.com/apache/nifi/pull/706#discussion_r72677845
  
--- Diff: 
nifi-nar-bundles/nifi-hive-bundle/nifi-hive-processors/src/main/java/org/apache/nifi/processors/hive/PutHiveStreaming.java
 ---
@@ -0,0 +1,657 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.hive;
+
+import com.google.common.util.concurrent.ThreadFactoryBuilder;
+import org.apache.avro.Schema;
+import org.apache.avro.file.DataFileStream;
+import org.apache.avro.generic.GenericDatumReader;
+import org.apache.avro.generic.GenericRecord;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.security.UserGroupInformation;
+import org.apache.hive.hcatalog.streaming.ConnectionError;
+import org.apache.hive.hcatalog.streaming.HiveEndPoint;
+import org.apache.hive.hcatalog.streaming.SerializationError;
+import org.apache.hive.hcatalog.streaming.StreamingException;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.components.Validator;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.hadoop.KerberosProperties;
+import org.apache.nifi.hadoop.SecurityUtil;
+import org.apache.nifi.logging.ComponentLog;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.util.NiFiProperties;
+import org.apache.nifi.util.StringUtils;
+import org.apache.nifi.util.hive.AuthenticationFailedException;
+import org.apache.nifi.util.hive.HiveConfigurator;
+import org.apache.nifi.util.hive.HiveOptions;
+import org.apache.nifi.util.hive.HiveUtils;
+import org.apache.nifi.util.hive.HiveWriter;
+import org.json.JSONException;
+import org.json.JSONObject;
+
+import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.Timer;
+import java.util.TimerTask;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.regex.Pattern;
+
+/**
+ * This processor utilizes the Hive Streaming capability to insert data 
from the flow into a Hive database table.
+ */
+@Tags({"hive", "streaming", "put", "database", "store"})
+@CapabilityDescription("This processor uses Hive Streaming to send flow 
file data to an Apache Hive table. The incoming flow file is expected to be in "
++ "Avro format and the table must exist in Hive. Please see the 
Hive documentation for requirements on the Hive table (format, partitions, 
etc.). "
++ "The partition values are extracted from the Avro record based 
on the names of the partition columns as specified in the processor. ")
+@WritesAttributes({
+@WritesAttribute(attribute = "hivestreaming.record.count", 
description = "The number of records from this flow file written using Hive 
Streaming.")
+})
+public 

[jira] [Commented] (NIFI-2305) Nodes run "Primary Node Only" processors when disconnected from cluster

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2305?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15397987#comment-15397987
 ] 

ASF GitHub Bot commented on NIFI-2305:
--

Github user JPercivall commented on the issue:

https://github.com/apache/nifi/pull/689
  
Reviewing this and came across a state that I'm not if its intended or not.

ncm and 2 node cluster with a flow GenerateFF (primary node only) -> 
LogAttribute
set a break point on line 3809 of FlowController.java to cause the primary 
node to stop sending heartbeats
after 45 seconds of not receiving heartbeats the NCM disconnects the node 
from the cluster
at that time observe that the FFs are no longer getting logged (assume 
because the GenerateFF is no longer being scheduled)

Also I cannot access the UI of the node that is disconnected.



> Nodes run "Primary Node Only" processors when disconnected from cluster
> ---
>
> Key: NIFI-2305
> URL: https://issues.apache.org/jira/browse/NIFI-2305
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Reporter: Mark Payne
>Assignee: Mark Payne
>Priority: Blocker
> Fix For: 1.0.0, 0.8.0
>
>
> Nodes that get disconnected from the cluster start running processors that 
> are considered 'primary node only'



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-1868) Add support for Hive Streaming

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-1868?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15397953#comment-15397953
 ] 

ASF GitHub Bot commented on NIFI-1868:
--

Github user bbende commented on a diff in the pull request:

https://github.com/apache/nifi/pull/706#discussion_r72673853
  
--- Diff: 
nifi-nar-bundles/nifi-hive-bundle/nifi-hive-processors/src/main/java/org/apache/nifi/processors/hive/PutHiveStreaming.java
 ---
@@ -0,0 +1,657 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.hive;
+
+import com.google.common.util.concurrent.ThreadFactoryBuilder;
+import org.apache.avro.Schema;
+import org.apache.avro.file.DataFileStream;
+import org.apache.avro.generic.GenericDatumReader;
+import org.apache.avro.generic.GenericRecord;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.security.UserGroupInformation;
+import org.apache.hive.hcatalog.streaming.ConnectionError;
+import org.apache.hive.hcatalog.streaming.HiveEndPoint;
+import org.apache.hive.hcatalog.streaming.SerializationError;
+import org.apache.hive.hcatalog.streaming.StreamingException;
+import org.apache.nifi.annotation.behavior.WritesAttribute;
+import org.apache.nifi.annotation.behavior.WritesAttributes;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.lifecycle.OnScheduled;
+import org.apache.nifi.annotation.lifecycle.OnStopped;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.components.ValidationResult;
+import org.apache.nifi.components.Validator;
+import org.apache.nifi.flowfile.FlowFile;
+import org.apache.nifi.hadoop.KerberosProperties;
+import org.apache.nifi.hadoop.SecurityUtil;
+import org.apache.nifi.logging.ComponentLog;
+import org.apache.nifi.processor.AbstractProcessor;
+import org.apache.nifi.processor.ProcessContext;
+import org.apache.nifi.processor.ProcessSession;
+import org.apache.nifi.processor.ProcessorInitializationContext;
+import org.apache.nifi.processor.Relationship;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processor.util.StandardValidators;
+import org.apache.nifi.util.NiFiProperties;
+import org.apache.nifi.util.StringUtils;
+import org.apache.nifi.util.hive.AuthenticationFailedException;
+import org.apache.nifi.util.hive.HiveConfigurator;
+import org.apache.nifi.util.hive.HiveOptions;
+import org.apache.nifi.util.hive.HiveUtils;
+import org.apache.nifi.util.hive.HiveWriter;
+import org.json.JSONException;
+import org.json.JSONObject;
+
+import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.Timer;
+import java.util.TimerTask;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.regex.Pattern;
+
+/**
+ * This processor utilizes the Hive Streaming capability to insert data 
from the flow into a Hive database table.
+ */
+@Tags({"hive", "streaming", "put", "database", "store"})
+@CapabilityDescription("This processor uses Hive Streaming to send flow 
file data to an Apache Hive table. The incoming flow file is expected to be in "
++ "Avro format and the table must exist in Hive. Please see the 
Hive documentation for requirements on the Hive table (format, partitions, 
etc.). "
++ "The partition values are extracted from the Avro record based 
on the names of the partition columns as 

[jira] [Updated] (NIFI-2421) Unable to copy/paste on standalone default instance

[Joseph Percivall (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-2421?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joseph Percivall updated NIFI-2421:
---
Resolution: Fixed
Status: Resolved  (was: Patch Available)

> Unable to copy/paste on standalone default instance
> ---
>
> Key: NIFI-2421
> URL: https://issues.apache.org/jira/browse/NIFI-2421
> Project: Apache NiFi
>  Issue Type: Bug
>Reporter: Joseph Percivall
>Assignee: Matt Gilman
>Priority: Blocker
> Fix For: 1.0.0
>
>
> start up default instance
> add component to graph
> attempt to copy and paste
> see error:
> Paste Error: 
> This NiFi is not configured to internally manage users, groups, and policies. 
> Please contact your system administrator.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-2419) Node could be elected Cluster Coordinator when disconnected from cluster

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2419?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15397938#comment-15397938
 ] 

ASF GitHub Bot commented on NIFI-2419:
--

GitHub user markap14 opened a pull request:

https://github.com/apache/nifi/pull/739

NIFI-2419: Ensure that if a node is disconnected that we unregister f…

…or 'cluster coordinator' and 'primary node' roles by updating 
FlowController to know that it is disconnected. Also removed dead code that was 
needed in the master-worker clustering paradigm but not for 
zero-master-clustering

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/markap14/nifi NIFI-2419

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/739.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #739






> Node could be elected Cluster Coordinator when disconnected from cluster
> 
>
> Key: NIFI-2419
> URL: https://issues.apache.org/jira/browse/NIFI-2419
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Affects Versions: 1.0.0
>Reporter: Mark Payne
>Assignee: Mark Payne
>Priority: Blocker
> Fix For: 1.0.0
>
>
> In my cluster, I see the following Node events for a particular node:
> 07/28/2016 06:17:29 UTC: Acquired role [Cluster Coordinator]
> 07/28/2016 04:53:30 UTC: Node Status changed from CONNECTING to DISCONNECTED 
> due to org.apache.nifi.controller.UninheritableFlowException: Failed to 
> connect node to cluster because local flow is different than cluster flow.
> So the Cluster Coordinator role was obtained even though the node is 
> disconnected. This shouldn't happen.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (NIFI-2419) Node could be elected Cluster Coordinator when disconnected from cluster

[Mark Payne (JIRA)]

 [ 
https://issues.apache.org/jira/browse/NIFI-2419?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mark Payne updated NIFI-2419:
-
Status: Patch Available  (was: In Progress)

> Node could be elected Cluster Coordinator when disconnected from cluster
> 
>
> Key: NIFI-2419
> URL: https://issues.apache.org/jira/browse/NIFI-2419
> Project: Apache NiFi
>  Issue Type: Bug
>  Components: Core Framework
>Affects Versions: 1.0.0
>Reporter: Mark Payne
>Assignee: Mark Payne
>Priority: Blocker
> Fix For: 1.0.0
>
>
> In my cluster, I see the following Node events for a particular node:
> 07/28/2016 06:17:29 UTC: Acquired role [Cluster Coordinator]
> 07/28/2016 04:53:30 UTC: Node Status changed from CONNECTING to DISCONNECTED 
> due to org.apache.nifi.controller.UninheritableFlowException: Failed to 
> connect node to cluster because local flow is different than cluster flow.
> So the Cluster Coordinator role was obtained even though the node is 
> disconnected. This shouldn't happen.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (NIFI-2421) Unable to copy/paste on standalone default instance

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/NIFI-2421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15397932#comment-15397932
 ] 

ASF subversion and git services commented on NIFI-2421:
---

Commit 01adb050f93ced1fe0ff85963265d286e1817a69 in nifi's branch 
refs/heads/master from [~mcgilman]
[ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=01adb05 ]

NIFI-2421: - Only attempting to clone policies when NiFI supports a 
configurable authorizer.

This closes #738

Signed-off-by: jpercivall 


> Unable to copy/paste on standalone default instance
> ---
>
> Key: NIFI-2421
> URL: https://issues.apache.org/jira/browse/NIFI-2421
> Project: Apache NiFi
>  Issue Type: Bug
>Reporter: Joseph Percivall
>Assignee: Matt Gilman
>Priority: Blocker
> Fix For: 1.0.0
>
>
> start up default instance
> add component to graph
> attempt to copy and paste
> see error:
> Paste Error: 
> This NiFi is not configured to internally manage users, groups, and policies. 
> Please contact your system administrator.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)