[GitHub] [nifi] natural commented on a change in pull request #3594: NIFI-3833 Support for Encrypted Flow File Repositories
natural commented on a change in pull request #3594: NIFI-3833 Support for Encrypted Flow File Repositories URL: https://github.com/apache/nifi/pull/3594#discussion_r323578236 ## File path: nifi-commons/nifi-write-ahead-log/src/main/java/org/apache/nifi/wali/HashMapSnapshot.java ## @@ -264,10 +267,11 @@ public synchronized void writeSnapshot(final SnapshotCapture snapshot) throws } // Write to the partial file. -try (final FileOutputStream fileOut = new FileOutputStream(getPartialFile()); Review comment: (in this specific case, however, moving the stream construction back into the try-with-resources declaration still causes test failures) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [nifi] natural commented on a change in pull request #3594: NIFI-3833 Support for Encrypted Flow File Repositories
natural commented on a change in pull request #3594: NIFI-3833 Support for Encrypted Flow File Repositories URL: https://github.com/apache/nifi/pull/3594#discussion_r323575686 ## File path: nifi-docs/src/main/asciidoc/administration-guide.adoc ## @@ -2477,6 +2477,13 @@ implementation. |`nifi.flowfile.repository.always.sync`|If set to `true`, any change to the repository will be synchronized to the disk, meaning that NiFi will ask the operating system not to cache the information. This is very expensive and can significantly reduce NiFi performance. However, if it is `false`, there could be the potential for data loss if either there is a sudden power loss or the operating system crashes. The default value is `false`. | + Encryption + +The FlowFile repository can be configured to encrypt all files as they are written to disk. To enable this encryption, +set the `nifi.flowfile.repository.always.key.1` property to a 16 or 32 bit value like this: Review comment: Great point! Still working on the key management + documentation bits. Will update when I've got more. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [nifi] natural commented on a change in pull request #3594: NIFI-3833 Support for Encrypted Flow File Repositories
natural commented on a change in pull request #3594: NIFI-3833 Support for
Encrypted Flow File Repositories
URL: https://github.com/apache/nifi/pull/3594#discussion_r323575365
##
File path:
nifi-commons/nifi-write-ahead-log/src/main/java/org/apache/nifi/wali/SimpleCipherOutputStream.java
##
@@ -0,0 +1,72 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.wali;
+
+import org.bouncycastle.crypto.io.CipherOutputStream;
+import org.bouncycastle.crypto.modes.AEADBlockCipher;
+
+import javax.crypto.SecretKey;
+import java.io.IOException;
+import java.io.OutputStream;
+
+/**
+ * This class extends {@link CipherOutputStream} with a static factory method
for constructing
+ * an output stream with an AEAD block cipher.
+ *
+ * Note that the {@link CipherOutputStream} implementation writes the MAC at
the end of the stream during `close`.
+ * If streams using this class aren't closed properly, the result may be a
stream without a MAC written, which
+ * causes a MAC authentication failure in the input stream.
+ *
+ */
+public class SimpleCipherOutputStream extends CipherOutputStream {
+/**
+ * Constructs an {@link OutputStream} from an existing {@link
OutputStream} and block cipher.
+ *
+ * @param out output stream to wrap.
+ * @param cipher block cipher, initialized for encryption.
+ */
+public SimpleCipherOutputStream(OutputStream out, AEADBlockCipher cipher) {
+super(out, cipher);
+}
+
+/**
+ * Static factory for wrapping an output stream with a block cipher.
+ *
+ * NB: this function eagerly writes the initial cipher values to the
plain output stream before returning the cipher stream.
+ *
+ * @param out output stream to wrap.
+ * @param key cipher key.
+ * @return wrapped output stream.
+ * @throws IOException if the stream cannot be written eagerly, or if the
cipher cannot be initialized.
+ */
+public static OutputStream wrapWithKey(OutputStream out, SecretKey key)
throws IOException {
+if (key == null) {
+return out;
Review comment:
We're checking for null keys at the call site now, so no more implicit
wrapping.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [nifi] natural commented on a change in pull request #3594: NIFI-3833 Support for Encrypted Flow File Repositories
natural commented on a change in pull request #3594: NIFI-3833 Support for
Encrypted Flow File Repositories
URL: https://github.com/apache/nifi/pull/3594#discussion_r323574842
##
File path:
nifi-commons/nifi-write-ahead-log/src/main/java/org/apache/nifi/wali/SimpleCipherInputStream.java
##
@@ -0,0 +1,104 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.wali;
+
+import org.bouncycastle.crypto.io.CipherInputStream;
+import org.bouncycastle.crypto.modes.AEADBlockCipher;
+
+import javax.crypto.SecretKey;
+import java.io.IOException;
+import java.io.InputStream;
+
+/**
+ * This class extends {@link CipherInputStream} with a static factory method
for constructing
+ * an input stream with an AEAD block cipher.
+ */
+public class SimpleCipherInputStream extends CipherInputStream {
+protected AEADBlockCipher cipher;
+
+/**
+ * Constructs an {@link InputStream} from an existing {@link InputStream}
and block cipher.
+ *
+ * @param in input stream to wrap.
+ * @param cipher block cipher, initialized for decryption.
+ */
+public SimpleCipherInputStream(InputStream in, AEADBlockCipher cipher) {
+super(in, cipher);
+this.cipher = cipher;
+}
+
+/**
+ * Static factory for wrapping an input stream with a block cipher.
+ *
+ * NB: this function eagerly reads the initial cipher values from the
plain input stream before returning the cipher stream.
+ *
+ * @param in input stream to wrap.
+ * @param key cipher key.
+ * @return wrapped input stream.
+ * @throws IOException if the stream cannot be read eagerly, or if the
cipher cannot be initialized.
+ */
+public static InputStream wrapWithKey(InputStream in, SecretKey key)
throws IOException {
+if (key == null) {
+return in;
+}
+
+if (in.markSupported()) {
+in.mark(0);
+}
+
+// Read the marker, the iv, and the aad in the same order as they're
written in the SimpleCipherOutputStream:
+try {
+final int marker = in.read();
+if (marker != SimpleCipherUtil.MARKER_BYTE) {
+if (in.markSupported()) {
+in.reset();
+}
+return in;
+}
+
+byte[] iv = new byte[SimpleCipherUtil.IV_BYTE_LEN];
+int len = in.read(iv);
+if (len != iv.length) {
+throw new IOException("Could not read IV.");
+}
+
+byte[] aad = new byte[SimpleCipherUtil.AAD_BYTE_LEN];
+len = in.read(aad);
+if (len != aad.length) {
+throw new IOException("Could not read AAD.");
+}
+
+AEADBlockCipher cipher = SimpleCipherUtil.initCipher(key, false,
iv, aad);
+return new SimpleCipherInputStream(in, cipher);
+
+} catch (final IOException ignored) {
Review comment:
Good catch, removed.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [nifi] natural commented on a change in pull request #3594: NIFI-3833 Support for Encrypted Flow File Repositories
natural commented on a change in pull request #3594: NIFI-3833 Support for
Encrypted Flow File Repositories
URL: https://github.com/apache/nifi/pull/3594#discussion_r323575201
##
File path:
nifi-commons/nifi-write-ahead-log/src/main/java/org/apache/nifi/wali/SimpleCipherInputStream.java
##
@@ -0,0 +1,104 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.wali;
+
+import org.bouncycastle.crypto.io.CipherInputStream;
+import org.bouncycastle.crypto.modes.AEADBlockCipher;
+
+import javax.crypto.SecretKey;
+import java.io.IOException;
+import java.io.InputStream;
+
+/**
+ * This class extends {@link CipherInputStream} with a static factory method
for constructing
+ * an input stream with an AEAD block cipher.
+ */
+public class SimpleCipherInputStream extends CipherInputStream {
+protected AEADBlockCipher cipher;
+
+/**
+ * Constructs an {@link InputStream} from an existing {@link InputStream}
and block cipher.
+ *
+ * @param in input stream to wrap.
+ * @param cipher block cipher, initialized for decryption.
+ */
+public SimpleCipherInputStream(InputStream in, AEADBlockCipher cipher) {
+super(in, cipher);
+this.cipher = cipher;
+}
+
+/**
+ * Static factory for wrapping an input stream with a block cipher.
+ *
+ * NB: this function eagerly reads the initial cipher values from the
plain input stream before returning the cipher stream.
+ *
+ * @param in input stream to wrap.
+ * @param key cipher key.
+ * @return wrapped input stream.
+ * @throws IOException if the stream cannot be read eagerly, or if the
cipher cannot be initialized.
+ */
+public static InputStream wrapWithKey(InputStream in, SecretKey key)
throws IOException {
+if (key == null) {
+return in;
+}
+
+if (in.markSupported()) {
+in.mark(0);
+}
+
+// Read the marker, the iv, and the aad in the same order as they're
written in the SimpleCipherOutputStream:
+try {
+final int marker = in.read();
+if (marker != SimpleCipherUtil.MARKER_BYTE) {
+if (in.markSupported()) {
+in.reset();
+}
+return in;
+}
+
+byte[] iv = new byte[SimpleCipherUtil.IV_BYTE_LEN];
+int len = in.read(iv);
+if (len != iv.length) {
+throw new IOException("Could not read IV.");
+}
+
+byte[] aad = new byte[SimpleCipherUtil.AAD_BYTE_LEN];
+len = in.read(aad);
+if (len != aad.length) {
+throw new IOException("Could not read AAD.");
+}
+
+AEADBlockCipher cipher = SimpleCipherUtil.initCipher(key, false,
iv, aad);
+return new SimpleCipherInputStream(in, cipher);
+
+} catch (final IOException ignored) {
+if (in.markSupported()) {
+in.reset();
+}
+return in;
+}
+}
+
+@Override
+public void close() throws IOException {
+try {
Review comment:
Ah, my bad for not using the specific exception. The base class
implementation throws `InvalidCipherTextIOException`, so I've updated the code
to catch that + rethrow the `IOException`.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [nifi] natural commented on a change in pull request #3594: NIFI-3833 Support for Encrypted Flow File Repositories
natural commented on a change in pull request #3594: NIFI-3833 Support for
Encrypted Flow File Repositories
URL: https://github.com/apache/nifi/pull/3594#discussion_r323574842
##
File path:
nifi-commons/nifi-write-ahead-log/src/main/java/org/apache/nifi/wali/SimpleCipherInputStream.java
##
@@ -0,0 +1,104 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.wali;
+
+import org.bouncycastle.crypto.io.CipherInputStream;
+import org.bouncycastle.crypto.modes.AEADBlockCipher;
+
+import javax.crypto.SecretKey;
+import java.io.IOException;
+import java.io.InputStream;
+
+/**
+ * This class extends {@link CipherInputStream} with a static factory method
for constructing
+ * an input stream with an AEAD block cipher.
+ */
+public class SimpleCipherInputStream extends CipherInputStream {
+protected AEADBlockCipher cipher;
+
+/**
+ * Constructs an {@link InputStream} from an existing {@link InputStream}
and block cipher.
+ *
+ * @param in input stream to wrap.
+ * @param cipher block cipher, initialized for decryption.
+ */
+public SimpleCipherInputStream(InputStream in, AEADBlockCipher cipher) {
+super(in, cipher);
+this.cipher = cipher;
+}
+
+/**
+ * Static factory for wrapping an input stream with a block cipher.
+ *
+ * NB: this function eagerly reads the initial cipher values from the
plain input stream before returning the cipher stream.
+ *
+ * @param in input stream to wrap.
+ * @param key cipher key.
+ * @return wrapped input stream.
+ * @throws IOException if the stream cannot be read eagerly, or if the
cipher cannot be initialized.
+ */
+public static InputStream wrapWithKey(InputStream in, SecretKey key)
throws IOException {
+if (key == null) {
+return in;
+}
+
+if (in.markSupported()) {
+in.mark(0);
+}
+
+// Read the marker, the iv, and the aad in the same order as they're
written in the SimpleCipherOutputStream:
+try {
+final int marker = in.read();
+if (marker != SimpleCipherUtil.MARKER_BYTE) {
+if (in.markSupported()) {
+in.reset();
+}
+return in;
+}
+
+byte[] iv = new byte[SimpleCipherUtil.IV_BYTE_LEN];
+int len = in.read(iv);
+if (len != iv.length) {
+throw new IOException("Could not read IV.");
+}
+
+byte[] aad = new byte[SimpleCipherUtil.AAD_BYTE_LEN];
+len = in.read(aad);
+if (len != aad.length) {
+throw new IOException("Could not read AAD.");
+}
+
+AEADBlockCipher cipher = SimpleCipherUtil.initCipher(key, false,
iv, aad);
+return new SimpleCipherInputStream(in, cipher);
+
+} catch (final IOException ignored) {
Review comment:
Ah, my bad for not using the specific exception. The base class
implementation throws `InvalidCipherTextIOException`, so I've updated the code
to catch that + rethrow the `IOException`.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [nifi] natural commented on a change in pull request #3594: NIFI-3833 Support for Encrypted Flow File Repositories
natural commented on a change in pull request #3594: NIFI-3833 Support for
Encrypted Flow File Repositories
URL: https://github.com/apache/nifi/pull/3594#discussion_r323574261
##
File path:
nifi-commons/nifi-write-ahead-log/src/main/java/org/apache/nifi/wali/SimpleCipherInputStream.java
##
@@ -0,0 +1,104 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.wali;
+
+import org.bouncycastle.crypto.io.CipherInputStream;
+import org.bouncycastle.crypto.modes.AEADBlockCipher;
+
+import javax.crypto.SecretKey;
+import java.io.IOException;
+import java.io.InputStream;
+
+/**
+ * This class extends {@link CipherInputStream} with a static factory method
for constructing
+ * an input stream with an AEAD block cipher.
+ */
+public class SimpleCipherInputStream extends CipherInputStream {
+protected AEADBlockCipher cipher;
+
+/**
+ * Constructs an {@link InputStream} from an existing {@link InputStream}
and block cipher.
+ *
+ * @param in input stream to wrap.
+ * @param cipher block cipher, initialized for decryption.
+ */
+public SimpleCipherInputStream(InputStream in, AEADBlockCipher cipher) {
+super(in, cipher);
+this.cipher = cipher;
+}
+
+/**
+ * Static factory for wrapping an input stream with a block cipher.
+ *
+ * NB: this function eagerly reads the initial cipher values from the
plain input stream before returning the cipher stream.
+ *
+ * @param in input stream to wrap.
+ * @param key cipher key.
+ * @return wrapped input stream.
+ * @throws IOException if the stream cannot be read eagerly, or if the
cipher cannot be initialized.
+ */
+public static InputStream wrapWithKey(InputStream in, SecretKey key)
throws IOException {
+if (key == null) {
+return in;
+}
+
+if (in.markSupported()) {
+in.mark(0);
+}
+
+// Read the marker, the iv, and the aad in the same order as they're
written in the SimpleCipherOutputStream:
+try {
+final int marker = in.read();
+if (marker != SimpleCipherUtil.MARKER_BYTE) {
+if (in.markSupported()) {
+in.reset();
+}
+return in;
+}
+
+byte[] iv = new byte[SimpleCipherUtil.IV_BYTE_LEN];
+int len = in.read(iv);
+if (len != iv.length) {
+throw new IOException("Could not read IV.");
+}
+
+byte[] aad = new byte[SimpleCipherUtil.AAD_BYTE_LEN];
+len = in.read(aad);
Review comment:
Same fix and appreciation.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [nifi] natural commented on a change in pull request #3594: NIFI-3833 Support for Encrypted Flow File Repositories
natural commented on a change in pull request #3594: NIFI-3833 Support for
Encrypted Flow File Repositories
URL: https://github.com/apache/nifi/pull/3594#discussion_r323574182
##
File path:
nifi-commons/nifi-write-ahead-log/src/main/java/org/apache/nifi/wali/SimpleCipherInputStream.java
##
@@ -0,0 +1,104 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.wali;
+
+import org.bouncycastle.crypto.io.CipherInputStream;
+import org.bouncycastle.crypto.modes.AEADBlockCipher;
+
+import javax.crypto.SecretKey;
+import java.io.IOException;
+import java.io.InputStream;
+
+/**
+ * This class extends {@link CipherInputStream} with a static factory method
for constructing
+ * an input stream with an AEAD block cipher.
+ */
+public class SimpleCipherInputStream extends CipherInputStream {
+protected AEADBlockCipher cipher;
+
+/**
+ * Constructs an {@link InputStream} from an existing {@link InputStream}
and block cipher.
+ *
+ * @param in input stream to wrap.
+ * @param cipher block cipher, initialized for decryption.
+ */
+public SimpleCipherInputStream(InputStream in, AEADBlockCipher cipher) {
+super(in, cipher);
+this.cipher = cipher;
+}
+
+/**
+ * Static factory for wrapping an input stream with a block cipher.
+ *
+ * NB: this function eagerly reads the initial cipher values from the
plain input stream before returning the cipher stream.
+ *
+ * @param in input stream to wrap.
+ * @param key cipher key.
+ * @return wrapped input stream.
+ * @throws IOException if the stream cannot be read eagerly, or if the
cipher cannot be initialized.
+ */
+public static InputStream wrapWithKey(InputStream in, SecretKey key)
throws IOException {
+if (key == null) {
+return in;
+}
+
+if (in.markSupported()) {
+in.mark(0);
+}
+
+// Read the marker, the iv, and the aad in the same order as they're
written in the SimpleCipherOutputStream:
+try {
+final int marker = in.read();
+if (marker != SimpleCipherUtil.MARKER_BYTE) {
+if (in.markSupported()) {
+in.reset();
+}
+return in;
+}
+
+byte[] iv = new byte[SimpleCipherUtil.IV_BYTE_LEN];
+int len = in.read(iv);
Review comment:
I *knew* this code smelled wrong when I wrote it, but it passed all of the
existing tests! So thanks for pointing out the better approach. Updated.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [nifi] natural commented on a change in pull request #3594: NIFI-3833 Support for Encrypted Flow File Repositories
natural commented on a change in pull request #3594: NIFI-3833 Support for
Encrypted Flow File Repositories
URL: https://github.com/apache/nifi/pull/3594#discussion_r323573964
##
File path:
nifi-commons/nifi-write-ahead-log/src/main/java/org/apache/nifi/wali/SimpleCipherInputStream.java
##
@@ -0,0 +1,104 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.wali;
+
+import org.bouncycastle.crypto.io.CipherInputStream;
+import org.bouncycastle.crypto.modes.AEADBlockCipher;
+
+import javax.crypto.SecretKey;
+import java.io.IOException;
+import java.io.InputStream;
+
+/**
+ * This class extends {@link CipherInputStream} with a static factory method
for constructing
+ * an input stream with an AEAD block cipher.
+ */
+public class SimpleCipherInputStream extends CipherInputStream {
+protected AEADBlockCipher cipher;
+
+/**
+ * Constructs an {@link InputStream} from an existing {@link InputStream}
and block cipher.
+ *
+ * @param in input stream to wrap.
+ * @param cipher block cipher, initialized for decryption.
+ */
+public SimpleCipherInputStream(InputStream in, AEADBlockCipher cipher) {
+super(in, cipher);
+this.cipher = cipher;
+}
+
+/**
+ * Static factory for wrapping an input stream with a block cipher.
+ *
+ * NB: this function eagerly reads the initial cipher values from the
plain input stream before returning the cipher stream.
+ *
+ * @param in input stream to wrap.
+ * @param key cipher key.
+ * @return wrapped input stream.
+ * @throws IOException if the stream cannot be read eagerly, or if the
cipher cannot be initialized.
+ */
+public static InputStream wrapWithKey(InputStream in, SecretKey key)
throws IOException {
+if (key == null) {
+return in;
+}
+
+if (in.markSupported()) {
+in.mark(0);
+}
+
+// Read the marker, the iv, and the aad in the same order as they're
written in the SimpleCipherOutputStream:
+try {
+final int marker = in.read();
+if (marker != SimpleCipherUtil.MARKER_BYTE) {
Review comment:
The original static method was (purposefully!) combining two operations:
check the initial byte to see if it's a cipher stream, and if so, create a
suitable cipher.
The replacement code splits these into two explicit steps with more
applicable names (i.e., `peekForMarker`). Hopefully that makes the intent
clearer!
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [nifi] natural commented on a change in pull request #3594: NIFI-3833 Support for Encrypted Flow File Repositories
natural commented on a change in pull request #3594: NIFI-3833 Support for
Encrypted Flow File Repositories
URL: https://github.com/apache/nifi/pull/3594#discussion_r323573295
##
File path:
nifi-commons/nifi-write-ahead-log/src/main/java/org/apache/nifi/wali/LengthDelimitedJournal.java
##
@@ -411,9 +417,10 @@ public JournalRecovery recoverRecords(final Map recordMap, final Set<
final double journalLength = journalFile.length();
try (final InputStream fis = new FileInputStream(journalFile);
-final InputStream bufferedIn = new BufferedInputStream(fis);
-final ByteCountingInputStream byteCountingIn = new
ByteCountingInputStream(bufferedIn);
-final DataInputStream in = new DataInputStream(byteCountingIn)) {
+ final InputStream bufferedIn = new BufferedInputStream(fis);
+ final InputStream plainIn =
SimpleCipherInputStream.wrapWithKey(bufferedIn, cipherKey);
+ final ByteCountingInputStream byteCountingIn = new
ByteCountingInputStream(plainIn);
Review comment:
Yeah, that's actually intentional at this point. Moving the
`ByteCountingInputStream` below the cipher stream caused lots of test failures.
I can swing back and re-investigate later if you'd prefer.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [nifi] natural commented on a change in pull request #3594: NIFI-3833 Support for Encrypted Flow File Repositories
natural commented on a change in pull request #3594: NIFI-3833 Support for
Encrypted Flow File Repositories
URL: https://github.com/apache/nifi/pull/3594#discussion_r323572526
##
File path:
nifi-commons/nifi-write-ahead-log/src/main/java/org/apache/nifi/wali/LengthDelimitedJournal.java
##
@@ -373,7 +378,8 @@ public synchronized void fsync() throws IOException {
try {
if (fileOut != null) {
-fileOut.getChannel().force(false);
+// Before transparent crypto, this was:
fileOut.getChannel().force(false);
Review comment:
I exposed the `FileOutputStream` as a member method and reverted the line
above.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [nifi] natural commented on a change in pull request #3594: NIFI-3833 Support for Encrypted Flow File Repositories
natural commented on a change in pull request #3594: NIFI-3833 Support for Encrypted Flow File Repositories URL: https://github.com/apache/nifi/pull/3594#discussion_r323572363 ## File path: nifi-commons/nifi-write-ahead-log/src/main/java/org/apache/nifi/wali/HashMapSnapshot.java ## @@ -264,10 +267,11 @@ public synchronized void writeSnapshot(final SnapshotCapture snapshot) throws } // Write to the partial file. -try (final FileOutputStream fileOut = new FileOutputStream(getPartialFile()); Review comment: My intent was to avoid having to check for a null cipher key at every call site, but as you point out, some of those sites are within a try-with-resources, making the static method that I implemented... questionable. I've replaced the static method with a ctor and some support methods, and updated all of the call sites with a null check. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [nifi] natural commented on issue #3594: NIFI-3833 Support for Encrypted Flow File Repositories
natural commented on issue #3594: NIFI-3833 Support for Encrypted Flow File Repositories URL: https://github.com/apache/nifi/pull/3594#issuecomment-530682987 > It's also important to consider how NiFi handles FlowFile swapping. Though the `FlowFileSwapManager` is not part of the FlowFile Repository, the default location to write data to is `./flowfile_repository/swap` and I think most users would expect that if encryption is enabled on the FlowFile Repository that any data written to the `swap` directory would also be encrypted. I feel this is a fair assumption, so we will need to be sure that we update the `FileSystemSwapManager` and any related code for that, as well. I missed `FlowFileSwapManager` and friends because I didn't look beyond the single package that had all the other FF bits in it. Thank you for pointing that out and helping me advert a mistake! Added transparent crypto support to those implementations. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [nifi] BoyillaGanesh commented on issue #3717: NIFI-6610 Disconnected Relationship support of Connect WebSocket processor in Nifi
BoyillaGanesh commented on issue #3717: NIFI-6610 Disconnected Relationship support of Connect WebSocket processor in Nifi URL: https://github.com/apache/nifi/pull/3717#issuecomment-530673894 could some one please review my code ? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [nifi-minifi] jzonthemtn commented on a change in pull request #168: MINIFI-512 Change bootstrap port command handling
jzonthemtn commented on a change in pull request #168: MINIFI-512 Change
bootstrap port command handling
URL: https://github.com/apache/nifi-minifi/pull/168#discussion_r323469127
##
File path:
minifi-bootstrap/src/main/java/org/apache/nifi/minifi/bootstrap/RunMiNiFi.java
##
@@ -1562,6 +1564,12 @@ void setAutoRestartNiFi(final boolean restart) {
}
void setMiNiFiCommandControlPort(final int port, final String secretKey)
throws IOException {
+
+if (this.secretKey != null && this.ccPort != UNINITIALIZED_CC_PORT) {
Review comment:
Just wondering if these will always be set together, or would `||` be more
appropriate?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [nifi-minifi] kevdoran opened a new pull request #168: MINIFI-512 Change bootstrap port command handling
kevdoran opened a new pull request #168: MINIFI-512 Change bootstrap port command handling URL: https://github.com/apache/nifi-minifi/pull/168 Thank you for submitting a contribution to Apache NiFi - MiNiFi. In order to streamline the review of the contribution we ask you to ensure the following steps have been taken: ### For all changes: - [ ] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message? - [ ] Does your PR title start with MINIFI- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [ ] Has your PR been rebased against the latest commit within the target branch (typically master)? - [ ] Is your initial contribution a single, squashed commit? ### For code changes: - [ ] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi-minifi folder? - [ ] Have you written or updated unit tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](https://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the LICENSE file, including the main LICENSE file under minifi-assembly? - [ ] If applicable, have you updated the NOTICE file, including the main NOTICE file found under minifi-assembly? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered? ### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [nifi] kevdoran opened a new pull request #3722: NIFI-6653 Change bootstrap port command handling
kevdoran opened a new pull request #3722: NIFI-6653 Change bootstrap port
command handling
URL: https://github.com/apache/nifi/pull/3722
Thank you for submitting a contribution to Apache NiFi.
Please provide a short description of the PR here:
Description of PR
_Enables X functionality; fixes bug NIFI-._
In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:
### For all changes:
- [ ] Is there a JIRA ticket associated with this PR? Is it referenced
in the commit message?
- [ ] Does your PR title start with **NIFI-** where is the JIRA
number you are trying to resolve? Pay particular attention to the hyphen "-"
character.
- [ ] Has your PR been rebased against the latest commit within the target
branch (typically `master`)?
- [ ] Is your initial contribution a single, squashed commit? _Additional
commits in response to PR reviewer feedback should be made on this branch and
pushed to allow change tracking. Do not `squash` or use `--force` when pushing
to allow for clean monitoring of changes._
### For code changes:
- [ ] Have you ensured that the full suite of tests is executed via `mvn
-Pcontrib-check clean install` at the root `nifi` folder?
- [ ] Have you written or updated unit tests to verify your changes?
- [ ] Have you verified that the full build is successful on both JDK 8 and
JDK 11?
- [ ] If adding new dependencies to the code, are these dependencies
licensed in a way that is compatible for inclusion under [ASF
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] If applicable, have you updated the `LICENSE` file, including the main
`LICENSE` file under `nifi-assembly`?
- [ ] If applicable, have you updated the `NOTICE` file, including the main
`NOTICE` file found under `nifi-assembly`?
- [ ] If adding new Properties, have you added `.displayName` in addition to
.name (programmatic access) for each of the new properties?
### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in which
it is rendered?
### Note:
Please ensure that once the PR is submitted, you check travis-ci for build
issues and submit an update to your PR as soon as possible.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [nifi-registry] kevdoran opened a new pull request #228: NIFIREG-318 Change bootstrap port command handling
kevdoran opened a new pull request #228: NIFIREG-318 Change bootstrap port command handling URL: https://github.com/apache/nifi-registry/pull/228 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Created] (NIFIREG-318) Only allow bootstrap port to be set during initial startup
Kevin Doran created NIFIREG-318: --- Summary: Only allow bootstrap port to be set during initial startup Key: NIFIREG-318 URL: https://issues.apache.org/jira/browse/NIFIREG-318 Project: NiFi Registry Issue Type: Improvement Reporter: Kevin Doran Assignee: Kevin Doran Fix For: 1.0.0 -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Created] (NIFI-6653) Only allow bootstrap port to be set during initial startup
Kevin Doran created NIFI-6653: - Summary: Only allow bootstrap port to be set during initial startup Key: NIFI-6653 URL: https://issues.apache.org/jira/browse/NIFI-6653 Project: Apache NiFi Issue Type: Improvement Components: Core Framework Reporter: Kevin Doran Assignee: Kevin Doran Fix For: 1.10.0 -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Assigned] (NIFI-6381) Make Parameters and Parameter Contexts searchable in UI
[ https://issues.apache.org/jira/browse/NIFI-6381?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Fellows reassigned NIFI-6381: Assignee: Robert Fellows > Make Parameters and Parameter Contexts searchable in UI > --- > > Key: NIFI-6381 > URL: https://issues.apache.org/jira/browse/NIFI-6381 > Project: Apache NiFi > Issue Type: Sub-task > Components: Core Framework >Reporter: Mark Payne >Assignee: Robert Fellows >Priority: Minor > -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Updated] (NIFI-6629) Usage alignment issue for invalid CS
[ https://issues.apache.org/jira/browse/NIFI-6629?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Gilman updated NIFI-6629: -- Fix Version/s: 1.10.0 Resolution: Fixed Status: Resolved (was: Patch Available) > Usage alignment issue for invalid CS > > > Key: NIFI-6629 > URL: https://issues.apache.org/jira/browse/NIFI-6629 > Project: Apache NiFi > Issue Type: Sub-task > Components: Core UI >Reporter: Scott Aslan >Assignee: Scott Aslan >Priority: Major > Fix For: 1.10.0 > > Attachments: Screen Shot 2019-09-05 at 4.00.51 PM.png > > Time Spent: 0.5h > Remaining Estimate: 0h > > !Screen Shot 2019-09-05 at 4.00.51 PM.png! > > It looks like the width of the link is incorrect for the CS name and is > causing the text to appear beneath the icon. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Updated] (NIFI-6652) UI - Fix overflowing text in variables dialog
[ https://issues.apache.org/jira/browse/NIFI-6652?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Gilman updated NIFI-6652: -- Attachment: Screen Shot 2019-09-11 at 2.19.18 PM.png > UI - Fix overflowing text in variables dialog > - > > Key: NIFI-6652 > URL: https://issues.apache.org/jira/browse/NIFI-6652 > Project: Apache NiFi > Issue Type: Bug > Components: Core UI >Reporter: Matt Gilman >Priority: Major > Attachments: Screen Shot 2019-09-11 at 2.19.18 PM.png > > > When the name of a referencing component overflows the available area, the > layout breaks. Need to update and add ellipsis or scrolling where appropriate. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Created] (NIFI-6652) UI - Fix overflowing text in variables dialog
Matt Gilman created NIFI-6652: - Summary: UI - Fix overflowing text in variables dialog Key: NIFI-6652 URL: https://issues.apache.org/jira/browse/NIFI-6652 Project: Apache NiFi Issue Type: Bug Components: Core UI Reporter: Matt Gilman When the name of a referencing component overflows the available area, the layout breaks. Need to update and add ellipsis or scrolling where appropriate. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (NIFI-6629) Usage alignment issue for invalid CS
[ https://issues.apache.org/jira/browse/NIFI-6629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16927922#comment-16927922 ] ASF subversion and git services commented on NIFI-6629: --- Commit c187292fd972f3e5ccd8100f5b6debba15cfd25c in nifi's branch refs/heads/master from Scott Aslan [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=c187292 ] [NIFI-6629] calc correct width for usage ellipsis This closes #3708 > Usage alignment issue for invalid CS > > > Key: NIFI-6629 > URL: https://issues.apache.org/jira/browse/NIFI-6629 > Project: Apache NiFi > Issue Type: Sub-task > Components: Core UI >Reporter: Scott Aslan >Assignee: Scott Aslan >Priority: Major > Attachments: Screen Shot 2019-09-05 at 4.00.51 PM.png > > Time Spent: 10m > Remaining Estimate: 0h > > !Screen Shot 2019-09-05 at 4.00.51 PM.png! > > It looks like the width of the link is incorrect for the CS name and is > causing the text to appear beneath the icon. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[GitHub] [nifi] mcgilman commented on issue #3708: [NIFI-6629] calc correct width for usage ellipsis
mcgilman commented on issue #3708: [NIFI-6629] calc correct width for usage ellipsis URL: https://github.com/apache/nifi/pull/3708#issuecomment-530522371 Thanks @scottyaslan! This has been merged to master. Thanks @rfellows for the review. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [nifi] asfgit closed pull request #3708: [NIFI-6629] calc correct width for usage ellipsis
asfgit closed pull request #3708: [NIFI-6629] calc correct width for usage ellipsis URL: https://github.com/apache/nifi/pull/3708 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [nifi] rfellows opened a new pull request #3721: NIFI-6634 - Indicate variables are no longer recommended and favor parameters
rfellows opened a new pull request #3721: NIFI-6634 - Indicate variables are no
longer recommended and favor parameters
URL: https://github.com/apache/nifi/pull/3721
Thank you for submitting a contribution to Apache NiFi.
Please provide a short description of the PR here:
Description of PR
Updates the message on the Variables dialog to indicate that Parameters are
now preferred. There is also a link to the documentation for more info on
Parameters.
This PR assumes that the documentation for parameters will be available in
the Users Guide at this location:
`nifi-docs/html/user-guide.html#Parameters`
That is being accounted for in another issue:
https://issues.apache.org/jira/browse/NIFI-6558
In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:
### For all changes:
- [X] Is there a JIRA ticket associated with this PR? Is it referenced
in the commit message?
- [X] Does your PR title start with **NIFI-** where is the JIRA
number you are trying to resolve? Pay particular attention to the hyphen "-"
character.
- [X] Has your PR been rebased against the latest commit within the target
branch (typically `master`)?
- [X] Is your initial contribution a single, squashed commit? _Additional
commits in response to PR reviewer feedback should be made on this branch and
pushed to allow change tracking. Do not `squash` or use `--force` when pushing
to allow for clean monitoring of changes._
### For code changes:
- [ ] Have you ensured that the full suite of tests is executed via `mvn
-Pcontrib-check clean install` at the root `nifi` folder?
- [ ] Have you written or updated unit tests to verify your changes?
- [ ] Have you verified that the full build is successful on both JDK 8 and
JDK 11?
- [ ] If adding new dependencies to the code, are these dependencies
licensed in a way that is compatible for inclusion under [ASF
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] If applicable, have you updated the `LICENSE` file, including the main
`LICENSE` file under `nifi-assembly`?
- [ ] If applicable, have you updated the `NOTICE` file, including the main
`NOTICE` file found under `nifi-assembly`?
- [ ] If adding new Properties, have you added `.displayName` in addition to
.name (programmatic access) for each of the new properties?
### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in which
it is rendered?
### Note:
Please ensure that once the PR is submitted, you check travis-ci for build
issues and submit an update to your PR as soon as possible.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Updated] (NIFI-6625) Updating a large number of parameters causes the usage listing alignment to be pushed down
[ https://issues.apache.org/jira/browse/NIFI-6625?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Gilman updated NIFI-6625: -- Resolution: Fixed Status: Resolved (was: Patch Available) > Updating a large number of parameters causes the usage listing alignment to > be pushed down > -- > > Key: NIFI-6625 > URL: https://issues.apache.org/jira/browse/NIFI-6625 > Project: Apache NiFi > Issue Type: Sub-task > Components: Core UI >Reporter: Scott Aslan >Assignee: Scott Aslan >Priority: Major > Fix For: 1.10.0 > > Attachments: Screen Shot 2019-09-05 at 1.46.19 PM.png > > Time Spent: 0.5h > Remaining Estimate: 0h > > !Screen Shot 2019-09-05 at 1.46.19 PM.png! > We probably need to add ellipsis to the csv list of parameter names with a > tooltip to prevent pushing down into the usage listing. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (NIFI-6625) Updating a large number of parameters causes the usage listing alignment to be pushed down
[ https://issues.apache.org/jira/browse/NIFI-6625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16927837#comment-16927837 ] ASF subversion and git services commented on NIFI-6625: --- Commit d9c8d0c5a7814619125332bfb69a72ad27023500 in nifi's branch refs/heads/master from Scott Aslan [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=d9c8d0c ] [NIFI-6625] add ellipsis and tooltip to list of parameters in usage list This closes #3707 > Updating a large number of parameters causes the usage listing alignment to > be pushed down > -- > > Key: NIFI-6625 > URL: https://issues.apache.org/jira/browse/NIFI-6625 > Project: Apache NiFi > Issue Type: Sub-task > Components: Core UI >Reporter: Scott Aslan >Assignee: Scott Aslan >Priority: Major > Fix For: 1.10.0 > > Attachments: Screen Shot 2019-09-05 at 1.46.19 PM.png > > Time Spent: 20m > Remaining Estimate: 0h > > !Screen Shot 2019-09-05 at 1.46.19 PM.png! > We probably need to add ellipsis to the csv list of parameter names with a > tooltip to prevent pushing down into the usage listing. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[GitHub] [nifi] asfgit closed pull request #3707: [NIFI-6625] add ellipsis to list of parameters in usage list
asfgit closed pull request #3707: [NIFI-6625] add ellipsis to list of parameters in usage list URL: https://github.com/apache/nifi/pull/3707 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [nifi] mcgilman commented on issue #3707: [NIFI-6625] add ellipsis to list of parameters in usage list
mcgilman commented on issue #3707: [NIFI-6625] add ellipsis to list of parameters in usage list URL: https://github.com/apache/nifi/pull/3707#issuecomment-530482620 Thanks @scottyaslan! This has been merged to master. Thanks @rfellows for the review! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Resolved] (NIFI-6639) Consistent checkbox and associated field behavior.
[ https://issues.apache.org/jira/browse/NIFI-6639?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Gilman resolved NIFI-6639. --- Fix Version/s: 1.10.0 Resolution: Fixed > Consistent checkbox and associated field behavior. > -- > > Key: NIFI-6639 > URL: https://issues.apache.org/jira/browse/NIFI-6639 > Project: Apache NiFi > Issue Type: Improvement > Components: Core UI >Reporter: Scott Aslan >Assignee: Scott Aslan >Priority: Major > Fix For: 1.10.0 > > Time Spent: 40m > Remaining Estimate: 0h > > The property table editor now clears out the value of the editor and disabled > it when the "Empty string set" checkbox is checked. When that checkbox is > unchecked if we know the previous value we repopulate the field. There is an > opportunity throughout the application to provide a similar UX. > * Create/Edit Parameter > * variable registry > * others??? -- This message was sent by Atlassian Jira (v8.3.2#803003)
[GitHub] [nifi] mcgilman commented on issue #3706: [NIFI-6639] consistent ux for checkboxes and their correspoinding field
mcgilman commented on issue #3706: [NIFI-6639] consistent ux for checkboxes and their correspoinding field URL: https://github.com/apache/nifi/pull/3706#issuecomment-530475329 Thanks @scottyaslan! This has been merged to master. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (NIFI-6639) Consistent checkbox and associated field behavior.
[ https://issues.apache.org/jira/browse/NIFI-6639?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16927818#comment-16927818 ] ASF subversion and git services commented on NIFI-6639: --- Commit 250e1b0297c353086e593ea14b6ee2f5f32eebce in nifi's branch refs/heads/master from Scott Aslan [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=250e1b0 ] [NIFI-6639] consistent ux for checkboxes and their correspoinding field This closes #3706 > Consistent checkbox and associated field behavior. > -- > > Key: NIFI-6639 > URL: https://issues.apache.org/jira/browse/NIFI-6639 > Project: Apache NiFi > Issue Type: Improvement > Components: Core UI >Reporter: Scott Aslan >Assignee: Scott Aslan >Priority: Major > Time Spent: 20m > Remaining Estimate: 0h > > The property table editor now clears out the value of the editor and disabled > it when the "Empty string set" checkbox is checked. When that checkbox is > unchecked if we know the previous value we repopulate the field. There is an > opportunity throughout the application to provide a similar UX. > * Create/Edit Parameter > * variable registry > * others??? -- This message was sent by Atlassian Jira (v8.3.2#803003)
[GitHub] [nifi] asfgit closed pull request #3706: [NIFI-6639] consistent ux for checkboxes and their correspoinding field
asfgit closed pull request #3706: [NIFI-6639] consistent ux for checkboxes and their correspoinding field URL: https://github.com/apache/nifi/pull/3706 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [nifi-minifi] apiri commented on issue #167: MINIFI-424: Centralise Bootstrap config
apiri commented on issue #167: MINIFI-424: Centralise Bootstrap config URL: https://github.com/apache/nifi-minifi/pull/167#issuecomment-530472856 @GCHQ-NiFi hey there, I've been away from work for some family stuff but will scope this out in the near future. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (NIFI-6327) Enhance NIFI Developer documentation to assist in developing NIFI in IDE
[ https://issues.apache.org/jira/browse/NIFI-6327?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16927790#comment-16927790 ] David Sargrad commented on NIFI-6327: - Never mind on the last comment.. I was struggling with a nar file project that had the wrong artifact id... I'm all good now. > Enhance NIFI Developer documentation to assist in developing NIFI in IDE > > > Key: NIFI-6327 > URL: https://issues.apache.org/jira/browse/NIFI-6327 > Project: Apache NiFi > Issue Type: Wish > Components: Core Framework >Affects Versions: 1.9.2 > Environment: LINUX: Centos 7 > ECLIPSE: Oxygen >Reporter: David Sargrad >Priority: Major > Attachments: image-2019-05-29-08-59-30-173.png, > image-2019-05-29-08-59-59-849.png, image-2019-05-29-09-00-39-263.png, > image-2019-05-29-09-01-47-311.png, image-2019-05-29-09-04-24-313.png, > image-2019-05-29-09-06-00-609.png, image-2019-05-31-07-29-48-733.png, > image-2019-05-31-07-30-09-097.png, image-2019-05-31-07-32-03-067.png, > image-2019-05-31-11-13-01-963.png, image-2019-05-31-11-14-01-380.png, > image-2019-05-31-11-33-50-040.png, image-2019-05-31-11-34-48-219.png, > image-2019-05-31-11-36-08-343.png, image-2019-06-01-09-19-46-853.png, > image-2019-06-01-09-32-35-143.png, image-2019-06-01-09-34-28-359.png, > image-2019-06-01-09-35-28-483.png, image-2019-06-01-09-40-38-299.png, > image-2019-06-01-09-42-13-371.png, image-2019-06-01-09-50-46-652.png, > image-2019-06-01-10-08-35-334.png, image-2019-06-01-10-19-30-506.png, > image-2019-06-01-10-20-04-869.png, image-2019-09-11-08-58-20-082.png, > image-2019-09-11-09-00-29-639.png, image-2019-09-11-09-05-05-837.png, > image-2019-09-11-09-52-46-872.png > > > I followed the following instructions to clone and build NIFI: > [https://nifi.apache.org/quickstart.html] > I've successfully built the system, using maven. > > !image-2019-05-29-09-01-47-311.png! > > However its not at all clear how to develop and run the system in an IDE such > as eclipse, or a "recommended IDE". > Currently I'm trying to get the system up and running in eclipse. > > However I'm seeing many problems (see image below). I'm guessing all of these > can be solved with a simple set of configuration commands. > Ultimately adding such details to the documentation would be useful. For now, > I'd appreciate if someone can help with the configuration I am missing. > > !image-2019-05-29-09-00-39-263.png! > > If I click on one of the failed projects, I see the following error. > > !image-2019-05-29-09-04-24-313.png! > > !image-2019-05-29-09-06-00-609.png! > > I've also noticed that many of the pom files show up with an error. The error > is seen below: > !image-2019-05-31-07-29-48-733.png! > > !image-2019-05-31-07-30-09-097.png! > This error shows up on the "parent" tag > !image-2019-05-31-07-32-03-067.png! > > I am using the following version of eclipse, and maven > > !image-2019-05-29-08-59-59-849.png! > > !image-2019-05-29-08-59-30-173.png! -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (NIFI-3956) Correct NiFi docs to remove OldestFlowFileFirstPrioritizer as being listed as default prioritizer.
[ https://issues.apache.org/jira/browse/NIFI-3956?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16927732#comment-16927732 ] Chih Han Yu commented on NIFI-3956: --- Hi, may I ask has this issue be solved or it's not a issue anymore? I still can see OldestFlowFileFirstPrioritizer is default prioritizer on user guide. > Correct NiFi docs to remove OldestFlowFileFirstPrioritizer as being listed as > default prioritizer. > -- > > Key: NIFI-3956 > URL: https://issues.apache.org/jira/browse/NIFI-3956 > Project: Apache NiFi > Issue Type: Bug > Components: Documentation & Website >Affects Versions: 1.2.0 >Reporter: Matthew Clarke >Priority: Minor > Labels: newbie > > The embedded NiFi documentation states in two places that the > OldestFlowFileFirstPrioritizer is the default prioritizer when nod is > selected on a connection. The actual default behavior is to order the data > in such a way as to optimize disk reads. This leads to unexpected behavior > for users who believe the oldest FlowFiles will always get processed first. > It is documented as default in both the "Overview" and User Guide sections of > the docs. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Comment Edited] (NIFI-6609) NiFi Docker can't load some JAR files
[
https://issues.apache.org/jira/browse/NIFI-6609?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16927714#comment-16927714
]
James Sevener edited comment on NIFI-6609 at 9/11/19 3:58 PM:
--
Changing
{code:java}
"${NIFI_HOME}/bin/nifi.sh" run &
{code}
to
{code:java}
"${NIFI_HOME}/bin/nifi.sh" start
{code}
in the start.sh file fixes this
was (Author: jamessevener):
Changing `"${NIFI_HOME}/bin/nifi.sh" run &` to `"${NIFI_HOME}/bin/nifi.sh"
start` in the start.sh file fixes this
> NiFi Docker can't load some JAR files
> -
>
> Key: NIFI-6609
> URL: https://issues.apache.org/jira/browse/NIFI-6609
> Project: Apache NiFi
> Issue Type: Bug
> Components: Docker
>Affects Versions: 1.9.2
> Environment: Docker (tested on Docker Desktop, Windows)
>Reporter: James Sevener
>Priority: Minor
>
> When adding the Splunk Java Logging JAR and dependencies, and introducing the
> appender to the logback.xml config, a ClassNotFound error is produced. This
> config and JARs were pulled directly from a working instance of NiFi running
> on RHEL. Also, a workaround is to set the entry point to /bin/bash and then
> run ./bin/nifi.sh. This then successfully starts and sends logs to Splunk.
>
> Reproduceable by building the dockerfile here:
> [https://github.com/jamessevener/nifi/tree/master/nifi-docker/dockerhub]
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Commented] (NIFI-6609) NiFi Docker can't load some JAR files
[
https://issues.apache.org/jira/browse/NIFI-6609?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16927714#comment-16927714
]
James Sevener commented on NIFI-6609:
-
Changing `"${NIFI_HOME}/bin/nifi.sh" run &` to `"${NIFI_HOME}/bin/nifi.sh"
start` in the start.sh file fixes this
> NiFi Docker can't load some JAR files
> -
>
> Key: NIFI-6609
> URL: https://issues.apache.org/jira/browse/NIFI-6609
> Project: Apache NiFi
> Issue Type: Bug
> Components: Docker
>Affects Versions: 1.9.2
> Environment: Docker (tested on Docker Desktop, Windows)
>Reporter: James Sevener
>Priority: Minor
>
> When adding the Splunk Java Logging JAR and dependencies, and introducing the
> appender to the logback.xml config, a ClassNotFound error is produced. This
> config and JARs were pulled directly from a working instance of NiFi running
> on RHEL. Also, a workaround is to set the entry point to /bin/bash and then
> run ./bin/nifi.sh. This then successfully starts and sends logs to Splunk.
>
> Reproduceable by building the dockerfile here:
> [https://github.com/jamessevener/nifi/tree/master/nifi-docker/dockerhub]
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Resolved] (NIFI-6567) HandleHttpRequest does not shutdown HTTP server in some circumstances
[ https://issues.apache.org/jira/browse/NIFI-6567?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Wicks resolved NIFI-6567. --- Resolution: Fixed PR Merged and Closed. > HandleHttpRequest does not shutdown HTTP server in some circumstances > - > > Key: NIFI-6567 > URL: https://issues.apache.org/jira/browse/NIFI-6567 > Project: Apache NiFi > Issue Type: Bug > Components: Extensions >Reporter: Peter Wicks >Assignee: Peter Wicks >Priority: Minor > Time Spent: 1h 10m > Remaining Estimate: 0h > > (Dependent on NIFI-6562) > If multiple cluster nodes are running on a single host, then there will be > port conflicts in HandleHttpRequest. To avoid this users may choose to run > in Primary Only scheduling mode. > If there is a primary node change, the processor does not properly shutdown > the old Http server instance before the new primary node tries to startup > it's own local copy. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (NIFI-6567) HandleHttpRequest does not shutdown HTTP server in some circumstances
[ https://issues.apache.org/jira/browse/NIFI-6567?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16927690#comment-16927690 ] ASF subversion and git services commented on NIFI-6567: --- Commit 758035b964c213649742b9072a5c3f4b057c63c3 in nifi's branch refs/heads/master from Hsin-Ying Lee [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=758035b ] NIFI-6567 HandleHttpRequest does not shutdown HTTP server in some cir… (#3673) * NIFI-6567 HandleHttpRequest does not shutdown HTTP server in some circumstances signed-off by: Peter Wicks > HandleHttpRequest does not shutdown HTTP server in some circumstances > - > > Key: NIFI-6567 > URL: https://issues.apache.org/jira/browse/NIFI-6567 > Project: Apache NiFi > Issue Type: Bug > Components: Extensions >Reporter: Peter Wicks >Assignee: Peter Wicks >Priority: Minor > Time Spent: 1h 10m > Remaining Estimate: 0h > > (Dependent on NIFI-6562) > If multiple cluster nodes are running on a single host, then there will be > port conflicts in HandleHttpRequest. To avoid this users may choose to run > in Primary Only scheduling mode. > If there is a primary node change, the processor does not properly shutdown > the old Http server instance before the new primary node tries to startup > it's own local copy. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (NIFI-6567) HandleHttpRequest does not shutdown HTTP server in some circumstances
[ https://issues.apache.org/jira/browse/NIFI-6567?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16927689#comment-16927689 ] ASF subversion and git services commented on NIFI-6567: --- Commit 758035b964c213649742b9072a5c3f4b057c63c3 in nifi's branch refs/heads/master from Hsin-Ying Lee [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=758035b ] NIFI-6567 HandleHttpRequest does not shutdown HTTP server in some cir… (#3673) * NIFI-6567 HandleHttpRequest does not shutdown HTTP server in some circumstances signed-off by: Peter Wicks > HandleHttpRequest does not shutdown HTTP server in some circumstances > - > > Key: NIFI-6567 > URL: https://issues.apache.org/jira/browse/NIFI-6567 > Project: Apache NiFi > Issue Type: Bug > Components: Extensions >Reporter: Peter Wicks >Assignee: Peter Wicks >Priority: Minor > Time Spent: 1h 10m > Remaining Estimate: 0h > > (Dependent on NIFI-6562) > If multiple cluster nodes are running on a single host, then there will be > port conflicts in HandleHttpRequest. To avoid this users may choose to run > in Primary Only scheduling mode. > If there is a primary node change, the processor does not properly shutdown > the old Http server instance before the new primary node tries to startup > it's own local copy. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[GitHub] [nifi] patricker merged pull request #3673: NIFI-6567 HandleHttpRequest does not shutdown HTTP server in some cir…
patricker merged pull request #3673: NIFI-6567 HandleHttpRequest does not shutdown HTTP server in some cir… URL: https://github.com/apache/nifi/pull/3673 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [nifi] patricker commented on issue #3673: NIFI-6567 HandleHttpRequest does not shutdown HTTP server in some cir…
patricker commented on issue #3673: NIFI-6567 HandleHttpRequest does not shutdown HTTP server in some cir… URL: https://github.com/apache/nifi/pull/3673#issuecomment-530433604 +1 LG This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [nifi] mattyb149 commented on a change in pull request #3684: NIFI-6295: Refactored NiFiRecordSerDe to handle nested complex types
mattyb149 commented on a change in pull request #3684: NIFI-6295: Refactored
NiFiRecordSerDe to handle nested complex types
URL: https://github.com/apache/nifi/pull/3684#discussion_r323303903
##
File path:
nifi-nar-bundles/nifi-hive-bundle/nifi-hive3-processors/src/main/java/org/apache/hive/streaming/NiFiRecordSerDe.java
##
@@ -187,152 +182,153 @@ private Object extractCurrentField(Record record,
RecordField field, TypeInfo fi
}
switch (primitiveCategory) {
case BYTE:
-Integer bIntValue = record.getAsInt(fieldName);
-val = bIntValue == null ? null : bIntValue.byteValue();
+Integer bIntValue =
DataTypeUtils.toInteger(fieldValue, fieldName);
+val = bIntValue.byteValue();
break;
case SHORT:
-Integer sIntValue = record.getAsInt(fieldName);
-val = sIntValue == null ? null :
sIntValue.shortValue();
+Integer sIntValue =
DataTypeUtils.toInteger(fieldValue, fieldName);
+val = sIntValue.shortValue();
break;
case INT:
-val = record.getAsInt(fieldName);
+val = DataTypeUtils.toInteger(fieldValue, fieldName);
break;
case LONG:
-val = record.getAsLong(fieldName);
+val = DataTypeUtils.toLong(fieldValue, fieldName);
break;
case BOOLEAN:
-val = record.getAsBoolean(fieldName);
+val = DataTypeUtils.toBoolean(fieldValue, fieldName);
break;
case FLOAT:
-val = record.getAsFloat(fieldName);
+val = DataTypeUtils.toFloat(fieldValue, fieldName);
break;
case DOUBLE:
-val = record.getAsDouble(fieldName);
+val = DataTypeUtils.toDouble(fieldValue, fieldName);
break;
case STRING:
case VARCHAR:
case CHAR:
-val = record.getAsString(fieldName);
+val = DataTypeUtils.toString(fieldValue, fieldName);
break;
case BINARY:
-Object[] array = record.getAsArray(fieldName);
-if (array == null) {
-return null;
+final ArrayDataType arrayDataType;
+if(fieldValue instanceof String) {
+// Treat this as an array of bytes
+arrayDataType = (ArrayDataType)
RecordFieldType.ARRAY.getArrayDataType(RecordFieldType.BYTE.getDataType());
+} else {
+arrayDataType = (ArrayDataType) fieldDataType;
}
+Object[] array = DataTypeUtils.toArray(fieldValue,
fieldName, arrayDataType.getElementType());
val = AvroTypeUtil.convertByteArray(array).array();
break;
case DATE:
-Date d = record.getAsDate(fieldName,
field.getDataType().getFormat());
-if(d != null) {
-org.apache.hadoop.hive.common.type.Date hiveDate =
new org.apache.hadoop.hive.common.type.Date();
-hiveDate.setTimeInMillis(d.getTime());
-val = hiveDate;
-} else {
-val = null;
-}
+Date d = DataTypeUtils.toDate(fieldValue, () ->
DataTypeUtils.getDateFormat(fieldDataType.getFormat()), fieldName);
+org.apache.hadoop.hive.common.type.Date hiveDate = new
org.apache.hadoop.hive.common.type.Date();
+hiveDate.setTimeInMillis(d.getTime());
+val = hiveDate;
break;
// ORC doesn't currently handle TIMESTAMPLOCALTZ
case TIMESTAMP:
-Timestamp ts =
DataTypeUtils.toTimestamp(record.getValue(fieldName), () ->
DataTypeUtils.getDateFormat(field.getDataType().getFormat()), fieldName);
-if(ts != null) {
-// Convert to Hive's Timestamp type
-org.apache.hadoop.hive.common.type.Timestamp
hivetimestamp = new org.apache.hadoop.hive.common.type.Timestamp();
-hivetimestamp.setTimeInMillis(ts.getTime(),
ts.getNanos());
-
[jira] [Commented] (NIFI-6642) JsonPath support for adding array element
[ https://issues.apache.org/jira/browse/NIFI-6642?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16927648#comment-16927648 ] ASF subversion and git services commented on NIFI-6642: --- Commit 9ec6314687dfc78ff28566770777e5ec0544f238 in nifi's branch refs/heads/master from mans2singh [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=9ec6314 ] NIFI-6642 - JsonPath support for adding array element NIFI-6642 - Updated json file used in tests Clarified function doc Signed-off-by: Matthew Burgess This closes #3711 > JsonPath support for adding array element > - > > Key: NIFI-6642 > URL: https://issues.apache.org/jira/browse/NIFI-6642 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Affects Versions: 1.9.2 > Environment: All >Reporter: Mans Singh >Assignee: Mans Singh >Priority: Minor > Labels: add, array, json, jsonpath > Fix For: 1.10.0 > > Original Estimate: 8h > Time Spent: 0.5h > Remaining Estimate: 7.5h > -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (NIFI-6642) JsonPath support for adding array element
[ https://issues.apache.org/jira/browse/NIFI-6642?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16927649#comment-16927649 ] ASF subversion and git services commented on NIFI-6642: --- Commit 9ec6314687dfc78ff28566770777e5ec0544f238 in nifi's branch refs/heads/master from mans2singh [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=9ec6314 ] NIFI-6642 - JsonPath support for adding array element NIFI-6642 - Updated json file used in tests Clarified function doc Signed-off-by: Matthew Burgess This closes #3711 > JsonPath support for adding array element > - > > Key: NIFI-6642 > URL: https://issues.apache.org/jira/browse/NIFI-6642 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Affects Versions: 1.9.2 > Environment: All >Reporter: Mans Singh >Assignee: Mans Singh >Priority: Minor > Labels: add, array, json, jsonpath > Fix For: 1.10.0 > > Original Estimate: 8h > Time Spent: 0.5h > Remaining Estimate: 7.5h > -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Resolved] (NIFI-6642) JsonPath support for adding array element
[ https://issues.apache.org/jira/browse/NIFI-6642?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Burgess resolved NIFI-6642. Resolution: Fixed > JsonPath support for adding array element > - > > Key: NIFI-6642 > URL: https://issues.apache.org/jira/browse/NIFI-6642 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Affects Versions: 1.9.2 > Environment: All >Reporter: Mans Singh >Assignee: Mans Singh >Priority: Minor > Labels: add, array, json, jsonpath > Fix For: 1.10.0 > > Original Estimate: 8h > Time Spent: 40m > Remaining Estimate: 7h 20m > -- This message was sent by Atlassian Jira (v8.3.2#803003)
[GitHub] [nifi] mattyb149 commented on issue #3711: NIFI-6642 - JsonPath support for adding array element
mattyb149 commented on issue #3711: NIFI-6642 - JsonPath support for adding array element URL: https://github.com/apache/nifi/pull/3711#issuecomment-530427251 +1 LGTM, ran contrib-check and a few tests on a live NiFi instance, verified the doc looks good (and added the word `scalar` for clarity after I verified the value must be scalar). Thanks for this addition! Merging to master This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [nifi] asfgit closed pull request #3711: NIFI-6642 - JsonPath support for adding array element
asfgit closed pull request #3711: NIFI-6642 - JsonPath support for adding array element URL: https://github.com/apache/nifi/pull/3711 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [nifi] mattyb149 commented on a change in pull request #3711: NIFI-6642 - JsonPath support for adding array element
mattyb149 commented on a change in pull request #3711: NIFI-6642 - JsonPath support for adding array element URL: https://github.com/apache/nifi/pull/3711#discussion_r323295151 ## File path: nifi-docs/src/main/asciidoc/expression-language-guide.adoc ## @@ -1698,6 +1698,61 @@ form of the updated JSON.# An empty subject value or a subject value with an invalid JSON document results in an exception bulletin. +[.function] +=== jsonPathAdd + +*Description*: [.description]#The `jsonPathAdd` function adds a value to an array at the specified JsonPath on Review comment: This should probably say `adds a scalar value`, I don't think it's possible to insert a JSON object right? I can add the word `scalar` during merge if you agree This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [nifi] matthew-formosa-gig opened a new pull request #3720: Nifi 6647 - Upgrading Apache Ignite version to 2.7.5
matthew-formosa-gig opened a new pull request #3720: Nifi 6647 - Upgrading
Apache Ignite version to 2.7.5
URL: https://github.com/apache/nifi/pull/3720
Description of PR
The version of Apache Ignite is currently set to 1.6.0 which is now over 3
years old. Upgrading to the latest version, that is 2.7.5.
### For all changes:
- [X] Is there a JIRA ticket associated with this PR? Is it referenced
in the commit message?
- [X] Does your PR title start with **NIFI-** where is the JIRA
number you are trying to resolve? Pay particular attention to the hyphen "-"
character.
- [X] Has your PR been rebased against the latest commit within the target
branch (typically `master`)?
- [X] Is your initial contribution a single, squashed commit? _Additional
commits in response to PR reviewer feedback should be made on this branch and
pushed to allow change tracking. Do not `squash` or use `--force` when pushing
to allow for clean monitoring of changes._
### For code changes:
- [X] Have you ensured that the full suite of tests is executed via `mvn
-Pcontrib-check clean install` at the root `nifi` folder?
- [X] Have you written or updated unit tests to verify your changes?
- [X] Have you verified that the full build is successful on both JDK 8 and
JDK 11?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Commented] (NIFI-6634) UI - Indicate variable are no longer recommended and favor parameters
[
https://issues.apache.org/jira/browse/NIFI-6634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16927622#comment-16927622
]
Andrew Lim commented on NIFI-6634:
--
[~rfellows] The URL of the section I am adding to the User Guide is:
{{nifi-docs/html/user-guide.html#Parameters}}
> UI - Indicate variable are no longer recommended and favor parameters
> -
>
> Key: NIFI-6634
> URL: https://issues.apache.org/jira/browse/NIFI-6634
> Project: Apache NiFi
> Issue Type: Sub-task
> Components: Core UI
>Reporter: Robert Fellows
>Assignee: Robert Fellows
>Priority: Major
>
> Variables less powerful than parameters. Specifically, they don't support
> sensitive values. On the Variables dialog, this should be conveyed to the
> user to help guide them to use parameters instead.
> One suggestion for wording:
> "Variables are still supported for compatibility purposes but they do not
> allow the same power as Parameters such as support for sensitive parameters.
> Variables will be removed in a later release so please change to using
> parameters."
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Assigned] (NIFI-6634) UI - Indicate variable are no longer recommended and favor parameters
[ https://issues.apache.org/jira/browse/NIFI-6634?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Fellows reassigned NIFI-6634: Assignee: Robert Fellows > UI - Indicate variable are no longer recommended and favor parameters > - > > Key: NIFI-6634 > URL: https://issues.apache.org/jira/browse/NIFI-6634 > Project: Apache NiFi > Issue Type: Sub-task > Components: Core UI >Reporter: Robert Fellows >Assignee: Robert Fellows >Priority: Major > > Variables less powerful than parameters. Specifically, they don't support > sensitive values. On the Variables dialog, this should be conveyed to the > user to help guide them to use parameters instead. > One suggestion for wording: > "Variables are still supported for compatibility purposes but they do not > allow the same power as Parameters such as support for sensitive parameters. > Variables will be removed in a later release so please change to using > parameters." -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (NIFI-3388) Provide encrypted repository implementations
[
https://issues.apache.org/jira/browse/NIFI-3388?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16927593#comment-16927593
]
Deepak commented on NIFI-3388:
--
Hi Andy,
Any updates on this by when it will be available?
Thanks
Deepak
> Provide encrypted repository implementations
>
>
> Key: NIFI-3388
> URL: https://issues.apache.org/jira/browse/NIFI-3388
> Project: Apache NiFi
> Issue Type: Epic
> Components: Core Framework
>Affects Versions: 1.1.1
>Reporter: Andy LoPresto
>Assignee: Andy LoPresto
>Priority: Critical
> Labels: data-integrity, encryption, repository, security
>
> Apache NiFi secures data within the application but the various
> *repositories* -- content, provenance, flowfile (aka attribute), and to a
> lesser extent bulletin, counter, component status, and log -- are stored
> unencrypted on disk. Thus far, OS-level access control policies and full disk
> encryption (FDE) have been recommended to secure these repositories. However,
> the underlying raw data can be viewed, and possibly manipulated, by a user
> with access to the repository files.
> With additional focus on data security (confidentiality, integrity, *and*
> authentication), especially as more users intend to deploy NiFi on
> third-party provisioned hardware and operating systems (AWS, Azure, etc.),
> further steps should be taken to secure the repository data which NiFi writes
> and reads.
> Each of the repository implementations adheres to an interface definition:
> * Content: {{ContentRepository}}
> ** {{FileSystemRepository}} *
> ** {{VolatileContentRepository}}
> ** {{MockContentRepository}}
> * Provenance: {{ProvenanceRepository extends ProvenanceEventRepository}}
> ** {{PersistentProvenanceRepository}} (to be deprecated via
> [NIFI-3356|https://issues.apache.org/jira/browse/NIFI-3356])*
> ** {{WriteAheadProvenanceRepository}} (to be introduced via NIFI-3356) *
> ** {{VolatileProvenanceRepository}}
> ** {{MockProvenanceRepository}}
> * Flowfile: {{FlowFileRepository}} / {{FlowFileEventRepository}}
> ** {{WriteAheadFlowFileRepository implements FlowFileRepository}} *
> ** {{VolatileFlowFileRepository implements FlowFileRepository}}
> ** {{MockFlowFileRepository implements FlowFileRepository}}
> ** {{RingBufferEventRepository implements FlowFileEventRepository}}
> The bulletin, counter, component status, and log repositories currently have
> only *volatile* implementations, and are not addressed in this ticket. The
> repository implementations noted above with an asterisk will have new
> implementations provided of the form
> {{EncryptedWriteAheadFlowFileRepository}}, extending the behavior of the
> existing repository and adding transparent encryption/decryption on
> serialization, following the existing interface contracts and thus invisible
> to the higher level code delegating these operations, aside from
> configuration requirements.
> There are substantial concerns to address in this approach.
> * Should the various repositories all be required to have the same
> encrypted/plaintext status (i.e. can they be encrypted independently)?
> * Should all encrypted repositories use the same encryption key, or should it
> be segmented by repository?
> * If a content or provenance repository has multiple shards, do they all
> require the same level of encryption? If not, can "sensitive" records be
> routed to an encrypted repository and "normal" records to a plaintext
> repository for performance reasons?
> * Can a plaintext repository have encryption enabled at any time? Can an
> encrypted repository have decryption removed?
> * Performance impact on reading and writing from the repositories is not yet
> captured
> ** The provenance repository requires many fast writes and reads during high
> performance and query operations
> ** The flowfile repository requires many fast writes and reads
> ** The content repository requires fewer reads and writes, but the current
> content repository stores multiple flowfile contents in the same ["sections"
> of the
> "containers"|https://nifi.apache.org/docs/nifi-docs/html/nifi-in-depth.html#deeper-view-content-claim]
> that make up the repository, so content claims will need to be encrypted
> separately to allow random-access seek and retrieve (i.e. if a 10 byte claim
> and a 10 GB claim are stored in the same section, the 10 B claim must be
> retrievable without reading 10 GB into memory to decrypt)
> * The provenance repository uses GZip compression to improve the use of disk
> space. Compression over encrypted data will yield close to zero improvement
> (as encryption intentionally generates near-random output, which means
> pattern recognition/entropy removal will have no effect), and compression
> before
[jira] [Comment Edited] (NIFI-6327) Enhance NIFI Developer documentation to assist in developing NIFI in IDE
[ https://issues.apache.org/jira/browse/NIFI-6327?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16927587#comment-16927587 ] David Sargrad edited comment on NIFI-6327 at 9/11/19 1:53 PM: -- Part of my problem is that intellij is loading the class file from the target folder of the source project.. not from the NAR file itself. So the breakpoints aren't lining up. !image-2019-09-11-09-52-46-872.png! was (Author: dsargrad): Part of my problem is that intellij is loading the class file from the target folder of the source project.. not from the NAR file itself. !image-2019-09-11-09-52-46-872.png! > Enhance NIFI Developer documentation to assist in developing NIFI in IDE > > > Key: NIFI-6327 > URL: https://issues.apache.org/jira/browse/NIFI-6327 > Project: Apache NiFi > Issue Type: Wish > Components: Core Framework >Affects Versions: 1.9.2 > Environment: LINUX: Centos 7 > ECLIPSE: Oxygen >Reporter: David Sargrad >Priority: Major > Attachments: image-2019-05-29-08-59-30-173.png, > image-2019-05-29-08-59-59-849.png, image-2019-05-29-09-00-39-263.png, > image-2019-05-29-09-01-47-311.png, image-2019-05-29-09-04-24-313.png, > image-2019-05-29-09-06-00-609.png, image-2019-05-31-07-29-48-733.png, > image-2019-05-31-07-30-09-097.png, image-2019-05-31-07-32-03-067.png, > image-2019-05-31-11-13-01-963.png, image-2019-05-31-11-14-01-380.png, > image-2019-05-31-11-33-50-040.png, image-2019-05-31-11-34-48-219.png, > image-2019-05-31-11-36-08-343.png, image-2019-06-01-09-19-46-853.png, > image-2019-06-01-09-32-35-143.png, image-2019-06-01-09-34-28-359.png, > image-2019-06-01-09-35-28-483.png, image-2019-06-01-09-40-38-299.png, > image-2019-06-01-09-42-13-371.png, image-2019-06-01-09-50-46-652.png, > image-2019-06-01-10-08-35-334.png, image-2019-06-01-10-19-30-506.png, > image-2019-06-01-10-20-04-869.png, image-2019-09-11-08-58-20-082.png, > image-2019-09-11-09-00-29-639.png, image-2019-09-11-09-05-05-837.png, > image-2019-09-11-09-52-46-872.png > > > I followed the following instructions to clone and build NIFI: > [https://nifi.apache.org/quickstart.html] > I've successfully built the system, using maven. > > !image-2019-05-29-09-01-47-311.png! > > However its not at all clear how to develop and run the system in an IDE such > as eclipse, or a "recommended IDE". > Currently I'm trying to get the system up and running in eclipse. > > However I'm seeing many problems (see image below). I'm guessing all of these > can be solved with a simple set of configuration commands. > Ultimately adding such details to the documentation would be useful. For now, > I'd appreciate if someone can help with the configuration I am missing. > > !image-2019-05-29-09-00-39-263.png! > > If I click on one of the failed projects, I see the following error. > > !image-2019-05-29-09-04-24-313.png! > > !image-2019-05-29-09-06-00-609.png! > > I've also noticed that many of the pom files show up with an error. The error > is seen below: > !image-2019-05-31-07-29-48-733.png! > > !image-2019-05-31-07-30-09-097.png! > This error shows up on the "parent" tag > !image-2019-05-31-07-32-03-067.png! > > I am using the following version of eclipse, and maven > > !image-2019-05-29-08-59-59-849.png! > > !image-2019-05-29-08-59-30-173.png! -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (NIFI-6327) Enhance NIFI Developer documentation to assist in developing NIFI in IDE
[ https://issues.apache.org/jira/browse/NIFI-6327?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16927587#comment-16927587 ] David Sargrad commented on NIFI-6327: - Part of my problem is that intellij is loading the class file from the target folder of the source project.. not from the NAR file itself. !image-2019-09-11-09-52-46-872.png! > Enhance NIFI Developer documentation to assist in developing NIFI in IDE > > > Key: NIFI-6327 > URL: https://issues.apache.org/jira/browse/NIFI-6327 > Project: Apache NiFi > Issue Type: Wish > Components: Core Framework >Affects Versions: 1.9.2 > Environment: LINUX: Centos 7 > ECLIPSE: Oxygen >Reporter: David Sargrad >Priority: Major > Attachments: image-2019-05-29-08-59-30-173.png, > image-2019-05-29-08-59-59-849.png, image-2019-05-29-09-00-39-263.png, > image-2019-05-29-09-01-47-311.png, image-2019-05-29-09-04-24-313.png, > image-2019-05-29-09-06-00-609.png, image-2019-05-31-07-29-48-733.png, > image-2019-05-31-07-30-09-097.png, image-2019-05-31-07-32-03-067.png, > image-2019-05-31-11-13-01-963.png, image-2019-05-31-11-14-01-380.png, > image-2019-05-31-11-33-50-040.png, image-2019-05-31-11-34-48-219.png, > image-2019-05-31-11-36-08-343.png, image-2019-06-01-09-19-46-853.png, > image-2019-06-01-09-32-35-143.png, image-2019-06-01-09-34-28-359.png, > image-2019-06-01-09-35-28-483.png, image-2019-06-01-09-40-38-299.png, > image-2019-06-01-09-42-13-371.png, image-2019-06-01-09-50-46-652.png, > image-2019-06-01-10-08-35-334.png, image-2019-06-01-10-19-30-506.png, > image-2019-06-01-10-20-04-869.png, image-2019-09-11-08-58-20-082.png, > image-2019-09-11-09-00-29-639.png, image-2019-09-11-09-05-05-837.png, > image-2019-09-11-09-52-46-872.png > > > I followed the following instructions to clone and build NIFI: > [https://nifi.apache.org/quickstart.html] > I've successfully built the system, using maven. > > !image-2019-05-29-09-01-47-311.png! > > However its not at all clear how to develop and run the system in an IDE such > as eclipse, or a "recommended IDE". > Currently I'm trying to get the system up and running in eclipse. > > However I'm seeing many problems (see image below). I'm guessing all of these > can be solved with a simple set of configuration commands. > Ultimately adding such details to the documentation would be useful. For now, > I'd appreciate if someone can help with the configuration I am missing. > > !image-2019-05-29-09-00-39-263.png! > > If I click on one of the failed projects, I see the following error. > > !image-2019-05-29-09-04-24-313.png! > > !image-2019-05-29-09-06-00-609.png! > > I've also noticed that many of the pom files show up with an error. The error > is seen below: > !image-2019-05-31-07-29-48-733.png! > > !image-2019-05-31-07-30-09-097.png! > This error shows up on the "parent" tag > !image-2019-05-31-07-32-03-067.png! > > I am using the following version of eclipse, and maven > > !image-2019-05-29-08-59-59-849.png! > > !image-2019-05-29-08-59-30-173.png! -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Updated] (NIFI-6327) Enhance NIFI Developer documentation to assist in developing NIFI in IDE
[ https://issues.apache.org/jira/browse/NIFI-6327?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Sargrad updated NIFI-6327: Attachment: image-2019-09-11-09-52-46-872.png > Enhance NIFI Developer documentation to assist in developing NIFI in IDE > > > Key: NIFI-6327 > URL: https://issues.apache.org/jira/browse/NIFI-6327 > Project: Apache NiFi > Issue Type: Wish > Components: Core Framework >Affects Versions: 1.9.2 > Environment: LINUX: Centos 7 > ECLIPSE: Oxygen >Reporter: David Sargrad >Priority: Major > Attachments: image-2019-05-29-08-59-30-173.png, > image-2019-05-29-08-59-59-849.png, image-2019-05-29-09-00-39-263.png, > image-2019-05-29-09-01-47-311.png, image-2019-05-29-09-04-24-313.png, > image-2019-05-29-09-06-00-609.png, image-2019-05-31-07-29-48-733.png, > image-2019-05-31-07-30-09-097.png, image-2019-05-31-07-32-03-067.png, > image-2019-05-31-11-13-01-963.png, image-2019-05-31-11-14-01-380.png, > image-2019-05-31-11-33-50-040.png, image-2019-05-31-11-34-48-219.png, > image-2019-05-31-11-36-08-343.png, image-2019-06-01-09-19-46-853.png, > image-2019-06-01-09-32-35-143.png, image-2019-06-01-09-34-28-359.png, > image-2019-06-01-09-35-28-483.png, image-2019-06-01-09-40-38-299.png, > image-2019-06-01-09-42-13-371.png, image-2019-06-01-09-50-46-652.png, > image-2019-06-01-10-08-35-334.png, image-2019-06-01-10-19-30-506.png, > image-2019-06-01-10-20-04-869.png, image-2019-09-11-08-58-20-082.png, > image-2019-09-11-09-00-29-639.png, image-2019-09-11-09-05-05-837.png, > image-2019-09-11-09-52-46-872.png > > > I followed the following instructions to clone and build NIFI: > [https://nifi.apache.org/quickstart.html] > I've successfully built the system, using maven. > > !image-2019-05-29-09-01-47-311.png! > > However its not at all clear how to develop and run the system in an IDE such > as eclipse, or a "recommended IDE". > Currently I'm trying to get the system up and running in eclipse. > > However I'm seeing many problems (see image below). I'm guessing all of these > can be solved with a simple set of configuration commands. > Ultimately adding such details to the documentation would be useful. For now, > I'd appreciate if someone can help with the configuration I am missing. > > !image-2019-05-29-09-00-39-263.png! > > If I click on one of the failed projects, I see the following error. > > !image-2019-05-29-09-04-24-313.png! > > !image-2019-05-29-09-06-00-609.png! > > I've also noticed that many of the pom files show up with an error. The error > is seen below: > !image-2019-05-31-07-29-48-733.png! > > !image-2019-05-31-07-30-09-097.png! > This error shows up on the "parent" tag > !image-2019-05-31-07-32-03-067.png! > > I am using the following version of eclipse, and maven > > !image-2019-05-29-08-59-59-849.png! > > !image-2019-05-29-08-59-30-173.png! -- This message was sent by Atlassian Jira (v8.3.2#803003)
[GitHub] [nifi] BoyillaGanesh commented on issue #3717: NIFI-6610 Disconnected Relationship support of Connect WebSocket processor in Nifi
BoyillaGanesh commented on issue #3717: NIFI-6610 Disconnected Relationship support of Connect WebSocket processor in Nifi URL: https://github.com/apache/nifi/pull/3717#issuecomment-530375691 waiting for the review to be done This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Comment Edited] (NIFI-6327) Enhance NIFI Developer documentation to assist in developing NIFI in IDE
[ https://issues.apache.org/jira/browse/NIFI-6327?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16927539#comment-16927539 ] David Sargrad edited comment on NIFI-6327 at 9/11/19 1:05 PM: -- I'm having a fair amount of success with single step debugging in my custom nifi processors. I use the remote debugging option to connect to the instance of NiFi that I've built from source. As you see I can then set a breakpoint right at the point where the server is started. !image-2019-09-11-08-58-20-082.png! II am struggling to find out the proper way to load my NAR file into my intellij project. I've tried to use the Project Structure Settings as follows. I add another library and then browse to my custom nifi project. In the following picture I've browsed to the trunk of "consume_solace_text" !image-2019-09-11-09-00-29-639.png! This sort of seems to work, but intellij seems to get confused by the .class and source files in that project. I am still able to single-step debug, but I sometimes have to unload and reload the project.. Is there a cleaner way for me to point to the "nar" file that contains my built custom processor !image-2019-09-11-09-05-05-837.png! was (Author: dsargrad): I'm having a fair amount of success with single step debugging in my custom nifi processors. I use the remote debugging option to connect to the instance of NiFi that I've built from source. As you see I can then set a breakpoint right at the point where the server is started. !image-2019-09-11-08-58-20-082.png! II am struggling to find out the proper way to load my NAR file into my intellij project. I've tried to use the Project Structure Settings as follows. I add another library and then browse to my custom nifi project. In the following picture I've browsed to the trunk of "consume_solace_text" !image-2019-09-11-09-00-29-639.png! This sort of seems to work, but intellij seems to get confused by the .class and source files in that project. I am still able to single-step debug, but I sometimes have to unload and reload the project.. Is there a cleaner way for me to point to the "nar" file that contains my built custom processor > Enhance NIFI Developer documentation to assist in developing NIFI in IDE > > > Key: NIFI-6327 > URL: https://issues.apache.org/jira/browse/NIFI-6327 > Project: Apache NiFi > Issue Type: Wish > Components: Core Framework >Affects Versions: 1.9.2 > Environment: LINUX: Centos 7 > ECLIPSE: Oxygen >Reporter: David Sargrad >Priority: Major > Attachments: image-2019-05-29-08-59-30-173.png, > image-2019-05-29-08-59-59-849.png, image-2019-05-29-09-00-39-263.png, > image-2019-05-29-09-01-47-311.png, image-2019-05-29-09-04-24-313.png, > image-2019-05-29-09-06-00-609.png, image-2019-05-31-07-29-48-733.png, > image-2019-05-31-07-30-09-097.png, image-2019-05-31-07-32-03-067.png, > image-2019-05-31-11-13-01-963.png, image-2019-05-31-11-14-01-380.png, > image-2019-05-31-11-33-50-040.png, image-2019-05-31-11-34-48-219.png, > image-2019-05-31-11-36-08-343.png, image-2019-06-01-09-19-46-853.png, > image-2019-06-01-09-32-35-143.png, image-2019-06-01-09-34-28-359.png, > image-2019-06-01-09-35-28-483.png, image-2019-06-01-09-40-38-299.png, > image-2019-06-01-09-42-13-371.png, image-2019-06-01-09-50-46-652.png, > image-2019-06-01-10-08-35-334.png, image-2019-06-01-10-19-30-506.png, > image-2019-06-01-10-20-04-869.png, image-2019-09-11-08-58-20-082.png, > image-2019-09-11-09-00-29-639.png, image-2019-09-11-09-05-05-837.png > > > I followed the following instructions to clone and build NIFI: > [https://nifi.apache.org/quickstart.html] > I've successfully built the system, using maven. > > !image-2019-05-29-09-01-47-311.png! > > However its not at all clear how to develop and run the system in an IDE such > as eclipse, or a "recommended IDE". > Currently I'm trying to get the system up and running in eclipse. > > However I'm seeing many problems (see image below). I'm guessing all of these > can be solved with a simple set of configuration commands. > Ultimately adding such details to the documentation would be useful. For now, > I'd appreciate if someone can help with the configuration I am missing. > > !image-2019-05-29-09-00-39-263.png! > > If I click on one of the failed projects, I see the following error. > > !image-2019-05-29-09-04-24-313.png! > > !image-2019-05-29-09-06-00-609.png! > > I've also noticed that many of the pom files show up with an error. The error > is seen below: > !image-2019-05-31-07-29-48-733.png! > > !image-2019-05-31-07-30-09-097.png! > This error shows up on the "parent" tag > !i
[jira] [Commented] (NIFI-6327) Enhance NIFI Developer documentation to assist in developing NIFI in IDE
[ https://issues.apache.org/jira/browse/NIFI-6327?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16927539#comment-16927539 ] David Sargrad commented on NIFI-6327: - I'm having a fair amount of success with single step debugging in my custom nifi processors. I use the remote debugging option to connect to the instance of NiFi that I've built from source. As you see I can then set a breakpoint right at the point where the server is started. !image-2019-09-11-08-58-20-082.png! II am struggling to find out the proper way to load my NAR file into my intellij project. I've tried to use the Project Structure Settings as follows. I add another library and then browse to my custom nifi project. In the following picture I've browsed to the trunk of "consume_solace_text" !image-2019-09-11-09-00-29-639.png! This sort of seems to work, but intellij seems to get confused by the .class and source files in that project. I am still able to single-step debug, but I sometimes have to unload and reload the project.. Is there a cleaner way for me to point to the "nar" file that contains my built custom processor > Enhance NIFI Developer documentation to assist in developing NIFI in IDE > > > Key: NIFI-6327 > URL: https://issues.apache.org/jira/browse/NIFI-6327 > Project: Apache NiFi > Issue Type: Wish > Components: Core Framework >Affects Versions: 1.9.2 > Environment: LINUX: Centos 7 > ECLIPSE: Oxygen >Reporter: David Sargrad >Priority: Major > Attachments: image-2019-05-29-08-59-30-173.png, > image-2019-05-29-08-59-59-849.png, image-2019-05-29-09-00-39-263.png, > image-2019-05-29-09-01-47-311.png, image-2019-05-29-09-04-24-313.png, > image-2019-05-29-09-06-00-609.png, image-2019-05-31-07-29-48-733.png, > image-2019-05-31-07-30-09-097.png, image-2019-05-31-07-32-03-067.png, > image-2019-05-31-11-13-01-963.png, image-2019-05-31-11-14-01-380.png, > image-2019-05-31-11-33-50-040.png, image-2019-05-31-11-34-48-219.png, > image-2019-05-31-11-36-08-343.png, image-2019-06-01-09-19-46-853.png, > image-2019-06-01-09-32-35-143.png, image-2019-06-01-09-34-28-359.png, > image-2019-06-01-09-35-28-483.png, image-2019-06-01-09-40-38-299.png, > image-2019-06-01-09-42-13-371.png, image-2019-06-01-09-50-46-652.png, > image-2019-06-01-10-08-35-334.png, image-2019-06-01-10-19-30-506.png, > image-2019-06-01-10-20-04-869.png, image-2019-09-11-08-58-20-082.png, > image-2019-09-11-09-00-29-639.png > > > I followed the following instructions to clone and build NIFI: > [https://nifi.apache.org/quickstart.html] > I've successfully built the system, using maven. > > !image-2019-05-29-09-01-47-311.png! > > However its not at all clear how to develop and run the system in an IDE such > as eclipse, or a "recommended IDE". > Currently I'm trying to get the system up and running in eclipse. > > However I'm seeing many problems (see image below). I'm guessing all of these > can be solved with a simple set of configuration commands. > Ultimately adding such details to the documentation would be useful. For now, > I'd appreciate if someone can help with the configuration I am missing. > > !image-2019-05-29-09-00-39-263.png! > > If I click on one of the failed projects, I see the following error. > > !image-2019-05-29-09-04-24-313.png! > > !image-2019-05-29-09-06-00-609.png! > > I've also noticed that many of the pom files show up with an error. The error > is seen below: > !image-2019-05-31-07-29-48-733.png! > > !image-2019-05-31-07-30-09-097.png! > This error shows up on the "parent" tag > !image-2019-05-31-07-32-03-067.png! > > I am using the following version of eclipse, and maven > > !image-2019-05-29-08-59-59-849.png! > > !image-2019-05-29-08-59-30-173.png! -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Updated] (NIFI-6327) Enhance NIFI Developer documentation to assist in developing NIFI in IDE
[ https://issues.apache.org/jira/browse/NIFI-6327?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Sargrad updated NIFI-6327: Attachment: image-2019-09-11-09-00-29-639.png > Enhance NIFI Developer documentation to assist in developing NIFI in IDE > > > Key: NIFI-6327 > URL: https://issues.apache.org/jira/browse/NIFI-6327 > Project: Apache NiFi > Issue Type: Wish > Components: Core Framework >Affects Versions: 1.9.2 > Environment: LINUX: Centos 7 > ECLIPSE: Oxygen >Reporter: David Sargrad >Priority: Major > Attachments: image-2019-05-29-08-59-30-173.png, > image-2019-05-29-08-59-59-849.png, image-2019-05-29-09-00-39-263.png, > image-2019-05-29-09-01-47-311.png, image-2019-05-29-09-04-24-313.png, > image-2019-05-29-09-06-00-609.png, image-2019-05-31-07-29-48-733.png, > image-2019-05-31-07-30-09-097.png, image-2019-05-31-07-32-03-067.png, > image-2019-05-31-11-13-01-963.png, image-2019-05-31-11-14-01-380.png, > image-2019-05-31-11-33-50-040.png, image-2019-05-31-11-34-48-219.png, > image-2019-05-31-11-36-08-343.png, image-2019-06-01-09-19-46-853.png, > image-2019-06-01-09-32-35-143.png, image-2019-06-01-09-34-28-359.png, > image-2019-06-01-09-35-28-483.png, image-2019-06-01-09-40-38-299.png, > image-2019-06-01-09-42-13-371.png, image-2019-06-01-09-50-46-652.png, > image-2019-06-01-10-08-35-334.png, image-2019-06-01-10-19-30-506.png, > image-2019-06-01-10-20-04-869.png, image-2019-09-11-08-58-20-082.png, > image-2019-09-11-09-00-29-639.png > > > I followed the following instructions to clone and build NIFI: > [https://nifi.apache.org/quickstart.html] > I've successfully built the system, using maven. > > !image-2019-05-29-09-01-47-311.png! > > However its not at all clear how to develop and run the system in an IDE such > as eclipse, or a "recommended IDE". > Currently I'm trying to get the system up and running in eclipse. > > However I'm seeing many problems (see image below). I'm guessing all of these > can be solved with a simple set of configuration commands. > Ultimately adding such details to the documentation would be useful. For now, > I'd appreciate if someone can help with the configuration I am missing. > > !image-2019-05-29-09-00-39-263.png! > > If I click on one of the failed projects, I see the following error. > > !image-2019-05-29-09-04-24-313.png! > > !image-2019-05-29-09-06-00-609.png! > > I've also noticed that many of the pom files show up with an error. The error > is seen below: > !image-2019-05-31-07-29-48-733.png! > > !image-2019-05-31-07-30-09-097.png! > This error shows up on the "parent" tag > !image-2019-05-31-07-32-03-067.png! > > I am using the following version of eclipse, and maven > > !image-2019-05-29-08-59-59-849.png! > > !image-2019-05-29-08-59-30-173.png! -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Updated] (NIFI-6327) Enhance NIFI Developer documentation to assist in developing NIFI in IDE
[ https://issues.apache.org/jira/browse/NIFI-6327?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Sargrad updated NIFI-6327: Attachment: image-2019-09-11-08-58-20-082.png > Enhance NIFI Developer documentation to assist in developing NIFI in IDE > > > Key: NIFI-6327 > URL: https://issues.apache.org/jira/browse/NIFI-6327 > Project: Apache NiFi > Issue Type: Wish > Components: Core Framework >Affects Versions: 1.9.2 > Environment: LINUX: Centos 7 > ECLIPSE: Oxygen >Reporter: David Sargrad >Priority: Major > Attachments: image-2019-05-29-08-59-30-173.png, > image-2019-05-29-08-59-59-849.png, image-2019-05-29-09-00-39-263.png, > image-2019-05-29-09-01-47-311.png, image-2019-05-29-09-04-24-313.png, > image-2019-05-29-09-06-00-609.png, image-2019-05-31-07-29-48-733.png, > image-2019-05-31-07-30-09-097.png, image-2019-05-31-07-32-03-067.png, > image-2019-05-31-11-13-01-963.png, image-2019-05-31-11-14-01-380.png, > image-2019-05-31-11-33-50-040.png, image-2019-05-31-11-34-48-219.png, > image-2019-05-31-11-36-08-343.png, image-2019-06-01-09-19-46-853.png, > image-2019-06-01-09-32-35-143.png, image-2019-06-01-09-34-28-359.png, > image-2019-06-01-09-35-28-483.png, image-2019-06-01-09-40-38-299.png, > image-2019-06-01-09-42-13-371.png, image-2019-06-01-09-50-46-652.png, > image-2019-06-01-10-08-35-334.png, image-2019-06-01-10-19-30-506.png, > image-2019-06-01-10-20-04-869.png, image-2019-09-11-08-58-20-082.png > > > I followed the following instructions to clone and build NIFI: > [https://nifi.apache.org/quickstart.html] > I've successfully built the system, using maven. > > !image-2019-05-29-09-01-47-311.png! > > However its not at all clear how to develop and run the system in an IDE such > as eclipse, or a "recommended IDE". > Currently I'm trying to get the system up and running in eclipse. > > However I'm seeing many problems (see image below). I'm guessing all of these > can be solved with a simple set of configuration commands. > Ultimately adding such details to the documentation would be useful. For now, > I'd appreciate if someone can help with the configuration I am missing. > > !image-2019-05-29-09-00-39-263.png! > > If I click on one of the failed projects, I see the following error. > > !image-2019-05-29-09-04-24-313.png! > > !image-2019-05-29-09-06-00-609.png! > > I've also noticed that many of the pom files show up with an error. The error > is seen below: > !image-2019-05-31-07-29-48-733.png! > > !image-2019-05-31-07-30-09-097.png! > This error shows up on the "parent" tag > !image-2019-05-31-07-32-03-067.png! > > I am using the following version of eclipse, and maven > > !image-2019-05-29-08-59-59-849.png! > > !image-2019-05-29-08-59-30-173.png! -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Created] (MINIFICPP-1029) Extensions built in docker builds should be customisable
Arpad Boda created MINIFICPP-1029: - Summary: Extensions built in docker builds should be customisable Key: MINIFICPP-1029 URL: https://issues.apache.org/jira/browse/MINIFICPP-1029 Project: Apache NiFi MiNiFi C++ Issue Type: Improvement Reporter: Arpad Boda Fix For: 0.8.0 Currently docker builds build a predefined set of extensions. We should be able to provide a way of customising this without manually editing dockerfile. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Resolved] (MINIFICPP-959) Review librdkafka thread safety
[ https://issues.apache.org/jira/browse/MINIFICPP-959?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nghia Le resolved MINIFICPP-959. Resolution: Fixed > Review librdkafka thread safety > --- > > Key: MINIFICPP-959 > URL: https://issues.apache.org/jira/browse/MINIFICPP-959 > Project: Apache NiFi MiNiFi C++ > Issue Type: Bug >Reporter: Daniel Bakai >Assignee: Nghia Le >Priority: Major > Time Spent: 2h > Remaining Estimate: 0h > > I don't think librdkafka is doing what it is supposed to around the > KafkaLease and the atomic spinlock combined with mutexes. > It is thread safe in a way that resources are only accessed by one thread at > a time, but the behaviour otherwise is I think not what was intended > (multiple threads can initialize a KafkaConnection after each other, for > example). -- This message was sent by Atlassian Jira (v8.3.2#803003)
[GitHub] [nifi-minifi] gruselglatz commented on issue #167: MINIFI-424: Centralise Bootstrap config
gruselglatz commented on issue #167: MINIFI-424: Centralise Bootstrap config URL: https://github.com/apache/nifi-minifi/pull/167#issuecomment-530312018 @GCHQ-NiFi Oh I understand, so do you have a hint for me, how i can use C2 and connect via SSL? Or do you have a Branch which i can Build to fix this problem? Thx This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Created] (NIFI-6651) Connections overlaps if directed to another already connected processor
Arthur Grigorjan created NIFI-6651: -- Summary: Connections overlaps if directed to another already connected processor Key: NIFI-6651 URL: https://issues.apache.org/jira/browse/NIFI-6651 Project: Apache NiFi Issue Type: Bug Components: Core UI Affects Versions: 1.9.2 Environment: version: "3" services: zookeeper: hostname: zookeeper2 container_name: zookeeper2 image: 'bitnami/zookeeper:latest' environment: - ALLOW_ANONYMOUS_LOGIN=yes nifi: image: apache/nifi:1.9.2 ports: - 8080 # Unsecured HTTP Web Port environment: - NIFI_WEB_HTTP_PORT=8080 - NIFI_CLUSTER_IS_NODE=true - NIFI_CLUSTER_NODE_PROTOCOL_PORT=8082 - NIFI_ZK_CONNECT_STRING=zookeeper2:2181 - NIFI_ELECTION_MAX_WAIT=1 min Reporter: Arthur Grigorjan Attachments: nifi-overlapping-connection-bug.gif When a connection from processor A to B is redirected from A to C and A to C already has a connection, then the two connections overlap. The user is not able to see that there are multiple connections. This problem doesn't occur when creating a new connection from A to C (see GIF attached). The path of a connection should be calulated the same way when redirecting a connection as it is when creating a new connection. Besides: Why is it not allowed to delete a connection of a running source but to redirect it to another destination? -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Created] (NIFI-6650) Add Infinispan cache processor
Sakina Shaikh created NIFI-6650: --- Summary: Add Infinispan cache processor Key: NIFI-6650 URL: https://issues.apache.org/jira/browse/NIFI-6650 Project: Apache NiFi Issue Type: New Feature Components: Extensions Affects Versions: 1.9.2 Reporter: Sakina Shaikh Add Infinispan cache server support. It has 2 processors # PutInInfinispan It puts value in cache server against the given cache key # GetFromInfinispan It gets the value from cache server against the given cache key -- This message was sent by Atlassian Jira (v8.3.2#803003)
[GitHub] [nifi-minifi] GCHQ-NiFi commented on issue #167: MINIFI-424: Centralise Bootstrap config
GCHQ-NiFi commented on issue #167: MINIFI-424: Centralise Bootstrap config URL: https://github.com/apache/nifi-minifi/pull/167#issuecomment-530295586 @gruselglatz - No, this pull request just simplifies the transformConfigFile function. If this is merged in, it'll be easier to override flow settings using the bootstrap.conf (like the C2 SSL override). I've got a fix for the C2 SSL ready to go, it just relies on this pull request to get merged in first. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [nifi] tpalfy commented on a change in pull request #3684: NIFI-6295: Refactored NiFiRecordSerDe to handle nested complex types
tpalfy commented on a change in pull request #3684: NIFI-6295: Refactored
NiFiRecordSerDe to handle nested complex types
URL: https://github.com/apache/nifi/pull/3684#discussion_r322828012
##
File path:
nifi-nar-bundles/nifi-hive-bundle/nifi-hive3-processors/src/main/java/org/apache/hive/streaming/NiFiRecordSerDe.java
##
@@ -187,152 +182,153 @@ private Object extractCurrentField(Record record,
RecordField field, TypeInfo fi
}
switch (primitiveCategory) {
case BYTE:
-Integer bIntValue = record.getAsInt(fieldName);
-val = bIntValue == null ? null : bIntValue.byteValue();
+Integer bIntValue =
DataTypeUtils.toInteger(fieldValue, fieldName);
+val = bIntValue.byteValue();
break;
case SHORT:
-Integer sIntValue = record.getAsInt(fieldName);
-val = sIntValue == null ? null :
sIntValue.shortValue();
+Integer sIntValue =
DataTypeUtils.toInteger(fieldValue, fieldName);
+val = sIntValue.shortValue();
break;
case INT:
-val = record.getAsInt(fieldName);
+val = DataTypeUtils.toInteger(fieldValue, fieldName);
break;
case LONG:
-val = record.getAsLong(fieldName);
+val = DataTypeUtils.toLong(fieldValue, fieldName);
break;
case BOOLEAN:
-val = record.getAsBoolean(fieldName);
+val = DataTypeUtils.toBoolean(fieldValue, fieldName);
break;
case FLOAT:
-val = record.getAsFloat(fieldName);
+val = DataTypeUtils.toFloat(fieldValue, fieldName);
break;
case DOUBLE:
-val = record.getAsDouble(fieldName);
+val = DataTypeUtils.toDouble(fieldValue, fieldName);
break;
case STRING:
case VARCHAR:
case CHAR:
-val = record.getAsString(fieldName);
+val = DataTypeUtils.toString(fieldValue, fieldName);
break;
case BINARY:
-Object[] array = record.getAsArray(fieldName);
-if (array == null) {
-return null;
+final ArrayDataType arrayDataType;
+if(fieldValue instanceof String) {
+// Treat this as an array of bytes
+arrayDataType = (ArrayDataType)
RecordFieldType.ARRAY.getArrayDataType(RecordFieldType.BYTE.getDataType());
+} else {
+arrayDataType = (ArrayDataType) fieldDataType;
}
+Object[] array = DataTypeUtils.toArray(fieldValue,
fieldName, arrayDataType.getElementType());
val = AvroTypeUtil.convertByteArray(array).array();
break;
case DATE:
-Date d = record.getAsDate(fieldName,
field.getDataType().getFormat());
-if(d != null) {
-org.apache.hadoop.hive.common.type.Date hiveDate =
new org.apache.hadoop.hive.common.type.Date();
-hiveDate.setTimeInMillis(d.getTime());
-val = hiveDate;
-} else {
-val = null;
-}
+Date d = DataTypeUtils.toDate(fieldValue, () ->
DataTypeUtils.getDateFormat(fieldDataType.getFormat()), fieldName);
+org.apache.hadoop.hive.common.type.Date hiveDate = new
org.apache.hadoop.hive.common.type.Date();
+hiveDate.setTimeInMillis(d.getTime());
+val = hiveDate;
break;
// ORC doesn't currently handle TIMESTAMPLOCALTZ
case TIMESTAMP:
-Timestamp ts =
DataTypeUtils.toTimestamp(record.getValue(fieldName), () ->
DataTypeUtils.getDateFormat(field.getDataType().getFormat()), fieldName);
-if(ts != null) {
-// Convert to Hive's Timestamp type
-org.apache.hadoop.hive.common.type.Timestamp
hivetimestamp = new org.apache.hadoop.hive.common.type.Timestamp();
-hivetimestamp.setTimeInMillis(ts.getTime(),
ts.getNanos());
-
[GitHub] [nifi] tpalfy commented on a change in pull request #3684: NIFI-6295: Refactored NiFiRecordSerDe to handle nested complex types
tpalfy commented on a change in pull request #3684: NIFI-6295: Refactored
NiFiRecordSerDe to handle nested complex types
URL: https://github.com/apache/nifi/pull/3684#discussion_r323125610
##
File path:
nifi-nar-bundles/nifi-hive-bundle/nifi-hive3-processors/src/test/java/org/apache/hive/streaming/TestNiFiRecordSerDe.java
##
@@ -0,0 +1,387 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hive.streaming;
+
+import org.apache.hadoop.hive.common.type.Date;
+import org.apache.hadoop.hive.common.type.HiveDecimal;
+import org.apache.hadoop.hive.common.type.Timestamp;
+import org.apache.hadoop.hive.serde.serdeConstants;
+import org.apache.hadoop.hive.serde2.SerDeException;
+import org.apache.hadoop.io.ObjectWritable;
+import org.apache.nifi.avro.AvroTypeUtil;
+import org.apache.nifi.serialization.SimpleRecordSchema;
+import org.apache.nifi.serialization.record.DataType;
+import org.apache.nifi.serialization.record.MapRecord;
+import org.apache.nifi.serialization.record.Record;
+import org.apache.nifi.serialization.record.RecordField;
+import org.apache.nifi.serialization.record.RecordFieldType;
+import org.apache.nifi.serialization.record.RecordSchema;
+import org.apache.nifi.util.MockComponentLog;
+import org.junit.Test;
+
+import java.nio.charset.StandardCharsets;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+
+import static org.junit.Assert.assertArrayEquals;
+import static org.junit.Assert.assertEquals;
+
+public class TestNiFiRecordSerDe {
Review comment:
I think in general this test could benefit from some changes:
1. Use more declarative assertions by defining an "expected" datastructure
2. Slim the the curly bracketed structures. Usually it's better for the
observer if a new line starts a significantly new concept.
I played around with this, here's what I came up with, please use of it as
much as you'd like:
```java
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.hive.streaming;
import org.apache.hadoop.hive.common.type.Date;
import org.apache.hadoop.hive.common.type.HiveDecimal;
import org.apache.hadoop.hive.common.type.Timestamp;
import org.apache.hadoop.hive.serde.serdeConstants;
import org.apache.hadoop.hive.serde2.SerDeException;
import org.apache.hadoop.io.ObjectWritable;
import org.apache.nifi.avro.AvroTypeUtil;
import org.apache.nifi.serialization.SimpleRecordSchema;
import org.apache.nifi.serialization.record.DataType;
import org.apache.nifi.serialization.record.MapRecord;
import org.apache.nifi.serialization.record.Record;
import org.apache.nifi.serialization.record.RecordField;
import org.apache.nifi.serialization.record.RecordFieldType;
import org.apache.nifi.serialization.record.RecordSchema;
import org.apache.nifi.util.MockComponentLog;
import org.junit.Test;
import java.nio.charset.StandardCharsets;
import java.util.Arrays;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
import java.util.Properties;
import static org.junit.Assert.assertArrayEquals;
import static org.junit.Assert.assertEquals;
public class TestNiFiRecordSerDe {
@Test
public void testSimpleFields() throws SerDeException {
NiFiRecordSerDe serDe = createSerDe(
[GitHub] [nifi] tpalfy commented on a change in pull request #3684: NIFI-6295: Refactored NiFiRecordSerDe to handle nested complex types
tpalfy commented on a change in pull request #3684: NIFI-6295: Refactored
NiFiRecordSerDe to handle nested complex types
URL: https://github.com/apache/nifi/pull/3684#discussion_r322827011
##
File path:
nifi-nar-bundles/nifi-hive-bundle/nifi-hive3-processors/src/main/java/org/apache/hive/streaming/NiFiRecordSerDe.java
##
@@ -187,152 +182,153 @@ private Object extractCurrentField(Record record,
RecordField field, TypeInfo fi
}
switch (primitiveCategory) {
case BYTE:
-Integer bIntValue = record.getAsInt(fieldName);
-val = bIntValue == null ? null : bIntValue.byteValue();
+Integer bIntValue =
DataTypeUtils.toInteger(fieldValue, fieldName);
+val = bIntValue.byteValue();
break;
case SHORT:
-Integer sIntValue = record.getAsInt(fieldName);
-val = sIntValue == null ? null :
sIntValue.shortValue();
+Integer sIntValue =
DataTypeUtils.toInteger(fieldValue, fieldName);
+val = sIntValue.shortValue();
break;
case INT:
-val = record.getAsInt(fieldName);
+val = DataTypeUtils.toInteger(fieldValue, fieldName);
break;
case LONG:
-val = record.getAsLong(fieldName);
+val = DataTypeUtils.toLong(fieldValue, fieldName);
break;
case BOOLEAN:
-val = record.getAsBoolean(fieldName);
+val = DataTypeUtils.toBoolean(fieldValue, fieldName);
break;
case FLOAT:
-val = record.getAsFloat(fieldName);
+val = DataTypeUtils.toFloat(fieldValue, fieldName);
break;
case DOUBLE:
-val = record.getAsDouble(fieldName);
+val = DataTypeUtils.toDouble(fieldValue, fieldName);
break;
case STRING:
case VARCHAR:
case CHAR:
-val = record.getAsString(fieldName);
+val = DataTypeUtils.toString(fieldValue, fieldName);
break;
case BINARY:
-Object[] array = record.getAsArray(fieldName);
-if (array == null) {
-return null;
+final ArrayDataType arrayDataType;
+if(fieldValue instanceof String) {
+// Treat this as an array of bytes
+arrayDataType = (ArrayDataType)
RecordFieldType.ARRAY.getArrayDataType(RecordFieldType.BYTE.getDataType());
+} else {
+arrayDataType = (ArrayDataType) fieldDataType;
}
+Object[] array = DataTypeUtils.toArray(fieldValue,
fieldName, arrayDataType.getElementType());
val = AvroTypeUtil.convertByteArray(array).array();
break;
case DATE:
-Date d = record.getAsDate(fieldName,
field.getDataType().getFormat());
-if(d != null) {
-org.apache.hadoop.hive.common.type.Date hiveDate =
new org.apache.hadoop.hive.common.type.Date();
-hiveDate.setTimeInMillis(d.getTime());
-val = hiveDate;
-} else {
-val = null;
-}
+Date d = DataTypeUtils.toDate(fieldValue, () ->
DataTypeUtils.getDateFormat(fieldDataType.getFormat()), fieldName);
+org.apache.hadoop.hive.common.type.Date hiveDate = new
org.apache.hadoop.hive.common.type.Date();
+hiveDate.setTimeInMillis(d.getTime());
+val = hiveDate;
break;
// ORC doesn't currently handle TIMESTAMPLOCALTZ
case TIMESTAMP:
-Timestamp ts =
DataTypeUtils.toTimestamp(record.getValue(fieldName), () ->
DataTypeUtils.getDateFormat(field.getDataType().getFormat()), fieldName);
-if(ts != null) {
-// Convert to Hive's Timestamp type
-org.apache.hadoop.hive.common.type.Timestamp
hivetimestamp = new org.apache.hadoop.hive.common.type.Timestamp();
-hivetimestamp.setTimeInMillis(ts.getTime(),
ts.getNanos());
-
[GitHub] [nifi-minifi] gruselglatz commented on issue #167: MINIFI-424: Centralise Bootstrap config
gruselglatz commented on issue #167: MINIFI-424: Centralise Bootstrap config URL: https://github.com/apache/nifi-minifi/pull/167#issuecomment-530289152 @GCHQ-NiFi Does this fix the issue, that Minifi cant connect via SSL to Nifi when Minifi-C2 is used? bootstrap.conf doesnt overwrite ssl settings in config.yml as intended, and every time C2 is refreshing the config, ssl wont connect anymore (because empty) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
