[
https://issues.apache.org/jira/browse/NIFI-3594?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15982057#comment-15982057
]
ASF GitHub Bot commented on NIFI-3594:
--------------------------------------
Github user alopresto commented on a diff in the pull request:
https://github.com/apache/nifi/pull/1686#discussion_r113075612
--- Diff:
nifi-nar-bundles/nifi-provenance-repository-bundle/nifi-persistent-provenance-repository/src/main/java/org/apache/nifi/provenance/EncryptedSchemaRecordReader.java
---
@@ -0,0 +1,154 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.provenance;
+
+import java.io.ByteArrayInputStream;
+import java.io.DataInputStream;
+import java.io.File;
+import java.io.IOException;
+import java.io.InputStream;
+import java.util.Collection;
+import java.util.Optional;
+import java.util.concurrent.TimeUnit;
+import org.apache.nifi.provenance.schema.LookupTableEventRecord;
+import org.apache.nifi.provenance.toc.TocReader;
+import org.apache.nifi.repository.schema.Record;
+import org.apache.nifi.stream.io.LimitingInputStream;
+import org.apache.nifi.stream.io.StreamUtils;
+import org.apache.nifi.util.timebuffer.LongEntityAccess;
+import org.apache.nifi.util.timebuffer.TimedBuffer;
+import org.apache.nifi.util.timebuffer.TimestampedLong;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class EncryptedSchemaRecordReader extends
EventIdFirstSchemaRecordReader {
+ private static final Logger logger =
LoggerFactory.getLogger(EncryptedSchemaRecordReader.class);
+
+ private static final int DEFAULT_DEBUG_FREQUENCY = 1_000_000;
+
+ private ProvenanceEventEncryptor provenanceEventEncryptor;
+
+ private static final TimedBuffer<TimestampedLong> decryptTimes = new
TimedBuffer<>(TimeUnit.SECONDS, 60, new LongEntityAccess());
+
+ private int debugFrequency = DEFAULT_DEBUG_FREQUENCY;
+ public static final int SERIALIZATION_VERSION = 1;
+
+ public static final String SERIALIZATION_NAME =
"EncryptedSchemaRecordWriter";
+
+ public EncryptedSchemaRecordReader(final InputStream inputStream,
final String filename, final TocReader tocReader, final int maxAttributeChars,
+ ProvenanceEventEncryptor
provenanceEventEncryptor) throws IOException {
+ this(inputStream, filename, tocReader, maxAttributeChars,
provenanceEventEncryptor, DEFAULT_DEBUG_FREQUENCY);
+ }
+
+ public EncryptedSchemaRecordReader(final InputStream inputStream,
final String filename, final TocReader tocReader, final int maxAttributeChars,
+ ProvenanceEventEncryptor
provenanceEventEncryptor, int debugFrequency) throws IOException {
+ super(inputStream, filename, tocReader, maxAttributeChars);
+ this.provenanceEventEncryptor = provenanceEventEncryptor;
+ this.debugFrequency = debugFrequency;
+ }
+
+ @Override
+ protected StandardProvenanceEventRecord nextRecord(final
DataInputStream in, final int serializationVersion) throws IOException {
+ verifySerializationVersion(serializationVersion);
+
+ final long byteOffset = getBytesConsumed();
+ final long eventId = in.readInt() + getFirstEventId();
+ final int recordLength = in.readInt();
+
+ return readRecord(in, eventId, byteOffset, recordLength);
+ }
+
+ private StandardProvenanceEventRecord readRecord(final DataInputStream
inputStream, final long eventId, final long startOffset, final int
recordLength) throws IOException {
+ try {
+ final InputStream limitedIn = new
LimitingInputStream(inputStream, recordLength);
+
+ byte[] encryptedSerializedBytes = new byte[recordLength];
+ DataInputStream encryptedInputStream = new
DataInputStream(limitedIn);
+ encryptedInputStream.readFully(encryptedSerializedBytes);
+
+ byte[] plainSerializedBytes =
decrypt(encryptedSerializedBytes, Long.toString(eventId));
+ InputStream plainStream = new
ByteArrayInputStream(plainSerializedBytes);
+
+ final Record eventRecord =
getRecordReader().readRecord(plainStream);
+ if (eventRecord == null) {
+ return null;
+ }
+
+ final StandardProvenanceEventRecord deserializedEvent =
LookupTableEventRecord.getEvent(eventRecord, getFilename(), startOffset,
getMaxAttributeLength(),
+ getFirstEventId(), getSystemTimeOffset(),
getComponentIds(), getComponentTypes(), getQueueIds(), getEventTypes());
+ deserializedEvent.setEventId(eventId);
+ return deserializedEvent;
+ } catch (EncryptionException e) {
+ logger.error("Encountered an error reading the record: ", e);
+ throw new IOException(e);
+ }
+ }
+
+ // TODO: Copied from EventIdFirstSchemaRecordReader to force
local/overridden readRecord()
--- End diff --
This was a note to myself/any reviewer about why I copied the entire method
body from the other reader implementation. If I just referred to the other
instance (i.e. comment out this entire method declaration), the operation will
fail (see `EncryptedSchemaRecordReaderWriterTest#testSkipToEvent()`) even
though the code is identical. If there is a better suggestion for how to fix
this, I am all ears.
> Implement encrypted provenance repository
> -----------------------------------------
>
> Key: NIFI-3594
> URL: https://issues.apache.org/jira/browse/NIFI-3594
> Project: Apache NiFi
> Issue Type: Sub-task
> Components: Core Framework
> Affects Versions: 1.1.1
> Reporter: Andy LoPresto
> Assignee: Andy LoPresto
> Labels: encryption, provenance, repository
>
> I am going to start with the provenance repository, as the new implementation
> of {{WriteAheadProvenanceRepository}} has the most recent design decisions
> and has not been available in a released version yet, so there should be
> minimal backward compatibility concerns.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
