urbandan commented on a change in pull request #4950:
URL: https://github.com/apache/nifi/pull/4950#discussion_r607548833
##########
File path:
nifi-nar-bundles/nifi-framework-bundle/nifi-stateless-bundle/nifi-stateless-api/src/main/java/org/apache/nifi/stateless/config/PropertiesFileEngineConfigurationParser.java
##########
@@ -78,6 +79,12 @@ public StatelessEngineConfiguration
parseEngineConfiguration(final File properti
throw new StatelessConfigurationException("Working Directory " +
workingDirectory.getAbsolutePath() + " specified in properties file does not
exist and could not be created");
}
+ final String extensionsDirectoryFilename =
properties.getProperty(EXTENSIONS_DIRECTORY);
+ final File extensionsDirectory = extensionsDirectoryFilename == null ?
narDirectory : new File(extensionsDirectoryFilename);
+ if (!extensionsDirectory.exists() && !extensionsDirectory.mkdirs()) {
Review comment:
Got it, thanks
##########
File path:
nifi-nar-bundles/nifi-framework-bundle/nifi-stateless-bundle/nifi-stateless-bootstrap/src/main/java/org/apache/nifi/stateless/bootstrap/StatelessBootstrap.java
##########
@@ -98,7 +99,12 @@ public static StatelessBootstrap bootstrap(final
StatelessEngineConfiguration en
// Unpack NARs
final long unpackStart = System.currentTimeMillis();
final Predicate<BundleCoordinate> narFilter = coordinate -> true;
- NarUnpacker.unpackNars(systemBundle, frameworkWorkingDir,
extensionsWorkingDir, null, narDirectories, false, false, false, narFilter);
+ NarUnpackLock.lock();
Review comment:
Is this lock also protecting the part when the nars are getting
downloaded? Can it cause issues if the download is not protected by it?
With this solution, do we avoid downloading the nars multiple times? (i.e.
if one task is blocked on getting the lock, will it realize that the sibling
task already downloaded and unpacked the necessary nars?)
##########
File path:
nifi-nar-bundles/nifi-framework-bundle/nifi-stateless-bundle/nifi-stateless-api/src/main/java/org/apache/nifi/stateless/engine/NarUnpackLock.java
##########
@@ -0,0 +1,58 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.stateless.engine;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.concurrent.locks.Lock;
+import java.util.concurrent.locks.ReentrantLock;
+
+/**
+ * If multiple Stateless dataflows are loaded concurrently within the same
JVM, we need to ensure that the dataflows
+ * do not stomp on one another when unpacking NAR's. To do that, we need a
mechanism by which a single lock can be shared
+ * across multiple classes, as the Extension Repository as well as the
bootstrap logic may attempt to unpack NARs.
+ * Because these classes exist across multiple modules, and because statically
defined locks at that level may not be enough
+ * (due to multiple classloders being used for the 'stateless nar'), we define
a singleton Lock within the nifi-stateless-api module.
+ * This lock should always be obtained before attempting to unpack nars.
+ */
+public class NarUnpackLock {
Review comment:
As an improvement, this could be providing locks on a per-directory
basis - so if we have multiple NiFi Connectors with separate nar directories,
they can progress without blocking each other.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
