nickwallen commented on a change in pull request #1383: METRON-1788 Batch
profiler pull profile information from zookeeper
URL: https://github.com/apache/metron/pull/1383#discussion_r276252816
##########
File path:
metron-analytics/metron-profiler-spark/src/main/java/org/apache/metron/profiler/spark/cli/BatchProfilerCLI.java
##########
@@ -96,6 +112,65 @@ public static void main(String[] args) throws IOException,
org.apache.commons.cl
LOG.info("Profiler produced {} profile measurement(s)", count);
}
+ /**
+ * Extracts profile information from a file or from zookeeper
+ * @param commandLine Command line information.
+ * @return Profile information
+ * @throws MissingOptionException if command line options are missing
+ * @throws IOException If there are disk or network issues retrieving
profiles
+ */
+ private static ProfilerConfig handleProfileDefinitions(CommandLine
commandLine) throws MissingOptionException, IOException {
+ final String PROFILE_LOCATION_ERROR =
+ "A single profile location (--profiles or --zookeeper) must be
specified";
+ ProfilerConfig profiles;
+
+ if ((!PROFILE_ZK.has(commandLine)) &&
(!PROFILE_DEFN_FILE.has(commandLine))) {
+ throw new MissingOptionException(PROFILE_LOCATION_ERROR);
+ }
+ if (PROFILE_ZK.has(commandLine) && PROFILE_DEFN_FILE.has(commandLine)) {
+ throw new IllegalArgumentException(PROFILE_LOCATION_ERROR);
+ }
+
+ if (PROFILE_ZK.has(commandLine)) {
+ try (final CuratorFramework zkClient = createZKClient(commandLine)) {
+ try {
+ profiles = handleProfileDefinitionsZK(zkClient);
+ } catch (Exception ex) {
+ throw new IOException(
+ String.format("Error reading configuration from Zookeeper
client %s",
+ zkClient.toString()),
+ ex);
+ }
+ }
+ } else {
+ profiles = handleProfileDefinitionsFile(commandLine);
+ }
+
+ // event time can specified via command line override
+ if (PROFILE_TIMESTAMP_FLD.has(commandLine)) {
Review comment:
I like the `--timestamp` option that you added. It will override the
timestamp, no matter if the profile definition is pulled from the filesystem or
Zk. I think that's a logical way to do it.
Can you make sure the docs reflect that? I think the README mentions that
it only overrides a profile pulled from Zk.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
