[
https://issues.apache.org/jira/browse/METRON-283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15838512#comment-15838512
]
ASF GitHub Bot commented on METRON-283:
---------------------------------------
Github user nickwallen commented on a diff in the pull request:
https://github.com/apache/incubator-metron/pull/421#discussion_r97868280
--- Diff:
metron-platform/metron-enrichment/src/main/java/org/apache/metron/enrichment/adapters/geo/GeoLiteDatabase.java
---
@@ -0,0 +1,184 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.enrichment.adapters.geo;
+
+import com.maxmind.db.CHMCache;
+import com.maxmind.geoip2.DatabaseReader;
+import com.maxmind.geoip2.exception.GeoIp2Exception;
+import com.maxmind.geoip2.model.CityResponse;
+import com.maxmind.geoip2.record.City;
+import com.maxmind.geoip2.record.Country;
+import com.maxmind.geoip2.record.Location;
+import com.maxmind.geoip2.record.Postal;
+import org.apache.commons.validator.routines.InetAddressValidator;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.net.InetAddress;
+import java.net.UnknownHostException;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Optional;
+import java.util.concurrent.locks.Lock;
+import java.util.concurrent.locks.ReentrantReadWriteLock;
+import java.util.zip.GZIPInputStream;
+
+public enum GeoLiteDatabase {
+ INSTANCE;
+
+ protected static final Logger LOG =
LoggerFactory.getLogger(GeoLiteDatabase.class);
+ public static final String GEO_HDFS_FILE = "geo.hdfs.file";
+ public static final String GEO_HDFS_FILE_DEFAULT =
"/apps/metron/geo/default/GeoLite2-City.mmdb.gz";
+
+ private static ReentrantReadWriteLock lock = new
ReentrantReadWriteLock();
+ private static final Lock readLock = lock.readLock();
+ private static final Lock writeLock = lock.writeLock();
+ private static InetAddressValidator ipvalidator = new
InetAddressValidator();
+ private static volatile String hdfsLoc = GEO_HDFS_FILE_DEFAULT;
+ private static DatabaseReader reader = null;
+
+ public synchronized void updateIfNecessary(Map<String, Object>
globalConfig) {
+ // Reload database if necessary (file changes on HDFS)
+ LOG.trace("[Metron] Determining if GeoIpDatabase update required");
+ String hdfsFile = GEO_HDFS_FILE_DEFAULT;
+ if (globalConfig != null) {
+ hdfsFile = (String) globalConfig.getOrDefault(GEO_HDFS_FILE,
GEO_HDFS_FILE_DEFAULT);
+ }
+
+ // Always update if we don't have a DatabaseReader
+ if (reader == null || !hdfsLoc.equals(hdfsFile)) {
--- End diff --
Does this null check need to be protected by a lock? I am really not sure.
> Migrate Geo Enrichment outside of MySQL
> ---------------------------------------
>
> Key: METRON-283
> URL: https://issues.apache.org/jira/browse/METRON-283
> Project: Metron
> Issue Type: Improvement
> Reporter: James Sirota
> Assignee: Justin Leet
> Priority: Minor
>
> We need to migrate our enrichment SQL store from MySQL to Phoenix or some
> other SQL on Hbase library. Or alternatively come up with a way to do this
> without using SQL. This way we don't have a dependency on MySQL and there is
> one less thing that we need to install on our platform
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
