[metron] branch master updated: METRON-1849 Elasticsearch Index Write Functionality Should be Shared (nickwallen) closes apache/metron#1254
This is an automated email from the ASF dual-hosted git repository.
nickallen pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/metron.git
The following commit(s) were added to refs/heads/master by this push:
new ec3b98f METRON-1849 Elasticsearch Index Write Functionality Should be
Shared (nickwallen) closes apache/metron#1254
ec3b98f is described below
commit ec3b98f762ce3726ed9a33abdb446957d1865dca
Author: nickwallen
AuthorDate: Tue Dec 11 12:59:08 2018 -0500
METRON-1849 Elasticsearch Index Write Functionality Should be Shared
(nickwallen) closes apache/metron#1254
---
.../elasticsearch/bulk/BulkDocumentWriter.java | 45 +++
.../bulk/BulkDocumentWriterResults.java| 68
.../bulk/ElasticsearchBulkDocumentWriter.java | 166 ++
.../metron/elasticsearch/bulk/WriteFailure.java| 48 +++
.../metron/elasticsearch/bulk/WriteSuccess.java| 36 +++
.../metron/elasticsearch/dao/ElasticsearchDao.java | 2 +-
.../elasticsearch/dao/ElasticsearchUpdateDao.java | 144 -
.../elasticsearch/writer/ElasticsearchWriter.java | 157 +
.../elasticsearch/writer/TupleBasedDocument.java | 44 +++
.../bulk/ElasticsearchBulkDocumentWriterTest.java | 178 ++
.../components/ElasticSearchComponent.java | 46 +--
.../writer/ElasticsearchWriterTest.java| 360 ++---
.../src/test/resources/log4j.properties| 0
13 files changed, 994 insertions(+), 300 deletions(-)
diff --git
a/metron-platform/metron-elasticsearch/src/main/java/org/apache/metron/elasticsearch/bulk/BulkDocumentWriter.java
b/metron-platform/metron-elasticsearch/src/main/java/org/apache/metron/elasticsearch/bulk/BulkDocumentWriter.java
new file mode 100644
index 000..34f543e
--- /dev/null
+++
b/metron-platform/metron-elasticsearch/src/main/java/org/apache/metron/elasticsearch/bulk/BulkDocumentWriter.java
@@ -0,0 +1,45 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.elasticsearch.bulk;
+
+import org.apache.metron.indexing.dao.update.Document;
+
+/**
+ * Writes documents to an index in bulk.
+ *
+ * @param The type of document to write.
+ */
+public interface BulkDocumentWriter {
+
+/**
+ * Add a document to the batch.
+ * @param document The document to write.
+ * @param index The name of the index to write to.
+ */
+void addDocument(D document, String index);
+
+/**
+ * @return The number of documents waiting to be written.
+ */
+int size();
+
+/**
+ * Write all documents in the batch.
+ */
+BulkDocumentWriterResults write();
+}
diff --git
a/metron-platform/metron-elasticsearch/src/main/java/org/apache/metron/elasticsearch/bulk/BulkDocumentWriterResults.java
b/metron-platform/metron-elasticsearch/src/main/java/org/apache/metron/elasticsearch/bulk/BulkDocumentWriterResults.java
new file mode 100644
index 000..90e5ce3
--- /dev/null
+++
b/metron-platform/metron-elasticsearch/src/main/java/org/apache/metron/elasticsearch/bulk/BulkDocumentWriterResults.java
@@ -0,0 +1,68 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.elasticsearch.bulk;
+
+import org.apache.metron.indexing.dao.update.Document;
+
+import java.util.ArrayList;
+import java.util.List;
+
+/**
+ * The result of writing documents in bulk using a {@link BulkDocumentWriter}.
+ * @param The type
svn commit: r31498 - in /dev/metron/0.7.0-RC1: ./ CHANGES LICENSE NOTICE apache-metron-0.7.0-rc1.tar.gz apache-metron-0.7.0-rc1.tar.gz.asc apache-metron-0.7.0-rc1.tar.gz.sha256 apache-metron-0.7.0-rc1
Author: leet Date: Tue Dec 11 19:35:08 2018 New Revision: 31498 Log: Adding artifacts for metron 0.7.0-RC1 Added: dev/metron/0.7.0-RC1/ dev/metron/0.7.0-RC1/CHANGES dev/metron/0.7.0-RC1/LICENSE dev/metron/0.7.0-RC1/NOTICE dev/metron/0.7.0-RC1/apache-metron-0.7.0-rc1.tar.gz (with props) dev/metron/0.7.0-RC1/apache-metron-0.7.0-rc1.tar.gz.asc dev/metron/0.7.0-RC1/apache-metron-0.7.0-rc1.tar.gz.sha256 dev/metron/0.7.0-RC1/apache-metron-0.7.0-rc1.tar.gz.sha512 Added: dev/metron/0.7.0-RC1/CHANGES == --- dev/metron/0.7.0-RC1/CHANGES (added) +++ dev/metron/0.7.0-RC1/CHANGES Tue Dec 11 19:35:08 2018 @@ -0,0 +1,98 @@ +METRON-1928 Bump Metron version to 0.7.0 for release. (justinleet) closes apache/metron#1293 +METRON-1931 Update dev utilities to support new repo location (rlenferink via justinleet) closes apache/metron#1295 +METRON-1922 Escaping incorrectly handled in current aesh version (justinleet) closes apache/metron#1291 +METRON-1867 Remove `/api/v1/update/replace` endpoint (nickwallen) closes apache/metron#1284 +METRON-1810 Storm Profiler Intermittent Test Failure (nickwallen) closes apache/metron#1289 +METRON-1909 Remove http filter from release utils changelog generation (justinleet) closes apache/metron#1283 +METRON-1869 Unable to Sort an Escalated Meta Alert (nickwallen) closes apache/metron#1280 +METRON-1889: Add any missing timestamp fields to unified enrichment topology (mmiklavc via mmiklavc) closes apache/metron#1286 +METRON-1913 metron-alert UI - Build broken by missing transitive dependency (tiborm via sardell) closes apache/metron#1285 +METRON-1845 Correct Test Data Load in Elasticsearch Integration Tests (nickwallen) closes apache/metron#1247 +METRON-1888 Default Topology Settings in MPack Cause Profiler to Stall (nickwallen) closes apache/metron#1276 +METRON-1887: Add logging to the ClasspathFunctionResolver (mmiklavc via mmiklavc) closes apache/metron#1274 +METRON-1873 Update Bootstrap version in Management UI (sardell) closes apache/metron#1267 +METRON-1825 Upgrade bro to 2.5.5 (JonZeolla via nickwallen) closes apache/metron#1237 +METRON-1890 Metron Vagrant should disable audio (ottobackwards) closes apache/metron#1277 +METRON-1874 Create a Parser Debugger (nickwallen) closes apache/metron#1265 +METRON-1880 Use Caffeine for Profiler Caching (nickwallen) closes apache/metron#1270 +METRON-1877 Nested IF ELSE statements can cause parse errors in Stellar (justinleet) closes apache/metron#1268 +METRON-1872 Move rat plugin away from snapshot version (justinleet) closes apache/metron#1264 +METRON-1875 Expose configurable global settings in the Alerts UI (merrimanr) closes apache/metron#1266 +METRON-1834: Migrate Elasticsearch from TransportClient to new Java REST API (mmiklavc via mmiklavc) closes apache/metron#1242 +METRON-1834: Migrate Elasticsearch from TransportClient to new Java REST API (cstella via mmiklavc) +METRON-1749 Update Angular to latest release in Management UI (sardell via nickwallen) closes apache/metron#1217 +METRON-1870 Intermittent Stellar REST test failures (merrimanr via nickwallen) closes apache/metron#1263 +METRON-1868 metron-committer-common incorrectly checking REPO_NAME (JonZeolla via jonzeolla) closes apache/metron#1260 +METRON-1740 Improve Palo Alto parser to handle CONFIG and SYSTEM syslog messages (liuy-tnz via nickwallen) closes apache/metron#1171 +METRON-1847 Create reusable script with functions from prepare-commit (ottobackwards) closes apache/metron#1248 +METRON-1850 Stellar REST function (merrimanr) closes apache/metron#1250 +METRON-1858 BasicFireEyeParser check style cleanup and optimization (ottobackwards) closes apache/metron#1255 +METRON-1864 Stellar date format test fails after daylight saving (ottobackwards) closes apache/metron#1258 +METRON-1861 METRON-1861: REST fails to start when LDAP enabled and 'Active Spring profiles' config is empty (anandsubbu via justinleet) closes apache/metron#1256 +METRON-1853: Add shutdown hook to Stellar BaseFunctionResolver (mmiklavc via mmiklavc) closes apache/metron#1251 +METRON-1857 Fix Metaalert Nested Alert Field Name in Index Template (nickwallen) closes apache/metron#1253 +METRON-1855: Make unified enrichment topology the default and deprecate split-join (mmiklavc via mmiklavc) closes apache/metron#1252 +METRON-1790 Unsubscribe from every observable in the pcap panel UI component (ruffle via nickwallen) closes apache/metron#1208 +METRON-1803: Integrate Cypress with Travis (tiborm via mmiklavc) closes apache/metron#1226 +METRON-1844 Allow for LDAP to be used for authentication and roles (justinleet) closes apache/metron#1246 +METRON-1844 Allow for LDAP to be used for authentication and roles (simonellistonball via
[metron] branch Metron_0.7.0 created (now 2263983)
This is an automated email from the ASF dual-hosted git repository. leet pushed a change to branch Metron_0.7.0 in repository https://gitbox.apache.org/repos/asf/metron.git. at 2263983 METRON-1928 Bump Metron version to 0.7.0 for release. (justinleet) closes apache/metron#1293 No new revisions were added by this update.
[metron] branch master updated: METRON-1928 Bump Metron version to 0.7.0 for release. (justinleet) closes apache/metron#1293
This is an automated email from the ASF dual-hosted git repository. leet pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/metron.git The following commit(s) were added to refs/heads/master by this push: new 2263983 METRON-1928 Bump Metron version to 0.7.0 for release. (justinleet) closes apache/metron#1293 2263983 is described below commit 2263983761e77b7eec52f70f5e8f8001bac83125 Author: justinleet AuthorDate: Tue Dec 11 09:41:40 2018 -0500 METRON-1928 Bump Metron version to 0.7.0 for release. (justinleet) closes apache/metron#1293 --- Upgrading.md | 2 +- metron-analytics/metron-maas-common/pom.xml | 2 +- metron-analytics/metron-maas-service/pom.xml | 2 +- metron-analytics/metron-profiler-client/pom.xml | 2 +- metron-analytics/metron-profiler-common/pom.xml | 2 +- metron-analytics/metron-profiler-repl/pom.xml | 2 +- metron-analytics/metron-profiler-spark/pom.xml| 2 +- metron-analytics/metron-profiler-storm/pom.xml| 2 +- metron-analytics/metron-statistics/pom.xml| 2 +- metron-analytics/pom.xml | 2 +- metron-contrib/metron-docker/pom.xml | 2 +- metron-contrib/metron-performance/pom.xml | 2 +- metron-contrib/pom.xml| 2 +- metron-deployment/amazon-ec2/conf/defaults.yml| 2 +- metron-deployment/ansible/playbooks/docker_probe_install.yml | 2 +- metron-deployment/ansible/roles/ambari_master/defaults/main.yml | 2 +- .../development/centos6/ansible/inventory/group_vars/all | 2 +- .../development/ubuntu14/ansible/inventory/group_vars/all | 2 +- metron-deployment/packaging/ambari/elasticsearch-mpack/pom.xml| 4 ++-- metron-deployment/packaging/ambari/metron-mpack/pom.xml | 4 ++-- .../packaging/ambari/metron-mpack/src/main/resources/mpack.json | 2 +- metron-deployment/packaging/docker/deb-docker/pom.xml | 4 ++-- metron-deployment/packaging/docker/rpm-docker/SPECS/metron.spec | 8 metron-deployment/packaging/docker/rpm-docker/pom.xml | 4 ++-- metron-deployment/pom.xml | 2 +- metron-interface/metron-alerts/package-lock.json | 2 +- metron-interface/metron-alerts/package.json | 2 +- metron-interface/metron-alerts/pom.xml| 2 +- metron-interface/metron-alerts/scripts/package.json | 2 +- metron-interface/metron-config/package-lock.json | 2 +- metron-interface/metron-config/package.json | 2 +- metron-interface/metron-config/pom.xml| 2 +- metron-interface/metron-config/scripts/package.json | 2 +- metron-interface/metron-rest-client/pom.xml | 2 +- metron-interface/metron-rest/pom.xml | 2 +- metron-interface/metron-rest/src/main/resources/application.yml | 2 +- metron-interface/pom.xml | 2 +- metron-platform/Performance-tuning-guide.md | 4 ++-- metron-platform/README.md | 2 +- metron-platform/elasticsearch-shaded/pom.xml | 2 +- metron-platform/metron-common/pom.xml | 2 +- metron-platform/metron-common/src/main/scripts/cluster_info.py| 2 +- metron-platform/metron-data-management/pom.xml| 2 +- metron-platform/metron-elasticsearch/pom.xml | 2 +- metron-platform/metron-enrichment/pom.xml | 2 +- metron-platform/metron-hbase-client/pom.xml | 2 +- metron-platform/metron-hbase/pom.xml | 2 +- metron-platform/metron-indexing/pom.xml | 2 +- metron-platform/metron-integration-test/pom.xml | 2 +- metron-platform/metron-job/pom.xml| 2 +- metron-platform/metron-management/pom.xml | 2 +- metron-platform/metron-parsers/pom.xml| 2 +- metron-platform/metron-pcap-backend/pom.xml | 2 +- .../metron-pcap-backend/src/main/scripts/pcap_zeppelin_run.sh | 2 +- metron-platform/metron-pcap/pom.xml | 2 +- metron-platform/metron-solr/pom.xml | 2 +- metron-platform/metron-storm-kafka-override/pom.xml | 2 +- metron-platform/metron-storm-kafka/pom.xml| 2 +- metron-platform/metron-test-utilities/pom.xml | 2 +-
