[jira] [Created] (METRON-1843) ElasticsearchUpdateIntegrationTest can fail sometimes
Otto Fowler created METRON-1843: --- Summary: ElasticsearchUpdateIntegrationTest can fail sometimes Key: METRON-1843 URL: https://issues.apache.org/jira/browse/METRON-1843 Project: Metron Issue Type: Bug Reporter: Otto Fowler Tests run: 3, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 16.437 sec <<< FAILURE! - in org.apache.metron.elasticsearch.integration.ElasticsearchUpdateIntegrationTest test(org.apache.metron.elasticsearch.integration.ElasticsearchUpdateIntegrationTest) Time elapsed: 6.205 sec <<< FAILURE! java.lang.AssertionError: Data store is not updated!. Actual: 0 at org.junit.Assert.fail(Assert.java:88) at org.junit.Assert.failEquals(Assert.java:185) at org.junit.Assert.assertNotEquals(Assert.java:199) at org.apache.metron.indexing.dao.UpdateIntegrationTest.test(UpdateIntegrationTest.java:132) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.runners.ParentRunner.run(ParentRunner.java:363) at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:283) at org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:173) at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:153) at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:128) at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:203) at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:155) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (METRON-1834) Migrate Elasticsearch from TransportClient to new Java REST API
[
https://issues.apache.org/jira/browse/METRON-1834?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16662448#comment-16662448
]
ASF GitHub Bot commented on METRON-1834:
Github user nickwallen commented on a diff in the pull request:
https://github.com/apache/metron/pull/1242#discussion_r227848817
--- Diff:
metron-platform/metron-elasticsearch/src/main/java/org/apache/metron/elasticsearch/client/ElasticsearchClient.java
---
@@ -0,0 +1,147 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.elasticsearch.client;
+
+import com.google.common.base.Joiner;
+import com.google.common.base.Splitter;
+import com.google.common.collect.Iterables;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import org.apache.commons.io.IOUtils;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.http.HttpEntity;
+import org.apache.http.entity.StringEntity;
+import org.apache.metron.common.utils.JSONUtils;
+import org.apache.metron.elasticsearch.utils.FieldMapping;
+import org.apache.metron.elasticsearch.utils.FieldProperties;
+import org.elasticsearch.client.Response;
+import org.elasticsearch.client.RestClient;
+import org.elasticsearch.client.RestHighLevelClient;
+
+/**
+ * Wrapper around the Elasticsearch REST clients. Exposes capabilities of
the low and high-level clients.
+ */
+public class ElasticsearchClient implements AutoCloseable{
+ private RestClient lowLevelClient;
+ private RestHighLevelClient highLevelClient;
+
+ public ElasticsearchClient(RestClient lowLevelClient,
RestHighLevelClient highLevelClient) {
+this.lowLevelClient = lowLevelClient;
+this.highLevelClient = highLevelClient;
+ }
+
+ public RestClient getLowLevelClient() {
+return lowLevelClient;
+ }
+
+ public RestHighLevelClient getHighLevelClient() {
+return highLevelClient;
+ }
+
+ @Override
+ public void close() throws IOException {
+if(lowLevelClient != null) {
+ lowLevelClient.close();
+}
+ }
+
+ public void putMapping(String index, String type, String source) throws
IOException {
--- End diff --
For example, I was just trying to use this. I assume type == docType, but
that's not completely clear to me.
> Migrate Elasticsearch from TransportClient to new Java REST API
> ---
>
> Key: METRON-1834
> URL: https://issues.apache.org/jira/browse/METRON-1834
> Project: Metron
> Issue Type: Improvement
>Reporter: Michael Miklavcic
>Assignee: Michael Miklavcic
>Priority: Major
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] metron issue #789: METRON-1233: Remove description of Global configuration f...
Github user mmiklavc commented on the issue: https://github.com/apache/metron/pull/789 @DimDroll @ottobackwards - All of our topologies pull in the global config. One such more recent example is that we now provide an option for specifying batching details for enrichment: ``` enrichment.writer.batchSize enrichment.writer.batchTimeout ``` The reason for this is that we don't have a global-only-per-topology type of configuration, with the exception parsers because of how they can be deployed independently. If you think there's some better clarification that could be made, I'm open to it. But I think the link between parsers, enrichment, indexing, and the global configs should be maintained as it is relevant. ---
[jira] [Commented] (METRON-1233) Remove description of Global configuration from Enrichment doc
[ https://issues.apache.org/jira/browse/METRON-1233?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16662449#comment-16662449 ] ASF GitHub Bot commented on METRON-1233: Github user mmiklavc commented on the issue: https://github.com/apache/metron/pull/789 @DimDroll @ottobackwards - All of our topologies pull in the global config. One such more recent example is that we now provide an option for specifying batching details for enrichment: ``` enrichment.writer.batchSize enrichment.writer.batchTimeout ``` The reason for this is that we don't have a global-only-per-topology type of configuration, with the exception parsers because of how they can be deployed independently. If you think there's some better clarification that could be made, I'm open to it. But I think the link between parsers, enrichment, indexing, and the global configs should be maintained as it is relevant. > Remove description of Global configuration from Enrichment doc > -- > > Key: METRON-1233 > URL: https://issues.apache.org/jira/browse/METRON-1233 > Project: Metron > Issue Type: Improvement >Reporter: Dima Kovalyov >Priority: Minor > > It confuses reader. Description of Global configuration is available ahead of > the enrichment and listed here: > https://metron.apache.org/current-book/metron-platform/metron-common/index.html > Global config is used for validation of the fields. > Enrichment is used for fields enrichment and probably can also be used for > validation of the fields (which is not covered in the doc). > From available doc I can see that they serve two different purposes. > So, I don't see any reason to mention global config in enrichment > description. What's the benefit of knowing about global configuration when > you want deep dive in enrichment? -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] metron pull request #1242: METRON-1834: Migrate Elasticsearch from Transport...
Github user nickwallen commented on a diff in the pull request:
https://github.com/apache/metron/pull/1242#discussion_r227847716
--- Diff:
metron-platform/metron-elasticsearch/src/main/java/org/apache/metron/elasticsearch/utils/ElasticsearchUtils.java
---
@@ -124,102 +129,146 @@ public static String getBaseIndexName(String
indexName) {
}
/**
- * Instantiates an Elasticsearch client based on es.client.class, if
set. Defaults to
- * org.elasticsearch.transport.client.PreBuiltTransportClient.
+ * Instantiates an Elasticsearch client
*
* @param globalConfiguration Metron global config
- * @return
+ * @return new es client
*/
- public static TransportClient getClient(Map
globalConfiguration) {
-Set customESSettings = new HashSet<>();
-customESSettings.addAll(Arrays.asList("es.client.class",
USERNAME_CONFIG_KEY, PWD_FILE_CONFIG_KEY));
-Settings.Builder settingsBuilder = Settings.builder();
-Map esSettings = getEsSettings(globalConfiguration);
-for (Map.Entry entry : esSettings.entrySet()) {
- String key = entry.getKey();
- String value = entry.getValue();
- if (!customESSettings.contains(key)) {
-settingsBuilder.put(key, value);
- }
-}
-settingsBuilder.put("cluster.name",
globalConfiguration.get("es.clustername"));
-settingsBuilder.put("client.transport.ping_timeout",
esSettings.getOrDefault("client.transport.ping_timeout","500s"));
-setXPackSecurityOrNone(settingsBuilder, esSettings);
-
-try {
- LOG.info("Number of available processors in Netty: {}",
NettyRuntimeWrapper.availableProcessors());
- // Netty sets available processors statically and if an attempt is
made to set it more than
- // once an IllegalStateException is thrown by
NettyRuntime.setAvailableProcessors(NettyRuntime.java:87)
- //
https://discuss.elastic.co/t/getting-availableprocessors-is-already-set-to-1-rejecting-1-illegalstateexception-exception/103082
- //
https://discuss.elastic.co/t/elasticsearch-5-4-1-availableprocessors-is-already-set/88036
- System.setProperty("es.set.netty.runtime.available.processors",
"false");
- TransportClient client =
createTransportClient(settingsBuilder.build(), esSettings);
- for (HostnamePort hp : getIps(globalConfiguration)) {
-client.addTransportAddress(
-new
InetSocketTransportAddress(InetAddress.getByName(hp.hostname), hp.port)
-);
- }
- return client;
-} catch (UnknownHostException exception) {
- throw new RuntimeException(exception);
-}
+ public static ElasticsearchClient getClient(Map
globalConfiguration) {
--- End diff --
I just found myself trying to instantiate a new `ElasticsearchClient` with
no idea how to go about doing that. Then I remember running across this method
in your PR. This seems really broadly useful.
Could we make this functionality more discoverable somehow? We could make
this a static method on `ElasticsearchClient`. Or just refer to this method in
the javadocs for `ElasticsearchClient`? Or some other way?
---
[jira] [Commented] (METRON-1834) Migrate Elasticsearch from TransportClient to new Java REST API
[
https://issues.apache.org/jira/browse/METRON-1834?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16662443#comment-16662443
]
ASF GitHub Bot commented on METRON-1834:
Github user nickwallen commented on a diff in the pull request:
https://github.com/apache/metron/pull/1242#discussion_r227847716
--- Diff:
metron-platform/metron-elasticsearch/src/main/java/org/apache/metron/elasticsearch/utils/ElasticsearchUtils.java
---
@@ -124,102 +129,146 @@ public static String getBaseIndexName(String
indexName) {
}
/**
- * Instantiates an Elasticsearch client based on es.client.class, if
set. Defaults to
- * org.elasticsearch.transport.client.PreBuiltTransportClient.
+ * Instantiates an Elasticsearch client
*
* @param globalConfiguration Metron global config
- * @return
+ * @return new es client
*/
- public static TransportClient getClient(Map
globalConfiguration) {
-Set customESSettings = new HashSet<>();
-customESSettings.addAll(Arrays.asList("es.client.class",
USERNAME_CONFIG_KEY, PWD_FILE_CONFIG_KEY));
-Settings.Builder settingsBuilder = Settings.builder();
-Map esSettings = getEsSettings(globalConfiguration);
-for (Map.Entry entry : esSettings.entrySet()) {
- String key = entry.getKey();
- String value = entry.getValue();
- if (!customESSettings.contains(key)) {
-settingsBuilder.put(key, value);
- }
-}
-settingsBuilder.put("cluster.name",
globalConfiguration.get("es.clustername"));
-settingsBuilder.put("client.transport.ping_timeout",
esSettings.getOrDefault("client.transport.ping_timeout","500s"));
-setXPackSecurityOrNone(settingsBuilder, esSettings);
-
-try {
- LOG.info("Number of available processors in Netty: {}",
NettyRuntimeWrapper.availableProcessors());
- // Netty sets available processors statically and if an attempt is
made to set it more than
- // once an IllegalStateException is thrown by
NettyRuntime.setAvailableProcessors(NettyRuntime.java:87)
- //
https://discuss.elastic.co/t/getting-availableprocessors-is-already-set-to-1-rejecting-1-illegalstateexception-exception/103082
- //
https://discuss.elastic.co/t/elasticsearch-5-4-1-availableprocessors-is-already-set/88036
- System.setProperty("es.set.netty.runtime.available.processors",
"false");
- TransportClient client =
createTransportClient(settingsBuilder.build(), esSettings);
- for (HostnamePort hp : getIps(globalConfiguration)) {
-client.addTransportAddress(
-new
InetSocketTransportAddress(InetAddress.getByName(hp.hostname), hp.port)
-);
- }
- return client;
-} catch (UnknownHostException exception) {
- throw new RuntimeException(exception);
-}
+ public static ElasticsearchClient getClient(Map
globalConfiguration) {
--- End diff --
I just found myself trying to instantiate a new `ElasticsearchClient` with
no idea how to go about doing that. Then I remember running across this method
in your PR. This seems really broadly useful.
Could we make this functionality more discoverable somehow? We could make
this a static method on `ElasticsearchClient`. Or just refer to this method in
the javadocs for `ElasticsearchClient`? Or some other way?
> Migrate Elasticsearch from TransportClient to new Java REST API
> ---
>
> Key: METRON-1834
> URL: https://issues.apache.org/jira/browse/METRON-1834
> Project: Metron
> Issue Type: Improvement
>Reporter: Michael Miklavcic
>Assignee: Michael Miklavcic
>Priority: Major
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (METRON-1563) Initial Feature Branch Work
[ https://issues.apache.org/jira/browse/METRON-1563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16662438#comment-16662438 ] ASF GitHub Bot commented on METRON-1563: Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/1014 ok, I updated the feature base to track master, and pulled that into this. I also added support for `:=` @cestella > Initial Feature Branch Work > --- > > Key: METRON-1563 > URL: https://issues.apache.org/jira/browse/METRON-1563 > Project: Metron > Issue Type: Sub-task >Reporter: Otto Fowler >Assignee: Otto Fowler >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] metron issue #1014: METRON-1563 : Base Stellar assign for feature branch
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/1014 ok, I updated the feature base to track master, and pulled that into this. I also added support for `:=` @cestella ---
[jira] [Commented] (METRON-1086) Create a Blockly-based user interface for Stellar
[ https://issues.apache.org/jira/browse/METRON-1086?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16662351#comment-16662351 ] ASF GitHub Bot commented on METRON-1086: Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/684 I actually think in the future this should be a stellar project / feature, then integrated after with metron. So that any ui can do stellar blockly > Create a Blockly-based user interface for Stellar > - > > Key: METRON-1086 > URL: https://issues.apache.org/jira/browse/METRON-1086 > Project: Metron > Issue Type: New Feature >Reporter: Ryan Merriman >Assignee: Ryan Merriman >Priority: Major > > Having a visual Stellar editor would be useful. Blockly is a Google library > for building visual programming editors: > https://developers.google.com/blockly/. This feature will integrate the two > and produce a reusable javascript widget for editing Stellar code. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] metron issue #684: DO NOT MERGE: METRON-1086: Create a Blockly-based user in...
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/684 I actually think in the future this should be a stellar project / feature, then integrated after with metron. So that any ui can do stellar blockly ---
[jira] [Commented] (METRON-1830) Re-implement Alerts dialog box without jQuery
[ https://issues.apache.org/jira/browse/METRON-1830?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16662338#comment-16662338 ] ASF GitHub Bot commented on METRON-1830: Github user tiborm commented on the issue: https://github.com/apache/metron/pull/1240 Thanks, Shane. +1 > Re-implement Alerts dialog box without jQuery > - > > Key: METRON-1830 > URL: https://issues.apache.org/jira/browse/METRON-1830 > Project: Metron > Issue Type: Bug >Reporter: Shane Ardell >Assignee: Shane Ardell >Priority: Minor > > Currently, the dialog box class in both UIs directly manipulates the DOM with > jQuery. This is problematic when using a framework like Angular because it > causes a disconnect with how Angular handles and tracks DOM changes. We also > don't take advantage of Angular's rendering engine. > The dialog box can and should be implemented as a component and a singleton > service (since we would never want to launch more than one modal at a time). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] metron issue #1240: METRON-1830: Re-implement Alerts dialog box without jQue...
Github user tiborm commented on the issue: https://github.com/apache/metron/pull/1240 Thanks, Shane. +1 ---
[jira] [Commented] (METRON-1563) Initial Feature Branch Work
[ https://issues.apache.org/jira/browse/METRON-1563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16662335#comment-16662335 ] ASF GitHub Bot commented on METRON-1563: Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/1014 I'll expand the assignment to include := in literal stellar, maybe that will bump some review > Initial Feature Branch Work > --- > > Key: METRON-1563 > URL: https://issues.apache.org/jira/browse/METRON-1563 > Project: Metron > Issue Type: Sub-task >Reporter: Otto Fowler >Assignee: Otto Fowler >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] metron issue #1014: METRON-1563 : Base Stellar assign for feature branch
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/1014 I'll expand the assignment to include := in literal stellar, maybe that will bump some review ---
[GitHub] metron issue #789: METRON-1233: Remove description of Global configuration f...
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/789 bump. Let's get a response on this @mmiklavc , @JonZeolla , @nickwallen ---
[jira] [Commented] (METRON-1086) Create a Blockly-based user interface for Stellar
[ https://issues.apache.org/jira/browse/METRON-1086?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16662331#comment-16662331 ] ASF GitHub Bot commented on METRON-1086: Github user merrimanr closed the pull request at: https://github.com/apache/metron/pull/684 > Create a Blockly-based user interface for Stellar > - > > Key: METRON-1086 > URL: https://issues.apache.org/jira/browse/METRON-1086 > Project: Metron > Issue Type: New Feature >Reporter: Ryan Merriman >Assignee: Ryan Merriman >Priority: Major > > Having a visual Stellar editor would be useful. Blockly is a Google library > for building visual programming editors: > https://developers.google.com/blockly/. This feature will integrate the two > and produce a reusable javascript widget for editing Stellar code. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] metron issue #684: DO NOT MERGE: METRON-1086: Create a Blockly-based user in...
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/684 We should close this until we are ready for a feature branch ---
[jira] [Commented] (METRON-1086) Create a Blockly-based user interface for Stellar
[ https://issues.apache.org/jira/browse/METRON-1086?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16662330#comment-16662330 ] ASF GitHub Bot commented on METRON-1086: Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/684 We should close this until we are ready for a feature branch > Create a Blockly-based user interface for Stellar > - > > Key: METRON-1086 > URL: https://issues.apache.org/jira/browse/METRON-1086 > Project: Metron > Issue Type: New Feature >Reporter: Ryan Merriman >Assignee: Ryan Merriman >Priority: Major > > Having a visual Stellar editor would be useful. Blockly is a Google library > for building visual programming editors: > https://developers.google.com/blockly/. This feature will integrate the two > and produce a reusable javascript widget for editing Stellar code. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] metron issue #526: Metron-846: Add E2E tests for metron management ui
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/526 please close this ---
[jira] [Commented] (METRON-1340) Improve e2e tests for metron alerts
[ https://issues.apache.org/jira/browse/METRON-1340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16662326#comment-16662326 ] ASF GitHub Bot commented on METRON-1340: Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/857 Please close this > Improve e2e tests for metron alerts > --- > > Key: METRON-1340 > URL: https://issues.apache.org/jira/browse/METRON-1340 > Project: Metron > Issue Type: Bug >Reporter: RaghuMitra >Assignee: RaghuMitra >Priority: Major > > Need to improve e2e tests in the following areas: > - Tests should not be flaky > - Remove the sleep ( This should implicitly make the tests run faster) > - Truncate HBase table 'metron_update' before starting the tests > - Improve the tests descriptions > - Run the tests headless if possible > - Check the node version and browser version before launching the tests > The expected behavior is that there are no intermittent failures. Acceptance > criteria: 5 consecutive runs without failures. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (METRON-1340) Improve e2e tests for metron alerts
[ https://issues.apache.org/jira/browse/METRON-1340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16662327#comment-16662327 ] ASF GitHub Bot commented on METRON-1340: Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/857 please close this > Improve e2e tests for metron alerts > --- > > Key: METRON-1340 > URL: https://issues.apache.org/jira/browse/METRON-1340 > Project: Metron > Issue Type: Bug >Reporter: RaghuMitra >Assignee: RaghuMitra >Priority: Major > > Need to improve e2e tests in the following areas: > - Tests should not be flaky > - Remove the sleep ( This should implicitly make the tests run faster) > - Truncate HBase table 'metron_update' before starting the tests > - Improve the tests descriptions > - Run the tests headless if possible > - Check the node version and browser version before launching the tests > The expected behavior is that there are no intermittent failures. Acceptance > criteria: 5 consecutive runs without failures. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] metron issue #857: METRON-1340: Improve e2e tests for metron alerts
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/857 please close this ---
[GitHub] metron issue #857: METRON-1340: Improve e2e tests for metron alerts
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/857 Please close this ---
[jira] [Created] (METRON-1842) PCAP UI: Implementing proper reactive event handling in PCAP
Tibor Meller created METRON-1842: Summary: PCAP UI: Implementing proper reactive event handling in PCAP Key: METRON-1842 URL: https://issues.apache.org/jira/browse/METRON-1842 Project: Metron Issue Type: Improvement Reporter: Tibor Meller The current implementation gives no chance to handle events in a reactive way. In my opinion, our real streams in PCAP panel are the following: * the stream of clicks on query button (leads to a request) * the stream of clicks on the cancel button (leads to a cancel request) The following three belongs to the PCAP service and consumed by the panel: * the stream of PCAP values (leads to rendering table) * a stream of status changes (leads to updating several UI controls) * a stream of percentages (leads to updating progress bar) We only want to unsubscribe from these streams when the user navigates away to the alert tab. If an error occurs, the job fails, user click cancel we like to continue observing them. Unfortunately, the current implementation creates new observables on every click instead of observing the stream of clicks. This leads to the fact that we unsubscribing and resubscribing all the time. Also makes this code hard to understand and explain any change. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (METRON-1841) PCAP UI: Refactor out PCAP querying logic from PCAP panel and move it to the service
Tibor Meller created METRON-1841: Summary: PCAP UI: Refactor out PCAP querying logic from PCAP panel and move it to the service Key: METRON-1841 URL: https://issues.apache.org/jira/browse/METRON-1841 Project: Metron Issue Type: Improvement Reporter: Tibor Meller Assignee: Tibor Meller In its current form, PCAP panel holds the actual querying logic of the PCAP service. Please make sure that nothing else than * the stream of PCAP values (leads to rendering table) * a stream of status changes (leads to updating several UI controls) * a stream of percentages (leads to updating progress bar) exposed by PCAP service and all the underlying logic encapsulated by it. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (METRON-1830) Re-implement Alerts dialog box without jQuery
[ https://issues.apache.org/jira/browse/METRON-1830?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16662285#comment-16662285 ] ASF GitHub Bot commented on METRON-1830: Github user sardell commented on the issue: https://github.com/apache/metron/pull/1240 @tiborm In regard to the 'Initial' state check, I was able to do what you suggested but only after switching from a BehaviorSubject to a Subject. I did this because Subjects do not return the current value on Subscription. It only triggers with the .next() call and returns the value you pass there. That was a really great suggestion on your part and made me realize I was using the wrong tool for the job. I also made changes based on the rest of your feedback, but they were straightforward and do not need further explanation here. Thanks for taking the time to review my PR! > Re-implement Alerts dialog box without jQuery > - > > Key: METRON-1830 > URL: https://issues.apache.org/jira/browse/METRON-1830 > Project: Metron > Issue Type: Bug >Reporter: Shane Ardell >Assignee: Shane Ardell >Priority: Minor > > Currently, the dialog box class in both UIs directly manipulates the DOM with > jQuery. This is problematic when using a framework like Angular because it > causes a disconnect with how Angular handles and tracks DOM changes. We also > don't take advantage of Angular's rendering engine. > The dialog box can and should be implemented as a component and a singleton > service (since we would never want to launch more than one modal at a time). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] metron issue #1240: METRON-1830: Re-implement Alerts dialog box without jQue...
Github user sardell commented on the issue: https://github.com/apache/metron/pull/1240 @tiborm In regard to the 'Initial' state check, I was able to do what you suggested but only after switching from a BehaviorSubject to a Subject. I did this because Subjects do not return the current value on Subscription. It only triggers with the .next() call and returns the value you pass there. That was a really great suggestion on your part and made me realize I was using the wrong tool for the job. I also made changes based on the rest of your feedback, but they were straightforward and do not need further explanation here. Thanks for taking the time to review my PR! ---
[jira] [Commented] (METRON-1834) Migrate Elasticsearch from TransportClient to new Java REST API
[ https://issues.apache.org/jira/browse/METRON-1834?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16662280#comment-16662280 ] ASF GitHub Bot commented on METRON-1834: Github user nickwallen commented on a diff in the pull request: https://github.com/apache/metron/pull/1242#discussion_r227783148 --- Diff: metron-platform/metron-elasticsearch/src/main/java/org/apache/metron/elasticsearch/dao/ElasticsearchColumnMetadataDao.java --- @@ -64,12 +62,12 @@ /** * An Elasticsearch administrative client. */ - private transient AdminClient adminClient; + private transient ElasticsearchClient adminClient; --- End diff -- Same in `ElasticsearchRetrieveLatestDao.transportClient`. > Migrate Elasticsearch from TransportClient to new Java REST API > --- > > Key: METRON-1834 > URL: https://issues.apache.org/jira/browse/METRON-1834 > Project: Metron > Issue Type: Improvement >Reporter: Michael Miklavcic >Assignee: Michael Miklavcic >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] metron pull request #1242: METRON-1834: Migrate Elasticsearch from Transport...
Github user nickwallen commented on a diff in the pull request: https://github.com/apache/metron/pull/1242#discussion_r227783148 --- Diff: metron-platform/metron-elasticsearch/src/main/java/org/apache/metron/elasticsearch/dao/ElasticsearchColumnMetadataDao.java --- @@ -64,12 +62,12 @@ /** * An Elasticsearch administrative client. */ - private transient AdminClient adminClient; + private transient ElasticsearchClient adminClient; --- End diff -- Same in `ElasticsearchRetrieveLatestDao.transportClient`. ---
[jira] [Commented] (METRON-1834) Migrate Elasticsearch from TransportClient to new Java REST API
[
https://issues.apache.org/jira/browse/METRON-1834?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16662275#comment-16662275
]
ASF GitHub Bot commented on METRON-1834:
Github user nickwallen commented on a diff in the pull request:
https://github.com/apache/metron/pull/1242#discussion_r227780851
--- Diff:
metron-platform/metron-elasticsearch/src/main/java/org/apache/metron/elasticsearch/client/ElasticsearchClient.java
---
@@ -0,0 +1,147 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.elasticsearch.client;
+
+import com.google.common.base.Joiner;
+import com.google.common.base.Splitter;
+import com.google.common.collect.Iterables;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import org.apache.commons.io.IOUtils;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.http.HttpEntity;
+import org.apache.http.entity.StringEntity;
+import org.apache.metron.common.utils.JSONUtils;
+import org.apache.metron.elasticsearch.utils.FieldMapping;
+import org.apache.metron.elasticsearch.utils.FieldProperties;
+import org.elasticsearch.client.Response;
+import org.elasticsearch.client.RestClient;
+import org.elasticsearch.client.RestHighLevelClient;
+
+/**
+ * Wrapper around the Elasticsearch REST clients. Exposes capabilities of
the low and high-level clients.
+ */
+public class ElasticsearchClient implements AutoCloseable{
--- End diff --
Actually, I don't see any uses of
`ElasticsearchClient.getLowLevelClient()`.
> Migrate Elasticsearch from TransportClient to new Java REST API
> ---
>
> Key: METRON-1834
> URL: https://issues.apache.org/jira/browse/METRON-1834
> Project: Metron
> Issue Type: Improvement
>Reporter: Michael Miklavcic
>Assignee: Michael Miklavcic
>Priority: Major
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] metron pull request #1242: METRON-1834: Migrate Elasticsearch from Transport...
Github user nickwallen commented on a diff in the pull request:
https://github.com/apache/metron/pull/1242#discussion_r227780851
--- Diff:
metron-platform/metron-elasticsearch/src/main/java/org/apache/metron/elasticsearch/client/ElasticsearchClient.java
---
@@ -0,0 +1,147 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.elasticsearch.client;
+
+import com.google.common.base.Joiner;
+import com.google.common.base.Splitter;
+import com.google.common.collect.Iterables;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import org.apache.commons.io.IOUtils;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.http.HttpEntity;
+import org.apache.http.entity.StringEntity;
+import org.apache.metron.common.utils.JSONUtils;
+import org.apache.metron.elasticsearch.utils.FieldMapping;
+import org.apache.metron.elasticsearch.utils.FieldProperties;
+import org.elasticsearch.client.Response;
+import org.elasticsearch.client.RestClient;
+import org.elasticsearch.client.RestHighLevelClient;
+
+/**
+ * Wrapper around the Elasticsearch REST clients. Exposes capabilities of
the low and high-level clients.
+ */
+public class ElasticsearchClient implements AutoCloseable{
--- End diff --
Actually, I don't see any uses of
`ElasticsearchClient.getLowLevelClient()`.
---
[jira] [Commented] (METRON-1834) Migrate Elasticsearch from TransportClient to new Java REST API
[ https://issues.apache.org/jira/browse/METRON-1834?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16662268#comment-16662268 ] ASF GitHub Bot commented on METRON-1834: Github user nickwallen commented on a diff in the pull request: https://github.com/apache/metron/pull/1242#discussion_r227774585 --- Diff: metron-platform/metron-elasticsearch/src/main/java/org/apache/metron/elasticsearch/dao/ElasticsearchColumnMetadataDao.java --- @@ -64,12 +62,12 @@ /** * An Elasticsearch administrative client. */ - private transient AdminClient adminClient; + private transient ElasticsearchClient adminClient; --- End diff -- Can we call this `client`,`elasticClient`, or `esClient` rather than retaining the original name `adminClient`, which doesn't make sense any longer? > Migrate Elasticsearch from TransportClient to new Java REST API > --- > > Key: METRON-1834 > URL: https://issues.apache.org/jira/browse/METRON-1834 > Project: Metron > Issue Type: Improvement >Reporter: Michael Miklavcic >Assignee: Michael Miklavcic >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (METRON-1834) Migrate Elasticsearch from TransportClient to new Java REST API
[
https://issues.apache.org/jira/browse/METRON-1834?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16662272#comment-16662272
]
ASF GitHub Bot commented on METRON-1834:
Github user nickwallen commented on a diff in the pull request:
https://github.com/apache/metron/pull/1242#discussion_r227778399
--- Diff:
metron-platform/metron-elasticsearch/src/main/java/org/apache/metron/elasticsearch/client/ElasticsearchClient.java
---
@@ -0,0 +1,147 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.elasticsearch.client;
+
+import com.google.common.base.Joiner;
+import com.google.common.base.Splitter;
+import com.google.common.collect.Iterables;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import org.apache.commons.io.IOUtils;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.http.HttpEntity;
+import org.apache.http.entity.StringEntity;
+import org.apache.metron.common.utils.JSONUtils;
+import org.apache.metron.elasticsearch.utils.FieldMapping;
+import org.apache.metron.elasticsearch.utils.FieldProperties;
+import org.elasticsearch.client.Response;
+import org.elasticsearch.client.RestClient;
+import org.elasticsearch.client.RestHighLevelClient;
+
+/**
+ * Wrapper around the Elasticsearch REST clients. Exposes capabilities of
the low and high-level clients.
+ */
+public class ElasticsearchClient implements AutoCloseable{
--- End diff --
Any wisdom or linkage to help developers understand when they should use
the high-level versus low-level client?
> Migrate Elasticsearch from TransportClient to new Java REST API
> ---
>
> Key: METRON-1834
> URL: https://issues.apache.org/jira/browse/METRON-1834
> Project: Metron
> Issue Type: Improvement
>Reporter: Michael Miklavcic
>Assignee: Michael Miklavcic
>Priority: Major
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (METRON-1834) Migrate Elasticsearch from TransportClient to new Java REST API
[ https://issues.apache.org/jira/browse/METRON-1834?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16662271#comment-16662271 ] ASF GitHub Bot commented on METRON-1834: Github user nickwallen commented on a diff in the pull request: https://github.com/apache/metron/pull/1242#discussion_r227776804 --- Diff: metron-platform/metron-elasticsearch/src/main/java/org/apache/metron/elasticsearch/writer/ElasticsearchWriter.java --- @@ -76,7 +76,8 @@ public BulkWriterResponse write(String sensorType, WriterConfiguration configura FieldNameConverter fieldNameConverter = FieldNameConverters.create(sensorType, configurations); final String indexPostfix = dateFormat.format(new Date()); -BulkRequestBuilder bulkRequest = client.prepareBulk(); +BulkRequest bulkRequest = new BulkRequest(); +//BulkRequestBuilder bulkRequest = client.prepareBulk(); --- End diff -- Can we remove this commented out line? > Migrate Elasticsearch from TransportClient to new Java REST API > --- > > Key: METRON-1834 > URL: https://issues.apache.org/jira/browse/METRON-1834 > Project: Metron > Issue Type: Improvement >Reporter: Michael Miklavcic >Assignee: Michael Miklavcic >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (METRON-1834) Migrate Elasticsearch from TransportClient to new Java REST API
[
https://issues.apache.org/jira/browse/METRON-1834?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16662269#comment-16662269
]
ASF GitHub Bot commented on METRON-1834:
Github user nickwallen commented on a diff in the pull request:
https://github.com/apache/metron/pull/1242#discussion_r227776412
--- Diff:
metron-platform/metron-elasticsearch/src/main/java/org/apache/metron/elasticsearch/utils/FieldMapping.java
---
@@ -0,0 +1,29 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.elasticsearch.utils;
+
+import org.apache.commons.collections4.map.AbstractMapDecorator;
+
+import java.util.HashMap;
+
+public class FieldMapping extends AbstractMapDecorator{
--- End diff --
What is this? Can you add javadoc?
> Migrate Elasticsearch from TransportClient to new Java REST API
> ---
>
> Key: METRON-1834
> URL: https://issues.apache.org/jira/browse/METRON-1834
> Project: Metron
> Issue Type: Improvement
>Reporter: Michael Miklavcic
>Assignee: Michael Miklavcic
>Priority: Major
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (METRON-1834) Migrate Elasticsearch from TransportClient to new Java REST API
[
https://issues.apache.org/jira/browse/METRON-1834?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16662273#comment-16662273
]
ASF GitHub Bot commented on METRON-1834:
Github user nickwallen commented on a diff in the pull request:
https://github.com/apache/metron/pull/1242#discussion_r227778927
--- Diff:
metron-platform/metron-elasticsearch/src/main/java/org/apache/metron/elasticsearch/client/ElasticsearchClient.java
---
@@ -0,0 +1,147 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.elasticsearch.client;
+
+import com.google.common.base.Joiner;
+import com.google.common.base.Splitter;
+import com.google.common.collect.Iterables;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import org.apache.commons.io.IOUtils;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.http.HttpEntity;
+import org.apache.http.entity.StringEntity;
+import org.apache.metron.common.utils.JSONUtils;
+import org.apache.metron.elasticsearch.utils.FieldMapping;
+import org.apache.metron.elasticsearch.utils.FieldProperties;
+import org.elasticsearch.client.Response;
+import org.elasticsearch.client.RestClient;
+import org.elasticsearch.client.RestHighLevelClient;
+
+/**
+ * Wrapper around the Elasticsearch REST clients. Exposes capabilities of
the low and high-level clients.
+ */
+public class ElasticsearchClient implements AutoCloseable{
+ private RestClient lowLevelClient;
+ private RestHighLevelClient highLevelClient;
+
+ public ElasticsearchClient(RestClient lowLevelClient,
RestHighLevelClient highLevelClient) {
+this.lowLevelClient = lowLevelClient;
+this.highLevelClient = highLevelClient;
+ }
+
+ public RestClient getLowLevelClient() {
+return lowLevelClient;
+ }
+
+ public RestHighLevelClient getHighLevelClient() {
+return highLevelClient;
+ }
+
+ @Override
+ public void close() throws IOException {
+if(lowLevelClient != null) {
+ lowLevelClient.close();
+}
+ }
+
+ public void putMapping(String index, String type, String source) throws
IOException {
+HttpEntity entity = new StringEntity(source);
+Response response = lowLevelClient.performRequest("PUT"
+, "/" + index + "/_mapping/" + type
+, Collections.emptyMap()
+, entity
+);
+
+if(response.getStatusLine().getStatusCode() != 200) {
+ String responseStr =
IOUtils.toString(response.getEntity().getContent());
+ throw new IllegalStateException("Got a " +
response.getStatusLine().getStatusCode() + " due to " + responseStr);
+}
+ }
+
+ public String[] getIndices() throws IOException {
--- End diff --
Are there unit tests specifically for this class? I don't see any, but
maybe I missed.
> Migrate Elasticsearch from TransportClient to new Java REST API
> ---
>
> Key: METRON-1834
> URL: https://issues.apache.org/jira/browse/METRON-1834
> Project: Metron
> Issue Type: Improvement
>Reporter: Michael Miklavcic
>Assignee: Michael Miklavcic
>Priority: Major
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (METRON-1834) Migrate Elasticsearch from TransportClient to new Java REST API
[
https://issues.apache.org/jira/browse/METRON-1834?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16662270#comment-16662270
]
ASF GitHub Bot commented on METRON-1834:
Github user nickwallen commented on a diff in the pull request:
https://github.com/apache/metron/pull/1242#discussion_r227773650
--- Diff:
metron-platform/metron-elasticsearch/src/main/java/org/apache/metron/elasticsearch/config/ElasticsearchClientOptions.java
---
@@ -0,0 +1,60 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.metron.elasticsearch.config;
+
+import org.apache.metron.common.configuration.ConfigOption;
+
+public enum ElasticsearchClientOptions implements ConfigOption {
--- End diff --
What is the difference between `ElasticsearchClientOptions` and
`ElasticsearchClientConfig`? Can you spell that out more clearly in the
javadoc, if needed.
> Migrate Elasticsearch from TransportClient to new Java REST API
> ---
>
> Key: METRON-1834
> URL: https://issues.apache.org/jira/browse/METRON-1834
> Project: Metron
> Issue Type: Improvement
>Reporter: Michael Miklavcic
>Assignee: Michael Miklavcic
>Priority: Major
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] metron pull request #1242: METRON-1834: Migrate Elasticsearch from Transport...
Github user nickwallen commented on a diff in the pull request:
https://github.com/apache/metron/pull/1242#discussion_r227778399
--- Diff:
metron-platform/metron-elasticsearch/src/main/java/org/apache/metron/elasticsearch/client/ElasticsearchClient.java
---
@@ -0,0 +1,147 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.elasticsearch.client;
+
+import com.google.common.base.Joiner;
+import com.google.common.base.Splitter;
+import com.google.common.collect.Iterables;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import org.apache.commons.io.IOUtils;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.http.HttpEntity;
+import org.apache.http.entity.StringEntity;
+import org.apache.metron.common.utils.JSONUtils;
+import org.apache.metron.elasticsearch.utils.FieldMapping;
+import org.apache.metron.elasticsearch.utils.FieldProperties;
+import org.elasticsearch.client.Response;
+import org.elasticsearch.client.RestClient;
+import org.elasticsearch.client.RestHighLevelClient;
+
+/**
+ * Wrapper around the Elasticsearch REST clients. Exposes capabilities of
the low and high-level clients.
+ */
+public class ElasticsearchClient implements AutoCloseable{
--- End diff --
Any wisdom or linkage to help developers understand when they should use
the high-level versus low-level client?
---
[GitHub] metron pull request #1242: METRON-1834: Migrate Elasticsearch from Transport...
Github user nickwallen commented on a diff in the pull request:
https://github.com/apache/metron/pull/1242#discussion_r227776412
--- Diff:
metron-platform/metron-elasticsearch/src/main/java/org/apache/metron/elasticsearch/utils/FieldMapping.java
---
@@ -0,0 +1,29 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.elasticsearch.utils;
+
+import org.apache.commons.collections4.map.AbstractMapDecorator;
+
+import java.util.HashMap;
+
+public class FieldMapping extends AbstractMapDecorator{
--- End diff --
What is this? Can you add javadoc?
---
[GitHub] metron pull request #1242: METRON-1834: Migrate Elasticsearch from Transport...
Github user nickwallen commented on a diff in the pull request:
https://github.com/apache/metron/pull/1242#discussion_r227778927
--- Diff:
metron-platform/metron-elasticsearch/src/main/java/org/apache/metron/elasticsearch/client/ElasticsearchClient.java
---
@@ -0,0 +1,147 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.elasticsearch.client;
+
+import com.google.common.base.Joiner;
+import com.google.common.base.Splitter;
+import com.google.common.collect.Iterables;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import org.apache.commons.io.IOUtils;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.http.HttpEntity;
+import org.apache.http.entity.StringEntity;
+import org.apache.metron.common.utils.JSONUtils;
+import org.apache.metron.elasticsearch.utils.FieldMapping;
+import org.apache.metron.elasticsearch.utils.FieldProperties;
+import org.elasticsearch.client.Response;
+import org.elasticsearch.client.RestClient;
+import org.elasticsearch.client.RestHighLevelClient;
+
+/**
+ * Wrapper around the Elasticsearch REST clients. Exposes capabilities of
the low and high-level clients.
+ */
+public class ElasticsearchClient implements AutoCloseable{
+ private RestClient lowLevelClient;
+ private RestHighLevelClient highLevelClient;
+
+ public ElasticsearchClient(RestClient lowLevelClient,
RestHighLevelClient highLevelClient) {
+this.lowLevelClient = lowLevelClient;
+this.highLevelClient = highLevelClient;
+ }
+
+ public RestClient getLowLevelClient() {
+return lowLevelClient;
+ }
+
+ public RestHighLevelClient getHighLevelClient() {
+return highLevelClient;
+ }
+
+ @Override
+ public void close() throws IOException {
+if(lowLevelClient != null) {
+ lowLevelClient.close();
+}
+ }
+
+ public void putMapping(String index, String type, String source) throws
IOException {
+HttpEntity entity = new StringEntity(source);
+Response response = lowLevelClient.performRequest("PUT"
+, "/" + index + "/_mapping/" + type
+, Collections.emptyMap()
+, entity
+);
+
+if(response.getStatusLine().getStatusCode() != 200) {
+ String responseStr =
IOUtils.toString(response.getEntity().getContent());
+ throw new IllegalStateException("Got a " +
response.getStatusLine().getStatusCode() + " due to " + responseStr);
+}
+ }
+
+ public String[] getIndices() throws IOException {
--- End diff --
Are there unit tests specifically for this class? I don't see any, but
maybe I missed.
---
[GitHub] metron pull request #1242: METRON-1834: Migrate Elasticsearch from Transport...
Github user nickwallen commented on a diff in the pull request: https://github.com/apache/metron/pull/1242#discussion_r227774585 --- Diff: metron-platform/metron-elasticsearch/src/main/java/org/apache/metron/elasticsearch/dao/ElasticsearchColumnMetadataDao.java --- @@ -64,12 +62,12 @@ /** * An Elasticsearch administrative client. */ - private transient AdminClient adminClient; + private transient ElasticsearchClient adminClient; --- End diff -- Can we call this `client`,`elasticClient`, or `esClient` rather than retaining the original name `adminClient`, which doesn't make sense any longer? ---
[GitHub] metron pull request #1242: METRON-1834: Migrate Elasticsearch from Transport...
Github user nickwallen commented on a diff in the pull request:
https://github.com/apache/metron/pull/1242#discussion_r227773650
--- Diff:
metron-platform/metron-elasticsearch/src/main/java/org/apache/metron/elasticsearch/config/ElasticsearchClientOptions.java
---
@@ -0,0 +1,60 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.metron.elasticsearch.config;
+
+import org.apache.metron.common.configuration.ConfigOption;
+
+public enum ElasticsearchClientOptions implements ConfigOption {
--- End diff --
What is the difference between `ElasticsearchClientOptions` and
`ElasticsearchClientConfig`? Can you spell that out more clearly in the
javadoc, if needed.
---
[GitHub] metron pull request #1242: METRON-1834: Migrate Elasticsearch from Transport...
Github user nickwallen commented on a diff in the pull request: https://github.com/apache/metron/pull/1242#discussion_r227776804 --- Diff: metron-platform/metron-elasticsearch/src/main/java/org/apache/metron/elasticsearch/writer/ElasticsearchWriter.java --- @@ -76,7 +76,8 @@ public BulkWriterResponse write(String sensorType, WriterConfiguration configura FieldNameConverter fieldNameConverter = FieldNameConverters.create(sensorType, configurations); final String indexPostfix = dateFormat.format(new Date()); -BulkRequestBuilder bulkRequest = client.prepareBulk(); +BulkRequest bulkRequest = new BulkRequest(); +//BulkRequestBuilder bulkRequest = client.prepareBulk(); --- End diff -- Can we remove this commented out line? ---
[GitHub] metron pull request #1242: METRON-1834: Migrate Elasticsearch from Transport...
Github user nickwallen commented on a diff in the pull request:
https://github.com/apache/metron/pull/1242#discussion_r227776572
--- Diff:
metron-platform/metron-elasticsearch/src/main/java/org/apache/metron/elasticsearch/utils/FieldProperties.java
---
@@ -0,0 +1,33 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.elasticsearch.utils;
+
+import org.apache.commons.collections4.map.AbstractMapDecorator;
+
+import java.util.HashMap;
+import java.util.Map;
+
+public class FieldProperties extends AbstractMapDecorator {
--- End diff --
Can you add javadoc on the class?
---
[jira] [Commented] (METRON-1801) Allow Customization of Elasticsearch Document ID
[ https://issues.apache.org/jira/browse/METRON-1801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16662230#comment-16662230 ] ASF GitHub Bot commented on METRON-1801: Github user nickwallen commented on the issue: https://github.com/apache/metron/pull/1218 This change was reverted [here](https://github.com/apache/metron/commit/0e037edad913955d3b6754ca9cf42b329cd84160). A new pull request will be opened with the functionality. See also [this mailing list thread](https://github.com/apache/metron/commit/0e037edad913955d3b6754ca9cf42b329cd84160). > Allow Customization of Elasticsearch Document ID > > > Key: METRON-1801 > URL: https://issues.apache.org/jira/browse/METRON-1801 > Project: Metron > Issue Type: Sub-task >Reporter: Nick Allen >Assignee: Nick Allen >Priority: Major > > The user should be able to customize the document ID that is set by the > client when indexing documents into Elasticsearch. The user should be able > to use the Metron GUID, define their own custom document ID, or choose to not > have the document ID set by the client. > > This task covers Elasticsearch only. An additional task should be created to > cover Solr. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] metron issue #1218: METRON-1801 Allow Customization of Elasticsearch Documen...
Github user nickwallen commented on the issue: https://github.com/apache/metron/pull/1218 This change was reverted [here](https://github.com/apache/metron/commit/0e037edad913955d3b6754ca9cf42b329cd84160). A new pull request will be opened with the functionality. See also [this mailing list thread](https://github.com/apache/metron/commit/0e037edad913955d3b6754ca9cf42b329cd84160). ---
[jira] [Commented] (METRON-1823) Refactor Elasticsearch Configuration Settings
[ https://issues.apache.org/jira/browse/METRON-1823?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16662227#comment-16662227 ] ASF GitHub Bot commented on METRON-1823: Github user nickwallen closed the pull request at: https://github.com/apache/metron/pull/1235 > Refactor Elasticsearch Configuration Settings > - > > Key: METRON-1823 > URL: https://issues.apache.org/jira/browse/METRON-1823 > Project: Metron > Issue Type: Bug >Reporter: Nick Allen >Priority: Major > > The various settings for the ElasticsearchWriter like 'es.ip', 'es.port', etc > are not all defined as constants in a single place. Refactor these so they > are all defined in a single location. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] metron pull request #1235: METRON-1823 Refactor Elasticsearch Configuration ...
Github user nickwallen closed the pull request at: https://github.com/apache/metron/pull/1235 ---
