[jira] [Commented] (METRON-1056) Get field types from Elasticsearch
[ https://issues.apache.org/jira/browse/METRON-1056?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16109668#comment-16109668 ] ASF GitHub Bot commented on METRON-1056: Github user asfgit closed the pull request at: https://github.com/apache/metron/pull/662 > Get field types from Elasticsearch > -- > > Key: METRON-1056 > URL: https://issues.apache.org/jira/browse/METRON-1056 > Project: Metron > Issue Type: Improvement >Reporter: Ryan Merriman > > We need a way to retrieve field types from Elasticsearch for a specified list > of indices. This will also be exposed through our indexing DAO layer with > Elasticsearch being the initial implementation. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (METRON-1056) Get field types from Elasticsearch
[ https://issues.apache.org/jira/browse/METRON-1056?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16109407#comment-16109407 ] ASF GitHub Bot commented on METRON-1056: Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/662 +1 > Get field types from Elasticsearch > -- > > Key: METRON-1056 > URL: https://issues.apache.org/jira/browse/METRON-1056 > Project: Metron > Issue Type: Improvement >Reporter: Ryan Merriman > > We need a way to retrieve field types from Elasticsearch for a specified list > of indices. This will also be exposed through our indexing DAO layer with > Elasticsearch being the initial implementation. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (METRON-1056) Get field types from Elasticsearch
[ https://issues.apache.org/jira/browse/METRON-1056?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16109378#comment-16109378 ] ASF GitHub Bot commented on METRON-1056: Github user james-sirota commented on a diff in the pull request: https://github.com/apache/metron/pull/662#discussion_r130675495 --- Diff: metron-platform/metron-indexing/src/test/java/org/apache/metron/indexing/dao/IndexingDaoIntegrationTest.java --- @@ -27,28 +28,32 @@ import org.json.simple.parser.ParseException; import org.junit.*; +import java.io.IOException; +import java.util.Arrays; +import java.util.Collections; import java.util.List; +import java.util.Map; --- End diff -- @ottobackwards are you ok with this PR? Can this get a +1 from you? > Get field types from Elasticsearch > -- > > Key: METRON-1056 > URL: https://issues.apache.org/jira/browse/METRON-1056 > Project: Metron > Issue Type: Improvement >Reporter: Ryan Merriman > > We need a way to retrieve field types from Elasticsearch for a specified list > of indices. This will also be exposed through our indexing DAO layer with > Elasticsearch being the initial implementation. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (METRON-1056) Get field types from Elasticsearch
[
https://issues.apache.org/jira/browse/METRON-1056?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16099706#comment-16099706
]
ASF GitHub Bot commented on METRON-1056:
Github user simonellistonball commented on a diff in the pull request:
https://github.com/apache/metron/pull/662#discussion_r129241629
--- Diff:
metron-interface/metron-rest/src/main/java/org/apache/metron/rest/controller/SearchController.java
---
@@ -45,4 +49,18 @@
ResponseEntity search(final @ApiParam(name =
"searchRequest", value = "Search request", required = true) @RequestBody
SearchRequest searchRequest) throws RestException {
return new ResponseEntity<>(searchService.search(searchRequest),
HttpStatus.OK);
}
+
--- End diff --
Feels like any asynchronicity should be at the service layer anyway for
other things to potentially benefit from, of course that would bubble up here,
but do we really expect the REST server to have to worry about high load?
> Get field types from Elasticsearch
> --
>
> Key: METRON-1056
> URL: https://issues.apache.org/jira/browse/METRON-1056
> Project: Metron
> Issue Type: Improvement
>Reporter: Ryan Merriman
>
> We need a way to retrieve field types from Elasticsearch for a specified list
> of indices. This will also be exposed through our indexing DAO layer with
> Elasticsearch being the initial implementation.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (METRON-1056) Get field types from Elasticsearch
[
https://issues.apache.org/jira/browse/METRON-1056?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16099136#comment-16099136
]
ASF GitHub Bot commented on METRON-1056:
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/metron/pull/662#discussion_r129156078
--- Diff:
metron-platform/metron-indexing/src/main/java/org/apache/metron/indexing/dao/search/FieldType.java
---
@@ -0,0 +1,52 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.indexing.dao.search;
+
+import com.fasterxml.jackson.annotation.JsonProperty;
+
+public enum FieldType {
+ @JsonProperty("string")
+ STRING("string"),
+ @JsonProperty("ip")
+ IP("ip"),
+ @JsonProperty("integer")
+ INTEGER("integer"),
+ @JsonProperty("long")
+ LONG("long"),
+ @JsonProperty("date")
+ DATE("date"),
+ @JsonProperty("float")
+ FLOAT("float"),
+ @JsonProperty("double")
+ DOUBLE("double"),
+ @JsonProperty("boolean")
+ BOOLEAN("boolean"),
+ @JsonProperty("other")
--- End diff --
Just to be clear, I don't think this should hold up this PR, but that we in
a general sense need to go into more detail.
> Get field types from Elasticsearch
> --
>
> Key: METRON-1056
> URL: https://issues.apache.org/jira/browse/METRON-1056
> Project: Metron
> Issue Type: Improvement
>Reporter: Ryan Merriman
>
> We need a way to retrieve field types from Elasticsearch for a specified list
> of indices. This will also be exposed through our indexing DAO layer with
> Elasticsearch being the initial implementation.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (METRON-1056) Get field types from Elasticsearch
[
https://issues.apache.org/jira/browse/METRON-1056?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16099117#comment-16099117
]
ASF GitHub Bot commented on METRON-1056:
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/metron/pull/662#discussion_r129152107
--- Diff:
metron-platform/metron-indexing/src/main/java/org/apache/metron/indexing/dao/search/FieldType.java
---
@@ -0,0 +1,52 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.indexing.dao.search;
+
+import com.fasterxml.jackson.annotation.JsonProperty;
+
+public enum FieldType {
+ @JsonProperty("string")
+ STRING("string"),
+ @JsonProperty("ip")
+ IP("ip"),
+ @JsonProperty("integer")
+ INTEGER("integer"),
+ @JsonProperty("long")
+ LONG("long"),
+ @JsonProperty("date")
+ DATE("date"),
+ @JsonProperty("float")
+ FLOAT("float"),
+ @JsonProperty("double")
+ DOUBLE("double"),
+ @JsonProperty("boolean")
+ BOOLEAN("boolean"),
+ @JsonProperty("other")
--- End diff --
At some point, if we are going to abstract these other systems, we need to
document our abstractions. I am not sure where though. Like stellar, the rest
api documentation prob. needs to be broken out with some better detail for
specific entry points, including references to outside systems the user is
expected to know.
> Get field types from Elasticsearch
> --
>
> Key: METRON-1056
> URL: https://issues.apache.org/jira/browse/METRON-1056
> Project: Metron
> Issue Type: Improvement
>Reporter: Ryan Merriman
>
> We need a way to retrieve field types from Elasticsearch for a specified list
> of indices. This will also be exposed through our indexing DAO layer with
> Elasticsearch being the initial implementation.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (METRON-1056) Get field types from Elasticsearch
[ https://issues.apache.org/jira/browse/METRON-1056?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16099091#comment-16099091 ] ASF GitHub Bot commented on METRON-1056: Github user merrimanr commented on a diff in the pull request: https://github.com/apache/metron/pull/662#discussion_r129147554 --- Diff: metron-platform/metron-indexing/src/test/java/org/apache/metron/indexing/dao/IndexingDaoIntegrationTest.java --- @@ -27,28 +28,32 @@ import org.json.simple.parser.ParseException; import org.junit.*; +import java.io.IOException; +import java.util.Arrays; +import java.util.Collections; import java.util.List; +import java.util.Map; --- End diff -- "location_point" is of type geo_field so it's falls into the "other" category. > Get field types from Elasticsearch > -- > > Key: METRON-1056 > URL: https://issues.apache.org/jira/browse/METRON-1056 > Project: Metron > Issue Type: Improvement >Reporter: Ryan Merriman > > We need a way to retrieve field types from Elasticsearch for a specified list > of indices. This will also be exposed through our indexing DAO layer with > Elasticsearch being the initial implementation. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (METRON-1056) Get field types from Elasticsearch
[
https://issues.apache.org/jira/browse/METRON-1056?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16099089#comment-16099089
]
ASF GitHub Bot commented on METRON-1056:
Github user merrimanr commented on a diff in the pull request:
https://github.com/apache/metron/pull/662#discussion_r129147257
--- Diff:
metron-platform/metron-indexing/src/main/java/org/apache/metron/indexing/dao/search/FieldType.java
---
@@ -0,0 +1,52 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.indexing.dao.search;
+
+import com.fasterxml.jackson.annotation.JsonProperty;
+
+public enum FieldType {
+ @JsonProperty("string")
+ STRING("string"),
+ @JsonProperty("ip")
+ IP("ip"),
+ @JsonProperty("integer")
+ INTEGER("integer"),
+ @JsonProperty("long")
+ LONG("long"),
+ @JsonProperty("date")
+ DATE("date"),
+ @JsonProperty("float")
+ FLOAT("float"),
+ @JsonProperty("double")
+ DOUBLE("double"),
+ @JsonProperty("boolean")
+ BOOLEAN("boolean"),
+ @JsonProperty("other")
--- End diff --
"other" just means it's not one of the other simple types in the enum. The
Elasticsearch Java API returns the search result as a Map type
which is also the return type of the controller so it's up to Jackson to
serialize it. We are not doing anything special to map values to types when
returning search results. So far I haven't seen Jackson have any problems with
the types being returned. From what I can tell Elasticsearch mostly sticks to
simple types for the values.
> Get field types from Elasticsearch
> --
>
> Key: METRON-1056
> URL: https://issues.apache.org/jira/browse/METRON-1056
> Project: Metron
> Issue Type: Improvement
>Reporter: Ryan Merriman
>
> We need a way to retrieve field types from Elasticsearch for a specified list
> of indices. This will also be exposed through our indexing DAO layer with
> Elasticsearch being the initial implementation.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (METRON-1056) Get field types from Elasticsearch
[
https://issues.apache.org/jira/browse/METRON-1056?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16099083#comment-16099083
]
ASF GitHub Bot commented on METRON-1056:
Github user merrimanr commented on a diff in the pull request:
https://github.com/apache/metron/pull/662#discussion_r129145507
--- Diff:
metron-interface/metron-rest/src/main/java/org/apache/metron/rest/controller/SearchController.java
---
@@ -45,4 +49,18 @@
ResponseEntity search(final @ApiParam(name =
"searchRequest", value = "Search request", required = true) @RequestBody
SearchRequest searchRequest) throws RestException {
return new ResponseEntity<>(searchService.search(searchRequest),
HttpStatus.OK);
}
+
--- End diff --
It's a legitimate concern but I think synchronous is ok here because
Elasticsearch is intended to provide results in real-time. The http javascript
APIs we use are asynchronous so it shouldn't make a difference in the UI. If
this were a batch operation (Hive, Map Reduce) I would agree that it should be
an async call.
> Get field types from Elasticsearch
> --
>
> Key: METRON-1056
> URL: https://issues.apache.org/jira/browse/METRON-1056
> Project: Metron
> Issue Type: Improvement
>Reporter: Ryan Merriman
>
> We need a way to retrieve field types from Elasticsearch for a specified list
> of indices. This will also be exposed through our indexing DAO layer with
> Elasticsearch being the initial implementation.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (METRON-1056) Get field types from Elasticsearch
[ https://issues.apache.org/jira/browse/METRON-1056?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16099072#comment-16099072 ] ASF GitHub Bot commented on METRON-1056: Github user merrimanr commented on a diff in the pull request: https://github.com/apache/metron/pull/662#discussion_r129143530 --- Diff: metron-interface/metron-rest/README.md --- @@ -353,6 +355,20 @@ Request and Response objects are JSON formatted. The JSON schemas are available * searchRequest - Search request * Returns: * 200 - Search results + --- End diff -- This isn't intended to be the source of truth for types that we support. I would think any data type someone may need should be supported in some way. This is restricted to (mostly) simple types and is driven by the need to format simple values in different ways. Anything else besides simple types is categorized as "other". If at any point there is a need to include more types this can easily be extended. The Elasticsearch API gives you either a number or string (only string in the case of the REST API) so we need more information about the type in Elasticsearch to format it correctly. Solr is similar. I will take another pass at the documentation to make this more obvious. > Get field types from Elasticsearch > -- > > Key: METRON-1056 > URL: https://issues.apache.org/jira/browse/METRON-1056 > Project: Metron > Issue Type: Improvement >Reporter: Ryan Merriman > > We need a way to retrieve field types from Elasticsearch for a specified list > of indices. This will also be exposed through our indexing DAO layer with > Elasticsearch being the initial implementation. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (METRON-1056) Get field types from Elasticsearch
[ https://issues.apache.org/jira/browse/METRON-1056?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16098508#comment-16098508 ] ASF GitHub Bot commented on METRON-1056: Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/metron/pull/662#discussion_r129052380 --- Diff: metron-interface/metron-rest/README.md --- @@ -353,6 +355,20 @@ Request and Response objects are JSON formatted. The JSON schemas are available * searchRequest - Search request * Returns: * 200 - Search results + --- End diff -- Such as complex types. This comes through here, because we are creating our own Field Type Enums > Get field types from Elasticsearch > -- > > Key: METRON-1056 > URL: https://issues.apache.org/jira/browse/METRON-1056 > Project: Metron > Issue Type: Improvement >Reporter: Ryan Merriman > > We need a way to retrieve field types from Elasticsearch for a specified list > of indices. This will also be exposed through our indexing DAO layer with > Elasticsearch being the initial implementation. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (METRON-1056) Get field types from Elasticsearch
[ https://issues.apache.org/jira/browse/METRON-1056?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16098505#comment-16098505 ] ASF GitHub Bot commented on METRON-1056: Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/metron/pull/662#discussion_r129051676 --- Diff: metron-interface/metron-rest/README.md --- @@ -353,6 +355,20 @@ Request and Response objects are JSON formatted. The JSON schemas are available * searchRequest - Search request * Returns: * 200 - Search results + --- End diff -- This is not a problem for this PR, but rather the documentation in general. The actual TYPES returned from the rest calls are not documented at all. > Get field types from Elasticsearch > -- > > Key: METRON-1056 > URL: https://issues.apache.org/jira/browse/METRON-1056 > Project: Metron > Issue Type: Improvement >Reporter: Ryan Merriman > > We need a way to retrieve field types from Elasticsearch for a specified list > of indices. This will also be exposed through our indexing DAO layer with > Elasticsearch being the initial implementation. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (METRON-1056) Get field types from Elasticsearch
[
https://issues.apache.org/jira/browse/METRON-1056?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16098503#comment-16098503
]
ASF GitHub Bot commented on METRON-1056:
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/metron/pull/662#discussion_r129054322
--- Diff:
metron-platform/metron-indexing/src/main/java/org/apache/metron/indexing/dao/search/FieldType.java
---
@@ -0,0 +1,52 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.indexing.dao.search;
+
+import com.fasterxml.jackson.annotation.JsonProperty;
+
+public enum FieldType {
+ @JsonProperty("string")
+ STRING("string"),
+ @JsonProperty("ip")
+ IP("ip"),
+ @JsonProperty("integer")
+ INTEGER("integer"),
+ @JsonProperty("long")
+ LONG("long"),
+ @JsonProperty("date")
+ DATE("date"),
+ @JsonProperty("float")
+ FLOAT("float"),
+ @JsonProperty("double")
+ DOUBLE("double"),
+ @JsonProperty("boolean")
+ BOOLEAN("boolean"),
+ @JsonProperty("other")
--- End diff --
What does other mean? Is it mapped to string in return values..
> Get field types from Elasticsearch
> --
>
> Key: METRON-1056
> URL: https://issues.apache.org/jira/browse/METRON-1056
> Project: Metron
> Issue Type: Improvement
>Reporter: Ryan Merriman
>
> We need a way to retrieve field types from Elasticsearch for a specified list
> of indices. This will also be exposed through our indexing DAO layer with
> Elasticsearch being the initial implementation.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (METRON-1056) Get field types from Elasticsearch
[ https://issues.apache.org/jira/browse/METRON-1056?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16098507#comment-16098507 ] ASF GitHub Bot commented on METRON-1056: Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/metron/pull/662#discussion_r129055024 --- Diff: metron-platform/metron-indexing/src/test/java/org/apache/metron/indexing/dao/IndexingDaoIntegrationTest.java --- @@ -27,28 +28,32 @@ import org.json.simple.parser.ParseException; import org.junit.*; +import java.io.IOException; +import java.util.Arrays; +import java.util.Collections; import java.util.List; +import java.util.Map; --- End diff -- How do we test unknown > Get field types from Elasticsearch > -- > > Key: METRON-1056 > URL: https://issues.apache.org/jira/browse/METRON-1056 > Project: Metron > Issue Type: Improvement >Reporter: Ryan Merriman > > We need a way to retrieve field types from Elasticsearch for a specified list > of indices. This will also be exposed through our indexing DAO layer with > Elasticsearch being the initial implementation. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (METRON-1056) Get field types from Elasticsearch
[ https://issues.apache.org/jira/browse/METRON-1056?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16098449#comment-16098449 ] ASF GitHub Bot commented on METRON-1056: Github user merrimanr commented on the issue: https://github.com/apache/metron/pull/662 Sure @ottobackwards. There is a similar api for Solr that will return field types for a list of collections. When we add Solr as an indexing option we will create a "SolrDao" class that will implement the "IndexDao.getColumnMetadata(List indices)" and "IndexDao.getCommonColumnMetadata(List indices)" methods. > Get field types from Elasticsearch > -- > > Key: METRON-1056 > URL: https://issues.apache.org/jira/browse/METRON-1056 > Project: Metron > Issue Type: Improvement >Reporter: Ryan Merriman > > We need a way to retrieve field types from Elasticsearch for a specified list > of indices. This will also be exposed through our indexing DAO layer with > Elasticsearch being the initial implementation. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
