RE: No Index Formation in Elastic Search

[Karan Gupta]

Hi Lionel,

I tried the below CURL which you sent me

curl -X PUT 'http:///griffin?pretty=true' -H 'Content-Type: 
application/json' -d  '{"mappings": {"accuracy": {"properties": {"name" : 
{"type": "keyword"},"tmst" : {"type": "long"},"value" : {"properties": 
{"total": {"type": "long"},"miss": {"type": "long"},"matched": {"type": 
"long"}}}'

When I try to GET the indexes, I can see that griffin index has been created in 
the elastic search. Then I ran the service jar again but I could not see DQ 
Metric getting populated.

Am I missing something here?

Thank you,
Karan Gupta



From: Lionel Liu 
Sent: Friday, May 4, 2018 6:29 PM
To: Karan Gupta 
Cc: dev@griffin.incubator.apache.org
Subject: Re: No Index Formation in Elastic Search

Hi Karan,

For accuracy, you can try mappings like this:

curl -XPUT 'http://:9200/griffin?pretty=true' -d  '{
  "mappings": {
"accuracy": {
  "properties": {
"name" : {"type": "keyword"},
"tmst" : {"type": "long"},
"value" : {
  "properties": {
 "total": {"type": "long"},
 "miss": {"type": "long"},
 "matched": {"type": "long"}
  }
}
}
  }
  }
}'

The metric schema is like this:

{

   "name": "accuracy",

   "tmst":152532060

   "value": {

  "total": 10,

  "miss": 200,

  "matched": 99800

   }

}



For profiling, you may need another mappings.

In our wiki, you can get the metric schema here:
https://cwiki.apache.org/confluence/display/GRIFFIN/4.+Metric+schema

As I know, ES doesn't need to create indices manually, it will create the 
mappings by the first value posted. That's what we do in our docker image, and 
it works.


Thanks,
Lionel


On Fri, May 4, 2018 at 7:33 PM, Karan Gupta 
> wrote:
Hi Lionel,

We are not using Docker Image, hence we want to set it up manually.
Could you provide us the “CREATE” statement for griffin indices along with 
“mappings”.


Thank you,
Karan Gupta

From: Lionel Liu >
Sent: Friday, May 4, 2018 2:56 PM

To: Karan Gupta >
Cc: dev@griffin.incubator.apache.org
Subject: Re: No Index Formation in Elastic Search

Hi Karan,

In our docker image, we only configured 'http.cors.enabled: true' and 
'http.cors.allow-origin: "*"' in elasticsearch.yml, as the Dockerfile: 
https://github.com/bhlx3lyx7/griffin-docker/blob/master/elasticsearch/Dockerfile
That's all the things we've done for ES configuration, without any other 
initialization. And when the spark application post metrics to ES directly, it 
succeed.

ES will generate the indices by the first value you post to it.

Thanks,
Lionel

On Fri, May 4, 2018 at 4:46 PM, Karan Gupta 
> wrote:
HI Lionel,

The metrics is being persisted in HDFS… This is good progress for us. Thank you 
for all your valuable help.

We created an index for Griffin but we were not sure about what mappings we 
should use.
Until we created this, we never got this index auto-created in ES…..
And now that we have created the index, there are errors which are suggestive 
of missing “mappings”

Is there an auto index create property that we need to enable somewhere in ES?
I could not find anything in the config yml file though….

Thank you,
Karan Gupta
From: Lionel Liu >
Sent: Friday, May 4, 2018 2:11 PM

To: Karan Gupta >
Cc: dev@griffin.incubator.apache.org
Subject: Re: No Index Formation in Elastic Search

Hi Karan,

For HTTP persistence, are the metrics persisted directly from “Spark”? (or) 
Griffin services writes into it?
[Answer] The metrics are persisted directly from spark application.

Our URL is like this: http://localhost:9200/griffin/accuracy (if it is from 
Griffin service, it will work…. But from Spark executors, it wont work as 
localhost resolves to executor host)
[Answer] I think you can modify "localhost" to the ip 

[GitHub] incubator-griffin issue #276: implement sample missing download service, see...

[vzhao]

Github user vzhao commented on the issue:

https://github.com/apache/incubator-griffin/pull/276
  
good comments!  I will add a API spec here, should download 100 sample 
records  
https://github.com/apache/incubator-griffin/blob/master/griffin-doc/service/api-guide.md


---

[GitHub] incubator-griffin issue #276: implement sample missing download service, see...

[guoyuepeng]

Github user guoyuepeng commented on the issue:

https://github.com/apache/incubator-griffin/pull/276
  
to download the sample file, where is the specification for the sample? 
say we only download the head 20 records?


---

[GitHub] incubator-griffin pull request #276: implement sample missing download servi...

[vzhao]

GitHub user vzhao opened a pull request:

https://github.com/apache/incubator-griffin/pull/276

implement sample missing download service, see details in GRIFFIN-177



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/vzhao/incubator-griffin master

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-griffin/pull/276.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #276


commit 98505363aa20b330a55e46c9ebb4979ccaf7dc97
Author: wenzhao 
Date:   2018-05-08T14:21:35Z

implement sample missing download service, see details in GRIFFIN-177




---