Hi Jaimin,
For Step1, when you say exposing metrics through the Ambari REST API... are
you talking about the metrics collector REST API, or through the Ambari
Server REST API?
I am able to see data through Phoenix, as an example:
+------------------------------------------+------------------------------------------+------------------------------------------+------------------------------------------+
| METRIC_NAME | HOSTNAME
| SERVER_TIME |
APP_ID |
+------------------------------------------+------------------------------------------+------------------------------------------+------------------------------------------+
| FlowFiles_Received_Last_5_mins | localhost
| 1438045869329 | NIFI
|
Then I try to use this API call:
http://localhost:6188/ws/v1/timeline/metrics?metricNames=FlowFiles_Received_Last_5_mins&appId=NIFI&hostname=localhost&startTime=1437870332000&endTime=1438129532000
and I get: {"metrics":[]}
Something must not be lining up with what I am sending over. Is SERVER_TIME
the field that has to fall between startTime and endTime?
-Bryan
On Mon, Jul 27, 2015 at 1:40 PM, Jaimin Jetly <[email protected]>
wrote:
> Hi Bryan,
>
>
> There are 2 steps in this that needs to be achieved.
>
>
> STEP-1: Exposing service metrics successfully through Ambari REST API
>
> STEP-2: Ambari UI displaying widgets comprised from newly exposed metrics
> via Ambari server.
>
>
>
> As step-1 is pre-requisite to step-2, can you confirm that you were able
> to achieve step-1 (exposing service metrics successfully through Ambari
> REST API) ?
>
>
> *NOTE:*
> /var/lib/ambari-server/resources/common-services/AMBARI_METRICS/0.1.0/metrics.json
> are the metrics specific to Ambari metrics service. If the new metrics
> that you want to expose are related to any other service then please
> edit/create metrics.json file in that specific service package and not in
> Ambari
> metrics service package. widgets.json also needs to be changed/added in the
> same service package and not at
> /var/lib/ambari-server/resources/stacks/HDP/2.0.6/widgets.json (unless
> you want to add system heatmaps for a stack that inherits HDP-2.0.6 stack).
>
>
>
> -- Thanks
>
> Jaimin
> ------------------------------
> *From:* Bryan Bende <[email protected]>
> *Sent:* Sunday, July 26, 2015 2:10 PM
>
> *To:* [email protected]
> *Subject:* Re: Posting Metrics to Ambari
>
>
> Hi Sid,
>
> Thanks for the pointers about how to add a metric to the UI. Based on
> those instructions I modified
> /var/lib/ambari-server/resources/common-services/AMBARI_METRICS/0.1.0/metrics.json
> and added the following based on the test metrics I posted:
>
> "metrics/SmokeTest/FakeMetric": {
>
> "metric": "AMBARI_METRICS.SmokeTest.FakeMetric",
>
> "pointInTime": true,
>
> "temporal": true
>
> }
>
> From digging around the filesystem there appears to be a widgets.json in
> /var/lib/ambari-server/resources/stacks/HDP/2.0.6/widgets.json. It looks
> like this file only contained the definitions of the heatmaps, so I wasn't
> sure if this was the right place, but just to see what happened I modified
> it as follows:
>
> 1) Added a whole new layout:
>
> http://pastebin.com/KqeT8xfe
>
> 2) Added a heatmap for the test metric:
>
> http://pastebin.com/AQDT7u6v
>
> Then I restarted the HDP VM but I don't see anything in the UI under
> Metric Actions -> Add, or under Heatmaps. Anything that seems completely
> wrong about what I did? Maybe I should be going down the route of defining
> a new service type for system I will be sending metrics from?
>
> Sorry to keep bothering with all these questions, I just don't have any
> previous experience with Ambari.
>
> Thanks,
>
> Bryan
>
> On Sun, Jul 26, 2015 at 12:10 AM, Siddharth Wagle <[email protected]>
> wrote:
>
>> The AMS API does not allow open ended queries so startTime and endTime
>> are required fields, the curl call should return the error code with the
>> apt response.
>>
>>
>> If this doesn't happen please go ahead and file a Jira.
>>
>>
>> Using AMS through Ambari UI after getting the plumbing work with
>> metrics.json completed would be much easier. The AMS API does need some
>> refinement. Jiras / Bugs are welcome.
>>
>>
>> -Sid
>>
>>
>>
>> ------------------------------
>> *From:* Siddharth Wagle <[email protected]>
>> *Sent:* Saturday, July 25, 2015 9:01 PM
>>
>> *To:* [email protected]
>> *Subject:* Re: Posting Metrics to Ambari
>>
>>
>> No dev work need only need to modify metrics.json file and then add
>> widget from UI.
>>
>>
>> Stack details:
>>
>> https://cwiki.apache.org/confluence/display/AMBARI/Stack+Defined+Metrics
>>
>>
>> UI specifics:
>>
>>
>> https://cwiki.apache.org/confluence/display/AMBARI/Enhanced+Service+Dashboard
>>
>>
>> -Sid
>>
>>
>> ------------------------------
>> *From:* Bryan Bende <[email protected]>
>> *Sent:* Saturday, July 25, 2015 7:10 PM
>> *To:* [email protected]
>> *Subject:* Re: Posting Metrics to Ambari
>>
>> Quick update, I was able to connect with the phoenix 4.2.2 client and I
>> did get results querying with:
>> SELECT * from METRIC_RECORD WHERE METRIC_NAME =
>> 'AMBARI_METRICS.SmokeTest.FakeMetric' order by SERVER_TIME desc limit 10;
>>
>> Now that I know the metrics are posting, I am less concerned about
>> querying through the REST API.
>>
>> Is there any way to get a custom metric added to the main page of
>> Ambari? or does this require development work?
>>
>> Thanks,
>>
>> Bryan
>>
>> On Sat, Jul 25, 2015 at 9:42 PM, Bryan Bende <[email protected]> wrote:
>>
>>> Hi Sid,
>>>
>>> Thanks for the suggestions. I turned on DEBUG for the metrics
>>> collector (had to do this through the Ambari UI configs section) and now I
>>> can see some activity... When I post a metric I see:
>>>
>>> 01:30:18,372 DEBUG [95266635@qtp-171166092-2 -
>>> /ws/v1/timeline/metrics] TimelineWebServices:270 - Storing metrics: {
>>>
>>> "metrics" : [ {
>>>
>>> "timestamp" : 1432075898000,
>>>
>>> "metricname" : "AMBARI_METRICS.SmokeTest.FakeMetric",
>>>
>>> "appid" : "amssmoketestfake",
>>>
>>> "hostname" : "localhost",
>>>
>>> "starttime" : 1432075898000,
>>>
>>> "metrics" : {
>>>
>>> "1432075898000" : 0.963781711428,
>>>
>>> "1432075899000" : 1.432075898E12
>>>
>>> }
>>>
>>> } ]
>>>
>>> }
>>>
>>> 01:30:18,372 DEBUG [95266635@qtp-171166092-2 - /ws/v1/timeline/metrics]
>>> DefaultPhoenixDataSource:67 - Metric store connection url:
>>> jdbc:phoenix:localhost:61181:/hbase
>>>
>>> 01:30:18,376 DEBUG [95266635@qtp-171166092-2 - /ws/v1/timeline/metrics]
>>> MutationState:361 - Sending 2 mutations for METRIC_RECORD with 8 key values
>>> of total size 925 bytes
>>>
>>> 01:30:18,380 DEBUG [95266635@qtp-171166092-2 - /ws/v1/timeline/metrics]
>>> MutationState:436 - Total time for batch call of 2 mutations into
>>> METRIC_RECORD: 3 ms
>>>
>>> 01:30:18,381 DEBUG [95266635@qtp-171166092-2 - /ws/v1/timeline/metrics]
>>> log:40 - RESPONSE /ws/v1/timeline/metrics 200
>>>
>>>
>>> So it looks like it posted successfully. Then I hit:
>>>
>>>
>>> http://localhost:6188/ws/v1/timeline/metrics?metricNames=AMBARI_METRICS.SmokeTest.FakeMetric
>>>
>>> and I see...
>>>
>>> 01:31:16,952 DEBUG [95266635@qtp-171166092-2 -
>>> /ws/v1/timeline/metrics?metricNames=AMBARI_METRICS.SmokeTest.FakeMetric]
>>> ParallelIterators:412 - Guideposts: ]
>>>
>>> 01:31:16,953 DEBUG [95266635@qtp-171166092-2 -
>>> /ws/v1/timeline/metrics?metricNames=AMBARI_METRICS.SmokeTest.FakeMetric]
>>> ParallelIterators:481 - The parallelScans:
>>> [[{"timeRange":[0,1437874276946],"batch":-1,"startRow":"AMBARI_METRICS.SmokeTest.FakeMetric","stopRow":"AMBARI_METRICS.SmokeTest.FakeMetric\\x01","loadColumnFamiliesOnDemand":null,"totalColumns":1,"cacheBlocks":true,"families":{"0":["ALL"]},"maxResultSize":-1,"maxVersions":1,"filter":"","caching":-1}]]
>>>
>>> 01:31:16,953 DEBUG [95266635@qtp-171166092-2 -
>>> /ws/v1/timeline/metrics?metricNames=AMBARI_METRICS.SmokeTest.FakeMetric]
>>> BaseQueryPlan:243 - Iterator ready: MergeSortTopNResultIterator [limit=1,
>>> count=0, orderByColumns=[METRIC_NAME DESC, SERVER_TIME DESC], ptr1=, ptr2=]
>>>
>>> 01:31:16,957 DEBUG [phoenix-1-thread-171] ParallelIterators:629 - Id:
>>> d0c9c381-f35f-48e6-b970-8b6d5997684b, Time: 3ms, Scan:
>>> {"timeRange":[0,1437874276946],"batch":-1,"startRow":"AMBARI_METRICS.SmokeTest.FakeMetric","stopRow":"AMBARI_METRICS.SmokeTest.FakeMetric\\x01","loadColumnFamiliesOnDemand":null,"totalColumns":1,"cacheBlocks":true,"families":{"0":["ALL"]},"maxResultSize":-1,"maxVersions":1,"filter":"","caching":10000}
>>>
>>> 01:31:16,959 DEBUG [95266635@qtp-171166092-2 -
>>> /ws/v1/timeline/metrics?metricNames=AMBARI_METRICS.SmokeTest.FakeMetric]
>>> PhoenixHBaseAccessor:552 - Aggregate records size: 0
>>>
>>> I'll see if I can get the phoenix client working and see what that
>>> returns.
>>>
>>> Thanks,
>>>
>>> Bryan
>>>
>>> On Fri, Jul 24, 2015 at 5:44 PM, Siddharth Wagle <[email protected]
>>> > wrote:
>>>
>>>> Hi Bryan,
>>>>
>>>>
>>>> Few things you can do:
>>>>
>>>>
>>>> 1. Turn on DEBUG mode by changing log4j.properties at,
>>>> /etc/ambari-metrics-collector/conf/
>>>>
>>>> This might reveal more info, I don't think we print every metrics
>>>> received to the log in 2.0 or 2.1, I did add this option if TRACE is
>>>> enabled to trunk recently.
>>>>
>>>>
>>>> 2. Connect using Phoenix directly and you can do a SELECT query like
>>>> this:
>>>>
>>>> SELECT * from METRIC_RECORD WHERE METRIC_NAME = '<your-metric-name>'
>>>> order by SERVER_TIME desc limit 10;
>>>>
>>>>
>>>> Instructions for connecting to Phoenix:
>>>>
>>>> https://cwiki.apache.org/confluence/display/AMBARI/Phoenix+Schema
>>>>
>>>>
>>>> 3. What API call are you making to get metrics?
>>>>
>>>> E.g.: http://
>>>> <ams-collector>:6188/ws/v1/timeline/metrics?metricNames=<your-metric-name>&startTime=<epoch>&endTime=<epoch>&hostname=<hostname>
>>>>
>>>>
>>>> -Sid
>>>>
>>>>
>>>> ------------------------------
>>>> *From:* Bryan Bende <[email protected]>
>>>> *Sent:* Friday, July 24, 2015 2:03 PM
>>>> *To:* [email protected]
>>>> *Subject:* Posting Metrics to Ambari
>>>>
>>>> I'm interested in sending metrics to Ambari and I've been looking at
>>>> the Metrics Collector REST API described here:
>>>>
>>>> https://cwiki.apache.org/confluence/display/AMBARI/Metrics+Collector+API+Specification
>>>>
>>>> I figured the easiest way to test it would be to get the latest HDP
>>>> Sandbox... so I downloaded and started it up. The Metrics Collector service
>>>> wasn't running so I started it, and also added port 6188 to the VM port
>>>> forwarding. From there I used the example POST on the Wiki page and made a
>>>> successful POST which got a 200 response. After that I tried the query, but
>>>> could never get any results to come back.
>>>>
>>>> I know this list is not specific to HDP, but I was wondering if
>>>> anyone has any suggestions as to what I can look at to figure out what is
>>>> happening with the data I am posting.
>>>>
>>>> I was watching the metrics collector log while posting and querying
>>>> and didn't see any activity besides the periodic aggregation.
>>>>
>>>> Any suggestions would be greatly appreciated.
>>>>
>>>> Thanks,
>>>>
>>>> Bran
>>>>
>>>
>>>
>>
>
