Hi, I tried to do a workaround but it was somehow unsuccessful.
We have the following structure:
struct VMUsageObject
{
1: string cache_id
2: string vm_id
3: double cpu_used
4: u64 mem_used
}
struct VMServiceObject
{
1: string cache_id
2: string vm_id
3: string service_id
4: string status
5: u64 memory
}
struct QoSInfoObject
{
1: string cache_id
2: u64 service_quality
3: double hit_ratio
}
struct ChannelUsersObject
{
1: string cache_id
2: string channel_id
3: u64 num_users
}
struct DeviceTypeUsersObject
{
1: string cache_id
2: string devicetype_id
3: u64 num_users
}
struct CDNAgent
{
1: string name (key="CDNObject")
2: optional bool deleted
3: optional list<VMUsageObject> VMUsage
(tags=".cache_id,.vm_id")
4: optional list<VMServiceObject> VMServices
(tags=".cache_id,.vm_id,.service_id")
5: optional list<QoSInfoObject> QoSInfo (tags=".cache_id")6: optional list<ChannelUsersObject> ChannelUsers (tags=".cache_id,.channel_id") 7: optional list<DeviceTypeUsersObject> DeviceTypeUsers (tags=".cache_id,.devicetype_id")
}
uve sandesh CDNTrace
{
1: CDNAgent data
}
By checking the CDN uve we get the reply that is shown in the adjoint
SandeshReply.json. Except that there is a table that doesn't contain
any value (VMServices), the rest is OK.
However, if we do the following query:
{
"end_time": "now",
"select_fields": [
"name", "T", "VMUsage.cache_id", "VMUsage.vm_id",
"VMUsage.cpu_used", "VMUsage.mem_used"
],
"sort_fields": [],
"start_time": "now-60m",
"table": "StatTable.CDNAgent.VMUsage",
"where": [
[
{
"name": "name",
"op": 7,
"suffix": null,
"value": "",
"value2": null
}
]
]
}
We get this reply:
Error: 'NoneType' object has no attribute 'iteritems'
Any ideas?
Best Regards,
Alberto.
Quoting Anish Mehta <[email protected]>:
Hi Alberto, For ObjectTables (Using "table":"ObjectAlberto" for queries) , the list and map should work fine. But for StatsTables, the answer is more complicated. Upto R2.1, we only record StatsTable samples for strings, ints and doubles. Starting in R2.2, we do support structs and lists-of-structs in StatTables. You would need to use the "tags" annotation against the structs and list of struct you want recorded in this way. Otherwise, the struct or list-of-structs will be ignored. Maps are not supported. For your example, it would work like this: struct ServiceInfo2 { 1: string sometag 2: u64 someval } struct VMInfoObject { 1: string cache_id 2: string vm_id 3: double cpu_used 4: u64 mem_used 6: list<ServiceInfo2> services2 (tags=".someval") } struct AlbertoAgent { 1: string name (key="ObjectAlberto") 2: optional bool deleted 3: optional list<VMInfoObject> VMInfo (tags=".cache_id,.vm_id") } You can query it in this way: "table": "StatTable.AlbertoAgent.VMInfo.services2", "where": [[ { "name": "name", "op": 1, "suffix": null, "value": "lvelascoTurbo", "value2": null }, { "name": "VMInfo.cache_id", "op": 1, "suffix": null, "value": "mycache", "value2": null }, { "name": "VMInfo.services2.sometag", "op": 1, "suffix": null, "value": "mytag", "value2": null } ]], "select_fields": [ "T=60", "COUNT(VMInfo.services2)", "SUM(VMInfo.services2.someval)", "VMInfo.vm_id" ], The functionally was added in this commit: https://github.com/Juniper/contrail-controller/commit/226b6230690d4ee7a6548 24bef0e9b667011f842 Regards, Anish On 5/13/15, 9:36 AM, "[email protected]" <[email protected]> wrote:Hi, Another question: Is it possible to have lists/maps and get them in a query? For example: struct VMInfoObject { 1: string cache_id 2: string vm_id 3: double cpu_used 4: u64 mem_used 5: map<string, ServiceInfo> services 6: list<ServiceInfo2> services2 } With: 3: optional list<VMInfoObject> VMInfo (tags=".cache_id,.vm_id") Can I get in a single query the services or the services2? We tried to query for "VMInfo.services" and "VMInfo.services2" with no positive result. Thanks for your support, Alberto. Quoting Anish Mehta <[email protected]>:Hi Alberto, Support for MAX and MIN is not in R2.1 or earlier release. You will see it if you use the R2.2 or mainline branches. If you aren't sure if your image has MAX and MIN, try to access this schema via the Analytics API: http://xx.xx.xx.xx:8081/analytics/table/StatTable.AnalyticsCpuState.cpu_i nf o/schema If you see MAX and MIN attributes here, they should work for your query as well. Regards, Anish On 5/12/15, 8:48 AM, "[email protected]" <[email protected]> wrote:Hi Anish, Thanks for your reply. I'm now doing the following query: { "end_time": "now", "select_fields": [ "T=30s", "COUNT(dTrace)", "SUM(dTrace.val1)", "SUM(dTrace.val2)", "SUM(dTrace.val3)", "MAX(dTrace.val2)", "MIN(dTrace.val3)" ], "sort_fields": [], "start_time": "now-1m", "table": "StatTable.AlbertoAgent.dTrace", "where": [ [ { "name": "name", "op": 1, "suffix": null, "value": "lvelascoTurbo", "value2": null } ] ] } And as result I get the following: { "value": [ { "COUNT(dTrace)": 22, "SUM(dTrace.val1)": 22838019078, "SUM(dTrace.val2)": 27778532625, "SUM(dTrace.val3)": 726, "T=": 1431445440000000 }, { "COUNT(dTrace)": 28, "SUM(dTrace.val1)": 30242613471, "SUM(dTrace.val2)": 36570568510, "SUM(dTrace.val3)": 924, "T=": 1431445470000000 } ] } As you can see it doesn't return neither the MAX nor the MIN. Are those aggregations defined for type u64? Best regards, Alberto. Quoting Anish Mehta <[email protected]>:Hi Alberto, The ³aggtype² annotation in the sandesh file refers to the Analytics GET API for UVEs. It is unrelated to the Analytics Query API. For the Analytics GET API, we support the following aggregations: aggtype=³sum² (for integers or doubles) aggtype=³union² (for lists of strings or structures) See this blog post for details: http://www.opencontrail.org/operational-state-in-the-opencontrail-syste m- uv e-user-visible-entities-through-analytics-api/ For the Analytics Query API on StatTables, we support the following aggregations for integers or doubles: SUM MAX MIN For your example, you are trying to use aggregations for StatTables queries. You do not need to use the ³aggtype² annotation at all. If you use SUM, MIN or MAX aggregations in the select part of the query, it should just work. Regards, Anish On 5/11/15, 2:38 AM, "[email protected]" <[email protected]> wrote:Hi, That was a silly mistake, excuse me. It works now! Now I have a doubt about aggregation types. I asked before about which kind of aggregations there are and it seems that there are SUM, AVG, UNION, MIN and MAX at least (correct me if I'm wrong). I want to have MIN and MAX along SUM in my structure. Is it possible? I'm trying to do it with the following sandesh file with no success: struct DataTrace { 1: u64 val1 2: u64 val2 3: u64 val3 (aggtype="sum,min,max") } struct AlbertoAgent { 1: string name (key="ObjectAlberto") 2: optional bool deleted 3: optional u64 cpu 4: optional list<DataTrace> dTrace (tags=".val1,.val2") } uve sandesh AlbertoTrace { 1: AlbertoAgent data } I haven't seen any error in the process. Where should I check if there is an error in this part of the environment? Thank you for your support. Best regards, Alberto. Quoting Anish Mehta <[email protected]>:Hi Alberto, You have used "dtrace" instead of "dTrace" for val1 and val2. As Raj stated in his email, we usually use T= in conjunction with at least one aggregate column like SUM(xyz) to get time series. E.g. [T=60 , SUM(dTrace.val1) , dTrace.val2] will give one time series for SUM(dTrace.val1) for each unique value of dTrace.val2. The API will allow you to specify [T=60 , dTrace.val1]. This will return all unique values of dTrace.val1 in each time period. The API will also allow you to specify [T, SUM(dTrace.val1)]. If there are multiple samples of dTrace.val1 for a given T, they will get aggregated together. Otherwise, each SUM(dTrace.val1) will only represent one dTrace.val1, and the query result will be equivalent to [T, dTrace.val1] Please see the Stats blog entry for more details. http://www.opencontrail.org/statistics-in-opencontrail-analytics/ Regards, Anish On May 7, 2015, at 6:12 AM, Raj Reddy <[email protected]<mailto:[email protected]>> wrote: You can either select "T=" & SUM(xyz) [this gives time series values] *or* "T" & XYZ [this gives raw values] One of the below will work "select_fields": [ "T=","COUNT(dTrace)", "SUM(dtrace.val1)", "SUM(dtrace.val2)" ], or "select_fields": [ "T","COUNT(dTrace)", "dtrace.val1", "dtrace.val2" ], - Raj On May 7, 2015, at 4:44 AM, <[email protected]<mailto:[email protected]>> wrote: Hi Megh and Anish, Thanks for your support. Making the query to the UVE I get the following: { "AlbertoAgent": { "cpu": 31337, "dTrace": [ { "StatTable.AlbertoAgent.dTrace": [ { "COUNT(dTrace)": 1, "SUM(dTrace.val1)": 1, "SUM(dTrace.val2)": 2 } ] } ] } } That message is produced by the following C++ code: AlbertoAgent vnAgent; vnAgent.set_name("lvelasco"); vnAgent.set_deleted(false); vnAgent.set_cpu(31337); std::vector<DataTrace> dt(1); dt[0].set_val1(1); dt[0].set_val2(2); vnAgent.set_cpu(31337); vnAgent.set_dTrace(dt); AlbertoTrace::Send(vnAgent); This code is executed in a loop fashion. Now I want to get the values produced over time by this agent so I did the following query: { "end_time": "now", "select_fields": [ "T","COUNT(dTrace)", "SUM(dtrace.val1)", "SUM(dtrace.val2)" ], "sort_fields": [], "start_time": "now-60m", "table": "StatTable.AlbertoAgent.dTrace", "where": [ [ { "name": "name", "op": 1, "suffix": null, "value": "lvelasco", "value2": null } ] ] } However it only returns me the "T","COUNT(dTrace)" part and not val1 and val2. What I'm doing wrong? Best regards, Alberto. Quoting Megh Bhatt <[email protected]<mailto:[email protected]>>: Hi Albero, trace is a keyword, please change name to data_trace. Also please have optional bool deleted as element number 2 and then you should be good to go Thanks Megh On May 6, 2015, at 6:57 AM, "[email protected]<mailto:[email protected]>" <[email protected]<mailto:[email protected]>> wrote: Hi anish, Thanks for your reply. In that example, aggregation type "union" is applied over all the elements of type ProcessCpuInfo? I have adapted my structure following your example: struct DataTrace { 1: u64 val1 2: u64 val2 } struct AlbertoAgent { 1: string name (key="ObjectAlberto") 2: optional u64 cpu 3: optional bool deleted 4: optional list<DataTrace> trace (tags=".val1,.val2") } uve sandesh AlbertoTrace { 1: AlbertoAgent data } However there is a syntax error that I'm not able to detect (Error message:(last token was 'trace')syntax error). Am I in the correct way? Where is the error? Thanks for your support, Alberto. Quoting Anish Mehta <[email protected]<mailto:[email protected]>>: Hi Alberto, I assume you are talking about the stats functionality, which is turned on using the "tags" annotation. I can help more if you send the sandesh file Meanwhile, I will demonstrate this feature using Analytics CPU Info. This is the sandesh file: struct AnalyticsCpuState { 1: string name (key="ObjectCollectorInfo") 2: optional bool deleted 3: optional list<cpuinfo.ProcessCpuInfo> cpu_info (tags=".module_id,.mem_virt,.cpu_share,.mem_res", aggtype="union") } uve sandesh AnalyticsCpuStateTrace { 1: AnalyticsCpuState data } The tags annotation will cause the "cpu_info" attribute to be recorded as a Stat Sample. This sample can then be queried via the Analytics Query API (the same API used for the object-log) The attributes listed in the tags annotation will be used for indexing. (the "where" part of the query) We implicitly index by the Source of message (Source of the Sandesh Generator) and the UVE Key. These fields are recorded as "name" and "Source". The other index fields (as per the tags annotation) in this case are cpu_info.module_id, cpu_info.mem_virt, cpu_info.cpu_share and cpu_info.mem_res. There are two other fields that will be present the sample - cpu_info.inst_id and cpu_info.module_id. These are not in the tags annotation, so we will not index by them. From the command line, you can use the "contrail-stats" command to exercise the Analytics Query API for stats. Here's an example of usage: contrail-stats --dtable AnalyticsCpuState.cpu_info --where "Source=*" --select "T=300" "COUNT(cpu_info)" "name" "cpu_info.module_id" "SUM(cpu_info.cpu_share)" --last 30m This results in the following Analytics API POST request: {"start_time": "now-30m", "sort_fields": [], "end_time": "now", "select_fields": ["T=300", "COUNT(cpu_info)", "name", "cpu_info.module_id", "SUM(cpu_info.cpu_share)"], "table": "StatTable.AnalyticsCpuState.cpu_info", "where": [[{"suffix": null, "value2": null, "name": "Source", "value": "", "op": 7}]]} You can get more information here: http://www.opencontrail.org/statistics-in-opencontrail-analytics/ This blog entry will also explain what "T" and "T=" mean. If you share your sandesh file, I can give you more specific pointers. Please reach out if you have other questions about this. Regards, Anish On 5/4/15, 3:39 AM, "[email protected]<mailto:[email protected]>" <[email protected]<mailto:[email protected]>> wrote: Hi, Any clues about how can I make a flow like the ones from Analytics node CPU info? I cannot figure how to do it with sandesh. Is it done using list type? Best Regards, Alberto. Quoting [email protected]<mailto:[email protected]>: Hi Anish, There was a mistake in the changes I did. Now it's correct and I get this: agutierrez@Mahalanobis:~/Documents/thesis/OpenContrail/query$ ./getQuery.sh /uves/alberto/lvelasco?flat % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 186 0 186 0 0 147 0 --:--:-- 0:00:01 --:--:-- 147 { "UveVirtualNetworkAgent": { "cpu": 31337, "cpu_info": [ { "StatTable.UveVirtualNetworkAgent.cpu_info": [ { "COUNT(cpu_info)": 1, "SUM(cpu_info.att1)": 2090, "SUM(cpu_info.att2)": 209 } ] } ] } } I modified slightly the structure to have a list inside. Now my question is the following: How can I make a flow like the ones from Analytics node CPU info? I have taken a look to some .sandesh files but I cannot figure out. Also, how can be "T" and "T=" specified? Thank you for your help. Best regards, Alberto. Quoting Anish Mehta <[email protected]<mailto:[email protected]>>: Hi Alberto. The schema looks correct. Both contrail-logs and the Query API use the _STAT_TABLES variable. They should work if you change that variable as directed. Please send me the output of "contrail-logs ?help?, and the contents of your viz/constants.py file. Lets work on the UVE part. This is what you can do: 1. Check if the UVE table type is available in contrail-analytics-api. Look at http://xx.xx.xx.xx:8081/analytics/uves Do you see an entry like this?: { href: "http://xx.xx.xx.xx:8081/analytics/uves/albertos";, name: ?albertos" }, 2. Send the UVE from your application. When you called InitGenerator, you can specify an Introspect port. I?ll assume you used 5555. After your application is up, and you have sent your UVE, check for this: http://xx.xx.xx.xx:5555/Snh_SandeshUVETypesReq? Do you see an entry for you structure ?TestAgent? You can go onto the link for TestAgent and the the last values you sent. Also, check your connection to the collector: http://xx.xx.xx.xx:5555/Snh_CollectorInfoRequest? 3. If all this worked, you should see the UVE at http://xx.xx.xx.xx:8081/analytics/uves/albertos Regards, Anish On 4/24/15, 9:16 AM, "[email protected]<mailto:[email protected]>" <[email protected]<mailto:[email protected]>> wrote: Hi, I compiled viz.sandesh using "Sandesh --gen py viz.sandesh" directly and I substituted the .py. I done this using the same structure as before but changing the name of the table and following your instructions, but I have problems with it. I get this schema as the table schema: { "columns": [ { "datatype": "int", "index": false, "name": "MessageTS", "select": null, "suffixes": null }, { "datatype": "string", "index": true, "name": "ObjectId", "select": null, "suffixes": null }, { "datatype": "string", "index": true, "name": "Source", "select": null, "suffixes": null }, { "datatype": "string", "index": true, "name": "ModuleId", "select": null, "suffixes": null }, { "datatype": "string", "index": true, "name": "Messagetype", "select": null, "suffixes": null }, { "datatype": "string", "index": false, "name": "ObjectLog", "select": null, "suffixes": null }, { "datatype": "string", "index": false, "name": "SystemLog", "select": null, "suffixes": null } ], "type": "OBJECT" } Does this make sense? Moreover, the following thing happens: contrail-logs --object-type alberto HTTP error code: 404 It seems to not be available as UVE. Also I have a question regarding to _STAT_TABLES variable in viz.sandesh. If I define the structure in this table, will it be available in the query part of the web? Thank you for your assistance, Alberto. Quoting [email protected]<mailto:[email protected]>: Hi Anish, Thank you for your reply. I'm having the following problem when compiling: ... Running virtualenv with interpreter testroot/bin/python New python executable in analytics_test/bin/python Installing Setuptools........................................................... . ... ..............................done. Installing Pip.................................................................. . ... ..............................................................done. /bin/bash -c "source /home/agutierrez/workspace/Contrail/opencontrail2/build/debug/analyti c s_t est/bin/activate; pip install --download-cache=/tmp/cache/agutierrez/systemless_test greenlet==0.4.1 gevent==0.13.8 eventlet==0.9.17 testtools==0.9.21 fixtures==0.3.12 requests>=1.1.0 lxml==2.3.3 geventhttpclient==1.0a prettytable==0.7.2 psutil==0.4.1 redis==2.7.1 xmltodict==0.2 thrift==0.8.0 bottle==0.11.6 mock==1.0.1 kafka-python==0.9.2 kazoo==1.3.1 stevedore" scons: *** [build/debug/opserver/test/greenlet] Error 1 scons: building terminated because of errors. Do you have any idea that what can be the problem or if there is any log file that could provide more information about the error? Best regards, Alberto. Quoting Anish Mehta <[email protected]<mailto:[email protected]>>: Hi Alberto, Adding your own table is possible. You need to add some entries to /controller/src/analytics/viz.sandesh It does not need the collector to be rebuilt, but it does require rebuilding contrail-analytics-api. I'm going to assume the table name "ObjectAlberto" (instead of "ObjectTableName") I'm going to make this UVE accessible via the Analytics API as follows: http://xx.xx.xx.xx:8081/analytics/uves/albertos I'm going to make the ObjectLog visible via the Analytics Query API as well (as you mentioned in your email) Based on this API, from the command line of the analytics node you will be able to do this: contrail-logs --object-type alberto 1. Changes in controller/src/analytics/viz.sandesh const string ALBERTO_TABLE = "ObjectAlberto" const string ALBERTO_DISPLAY_NAME = "Alberto" ... const map<string, string> UVE_MAP = { ... "alberto" : ALBERTO_TABLE, } ... const map<string, objtable_info> _OBJECT_TABLES = { ALBERTO_TABLE : { 'objtable_display_name' : ALBERTO_DISPLAY_NAME 'log_query_name' : "alberto" } } 2. Rebuild contrail-analytics-api sb/controller/src$ scons -u opserver 3. Install and test on target You should rebuild the contrail-analytics package and install it on your target. For this kind of change, there is an alternative installation method as well, given below: This is the generated file that will change when you rebuild: sb/controller/src$ ls ../../build/debug/opserver/opserver/sandesh/viz/constants.py You can copy it onto your analytics node. It will be at a location such as: /usr/lib/python2.7/dist-packages/opserver/sandesh/viz/constants.py After overwriting the file (you should take a backup first, just in case), restart contrail-analytics-api: service contrail-analytics-api restart 4. Test the UVE Analytics API and Analytics Query API after starting your application and connecting to the collector. Regards, Anish On 4/23/15, 1:34 AM, "[email protected]<mailto:[email protected]>" <[email protected]<mailto:[email protected]>> wrote: Hello Anish, Thanks for your reply. What we are going to do is a little application that sends it's own metrics to the collector so we can retrieve that data using de Analytics API. For example: struct TestAgent { 1: string name(key="ObjectTableName") 2: optional i32 valueToBeSent1 } uve sandesh TestTrage { 1: TestAgent data; } This kind of message should be stored in some way in Cassandra databases and be available to the API users. I understand that (key="ObjectTableName") makes reference to a new table with that name that can be queried from the API. For me that is the concept of "custom table", tables that are administrator/developer defined different from base contrail tables. It would be retrieved with a POST query to the API similar to this: { "end_time": "now", "select_fields": [ "valueToBeSent1" ], "sort_fields": [], "start_time": "now-60m", "table": "ObjectTableName", "where": [ [ { "name": "Source", "op": 1, "suffix": null, "value": "GeneratorApplication", "value2": null } ] ] } Is this possible? If it is, do I have to do any step apart from send the structure using the code generated by Sandesh compiler? If it is not possible, is there any workaround? The only way that comes to my mind is that we could query directly to message table, filter the messages we don't want and process the XMLs but that solution wouldn't be clean. I hope I explained it correctly. Thank you for your assistance. Best regards, Alberto Guti?rrez. Quoting Anish Mehta <[email protected]<mailto:[email protected]>>: The schemas for new Sandesh types do not need to be compiled or loaded onto the Collector. The collector gets XML-encoded messages from the generators and stores them according to their sandesh types and the contained annotations. Sandesh Types and how they work are explained here: http://www.opencontrail.org/sandesh-a-sdn-analytics-interface/ I^1m not sure what you mean by ^3custom tables^2. Please send us a sample sandesh file and describe what you are trying to do, and we can discuss further. Regards, Anish On 4/22/15, 6:53 AM, "[email protected]<mailto:[email protected]>" <[email protected]<mailto:[email protected]>> wrote: Dear all, When creating a new sandesh structure, do I need to send somehow the schema to the collectors so they will understand it or it will be added automatically with the first message it recieves? Thanks for your attention, Alberto. _______________________________________________ Dev mailing list [email protected]<mailto:[email protected]> http://lists.opencontrail.org/mailman/listinfo/dev_lists.opencontr ail .or g _______________________________________________ Dev mailing list [email protected]<mailto:[email protected]> http://lists.opencontrail.org/mailman/listinfo/dev_lists.opencontrail . org _______________________________________________ Dev mailing list [email protected]<mailto:[email protected]> http://lists.opencontrail.org/mailman/listinfo/dev_lists.opencontrail .o rg _______________________________________________ Dev mailing list [email protected]<mailto:[email protected]> http://lists.opencontrail.org/mailman/listinfo/dev_lists.opencontrail .o rg _______________________________________________ Dev mailing list [email protected]<mailto:[email protected]> http://lists.opencontrail.org/mailman/listinfo/dev_lists.opencontrail .o rg
SandeshReply.json
Description: application/json
_______________________________________________ Dev mailing list [email protected] http://lists.opencontrail.org/mailman/listinfo/dev_lists.opencontrail.org
