I have implemented a Distributed Transaction Logging library with Tree like Structure as mention in Google Dapper( http://research.google.com/pubs/pub36356.html) and eBay CAL Transaction Logging Framework( http://devopsdotcom.files.wordpress.com/2012/11/screen-shot-2012-11-11-at-10-06-39-am.png).
The following is log format. *Log Format* TIMESTAMP HOSTNAME DATACENTER ENVIRONMENT EVENT_GUID PARENT_GUID TRACE_GUID APPLICATION_ID TREE_LEVEL TRANSACTION_TYPE TRANSACTION_NAME STATUS_CODE DURATION(in ms) PAYLOAD(key1=value2,key2=value2) *GUID HEX NUMBER FORMAT* MURMER_HASH(HOSTNAME + DATACENTER + ENVIRONMENT)-JVM_THREAD_ID-(TIME_STAMP+Atomic Counter) The log format is like Relational Database Tables. Here is list of search query we will be doing: Also I would like to know how I can index elastic search payload data so user specify some expression like in payload (duration > 1000) then, Elastic Search will bring all the loglines that satisfy condition. Also, I would like to index Payload as Name=Value pair so user can query (key3=value2 or key4 = *exception*) some sort of regular expression. Please let me know if this can be achieved. Any help pointer would be great.. Please note the Payload can be any string name or value(can be integer, or decimal or string). Payload search example: // (Duration > 1000) and (key1 > 10 or key1 < 100) etc Finally I would be integrating this with Kibana UI so we can have nice UI for searching and sorting etc. I am new to Elastic Search and any pointer will be great. We will be using this in walmartlabs with around 100 machine elastic search cluster and ~6000 FE servers logging data into Kafka and we will index from the Kafka into Elastic Search. Any pointer will be great. Thanks, Bhavesh -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/79f04627-29ac-42aa-9170-35c22ef76ffa%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
