Hi Supun, We should index all the information from the messages, including the data payload as well, where this data will be very valuable in the search operations later. This will not become that much of a big problem, since the indexing happens in the background, and is not in the data publishing critical path. And not to mention, the indexing is also pretty fast. Also, disk space is a different concern, and most probably is not a big problem, and for a very large number of messages throughout a large period of time, if needed, we can of course configure data purging. And also, yeah, when data is purging, it will also delete the corresponding index entries as well, so the raw data in the database, and also the index data as well will be deleted.
Cheers, Anjana. On Tue, Mar 29, 2016 at 10:48 PM, Supun Sethunga <[email protected]> wrote: > Hi all, > > As came-up during previous meetings, there was a requirement to do > $subject, when we list down the messages come to a Proxy/API/etc.. in a > given time range. If we are to allow $subject by *payload*, we need to > index the payloads, at esb-analytics (DAS) side. And understandably, since > payloads can be pretty large, indexing payloads for each and every message > would eat-up the disk space (and any possible performance overhead? may be > Gimantha can fill in..). > > Thus, are we going ahead with searching the messages by payloads, or do we > have any alternative search mechanism? > > Also, would like to know, when we are purging the raw data in DAS (say, > per-event-data came few months back), does it also clear any indexed > information as well, which originated from the purged data? > > Regards, > Supun > -- > *Supun Sethunga* > Software Engineer > WSO2, Inc. > http://wso2.com/ > lean | enterprise | middleware > Mobile : +94 716546324 > -- *Anjana Fernando* Senior Technical Lead WSO2 Inc. | http://wso2.com lean . enterprise . middleware
_______________________________________________ Architecture mailing list [email protected] https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
