On Wed, Jul 4, 2012 at 12:30 PM, Srinath Perera <[email protected]> wrote:
> Hi All, > > Sorry for late reply > > As I see it, secondary indexes are not a part of the agent API, which is > about collecting the data. > > Secondary index is about analyzing / presenting the data. So if user > need secondary indexes, they will have to go to Cassandra directly, and > define them. I think we can do the same in logging impl. > > Yes, after last friday stratos-logging review, we decided to have real time logs from the memory and to view older logs using bam analytics. So we don't have this requirement anymore, We are writing a separate log tool kit for BAM so that users can download long running logs separately. > --Srinath > > > On Fri, Jun 22, 2012 at 12:34 AM, Suhothayan Sriskandarajah <[email protected] > > wrote: > >> >> >> On Fri, Jun 22, 2012 at 2:36 PM, Amani Soysa <[email protected]> wrote: >> >>> >>> >>> On Fri, Jun 22, 2012 at 2:29 PM, Tharindu Mathew <[email protected]>wrote: >>> >>>> This can be a useful feature for realtime requirements. But we need to >>>> follow some proper convention as this changes the event stream definition >>>> This is then an extensive change, and we are close to feature freezing. >>>> >>>> On Fri, Jun 22, 2012 at 2:15 PM, Amani Soysa <[email protected]> wrote: >>>> >>>>> >>>>> >>>>> On Fri, Jun 22, 2012 at 2:02 PM, Deependra Ariyadewa <[email protected]>wrote: >>>>> >>>>>> >>>>>> >>>>>> On Fri, Jun 22, 2012 at 1:45 PM, Amani Soysa <[email protected]> wrote: >>>>>> >>>>>>> >>>>>>> >>>>>>> On Fri, Jun 22, 2012 at 1:24 PM, Tharindu Mathew >>>>>>> <[email protected]>wrote: >>>>>>> >>>>>>>> This will be useful for folks who want real time data access, but >>>>>>>> BAM is not designed to be real time. I don't want the Agent API to be >>>>>>>> specific to Cassandra, either. >>>>>>>> >>>>>>>> There should be a clean way to do this. How did you decide to do it >>>>>>>> this way? Was there a discussion? >>>>>>>> >>>>>>> Yes there was a discussion on this some time back on Architecture >>>>>>> -"RFC: Architecture for Stratos Log Processing" where we decided to >>>>>>> push >>>>>>> logs to bam event receiver through the publisher and view logs using >>>>>>> hector >>>>>>> api. >>>>>>> >>>>>> >>>>>> Initially we tried to use flume as the Stratos log collector/manager >>>>>> but we stop flume evaluation because BAM's capability to cover the same >>>>>> use >>>>>> case. >>>>>> >>>>>> There are several workarounds like create relevant keyspaces in the >>>>>> tenant creation or create extended event receiver only for logging. >>>>>> >>>>> If we do it in the tenant creation time their will be around 10 >>>>> keyspaces per each tenant (given that for each server, we create a >>>>> keyspace >>>>> as I explained earlier) So even a user doesn't use a particular server >>>>> their will be a keyspace for it. So if there are 1000 tenants there will >>>>> be >>>>> 10 000 keyspace (even if some keyspaces are not used at all) So I think >>>>> its >>>>> better to create keyspaces when ever logs are publishing to that >>>>> particular >>>>> keyspace. >>>>> >>>> 10 keryspaces per tenant?? Are you sure that's right... >>>> >>> Yes for each sever (maybe more than 10 :) ) >>> ie - data services,appserver,esb,mb,cep,bps,brs etc all the products we >>> offer for stratos deployment , because we store logs for each server in >>> different keyspace(if not we have to keep everything in a single keyspace >>> per tenant and it will be an expensive search when we filter the logs from >>> the server level because we give users server specific logs) >>> In our earlier syslog implementation we divided logs per each server for >>> fast result (as logs generates). Thats why its very important to not to >>> create keyspaces if users are not using a particular product. >>> >> >> I also agree with tharindu, that we should not make stream definition >> Cassandra specific, since this can also be used for JDBC data-store or >> InMemory data-store for CEP. >> >> Since this is an Cassandra specific issue and since we know the stream >> definition in advance, I believe its appropriate to have a Cassandra data >> store configuration which maps the stream definition to the appropriate >> Cassandra create key store query. Through this when the client request sent >> for defineEventStream the Cassandra data-store can first checks if it match >> one of the entry in the configuration, if so it runs the create key store >> query given in the configuration to create the key store with indexes, else >> it will create the key store in the normal way. >> Through this we can also restrict the number of unnecessary key store >> creation. >> >> Regards >> Suho >> >>> >>>>>> Thanks, >>>>>> >>>>>> Deependra. >>>>>> >>>>>>> >>>>>>>> On Fri, Jun 22, 2012 at 8:45 AM, Amani Soysa <[email protected]>wrote: >>>>>>>> >>>>>>>>> Hi, >>>>>>>>> >>>>>>>>> Currently we are sending LogEvent data through bam data publisher >>>>>>>>> to bam event receiver using a custom log4j appender. And we retrieve >>>>>>>>> logs >>>>>>>>> using the hector API for the carbon log viewer. However, we need to >>>>>>>>> have >>>>>>>>> secondary indexes for several columns so that we can filter log >>>>>>>>> information >>>>>>>>> for a given column ( such as date, applicationName, priority,logger >>>>>>>>> etc) >>>>>>>>> when creating the data publisher (keyspace). From the current Bam Data >>>>>>>>> publisher implementation we cannot do secondary indexing all we can >>>>>>>>> do is >>>>>>>>> define the column name and the data type of that column, and reciver >>>>>>>>> creates the keyspaces for given columns with their data types. >>>>>>>>> >>>>>>>>> streamId = dataPublisher.defineEventStream("{" + " >>>>>>>>> 'name':'org.wso2.carbon.logging.$tenantId.$serverName'," >>>>>>>>> + " 'version':'1.0.0'," + " 'nickName': >>>>>>>>> 'Logs'," >>>>>>>>> + " 'description': 'Logging Event'," + " >>>>>>>>> 'metaData':[" >>>>>>>>> + " {'name':'clientType','type':'STRING'}" >>>>>>>>> + " ]," >>>>>>>>> + " 'payloadData':[" >>>>>>>>> + " {'name':'tenantID','type':' >>>>>>>>> STRING'}," >>>>>>>>> + " {'name':'serverName','type':' >>>>>>>>> STRING'}," >>>>>>>>> + " {'name':'appName','type':' >>>>>>>>> STRING'}," >>>>>>>>> + " {'name':'logTime','type':' >>>>>>>>> LONG'}," >>>>>>>>> + " {'name':'logger','type':' >>>>>>>>> STRING'}," >>>>>>>>> + " {'name':'priority','type':' >>>>>>>>> STRING'}," >>>>>>>>> + " {'name':'message','type':' >>>>>>>>> STRING'}," >>>>>>>>> + " {'name':'ip','type':'STRING'}, >>>>>>>>> " >>>>>>>>> + " {'name':'stacktrace','type':' >>>>>>>>> STRING'}," >>>>>>>>> + " {'name':'instance','type':' >>>>>>>>> STRING'}" >>>>>>>>> + " ]" >>>>>>>>> + "}"); >>>>>>>>> >>>>>>>>> Is it possible to have a cassandra specific event receiver (for >>>>>>>>> logging purposes) so that we can create key spaces with secondary >>>>>>>>> indexes?[ >>>>>>>>> 1 <https://wso2.org/jira/browse/CARBON-13468>] and it will create >>>>>>>>> keyspaces when ever logs are published . Or do we need to create >>>>>>>>> keyspaces >>>>>>>>> at tenant creation time?. For a given tenant we need to create several >>>>>>>>> keyspaces, depending on the server (and if possible for applications >>>>>>>>> as >>>>>>>>> well so we can have better performance when viewing logs). >>>>>>>>> ie - keyspace1 - org_wso2_logging_tenant1_application_server >>>>>>>>> (store AS specific logs) >>>>>>>>> keyspace2 - >>>>>>>>> org_wso2_logging_tenant1_data_services_server (store DSS specific >>>>>>>>> logs) >>>>>>>>> >>>>>>>>> Please note that we cannot use BAM analytics to view logs because >>>>>>>>> we need a real time log-viwer. >>>>>>>>> >>>>>>>>> [1] - https://wso2.org/jira/browse/CARBON-13468 >>>>>>>>> >>>>>>>>> Regards, >>>>>>>>> Amani >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> Regards, >>>>>>>> >>>>>>>> Tharindu >>>>>>>> >>>>>>>> blog: http://mackiemathew.com/ >>>>>>>> M: +94777759908 >>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> Deependra Ariyadewa >>>>>> WSO2, Inc. http://wso2.com/ http://wso2.org >>>>>> >>>>>> email [email protected]; cell +94 71 403 5996 ; >>>>>> Blog http://risenfall.wordpress.com/ >>>>>> PGP info: KeyID: 'DC627E6F' >>>>>> >>>>>> >>>>> >>>> >>>> >>>> -- >>>> Regards, >>>> >>>> Tharindu >>>> >>>> blog: http://mackiemathew.com/ >>>> M: +94777759908 >>>> >>>> >>> >> >> >> -- >> *S. Suhothayan >> * >> Software Engineer, >> Data Technologies Team, >> *WSO2, Inc. **http://wso2.com >> <http://wso2.com/>* >> *lean.enterprise.middleware.* >> >> *email: **[email protected]* <[email protected]>* cell: (+94) 779 756 757 >> blog: **http://suhothayan.blogspot.com/*<http://suhothayan.blogspot.com/> >> * >> twitter: **http://twitter.com/suhothayan* <http://twitter.com/suhothayan> >> * >> linked-in: **http://lk.linkedin.com/in/suhothayan* >> * >> * >> >> > > > -- > ============================ > Srinath Perera, Ph.D. > Senior Software Architect, WSO2 Inc. > Visiting Faculty, University of Moratuwa > Member, Apache Software Foundation > Research Scientist, Lanka Software Foundation > Blog: http://srinathsview.blogspot.com/ > Photos: http://www.flickr.com/photos/hemapani/ > Phone: 0772360902 >
_______________________________________________ Dev mailing list [email protected] http://wso2.org/cgi-bin/mailman/listinfo/dev
