Shall we try to write a hive query which takes all the logs for a given tenant and group by the application name and depending on the application name we send it to different file locations. I am not sure how practical this is from hive side.
If not +1 for the second approach. I don’t think there’s anything wrong running multiple hive scripts (since we will be executing a single hive execution task) and there will be multiple map reduce jobs running per each app for a given tenant. And this task will happen in the back ground on a separate thread at a given scheduled time which will not impact any performance. On Mon, Sep 23, 2013 at 3:16 PM, Janaka Ranabahu <[email protected]> wrote: > Hi All, > > Currently the log Summarizer can only archive log files per tenant. We > need to extend that facility and archive files per tenant application. What > is the best way to achieve this? > > We have tried the following. > 1. Append the appName to the Cassandra column family name. This is the > easiest method but it leads to issues when there are long application names > and when there are application names with characters like "/". So we didn't > continue on that approach. > > 2. Modify the existing log summarizer code to run per tenant application. > How we are doing that is that, we use a hive query to get the app names > from BAM and create queries per each tenant application. This is done by > extending the current code base and appending the application name to the > query template. > This method generates the expected results but the number of jobs that > runs is very high. For Ex: if there are 10 apps per tenant and if there are > 3 tenants there would be close to 33 queries that executes. > > Therefore we need to clarify about the best approach to follow to achieve > this task. > > Thanks, > Janaka > > -- > *Janaka Ranabahu* > Senior Software Engineer; WSO2 Inc.; http://wso2.com* > > E-mail: [email protected] > **M: **+94 718370861** > > *Lean . Enterprise . Middleware > -- Amani Soysa Senior Software Engineer Mobile: +94772325528 WSO2, Inc. | http://wso2.com/ Lean . Enterprise . Middleware
_______________________________________________ Architecture mailing list [email protected] https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
