Hi Dulitha, On Fri, Jun 24, 2016 at 8:11 PM, Dulitha Wijewantha <[email protected]> wrote:
> Hi Inosh, > If epoach time is received to the server I don't understand the > limitation? Is it because Siddhi time window doesn't operate in UTC time? > As I explained above, using received timestamp we extract time attributes such as year, month, day, hour, etc. and do per <time unit> based aggregations (such as per minute/hour/day counts). At the moment those time attributes are extracted based on the local time zone DAS server running. So our window boundaries are based on local time. We need to handle this cleanly for both real time time and batch analytics [1] [2]. https://wso2.org/jira/browse/CEP-1519 https://wso2.org/jira/browse/DAS-436 > Cheers~ > > On Tue, Jun 14, 2016 at 7:00 AM, Inosh Goonewardena <[email protected]> > wrote: > >> Hi Sajith, >> >> >> On Tue, Jun 14, 2016 at 12:21 PM, Sajith Ravindra <[email protected]> >> wrote: >> >>> Hi Inosh, >>> >>> In order to do this we might probably need to update the data publishers >>> to send their time zones as meta data to do the time zone conversion. >>> >> >> We receive epoch time in event timestamp field from data publishers. So >> publisher time zone is not required here. >> >> However, we have decided to not pursue with UTC time unit approach due to >> the time window boundary issue explained in above. >> >> >>> >>> In cases like daylight saving changes this information will be required >>> to correctly map the incoming time to UTC >>> >>> Also, isn't it an option to make data agents to publish their data in >>> UTC? >>> >>> On Jun 13, 2016, at 10:20 PM, Inosh Goonewardena <[email protected]> wrote: >>> >>> Hi Ruwan, >>> >>> >>> On Mon, Jun 13, 2016 at 7:13 PM, Ruwan Abeykoon <[email protected]> wrote: >>> >>>> Hi All, >>>> +1 on storing time in UTC. >>>> >>>> However this will cause some minor (but annoying) discrepancies to >>>> occur when we do windowing on "per-hour" basis, when the time zone >>>> difference between UTC and the server is not full hour. e.g. +5:30 UTC. >>>> Is there any solution for that? >>>> >>> >>> Yes. This could be a problem. For example, UTC time window [00.00 - >>> 01.00] is equals to [05.30 - 06.30] IST. And also if a timezone conversion >>> happen in a dashboard level, this again could affect graphs too. So need to >>> check those areas before moving ahead. >>> >>> >>>> >>>> Cheers, >>>> Ruwan >>>> >>>> On Mon, Jun 13, 2016 at 6:10 PM, Inosh Goonewardena <[email protected]> >>>> wrote: >>>> >>>>> Hi, >>>>> >>>>> In analytics use cases we use a couple of extensions to extract >>>>> attributes (year, month, day, hour, etc.) from the event timestamp. In >>>>> product anaytics implementations, we mostly use time:extract() Siddhi >>>>> extension [1] in real time scenarios and DateTimeUDF in batch scenarios >>>>> [2]. However, both those extensions give output time units based on the >>>>> local time zone of the server on which DAS is running. >>>>> >>>>> For example, if the DAS is running in a server which has the timezone >>>>> set as IST, for timestamp 1461135538669 (2016/04/20 06:58:58 GMT) output >>>>> time units are resolved as 2016, 04, 20, 12, 28. >>>>> >>>>> In most of the scenarios, analytics is implemented in per <time unit> >>>>> basis, i.e., we maintain summary tables for per minute, per hour, per day, >>>>> per month. These summary tables has columns for year, month, date, hour, >>>>> etc. Since aforementioned extensions are giving time units based on local >>>>> timezone what we store in there are are local time units. >>>>> >>>>> IMO, we should store UTC time units instead, since it is better to >>>>> maintain time units uniformly without depending on the time zone of the >>>>> server DAS is running. We have also found that this inconsistency is >>>>> capable of producing issues in incremental data processing. >>>>> >>>>> Shall we extend our analytics extensions to support UTC time units >>>>> throughout? >>>>> >>>>> [1] >>>>> https://github.com/wso2/siddhi/blob/master/modules/siddhi-extensions/time/src/main/java/org/wso2/siddhi/extension/time/ExtractAttributesFunctionExtension.java >>>>> [2] >>>>> https://github.com/wso2/shared-analytics/blob/master/components/spark-udf/org.wso2.carbon.analytics.shared.spark.common.udf/src/main/java/org/wso2/carbon/analytics/shared/common/udf/DateTimeUDF.java >>>>> >>>>> -- >>>>> Thanks & Regards, >>>>> >>>>> Inosh Goonewardena >>>>> Associate Technical Lead- WSO2 Inc. >>>>> Mobile: +94779966317 >>>>> >>>>> _______________________________________________ >>>>> Architecture mailing list >>>>> [email protected] >>>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>>>> >>>>> >>>> >>>> >>>> -- >>>> >>>> *Ruwan Abeykoon* >>>> *Associate Director/Architect**,* >>>> *WSO2, Inc. http://wso2.com <http://wso2.com/> * >>>> *lean.enterprise.middleware.* >>>> >>>> email: [email protected] >>>> >>>> _______________________________________________ >>>> Architecture mailing list >>>> [email protected] >>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>>> >>>> >>> >>> >>> -- >>> Thanks & Regards, >>> >>> Inosh Goonewardena >>> Associate Technical Lead- WSO2 Inc. >>> Mobile: +94779966317 >>> >>> _______________________________________________ >>> Architecture mailing list >>> [email protected] >>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>> >>> >>> _______________________________________________ >>> Architecture mailing list >>> [email protected] >>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>> >>> >> >> >> -- >> Thanks & Regards, >> >> Inosh Goonewardena >> Associate Technical Lead- WSO2 Inc. >> Mobile: +94779966317 >> >> _______________________________________________ >> Architecture mailing list >> [email protected] >> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >> >> > > > -- > Dulitha Wijewantha (Chan) > Software Engineer - Mobile Development > WSO2 Inc > Lean.Enterprise.Middleware > * ~Email [email protected] <[email protected]>* > * ~Mobile +94712112165 <%2B94712112165>* > * ~Website dulitha.me <http://dulitha.me>* > * ~Twitter @dulitharw <https://twitter.com/dulitharw>* > *~Github @dulichan <https://github.com/dulichan>* > *~SO @chan <http://stackoverflow.com/users/813471/chan>* > > _______________________________________________ > Architecture mailing list > [email protected] > https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture > > -- Thanks & Regards, Inosh Goonewardena Associate Technical Lead- WSO2 Inc. Mobile: +94779966317
_______________________________________________ Architecture mailing list [email protected] https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
