Hi Sajith,
On Tue, Jun 14, 2016 at 12:21 PM, Sajith Ravindra <[email protected]> wrote: > Hi Inosh, > > In order to do this we might probably need to update the data publishers > to send their time zones as meta data to do the time zone conversion. > We receive epoch time in event timestamp field from data publishers. So publisher time zone is not required here. However, we have decided to not pursue with UTC time unit approach due to the time window boundary issue explained in above. > > In cases like daylight saving changes this information will be required to > correctly map the incoming time to UTC > > Also, isn't it an option to make data agents to publish their data in UTC? > > On Jun 13, 2016, at 10:20 PM, Inosh Goonewardena <[email protected]> wrote: > > Hi Ruwan, > > > On Mon, Jun 13, 2016 at 7:13 PM, Ruwan Abeykoon <[email protected]> wrote: > >> Hi All, >> +1 on storing time in UTC. >> >> However this will cause some minor (but annoying) discrepancies to occur >> when we do windowing on "per-hour" basis, when the time zone difference >> between UTC and the server is not full hour. e.g. +5:30 UTC. >> Is there any solution for that? >> > > Yes. This could be a problem. For example, UTC time window [00.00 - 01.00] > is equals to [05.30 - 06.30] IST. And also if a timezone conversion happen > in a dashboard level, this again could affect graphs too. So need to check > those areas before moving ahead. > > >> >> Cheers, >> Ruwan >> >> On Mon, Jun 13, 2016 at 6:10 PM, Inosh Goonewardena <[email protected]> >> wrote: >> >>> Hi, >>> >>> In analytics use cases we use a couple of extensions to extract >>> attributes (year, month, day, hour, etc.) from the event timestamp. In >>> product anaytics implementations, we mostly use time:extract() Siddhi >>> extension [1] in real time scenarios and DateTimeUDF in batch scenarios >>> [2]. However, both those extensions give output time units based on the >>> local time zone of the server on which DAS is running. >>> >>> For example, if the DAS is running in a server which has the timezone >>> set as IST, for timestamp 1461135538669 (2016/04/20 06:58:58 GMT) output >>> time units are resolved as 2016, 04, 20, 12, 28. >>> >>> In most of the scenarios, analytics is implemented in per <time unit> >>> basis, i.e., we maintain summary tables for per minute, per hour, per day, >>> per month. These summary tables has columns for year, month, date, hour, >>> etc. Since aforementioned extensions are giving time units based on local >>> timezone what we store in there are are local time units. >>> >>> IMO, we should store UTC time units instead, since it is better to >>> maintain time units uniformly without depending on the time zone of the >>> server DAS is running. We have also found that this inconsistency is >>> capable of producing issues in incremental data processing. >>> >>> Shall we extend our analytics extensions to support UTC time units >>> throughout? >>> >>> [1] >>> https://github.com/wso2/siddhi/blob/master/modules/siddhi-extensions/time/src/main/java/org/wso2/siddhi/extension/time/ExtractAttributesFunctionExtension.java >>> [2] >>> https://github.com/wso2/shared-analytics/blob/master/components/spark-udf/org.wso2.carbon.analytics.shared.spark.common.udf/src/main/java/org/wso2/carbon/analytics/shared/common/udf/DateTimeUDF.java >>> >>> -- >>> Thanks & Regards, >>> >>> Inosh Goonewardena >>> Associate Technical Lead- WSO2 Inc. >>> Mobile: +94779966317 >>> >>> _______________________________________________ >>> Architecture mailing list >>> [email protected] >>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>> >>> >> >> >> -- >> >> *Ruwan Abeykoon* >> *Associate Director/Architect**,* >> *WSO2, Inc. http://wso2.com <http://wso2.com/> * >> *lean.enterprise.middleware.* >> >> email: [email protected] >> >> _______________________________________________ >> Architecture mailing list >> [email protected] >> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >> >> > > > -- > Thanks & Regards, > > Inosh Goonewardena > Associate Technical Lead- WSO2 Inc. > Mobile: +94779966317 > > _______________________________________________ > Architecture mailing list > [email protected] > https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture > > > _______________________________________________ > Architecture mailing list > [email protected] > https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture > > -- Thanks & Regards, Inosh Goonewardena Associate Technical Lead- WSO2 Inc. Mobile: +94779966317
_______________________________________________ Architecture mailing list [email protected] https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
