Hi Inosh, In order to do this we might probably need to update the data publishers to send their time zones as meta data to do the time zone conversion.
In cases like daylight saving changes this information will be required to correctly map the incoming time to UTC Also, isn't it an option to make data agents to publish their data in UTC? > On Jun 13, 2016, at 10:20 PM, Inosh Goonewardena <[email protected]> wrote: > > Hi Ruwan, > > >> On Mon, Jun 13, 2016 at 7:13 PM, Ruwan Abeykoon <[email protected]> wrote: >> Hi All, >> +1 on storing time in UTC. >> >> However this will cause some minor (but annoying) discrepancies to occur >> when we do windowing on "per-hour" basis, when the time zone difference >> between UTC and the server is not full hour. e.g. +5:30 UTC. >> Is there any solution for that? > > Yes. This could be a problem. For example, UTC time window [00.00 - 01.00] is > equals to [05.30 - 06.30] IST. And also if a timezone conversion happen in a > dashboard level, this again could affect graphs too. So need to check those > areas before moving ahead. > >> >> Cheers, >> Ruwan >> >>> On Mon, Jun 13, 2016 at 6:10 PM, Inosh Goonewardena <[email protected]> wrote: >>> Hi, >>> >>> In analytics use cases we use a couple of extensions to extract attributes >>> (year, month, day, hour, etc.) from the event timestamp. In product >>> anaytics implementations, we mostly use time:extract() Siddhi extension [1] >>> in real time scenarios and DateTimeUDF in batch scenarios [2]. However, >>> both those extensions give output time units based on the local time zone >>> of the server on which DAS is running. >>> >>> For example, if the DAS is running in a server which has the timezone set >>> as IST, for timestamp 1461135538669 (2016/04/20 06:58:58 GMT) output time >>> units are resolved as 2016, 04, 20, 12, 28. >>> >>> In most of the scenarios, analytics is implemented in per <time unit> >>> basis, i.e., we maintain summary tables for per minute, per hour, per day, >>> per month. These summary tables has columns for year, month, date, hour, >>> etc. Since aforementioned extensions are giving time units based on local >>> timezone what we store in there are are local time units. >>> >>> IMO, we should store UTC time units instead, since it is better to maintain >>> time units uniformly without depending on the time zone of the server DAS >>> is running. We have also found that this inconsistency is capable of >>> producing issues in incremental data processing. >>> >>> Shall we extend our analytics extensions to support UTC time units >>> throughout? >>> >>> [1] >>> https://github.com/wso2/siddhi/blob/master/modules/siddhi-extensions/time/src/main/java/org/wso2/siddhi/extension/time/ExtractAttributesFunctionExtension.java >>> [2] >>> https://github.com/wso2/shared-analytics/blob/master/components/spark-udf/org.wso2.carbon.analytics.shared.spark.common.udf/src/main/java/org/wso2/carbon/analytics/shared/common/udf/DateTimeUDF.java >>> >>> -- >>> Thanks & Regards, >>> >>> Inosh Goonewardena >>> Associate Technical Lead- WSO2 Inc. >>> Mobile: +94779966317 >>> >>> _______________________________________________ >>> Architecture mailing list >>> [email protected] >>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >> >> >> >> -- >> Ruwan Abeykoon >> Associate Director/Architect, >> WSO2, Inc. http://wso2.com >> lean.enterprise.middleware. >> >> email: [email protected] >> >> _______________________________________________ >> Architecture mailing list >> [email protected] >> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture > > > > -- > Thanks & Regards, > > Inosh Goonewardena > Associate Technical Lead- WSO2 Inc. > Mobile: +94779966317 > _______________________________________________ > Architecture mailing list > [email protected] > https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
_______________________________________________ Architecture mailing list [email protected] https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
