Re: [QGIS-Developer] QGIS, DateTime and TimeZones
On Tue, 8 Sep 2020 at 23:40, Richard Duivenvoorde wrote: > > Hi Devs, > > I was hitting some timezone issues when I received some csv data in which > there are timestamps like: > "2020-07-25 20:21:38 UTC" > Apparently if you read this into QGIS, the UTC IS interpreted, and you get a > DateTime in your data which is (in my case) -2 hours(CET) off (well in some > parts of QGIS) > > This plays havoc if you for example are going to use this data in the > timecontroller or expressions, see > https://github.com/qgis/QGIS/issues/38647 > > So I think that now the TimeController is part of QGIS, and people will start > to play with Time-based data more and more (from TimeZone aware datasets like > csv and postgres) we will run into issues like above. > > Is there something we can do to make QGIS timezone aware? > OR should we stay far away from it (timezones are hell)... > OR should we wait with it after QGIS 4 This is a good discussion, and a very complex issue. Here's some summary points from a recent off-list discussion I had regarding this very topic: - we don't expose time zones anywhere in qgis. while the underlying QDateTime class has good support for timezone handling, there's no-where in QGIS you can "see " the time zone for a datetime value, and there's no functionality present in -the expression engine or processing to work with time zones/manipulate them. - different data providers are handling time zones differently, but most are just giving date time values as a "local time"/"no time zone available" value. As described above, this inconsistency leads to issues when you compare values from different sources. Even though the look the same in QGIS, they aren't!! - some underlying data sources just don't have any time zone capabilities, or have limited capabilities. E.g. GDAL has a flag if a datetime is UTC or "not utc/unknown", and that's it. I think we have no choice but to start exposing time zone information to users everywhere we have dates, and move to treating dates without a timezone as being in "local time" (which may cause issues if projects + data are shared between users in different time zones). We should add timezone functionality to the expression engine, and algorithms for converting time zones to processing. We'd also need to add a "manual time zone override" option to temporal enabled layers -- so for those sources with ambiguous time zone information the user has some capacity to manually pick the correct time zone. It's definitely achievable, but it's not a trivial amount of work. I'd estimate we're looking at ~5-7 days to do this properly (and it needs to be done properly, covering all uses of datetime values all throughout QGIS and the different data providers! Hacking in one-by-one fixes in individual data providers will just make the situation worse)... Nyall > > Regards, > > Richard Duivenvoorde > > > > ___ > QGIS-Developer mailing list > QGIS-Developer@lists.osgeo.org > List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer > Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer ___ QGIS-Developer mailing list QGIS-Developer@lists.osgeo.org List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Re: [QGIS-Developer] QGIS, DateTime and TimeZones
I recently had some issues concerning datetimes. Like in real life, I had to merge data from different sources, csv and gpx, with some duplicate entries. After a while I wondered why I couldn't find those duplicates. For some reasons, the time values where interpreted differently while merging, depending on the source and formatting, so I got an offset of 2 hours for values which should have been identical. So there seems to be some black magic going on under the hood which I can't influence when I would not want to change all entries by hand to make them behave consistently. Just my 2 cents Bernd On 08.09.20 15:40, Richard Duivenvoorde wrote: Hi Devs, I was hitting some timezone issues when I received some csv data in which there are timestamps like: "2020-07-25 20:21:38 UTC" Apparently if you read this into QGIS, the UTC IS interpreted, and you get a DateTime in your data which is (in my case) -2 hours(CET) off (well in some parts of QGIS) This plays havoc if you for example are going to use this data in the timecontroller or expressions, see https://github.com/qgis/QGIS/issues/38647 So I think that now the TimeController is part of QGIS, and people will start to play with Time-based data more and more (from TimeZone aware datasets like csv and postgres) we will run into issues like above. Is there something we can do to make QGIS timezone aware? OR should we stay far away from it (timezones are hell)... OR should we wait with it after QGIS 4 Regards, Richard Duivenvoorde ___ QGIS-Developer mailing list QGIS-Developer@lists.osgeo.org List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer ___ QGIS-Developer mailing list QGIS-Developer@lists.osgeo.org List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Re: [QGIS-Developer] QGIS, DateTime and TimeZones
Richard, I would second the desire to make QGIS more timezone aware. QDateTime supports time zones so QGIS does support it, but not much effort has gone into really supporting it. I currently have a task to do a QGIS date/time conversion tool with algorithms to convert between different time formats including EPOCH time. This was something that I was planning on starting soon. Like you said though time zones are going to be a real pain especially if you want to take a coordinate and convert it to a time zone. There are some python libraries I have been investigating to help with the task. Calvin On Tue, Sep 8, 2020 at 9:40 AM Richard Duivenvoorde wrote: > Hi Devs, > > I was hitting some timezone issues when I received some csv data in which > there are timestamps like: > "2020-07-25 20:21:38 UTC" > Apparently if you read this into QGIS, the UTC IS interpreted, and you get > a DateTime in your data which is (in my case) -2 hours(CET) off (well in > some parts of QGIS) > > This plays havoc if you for example are going to use this data in the > timecontroller or expressions, see > https://github.com/qgis/QGIS/issues/38647 > > So I think that now the TimeController is part of QGIS, and people will > start to play with Time-based data more and more (from TimeZone aware > datasets like csv and postgres) we will run into issues like above. > > Is there something we can do to make QGIS timezone aware? > OR should we stay far away from it (timezones are hell)... > OR should we wait with it after QGIS 4 > > Regards, > > Richard Duivenvoorde > > > > ___ > QGIS-Developer mailing list > QGIS-Developer@lists.osgeo.org > List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer > Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer ___ QGIS-Developer mailing list QGIS-Developer@lists.osgeo.org List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer
[QGIS-Developer] QGIS, DateTime and TimeZones
Hi Devs, I was hitting some timezone issues when I received some csv data in which there are timestamps like: "2020-07-25 20:21:38 UTC" Apparently if you read this into QGIS, the UTC IS interpreted, and you get a DateTime in your data which is (in my case) -2 hours(CET) off (well in some parts of QGIS) This plays havoc if you for example are going to use this data in the timecontroller or expressions, see https://github.com/qgis/QGIS/issues/38647 So I think that now the TimeController is part of QGIS, and people will start to play with Time-based data more and more (from TimeZone aware datasets like csv and postgres) we will run into issues like above. Is there something we can do to make QGIS timezone aware? OR should we stay far away from it (timezones are hell)... OR should we wait with it after QGIS 4 Regards, Richard Duivenvoorde ___ QGIS-Developer mailing list QGIS-Developer@lists.osgeo.org List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Re: [QGIS-Developer] py3_env in Windows installers
Hi Richard, On Tue, 08. Sep 2020 at 10:45:21 +0200, Richard Duivenvoorde wrote: > On 9/8/20 10:32 AM, Jürgen E. Fischer wrote: > > > I know this was to be able to handle both py2 and py3, but I do not think > > > we need to use python2 anymore is it? > > py2 is still available in osgeo4w - although we don't use it anymore for > > QGIS. > IS it still used by other osgeo4w components? Would be nice to clean it out > if that was possible (as py2 not supported etc etc)? None non-obsolete components at least. That probably also applies to the VC 2008 runtime. > > > In that case I would propose to make py3_env part of the osgeo4w.bat file > > > so we can instruct people to just go to the osgeo4w bat file to eventually > > > pip install modules usable in QGIS? > > You could also just move py3_env.bat to etc\ini\python3-core.bat and create > > an > > empty py3_env.bat. > That is for this user yes? Or are you talking about in the sources? Which user?I'm talking about osgeo4w. There's a etc/ini/python-core.bat in python-core (2) that makes python2 default if it's installed and that used to be the default installed python. py3_env.bat was added to switch from there to python3. If we were going to change that, the contents of py3_env.bat should be moved to etc/ini - so that it's sourced by o4w_env.bat, which all the batch files that start applications source. The osgeo4w shell itself doesn't use any python. All of the above is osgeo4w and not QGIS. Jürgen -- Jürgen E. Fischer norBIT GmbH Tel. +49-4931-918175-31 Dipl.-Inf. (FH) Rheinstraße 13 Fax. +49-4931-918175-50 Software Engineer D-26506 Nordenhttps://www.norbit.de QGIS release manager (PSC) GermanyIRC: jef on FreeNode signature.asc Description: PGP signature ___ QGIS-Developer mailing list QGIS-Developer@lists.osgeo.org List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Re: [QGIS-Developer] py3_env in Windows installers
On 9/8/20 10:32 AM, Jürgen E. Fischer wrote: >> I know this was to be able to handle both py2 and py3, but I do not think we >> need to use python2 anymore is it? > > py2 is still available in osgeo4w - although we don't use it anymore for QGIS. IS it still used by other osgeo4w components? Would be nice to clean it out if that was possible (as py2 not supported etc etc)? >> In that case I would propose to make py3_env part of the osgeo4w.bat file so >> we can instruct people to just go to the osgeo4w bat file to eventually pip >> install modules usable in QGIS? > > You could also just move py3_env.bat to etc\ini\python3-core.bat and create an > empty py3_env.bat. That is for this user yes? Or are you talking about in the sources? I was searching for those files (py3_env etc) in sources: https://github.com/qgis/QGIS/tree/master/ms-windows also googled https://trac.osgeo.org/osgeo4w but could not find it. Regards, Richard Duivenvoorde ___ QGIS-Developer mailing list QGIS-Developer@lists.osgeo.org List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Re: [QGIS-Developer] py3_env in Windows installers
Hi Richard, On Tue, 08. Sep 2020 at 09:05:05 +0200, Richard Duivenvoorde wrote: > If you start the osgeo4w command prompt, both in standalone and osgeo4w > version, you have to run py3_env before being able to use (the QGIS used) > python there (or am I wrong. I tested by running osgeo4w.bat within a 3.14 > standalone install) > I know this was to be able to handle both py2 and py3, but I do not think we > need to use python2 anymore is it? py2 is still available in osgeo4w - although we don't use it anymore for QGIS. > In that case I would propose to make py3_env part of the osgeo4w.bat file so > we can instruct people to just go to the osgeo4w bat file to eventually pip > install modules usable in QGIS? You could also just move py3_env.bat to etc\ini\python3-core.bat and create an empty py3_env.bat. Jürgen -- Jürgen E. Fischer norBIT GmbH Tel. +49-4931-918175-31 Dipl.-Inf. (FH) Rheinstraße 13 Fax. +49-4931-918175-50 Software Engineer D-26506 Nordenhttps://www.norbit.de QGIS release manager (PSC) GermanyIRC: jef on FreeNode signature.asc Description: PGP signature norBIT Gesellschaft fuer Unternehmensberatung und Informationssysteme mbH Rheinstrasse 13, 26506 Norden GF: Juergen Fischer, Nils Kutscher HR: Amtsgericht Aurich HRB 100827 Datenschutzerklaerung: https://www.norbit.de/83/ ___ QGIS-Developer mailing list QGIS-Developer@lists.osgeo.org List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer
[QGIS-Developer] Fwd: [Qgis-user] Latest runtime version (2008?)
Hi Lukas, I am forwarding your message to the Developers list https://lists.osgeo.org/mailman/listinfo/qgis-developer I'm not sure if the 2008 runtime is actually being used (for 3.10), I hope Juergen can maybe answer this question. Regards, Richard Duivenvoorde Forwarded Message Subject:[Qgis-user] Latest runtime version Date: Tue, 8 Sep 2020 09:57:17 +0200 From: Lukas Fiddrich To: qgis-u...@lists.osgeo.org Dear QGIS Community, Can you help me with the runtime version of QGIS 3? As For using QGIS at my workplace, it need to get permitted by the IT-Department. They figured out, that QGIS 3.10.9 has an outdated C++ runtime environment (Microsoft Visual C++ 2008 Redistributable - x64 9.0.30729.17). Is there any newer or older QGIS 3 version with a still supported C++ runtime environment? If I am wrong here with this question, could you give me a hint where I can adress this question correctly? Kind regards Lukas ___ Qgis-user mailing list qgis-u...@lists.osgeo.org List info: https://lists.osgeo.org/mailman/listinfo/qgis-user Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user___ QGIS-Developer mailing list QGIS-Developer@lists.osgeo.org List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer
[QGIS-Developer] py3_env in Windows installers
Hi Devs, If you start the osgeo4w command prompt, both in standalone and osgeo4w version, you have to run py3_env before being able to use (the QGIS used) python there (or am I wrong. I tested by running osgeo4w.bat within a 3.14 standalone install) I know this was to be able to handle both py2 and py3, but I do not think we need to use python2 anymore is it? In that case I would propose to make py3_env part of the osgeo4w.bat file so we can instruct people to just go to the osgeo4w bat file to eventually pip install modules usable in QGIS? Regards, Richard Duivenvoorde ps I'm planning to write a blog post about using the command lines from withing QGIS installers, as I think it can be very helpfull for a lot of people.. ___ QGIS-Developer mailing list QGIS-Developer@lists.osgeo.org List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer
