Re: [Qgis-user] [QGIS-Developer] Model for Temporal Measurement (vector) Datasets ?

[Anita Graser via Qgis-user]

Hi Richard, Hi Tim,

Here's what I could come up with:
https://anitagraser.com/2022/10/01/visualizing-iot-time-series-with-qgis-mobilitydb/

I think it's promising but lacks GUI support.

Regards,

Anita


On 05.09.2022 01:34, Tim Sutton wrote:

Hi Richard

Thanks for clarifying your ideas. Could we not implement something
similar to how WMS-T works in that we have a PostgreSQL-T provider
extension for example that passes a time filter to the underlying data
base request. In another system we are build for a client we have some
logic to fetch min/max/mean/sum/etc from a time slice so we could have
something similar such that each point has only one value for the
current time slice.

Not sure if I make myself clear or not...


Regards

Tim

On Sun, Sep 4, 2022 at 4:22 PM Anita Graser via QGIS-Developer
 wrote:

Hi Richard,

On 02.09.2022 13:07, Richard Duivenvoorde wrote:
> Anybody is aware of a model in which data owns a location, but THAT
> location has a (growing) set of [timestamp, value] observation
data to
> it

MobilityDB has temporal temporal bools, ints, floats, texts, and
geometries: https://docs.mobilitydb.com/MobilityDB/develop/ch03.html

So that would cover the storage question but the visualization issue
remains until we implement support for these types in Temporal
Controller.

Regards,

Anita


___
QGIS-Developer mailing list
qgis-develo...@lists.osgeo.org
List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer



--
--
​

Tim Sutton
Visit http://kartoza.com to find out about open source:
 * Desktop GIS programming services
 * Geospatial web development
* GIS Training
* Consulting Services
Tim is a member of the QGIS Project Steering Committee
---
___
Qgis-user mailing list
Qgis-user@lists.osgeo.org
List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user

Re: [Qgis-user] [QGIS-Developer] Model for Temporal Measurement (vector) Datasets ?

[Tim Sutton via Qgis-user]

Hi Richard

Thanks for clarifying your ideas. Could we not implement something similar
to how WMS-T works in that we have a PostgreSQL-T provider extension for
example that passes a time filter to the underlying data base request. In
another system we are build for a client we have some logic to fetch
min/max/mean/sum/etc from a time slice so we could have something similar
such that each point has only one value for the current time slice.

Not sure if I make myself clear or not...


Regards

Tim

On Sun, Sep 4, 2022 at 4:22 PM Anita Graser via QGIS-Developer <
qgis-develo...@lists.osgeo.org> wrote:

> Hi Richard,
>
> On 02.09.2022 13:07, Richard Duivenvoorde wrote:
> > Anybody is aware of a model in which data owns a location, but THAT
> > location has a (growing) set of [timestamp, value] observation data to
> > it
>
> MobilityDB has temporal temporal bools, ints, floats, texts, and
> geometries: https://docs.mobilitydb.com/MobilityDB/develop/ch03.html
>
> So that would cover the storage question but the visualization issue
> remains until we implement support for these types in Temporal Controller.
>
> Regards,
>
> Anita
>
>
> ___
> QGIS-Developer mailing list
> qgis-develo...@lists.osgeo.org
> List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
> Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer
>


-- 
--
​

Tim Sutton
Visit http://kartoza.com to find out about open source:
 * Desktop GIS programming services
 * Geospatial web development
* GIS Training
* Consulting Services
Tim is a member of the QGIS Project Steering Committee
---
___
Qgis-user mailing list
Qgis-user@lists.osgeo.org
List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user

Re: [Qgis-user] [QGIS-Developer] Model for Temporal Measurement (vector) Datasets ?

[Brent Wood via Qgis-user]

Hi Richard,

I think the underlying issue here is that QGIS does not do data management... 
It just accesses data (whether not managed, managed or mismanaged).

You are correct, the traditional GIS model of a feature with attributes does 
not map to the real world very well in many use cases, and timeseries data is 
one. GIS is seldom used to manage data these days, with very powerful spatially 
enabled databases readily available. To use your IOT approach where location is 
just an attribute, in a relational db, the location IS just another attribute. 
Unlike the GIS model, where the spatial feature is somehow different, here it 
isn't. A record can have multiple geometries, times, dates, strings, numerics, 
etc... like start time, end time, start location, end location, trackline 
between them, if you want... impossible without nasty hacks in a GIS centric 
data store.

So you have a pretty standard situation where you have a set of sites where 
data is captured in an ongoing basis.

I have designed Postgis databases to do this... most recently, when 
Postgres/Postgis was staring to slow down a bit with 600,000,000 readings I 
moved to Postgres/Postgis/TimescaleDB which is returning typical query results 
in 10's of milliseconds with 4 billion readings.

You don't just have sites with readings, you have sites, sensors, calibrations, 
instruments, readings, personnel, etc... there is a whole lot of metadata/data 
pertaining to your setup that should probably be managed in a database. I don't 
know if you will have enough data to justify using TimescaleDB, but you should 
be using some sort of db to manage your data effectively & efficiently. And 
Postgis/Postgres is a hard combination to beat as a tool for doing this well.

Then you point QGIS at your database for the mapping, cartography, 
visualisation, etc. You can use database views to simplify queries, but there 
are issues with QGIS and views. I find using the QGIS DB manager with a query 
instantiated as a QGIS layer works much better than accessing a view in many 
cases.

And for an alternative mapping of IOT data to a data model, the Object 
Relational approach provided by Postgres supports non-relational structures 
like key/value (hstore) or jsonb data storage.


52North have a (somewhat complex) Postgis database design to provide a data 
store for their SOS (Sensor Observation Service) software. Perhaps overkill for 
your use case, but a very complete and robust solution. See: 
https://www.researchgate.net/figure/The-52-North-SOS-schema_fig14_327124727


Hope this helps, there are certainly effective ways to do what you want out 
there.


Cheers

Brent Wood

Principal Technician, Fisheries
NIWA
DDI:  +64 (4) 3860529



On 2 Sep 2022, at 13:07, Richard Duivenvoorde via QGIS-Developer 
 wrote:

Hi All,

Anybody is aware of a model in which data owns a location, but THAT location 
has a (growing) set of [timestamp, value] observation data to it

In GIS/QGIS everything is tabular (talking about vector here, I know meshes can 
have more shapes).

But in the IOT world the 'location' is 'just an attribute', and the 'other' 
data is actually more of interest (and growing in time).

Currently to view measurements, 'we' often replicate the location for every 
value/measurement: we create a 'Feature' for every time step (eg for example 
via a WFS)

The OGC has a SensorThingsApi (STA) standard, in which for given location you 
can request all values/observations (of filter a certain sensortype/parameter). 
So one geometry has a full table of data to it.

BUT to 'work' with that data in QGIS, you always have to 'flatten' it, one way 
or another, (I think), for every time step: create a feature with: location, 
time, value...

My Question:

- isn't there some model (or can we come up with one) in which a location can 
actually have a range of time/value data (like some table join like), and when 
you use the Temporal Controller, you style your layer by RE-using the one 
location, but getting the value for current Timestamp (in the TimeController). 
Some sort of indirect filtering?

- or is this the mesh model (I know netcdf's in which you have often predefined 
meshes/location and the data/time vector is growing)?
And would a solution be to have some kind of in memory vector -> mesh loader or 
so???

I hope this makes sense to others. Any input appreciated.

Regards,

Richard Duivenvoorde

PS, if you want to play with SensorThingsApi:
- install: https://github.com/AirBreak-UIA/SensorThingsAPI_QGIS-plugin
- connect it to: 
https://airquality-frost.k8s.ilt-dmz.iosb.fraunhofer.de/v1.1/Locations
(around 5000 location and 500-million 'observations')
- load all locations in one layer, and click on a location
My point: the data is then viewable/selectable by table and graph, but NOT 
loadable in a sense way (yet) in QGIS...
___
QGIS-Developer mailing list

Re: [Qgis-user] [QGIS-Developer] Model for Temporal Measurement (vector) Datasets ?

[Richard Duivenvoorde via Qgis-user]

On 9/4/22 15:27, Matthias Kuhn wrote:

Hi Richard,

It seems to me this should be possible using:

1. Two tables:
- Points
- Measurements (Timestamp, Measured attribute)

2.  A view (Points LEFT JOIN Measurements)



What would be the actual result in your case (eg in the case of 2 points with 
100 measurements each)?

That would be 2 records/features isn't it? With..., I dont know, the last join?
OR 200 records (which is the actual way QGIS/GIS is handling this?

Note that I'm not talking about storage here.
I'm looking in a way, the temporal controller (or something else) can 
(separately from the geometries) request the timestamp/value data.
For example: FIRST requests all Locations of Sensors,
Then (based on the status of some (temporal) controller) request a subset of 
the values/timestamps.

I'm looking into this, because I'm searching for a way to handle massive 
datasets, in which it is such overkill to again and again sent over the 
geometries (be it points or polygons)...
Or another use case: I want points on the map, and timestamps/values flying in 
realtime as they are measured...
Or: you have a giant km grid covering the whole of europe, and data (per hour) 
for 7 days of data, and you can request (remotely!) the value for every grid 
cell per 10 minutes (without having to request the geometry every time...)

Regards & all thanks for the input,

Regards,

Richard



___
Qgis-user mailing list
Qgis-user@lists.osgeo.org
List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user

Re: [Qgis-user] [QGIS-Developer] Model for Temporal Measurement (vector) Datasets ?

[Richard Duivenvoorde via Qgis-user]

On 9/4/22 14:35, Tim Sutton wrote:

Hi Richard

Don’t people normally use things like https://www.timescale.com/ 


For this? In any case some of our clients that are doing things like vehicle 
tracking are…


Yes to *store* it, but my point is: given 2 points with 100 timesteps, QGIS 
will create 200 features for it to be able to display it...
So to 'view' that data in (Q)GIS, you have to flatten it to a 'table' with 200 
features...

While the actual interesting part is the value and the time, NOT those 2 xy 
fields... (the case of moving vehicles is slightly different, there the xy 
values also change)...
Maybe I'm in the wrong crowd here to tell the geoms are not so much of interest 
in my case ;-)

I'm looking for a way/model (non tabular) that QGIS creates 2 
geometries/features, and then will style/paint it based on the timestamps and 
values

The use case here: having 5000 sensors in Europe, having 500.000.000 
measurements in total, I don not want to create 500.000.000 features.

I'd preferred to create those 5000 features, and then get the 
geom-id/value/timestamp for a given filter range and repaint my canvas.

Something like 'dynamic' joins: the geometries will stay the same, but the 
'joined' columns/table will change based on the 'temporal filter' requested.

Sorry if I can not make myself more clear (or making it more complex that it 
maybe is...) :-)

Regards,

Richard Duivenvoorde
___
Qgis-user mailing list
Qgis-user@lists.osgeo.org
List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user

Re: [Qgis-user] [QGIS-Developer] Model for Temporal Measurement (vector) Datasets ?

[Tim Sutton via Qgis-user]

Hi Richard

Don’t people normally use things like https://www.timescale.com/

For this? In any case some of our clients that are doing things like vehicle 
tracking are…

Regards 

Tim

Sent from my iPhone

> On 2 Sep 2022, at 13:07, Richard Duivenvoorde via QGIS-Developer 
>  wrote:
> 
> Hi All,
> 
> Anybody is aware of a model in which data owns a location, but THAT location 
> has a (growing) set of [timestamp, value] observation data to it
> 
> In GIS/QGIS everything is tabular (talking about vector here, I know meshes 
> can have more shapes).
> 
> But in the IOT world the 'location' is 'just an attribute', and the 'other' 
> data is actually more of interest (and growing in time).
> 
> Currently to view measurements, 'we' often replicate the location for every 
> value/measurement: we create a 'Feature' for every time step (eg for example 
> via a WFS)
> 
> The OGC has a SensorThingsApi (STA) standard, in which for given location you 
> can request all values/observations (of filter a certain 
> sensortype/parameter). So one geometry has a full table of data to it.
> 
> BUT to 'work' with that data in QGIS, you always have to 'flatten' it, one 
> way or another, (I think), for every time step: create a feature with: 
> location, time, value...
> 
> My Question:
> 
> - isn't there some model (or can we come up with one) in which a location can 
> actually have a range of time/value data (like some table join like), and 
> when you use the Temporal Controller, you style your layer by RE-using the 
> one location, but getting the value for current Timestamp (in the 
> TimeController). Some sort of indirect filtering?
> 
> - or is this the mesh model (I know netcdf's in which you have often 
> predefined meshes/location and the data/time vector is growing)?
> And would a solution be to have some kind of in memory vector -> mesh loader 
> or so???
> 
> I hope this makes sense to others. Any input appreciated.
> 
> Regards,
> 
> Richard Duivenvoorde
> 
> PS, if you want to play with SensorThingsApi:
> - install: https://github.com/AirBreak-UIA/SensorThingsAPI_QGIS-plugin
> - connect it to: 
> https://airquality-frost.k8s.ilt-dmz.iosb.fraunhofer.de/v1.1/Locations
> (around 5000 location and 500-million 'observations')
> - load all locations in one layer, and click on a location
> My point: the data is then viewable/selectable by table and graph, but NOT 
> loadable in a sense way (yet) in QGIS...
> ___
> QGIS-Developer mailing list
> qgis-develo...@lists.osgeo.org
> List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
> Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer
___
Qgis-user mailing list
Qgis-user@lists.osgeo.org
List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user