Re: [QGIS-Developer] QGIS "fast" mode

2020-03-20 Thread Alessandro Pasotti
On Fri, Mar 20, 2020 at 9:08 AM Julien Cabieces
 wrote:
>
>
> Hi,
>
> > Hi, can someone explain what is the real logic currently coded for trust
> > option?
>
> It calls the QgsProject::setTrustLayerMetaData(bool) which says
>
> > Sets the trust option allowing to indicate if the extent has to be read 
> > from the XML document when data source has no metadata or if the data 
> > provider has to determine it.
> > Moreover, when this option is activated, primary key unicity is not checked 
> > for views and materialized views with Postgres provider.
>
> And from what I see from the code it does what it says, no less, no
> more. Only for PostGres.
>
> As a result the variable checkPrimaryKeyUnicity is set to 1 in the uri.
>
> It also add a trust node in project configuration for every layer, but I
> don't see where it's used in QGIS code (I maybe miss something)
>

So far so good, but I'd really like to stop having all these
differences among providers, there is too much Postgres specific
implementation in QGIS now.

At least for use estimated metadata, and "trust" options we should
agree on a minimal set of provider-agnostic options that the different
providers can support and expose through capability flags.

The options should be ideally settable per-layer (as part of the data
source URI) and with a per-project default that applies when there is
no such information in the individual layer URI.

My main concern is that we currently have a whole lot of
undocumented/hidden behaviors and slight variations among different
providers and we should really try hard to homogenize them and
eventually converge towards a unified interface with more shared
capabilities.

Kind regards

-- 
Alessandro Pasotti
w3:   www.itopen.it
___
QGIS-Developer mailing list
QGIS-Developer@lists.osgeo.org
List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer

Re: [QGIS-Developer] QGIS "fast" mode

2020-03-20 Thread Julien Cabieces

Hi, 

> Hi, can someone explain what is the real logic currently coded for trust
> option? 

It calls the QgsProject::setTrustLayerMetaData(bool) which says

> Sets the trust option allowing to indicate if the extent has to be read from 
> the XML document when data source has no metadata or if the data provider has 
> to determine it.
> Moreover, when this option is activated, primary key unicity is not checked 
> for views and materialized views with Postgres provider.

And from what I see from the code it does what it says, no less, no
more. Only for PostGres.

As a result the variable checkPrimaryKeyUnicity is set to 1 in the uri.

It also add a trust node in project configuration for every layer, but I
don't see where it's used in QGIS code (I maybe miss something)

Regards,
Julien
___
QGIS-Developer mailing list
QGIS-Developer@lists.osgeo.org
List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer

Re: [QGIS-Developer] QGIS "fast" mode

2020-03-20 Thread René-Luc Dhont

Hi Régis,

The trust option is used in QgProject with 
QgsVectorLayer::setReadExtentFromXml

https://github.com/qgis/QGIS/blob/master/src/core/qgsproject.cpp#L3304

Then in the case of a postgres layer, the checkUnicityKey is disabled
https://github.com/qgis/QGIS/blob/master/src/core/qgsvectorlayer.cpp#L1684

And it's all. The trust option only avoid to calculate layer Extent and 
verify primary key for postgres layers.


The estimatedMetadata layer option is more common in provider. It will 
be great to can check/uncheck this option.


Regards
René-luc

Le 19/03/2020 à 19:30, Régis Haubourg a écrit :
Hi, can someone explain what is the real logic currently coded for 
trust option? It feels like this topic raises again each year for 5 
years now, and we have regressions and what it is supposed to do.


It was funded at start to not check at all any metadata on the 
datasource and read only qgs informations

Did it change for some reason ? Why?

It should be adequate even for big databases of QGIS stores all the 
required informations.


Regards
Regis



___
QGIS-Developer mailing list
QGIS-Developer@lists.osgeo.org
List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer

Re: [QGIS-Developer] QGIS "fast" mode

2020-03-19 Thread Régis Haubourg
Hi, can someone explain what is the real logic currently coded for trust
option? It feels like this topic raises again each year for 5 years now,
and we have regressions and what it is supposed to do.

It was funded at start to not check at all any metadata on the datasource
and read only qgs informations
Did it change for some reason ? Why?

It should be adequate even for big databases of QGIS stores all the
required informations.

Regards
Regis

Le jeu. 19 mars 2020 à 18:05, René-Luc Dhont  a écrit :

> Hi Tomas,
>
> The way trust option works is not enough for big databases and big
> project with more than 100 layers. It is what Michaël and I experiment.
>
> Then the problem with changes data is more about the layer extent. For
> example a natural observations layer is designed to accept data on a
> territory but t the start of the project the layer extent is null and
> will growing with data inserted. The trust option cannot be used at the
> project start unless the user can set the available extent, or can
> defined the trust option for each layer.
>
> It will be great to set trust option at the layer level so QGIS can
> trust the layer information provided by the XML : QLR or QGS content
> It will also be great to can set some technical metadata like a layer
> available extent as the geographic area for which a selected CRS is
> valid for use.
>
> Regards,
> René-Luc
>
> Le 18/03/2020 à 22:01, Tomas Straupis a écrit :
> > 2020-03-18, tr, 21:41 kimaidou rašė:
> >> # only few requests are avoided as you pointed out so the performance
> improves "only" a bit
> >In large databases those few requests take minutes and sometimes
> > even hours... For servers even 30 seconds are too much when you're
> > trying to add a new QGIS server process to existing say 10 while on
> > high load doing 500 requests per second.
> >
> >> # sometimes you have some layers with changing data, and there is
> actually no way to trust only a subset of layers inside the project
> >1. If geometry types are changing by design, then checking geometry
> > type upon project loading is not enough. Then you need to ALWAYS
> > filter your results by requested geometry type. But only if it is the
> > case of varying geometry types. In my opinion, developer of the
> > layer/query should know beforehand if it is possible for geometry
> > types to variate and then create a view filtering on geometry type (or
> > it could be an option in QGIS ~"filter on geometry type").
> >
> >2. If database schemas are changing on-line then I would say
> > something is very wrong with the IT processes. Changes should start on
> > dev environment where data changes and QGIS project changes as well.
> > Then changes to db structure go to other environments up to production
> > in patches TOGETHER with updated QGIS project. System (in this case
> > QGIS) should never (in my experience) try to "fix" the problem because
> > it does not know all required information: maybe the project was
> > opened in incorrect environment, maybe it is a temporary problem,
> > maybe the real table is missing and you're accidentally trying to
> > query incorrect table which was never supposed to be queried -> strict
> > rules/control. Of course there could be a button "refresh" on a layer
> > to allow operator to re-query layer information upon manual request.
> >
> >> Do you think it would be interesting to have the trust option per layer
> and not only per project?
> >Theoretically there could be very different data sources, but in my
> > opinion if organisation uses strict IT processes then all layers will
> > be strictly managed (you would rarely/never have direct access to the
> > database of a different company/institution with different/weaker
> > processes). And vice versa.
> >
>
> ___
> QGIS-Developer mailing list
> QGIS-Developer@lists.osgeo.org
> List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
> Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer
___
QGIS-Developer mailing list
QGIS-Developer@lists.osgeo.org
List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer

Re: [QGIS-Developer] QGIS "fast" mode

2020-03-19 Thread René-Luc Dhont

Hi Tomas,

The way trust option works is not enough for big databases and big 
project with more than 100 layers. It is what Michaël and I experiment.


Then the problem with changes data is more about the layer extent. For 
example a natural observations layer is designed to accept data on a 
territory but t the start of the project the layer extent is null and 
will growing with data inserted. The trust option cannot be used at the 
project start unless the user can set the available extent, or can 
defined the trust option for each layer.


It will be great to set trust option at the layer level so QGIS can 
trust the layer information provided by the XML : QLR or QGS content
It will also be great to can set some technical metadata like a layer 
available extent as the geographic area for which a selected CRS is 
valid for use.


Regards,
René-Luc

Le 18/03/2020 à 22:01, Tomas Straupis a écrit :

2020-03-18, tr, 21:41 kimaidou rašė:

# only few requests are avoided as you pointed out so the performance improves 
"only" a bit

   In large databases those few requests take minutes and sometimes
even hours... For servers even 30 seconds are too much when you're
trying to add a new QGIS server process to existing say 10 while on
high load doing 500 requests per second.


# sometimes you have some layers with changing data, and there is actually no 
way to trust only a subset of layers inside the project

   1. If geometry types are changing by design, then checking geometry
type upon project loading is not enough. Then you need to ALWAYS
filter your results by requested geometry type. But only if it is the
case of varying geometry types. In my opinion, developer of the
layer/query should know beforehand if it is possible for geometry
types to variate and then create a view filtering on geometry type (or
it could be an option in QGIS ~"filter on geometry type").

   2. If database schemas are changing on-line then I would say
something is very wrong with the IT processes. Changes should start on
dev environment where data changes and QGIS project changes as well.
Then changes to db structure go to other environments up to production
in patches TOGETHER with updated QGIS project. System (in this case
QGIS) should never (in my experience) try to "fix" the problem because
it does not know all required information: maybe the project was
opened in incorrect environment, maybe it is a temporary problem,
maybe the real table is missing and you're accidentally trying to
query incorrect table which was never supposed to be queried -> strict
rules/control. Of course there could be a button "refresh" on a layer
to allow operator to re-query layer information upon manual request.


Do you think it would be interesting to have the trust option per layer and not 
only per project?

   Theoretically there could be very different data sources, but in my
opinion if organisation uses strict IT processes then all layers will
be strictly managed (you would rarely/never have direct access to the
database of a different company/institution with different/weaker
processes). And vice versa.



___
QGIS-Developer mailing list
QGIS-Developer@lists.osgeo.org
List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer

Re: [QGIS-Developer] QGIS "fast" mode

2020-03-18 Thread Tomas Straupis
2020-03-18, tr, 21:41 kimaidou rašė:
> # only few requests are avoided as you pointed out so the performance 
> improves "only" a bit

  In large databases those few requests take minutes and sometimes
even hours... For servers even 30 seconds are too much when you're
trying to add a new QGIS server process to existing say 10 while on
high load doing 500 requests per second.

> # sometimes you have some layers with changing data, and there is actually no 
> way to trust only a subset of layers inside the project

  1. If geometry types are changing by design, then checking geometry
type upon project loading is not enough. Then you need to ALWAYS
filter your results by requested geometry type. But only if it is the
case of varying geometry types. In my opinion, developer of the
layer/query should know beforehand if it is possible for geometry
types to variate and then create a view filtering on geometry type (or
it could be an option in QGIS ~"filter on geometry type").

  2. If database schemas are changing on-line then I would say
something is very wrong with the IT processes. Changes should start on
dev environment where data changes and QGIS project changes as well.
Then changes to db structure go to other environments up to production
in patches TOGETHER with updated QGIS project. System (in this case
QGIS) should never (in my experience) try to "fix" the problem because
it does not know all required information: maybe the project was
opened in incorrect environment, maybe it is a temporary problem,
maybe the real table is missing and you're accidentally trying to
query incorrect table which was never supposed to be queried -> strict
rules/control. Of course there could be a button "refresh" on a layer
to allow operator to re-query layer information upon manual request.

> Do you think it would be interesting to have the trust option per layer and 
> not only per project?

  Theoretically there could be very different data sources, but in my
opinion if organisation uses strict IT processes then all layers will
be strictly managed (you would rarely/never have direct access to the
database of a different company/institution with different/weaker
processes). And vice versa.

-- 
Tomas
___
QGIS-Developer mailing list
QGIS-Developer@lists.osgeo.org
List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer

Re: [QGIS-Developer] QGIS "fast" mode

2020-03-18 Thread kimaidou
Hi all

Thanks for raising this issue and for your proposal of patch. I agree with
you that in many cases, completely trusting the project for columns, srid,
and all other things needed by Qgis would improve performances a lot in
project loading, and for Qgis server getcapabilities requests.

I do not use often the trust option for two reasons

# only few requests are avoided as you pointed out so the performance
improves "only" a bit
# sometimes you have some layers with changing data, and there is actually
no way to trust only a subset of layers inside the project

Do you think it would be interesting to have the trust option per layer and
not only per project?

Regards
Michael

Le mardi 17 mars 2020, Julien Cabieces  a
écrit :
>
> Hi
>
>>   If I understand correctly, I can change my patch to skip geometry
>> type and srid check when they are saved in project already and when
>> option "trust project" is switched on?
>> (https://github.com/qgis/QGIS/pull/34738)
>
> Yes, I think it looks like the appropriate way of doing it
> ___
> QGIS-Developer mailing list
> QGIS-Developer@lists.osgeo.org
> List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
> Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer
___
QGIS-Developer mailing list
QGIS-Developer@lists.osgeo.org
List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer

Re: [QGIS-Developer] QGIS "fast" mode

2020-03-17 Thread Julien Cabieces

Hi

>   If I understand correctly, I can change my patch to skip geometry
> type and srid check when they are saved in project already and when
> option "trust project" is switched on?
> (https://github.com/qgis/QGIS/pull/34738)

Yes, I think it looks like the appropriate way of doing it
___
QGIS-Developer mailing list
QGIS-Developer@lists.osgeo.org
List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer

Re: [QGIS-Developer] QGIS "fast" mode

2020-03-16 Thread Tomas Straupis
2020-03-16, pr, 10:48 Julien Cabieces rašė:
> This function is only for PostGres (Tomas use an Oracle database if I
> remember correctly).

  Actually I use PostgreSQL more, but this particular performance
problem is with Oracle database.
  And if enough people agree with the principle, I think it could be
possible to remove unnecessary queries from all providers (in time).

> And from what I inderstand from the code, It only prevents from checking
> for unique colomn in a view or materialized view.
>
> However, it looks the good option to deal with this use case, I mean no
> check (geometry type, srid, key column...) as soon as this option is set.

  That would make my patch much more simpler ;-)
  If I understand correctly, I can change my patch to skip geometry
type and srid check when they are saved in project already and when
option "trust project" is switched on?
(https://github.com/qgis/QGIS/pull/34738)

-- 
Tomas
___
QGIS-Developer mailing list
QGIS-Developer@lists.osgeo.org
List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer

Re: [QGIS-Developer] QGIS "fast" mode

2020-03-16 Thread Julien Cabieces

Hi all,

This function is only for PostGres (Tomas use an Oracle database if I
remember correctly).

And from what I inderstand from the code, It only prevents from checking
for unique colomn in a view or materialized view.

However, it looks the good option to deal with this use case, I mean no
check (geometry type, srid, key column...) as soon as this option is set.

Regards,
Julien

> Hi Tomas.
> Check the project property option named "trust". This is exactly what it is
> supposed to do.
> I admit the wording is not nice, any better idea is welcome.
> Best regards
>
>
> Le sam. 14 mars 2020 à 12:55, Tomas Straupis  a
> écrit :
>
>> Hello
>>
>>   I would like to know opinion of QGIS developer community on the
>> question of "self-fixing queries" in the context of database layers.
>>
>>   When you add a new database layer, QGIS queries required data, asks
>> user for other information and then saves it in the project file.
>>   Now when you re-open the project, QGIS is re-querying a lot of data
>> which is already saved in the project: geometry type, srid, attributes
>> (table columns/types).
>>   The downsides are:
>>   * these "re-checking" queries take considerable time on large
>> databases: with millions of records from 1minute up to several hours,
>> database is tuned - WMS/WFS queries finish in ~100ms.
>>   * in QGIS server environment these re-checking queries are executed
>> each time when apache launches a new QGIS server process
>>   * QGIS might by itself (without human interaction) decide to  work
>> differently compared to what it was asked to do when the project was
>> created if it finds that "something has changed"
>>   * re-checking during project opening is not 100% correct, as if
>> changes are possible in principle, then data/schema could change
>> during the time QGIS (Desktop/Server) is open/running
>>   * in environments other than development (testing, qa, production),
>> database schema should never change without some clear procedure which
>> should also include updating QGIS project files and testing them as
>> required, therefore such re-checkings should never be required on
>> testing/production environments.
>>
>>   I do understand that the current situation/understanding is
>> different from mine, so in order to satisfy everybody, maybe it would
>> be feasible to add say an environment variable like QGIS_FAST=TRUE (or
>> QGIS_PRODUCTION=TRUE) and if this is set - re-checking queries could
>> be skipped? We've been running modified QGIS without re-checking in
>> production for months and are very happy with results. This should
>> also improve the benchmarks of QGIS Server as it is known that initial
>> opening of the project takes considerable amount of time.
>>
>>   What is your opinion?
>>
>>   Thank you
>>
>> --
>> Tomas
>> ___
>> QGIS-Developer mailing list
>> QGIS-Developer@lists.osgeo.org
>> List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
>> Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer
> ___
> QGIS-Developer mailing list
> QGIS-Developer@lists.osgeo.org
> List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
> Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer

___
QGIS-Developer mailing list
QGIS-Developer@lists.osgeo.org
List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer

Re: [QGIS-Developer] QGIS "fast" mode

2020-03-14 Thread Tomas Straupis
2020-03-14, št, 13:58 Régis Haubourg rašė:
> Check the project property option named "trust". This is exactly what it is 
> supposed to do.
> I admit the wording is not nice, any better idea is welcome.

  You mean this one: "Trust project when data source has no metadata"?
  I read the wording that it tries to check metadata and if that is
unsuccessful - trusts project data. My idea is to not even try
re-checking in metadata or anywhere else - always use data from
project and only execute queries for actual job to be done (also do
not try to filter on geometry type etc).

  Practical example: even simple things like re-checking table columns
in metadata (oracle data dictionary) could take 10-15 seconds when we
have ~100 layers. And on very high load these 10-15 seconds are
unwanted (and they will never find anything different than data in
project file).

> @Tomas: are you sure it is doing full table scans? Most database have
> some 'metadata' or cached metadata tables which could be used for that.

  For geometry checking in our scenario it is doing very badly because
data comes from a view which is optimised to be used with spatial
filter (spatial queries take ~100ms), but when we try to query
distinct geometry types (even from random 100 rows) usage schema is
different and it takes ages. Haven't checked but I'm pretty sure it
involves full scans because it takes hours and we do not want to tune
the view with hints for geometry type checking as that will decrease
performance of actually useful queries fetching features.
___
QGIS-Developer mailing list
QGIS-Developer@lists.osgeo.org
List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer

Re: [QGIS-Developer] QGIS "fast" mode

2020-03-14 Thread Richard Duivenvoorde
Hi,

Related: having projects with several WMS or WFS server layers also are
opening very slowly (even when NOT visible). It would be nice to have
'lazy' loading, so only load layer which are visible?

@Tomas: are you sure it is doing full table scans? Most database have
some 'metadata' or cached metadata tables which could be used for that.

Regards,

Richard Duivenvoorde

On 3/14/20 12:55 PM, Tomas Straupis wrote:
> Hello
> 
>   I would like to know opinion of QGIS developer community on the
> question of "self-fixing queries" in the context of database layers.
> 
>   When you add a new database layer, QGIS queries required data, asks
> user for other information and then saves it in the project file.
>   Now when you re-open the project, QGIS is re-querying a lot of data
> which is already saved in the project: geometry type, srid, attributes
> (table columns/types).
>   The downsides are:
>   * these "re-checking" queries take considerable time on large
> databases: with millions of records from 1minute up to several hours,
> database is tuned - WMS/WFS queries finish in ~100ms.
>   * in QGIS server environment these re-checking queries are executed
> each time when apache launches a new QGIS server process
>   * QGIS might by itself (without human interaction) decide to  work
> differently compared to what it was asked to do when the project was
> created if it finds that "something has changed"
>   * re-checking during project opening is not 100% correct, as if
> changes are possible in principle, then data/schema could change
> during the time QGIS (Desktop/Server) is open/running
>   * in environments other than development (testing, qa, production),
> database schema should never change without some clear procedure which
> should also include updating QGIS project files and testing them as
> required, therefore such re-checkings should never be required on
> testing/production environments.
> 
>   I do understand that the current situation/understanding is
> different from mine, so in order to satisfy everybody, maybe it would
> be feasible to add say an environment variable like QGIS_FAST=TRUE (or
> QGIS_PRODUCTION=TRUE) and if this is set - re-checking queries could
> be skipped? We've been running modified QGIS without re-checking in
> production for months and are very happy with results. This should
> also improve the benchmarks of QGIS Server as it is known that initial
> opening of the project takes considerable amount of time.
> 
>   What is your opinion?
> 
>   Thank you
> 

___
QGIS-Developer mailing list
QGIS-Developer@lists.osgeo.org
List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer

Re: [QGIS-Developer] QGIS "fast" mode

2020-03-14 Thread Régis Haubourg
Hi Tomas.
Check the project property option named "trust". This is exactly what it is
supposed to do.
I admit the wording is not nice, any better idea is welcome.
Best regards


Le sam. 14 mars 2020 à 12:55, Tomas Straupis  a
écrit :

> Hello
>
>   I would like to know opinion of QGIS developer community on the
> question of "self-fixing queries" in the context of database layers.
>
>   When you add a new database layer, QGIS queries required data, asks
> user for other information and then saves it in the project file.
>   Now when you re-open the project, QGIS is re-querying a lot of data
> which is already saved in the project: geometry type, srid, attributes
> (table columns/types).
>   The downsides are:
>   * these "re-checking" queries take considerable time on large
> databases: with millions of records from 1minute up to several hours,
> database is tuned - WMS/WFS queries finish in ~100ms.
>   * in QGIS server environment these re-checking queries are executed
> each time when apache launches a new QGIS server process
>   * QGIS might by itself (without human interaction) decide to  work
> differently compared to what it was asked to do when the project was
> created if it finds that "something has changed"
>   * re-checking during project opening is not 100% correct, as if
> changes are possible in principle, then data/schema could change
> during the time QGIS (Desktop/Server) is open/running
>   * in environments other than development (testing, qa, production),
> database schema should never change without some clear procedure which
> should also include updating QGIS project files and testing them as
> required, therefore such re-checkings should never be required on
> testing/production environments.
>
>   I do understand that the current situation/understanding is
> different from mine, so in order to satisfy everybody, maybe it would
> be feasible to add say an environment variable like QGIS_FAST=TRUE (or
> QGIS_PRODUCTION=TRUE) and if this is set - re-checking queries could
> be skipped? We've been running modified QGIS without re-checking in
> production for months and are very happy with results. This should
> also improve the benchmarks of QGIS Server as it is known that initial
> opening of the project takes considerable amount of time.
>
>   What is your opinion?
>
>   Thank you
>
> --
> Tomas
> ___
> QGIS-Developer mailing list
> QGIS-Developer@lists.osgeo.org
> List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
> Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer
___
QGIS-Developer mailing list
QGIS-Developer@lists.osgeo.org
List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer

[QGIS-Developer] QGIS "fast" mode

2020-03-14 Thread Tomas Straupis
Hello

  I would like to know opinion of QGIS developer community on the
question of "self-fixing queries" in the context of database layers.

  When you add a new database layer, QGIS queries required data, asks
user for other information and then saves it in the project file.
  Now when you re-open the project, QGIS is re-querying a lot of data
which is already saved in the project: geometry type, srid, attributes
(table columns/types).
  The downsides are:
  * these "re-checking" queries take considerable time on large
databases: with millions of records from 1minute up to several hours,
database is tuned - WMS/WFS queries finish in ~100ms.
  * in QGIS server environment these re-checking queries are executed
each time when apache launches a new QGIS server process
  * QGIS might by itself (without human interaction) decide to  work
differently compared to what it was asked to do when the project was
created if it finds that "something has changed"
  * re-checking during project opening is not 100% correct, as if
changes are possible in principle, then data/schema could change
during the time QGIS (Desktop/Server) is open/running
  * in environments other than development (testing, qa, production),
database schema should never change without some clear procedure which
should also include updating QGIS project files and testing them as
required, therefore such re-checkings should never be required on
testing/production environments.

  I do understand that the current situation/understanding is
different from mine, so in order to satisfy everybody, maybe it would
be feasible to add say an environment variable like QGIS_FAST=TRUE (or
QGIS_PRODUCTION=TRUE) and if this is set - re-checking queries could
be skipped? We've been running modified QGIS without re-checking in
production for months and are very happy with results. This should
also improve the benchmarks of QGIS Server as it is known that initial
opening of the project takes considerable amount of time.

  What is your opinion?

  Thank you

-- 
Tomas
___
QGIS-Developer mailing list
QGIS-Developer@lists.osgeo.org
List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer