Re: [QGIS-Developer] Discussing qgd files

[Régis Haubourg]

Hi,
just a quick followup about qgd file. Paul just merged that PR
https://github.com/qgis/QGIS/pull/6717 and qgd are now only created when
needed.
Thanks Andreas for raising the issue !


2018-03-07 18:45 GMT+01:00 Carlo A. Bertelli (Charta s.r.l.) <
carlo.berte...@gmail.com>:

> A disruptive "moreover approach" suggests some reflection about the file
>  database approach.
>
> I see several places where handling metadata in the database could provide
> a better but possibly conflicting solution.
> This happens for instance with styles stored on the database that may
> easily be overwritten by opening a modified project (if you work alone it's
> wonderful, but cooperating is a nightmare), but may happen also with any
> other "intelligent" provision that uses XML and/or persistent storage.
>
> I think this should not prevent a good solution about metadata. A map is
> not made of several layers overlaid with their styles only, it's something
> more and it's not completely satisfactory to reduce this to a project file
> that helps a single user to keep working on it or store the printing styles
> (sorry, I simplify too much, I know). Maybe QGD files are a starting point
> to design a better solution because we are embracing a three level storage
> system: files, light database (SQLite/Spatialite), DBMS (mainly
> PostgreSQL/PostGIS, but even others). This solution could lead us to a
> careful choice or to more flexibility (complexity?).
>
> Maybe this further flexibility is needed. I raise a very peculiar case: we
> work with historical maps and sometimes I georeference maps without being
> certain about my sources, sometimes the source itself is made of parts that
> ask two set of conflicting reference points for the same file. I'm forced
> to make a symlink on the filesystem to have the same file georeferenced in
> two ways. Why the different point sets cannot be stored on database? The
> reference points are points on earth and frequently I need to reuse them
> (what about adjacent tables? Maybe snapping on them could help), why not
> storing them in the database?
>
> QGIS Server is an amazing tool, but why using monolithic and completely
> proprietary XML file while several other applications could benefit from
> more generic metadata stored in the database? That is already done for
> styles which store SLD values besides the Qt ones, but it could be extended
> to other areas.
>
> It's reasonable to blame my "moreover approach", but take what you think
> QGIS could benefit of.
> c
>
> --
> --
> Carlo A. Bertelli
>Charta servizi e sistemi per il territorio e la storia ambientale srl
>   Dipendenze del palazzo Doria,
>   vc. alla Chiesa della Maddalena 9/2 16124  Genova (Italy)
>   tel./fax +39(0)10 2475439  +39 0108566195  mobile:+39 393 1590711
>e-mail: berte...@chartasrl.eu  http://www.chartasrl.eu
> --
>
>
>
>
> ___
> QGIS-Developer mailing list
> QGIS-Developer@lists.osgeo.org
> List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
> Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer
>
___
QGIS-Developer mailing list
QGIS-Developer@lists.osgeo.org
List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer

Re: [QGIS-Developer] Discussing qgd files

[Carlo A. Bertelli (Charta s.r.l.)]

A disruptive "moreover approach" suggests some reflection about the file
 database approach.

I see several places where handling metadata in the database could provide
a better but possibly conflicting solution.
This happens for instance with styles stored on the database that may
easily be overwritten by opening a modified project (if you work alone it's
wonderful, but cooperating is a nightmare), but may happen also with any
other "intelligent" provision that uses XML and/or persistent storage.

I think this should not prevent a good solution about metadata. A map is
not made of several layers overlaid with their styles only, it's something
more and it's not completely satisfactory to reduce this to a project file
that helps a single user to keep working on it or store the printing styles
(sorry, I simplify too much, I know). Maybe QGD files are a starting point
to design a better solution because we are embracing a three level storage
system: files, light database (SQLite/Spatialite), DBMS (mainly
PostgreSQL/PostGIS, but even others). This solution could lead us to a
careful choice or to more flexibility (complexity?).

Maybe this further flexibility is needed. I raise a very peculiar case: we
work with historical maps and sometimes I georeference maps without being
certain about my sources, sometimes the source itself is made of parts that
ask two set of conflicting reference points for the same file. I'm forced
to make a symlink on the filesystem to have the same file georeferenced in
two ways. Why the different point sets cannot be stored on database? The
reference points are points on earth and frequently I need to reuse them
(what about adjacent tables? Maybe snapping on them could help), why not
storing them in the database?

QGIS Server is an amazing tool, but why using monolithic and completely
proprietary XML file while several other applications could benefit from
more generic metadata stored in the database? That is already done for
styles which store SLD values besides the Qt ones, but it could be extended
to other areas.

It's reasonable to blame my "moreover approach", but take what you think
QGIS could benefit of.
c

-- 
--
Carlo A. Bertelli
   Charta servizi e sistemi per il territorio e la storia ambientale srl
  Dipendenze del palazzo Doria,
  vc. alla Chiesa della Maddalena 9/2 16124  Genova (Italy)
  tel./fax +39(0)10 2475439  +39 0108566195  mobile:+39 393 1590711
   e-mail: berte...@chartasrl.eu  http://www.chartasrl.eu
--
___
QGIS-Developer mailing list
QGIS-Developer@lists.osgeo.org
List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer

Re: [QGIS-Developer] Discussing qgd files

[Régis Haubourg]

2018-03-07 17:10 GMT+01:00 Jürgen E. Fischer :

> Hi Andreas,
>
> On Mon, 05. Mar 2018 at 16:03:08 +0100, Andreas Neumann wrote:
> > In QGIS 3 there is always a qgd file created alongside with QGIS
> > projects. I heard that they are used as auxilliary file to store project
> > specific labels - right?
>
> Sorry for not checking myself.  Are the qgd only written on project save?
>
> Just asking because there's check to avoid overwritting a project that
> meanwhile was changed by someone.  Are qgd implicitly covered by that check
> too?
>

Uhuh, excellent question! I'll check that with Paul. As this is a sqlite
database, I think that we also have to check the file locking policy (
https://www.sqlite.org/lockingv3.html)

 Régis


>
> Jürgen
>
> --
> Jürgen E. Fischer   norBIT GmbH Tel.
> +49-4931-918175-31
> Dipl.-Inf. (FH) Rheinstraße 13  Fax.
> +49-4931-918175-50
> Software Engineer   D-26506 Norden
> http://www.norbit.de
> QGIS release manager (PSC)  GermanyIRC: jef on FreeNode
>
> ___
> QGIS-Developer mailing list
> QGIS-Developer@lists.osgeo.org
> List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
> Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer
>
___
QGIS-Developer mailing list
QGIS-Developer@lists.osgeo.org
List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer

Re: [QGIS-Developer] Discussing qgd files

[Jürgen E . Fischer]

Hi Andreas,

On Mon, 05. Mar 2018 at 16:03:08 +0100, Andreas Neumann wrote:
> In QGIS 3 there is always a qgd file created alongside with QGIS
> projects. I heard that they are used as auxilliary file to store project
> specific labels - right? 

Sorry for not checking myself.  Are the qgd only written on project save?

Just asking because there's check to avoid overwritting a project that
meanwhile was changed by someone.  Are qgd implicitly covered by that check
too?


Jürgen

-- 
Jürgen E. Fischer   norBIT GmbH Tel. +49-4931-918175-31
Dipl.-Inf. (FH) Rheinstraße 13  Fax. +49-4931-918175-50
Software Engineer   D-26506 Norden http://www.norbit.de
QGIS release manager (PSC)  GermanyIRC: jef on FreeNode


signature.asc
Description: PGP signature
___
QGIS-Developer mailing list
QGIS-Developer@lists.osgeo.org
List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer

Re: [QGIS-Developer] Discussing qgd files

[Nathan Woodrow]

Haha. I was going to start prototyping this idea next week.  I will start a
QEP at some stage this week to collect ideas.

- Nathan

On Tue, Mar 6, 2018 at 6:31 PM, Paolo Cavallini 
wrote:

> Il 06/03/2018 09:29, Alessandro Pasotti ha scritto:
> > Maybe out of scope but I'd like to see an abstraction on top of
> > QgsProject to be able to load the project from different sources, for
> > instance a REST API, or a gpkg.
>
> right - or a PostgeSQL, as well.
> all the best.
>
> --
> Paolo Cavallini - www.faunalia.eu
> QGIS & PostGIS courses: http://www.faunalia.eu/training.html
> https://www.google.com/trends/explore?date=all=IT=qgis,arcgis
> ___
> QGIS-Developer mailing list
> QGIS-Developer@lists.osgeo.org
> List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
> Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer
>
___
QGIS-Developer mailing list
QGIS-Developer@lists.osgeo.org
List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer

Re: [QGIS-Developer] Discussing qgd files

[Paolo Cavallini]

Il 06/03/2018 09:29, Alessandro Pasotti ha scritto:
> Maybe out of scope but I'd like to see an abstraction on top of
> QgsProject to be able to load the project from different sources, for
> instance a REST API, or a gpkg.

right - or a PostgeSQL, as well.
all the best.

-- 
Paolo Cavallini - www.faunalia.eu
QGIS & PostGIS courses: http://www.faunalia.eu/training.html
https://www.google.com/trends/explore?date=all=IT=qgis,arcgis
___
QGIS-Developer mailing list
QGIS-Developer@lists.osgeo.org
List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer

Re: [QGIS-Developer] Discussing qgd files

[Alessandro Pasotti]

Maybe out of scope but I'd like to see an abstraction on top of QgsProject
to be able to load the project from different sources, for instance a REST
API, or a gpkg.


On Mon, Mar 5, 2018 at 9:20 PM, Régis Haubourg 
wrote:

> Hi Andreas,
> Good point indeed.
> In fact, the first implementations where created the database only when
> needed and were also spatialite. It was slowing down a lot the workflow,
> but now you raise the point, I think this was more linked to the spatialite
> db creation time. I'll ask Paul if creating the sqlite .qgd file can only
> be done when first required.
>
> Another point, we kept the qgz format optional, but I think that in the
> end, we probably should discuss if this is not too complicated. We could
> switch to qgz as default format, so that users see only one file, don't get
> bothered with qgd or any associated file anymore.
> Thoughts?
> Régis
>
>
> 2018-03-05 16:03 GMT+01:00 Andreas Neumann :
>
>> Hi,
>>
>> In QGIS 3 there is always a qgd file created alongside with QGIS
>> projects. I heard that they are used as auxilliary file to store project
>> specific labels - right?
>>
>> So far fine - but why are these files always created, even if the file
>> isn't used? Couldn't we create them on demand when they are really needed?
>> Currently, they are always created. I almost never use this feature - yet
>> these files are always created.
>>
>> I would assume that these extra files are a source of confusion to our
>> users - and if the project is renamed - most likely they don't know that
>> this file belongs to the same project and should be renamed as well -
>> almost as bad as with shapefiles ...
>>
>> Thanks for your thoughts on this matter,
>>
>> Andreas
>>
>>
>>
>> ___
>> QGIS-Developer mailing list
>> QGIS-Developer@lists.osgeo.org
>> List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
>> Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer
>>
>
>
> ___
> QGIS-Developer mailing list
> QGIS-Developer@lists.osgeo.org
> List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
> Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer
>



-- 
Alessandro Pasotti
w3:   www.itopen.it
___
QGIS-Developer mailing list
QGIS-Developer@lists.osgeo.org
List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer

Re: [QGIS-Developer] Discussing qgd files

[Régis Haubourg]

Cool, let's do that !
 (Paul is on vacations right now)

Regards
Régis

2018-03-06 0:36 GMT+01:00 Nyall Dawson :

> On 6 March 2018 at 06:59, Andreas Neumann  wrote:
> > Hi Régis,
> >
> > Thanks for asking Paul to see if the .qgd files can be created only "on
> > demand" when necessary.
>
> +1
>
>
> > If there aren't any performance problems with .qgz in comparison with
> .qgs
> > files, then I would be fine switching to the zipped version by default.
>
> Big +1
>
> Nyall
>
___
QGIS-Developer mailing list
QGIS-Developer@lists.osgeo.org
List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer

Re: [QGIS-Developer] Discussing qgd files

[Andreas Neumann]

Hi Régis,

Thanks for asking Paul to see if the .qgd files can be created only "on 
demand" when necessary.


If there aren't any performance problems with .qgz in comparison with 
.qgs files, then I would be fine switching to the zipped version by default.


Andreas



On 05.03.2018 21:20, Régis Haubourg wrote:

Hi Andreas,
Good point indeed.
In fact, the first implementations where created the database only 
when needed and were also spatialite. It was slowing down a lot the 
workflow, but now you raise the point, I think this was more linked to 
the spatialite db creation time. I'll ask Paul if creating the sqlite 
.qgd file can only be done when first required.


Another point, we kept the qgz format optional, but I think that in 
the end, we probably should discuss if this is not too complicated. We 
could switch to qgz as default format, so that users see only one 
file, don't get bothered with qgd or any associated file anymore.

Thoughts?
Régis


2018-03-05 16:03 GMT+01:00 Andreas Neumann >:


Hi,

In QGIS 3 there is always a qgd file created alongside with QGIS
projects. I heard that they are used as auxilliary file to store
project specific labels - right?

So far fine - but why are these files always created, even if the
file isn't used? Couldn't we create them on demand when they are
really needed? Currently, they are always created. I almost never
use this feature - yet these files are always created.

I would assume that these extra files are a source of confusion to
our users - and if the project is renamed - most likely they don't
know that this file belongs to the same project and should be
renamed as well - almost as bad as with shapefiles ...

Thanks for your thoughts on this matter,

Andreas



___
QGIS-Developer mailing list
QGIS-Developer@lists.osgeo.org 
List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer

Unsubscribe:
https://lists.osgeo.org/mailman/listinfo/qgis-developer





___
QGIS-Developer mailing list
QGIS-Developer@lists.osgeo.org
List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer

Re: [QGIS-Developer] Discussing qgd files

[Régis Haubourg]

Hi Andreas,
Good point indeed.
In fact, the first implementations where created the database only when
needed and were also spatialite. It was slowing down a lot the workflow,
but now you raise the point, I think this was more linked to the spatialite
db creation time. I'll ask Paul if creating the sqlite .qgd file can only
be done when first required.

Another point, we kept the qgz format optional, but I think that in the
end, we probably should discuss if this is not too complicated. We could
switch to qgz as default format, so that users see only one file, don't get
bothered with qgd or any associated file anymore.
Thoughts?
Régis


2018-03-05 16:03 GMT+01:00 Andreas Neumann :

> Hi,
>
> In QGIS 3 there is always a qgd file created alongside with QGIS projects.
> I heard that they are used as auxilliary file to store project specific
> labels - right?
>
> So far fine - but why are these files always created, even if the file
> isn't used? Couldn't we create them on demand when they are really needed?
> Currently, they are always created. I almost never use this feature - yet
> these files are always created.
>
> I would assume that these extra files are a source of confusion to our
> users - and if the project is renamed - most likely they don't know that
> this file belongs to the same project and should be renamed as well -
> almost as bad as with shapefiles ...
>
> Thanks for your thoughts on this matter,
>
> Andreas
>
>
>
> ___
> QGIS-Developer mailing list
> QGIS-Developer@lists.osgeo.org
> List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
> Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer
>
___
QGIS-Developer mailing list
QGIS-Developer@lists.osgeo.org
List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer