Re: [QGIS-Developer] QGIS server deployment

[Patrick Valsecchi]

On the camptocamp side, we use 1) within a docker image:
https://cloud.docker.com/u/camptocamp/repository/docker/camptocamp/qgis-server

What I strongly recommend is to add a FcgidMaxRequestsPerProcess with a
value of 1000 (work around some potential  memory leaks in QGIS). I'd say
to set the number of processes to the number of CPU cores.

On Mon, Feb 25, 2019 at 11:29 AM René-Luc Dhont  wrote:

> Hi Walter,
>
> We, at 3Liz, use the 3) with supervisord and the number of process is
> based on the number of preocessors.
>
> But we also used for QGIS3 a 4th way, a python server based on tronado:
> https://github.com/3liz/py-qgis-server
>
> Regards,
>
> René-Luc
> https://github.com/rldhont
>
> Le 25/02/2019 à 11:08, Walter Lorenzetti a écrit :
>
> Hi all,
>
> I'm not a sysadmin but for our customers I'd like find the better way for
> deploy QGIS-server 3.
>
> I try at least 3 ways:
>
> 1) Apache2 + libapache2-mod-fcgid
>
> 2) Nginx + fcgiwrap
>
> 3) Nginx + QGIS-Server working by socket/service (systemd)
>
> By delveloper side, what do you think is the best?
>
> I'd like to much 3) solution, but I found some problems, in particular,
> how many sockects I've to create watching at my server? (Number of
> processors )
>
> For 1) and 2) have you experiences on performance and tuning?
>
> Thanks in advance and thanks for work!
>
> W
> --
>
> Walter Lorenzetti phD
> email: lorenze...@gis3w.it
> skype: aiki74
> twitter:w_lorenzetti 
> g+:aiki74 
> Tel/Cell: (+39) 347-6597931
> Viale Verdi 24 - 51016 Montecatini Terme (PT)
> Nuovi corsi QGIS e GFOSS
> 
>
> ___
> QGIS-Developer mailing listqgis-develo...@lists.osgeo.org
> List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
> Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer
>
>
> ___
> QGIS-Developer mailing list
> QGIS-Developer@lists.osgeo.org
> List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
> Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer
___
QGIS-Developer mailing list
QGIS-Developer@lists.osgeo.org
List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer

Re: [QGIS-Developer] Sentry support for QGIS crashes / minidumps

[Patrick Valsecchi]

Hi,

Would be very useful for QGIS server. Good idea!

CU

On Thu, May 3, 2018 at 9:10 AM, Nathan Woodrow  wrote:

> Hey Tim,
>
> I added a crash handler in 3.0 for Windows at least and it lead to a few
> good fixes.  I will checkout the stuff you posted to see if we can
> intergrate it into what I already have there.
>
> Nathan
>
> On Thu, 3 May 2018, 4:43 pm Tim Sutton,  wrote:
>
>> Hi All
>>
>>
>> For many years we have used sentry (http://sentry.io) in our python
>> projects to systematically collect, review and prioritise issues raised in
>> our python projects. Sentry.io is open source (plus they offer a
>> commercially hosted service). At Kartoza we run our own instance under
>> docker / rancher.
>>
>> In the last Nødebo hackfest, we discussed the possibility of using
>> something like Sentry for QGIS so that we could better understand where our
>> users encounter crashes and proactively fix them. I subsequently went and
>> researched whether Sentry.io has support for mini dumps / c++ crash
>> handling and at the time it did not have so I parked thinking about it
>> there.
>>
>> Yesterday I got a sentry updates newsletter and noticed that they have
>> now added C++  support (currently in beta):
>>
>> https://blog.sentry.io/2018/04/17/introducing-minidump-support
>>
>> I know I am not alone when I run training courses offering this advice:
>> “Save your project regularly, QGIS will probably crash at some point”. I
>> really hate saying that but we have never had a systematic way of seeing
>> where QGIS is crashing for our users and fixing this crash points. As we
>> lead up to 3.4 LTR later this year, having good crash metrics and fixing
>> the most common crash points will allow us to have a release that we can be
>> confident works well for our users with out crashing during trivial
>> operations.
>>
>> Perhaps one of our fine developers might like to pitch this as a QGIS
>> Grant proposal (submission period closes 13 May 2018)?
>>
>> http://blog.qgis.org/2018/04/15/qgis-grants-3-call-for-
>> grant-proposals-2018/
>>
>> I’d be happy to help set up our own sentry.io instance on QGIS
>> infrastructure. We could also use that instance to receive tracebacks from
>> the python bits of our code….
>>
>> Regards
>>
>> Tim
>>
>> —
>>
>>
>>
>>
>>
>>
>>
>> *Tim Sutton*
>>
>> *Co-founder:* Kartoza
>> *Ex Project chair:* QGIS.org
>>
>> Visit http://kartoza.com to find out about open source:
>>
>> Desktop GIS programming services
>> Geospatial web development
>> GIS Training
>> Consulting Services
>>
>> *Skype*: timlinux
>> *IRC:* timlinux on #qgis at freenode.net
>>
>> ___
>> QGIS-Developer mailing list
>> QGIS-Developer@lists.osgeo.org
>> List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
>> Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer
>
>
> ___
> QGIS-Developer mailing list
> QGIS-Developer@lists.osgeo.org
> List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
> Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer
>
___
QGIS-Developer mailing list
QGIS-Developer@lists.osgeo.org
List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer

Re: [QGIS-Developer] What are "Key/value" and "List" widgets in attributes form?

[Patrick Valsecchi]

Hi

Good analysis and sorry for my slow response.

The widgets are not directly tied to PG. They will work as long as the
providers support QVariant::List or QVariant::Map datatypes.

The spatialite provider supports QVariant::List, encoded as JSON:
https://github.com/qgis/QGIS/commit/6260f9dea5293ec082ffe7cfae3211e914021288

CU


On Thu, Mar 29, 2018 at 5:58 PM, DelazJ  wrote:

> Hi,
>
> Thanks Nyall. Indeed, these are PG provider tools. I dug a bit and found
> out that list is designed for array field type while key/value refers to
> hstore field type.
> Doc is updated (might be light but better than nothing imho).
>
> Harrissou
>
> 2018-03-24 1:18 GMT+01:00 Nyall Dawson :
>
>> On 23 March 2018 at 00:07, DelazJ  wrote:
>> > Hi all,
>> >
>> > In the vector layer properties dialog, Attributes form tab, I fail to
>> > understand/find what enables those two widgets, hence what they are
>> supposed
>> > to do and how they could work.
>> >
>> > If someone has some information, thanks for sharing.
>> > Documenting this section is blocked by these information (see
>> > https://github.com/qgis/QGIS-Documentation/pull/2437/files)
>>
>> I suspect these are only available for list/map field types - which as
>> far as I'm aware are only available for the postgres provider.
>>
>> Nyall
>>
>> >
>> >
>> > Thanks,
>> > Harrissou
>> >
>> > ___
>> > QGIS-Developer mailing list
>> > QGIS-Developer@lists.osgeo.org
>> > List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
>> > Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer
>>
>
>
> ___
> QGIS-Developer mailing list
> QGIS-Developer@lists.osgeo.org
> List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
> Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer
>
___
QGIS-Developer mailing list
QGIS-Developer@lists.osgeo.org
List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer

Re: [QGIS-Developer] QGIS Server request extra parameters as variables for expressions

[Patrick Valsecchi]

You could do that using the filter parameter with something like that:
FILTER=layer_name:"floor" = 3

On Thu, Nov 23, 2017 at 2:38 PM, Matthias Kuhn  wrote:

> Like WMS dimension?
>
> http://mapserver.org/ogc/wms_dimension.html
>
> Matthias
>
>
> On 11/23/17 2:11 PM, Arnaud Morvan wrote:
> > Hello,
> >
> > I would like GetMap and GetFeature request extra parameters to be
> > accessible as variables in server.
> >
> > So we could define, in QGIS project, a variable "floor", used to
> > filter layers.
> >
> > On desktop side, we could have a desktop plugin named FloorSlider to
> > change the "floor" value.
> >
> > And on the server side, the floor value could be passed as extra
> > parameter.
> >
> > So it would be easy to handle multi-floor layers in a project file.
> >
> >
> > Arnaud Morvan
> > Ingénieur logiciel
> > Tél: +33 (0)4 58 48 20 32
> >
> > Camptocamp France SAS
> > 18 rue du Lac Saint André
> > Savoie Technolac - Bâtiment Le Dauphin
> > F-73370 Le Bourget du Lac
> > http://www.camptocamp.com
> >
> > Le 22/11/2017 à 18:05, René-Luc Dhont a écrit :
> >> Hi Arnaud,
> >>
> >> Is it like updated a QGIS project variables ?
> >>
> >> Regards,
> >>
> >> René-Luc
> >>
> >>
> >> Le 22/11/2017 à 15:31, Arnaud Morvan a écrit :
> >>> Hello,
> >>>
> >>> With mapserver, request extra parameters are accessible as template
> >>> variables in mapfile.
> >>>
> >>> I would like to implement the same possibility in QGIS Server, extra
> >>> parameters may be accessible in project as expression variables.
> >>>
> >>> For example : I would like to pass in a GetMap request an extra
> >>> parameter FLOOR.
> >>>
> >>> This parameter could be handled by QGIS Server as a variable value,
> >>> so this could be used in the project to filter some layers using an
> >>> expression.
> >>>
> >>> Do you think this could be acceptable directly in QGIS Server, or
> >>> may I have to wrote a plugin.
> >>>
> >>> Maybe this type of plugin already exists ?
> >>>
> >>> Note that I'm not familiar with server part.
> >>>
> >>> Best regards
> >>>
> >>
> >> ___
> >> QGIS-Developer mailing list
> >> QGIS-Developer@lists.osgeo.org
> >> List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
> >> Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer
> >
> > ___
> > QGIS-Developer mailing list
> > QGIS-Developer@lists.osgeo.org
> > List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
> > Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer
> >
>
> ___
> QGIS-Developer mailing list
> QGIS-Developer@lists.osgeo.org
> List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
> Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer
>
___
QGIS-Developer mailing list
QGIS-Developer@lists.osgeo.org
List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer

Re: [QGIS-Developer] Travis timeout

[Patrick Valsecchi]

OK, one step further. But I must be missing something because the tests
in tests/src/python/test_qgsserver_wms.py are just failing. From the looks
of the diff, it must be a font that is different. See the image in
attachment.
Any idea how to fix that?

On Wed, Nov 8, 2017 at 9:26 AM, Alessandro Pasotti <apaso...@gmail.com>
wrote:

> Hi Patrick,
>
> to run python tests locally you can either use ctest from the build dir or
> create a wrapper script that set paths and env vars before launching python
> and the script.
>
> Here is mine (you probably need to adapt it): http://dpaste.com/1C3FMZM
>
> Recently Matthias also explained on this list how to run python tests
> inside QtCreator.
>
> Cheers
>
>
> On Wed, Nov 8, 2017 at 9:22 AM, Patrick Valsecchi <
> patrick.valsec...@camptocamp.com> wrote:
>
>> Hi,
>>
>> I'm trying to work on a feature in the master branch, but Travis constantly
>> times out on my builds
>> <https://travis-ci.org/pvalsecc/QGIS/jobs/297999069>.
>>
>> Did I miss something?
>>
>> Plus, I'm trying to run UTs on my machine, but the doc must be outdated.
>> How do I run the tests in tests/src/python/test_qgsserver_wms.py locally?
>>
>> Thanks for your help
>>
>> ___
>> QGIS-Developer mailing list
>> QGIS-Developer@lists.osgeo.org
>> List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
>> Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer
>>
>
>
>
> --
> Alessandro Pasotti
> w3:   www.itopen.it
>
___
QGIS-Developer mailing list
QGIS-Developer@lists.osgeo.org
List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer

[QGIS-Developer] Travis timeout

[Patrick Valsecchi]

Hi,

I'm trying to work on a feature in the master branch, but Travis constantly
times out on my builds .

Did I miss something?

Plus, I'm trying to run UTs on my machine, but the doc must be outdated.
How do I run the tests in tests/src/python/test_qgsserver_wms.py locally?

Thanks for your help
___
QGIS-Developer mailing list
QGIS-Developer@lists.osgeo.org
List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer

Re: [Qgis-developer] Visualisation of relations

[Patrick Valsecchi]

OK, I've created a PR for auto-detecting N-1 relations and pick the
RelationReference widget for the foreign key fields.
https://github.com/qgis/QGIS/pull/3699

For the deep relations UX problem, I really don't know. Do we have UX
experts among the QGIS devs?

Thanks.

On Mon, Oct 31, 2016 at 1:51 PM, Matthias Kuhn <matth...@opengis.ch> wrote:

> On 10/31/2016 01:26 PM, Patrick Valsecchi wrote:
> > Matthias,
> >
> > Hummm... I don't know about the tabs. Initially I was thinking it would
> > be a good idea, then I tried to imagine how it would look like for
> > linked layers within linked layers. We would have two choices:
> >
> >   * Have QTabWidget within QTabWidget: tabs within tabs is a UX nightmare
> >   * Switch to the current way of representing relations for
> > sub-relations: that would lead to user confusion because we have two
> > ways of representing relations.
> >
> > What do you think? To me, tabs is not a good idea.
>
> I think the (current) groupbox within groupbox approach suffers from the
> same UX defficiencies. At least I get confused regularly.
>
> Maybe
>
> * toplevel tabs
> * first sublevel groupbox
> * second sublevel ...
>   groupbox again?
>   skip?
>   button to open in separate window?
>
> Matthias
>
___
Qgis-developer mailing list
Qgis-developer@lists.osgeo.org
List info: http://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: http://lists.osgeo.org/mailman/listinfo/qgis-developer

Re: [Qgis-developer] Visualisation of relations

[Patrick Valsecchi]

Matthias,

Hummm... I don't know about the tabs. Initially I was thinking it would be
a good idea, then I tried to imagine how it would look like for linked
layers within linked layers. We would have two choices:

   - Have QTabWidget within QTabWidget: tabs within tabs is a UX nightmare
   - Switch to the current way of representing relations for sub-relations:
   that would lead to user confusion because we have two ways of representing
   relations.

What do you think? To me, tabs is not a good idea.

Thanks

On Mon, Oct 31, 2016 at 10:06 AM, Matthias Kuhn <matth...@opengis.ch> wrote:

> Hi Patrick,
>
> for the side that contains the foreign key, there is the "relation
> reference" widget which has a configuration option "show embedded form"
> (I haven't used it in years and a quick check on it didn't work, so
> maybe it needs fixing).
>
> Agreed, collapsed by default makes sense for huge models. For small ones
> it's a bit strange though. Another approach would be to put them on
> separate tabs by default, I think that could look quite nice also. What
> do you think?
>
> Regards
> Matthias
>
> On 10/31/2016 09:54 AM, Patrick Valsecchi wrote:
> > Hi Matthias,
> >
> > My screenshot shows only 1:N links. N:1 links are not supported. By N:1,
> > I mean the relations from the side that contains the foreign key. If I
> > open the form for the "appart" layer, it doesn't show the linked
> "maison".
> >
> > Yes, the collapsed state is remembered, but the default should be
> > collapsed. If the default is to open everything, the GUI is going to
> > explode if we have hundreds of linked tables.
> >
> > Thanks and CU.
> >
> > On Sun, Oct 16, 2016 at 8:57 AM, Matthias Kuhn <matth...@opengis.ch
> > <mailto:matth...@opengis.ch>> wrote:
> >
> > Hi Patrick,
> >
> > Making forms and relations more usable is always welcome.
> >
> > What exactly are the problems you have with the current solution?
> > I see that you mentioned the lack of the N:1 side which is available
> > including add feature, add link, remove feature, remove link pretty
> much
> > the way you describe it. On [1] there is some explanation I wrote on
> the
> > first implementation.
> >
> > I think you should have found that since it's shown on the screenshot
> > which is attached to your mail.
> >
> > Also lazy loading (load when first visible) was introduced once for
> > performance reasons and the collapsed state of the group boxes
> > containing the QgsRelationEditor widget is remembered.
> >
> > So I think that functionality-wise, most of it should be there
> already.
> > With a lot of air left for improvement on the usability side.
> >
> > Cheers
> > Matthias
> >
> > [1] http://blog.vitu.ch/10112013-1201/qgis-relations
> > <http://blog.vitu.ch/10112013-1201/qgis-relations>
> >
> >
> > On 10/07/2016 09:27 AM, Patrick Valsecchi wrote:
> > > Hi,
> > >
> > > I'm tasked with making QGIS a bit more usable with complex database
> > > schemas having a lot of relations (up to hundreds of linked
> tables). The
> > > INSPIRE people were a bit too inspired when creating their data
> schemas
> > > and now we have to try to make QGIS able to cope with that.
> > >
> > > My concerns with the current situation (as of QGIS master) are:
> > >
> > >   * We can specify the relations between the layers at the project
> > level
> > > (it's now easier with the auto-discover feature for PostGIS and
> > > Spatialite). But those are only showing in the
> QgsAttributeForm for
> > > the 1-N side (the side that doesn't have the foreign key). Why
> not
> > > on the N-1 side?
> > >   * For showing the N-1 side in the QgsAttributeForm, one can
> define a
> > > Join in the layer's properties, but I don't see the point of
> having
> > > to define it here as well when we have already the relations
> info at
> > > the project level. I see a use for special joins, but for
> relations,
> > > I don't see why we have to define it twice. And the way it's
> > > displayed is not allowing to create joins or edit the joined
> fields.
> > >   * I let you imagine the look of the feature attribute form when
> > there
> > > are hundreds of directly an

Re: [Qgis-developer] Visualisation of relations

[Patrick Valsecchi]

Hi Matthias,

My screenshot shows only 1:N links. N:1 links are not supported. By N:1, I
mean the relations from the side that contains the foreign key. If I open
the form for the "appart" layer, it doesn't show the linked "maison".

Yes, the collapsed state is remembered, but the default should be
collapsed. If the default is to open everything, the GUI is going to
explode if we have hundreds of linked tables.

Thanks and CU.

On Sun, Oct 16, 2016 at 8:57 AM, Matthias Kuhn <matth...@opengis.ch> wrote:

> Hi Patrick,
>
> Making forms and relations more usable is always welcome.
>
> What exactly are the problems you have with the current solution?
> I see that you mentioned the lack of the N:1 side which is available
> including add feature, add link, remove feature, remove link pretty much
> the way you describe it. On [1] there is some explanation I wrote on the
> first implementation.
>
> I think you should have found that since it's shown on the screenshot
> which is attached to your mail.
>
> Also lazy loading (load when first visible) was introduced once for
> performance reasons and the collapsed state of the group boxes
> containing the QgsRelationEditor widget is remembered.
>
> So I think that functionality-wise, most of it should be there already.
> With a lot of air left for improvement on the usability side.
>
> Cheers
> Matthias
>
> [1] http://blog.vitu.ch/10112013-1201/qgis-relations
>
>
> On 10/07/2016 09:27 AM, Patrick Valsecchi wrote:
> > Hi,
> >
> > I'm tasked with making QGIS a bit more usable with complex database
> > schemas having a lot of relations (up to hundreds of linked tables). The
> > INSPIRE people were a bit too inspired when creating their data schemas
> > and now we have to try to make QGIS able to cope with that.
> >
> > My concerns with the current situation (as of QGIS master) are:
> >
> >   * We can specify the relations between the layers at the project level
> > (it's now easier with the auto-discover feature for PostGIS and
> > Spatialite). But those are only showing in the QgsAttributeForm for
> > the 1-N side (the side that doesn't have the foreign key). Why not
> > on the N-1 side?
> >   * For showing the N-1 side in the QgsAttributeForm, one can define a
> > Join in the layer's properties, but I don't see the point of having
> > to define it here as well when we have already the relations info at
> > the project level. I see a use for special joins, but for relations,
> > I don't see why we have to define it twice. And the way it's
> > displayed is not allowing to create joins or edit the joined fields.
> >   * I let you imagine the look of the feature attribute form when there
> > are hundreds of directly and indirectly linked tabled. This is just
> > not usable if we display all of them directly like that. Just look
> > at the attached screen shot that shows what happens by default with
> > only 3 tables. It's already a mess.
> >
> > Now, what I propose is:
> >
> >   * Not expand the relation widget (QgsCollapsibleGroupBox) by default
> > and build it's content only when it is expanded the first time
> > (think of what would happen when you have loops in the schema).
> >   * Show N-1 relations as well, in a collapsed by default
> > QgsCollapsibleGroupBox, including a way to add a new linked entry,
> > remove the link (put the FK to NULL) and delete it.
> >   * Add a button to open a related feature in a new window.
> >
> > What do you guys think?
> >
> > Thanks.
> >
> >
> >
> > ___
> > Qgis-developer mailing list
> > Qgis-developer@lists.osgeo.org
> > List info: http://lists.osgeo.org/mailman/listinfo/qgis-developer
> > Unsubscribe: http://lists.osgeo.org/mailman/listinfo/qgis-developer
> >
> ___
> Qgis-developer mailing list
> Qgis-developer@lists.osgeo.org
> List info: http://lists.osgeo.org/mailman/listinfo/qgis-developer
> Unsubscribe: http://lists.osgeo.org/mailman/listinfo/qgis-developer
>
___
Qgis-developer mailing list
Qgis-developer@lists.osgeo.org
List info: http://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: http://lists.osgeo.org/mailman/listinfo/qgis-developer

Re: [Qgis-developer] Visualisation of relations

[Patrick Valsecchi]

No answer. I think I'll move forward with my proposal.

Thanks.

On Fri, Oct 7, 2016 at 9:27 AM, Patrick Valsecchi <
patrick.valsec...@camptocamp.com> wrote:

> Hi,
>
> I'm tasked with making QGIS a bit more usable with complex database
> schemas having a lot of relations (up to hundreds of linked tables). The
> INSPIRE people were a bit too inspired when creating their data schemas and
> now we have to try to make QGIS able to cope with that.
>
> My concerns with the current situation (as of QGIS master) are:
>
>- We can specify the relations between the layers at the project level
>(it's now easier with the auto-discover feature for PostGIS and
>Spatialite). But those are only showing in the QgsAttributeForm for the 1-N
>side (the side that doesn't have the foreign key). Why not on the N-1 side?
>- For showing the N-1 side in the QgsAttributeForm, one can define a
>Join in the layer's properties, but I don't see the point of having to
>define it here as well when we have already the relations info at the
>project level. I see a use for special joins, but for relations, I don't
>see why we have to define it twice. And the way it's displayed is not
>allowing to create joins or edit the joined fields.
>- I let you imagine the look of the feature attribute form when there
>are hundreds of directly and indirectly linked tabled. This is just not
>usable if we display all of them directly like that. Just look at the
>attached screen shot that shows what happens by default with only 3 tables.
>It's already a mess.
>
> Now, what I propose is:
>
>- Not expand the relation widget (QgsCollapsibleGroupBox) by default
>and build it's content only when it is expanded the first time (think of
>what would happen when you have loops in the schema).
>- Show N-1 relations as well, in a collapsed by default
>QgsCollapsibleGroupBox, including a way to add a new linked entry, remove
>the link (put the FK to NULL) and delete it.
>- Add a button to open a related feature in a new window.
>
> What do you guys think?
>
> Thanks.
>
___
Qgis-developer mailing list
Qgis-developer@lists.osgeo.org
List info: http://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: http://lists.osgeo.org/mailman/listinfo/qgis-developer

Re: [Qgis-developer] A different behavior in Qt5 beaks a lot of things.

[Patrick Valsecchi]

hummm... in fact I start to think the problem is there from the start. I've
just made it more visible with my "better default widget" feature.
The operations to reproduce:

   - new project
   - add a postgis layer with integer attributes
   - open the attribute form and see that everything is OK
   - edit the layer properties
   - have a look at the fields
   - click OK (will make sure there are widgetv2config in the XML)
   - save the project somewhere
   - open the project
   - open the attribute form and see that all the integer fields are 0

OK, I'm working on a fix for the widgets that now appears by default.

On Fri, Oct 7, 2016 at 11:27 AM, Patrick Valsecchi <
patrick.valsec...@camptocamp.com> wrote:

> Hi,
>
> I was investigating a new bug when editing a feature's attribute with the
> current master.
> For example, if you don't touch the configuration of a layer containing
> integer fields, save the project, exit, re-open the project, when you edit
> the attribute of this layer, all the integer fields are forced to 0.
>
> The problem must come from a different behavior with Qt5. Before, the
> field config for text was saved like that:
>  notNull="0"/>
>
> Now, it's saved like that:
>  constraintDescription="" fieldEditable="1" notNull="0" Max="" Step=""
> constraint=""/>
>
> See? All the non-boolean fields that are not specified are now there with
> an empty value.
>
> My guess is that QDomElement::setAttribute was a no-op for QString::null
> and now it adds an attribute with an empty string. I don't know. But it's a
> mess now. Code in the QgsEditorWrapper will end up not using default values
> when they do stuff like:
> const QString displayFormat = config( "display_format",
> QgsDateTimeEditConfig::defaultFormat( field().type() ) ).toString();
>
> The config entry being always there, but empty when not specified, the
> default will never be taken. It's then same for range widget's min and max
> values then ends up being 0 (my original bug).
>
> And I'm afraid the edit widgets are not the only ones suffering from those
> problems.
>
> Am I the only one seeing that? What do we do, now?
>
> Thanks
>
___
Qgis-developer mailing list
Qgis-developer@lists.osgeo.org
List info: http://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: http://lists.osgeo.org/mailman/listinfo/qgis-developer

[Qgis-developer] A different behavior in Qt5 beaks a lot of things.

[Patrick Valsecchi]

Hi,

I was investigating a new bug when editing a feature's attribute with the
current master.
For example, if you don't touch the configuration of a layer containing
integer fields, save the project, exit, re-open the project, when you edit
the attribute of this layer, all the integer fields are forced to 0.

The problem must come from a different behavior with Qt5. Before, the field
config for text was saved like that:


Now, it's saved like that:


See? All the non-boolean fields that are not specified are now there with
an empty value.

My guess is that QDomElement::setAttribute was a no-op for QString::null
and now it adds an attribute with an empty string. I don't know. But it's a
mess now. Code in the QgsEditorWrapper will end up not using default values
when they do stuff like:
const QString displayFormat = config( "display_format",
QgsDateTimeEditConfig::defaultFormat( field().type() ) ).toString();

The config entry being always there, but empty when not specified, the
default will never be taken. It's then same for range widget's min and max
values then ends up being 0 (my original bug).

And I'm afraid the edit widgets are not the only ones suffering from those
problems.

Am I the only one seeing that? What do we do, now?

Thanks
___
Qgis-developer mailing list
Qgis-developer@lists.osgeo.org
List info: http://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: http://lists.osgeo.org/mailman/listinfo/qgis-developer

Re: [Qgis-developer] Decoupling core/data and GUI part of widgets in 3.0

[Patrick Valsecchi]

On Wed, Aug 31, 2016 at 4:38 PM, Matthias Kuhn <matth...@opengis.ch> wrote:

> On 08/31/2016 04:12 PM, Patrick Valsecchi wrote:
> > That would force the plugin writters to provide two plugins:
>
> Not really, you can add two register calls in one plugin.
>
> Then the plugin woud only work for QGIS desktop and not for QGIS server...
___
Qgis-developer mailing list
Qgis-developer@lists.osgeo.org
List info: http://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: http://lists.osgeo.org/mailman/listinfo/qgis-developer

Re: [Qgis-developer] Decoupling core/data and GUI part of widgets in 3.0

[Patrick Valsecchi]

Hi,

On Wed, Aug 31, 2016 at 1:20 PM, Matthias Kuhn  wrote:

> On 08/31/2016 12:41 PM, Nyall Dawson wrote:
> > Just to clarify - are you proposing that only a map of widget
> > configuration is moved to core? I'd say the representValue function
> > for each widget should also should be moved across, otherwise we'll
> > end up with multiple code paths reimplementing the logic from each
> > widget + plugins having to redo this themselves.
>
> Good point,
>
> that will also be nice to have in core.
>
> In this case we'll end up with two registries and two factories
> (although only a few widgets will need core functionality).
> Or did you have a different idea in mind?
>
>

That would force the plugin writters to provide two plugins: one for core
that could be used for qgis server as well and one for gui. Then how do we
make sure the user install the two plugins at the same version? I guess the
GUI plugin will depend on the core one...

I do agree that representValue would be useful in core, but that has some
cost to split the thing in two. Or maybe reprensent value is something
totally different from the widgets for editing the fields and they don't
have to be in 1-1 relation.

CU
___
Qgis-developer mailing list
Qgis-developer@lists.osgeo.org
List info: http://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: http://lists.osgeo.org/mailman/listinfo/qgis-developer

[Qgis-developer] Editor Widget for hstore (QVariantMap)

[Patrick Valsecchi]

Hi,

I'm working on [1] and I'm adding a widget for editing a key/value field. I
want it to look like that in the form view (yes I still have some details
to iron out):
[image: Inline image 1]

But, for the table view, this widget is too big to fit in a table cell. So
want I envision is too show the keys/values in a simple textual form and
when the user clicks on the cell to edit it, to pop up a window with the
widget as shown in the form view.

What do you guys think?

I've looked at the code and found nothing with a similar behavior for
editor widgets. Did I miss something or do I have to refactor a bit the
code to add the possibility to have a different widget for the table view?

Thanks for your help.

[1] https://github.com/qgis/QGIS-Enhancement-Proposals/issues/66
___
Qgis-developer mailing list
Qgis-developer@lists.osgeo.org
List info: http://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: http://lists.osgeo.org/mailman/listinfo/qgis-developer

Re: [Qgis-developer] Perfs: a lot of WKB conversions

[Patrick Valsecchi]

Hi,

I'll take the opportunity of being at the codesprint to try to reduce the
amount of WKB usage in the code.

For those being at the code sprint, don't hesitate do drop by if you have
recommendations or just want to talk.

CU

On Tue, Jul 19, 2016 at 10:52 AM, Martin Dobias <wonder...@gmail.com> wrote:

> Hi Patrick
>
> On Tue, Jul 19, 2016 at 9:58 AM, Patrick Valsecchi
> <patrick.valsec...@camptocamp.com> wrote:
> > Hi,
> >
> > I was looking at the perfs of qgis server and I did a small profiling of
> a
> > GetMap on a road layer (linestring) with QGIS configured with defaults.
> We
> > spend a lot of time converting to and from WKB. For example, for rending
> 67k
> > features (3 GetMaps), we do this for each feature:
>
> Just curious - what did you use for profiling? Release or Debug build?
>
>
> > Now, simplifying all the WKB parse/write will gain maybe 10-15% in global
> > perfs for this case. But there are maybe reasons for that (memory usage
> or I
> > don't know)?
>
> For a long time, WKB was the only way to store geometries in QGIS (it
> started as PostGIS viewer after all :-). Later GEOS representation was
> added and more recently the new geometry classes, so now the
> geometries are stored natively in QgsLineStringV2 and others, but
> there is still a fair amount of conversions going on in the background
> as various parts still use WKB representation. So no reason really for
> juggling with WKB apart from historical reasons...
>
> It would be good to convert the code to use native representation
> everywhere and use WKB only for import/export.
>
>
> > Why not avoiding simplifying a linestring if its bbox is in the same
> > ballpark as the bbox we are rendering? Or maybe this logic: if bboxLS.w /
> > nbPoints > pixelW*2 or bboxLS.h / nbPoints > pixelH*2, then we don't do
> > simplification. The idea being points are roughly equally spaced in a
> > linestring.
>
> Sounds quite interesting - would be good to get some numbers how much
> this would help.
>
>
> > Why storing the points as two arrays in QgsLineStringV2 instead of
> directly
> > a QPolygonF?
>
> Nyall has explained already...
>
>
> > I've been focusing only on linestrings for the moment. But I guess the
> same
> > applies to the other types of geometry.
>
> Yes, but obviously the results from profiling vary quite a lot
> depending on provider type and used symbology.
>
>
> > I know it's a lot of questions for a single email. But I'm new to this
> code
> > and have no knowledge of its history and the choices/tradeoffs that were
> > made.
>
> No problem!
>
> Cheers
> Martin
>
___
Qgis-developer mailing list
Qgis-developer@lists.osgeo.org
List info: http://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: http://lists.osgeo.org/mailman/listinfo/qgis-developer

Re: [Qgis-developer] JSON parsing

[Patrick Valsecchi]

Hi Nyall,

On Fri, Aug 19, 2016 at 11:11 AM, Nyall Dawson 
wrote:

> Just wondering - why not XML in the database? It's how styles are stored,
> so it'd be nice to keep consistency.
>

The widget config is a QMap and it was easier to convert
a JSON into that. But yes, this is a good idea (eventhough I hate XML).

OK, switching to that.

Thanks.
___
Qgis-developer mailing list
Qgis-developer@lists.osgeo.org
List info: http://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: http://lists.osgeo.org/mailman/listinfo/qgis-developer

[Qgis-developer] JSON parsing

[Patrick Valsecchi]

Hello,

For my editor widget stuff, I need to parse JSON (to get the stored the
widget config from the DB). I've seen that the arcgirest provider uses
qjson for that, so I've tried to use it. Works fine on my machine, but
fails to compile on Travis (missing include file for qjson/parser.h). Now,
I've seen that the arcgisrest provider is not compiled in Travis. That
explains why it is green without my patch.

Now, can I add qjson-dev as dependency to the Travis build in master?

For QT5, there is a JSON parser included, so I could #ifdef the hell out of
it and use the Qt parser if in Qt5.

What do you guys think?

Thanks.
___
Qgis-developer mailing list
Qgis-developer@lists.osgeo.org
List info: http://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: http://lists.osgeo.org/mailman/listinfo/qgis-developer

Re: [Qgis-developer] Perfs: a lot of WKB conversions

[Patrick Valsecchi]

On Tue, Jul 19, 2016 at 11:38 AM, Even Rouault 
wrote:

> I'm not sure to have followed your computation, but it is wrong somewhere
> (must be a km vs m confusion).
>

Yes indeed. Shame on me. The earth circumference is 40'000km, not 40'000m.
___
Qgis-developer mailing list
Qgis-developer@lists.osgeo.org
List info: http://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: http://lists.osgeo.org/mailman/listinfo/qgis-developer

Re: [Qgis-developer] Perfs: a lot of WKB conversions

[Patrick Valsecchi]

Hi Martin,

On Tue, Jul 19, 2016 at 10:52 AM, Martin Dobias  wrote:

> Just curious - what did you use for profiling? Release or Debug build?
>

When trying to optimize code like QGIS server, I tend to do that:

   1. Take a scenario that matches the realworld, run it with a RELEASE
   build on a machine matching somehow what is the usual production machine,
   then:
  1. do a warmup round to make sure the stuff as reached cruise speed
  and throw away the measurements.
  2. measuring global real times for several queries (2 minutes full
  speed) and look at the average response time and the distribution.
  2. I did the same measurements for other servers (MapServer and
   GeoServer), just for reference.
   3. When that is done, and only when that is done, I take a single query,
   that I run against QGIS compiled in DEBUG (to have symbols and to be able
   to follow what's happening) and run it under valgrind (callgrind mode).
   4. Then, after identifying trouble zones, I change the code and run step
   1 to see if there is any change. Keep the change if the improvement is
   measurable worth the added complexity.
   5. Repeat step 3 until you run out of budget and/or the improvements are
   becoming too small to be measured.

So yes, my per function numbers are not very accurate, but they are a very
good indication of where time is spent. But, I still have the global
numbers with a RELEASE build to see if changes are improving things.


> Sounds quite interesting - would be good to get some numbers how much
> this would help.
>

My plan is to find some time to do that. It shouldn't take too much time.


> Yes, but obviously the results from profiling vary quite a lot
> depending on provider type and used symbology.
>

The provider, at least PostGIS on the same machine, is not contributing too
much on the time spent to do a simple GetMap, from what I've seen. But yes,
when optimizing, you have to choose a scenario and it impacts a lot what
you see.

Thanks for the history lesson. It's always interesting to understand from
where a code comes from.

CU
___
Qgis-developer mailing list
Qgis-developer@lists.osgeo.org
List info: http://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: http://lists.osgeo.org/mailman/listinfo/qgis-developer

Re: [Qgis-developer] Perfs: a lot of WKB conversions

[Patrick Valsecchi]

Hi Nyall,

On Tue, Jul 19, 2016 at 10:25 AM, Nyall Dawson 
wrote:

> See https://github.com/qgis/qgis3.0_api/issues/15
>
>
Cool, that would force us to use the coordinates directly instead of the
WKB.


> We can't use a QPolygonF, as it uses qreal for storage. For certain
> architectures this is float, not double, and so is unsuitable for
> storing coordinates.
>

I was a bit sceptical, so I did a quick computation. A 32 bit float allows
an accuracy of 3mm with WGS84 for coordinates around 180°:
180 takes 8 bits out of the 23 bits of fraction.  The remaining 15 bits can
split the earth circumference into chunks of 3mm (4/(2^15)). I'm maybe
off by +1 bit of precision (the fraction has an implicit 1 at MSB), so that
would be 1.5mm.

Other coordinate systems have offsets. For example CH1903+ which has an
offset of 2e6 that is reducing the precision of a float coordinate to 13cm.

Makes sense to use 64bits floats.

But then, we transform into screen coordinates after we converted to
QPolygonF. We loose the precision anyway. Isn't that a problem? But I'm
digressing and this is a problem only for ARM.

I think the first logical step would be converting the clipper to work
> directly with QgsAbstractGeometryV2, so that QgsSymbol can avoid one
> of the to/from conversions at least.
>

IMHO, implementing my proposed heuristic for not simplifying some
geometries would lead to better gains and is less changes to the code.
Usually, features that are around a few pixel of size are not rendered.
Their class should be filtered out by the scale range, if the layer style
is correctly tuned.

Thanks
___
Qgis-developer mailing list
Qgis-developer@lists.osgeo.org
List info: http://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: http://lists.osgeo.org/mailman/listinfo/qgis-developer

[Qgis-developer] Perfs: a lot of WKB conversions

[Patrick Valsecchi]

Hi,

I was looking at the perfs of qgis server and I did a small profiling of a
GetMap on a road layer (linestring) with QGIS configured with defaults. We
spend a lot of time converting to and from WKB. For example, for rending
67k features (3 GetMaps), we do this for each feature:

   - When reading the features from the DB, *we parse the WKB* and convert
   it to a QgsLineStringV2 (array of x and array of y)
   - Then, in QgsSymbolV2, we get the WKB from the QgsLineStringV2 (OK, no
   conversion, we kept the WKB)
   - Then QgsClipper::clippedLineWKB uses the QgsConstWkbSimplifierPtr
   which uses QgsMapToPixelSimplifier::simplifyPoints:
  - *Parse the WKB* to get the bounding box
  - *Parse the WKB* to read the points, simplify them and *write a new
  WKB* with the simplified geometry
  - *Parse the simplified WKB* to read the points into a QPolygonF

So that is 4 parsing of WKB and 1 write WKB instead of the single parse
that is really needed. I have trouble getting exact numbers but we spend
roughly 15% of our time doing just that. For example, we spend globally
4.3% of our time calling 1M times QgsConstWkbPtr::verifyBound.

A few numbers (% are relative to the caller and I show only the major
stuff):

   - QgsWMSServer::executeRequest
  - 83% in getMap ->QgsVectorLayerRenderer::drawRendererV2
 - 69% in renderFeature -> QgsSymbolV2::renderFeature
 - 52% in QgsLineSymbolV2::renderFeature (mostly Qt rendering stuff)
- 35% in QgsSymbolV2::_getLineString ->
QgsMapToPixelSimplifier::simplifyPoints
   - 46% simplifyWkbGeometry
   - 36% calculateBoundingBox
   - 11% QgsConstWkbPtr::operator>>
   - 25% in QgsFeatureIterator::nextFeature ->
 QgsPostgresFeatureIterator::fetchFeature
 - 79% in QgsPostgresFeatureIterator
- 4% QgsPostgresConn::PGgetResult
- others...
- 17% in QgsHttpRequestHandler::setGetMapResponse

Now, simplifying all the WKB parse/write will gain maybe 10-15% in global
perfs for this case. But there are maybe reasons for that (memory usage or
I don't know)?
Why not avoiding simplifying a linestring if its bbox is in the same
ballpark as the bbox we are rendering? Or maybe this logic: if bboxLS.w /
nbPoints > pixelW*2 or bboxLS.h / nbPoints > pixelH*2, then we don't do
simplification. The idea being points are roughly equally spaced in a
linestring.

Why storing the points as two arrays in QgsLineStringV2 instead of directly
a QPolygonF?

I've been focusing only on linestrings for the moment. But I guess the same
applies to the other types of geometry.

I know it's a lot of questions for a single email. But I'm new to this code
and have no knowledge of its history and the choices/tradeoffs that were
made.

Thanks.


PS:

I think I've found what I'll do at Bonn's Hackfest ;-)
___
Qgis-developer mailing list
Qgis-developer@lists.osgeo.org
List info: http://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: http://lists.osgeo.org/mailman/listinfo/qgis-developer

[Qgis-developer] QGIS for INSPIRE

[Patrick Valsecchi]

Hi,

We (titellus and camptocamp) are going to start working on adding a few
functionalities needed for INSPIRE to QGIS core.  Before starting, we’d
like to tell the community what we plan to do (maybe, some are optional)
and give a chance to everybody to comment.


   -

   Advanced attribute types support
   -

  Add support for viewing/editing maps (key/value) and arrays
  -

 Usage of native type for postgis
 -

 Usage of string representation for sqlite (for example JSON)
 -

  Add a few expression functions for accessing those types:
  array_get(idx), array_length, array_join(separator) / dict_get(key), etc
  -

  Add a few function to search in those types: array_search(),
  dict_search()
  -

  Detect a date column and use the proper widget for editing those
  -

  Same for other types
  -

  Allow plugins to provide custom widgets for specific types
  automatically.
  -

   Fix bug where null dates are shown as the current time instead of null
   in forms and saved like that, even for features that are not modified
   -

   Custom types widgets, per type
   -

   Auto discover database joins:
   -

  Add FK (Foreign Key) discovery to the postgis and spatialite specific
  code.
  -

  Improve QGIS when creating relations according to the FK in the DB:
  view existing relations in the DB and select those to add
automatically to
  the project.
  -

  Improve QGIS when creating joins in layers according to the FK in the
  DB: view existing relations with the current layer and select
those to add
  automatically.


If somebody is already working on one of those features or is having
recommendations/comments/questions, please tell us.

Is there any point that would require a QEP?

Thanks
___
Qgis-developer mailing list
Qgis-developer@lists.osgeo.org
List info: http://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: http://lists.osgeo.org/mailman/listinfo/qgis-developer