subject:"Re\: \[Qgis\-developer\] Fwd\:"

Re: [QGIS-Developer] Fwd: Question about a plugin. POC attached.

2023-04-15 Thread Giordano Cetti via QGIS-Developer

Dear Tim,

It seems that 99% of developers were asking themselves the same question
and hesitated to examine the code.

Regardless, I think it's already good to have shared this small proof of
concept with the community, demonstrating an integration between Django and
QGIS, especially considering that it has been implemented with a rather
robust certificate verification mechanism.

So, rest assured, my question does not hide an intention contrary to GPL
policy, but I remain curious to understand the boundary between a process
considered external and a process considered internal to QGIS, and
therefore subject to the same GPL license. If I ever present a project to a
client, I would like to be able to tell them that I applied a defense
mechanism (albeit simple, such as compilation) to prevent the leakage of
endpoints called by the REST APIs.

However, I think I have answered myself: after researching the issue
online, I found that calling QGSTask in a function is considered an
internal process to QGIS and thus subject to GPL. The subsequent question
that emerged and that I still cannot answer is: would it be acceptable if
only the run() function or a subsequent function were compiled, just enough
to close the REST parameters? There probably isn't a definitive answer;
could that portion of code be considered an extension of pycurl+certifi?
Maybe yes, maybe no. It matters little, I guess.

Thank you for the time you spent writing this request for clarification.

Best regards,

Giordano Cetti

CTO @ByCloudSRL

pec: bycl...@pec.it
giordano.ce...@bycloud.eu
https://bycloud.eu

Il giorno sab 15 apr 2023 alle 22:39 Tim Sutton  ha
scritto:

> Hi
>
> I havent looked at your code, but perhaps you could add to your original
> message an explanation of why you don't just distribute it as a py /
> uncompiled python source file?
>
> Regards
>
> Tim
>
> On Tue, Apr 11, 2023 at 6:48 PM Giordano Cetti via QGIS-Developer <
> qgis-developer@lists.osgeo.org> wrote:
>
>> Greetings to QGIS developers,
>>
>> I would greatly appreciate it if someone could check if the attached
>> plugin complies with the licenses compared to the imported modules.
>>
>> In this specific attached plugin, all the source code is open, but the
>> question is: what if it comes with the following file:
>> *'djangorest/include/djangorest_compiled.py'* really compiled as a .pyd
>> file? Would it violate the GPL license terms or is it just enough to be
>> considered as an acceptable external process?
>>
>> The .pyd will just import QgsTask from qgis.core and use a session_path
>> received as a text created by the open source part of the plugin
>> using QgsProcessingUtils.
>>
>> The attached one is a very simple plugin made for this demonstration, you
>> can click on the button, just type google.it, and it will download the
>> 404 page found ( because it appends some string that google doesn't serve
>> as content ). For user and password fields: just type anything, they are
>> not used in the POC but there's still a check active on fields population.
>> The only action the plugin will do is just a single pycurl GET request to
>> the address specified.
>>
>> Any advice will be appreciated.
>> Thanks
>>
>> *ATTACHMENT BLOCKED BY GOOGLE SCAN BECAUSE IT INCLUDES PYCURL AND CERTIFI
>> LIBRARY SO I SHARE USING GOOGLE DRIVE LINK. I TESTED IT ONLY ON QGIS 3.10*
>>
>> *https://drive.google.com/file/d/1qS7h3LaZ6BlBAW3AItEjM5swpDocHbue/view?usp=sharing
>> *
>>
>> ___
>> QGIS-Developer mailing list
>> QGIS-Developer@lists.osgeo.org
>> List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
>> Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer
>>
>
>
> --
>
> --
> 
>
> Tim Sutton
> Kartoza Co-Founder
> Visit http://kartoza.com to find out about open source:
>  * Desktop GIS programming services
>  * Geospatial web development
> * GIS Training
> * Consulting Services
> Tim is a member of the QGIS Project Steering Committee
>
> ---
>
___
QGIS-Developer mailing list
QGIS-Developer@lists.osgeo.org
List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer

Re: [QGIS-Developer] Fwd: Question about a plugin. POC attached.

2023-04-15 Thread Tim Sutton via QGIS-Developer

Hi

I havent looked at your code, but perhaps you could add to your original
message an explanation of why you don't just distribute it as a py /
uncompiled python source file?

Regards

Tim

On Tue, Apr 11, 2023 at 6:48 PM Giordano Cetti via QGIS-Developer <
qgis-developer@lists.osgeo.org> wrote:

> Greetings to QGIS developers,
>
> I would greatly appreciate it if someone could check if the attached
> plugin complies with the licenses compared to the imported modules.
>
> In this specific attached plugin, all the source code is open, but the
> question is: what if it comes with the following file:
> *'djangorest/include/djangorest_compiled.py'* really compiled as a .pyd
> file? Would it violate the GPL license terms or is it just enough to be
> considered as an acceptable external process?
>
> The .pyd will just import QgsTask from qgis.core and use a session_path
> received as a text created by the open source part of the plugin
> using QgsProcessingUtils.
>
> The attached one is a very simple plugin made for this demonstration, you
> can click on the button, just type google.it, and it will download the
> 404 page found ( because it appends some string that google doesn't serve
> as content ). For user and password fields: just type anything, they are
> not used in the POC but there's still a check active on fields population.
> The only action the plugin will do is just a single pycurl GET request to
> the address specified.
>
> Any advice will be appreciated.
> Thanks
>
> *ATTACHMENT BLOCKED BY GOOGLE SCAN BECAUSE IT INCLUDES PYCURL AND CERTIFI
> LIBRARY SO I SHARE USING GOOGLE DRIVE LINK. I TESTED IT ONLY ON QGIS 3.10*
>
> *https://drive.google.com/file/d/1qS7h3LaZ6BlBAW3AItEjM5swpDocHbue/view?usp=sharing
> *
>
> ___
> QGIS-Developer mailing list
> QGIS-Developer@lists.osgeo.org
> List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
> Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer
>


-- 
--


Tim Sutton
Kartoza Co-Founder
Visit http://kartoza.com to find out about open source:
 * Desktop GIS programming services
 * Geospatial web development
* GIS Training
* Consulting Services
Tim is a member of the QGIS Project Steering Committee
---
___
QGIS-Developer mailing list
QGIS-Developer@lists.osgeo.org
List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer

Re: [QGIS-Developer] Fwd: [gdal-dev] Announcing SOZip: Seek-Optimized profile for the .zip format

2023-01-09 Thread Even Rouault via QGIS-Developer


> Very cool project, Even! (Why not "Cloud-Optimized" though?  )

The term is heavily loaded and quite vague. SOZip doesn't make something 
cloud unfriendly magically cloud friendly. If you need to seek at 100 
different locations in a (uncompressed) file, you'll need to also seek 
at 100 locations in its SOZip compressed version (which is 
excruciatingly slow for a regular non-optimized ZIP). SOZip will shine 
most for local use cases, where this seeking in the underlying 
compressed stream is a cheap operation. For remote use, your mileage may 
vary.


--
http://www.spatialys.com
My software is free, but my time generally not.

___
QGIS-Developer mailing list
QGIS-Developer@lists.osgeo.org
List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer

Re: [QGIS-Developer] Fwd: [gdal-dev] Announcing SOZip: Seek-Optimized profile for the .zip format

2023-01-09 Thread WhereGroup


Am 09.01.23 um 16:54 schrieb Richard Duivenvoorde via QGIS-Developer:


Anybody has experience with zipping "lrge zipped GeoPackages"? ;-) 
Is that useful? I just tested a 885MB mbtiles file (I know not 
geopackage...but still sqlite isnt't it?), and that ended up in 868MB. 


For *raster* GPKGs/MBTiles it won't do much. JPG and PNG tiles are 
already compressed and DEFLATE on top won't do anything interesting 
unless there is lots of identical tiles inside

For *vector* data it should offer considerable savings in most cases.

E. g.:

114M Address_25832_all.gpkg
16M Address_25832_all.gpkg.zip  # standard zip, not sozip

Or see https://github.com/sozip/sozip-examples for actual sozip examples.

Very cool project, Even! (Why not "Cloud-Optimized" though? ;) )

Cheers, Hannes

___
QGIS-Developer mailing list
QGIS-Developer@lists.osgeo.org
List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer

Re: [QGIS-Developer] Fwd: [gdal-dev] Announcing SOZip: Seek-Optimized profile for the .zip format

2023-01-09 Thread Even Rouault via QGIS-Developer



Le 09/01/2023 à 16:54, Richard Duivenvoorde a écrit :

Hi Even,

Cool!

Out of curiosity: I thought sqlite/geopackage was already relatively 
'skinny' packed? Am I wrong?
Well, SQLite does some packing of integer or floating-point values, but 
nothing on strings or blobs (which means no compression o the WKB 
extended geometries of GeoPackage). You can get easily a 3x compression 
ratio when zipping GeoPackage vector files (of course depends a lot on 
what you put in them)


Anybody has experience with zipping "lrge zipped GeoPackages"? ;-) 
Is that useful? I just tested a 885MB mbtiles file (I know not 
geopackage...but still sqlite isnt't it?), and that ended up in 868MB.


I have successfully tested a ~30 GB uncompressed GeoPackage vector file 
compressed as ~10 GB as SOZip.


MBTiles holds PNG or JPEG tiles which are already compressed, so no 
surprise that the Deflate compression of ZIP doesn't help.




OR... are we talking about sets of geopackages?


The SOZip spec is all about laaarge-files-in-zip. Considerations of 
many-files-in-zip are not covered by it (they are orthogonal)



--
http://www.spatialys.com
My software is free, but my time generally not.

___
QGIS-Developer mailing list
QGIS-Developer@lists.osgeo.org
List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer

Re: [QGIS-Developer] Fwd: [gdal-dev] Announcing SOZip: Seek-Optimized profile for the .zip format

2023-01-09 Thread Richard Duivenvoorde via QGIS-Developer

Hi Even,

Cool!

Out of curiosity: I thought sqlite/geopackage was already relatively 'skinny'
packed? Am I wrong?

Anybody has experience with zipping "lrge zipped GeoPackages"? ;-) Is that
useful? I just tested a 885MB mbtiles file (I know not geopackage...but still sqlite
isnt't it?), and that ended up in 868MB.

OR... are we talking about sets of geopackages?

Regards,

Richard Duivenvoorde

On 1/9/23 16:42, Even Rouault via QGIS-Developer wrote:

Sorry for cross-posting, but very relevant topic for QGIS. To make it short,
pending compressing .zip files in the SOZip way, it is possible to directly
read lrge zipped GeoPackage files (or Shapefiles for nostalgic) from QGIS
without prior decompression

Even

Message transféré
Sujet : [gdal-dev] Announcing SOZip: Seek-Optimized profile for the
.zip format
Date : Mon, 9 Jan 2023 15:19:07 +0100
De :Even Rouault
Pour : gdal-...@lists.osgeo.org

Hi,

It is my pleasure to announce (
https://github.com/sozip/sozip-spec/blob/master/blog/01-announcement.md ) the
initial release of the specification (
https://github.com/sozip/sozip-spec/blob/master/sozip_specification.md ) for
the SOZip (Seek-Optimized Zip) profile to the ZIP file format, as well as its
GDAL implementation.

What is SOZip ?
--

A Seek-Optimized ZIP file (SOZip) is a ZIP file that contains one or several
Deflate-compressed files that are organized and annotated such that a
SOZip-aware reader can perform very fast random access (seek) within a
compressed file.

SOZip makes it possible to access large compressed files directly from a .zip
file without prior decompression. It is not a new file format, but a profile of
the existing ZIP format, done in a fully backward compatible way. ZIP readers
that are non-SOZip aware can read a SOZip-enabled file normally and ignore the
extended features that support efficient seek capability.

Use cases
--

The SOZip specification is intended to be general purpose / not domain
specific. It was first developed to serve geospatial use cases, which commonly
have large compressed files inside of ZIP archives. In particular, it makes it
possible for users to read large GIS files using the Shapefile, GeoPackage or
FlatGeobuf formats (which have no native provision for compression) compressed
in .zip files without prior decompression.

Efficient random access and selective decompression are a requirement to
provide acceptable performance in many usage scenarios: spatial index
filtering, access to a feature by its identifier, etc.

Performance
--

SOZip is efficient:

* The overhead of using a file from a SOZip archive, compared to using it
uncompressed, is of the order of 10% for common read operations.
* Generation of a SOZip file can be much faster than regular ZIP generation
when using multithreading.
* SOZip files are typically only ~ 5% larger than regular ZIPs (dependent on
content, and chunk size)

Have a look at benchmarking results:
https://github.com/sozip/sozip-spec/blob/master/README.md#benchmarking

Other ZIP related specification
--

The SOZip GitHub organization also hosts the KeyValuePairs extra-field
specification (
https://github.com/sozip/keyvaluepairs-spec/blob/master/zip_keyvalue_extra_field_specification.md
), to be able to encode arbitrary key-value pairs of metadata associated with
a file within a ZIP. For example to store the Content-Type of a file.

How does this relate to GDAL ?
---

Pull request https://github.com/OSGeo/gdal/pull/7042 has been submitted with
the following enhancements:

* The /vsizip/ virtual file system uses the SOZip index to perform fast
random access within a compressed SOZip-enabled file.

* The Shapefile and GPKG drivers can directly generate SOZip-enabled
.shz/.shp.zip or .gpkg.zip files.

* Addition of the CPLAddFileInZip() C function that can compress a file and add
it to an new or existing ZIP file, and enable the SOZip optimization when
relevant.

* The existed VSIGetFileMetadata() method can be called on a filename of
the form /vsizip/path/to/the/file.zip/path/inside/the/zip/file and
with domain = "ZIP" to get information if a SOZip index is available for
that file.

* The sozip
(https://github.com/rouault/gdal/blob/sozip/doc/source/programs/sozip.rst) new
command line utility
can be used to create a seek-optimized ZIP file, to append files to an
existing ZIP file, list the
contents of a ZIP file and display the SOZip optimization status or
validate a SOZip file.

1 2 3 4 >

1 - 100 of 312 matches

Mail list logo