[gdal-dev] GDAL /vsistdin/ 1MB limit

[Romeo Alvaraz]

Hi there, 

I've got a PDF geo-referencing implementation in place which makes use of
the gdal_translate utility.  Recently I've been testing out and
implementation to read in the input content from standard input stream using
the  /vsistdin/ handler.  

According to the documentation for /vsistdin/ handler -  "Full seek in the
first MB of a file is possible" and I've encountered the following error(s)
when the content I've attempted to pipe in is greater than 1MB. 

ERROR 6: Seek(SEEK_END) unsupported on /vsistdin when stdin > 1 MB
ERROR 6: backward Seek() unsupported on /vsistdin above first MB

Would this be a hard limit of 1MB placed the content size which can be piped
in via /vsistdin/(the source code and the error message suggests it migh me)
or  can it be configured? Has anyone encountered this before? 

Any suggestions will be appreciated

Thanks. 



Platform: Windows 64-bit
Version: GDAL 2.4.4, released 2020/01/08 sourced from
https://www.gisinternals.com/release.php




--
Sent from: http://osgeo-org.1560.x6.nabble.com/GDAL-Dev-f3742093.html
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] New JP2Grok Driver for JPEG 2000

[Aaron Boxer]

Hi Sean and Even,
Correct, the current JP2Grok does not support the generation of Part 15
code streams.
It's trivial to add, as it just involves adding a new encoding mode, but I
was waiting for a merge
before looking into new driver features.

Also, yes, OpenJPH is being used for the new entropy encoder. As OpenJPH
library is in its
infancy and is missing many features of a mature implementation, I would
recommend
the best approach to be integrating OpenJPH into OpenJPEG and upgrading the
gdal driver
accordingly.

Cheers,
Aaron

On Mon, May 3, 2021 at 4:40 PM Even Rouault 
wrote:

> Sean,
>
> my quick look at the JP2Grok driver code is that it doesn't generate the
> HTJ2K / Part 15 variant of JPEG2000, at least there's nothing suggesting
> that (didn't try running it). I guess that there would be a need for a
> dedicated creation option in the driver for that to happen, so there
> shouldn't be interoperability issues of the files it generates with other
> JPEG2000 capable drivers.
>
> I'm not sure which support of HTJ2K the ECW and MrSID SDKs have. I believe
> there's such support Kakadu in recent releases, but might be subject to
> extra fees (or perhaps for  faster support)
>
> There's a BSD 2 licensed implementation of HTJ2K in
> https://github.com/aous72/OpenJPH . From what I see, Grok HTJ2K support
> has incorporated some of that code.  So for a BSD 2 compatible use in GDAL,
> either a GDAL driver should be built from OpenJPH as a standalone library,
> or relevant parts of OpenJPH should be "ported" to OpenJPEG.
>
> Even
> Le 03/05/2021 à 22:17, Sean Gillies via gdal-dev a écrit :
>
> Hi  Aaron, and everyone,
>
> It seems like interoperability could be harmed if we release GDAL with a
> JP2 driver that writes JPEG 2000 files that the main open source JP2 driver
> can't read. Would it make sense to add compatibility to OpenJPEG before the
> PR gets merged? Or are we already in a state of inoperability between
> JP2ECW, JP2MRSID, and JP2OpenJPEG?
>
> On Mon, May 3, 2021 at 1:52 PM Aaron Boxer  wrote:
>
>> Hi Jean-Roc,
>> Good question - unfortunately OpenJPEG doesn't currently support HTJ2K,
>> it only supports JPEG 2000 Part 1. I am sure that this will eventually be
>> integrated,
>> but it isn't there at the moment.
>> Cheers,
>> Aaron
>>
>> On Mon, May 3, 2021 at 12:05 PM Jean-Roc Morreale (ml) <
>> jrmorreale...@enoreth.net> wrote:
>>
>>> Hi Aaron,
>>>
>>> Is the HTJ2K's output still readable by a regular GDAL/OpenJPEG ?
>>>
>>> Regards,
>>> Jean-Roc Morreale
>>
>>
> --
> Sean Gillies
>
> ___
> gdal-dev mailing 
> listgdal-dev@lists.osgeo.orghttps://lists.osgeo.org/mailman/listinfo/gdal-dev
>
> -- http://www.spatialys.com
> My software is free, but my time generally not.
>
> ___
> gdal-dev mailing list
> gdal-dev@lists.osgeo.org
> https://lists.osgeo.org/mailman/listinfo/gdal-dev
>
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] New JP2Grok Driver for JPEG 2000

[Even Rouault]

Sean,

my quick look at the JP2Grok driver code is that it doesn't generate the 
HTJ2K / Part 15 variant of JPEG2000, at least there's nothing suggesting 
that (didn't try running it). I guess that there would be a need for a 
dedicated creation option in the driver for that to happen, so there 
shouldn't be interoperability issues of the files it generates with 
other JPEG2000 capable drivers.


I'm not sure which support of HTJ2K the ECW and MrSID SDKs have. I 
believe there's such support Kakadu in recent releases, but might be 
subject to extra fees (or perhaps for  faster support)


There's a BSD 2 licensed implementation of HTJ2K in 
https://github.com/aous72/OpenJPH . From what I see, Grok HTJ2K support 
has incorporated some of that code.  So for a BSD 2 compatible use in 
GDAL, either a GDAL driver should be built from OpenJPH as a standalone 
library, or relevant parts of OpenJPH should be "ported" to OpenJPEG.


Even

Le 03/05/2021 à 22:17, Sean Gillies via gdal-dev a écrit :

Hi  Aaron, and everyone,

It seems like interoperability could be harmed if we release GDAL with 
a JP2 driver that writes JPEG 2000 files that the main open source JP2 
driver can't read. Would it make sense to add compatibility to 
OpenJPEG before the PR gets merged? Or are we already in a state of 
inoperability between JP2ECW, JP2MRSID, and JP2OpenJPEG?


On Mon, May 3, 2021 at 1:52 PM Aaron Boxer > wrote:


Hi Jean-Roc,
Good question - unfortunately OpenJPEG doesn't currently support
HTJ2K,
it only supports JPEG 2000 Part 1. I am sure that this will
eventually be integrated,
but it isn't there at the moment.
Cheers,
Aaron

On Mon, May 3, 2021 at 12:05 PM Jean-Roc Morreale (ml)
mailto:jrmorreale...@enoreth.net>> wrote:

Hi Aaron,

Is the HTJ2K's output still readable by a regular GDAL/OpenJPEG ?

Regards,
Jean-Roc Morreale


--
Sean Gillies

___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev


--
http://www.spatialys.com
My software is free, but my time generally not.

___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] New JP2Grok Driver for JPEG 2000

[Sean Gillies via gdal-dev]

Hi  Aaron, and everyone,

It seems like interoperability could be harmed if we release GDAL with a
JP2 driver that writes JPEG 2000 files that the main open source JP2 driver
can't read. Would it make sense to add compatibility to OpenJPEG before the
PR gets merged? Or are we already in a state of inoperability between
JP2ECW, JP2MRSID, and JP2OpenJPEG?

On Mon, May 3, 2021 at 1:52 PM Aaron Boxer  wrote:

> Hi Jean-Roc,
> Good question - unfortunately OpenJPEG doesn't currently support HTJ2K,
> it only supports JPEG 2000 Part 1. I am sure that this will eventually be
> integrated,
> but it isn't there at the moment.
> Cheers,
> Aaron
>
> On Mon, May 3, 2021 at 12:05 PM Jean-Roc Morreale (ml) <
> jrmorreale...@enoreth.net> wrote:
>
>> Hi Aaron,
>>
>> Is the HTJ2K's output still readable by a regular GDAL/OpenJPEG ?
>>
>> Regards,
>> Jean-Roc Morreale
>
>
-- 
Sean Gillies
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] New JP2Grok Driver for JPEG 2000

[Aaron Boxer]

Hi Jean-Roc,
Good question - unfortunately OpenJPEG doesn't currently support HTJ2K,
it only supports JPEG 2000 Part 1. I am sure that this will eventually be
integrated,
but it isn't there at the moment.
Cheers,
Aaron

On Mon, May 3, 2021 at 12:05 PM Jean-Roc Morreale (ml) <
jrmorreale...@enoreth.net> wrote:

> Hi Aaron,
>
> Is the HTJ2K's output still readable by a regular GDAL/OpenJPEG ?
>
> Regards,
> Jean-Roc Morreale
>
>
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] New JP2Grok Driver for JPEG 2000

[Jean-Roc Morreale (ml)]

Hi Aaron,

Is the HTJ2K's output still readable by a regular GDAL/OpenJPEG ?

Regards,
Jean-Roc Morreale

___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] Mutexes

[Even Rouault]

Andrew,

I don't see myself much value to the mutexes with timeout, at least in 
how they are used within GDAL. What you underline was actually reported 
in https://github.com/OSGeo/gdal/issues/3631. The fix would be to use a 
non-timeout mutex


Even

Le 03/05/2021 à 17:23, Andrew Bell a écrit :

Hi,

I'm looking at code in gdalwarpkernel.cpp and there are calls to 
CPLAcquireMutex that take a "timeout" argument. From looking at 
cpl_multiproc, on a non-windows system it seems the timeout is 
ignored, but on WIN32 it is respected in the call, but the return 
value is generally ignored, meaning that the subsequent code could run 
without the lock having been acquired.


This seems like strange behavior. Is there a reason for the different 
behavior on Windows and not-Windows? Am I missing something?


Thanks,

--
Andrew Bell
andrew.bell...@gmail.com 

___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev


--
http://www.spatialys.com
My software is free, but my time generally not.

___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

[gdal-dev] Mutexes

[Andrew Bell]

Hi,

I'm looking at code in gdalwarpkernel.cpp and there are calls to
CPLAcquireMutex that take a "timeout" argument. From looking at
cpl_multiproc, on a non-windows system it seems the timeout is ignored, but
on WIN32 it is respected in the call, but the return value is generally
ignored, meaning that the subsequent code could run without the lock having
been acquired.

This seems like strange behavior. Is there a reason for the different
behavior on Windows and not-Windows? Am I missing something?

Thanks,

-- 
Andrew Bell
andrew.bell...@gmail.com
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

[gdal-dev] New JP2Grok Driver for JPEG 2000

[Aaron Boxer]

Hi Folks,

I would like to draw people's attention once again to the pending JPEG 2000
driver PR
https://github.com/OSGeo/gdal/pull/3449
which was opened 3 months ago.  Since I last posted, the driver now has an
autotest script courtesy of Frank Warmerdam, and all tests are passing.

In a nutshell, performance is 2-5 times faster than the only other viable
open source driver, from OpenJPEG, in several common geospatial work flows.
And performance rises by around 2X if the new HTJ2K entropy coder is used.
The AGPL license will not work for everyone, which is why the driver is
disabled by default.

We've already discussed this at length in the previous thread and in the
PR, but if you would like to see this new driver merged, please head over
to the PR and give it a thumbs up. Or, if you don't want this driver
merged, please share your feedback on the PR.

Kind Regards,
Aaron
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] [gdal-announce] GDAL 3.3.0 is released

[Even Rouault]

Idan warned me that there was a typo in the URL to the doc. Here's the 
fixed one:


  * https://download.osgeo.org/gdal/3.3.0/gdal330doc.zip - 
documentation / website


Le 03/05/2021 à 14:48, Even Rouault a écrit :

Hi,

On behalf of the GDAL/OGR development team and community, I am pleased to
announce the release of GDAL/OGR 3.3.0. GDAL/OGR is a C++ geospatial data
access library for raster and vector file formats, databases and web 
services.

It includes bindings for several languages, and a variety of command line
tools.

   http://gdal.org/

The 3.3.0 release is a new feature release with the following highlights:

* RFC 77 
(https://gdal.org/development/rfc/rfc77_drop_python2_support.html): 
Drop Python 2 support in favor of Python 3.6 (#3142)
* RFC 78 
(https://gdal.org/development/rfc/rfc78_gdal_utils_package.html): Add 
a gdal-utils Python package
* New driver: STACTA (https://gdal.org/drivers/raster/stacta.html): 
raster driver to read Spatio-Temporal Asset Catalog Tiled Assets

* Add /vsiadls/ virtual file system for Azure Data Lake Storage Gen2
* Improved drivers: DIMAP, NITF
* Number of improvements in Python bindings
 *Add automatic loading of configuration options from a file
* Add support for enumerated, constraint and glob field domains in 
MEM, FileGDB/OpenFileGDB and GeoPackage drivers

* Deprecation:
  - Disable by default raster drivers DODS, JPEG2000, JPEGLS, 
MG4LIDAR, FUJIBAS, IDA, INGR, ZMAP and vector driver ARCGEN, 
ArcObjects, CLOUDANT, COUCHDB, DB2, DODS, FME, GEOMEDIA, GTM, INGRES, 
MONGODB, REC, WALK at runtime, unless the 
GDAL_ENABLE_DEPRECATED_DRIVER_{drivername} configuration option is set 
to YES. Those drivers are planned for removal in GDAL 3.5
  - Perl bindings are deprecated. Removal planned for GDAL 3.5. Use 
Geo::GDAL::FFI instead
* Removal of BNA, AeronavFAA, HTF, OpenAir, SEGUKOOA, SEGY, SUA, 
XPlane, BPG, E00GRID, EPSILON, IGNFHeightASCIIGrid, NTV1 drivers. 
Moved to (unsupported) https://github.com/OSGeo/gdal-extra-drivers 
repository.


More complete information on the new features and fixes in the 3.3.0 
release can be found at:


  https://github.com/OSGeo/gdal/blob/v3.3.0/gdal/NEWS

The release can be downloaded from:
  * https://download.osgeo.org/gdal/3.3.0/gdal320.zip - source as a zip
  * https://download.osgeo.org/gdal/3.3.0/gdal-3.3.0.tar.gz - source 
as .tar.gz
  * https://download.osgeo.org/gdal/3.3.0/gdal-3.3.0.tar.xz - source 
as .tar.xz
  * https://download.osgeo.org/gdal/3.3.0/gdal-grass-3.3.0.tar.gz - 
GDAL-GRASS plugin
  * https://download.osgeo.org/gdal/3.3.0/gdalautotest-3.3.0.tar.gz - 
test suite
  * https://download.osgeo.org/gdal/3.3.0/gdal320doc.zip - 
documentation / website


Best regards,

Even


--
http://www.spatialys.com
My software is free, but my time generally not.

___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

[gdal-dev] GDAL 3.3.0 is released

[Even Rouault]

Hi,

On behalf of the GDAL/OGR development team and community, I am pleased to
announce the release of GDAL/OGR 3.3.0. GDAL/OGR is a C++ geospatial data
access library for raster and vector file formats, databases and web services.
It includes bindings for several languages, and a variety of command line
tools.

   http://gdal.org/

The 3.3.0 release is a new feature release with the following highlights:

* RFC 77 (https://gdal.org/development/rfc/rfc77_drop_python2_support.html): 
Drop Python 2 support in favor of Python 3.6 (#3142)
* RFC 78 (https://gdal.org/development/rfc/rfc78_gdal_utils_package.html): Add 
a gdal-utils Python package
* New driver: STACTA (https://gdal.org/drivers/raster/stacta.html): raster 
driver to read Spatio-Temporal Asset Catalog Tiled Assets
* Add /vsiadls/ virtual file system for Azure Data Lake Storage Gen2
* Improved drivers: DIMAP, NITF
* Number of improvements in Python bindings
 *Add automatic loading of configuration options from a file
* Add support for enumerated, constraint and glob field domains in MEM, 
FileGDB/OpenFileGDB and GeoPackage drivers
* Deprecation:
  - Disable by default raster drivers DODS, JPEG2000, JPEGLS, MG4LIDAR, 
FUJIBAS, IDA, INGR, ZMAP and vector driver ARCGEN, ArcObjects, CLOUDANT, 
COUCHDB, DB2, DODS, FME, GEOMEDIA, GTM, INGRES, MONGODB, REC, WALK at runtime, 
unless the GDAL_ENABLE_DEPRECATED_DRIVER_{drivername} configuration option is 
set to YES. Those drivers are planned for removal in GDAL 3.5
  - Perl bindings are deprecated. Removal planned for GDAL 3.5. Use 
Geo::GDAL::FFI instead
* Removal of BNA, AeronavFAA, HTF, OpenAir, SEGUKOOA, SEGY, SUA, XPlane, BPG, 
E00GRID, EPSILON, IGNFHeightASCIIGrid, NTV1 drivers. Moved to (unsupported) 
https://github.com/OSGeo/gdal-extra-drivers repository.

More complete information on the new features and fixes in the 3.3.0 release 
can be found at:

  https://github.com/OSGeo/gdal/blob/v3.3.0/gdal/NEWS

The release can be downloaded from:
  * https://download.osgeo.org/gdal/3.3.0/gdal320.zip - source as a zip
  * https://download.osgeo.org/gdal/3.3.0/gdal-3.3.0.tar.gz - source as .tar.gz
  * https://download.osgeo.org/gdal/3.3.0/gdal-3.3.0.tar.xz - source as .tar.xz
  * https://download.osgeo.org/gdal/3.3.0/gdal-grass-3.3.0.tar.gz - GDAL-GRASS 
plugin
  * https://download.osgeo.org/gdal/3.3.0/gdalautotest-3.3.0.tar.gz - test suite
  * https://download.osgeo.org/gdal/3.3.0/gdal320doc.zip - documentation / 
website

Best regards,

Even

--
http://www.spatialys.com

___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] Motion: promote GDAL 3.3.0 RC1

[Even Rouault]

Hello,


Motion:

Adopt GDAL 3.3.0 RC1 as final 3.3.0 release



Adopted with +1 from PSC members HowardB, DanielM, JukkaR and myself

Even

--
http://www.spatialys.com
My software is free, but my time generally not.

___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] Reading binary data from FileGDB

[srweal]

Hey, thanks so much for the response Paul. I feel kind of useless with this
stuff but want to get more involved so I can contribute to GDAL eventually
(where possible). 

This code looks like it'd do exactly what I need and would be great to see
it end up within the source. Is that easy to create the PR for? I have been
using the compiled Windows binaries from the GISInternals site, but as you
mention, it'll take ages for changes to filter through to those releases.

I'd really like to build this out myself from your fork, but I'm only just
learning about the GDAL source and how to build it for use in
Windows/VS2019. I have managed to get it atleast building today and compiled
a gdal303.dll file, but can't really understand how to get C# wrappers, etc.
out. 

Are you able to send through the nmake.opt file you use, or else some dot
points about the build process required to get this going within VS/C#?

Previously I've just included pre-compiled gdal_csharp.dll, ogr_csharp.dll
and osr_csharp.dll files into my applications.

Thanks again. Be well.



--
Sent from: http://osgeo-org.1560.x6.nabble.com/GDAL-Dev-f3742093.html
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev