Re: [gdal-dev] Proposal for GDAL Driver: EOPF Zarr (Earth Observation Product Format)

[Even Rouault via gdal-dev]

Yuvraj,


Having the spec publicly available seems to be a pre-condition before 
submitting a pull request implementing it, otherwise this will make 
review close to impossible.



Regarding a separate EOPF driver vs enhancing the Zarr one, I don't have 
enough elements to make a judgement. There are other options:


- the EOPF driver uses internally the Zarr one. It seems to be the most 
attractive option assuming that EOPF Zarr is valid Zarr (which I hope it is)


- if the above hypothesis isn't completely true, the Zarr driver code is 
restructured so that relevant parts can be reused by EOPF Zarr.



In any case, we want to avoid copy&paste of code between the 2 drivers 
to ease maintenance.



Even


Le 02/04/2025 à 13:50, Adagale Yuvraj Bhagwan a écrit :

Hi Even,

1. Maturity of the EOPF Enhancements
EOPF (Earth Observation Processing Framework) is still in a relatively 
early stage of development. While the core concepts have been tested 
in prototype systems, EOPF’s final production readiness is not yet 
established. We are currently refining specifications and partial read 
logic, with an aim to gradually move toward production use in the 
coming months.


2. Location of the Format Specification
The EOPF format specification is outlined in internal documentation 
maintained by the EOPF development team (originally used in ESA’s EOPF 
Sample Service activities). We plan to release a public-facing version 
of this spec once it stabilizes, so that interested developers can 
reference it directly.


3. Why Not Enhance the Existing Zarr Driver?
While the EOPF data layout leverages Zarr-like chunking, it carries 
additional, domain-specific metadata and conventions—such as 
specialized Earth Observation fields, naming conventions, and possibly 
integration with SAFE or STAC metadata. These specialized aspects 
don’t neatly map onto the existing Zarr driver’s scope.


 *
*Technical*: Enhancing the Zarr driver to interpret EOPF’s custom
logic and metadata might introduce significant complexity or break
typical Zarr assumptions.
 *
*Maintainability*: A standalone EOPF driver is less fragile, as we
can manage EOPF’s domain rules independently of any future Zarr
driver changes.
 *
*Flexibility*: This also allows the EOPF plugin to evolve at a
different pace than the generic Zarr driver.


4. Current Skeleton Status
Yes, the posted code is indeed just a skeleton right now. It 
demonstrates driver registration, basic dataset creation, and minimal 
chunk-read placeholders. This is not yet production-ready. Our 
near-term roadmap includes fleshing out actual reading of Zarr chunks, 
robust metadata handling, and multi-band support. The skeleton is a 
starting point that will expand into a more complete EOPF driver in 
subsequent development stages.


Thanks,
Yuvraj


*From:* Even Rouault 
*Sent:* Tuesday, April 1, 2025 9:41 PM
*To:* Adagale Yuvraj Bhagwan ; 
gdal-dev@lists.osgeo.org 
*Subject:* Re: [gdal-dev] Proposal for GDAL Driver: EOPF Zarr (Earth 
Observation Product Format)



You don't often get email from even.roua...@spatialys.com. Learn why 
this is important <https://aka.ms/LearnAboutSenderIdentification>



Hi Yuvraj,


A few questions:

- what is the maturity of the EOPF enhancements ?

- where are they described ? (i.e. the format specification)

- why not enhancing the existing Zarr driver ?

- from what I can currently see in 
https://github.com/EOPF-Sample-Service/GDAL-ZARR-EOPF/tree/main/src 
<https://github.com/EOPF-Sample-Service/GDAL-ZARR-EOPF/tree/main/src>, 
this seems more to be a skeleton of a driver that something production 
ready



Even



Le 01/04/2025 à 18:45, Adagale Yuvraj Bhagwan via gdal-dev a écrit :

Hello GDAL Community,
We’re developing a GDAL driver for the Earth Observation Product 
Format (EOPF), a cloud-optimized Zarr-based format tailored for 
large-scale EO data.
This driver aims to enable seamless access to EOPF datasets and their 
metadata through GDAL, supporting features like chunked I/O, and  
compatibility with STAC metadata.

Key features:
- Support for Zarr V2/V3 structures with EOPF-specific enhancements.
- Integration with cloud storage (S3, GCS, etc.).
- Alignment with ESA/Copernicus data standards.
We’d appreciate your feedback on integration requirements and best 
practices. The code is available at 
EOPF-Sample-Service/GDAL-ZARR-EOPF 
<https://github.com/EOPF-Sample-Service/GDAL-ZARR-EOPF>, and we plan 
to submit a PR soon.

Best regards,

*Yuvraj Adagale*

*Eurac Research*


*Researcher*

Institute for Earth Observation
*T* +39 344 584 4031

yuvraj.adag...@eurac.edu <mailto:yuvraj.adag...@eurac.edu>

Drususallee/Viale Druso 1

I-39100 Bozen/Bolzano

Legal Seat

Drususallee/Viale Druso 1

I-39100 Bozen/Bolzano
*_www.eurac.edu <http://www.eurac.edu/>_*

*_Facebook <http

Re: [gdal-dev] Proposal for GDAL Driver: EOPF Zarr (Earth Observation Product Format)

[Adagale Yuvraj Bhagwan via gdal-dev]

Hi Even,

1. Maturity of the EOPF Enhancements
EOPF (Earth Observation Processing Framework) is still in a relatively early 
stage of development. While the core concepts have been tested in prototype 
systems, EOPF’s final production readiness is not yet established. We are 
currently refining specifications and partial read logic, with an aim to 
gradually move toward production use in the coming months.

2. Location of the Format Specification
The EOPF format specification is outlined in internal documentation maintained 
by the EOPF development team (originally used in ESA’s EOPF Sample Service 
activities). We plan to release a public-facing version of this spec once it 
stabilizes, so that interested developers can reference it directly.

3. Why Not Enhance the Existing Zarr Driver?
While the EOPF data layout leverages Zarr-like chunking, it carries additional, 
domain-specific metadata and conventions—such as specialized Earth Observation 
fields, naming conventions, and possibly integration with SAFE or STAC 
metadata. These specialized aspects don’t neatly map onto the existing Zarr 
driver’s scope.


  *
Technical: Enhancing the Zarr driver to interpret EOPF’s custom logic and 
metadata might introduce significant complexity or break typical Zarr 
assumptions.
  *
Maintainability: A standalone EOPF driver is less fragile, as we can manage 
EOPF’s domain rules independently of any future Zarr driver changes.
  *
Flexibility: This also allows the EOPF plugin to evolve at a different pace 
than the generic Zarr driver.

4. Current Skeleton Status
Yes, the posted code is indeed just a skeleton right now. It demonstrates 
driver registration, basic dataset creation, and minimal chunk-read 
placeholders. This is not yet production-ready. Our near-term roadmap includes 
fleshing out actual reading of Zarr chunks, robust metadata handling, and 
multi-band support. The skeleton is a starting point that will expand into a 
more complete EOPF driver in subsequent development stages.

Thanks,
Yuvraj


From: Even Rouault 
Sent: Tuesday, April 1, 2025 9:41 PM
To: Adagale Yuvraj Bhagwan ; gdal-dev@lists.osgeo.org 

Subject: Re: [gdal-dev] Proposal for GDAL Driver: EOPF Zarr (Earth Observation 
Product Format)

You don't often get email from even.roua...@spatialys.com. Learn why this is 
important<https://aka.ms/LearnAboutSenderIdentification>

Hi Yuvraj,


A few questions:

- what is the maturity of the EOPF enhancements ?

- where are they described ? (i.e. the format specification)

- why not enhancing the existing Zarr driver ?

- from what I can currently see in 
https://github.com/EOPF-Sample-Service/GDAL-ZARR-EOPF/tree/main/src, this seems 
more to be a skeleton of a driver that something production ready


Even



Le 01/04/2025 à 18:45, Adagale Yuvraj Bhagwan via gdal-dev a écrit :
Hello GDAL Community,

We’re developing a GDAL driver for the Earth Observation Product Format (EOPF), 
a cloud-optimized Zarr-based format tailored for large-scale EO data.
This driver aims to enable seamless access to EOPF datasets and their metadata 
through GDAL, supporting features like chunked I/O, and  compatibility with 
STAC metadata.

Key features:
- Support for Zarr V2/V3 structures with EOPF-specific enhancements.
- Integration with cloud storage (S3, GCS, etc.).
- Alignment with ESA/Copernicus data standards.

We’d appreciate your feedback on integration requirements and best practices. 
The code is available at 
EOPF-Sample-Service/GDAL-ZARR-EOPF<https://github.com/EOPF-Sample-Service/GDAL-ZARR-EOPF>,
 and we plan to submit a PR soon.

Best regards,

Yuvraj Adagale

Eurac Research

Researcher

Institute for Earth Observation
T +39 344 584 4031

yuvraj.adag...@eurac.edu<mailto:yuvraj.adag...@eurac.edu>



Drususallee/Viale Druso 1

I-39100 Bozen/Bolzano



Legal Seat

Drususallee/Viale Druso 1

I-39100 Bozen/Bolzano
www.eurac.edu<http://www.eurac.edu/>



Facebook<https://facebook.com/eurac.research> | 
YouTube<https://www.youtube.com/EURACtv> | X<https://twitter.com/eurac> | 
LinkedIn<https://www.linkedin.com/company/euracresearch> | 
Instagram<https://www.instagram.com/euracresearch/> | CV





[signature_1401579056]<https://www.eurac.edu/en>



According to regulation (EU) 2016/679 this transmission is intended only

for the use of the addressee and may contain confidential information.

If you receive this transmission in error, please notify the sender immediately

by email and delete all copies of this message and any attachments.



Diese Nachricht ist im Sinne der Verordnung (EU) 2016/679 ausschließlich für

den Adressaten bestimmt und kann vertrauliche Informationen enthalten.

Sollten Sie diese Nachricht irrtümlich erhalten haben, bitten wir Sie, den

Absender darüber unverzüglich per E-Mail in Kenntnis zu setzen sowie die

Nachricht und etwaige Kopien und Anlagen zu vernichten.



Ai sensi del Regolamento 

Re: [gdal-dev] Proposal for GDAL Driver: EOPF Zarr (Earth Observation Product Format)

[Even Rouault via gdal-dev]

Hi Yuvraj,


A few questions:

- what is the maturity of the EOPF enhancements ?

- where are they described ? (i.e. the format specification)

- why not enhancing the existing Zarr driver ?

- from what I can currently see in 
https://github.com/EOPF-Sample-Service/GDAL-ZARR-EOPF/tree/main/src, 
this seems more to be a skeleton of a driver that something production ready



Even



Le 01/04/2025 à 18:45, Adagale Yuvraj Bhagwan via gdal-dev a écrit :

Hello GDAL Community,
We’re developing a GDAL driver for the Earth Observation Product 
Format (EOPF), a cloud-optimized Zarr-based format tailored for 
large-scale EO data.
This driver aims to enable seamless access to EOPF datasets and their 
metadata through GDAL, supporting features like chunked I/O, and  
compatibility with STAC metadata.

Key features:
- Support for Zarr V2/V3 structures with EOPF-specific enhancements.
- Integration with cloud storage (S3, GCS, etc.).
- Alignment with ESA/Copernicus data standards.
We’d appreciate your feedback on integration requirements and best 
practices. The code is available at EOPF-Sample-Service/GDAL-ZARR-EOPF 
, and we plan 
to submit a PR soon.

Best regards,

*Yuvraj Adagale*

*Eurac Research*


*Researcher*

Institute for Earth Observation
*T* +39 344 584 4031

yuvraj.adag...@eurac.edu

Drususallee/Viale Druso 1

I-39100 Bozen/Bolzano

Legal Seat

Drususallee/Viale Druso 1

I-39100 Bozen/Bolzano
*_www.eurac.edu _*

*_Facebook _ | _YouTube 
_ | _X _ | 
_LinkedIn _ | 
_Instagram _| CV*


_signature_1401579056 _

According to regulation (EU) 2016/679 this transmission is intended only

for the use of the addressee and may contain confidential information.

If you receive this transmission in error, please notify the sender 
immediately


by email and delete all copies of this message and any attachments.

Diese Nachricht ist im Sinne der Verordnung (EU) 2016/679 
ausschließlich für


den Adressaten bestimmt und kann vertrauliche Informationen enthalten.

Sollten Sie diese Nachricht irrtümlich erhalten haben, bitten wir Sie, den

Absender darüber unverzüglich per E-Mail in Kenntnis zu setzen sowie die

Nachricht und etwaige Kopien und Anlagen zu vernichten.

Ai sensi del Regolamento UE 679/2016 questo messaggio è ad uso esclusivo

del destinatario e può contenere informazioni riservate. Qualora Le fosse

pervenuto per errore, Le chiediamo gentilmente di comunicarcelo 


immediatamente via e-mail ed eliminare qualsiasi copia e allegato.



___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev


--
http://www.spatialys.com
My software is free, but my time generally not.
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev