Re: [gdal-dev] core dump on dir info

2024-02-06 Thread Michael Sumner via gdal-dev 
I don't understand how jammy is "old" when the full build is itself using
"BASE_IMAGE=ubuntu:22.04"

But, I'm out of my depth in these emails and trying to learn, thanks!

Cheers, Mike




On Wed, 7 Feb 2024, 00:46 Javier Jimenez Shaw,  wrote:

> Could you set up your VMs to include those SSE instructions? I think that
> keeping VMs that "old" configured is a source of problems using
> pre-compiled binaries.
> The same way GDAL updates dependencies of compilers and other libraries to
> something more modern (but not too modern), those SSE instructions should
> be updated.
>
> @Even knowing now that the "old" hardware is "virtually old", should we
> remove AVX2 compatibility from ubuntu-full-latest? I do not know how much
> is the performance impact.
>
> Cheers,
> Javier.
>
> On Mon, 5 Feb 2024 at 21:54, Michael Sumner  wrote:
>
>> yes, jammy VM on openstack is the host (and is where I run pretty much
>> everything, though will increasingly use AWS).
>>
>> Thanks for the note, I'll try on other systems too. We need a
>> security-allow set for vsicurl to work so if there are other little details
>> I'll be keen to flush them out.
>>
>> Cheers, Mike
>>
>> On Mon, 5 Feb 2024, 18:44 Javier Jimenez Shaw, 
>> wrote:
>>
>>> Hi Mike
>>>
>>> Out of curiosity, are run running it in a virtual machine?
>>> A few year ago I had problems running a program in a virtual machine
>>> (virtualbox, but I read it happens in others) due to a missing SSE
>>> instruction. The solution there was to "enable" the missing instructions in
>>> the virtual machine configuration (that I don't know why it was not the
>>> default).
>>>
>>> Cheers
>>>
>>>
>>> On Mon, 5 Feb 2024, 02:04 Even Rouault via gdal-dev, <
>>> gdal-dev@lists.osgeo.org> wrote:
>>>
 ghcr.io/osgeo/gdal:ubuntu-full-latest has been regenerated with the
 rebuild of TileDB without AVX2. I've also enabled the
 drivers-with-external-depencies-built-as-plugin GDAL build mode, so it is
 easy to just remove a given plugin by deleting the corresponding .so in
 /usr/lib/x86_64-linux-gnu/gdalplugins

 Even
 Le 04/02/2024 à 22:51, Michael Sumner a écrit :

 indeed there's no avx2:

 cat /proc/cpuinfo|grep sse|head -n 1
 flags   : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge
 mca cmov pat pse36 clflush mmx fxsr sse sse2 syscall nx mmxext fxsr_opt
 pdpe1gb rdtscp lm rep_good nopl cpuid extd_apicid tsc_known_freq pni
 pclmulqdq ssse3 fma cx16 sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes
 xsave avx f16c hypervisor lahf_lm cmp_legacy svm cr8_legacy abm sse4a
 misalignsse 3dnowprefetch osvw xop fma4 tbm perfctr_core ssbd ibpb vmmcall
 tsc_adjust bmi1 virt_ssbd arat npt nrip_save arch_capabilities

 Cheers, Mike


___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] core dump on dir info

2024-02-06 Thread Laurențiu Nicola via gdal-dev 
On Tue, Feb 6, 2024, at 16:04, Even Rouault via gdal-dev wrote:
> Le 06/02/2024 à 14:46, Javier Jimenez Shaw a écrit :
> I remember older VirtualBox versions indeed had issues to propage AVX to 
> guest, but I believed that has been fixed now. I'm not aware of issues 
> with SSEx not being propagate.

Not the same thing, but I've seen a VM where loading numpy was crashing because 
of a misreported CPU model. OpenBLAS got "AMD Opteron 63xx class CPU" from 
CPUID and assumed FMA4 support without actually doing feature detection. But 
the CPU was actually an EPYC, which have no FMA4. I filed an issue against 
OpenBLAS, but they weren't interested in fixing it.

So it's still possible to get a misconfigured VM, especially if you're running 
an older libvirt. This was also on OpenStack AFAIK.

Laurentiu
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] core dump on dir info

2024-02-06 Thread Even Rouault via gdal-dev 


Le 06/02/2024 à 14:46, Javier Jimenez Shaw a écrit :

Could you set up your VMs to include those SSE instructions?
I remember older VirtualBox versions indeed had issues to propage AVX to 
guest, but I believed that has been fixed now. I'm not aware of issues 
with SSEx not being propagate.
I think that keeping VMs that "old" configured is a source of problems 
using pre-compiled binaries.
The same way GDAL updates dependencies of compilers and other 
libraries to something more modern (but not too modern), those SSE 
instructions should be updated.


@Even knowing now that the "old" hardware is "virtually old", should 
we remove AVX2 compatibility from ubuntu-full-latest? I do not know 
how much is the performance impact.


Well, GDAL itself is built with default compiler flags, so only SSE2 on 
x86_64 (at least currently, but there seems to be a move by some distros 
to switch to SSE4.2 as the baseline)  (some parts of GDAL can use higher 
SSE or AVX instruction sets, but they only do so by checking first 
runtime CPU capabilities)


The tiledb issue was unfortunate in that its default config enables AVX2 
and you have to explicitly disable it. On the ubuntu-full image, we also 
use the Apache .deb for libarrow, and I believe they use SSE4.2.


Even


--
http://www.spatialys.com
My software is free, but my time generally not.

___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] core dump on dir info

2024-02-06 Thread Javier Jimenez Shaw via gdal-dev 
Could you set up your VMs to include those SSE instructions? I think that
keeping VMs that "old" configured is a source of problems using
pre-compiled binaries.
The same way GDAL updates dependencies of compilers and other libraries to
something more modern (but not too modern), those SSE instructions should
be updated.

@Even knowing now that the "old" hardware is "virtually old", should we
remove AVX2 compatibility from ubuntu-full-latest? I do not know how much
is the performance impact.

Cheers,
Javier.

On Mon, 5 Feb 2024 at 21:54, Michael Sumner  wrote:

> yes, jammy VM on openstack is the host (and is where I run pretty much
> everything, though will increasingly use AWS).
>
> Thanks for the note, I'll try on other systems too. We need a
> security-allow set for vsicurl to work so if there are other little details
> I'll be keen to flush them out.
>
> Cheers, Mike
>
> On Mon, 5 Feb 2024, 18:44 Javier Jimenez Shaw,  wrote:
>
>> Hi Mike
>>
>> Out of curiosity, are run running it in a virtual machine?
>> A few year ago I had problems running a program in a virtual machine
>> (virtualbox, but I read it happens in others) due to a missing SSE
>> instruction. The solution there was to "enable" the missing instructions in
>> the virtual machine configuration (that I don't know why it was not the
>> default).
>>
>> Cheers
>>
>>
>> On Mon, 5 Feb 2024, 02:04 Even Rouault via gdal-dev, <
>> gdal-dev@lists.osgeo.org> wrote:
>>
>>> ghcr.io/osgeo/gdal:ubuntu-full-latest has been regenerated with the
>>> rebuild of TileDB without AVX2. I've also enabled the
>>> drivers-with-external-depencies-built-as-plugin GDAL build mode, so it is
>>> easy to just remove a given plugin by deleting the corresponding .so in
>>> /usr/lib/x86_64-linux-gnu/gdalplugins
>>>
>>> Even
>>> Le 04/02/2024 à 22:51, Michael Sumner a écrit :
>>>
>>> indeed there's no avx2:
>>>
>>> cat /proc/cpuinfo|grep sse|head -n 1
>>> flags   : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge
>>> mca cmov pat pse36 clflush mmx fxsr sse sse2 syscall nx mmxext fxsr_opt
>>> pdpe1gb rdtscp lm rep_good nopl cpuid extd_apicid tsc_known_freq pni
>>> pclmulqdq ssse3 fma cx16 sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes
>>> xsave avx f16c hypervisor lahf_lm cmp_legacy svm cr8_legacy abm sse4a
>>> misalignsse 3dnowprefetch osvw xop fma4 tbm perfctr_core ssbd ibpb vmmcall
>>> tsc_adjust bmi1 virt_ssbd arat npt nrip_save arch_capabilities
>>>
>>> Cheers, Mike
>>>
>>>
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] core dump on dir info

2024-02-05 Thread Michael Sumner via gdal-dev 
yes, jammy VM on openstack is the host (and is where I run pretty much
everything, though will increasingly use AWS).

Thanks for the note, I'll try on other systems too. We need a
security-allow set for vsicurl to work so if there are other little details
I'll be keen to flush them out.

Cheers, Mike

On Mon, 5 Feb 2024, 18:44 Javier Jimenez Shaw,  wrote:

> Hi Mike
>
> Out of curiosity, are run running it in a virtual machine?
> A few year ago I had problems running a program in a virtual machine
> (virtualbox, but I read it happens in others) due to a missing SSE
> instruction. The solution there was to "enable" the missing instructions in
> the virtual machine configuration (that I don't know why it was not the
> default).
>
> Cheers
>
>
> On Mon, 5 Feb 2024, 02:04 Even Rouault via gdal-dev, <
> gdal-dev@lists.osgeo.org> wrote:
>
>> ghcr.io/osgeo/gdal:ubuntu-full-latest has been regenerated with the
>> rebuild of TileDB without AVX2. I've also enabled the
>> drivers-with-external-depencies-built-as-plugin GDAL build mode, so it is
>> easy to just remove a given plugin by deleting the corresponding .so in
>> /usr/lib/x86_64-linux-gnu/gdalplugins
>>
>> Even
>> Le 04/02/2024 à 22:51, Michael Sumner a écrit :
>>
>> indeed there's no avx2:
>>
>> cat /proc/cpuinfo|grep sse|head -n 1
>> flags   : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge
>> mca cmov pat pse36 clflush mmx fxsr sse sse2 syscall nx mmxext fxsr_opt
>> pdpe1gb rdtscp lm rep_good nopl cpuid extd_apicid tsc_known_freq pni
>> pclmulqdq ssse3 fma cx16 sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes
>> xsave avx f16c hypervisor lahf_lm cmp_legacy svm cr8_legacy abm sse4a
>> misalignsse 3dnowprefetch osvw xop fma4 tbm perfctr_core ssbd ibpb vmmcall
>> tsc_adjust bmi1 virt_ssbd arat npt nrip_save arch_capabilities
>>
>> Cheers, Mike
>>
>>
>>
>> On Sun, Feb 4, 2024 at 10:55 PM Even Rouault 
>> wrote:
>>
>>> ok, so I believe this is the AVX2 issue I was talking about, as I
>>> realize that enabling AVX2 is the default mode when TileDB is built from
>>> source (which the Docker image does), and must be explicitly disabled with
>>> "./bootstrap --disable-avx2" (I've just changed the build recipe to include
>>> that, will take effect next time the images are refreshed)
>>>
>>> To confirm, can you send or just check the output of : cat
>>> /proc/cpuinfo|grep sse|head -n 1
>>>
>>> If there is no "avx2" in it, this is at 99.9% the reason of the issue.
>>>
>>> Even
>>> Le 04/02/2024 à 06:20, Michael Sumner a écrit :
>>>
>>> skipping TileDB does fix:
>>>
>>> ogr2ogr /tmp/newdir
>>> https://github.com/SymbolixAU/geojsonsf/raw/master/inst/examples/geo_melbourne.geojson
>>>  -f
>>> "ESRI Shapefile"
>>> export GDAL_SKIP=TileDB
>>> ogrinfo /tmp/newdir/
>>> INFO: Open of `/tmp/newdir/'
>>>   using driver `ESRI Shapefile' successful.
>>> 1: geo_melbourne (Polygon)
>>>
>>> unset GDAL_SKIP
>>> ogrinfo /tmp/newdir/
>>> Illegal instruction (core dumped)
>>>
>>> I failed to explain that I'm using gdal containers from the repo:
>>>
>>> docker run --rm -ti ghcr.io/osgeo/gdal:ubunt
>>>
>>> u-full-latest
>>>
>>> apt update
>>> apt install -y gdb
>>>
>>> Here's the output of under gdb as you suggested, there was a lot so I
>>> put it on a gist:
>>> https://gist.github.com/mdsumner/839ae6e05ededf640b65bfee3a20a4c0
>>>
>>> gdb --args ogrinfo /tmp/newdir/
>>> > run
>>> > thread apply all bt
>>>
>>> Thanks!
>>>
>>>
>>>
>>>
>>>
>>> On Sat, Feb 3, 2024 at 7:49 PM Even Rouault 
>>> wrote:
>>>
 - When it crashes under gdb, type "thread apply all bt" to get the
 stack trace of all threads

 - I suspect there is a connection with
 https://github.com/OSGeo/gdal/pull/9170 , but that pull request
 wouldn't help here as "/tmp/newdir" could be a valid connection to TileDB

 - how did you get TileDB installed? It looks to be packaged? Which
 distribution do you use?

 - SIGILL reminds me of issues with some TileDB builds using the AVX2
 instruction set by default, which could cause some crash on host CPUs that
 don't have AVX2 (unlikely on recent hardware though)

 - Setting GDAL_SKIP=TileDB should be a workaround


 Le 03/02/2024 à 07:15, Michael Sumner a écrit :

 Thanks Even, so there's something about tiledb under gdb (or maybe I am
 mangling the context,  I will try variants of the host I'm using). Run with
 valgrind included below.

 gdb --args ogrinfo /tmp/newdir/
 ...
 (gdb) run
 Starting program: /usr/local/bin/ogrinfo /tmp/newdir/
 [Thread debugging using libthread_db enabled]
 Using host libthread_db library
 "/lib/x86_64-linux-gnu/libthread_db.so.1".
 [New Thread 0x7fffe7757640 (LWP 988)]
 [New Thread 0x7fffe6f56640 (LWP 989)]
 [New Thread 0x7fffde755640 (LWP 990)]
 [New Thread 0x7fffd5f54640 (LWP 991)]
 [New Thread 0x7fffc5753640 (LWP 992)]
 [New Thread 0x7fffc4f52640 (LWP 993)]
 [New Thread 0x7fffb4751640 (LWP 9

Re: [gdal-dev] core dump on dir info

2024-02-04 Thread Javier Jimenez Shaw via gdal-dev 
Hi Mike

Out of curiosity, are run running it in a virtual machine?
A few year ago I had problems running a program in a virtual machine
(virtualbox, but I read it happens in others) due to a missing SSE
instruction. The solution there was to "enable" the missing instructions in
the virtual machine configuration (that I don't know why it was not the
default).

Cheers


On Mon, 5 Feb 2024, 02:04 Even Rouault via gdal-dev, <
gdal-dev@lists.osgeo.org> wrote:

> ghcr.io/osgeo/gdal:ubuntu-full-latest has been regenerated with the
> rebuild of TileDB without AVX2. I've also enabled the
> drivers-with-external-depencies-built-as-plugin GDAL build mode, so it is
> easy to just remove a given plugin by deleting the corresponding .so in
> /usr/lib/x86_64-linux-gnu/gdalplugins
>
> Even
> Le 04/02/2024 à 22:51, Michael Sumner a écrit :
>
> indeed there's no avx2:
>
> cat /proc/cpuinfo|grep sse|head -n 1
> flags   : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca
> cmov pat pse36 clflush mmx fxsr sse sse2 syscall nx mmxext fxsr_opt pdpe1gb
> rdtscp lm rep_good nopl cpuid extd_apicid tsc_known_freq pni pclmulqdq
> ssse3 fma cx16 sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes xsave avx
> f16c hypervisor lahf_lm cmp_legacy svm cr8_legacy abm sse4a misalignsse
> 3dnowprefetch osvw xop fma4 tbm perfctr_core ssbd ibpb vmmcall tsc_adjust
> bmi1 virt_ssbd arat npt nrip_save arch_capabilities
>
> Cheers, Mike
>
>
>
> On Sun, Feb 4, 2024 at 10:55 PM Even Rouault 
> wrote:
>
>> ok, so I believe this is the AVX2 issue I was talking about, as I realize
>> that enabling AVX2 is the default mode when TileDB is built from source
>> (which the Docker image does), and must be explicitly disabled with
>> "./bootstrap --disable-avx2" (I've just changed the build recipe to include
>> that, will take effect next time the images are refreshed)
>>
>> To confirm, can you send or just check the output of : cat
>> /proc/cpuinfo|grep sse|head -n 1
>>
>> If there is no "avx2" in it, this is at 99.9% the reason of the issue.
>>
>> Even
>> Le 04/02/2024 à 06:20, Michael Sumner a écrit :
>>
>> skipping TileDB does fix:
>>
>> ogr2ogr /tmp/newdir
>> https://github.com/SymbolixAU/geojsonsf/raw/master/inst/examples/geo_melbourne.geojson
>>  -f
>> "ESRI Shapefile"
>> export GDAL_SKIP=TileDB
>> ogrinfo /tmp/newdir/
>> INFO: Open of `/tmp/newdir/'
>>   using driver `ESRI Shapefile' successful.
>> 1: geo_melbourne (Polygon)
>>
>> unset GDAL_SKIP
>> ogrinfo /tmp/newdir/
>> Illegal instruction (core dumped)
>>
>> I failed to explain that I'm using gdal containers from the repo:
>>
>> docker run --rm -ti ghcr.io/osgeo/gdal:ubunt
>>
>> u-full-latest
>>
>> apt update
>> apt install -y gdb
>>
>> Here's the output of under gdb as you suggested, there was a lot so I put
>> it on a gist:
>> https://gist.github.com/mdsumner/839ae6e05ededf640b65bfee3a20a4c0
>>
>> gdb --args ogrinfo /tmp/newdir/
>> > run
>> > thread apply all bt
>>
>> Thanks!
>>
>>
>>
>>
>>
>> On Sat, Feb 3, 2024 at 7:49 PM Even Rouault 
>> wrote:
>>
>>> - When it crashes under gdb, type "thread apply all bt" to get the stack
>>> trace of all threads
>>>
>>> - I suspect there is a connection with
>>> https://github.com/OSGeo/gdal/pull/9170 , but that pull request
>>> wouldn't help here as "/tmp/newdir" could be a valid connection to TileDB
>>>
>>> - how did you get TileDB installed? It looks to be packaged? Which
>>> distribution do you use?
>>>
>>> - SIGILL reminds me of issues with some TileDB builds using the AVX2
>>> instruction set by default, which could cause some crash on host CPUs that
>>> don't have AVX2 (unlikely on recent hardware though)
>>>
>>> - Setting GDAL_SKIP=TileDB should be a workaround
>>>
>>>
>>> Le 03/02/2024 à 07:15, Michael Sumner a écrit :
>>>
>>> Thanks Even, so there's something about tiledb under gdb (or maybe I am
>>> mangling the context,  I will try variants of the host I'm using). Run with
>>> valgrind included below.
>>>
>>> gdb --args ogrinfo /tmp/newdir/
>>> ...
>>> (gdb) run
>>> Starting program: /usr/local/bin/ogrinfo /tmp/newdir/
>>> [Thread debugging using libthread_db enabled]
>>> Using host libthread_db library
>>> "/lib/x86_64-linux-gnu/libthread_db.so.1".
>>> [New Thread 0x7fffe7757640 (LWP 988)]
>>> [New Thread 0x7fffe6f56640 (LWP 989)]
>>> [New Thread 0x7fffde755640 (LWP 990)]
>>> [New Thread 0x7fffd5f54640 (LWP 991)]
>>> [New Thread 0x7fffc5753640 (LWP 992)]
>>> [New Thread 0x7fffc4f52640 (LWP 993)]
>>> [New Thread 0x7fffb4751640 (LWP 994)]
>>> [New Thread 0x7fffabf50640 (LWP 995)]
>>> [New Thread 0x7fffab74f640 (LWP 996)]
>>> [New Thread 0x7fffa2f4e640 (LWP 997)]
>>> [New Thread 0x7fff9a74d640 (LWP 998)]
>>> [New Thread 0x7fff91f4c640 (LWP 999)]
>>> [New Thread 0x7fff8974b640 (LWP 1000)]
>>> [New Thread 0x7fff78f4a640 (LWP 1001)]
>>> [New Thread 0x7fff78749640 (LWP 1002)]
>>> [New Thread 0x7fff6f5ff640 (LWP 1003)]
>>>
>>> Thread 1 "ogrinfo" received signal SIGILL, Illegal instruction.
>>> 0x73773c9e in tiledb::common::Thre

Re: [gdal-dev] core dump on dir info

2024-02-04 Thread Michael Sumner via gdal-dev 
Excellent, thanks Even - confirmed at my end and on my original use-case.

Cheers, Mike



On Mon, Feb 5, 2024 at 12:04 PM Even Rouault 
wrote:

> ghcr.io/osgeo/gdal:ubuntu-full-latest has been regenerated with the
> rebuild of TileDB without AVX2. I've also enabled the
> drivers-with-external-depencies-built-as-plugin GDAL build mode, so it is
> easy to just remove a given plugin by deleting the corresponding .so in
> /usr/lib/x86_64-linux-gnu/gdalplugins
>
> Even
> Le 04/02/2024 à 22:51, Michael Sumner a écrit :
>
> indeed there's no avx2:
>
> cat /proc/cpuinfo|grep sse|head -n 1
> flags   : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca
> cmov pat pse36 clflush mmx fxsr sse sse2 syscall nx mmxext fxsr_opt pdpe1gb
> rdtscp lm rep_good nopl cpuid extd_apicid tsc_known_freq pni pclmulqdq
> ssse3 fma cx16 sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes xsave avx
> f16c hypervisor lahf_lm cmp_legacy svm cr8_legacy abm sse4a misalignsse
> 3dnowprefetch osvw xop fma4 tbm perfctr_core ssbd ibpb vmmcall tsc_adjust
> bmi1 virt_ssbd arat npt nrip_save arch_capabilities
>
> Cheers, Mike
>
>
>
> On Sun, Feb 4, 2024 at 10:55 PM Even Rouault 
> wrote:
>
>> ok, so I believe this is the AVX2 issue I was talking about, as I realize
>> that enabling AVX2 is the default mode when TileDB is built from source
>> (which the Docker image does), and must be explicitly disabled with
>> "./bootstrap --disable-avx2" (I've just changed the build recipe to include
>> that, will take effect next time the images are refreshed)
>>
>> To confirm, can you send or just check the output of : cat
>> /proc/cpuinfo|grep sse|head -n 1
>>
>> If there is no "avx2" in it, this is at 99.9% the reason of the issue.
>>
>> Even
>> Le 04/02/2024 à 06:20, Michael Sumner a écrit :
>>
>> skipping TileDB does fix:
>>
>> ogr2ogr /tmp/newdir
>> https://github.com/SymbolixAU/geojsonsf/raw/master/inst/examples/geo_melbourne.geojson
>>  -f
>> "ESRI Shapefile"
>> export GDAL_SKIP=TileDB
>> ogrinfo /tmp/newdir/
>> INFO: Open of `/tmp/newdir/'
>>   using driver `ESRI Shapefile' successful.
>> 1: geo_melbourne (Polygon)
>>
>> unset GDAL_SKIP
>> ogrinfo /tmp/newdir/
>> Illegal instruction (core dumped)
>>
>> I failed to explain that I'm using gdal containers from the repo:
>>
>> docker run --rm -ti ghcr.io/osgeo/gdal:ubunt
>>
>> u-full-latest
>>
>> apt update
>> apt install -y gdb
>>
>> Here's the output of under gdb as you suggested, there was a lot so I put
>> it on a gist:
>> https://gist.github.com/mdsumner/839ae6e05ededf640b65bfee3a20a4c0
>>
>> gdb --args ogrinfo /tmp/newdir/
>> > run
>> > thread apply all bt
>>
>> Thanks!
>>
>>
>>
>>
>>
>> On Sat, Feb 3, 2024 at 7:49 PM Even Rouault 
>> wrote:
>>
>>> - When it crashes under gdb, type "thread apply all bt" to get the stack
>>> trace of all threads
>>>
>>> - I suspect there is a connection with
>>> https://github.com/OSGeo/gdal/pull/9170 , but that pull request
>>> wouldn't help here as "/tmp/newdir" could be a valid connection to TileDB
>>>
>>> - how did you get TileDB installed? It looks to be packaged? Which
>>> distribution do you use?
>>>
>>> - SIGILL reminds me of issues with some TileDB builds using the AVX2
>>> instruction set by default, which could cause some crash on host CPUs that
>>> don't have AVX2 (unlikely on recent hardware though)
>>>
>>> - Setting GDAL_SKIP=TileDB should be a workaround
>>>
>>>
>>> Le 03/02/2024 à 07:15, Michael Sumner a écrit :
>>>
>>> Thanks Even, so there's something about tiledb under gdb (or maybe I am
>>> mangling the context,  I will try variants of the host I'm using). Run with
>>> valgrind included below.
>>>
>>> gdb --args ogrinfo /tmp/newdir/
>>> ...
>>> (gdb) run
>>> Starting program: /usr/local/bin/ogrinfo /tmp/newdir/
>>> [Thread debugging using libthread_db enabled]
>>> Using host libthread_db library
>>> "/lib/x86_64-linux-gnu/libthread_db.so.1".
>>> [New Thread 0x7fffe7757640 (LWP 988)]
>>> [New Thread 0x7fffe6f56640 (LWP 989)]
>>> [New Thread 0x7fffde755640 (LWP 990)]
>>> [New Thread 0x7fffd5f54640 (LWP 991)]
>>> [New Thread 0x7fffc5753640 (LWP 992)]
>>> [New Thread 0x7fffc4f52640 (LWP 993)]
>>> [New Thread 0x7fffb4751640 (LWP 994)]
>>> [New Thread 0x7fffabf50640 (LWP 995)]
>>> [New Thread 0x7fffab74f640 (LWP 996)]
>>> [New Thread 0x7fffa2f4e640 (LWP 997)]
>>> [New Thread 0x7fff9a74d640 (LWP 998)]
>>> [New Thread 0x7fff91f4c640 (LWP 999)]
>>> [New Thread 0x7fff8974b640 (LWP 1000)]
>>> [New Thread 0x7fff78f4a640 (LWP 1001)]
>>> [New Thread 0x7fff78749640 (LWP 1002)]
>>> [New Thread 0x7fff6f5ff640 (LWP 1003)]
>>>
>>> Thread 1 "ogrinfo" received signal SIGILL, Illegal instruction.
>>> 0x73773c9e in tiledb::common::ThreadPool::ThreadPool(unsigned
>>> long) () from /lib/x86_64-linux-gnu/libtiledb.so.2.16
>>>
>>>
>>>
>>>
>>> valgrind -s ogrinfo /tmp/newdir
>>> ==704== Memcheck, a memory error detector
>>> ==704== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
>>> ==704== Using Valgrind-3.18.1 and LibVEX; rerun with

Re: [gdal-dev] core dump on dir info

2024-02-04 Thread Even Rouault via gdal-dev 
ghcr.io/osgeo/gdal:ubuntu-full-latest has been regenerated with the 
rebuild of TileDB without AVX2. I've also enabled the 
drivers-with-external-depencies-built-as-plugin GDAL build mode, so it 
is easy to just remove a given plugin by deleting the corresponding .so 
in /usr/lib/x86_64-linux-gnu/gdalplugins


Even

Le 04/02/2024 à 22:51, Michael Sumner a écrit :

indeed there's no avx2:

cat /proc/cpuinfo|grep sse|head -n 1
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge 
mca cmov pat pse36 clflush mmx fxsr sse sse2 syscall nx mmxext 
fxsr_opt pdpe1gb rdtscp lm rep_good nopl cpuid extd_apicid 
tsc_known_freq pni pclmulqdq ssse3 fma cx16 sse4_1 sse4_2 x2apic 
popcnt tsc_deadline_timer aes xsave avx f16c hypervisor lahf_lm 
cmp_legacy svm cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw xop 
fma4 tbm perfctr_core ssbd ibpb vmmcall tsc_adjust bmi1 virt_ssbd arat 
npt nrip_save arch_capabilities


Cheers, Mike



On Sun, Feb 4, 2024 at 10:55 PM Even Rouault 
 wrote:


ok, so I believe this is the AVX2 issue I was talking about, as I
realize that enabling AVX2 is the default mode when TileDB is
built from source (which the Docker image does), and must be
explicitly disabled with "./bootstrap --disable-avx2" (I've just
changed the build recipe to include that, will take effect next
time the images are refreshed)

To confirm, can you send or just check the output of : cat
/proc/cpuinfo|grep sse|head -n 1

If there is no "avx2" in it, this is at 99.9% the reason of the issue.

Even

Le 04/02/2024 à 06:20, Michael Sumner a écrit :

skipping TileDB does fix:

ogr2ogr /tmp/newdir

https://github.com/SymbolixAU/geojsonsf/raw/master/inst/examples/geo_melbourne.geojson
 -f
"ESRI Shapefile"
export GDAL_SKIP=TileDB
ogrinfo /tmp/newdir/
INFO: Open of `/tmp/newdir/'
      using driver `ESRI Shapefile' successful.
1: geo_melbourne (Polygon)

unset GDAL_SKIP
ogrinfo /tmp/newdir/
Illegal instruction (core dumped)

I failed to explain that I'm using gdal containers from the repo:

docker run --rm -ti ghcr.io/osgeo/gdal:ubunt


u-full-latest

apt update
apt install -y gdb

Here's the output of under gdb as you suggested, there was a lot
so I put it on a gist:
https://gist.github.com/mdsumner/839ae6e05ededf640b65bfee3a20a4c0

gdb --args ogrinfo /tmp/newdir/
> run
> thread apply all bt

Thanks!





On Sat, Feb 3, 2024 at 7:49 PM Even Rouault
 wrote:

- When it crashes under gdb, type "thread apply all bt" to
get the stack trace of all threads

- I suspect there is a connection with
https://github.com/OSGeo/gdal/pull/9170 , but that pull
request wouldn't help here as "/tmp/newdir" could be a valid
connection to TileDB

- how did you get TileDB installed? It looks to be packaged?
Which distribution do you use?

- SIGILL reminds me of issues with some TileDB builds using
the AVX2 instruction set by default, which could cause some
crash on host CPUs that don't have AVX2 (unlikely on recent
hardware though)

- Setting GDAL_SKIP=TileDB should be a workaround


Le 03/02/2024 à 07:15, Michael Sumner a écrit :

Thanks Even, so there's something about tiledb under gdb (or
maybe I am mangling the context,  I will try variants of the
host I'm using). Run with valgrind included below.

gdb --args ogrinfo /tmp/newdir/
...
(gdb) run
Starting program: /usr/local/bin/ogrinfo /tmp/newdir/
[Thread debugging using libthread_db enabled]
Using host libthread_db library
"/lib/x86_64-linux-gnu/libthread_db.so.1".
[New Thread 0x7fffe7757640 (LWP 988)]
[New Thread 0x7fffe6f56640 (LWP 989)]
[New Thread 0x7fffde755640 (LWP 990)]
[New Thread 0x7fffd5f54640 (LWP 991)]
[New Thread 0x7fffc5753640 (LWP 992)]
[New Thread 0x7fffc4f52640 (LWP 993)]
[New Thread 0x7fffb4751640 (LWP 994)]
[New Thread 0x7fffabf50640 (LWP 995)]
[New Thread 0x7fffab74f640 (LWP 996)]
[New Thread 0x7fffa2f4e640 (LWP 997)]
[New Thread 0x7fff9a74d640 (LWP 998)]
[New Thread 0x7fff91f4c640 (LWP 999)]
[New Thread 0x7fff8974b640 (LWP 1000)]
[New Thread 0x7fff78f4a640 (LWP 1001)]
[New Thread 0x7fff78749640 (LWP 1002)]
[New Thread 0x7fff6f5ff640 (LWP 1003)]

Thread 1 "ogrinfo" received signal SIGILL, Illegal instruction.
0x73773c9e in
tiledb::common::ThreadPool::ThreadPool(unsigned long) ()
from /lib/x86_64-linux-gnu/libtiledb.so.2.16




valgrind -s ogrinfo /tmp/newdir
==704== Memcheck, a memory error detector
==704== Copyright (C) 2002-2017, and GNU GPL'd, by Julian
Seward et al.
==704== Using V

Re: [gdal-dev] core dump on dir info

2024-02-04 Thread Michael Sumner via gdal-dev 
indeed there's no avx2:

cat /proc/cpuinfo|grep sse|head -n 1
flags   : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca
cmov pat pse36 clflush mmx fxsr sse sse2 syscall nx mmxext fxsr_opt pdpe1gb
rdtscp lm rep_good nopl cpuid extd_apicid tsc_known_freq pni pclmulqdq
ssse3 fma cx16 sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes xsave avx
f16c hypervisor lahf_lm cmp_legacy svm cr8_legacy abm sse4a misalignsse
3dnowprefetch osvw xop fma4 tbm perfctr_core ssbd ibpb vmmcall tsc_adjust
bmi1 virt_ssbd arat npt nrip_save arch_capabilities

Cheers, Mike



On Sun, Feb 4, 2024 at 10:55 PM Even Rouault 
wrote:

> ok, so I believe this is the AVX2 issue I was talking about, as I realize
> that enabling AVX2 is the default mode when TileDB is built from source
> (which the Docker image does), and must be explicitly disabled with
> "./bootstrap --disable-avx2" (I've just changed the build recipe to include
> that, will take effect next time the images are refreshed)
>
> To confirm, can you send or just check the output of : cat
> /proc/cpuinfo|grep sse|head -n 1
>
> If there is no "avx2" in it, this is at 99.9% the reason of the issue.
>
> Even
> Le 04/02/2024 à 06:20, Michael Sumner a écrit :
>
> skipping TileDB does fix:
>
> ogr2ogr /tmp/newdir
> https://github.com/SymbolixAU/geojsonsf/raw/master/inst/examples/geo_melbourne.geojson
>  -f
> "ESRI Shapefile"
> export GDAL_SKIP=TileDB
> ogrinfo /tmp/newdir/
> INFO: Open of `/tmp/newdir/'
>   using driver `ESRI Shapefile' successful.
> 1: geo_melbourne (Polygon)
>
> unset GDAL_SKIP
> ogrinfo /tmp/newdir/
> Illegal instruction (core dumped)
>
> I failed to explain that I'm using gdal containers from the repo:
>
> docker run --rm -ti ghcr.io/osgeo/gdal:ubunt
>
> u-full-latest
>
> apt update
> apt install -y gdb
>
> Here's the output of under gdb as you suggested, there was a lot so I put
> it on a gist:
> https://gist.github.com/mdsumner/839ae6e05ededf640b65bfee3a20a4c0
>
> gdb --args ogrinfo /tmp/newdir/
> > run
> > thread apply all bt
>
> Thanks!
>
>
>
>
>
> On Sat, Feb 3, 2024 at 7:49 PM Even Rouault 
> wrote:
>
>> - When it crashes under gdb, type "thread apply all bt" to get the stack
>> trace of all threads
>>
>> - I suspect there is a connection with
>> https://github.com/OSGeo/gdal/pull/9170 , but that pull request wouldn't
>> help here as "/tmp/newdir" could be a valid connection to TileDB
>>
>> - how did you get TileDB installed? It looks to be packaged? Which
>> distribution do you use?
>>
>> - SIGILL reminds me of issues with some TileDB builds using the AVX2
>> instruction set by default, which could cause some crash on host CPUs that
>> don't have AVX2 (unlikely on recent hardware though)
>>
>> - Setting GDAL_SKIP=TileDB should be a workaround
>>
>>
>> Le 03/02/2024 à 07:15, Michael Sumner a écrit :
>>
>> Thanks Even, so there's something about tiledb under gdb (or maybe I am
>> mangling the context,  I will try variants of the host I'm using). Run with
>> valgrind included below.
>>
>> gdb --args ogrinfo /tmp/newdir/
>> ...
>> (gdb) run
>> Starting program: /usr/local/bin/ogrinfo /tmp/newdir/
>> [Thread debugging using libthread_db enabled]
>> Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
>> [New Thread 0x7fffe7757640 (LWP 988)]
>> [New Thread 0x7fffe6f56640 (LWP 989)]
>> [New Thread 0x7fffde755640 (LWP 990)]
>> [New Thread 0x7fffd5f54640 (LWP 991)]
>> [New Thread 0x7fffc5753640 (LWP 992)]
>> [New Thread 0x7fffc4f52640 (LWP 993)]
>> [New Thread 0x7fffb4751640 (LWP 994)]
>> [New Thread 0x7fffabf50640 (LWP 995)]
>> [New Thread 0x7fffab74f640 (LWP 996)]
>> [New Thread 0x7fffa2f4e640 (LWP 997)]
>> [New Thread 0x7fff9a74d640 (LWP 998)]
>> [New Thread 0x7fff91f4c640 (LWP 999)]
>> [New Thread 0x7fff8974b640 (LWP 1000)]
>> [New Thread 0x7fff78f4a640 (LWP 1001)]
>> [New Thread 0x7fff78749640 (LWP 1002)]
>> [New Thread 0x7fff6f5ff640 (LWP 1003)]
>>
>> Thread 1 "ogrinfo" received signal SIGILL, Illegal instruction.
>> 0x73773c9e in tiledb::common::ThreadPool::ThreadPool(unsigned
>> long) () from /lib/x86_64-linux-gnu/libtiledb.so.2.16
>>
>>
>>
>>
>> valgrind -s ogrinfo /tmp/newdir
>> ==704== Memcheck, a memory error detector
>> ==704== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
>> ==704== Using Valgrind-3.18.1 and LibVEX; rerun with -h for copyright info
>> ==704== Command: ogrinfo /tmp/newdir
>> ==704==
>> INFO: Open of `/tmp/newdir'
>>   using driver `ESRI Shapefile' successful.
>> 1: geo_melbourne (Polygon)
>> ==704==
>> ==704== HEAP SUMMARY:
>> ==704== in use at exit: 25,486 bytes in 216 blocks
>> ==704==   total heap usage: 15,761 allocs, 15,545 frees, 2,390,169 bytes
>> allocated
>> ==704==
>> ==704== LEAK SUMMARY:
>> ==704==definitely lost: 0 bytes in 0 blocks
>> ==704==indirectly lost: 0 bytes in 0 blocks
>> ==704==  possibly lost: 544 bytes in 1 blocks
>> ==704==still reachable: 24,942 bytes in 215 blocks
>> ==704== suppressed: 0 bytes in 0 blocks

Re: [gdal-dev] core dump on dir info

2024-02-04 Thread Even Rouault via gdal-dev 
ok, so I believe this is the AVX2 issue I was talking about, as I 
realize that enabling AVX2 is the default mode when TileDB is built from 
source (which the Docker image does), and must be explicitly disabled 
with "./bootstrap --disable-avx2" (I've just changed the build recipe to 
include that, will take effect next time the images are refreshed)


To confirm, can you send or just check the output of : cat 
/proc/cpuinfo|grep sse|head -n 1


If there is no "avx2" in it, this is at 99.9% the reason of the issue.

Even

Le 04/02/2024 à 06:20, Michael Sumner a écrit :

skipping TileDB does fix:

ogr2ogr /tmp/newdir 
https://github.com/SymbolixAU/geojsonsf/raw/master/inst/examples/geo_melbourne.geojson -f 
"ESRI Shapefile"

export GDAL_SKIP=TileDB
ogrinfo /tmp/newdir/
INFO: Open of `/tmp/newdir/'
      using driver `ESRI Shapefile' successful.
1: geo_melbourne (Polygon)

unset GDAL_SKIP
ogrinfo /tmp/newdir/
Illegal instruction (core dumped)

I failed to explain that I'm using gdal containers from the repo:

docker run --rm -ti ghcr.io/osgeo/gdal:ubunt 



u-full-latest

apt update
apt install -y gdb

Here's the output of under gdb as you suggested, there was a lot so I 
put it on a gist: 
https://gist.github.com/mdsumner/839ae6e05ededf640b65bfee3a20a4c0


gdb --args ogrinfo /tmp/newdir/
> run
> thread apply all bt

Thanks!





On Sat, Feb 3, 2024 at 7:49 PM Even Rouault 
 wrote:


- When it crashes under gdb, type "thread apply all bt" to get the
stack trace of all threads

- I suspect there is a connection with
https://github.com/OSGeo/gdal/pull/9170 , but that pull request
wouldn't help here as "/tmp/newdir" could be a valid connection to
TileDB

- how did you get TileDB installed? It looks to be packaged? Which
distribution do you use?

- SIGILL reminds me of issues with some TileDB builds using the
AVX2 instruction set by default, which could cause some crash on
host CPUs that don't have AVX2 (unlikely on recent hardware though)

- Setting GDAL_SKIP=TileDB should be a workaround


Le 03/02/2024 à 07:15, Michael Sumner a écrit :

Thanks Even, so there's something about tiledb under gdb (or
maybe I am mangling the context,  I will try variants of the host
I'm using). Run with valgrind included below.

gdb --args ogrinfo /tmp/newdir/
...
(gdb) run
Starting program: /usr/local/bin/ogrinfo /tmp/newdir/
[Thread debugging using libthread_db enabled]
Using host libthread_db library
"/lib/x86_64-linux-gnu/libthread_db.so.1".
[New Thread 0x7fffe7757640 (LWP 988)]
[New Thread 0x7fffe6f56640 (LWP 989)]
[New Thread 0x7fffde755640 (LWP 990)]
[New Thread 0x7fffd5f54640 (LWP 991)]
[New Thread 0x7fffc5753640 (LWP 992)]
[New Thread 0x7fffc4f52640 (LWP 993)]
[New Thread 0x7fffb4751640 (LWP 994)]
[New Thread 0x7fffabf50640 (LWP 995)]
[New Thread 0x7fffab74f640 (LWP 996)]
[New Thread 0x7fffa2f4e640 (LWP 997)]
[New Thread 0x7fff9a74d640 (LWP 998)]
[New Thread 0x7fff91f4c640 (LWP 999)]
[New Thread 0x7fff8974b640 (LWP 1000)]
[New Thread 0x7fff78f4a640 (LWP 1001)]
[New Thread 0x7fff78749640 (LWP 1002)]
[New Thread 0x7fff6f5ff640 (LWP 1003)]

Thread 1 "ogrinfo" received signal SIGILL, Illegal instruction.
0x73773c9e in
tiledb::common::ThreadPool::ThreadPool(unsigned long) () from
/lib/x86_64-linux-gnu/libtiledb.so.2.16




valgrind -s ogrinfo /tmp/newdir
==704== Memcheck, a memory error detector
==704== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward
et al.
==704== Using Valgrind-3.18.1 and LibVEX; rerun with -h for
copyright info
==704== Command: ogrinfo /tmp/newdir
==704==
INFO: Open of `/tmp/newdir'
      using driver `ESRI Shapefile' successful.
1: geo_melbourne (Polygon)
==704==
==704== HEAP SUMMARY:
==704==     in use at exit: 25,486 bytes in 216 blocks
==704==   total heap usage: 15,761 allocs, 15,545 frees,
2,390,169 bytes allocated
==704==
==704== LEAK SUMMARY:
==704==    definitely lost: 0 bytes in 0 blocks
==704==    indirectly lost: 0 bytes in 0 blocks
==704==      possibly lost: 544 bytes in 1 blocks
==704==    still reachable: 24,942 bytes in 215 blocks
==704==         suppressed: 0 bytes in 0 blocks
==704== Rerun with --leak-check=full to see details of leaked memory
==704==
==704== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0
from 0)


ogrinfo /tmp/newdir
Illegal instruction (core dumped)

Cheers, Mike




On Sat, Feb 3, 2024 at 12:46 PM Even Rouault
 wrote:

Michael,

I'm wondering if there not might be something wrong with your
build or runtime environment. Or there's something subtle,
because that works fine for me with my dev build or in the
ghcr.io/osgeo/gdal:alpine-normal-3.8.3

Re: [gdal-dev] core dump on dir info

2024-02-03 Thread Michael Sumner via gdal-dev 
skipping TileDB does fix:

ogr2ogr /tmp/newdir
https://github.com/SymbolixAU/geojsonsf/raw/master/inst/examples/geo_melbourne.geojson
-f
"ESRI Shapefile"
export GDAL_SKIP=TileDB
ogrinfo /tmp/newdir/
INFO: Open of `/tmp/newdir/'
  using driver `ESRI Shapefile' successful.
1: geo_melbourne (Polygon)

unset GDAL_SKIP
ogrinfo /tmp/newdir/
Illegal instruction (core dumped)

I failed to explain that I'm using gdal containers from the repo:

docker run --rm -ti ghcr.io/osgeo/gdal:ubunt

u-full-latest

apt update
apt install -y gdb

Here's the output of under gdb as you suggested, there was a lot so I put
it on a gist:
https://gist.github.com/mdsumner/839ae6e05ededf640b65bfee3a20a4c0

gdb --args ogrinfo /tmp/newdir/
> run
> thread apply all bt

Thanks!





On Sat, Feb 3, 2024 at 7:49 PM Even Rouault 
wrote:

> - When it crashes under gdb, type "thread apply all bt" to get the stack
> trace of all threads
>
> - I suspect there is a connection with
> https://github.com/OSGeo/gdal/pull/9170 , but that pull request wouldn't
> help here as "/tmp/newdir" could be a valid connection to TileDB
>
> - how did you get TileDB installed? It looks to be packaged? Which
> distribution do you use?
>
> - SIGILL reminds me of issues with some TileDB builds using the AVX2
> instruction set by default, which could cause some crash on host CPUs that
> don't have AVX2 (unlikely on recent hardware though)
>
> - Setting GDAL_SKIP=TileDB should be a workaround
>
>
> Le 03/02/2024 à 07:15, Michael Sumner a écrit :
>
> Thanks Even, so there's something about tiledb under gdb (or maybe I am
> mangling the context,  I will try variants of the host I'm using). Run with
> valgrind included below.
>
> gdb --args ogrinfo /tmp/newdir/
> ...
> (gdb) run
> Starting program: /usr/local/bin/ogrinfo /tmp/newdir/
> [Thread debugging using libthread_db enabled]
> Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
> [New Thread 0x7fffe7757640 (LWP 988)]
> [New Thread 0x7fffe6f56640 (LWP 989)]
> [New Thread 0x7fffde755640 (LWP 990)]
> [New Thread 0x7fffd5f54640 (LWP 991)]
> [New Thread 0x7fffc5753640 (LWP 992)]
> [New Thread 0x7fffc4f52640 (LWP 993)]
> [New Thread 0x7fffb4751640 (LWP 994)]
> [New Thread 0x7fffabf50640 (LWP 995)]
> [New Thread 0x7fffab74f640 (LWP 996)]
> [New Thread 0x7fffa2f4e640 (LWP 997)]
> [New Thread 0x7fff9a74d640 (LWP 998)]
> [New Thread 0x7fff91f4c640 (LWP 999)]
> [New Thread 0x7fff8974b640 (LWP 1000)]
> [New Thread 0x7fff78f4a640 (LWP 1001)]
> [New Thread 0x7fff78749640 (LWP 1002)]
> [New Thread 0x7fff6f5ff640 (LWP 1003)]
>
> Thread 1 "ogrinfo" received signal SIGILL, Illegal instruction.
> 0x73773c9e in tiledb::common::ThreadPool::ThreadPool(unsigned
> long) () from /lib/x86_64-linux-gnu/libtiledb.so.2.16
>
>
>
>
> valgrind -s ogrinfo /tmp/newdir
> ==704== Memcheck, a memory error detector
> ==704== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
> ==704== Using Valgrind-3.18.1 and LibVEX; rerun with -h for copyright info
> ==704== Command: ogrinfo /tmp/newdir
> ==704==
> INFO: Open of `/tmp/newdir'
>   using driver `ESRI Shapefile' successful.
> 1: geo_melbourne (Polygon)
> ==704==
> ==704== HEAP SUMMARY:
> ==704== in use at exit: 25,486 bytes in 216 blocks
> ==704==   total heap usage: 15,761 allocs, 15,545 frees, 2,390,169 bytes
> allocated
> ==704==
> ==704== LEAK SUMMARY:
> ==704==definitely lost: 0 bytes in 0 blocks
> ==704==indirectly lost: 0 bytes in 0 blocks
> ==704==  possibly lost: 544 bytes in 1 blocks
> ==704==still reachable: 24,942 bytes in 215 blocks
> ==704== suppressed: 0 bytes in 0 blocks
> ==704== Rerun with --leak-check=full to see details of leaked memory
> ==704==
> ==704== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)
>
>
> ogrinfo /tmp/newdir
> Illegal instruction (core dumped)
>
> Cheers, Mike
>
>
>
>
> On Sat, Feb 3, 2024 at 12:46 PM Even Rouault 
> wrote:
>
>> Michael,
>>
>> I'm wondering if there not might be something wrong with your build or
>> runtime environment. Or there's something subtle, because that works fine
>> for me with my dev build or in the ghcr.io/osgeo/gdal:alpine-normal-3.8.3
>> Docker image
>>
>> Try running "valgrind ogrinfo /tmp/newdir/" or "gdb --args ogrinfo
>> /tmp/newdir/" (type "run") to get more useful information
>>
>> Even
>> Le 03/02/2024 à 02:35, Michael Sumner via gdal-dev a écrit :
>>
>> I'm getting Illegal instruction / core dumped on ogrinfo of a directory:
>>
>> ogr2ogr /tmp/newdir
>> https://github.com/SymbolixAU/geojsonsf/raw/master/inst/examples/geo_melbourne.geojson
>> -f "ESRI Shapefile"
>>
>> ogrinfo /tmp/newdir/
>> Illegal instruction (core dumped)
>>
>> I've worked back through some docker images and it wasn't a problem in
>> 3.6.0, but I'm getting it since 3.7.0 - or I'm doing something wrong
>> entirely.
>>
>> Cheers, Mike
>>
>>
>> --
>> Michael Sumner
>> Software and Database Engineer
>> Australian Antarctic Division
>> Hobart, Australia
>> e-mail: mdsum.

Re: [gdal-dev] core dump on dir info

2024-02-03 Thread Even Rouault via gdal-dev 
- When it crashes under gdb, type "thread apply all bt" to get the stack 
trace of all threads


- I suspect there is a connection with 
https://github.com/OSGeo/gdal/pull/9170 , but that pull request wouldn't 
help here as "/tmp/newdir" could be a valid connection to TileDB


- how did you get TileDB installed? It looks to be packaged? Which 
distribution do you use?


- SIGILL reminds me of issues with some TileDB builds using the AVX2 
instruction set by default, which could cause some crash on host CPUs 
that don't have AVX2 (unlikely on recent hardware though)


- Setting GDAL_SKIP=TileDB should be a workaround


Le 03/02/2024 à 07:15, Michael Sumner a écrit :
Thanks Even, so there's something about tiledb under gdb (or maybe I 
am mangling the context,  I will try variants of the host I'm using). 
Run with valgrind included below.


gdb --args ogrinfo /tmp/newdir/
...
(gdb) run
Starting program: /usr/local/bin/ogrinfo /tmp/newdir/
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
[New Thread 0x7fffe7757640 (LWP 988)]
[New Thread 0x7fffe6f56640 (LWP 989)]
[New Thread 0x7fffde755640 (LWP 990)]
[New Thread 0x7fffd5f54640 (LWP 991)]
[New Thread 0x7fffc5753640 (LWP 992)]
[New Thread 0x7fffc4f52640 (LWP 993)]
[New Thread 0x7fffb4751640 (LWP 994)]
[New Thread 0x7fffabf50640 (LWP 995)]
[New Thread 0x7fffab74f640 (LWP 996)]
[New Thread 0x7fffa2f4e640 (LWP 997)]
[New Thread 0x7fff9a74d640 (LWP 998)]
[New Thread 0x7fff91f4c640 (LWP 999)]
[New Thread 0x7fff8974b640 (LWP 1000)]
[New Thread 0x7fff78f4a640 (LWP 1001)]
[New Thread 0x7fff78749640 (LWP 1002)]
[New Thread 0x7fff6f5ff640 (LWP 1003)]

Thread 1 "ogrinfo" received signal SIGILL, Illegal instruction.
0x73773c9e in tiledb::common::ThreadPool::ThreadPool(unsigned 
long) () from /lib/x86_64-linux-gnu/libtiledb.so.2.16





valgrind -s ogrinfo /tmp/newdir
==704== Memcheck, a memory error detector
==704== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
==704== Using Valgrind-3.18.1 and LibVEX; rerun with -h for copyright info
==704== Command: ogrinfo /tmp/newdir
==704==
INFO: Open of `/tmp/newdir'
      using driver `ESRI Shapefile' successful.
1: geo_melbourne (Polygon)
==704==
==704== HEAP SUMMARY:
==704==     in use at exit: 25,486 bytes in 216 blocks
==704==   total heap usage: 15,761 allocs, 15,545 frees, 2,390,169 
bytes allocated

==704==
==704== LEAK SUMMARY:
==704==    definitely lost: 0 bytes in 0 blocks
==704==    indirectly lost: 0 bytes in 0 blocks
==704==      possibly lost: 544 bytes in 1 blocks
==704==    still reachable: 24,942 bytes in 215 blocks
==704==         suppressed: 0 bytes in 0 blocks
==704== Rerun with --leak-check=full to see details of leaked memory
==704==
==704== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)


ogrinfo /tmp/newdir
Illegal instruction (core dumped)

Cheers, Mike




On Sat, Feb 3, 2024 at 12:46 PM Even Rouault 
 wrote:


Michael,

I'm wondering if there not might be something wrong with your
build or runtime environment. Or there's something subtle, because
that works fine for me with my dev build or in the
ghcr.io/osgeo/gdal:alpine-normal-3.8.3
 Docker image

Try running "valgrind ogrinfo /tmp/newdir/" or "gdb --args ogrinfo
/tmp/newdir/" (type "run") to get more useful information

Even

Le 03/02/2024 à 02:35, Michael Sumner via gdal-dev a écrit :

I'm getting Illegal instruction / core dumped on ogrinfo of a
directory:

ogr2ogr /tmp/newdir

https://github.com/SymbolixAU/geojsonsf/raw/master/inst/examples/geo_melbourne.geojson
-f "ESRI Shapefile"

ogrinfo /tmp/newdir/
Illegal instruction (core dumped)

I've worked back through some docker images and it wasn't a
problem in 3.6.0, but I'm getting it since 3.7.0 - or I'm doing
something wrong entirely.

Cheers, Mike


-- 
Michael Sumner

Software and Database Engineer
Australian Antarctic Division
Hobart, Australia
e-mail: mdsum...@gmail.com

___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev


-- 
http://www.spatialys.com

My software is free, but my time generally not.



--
Michael Sumner
Software and Database Engineer
Australian Antarctic Division
Hobart, Australia
e-mail: mdsum...@gmail.com


--
http://www.spatialys.com
My software is free, but my time generally not.
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] core dump on dir info

2024-02-02 Thread Michael Sumner via gdal-dev 
Thanks Even, so there's something about tiledb under gdb (or maybe I am
mangling the context,  I will try variants of the host I'm using). Run with
valgrind included below.

gdb --args ogrinfo /tmp/newdir/
...
(gdb) run
Starting program: /usr/local/bin/ogrinfo /tmp/newdir/
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
[New Thread 0x7fffe7757640 (LWP 988)]
[New Thread 0x7fffe6f56640 (LWP 989)]
[New Thread 0x7fffde755640 (LWP 990)]
[New Thread 0x7fffd5f54640 (LWP 991)]
[New Thread 0x7fffc5753640 (LWP 992)]
[New Thread 0x7fffc4f52640 (LWP 993)]
[New Thread 0x7fffb4751640 (LWP 994)]
[New Thread 0x7fffabf50640 (LWP 995)]
[New Thread 0x7fffab74f640 (LWP 996)]
[New Thread 0x7fffa2f4e640 (LWP 997)]
[New Thread 0x7fff9a74d640 (LWP 998)]
[New Thread 0x7fff91f4c640 (LWP 999)]
[New Thread 0x7fff8974b640 (LWP 1000)]
[New Thread 0x7fff78f4a640 (LWP 1001)]
[New Thread 0x7fff78749640 (LWP 1002)]
[New Thread 0x7fff6f5ff640 (LWP 1003)]

Thread 1 "ogrinfo" received signal SIGILL, Illegal instruction.
0x73773c9e in tiledb::common::ThreadPool::ThreadPool(unsigned long)
() from /lib/x86_64-linux-gnu/libtiledb.so.2.16




valgrind -s ogrinfo /tmp/newdir
==704== Memcheck, a memory error detector
==704== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
==704== Using Valgrind-3.18.1 and LibVEX; rerun with -h for copyright info
==704== Command: ogrinfo /tmp/newdir
==704==
INFO: Open of `/tmp/newdir'
  using driver `ESRI Shapefile' successful.
1: geo_melbourne (Polygon)
==704==
==704== HEAP SUMMARY:
==704== in use at exit: 25,486 bytes in 216 blocks
==704==   total heap usage: 15,761 allocs, 15,545 frees, 2,390,169 bytes
allocated
==704==
==704== LEAK SUMMARY:
==704==definitely lost: 0 bytes in 0 blocks
==704==indirectly lost: 0 bytes in 0 blocks
==704==  possibly lost: 544 bytes in 1 blocks
==704==still reachable: 24,942 bytes in 215 blocks
==704== suppressed: 0 bytes in 0 blocks
==704== Rerun with --leak-check=full to see details of leaked memory
==704==
==704== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)


ogrinfo /tmp/newdir
Illegal instruction (core dumped)

Cheers, Mike




On Sat, Feb 3, 2024 at 12:46 PM Even Rouault 
wrote:

> Michael,
>
> I'm wondering if there not might be something wrong with your build or
> runtime environment. Or there's something subtle, because that works fine
> for me with my dev build or in the ghcr.io/osgeo/gdal:alpine-normal-3.8.3
> Docker image
>
> Try running "valgrind ogrinfo /tmp/newdir/" or "gdb --args ogrinfo
> /tmp/newdir/" (type "run") to get more useful information
>
> Even
> Le 03/02/2024 à 02:35, Michael Sumner via gdal-dev a écrit :
>
> I'm getting Illegal instruction / core dumped on ogrinfo of a directory:
>
> ogr2ogr /tmp/newdir
> https://github.com/SymbolixAU/geojsonsf/raw/master/inst/examples/geo_melbourne.geojson
> -f "ESRI Shapefile"
>
> ogrinfo /tmp/newdir/
> Illegal instruction (core dumped)
>
> I've worked back through some docker images and it wasn't a problem in
> 3.6.0, but I'm getting it since 3.7.0 - or I'm doing something wrong
> entirely.
>
> Cheers, Mike
>
>
> --
> Michael Sumner
> Software and Database Engineer
> Australian Antarctic Division
> Hobart, Australia
> e-mail: mdsum...@gmail.com
>
> ___
> gdal-dev mailing 
> listgdal-dev@lists.osgeo.orghttps://lists.osgeo.org/mailman/listinfo/gdal-dev
>
> -- http://www.spatialys.com
> My software is free, but my time generally not.
>
>

-- 
Michael Sumner
Software and Database Engineer
Australian Antarctic Division
Hobart, Australia
e-mail: mdsum...@gmail.com
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] core dump on dir info

2024-02-02 Thread Even Rouault via gdal-dev 

Michael,

I'm wondering if there not might be something wrong with your build or 
runtime environment. Or there's something subtle, because that works 
fine for me with my dev build or in the 
ghcr.io/osgeo/gdal:alpine-normal-3.8.3 Docker image


Try running "valgrind ogrinfo /tmp/newdir/" or "gdb --args ogrinfo 
/tmp/newdir/" (type "run") to get more useful information


Even

Le 03/02/2024 à 02:35, Michael Sumner via gdal-dev a écrit :

I'm getting Illegal instruction / core dumped on ogrinfo of a directory:

ogr2ogr /tmp/newdir 
https://github.com/SymbolixAU/geojsonsf/raw/master/inst/examples/geo_melbourne.geojson 
-f "ESRI Shapefile"


ogrinfo /tmp/newdir/
Illegal instruction (core dumped)

I've worked back through some docker images and it wasn't a problem in 
3.6.0, but I'm getting it since 3.7.0 - or I'm doing something wrong 
entirely.


Cheers, Mike


--
Michael Sumner
Software and Database Engineer
Australian Antarctic Division
Hobart, Australia
e-mail: mdsum...@gmail.com

___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev


--
http://www.spatialys.com
My software is free, but my time generally not.
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev