Bug#818757: [Debian-med-packaging] Bug#818757: orthanc-postgresql: does not start

[Sebastien Jodogne]

>> After downloading the source code of each package, create a "Build"
>> directory inside, and run:
>>
>> # cmake .. -DSTATIC_BUILD=ON -DCMAKE_BUILD_TYPE=Debug
>> # make
>
> Attached a full backtrace of a stock Debian Orthanc 1.1.0
> with o-pg 2.0 (it is lacking symbols so might be of limited
> use).

Thanks, but unfortunately, this log does not show anything useful for 
debugging because of the lack of symbols.


Couldn't you try and statically link Orthanc and Orthanc-PG in debug 
mode following the commands written above? Or, at least, compile the 
Debian package from source and modify the "debian/rules" so as to add 
"-DCMAKE_BUILD_TYPE=Debug"?


Also, do you remember which versions of Orthanc and Orthanc-PG generated 
the database you try and upgrade?



> Any chance a -dbgsyms package can be provided ?

I have no idea of how this can be done in Debian together with CMake; I 
am unable to understand the wiki [1]. Furthermore, besides keeping the 
symbols, it would be really important to compile in Debug mode, 
otherwise the trace would be very hard to understand.


Sébastien-


[1] https://wiki.debian.org/DebugPackage

Bug#818757: [Debian-med-packaging] Bug#818757: orthanc-postgresql: does not start

[Karsten Hilbert]

On Wed, Jun 29, 2016 at 08:07:17AM +0200, Sebastien Jodogne wrote:

> What I would need would be a full backtrace of the C++ code. This requires a
> fresh build in debug mode, with static linking, of both Orthanc 1.1.0 and
> PostgreSQL plugin 2.0. Static linking allows to become independent of a
> potential instability of your Debian setup.
> 
> After downloading the source code of each package, create a "Build"
> directory inside, and run:
> 
> # cmake .. -DSTATIC_BUILD=ON -DCMAKE_BUILD_TYPE=Debug
> # make
> 
> Obviously, make sure that your Orthanc configuration points to the newly
> built PostgreSQL plugin.
> 
> Then, please post the full gdb backtrace (command "bt") of the crash when
> you try and run [Orthanc]

Attached a full backtrace of a stock Debian Orthanc 1.1.0
with o-pg 2.0 (it is lacking symbols so might be of limited
use).

Any chance a -dbgsyms package can be provided ?

Karsten
-- 
GPG key ID E4071346 @ eu.pool.sks-keyservers.net
E167 67FD A291 2BEA 73BD  4537 78B9 A9F9 E407 1346
Skript gestartet auf Mi 29 Jun 2016 14:40:04 CEST
root@hermes:~/bin# gdb --args /usr/sbin/Orthanc --trace 
--logdir=/var/log/orthanc /etc/orthanc/
GNU gdb (Debian 7.11.1-2) 7.11.1
Copyright (C) 2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later 
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "i686-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
.
Find the GDB manual and other documentation resources online at:
.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /usr/sbin/Orthanc...(no debugging symbols found)...done.
(gdb) run
Starting program: /usr/sbin/Orthanc --trace --logdir=/var/log/orthanc 
/etc/orthanc/
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/i386-linux-gnu/libthread_db.so.1".
*** stack smashing detected ***: /usr/sbin/Orthanc terminated
=== Backtrace: =
/lib/i386-linux-gnu/libc.so.6(+0x6929b)[0xb717629b]
/lib/i386-linux-gnu/libc.so.6(__fortify_fail+0x37)[0xb7205eb7]
/lib/i386-linux-gnu/libc.so.6(+0xf8e78)[0xb7205e78]
/usr/share/orthanc/plugins/libOrthancPostgreSQLStorage.so(+0x11284)[0xb7fac284]
/usr/share/orthanc/plugins/libOrthancPostgreSQLStorage.so(+0xa9bf)[0xb7fa59bf]
/usr/share/orthanc/plugins/libOrthancPostgreSQLStorage.so(OrthancPluginInitialize+0xab)[0xb7fabd7b]
/usr/sbin/Orthanc(+0xd8007)[0x800d8007]
/usr/sbin/Orthanc(+0xd8b70)[0x800d8b70]
/usr/sbin/Orthanc(+0xd8161)[0x800d8161]
/usr/sbin/Orthanc(main+0x4cd)[0x8002400d]
/lib/i386-linux-gnu/libc.so.6(__libc_start_main+0xf7)[0xb7125517]
/usr/sbin/Orthanc(+0x2b568)[0x8002b568]
=== Memory map: 
8000-802ad000 r-xp  08:01 6288261/usr/sbin/Orthanc
802ad000-802b1000 r--p 002ac000 08:01 6288261/usr/sbin/Orthanc
802b1000-802b2000 rw-p 002b 08:01 6288261/usr/sbin/Orthanc
802b2000-804e7000 rw-p  00:00 0  [heap]
b4978000-b49a8000 r-xp  08:01 6898193
/usr/lib/i386-linux-gnu/libpq.so.5.8
b49a8000-b49aa000 r--p 0002f000 08:01 6898193
/usr/lib/i386-linux-gnu/libpq.so.5.8
b49aa000-b49ab000 rw-p 00031000 08:01 6898193
/usr/lib/i386-linux-gnu/libpq.so.5.8
b49ab000-b49df000 r-xp  08:01 6899772
/usr/lib/i386-linux-gnu/libjsoncpp.so.0.10.5
b49df000-b49e r--p 00033000 08:01 6899772
/usr/lib/i386-linux-gnu/libjsoncpp.so.0.10.5
b49e-b49e1000 rw-p 00034000 08:01 6899772
/usr/lib/i386-linux-gnu/libjsoncpp.so.0.10.5
b4a34000-b4a3d000 rw-p  00:00 0 
b4a3d000-b4a44000 r-xp  08:01 6899295
/usr/lib/i386-linux-gnu/libffi.so.6.0.4
b4a44000-b4a45000 r--p 6000 08:01 6899295
/usr/lib/i386-linux-gnu/libffi.so.6.0.4
b4a45000-b4a46000 rw-p 7000 08:01 6899295
/usr/lib/i386-linux-gnu/libffi.so.6.0.4
b4a46000-b4a5a000 r-xp  08:01 8151438
/lib/i386-linux-gnu/libgpg-error.so.0.19.0
b4a5a000-b4a5b000 r--p 00013000 08:01 8151438
/lib/i386-linux-gnu/libgpg-error.so.0.19.0
b4a5b000-b4a5c000 rw-p 00014000 08:01 8151438
/lib/i386-linux-gnu/libgpg-error.so.0.19.0
b4a5c000-b4a6e000 r-xp  08:01 6899143
/usr/lib/i386-linux-gnu/libtasn1.so.6.5.2
b4a6e000-b4a6f000 ---p 00012000 08:01 6899143
/usr/lib/i386-linux-gnu/libtasn1.so.6.5.2
b4a6f000-b4a7 r--p 00012000 08:01 6899143
/usr/lib/i386-linux-gnu/libtasn1.so.6.5.2
b4a7-b4a71000 rw-p 00013000 08:01 6899143
/usr/lib/i386-linux-gnu/libtasn1.so.6.5.2
b4a71000-b4a72000 rw-p  00:00 0 
b4a72000-b4ace000 r-xp  08:01 6898110
/usr/lib/i386-linux-gnu/libp11-kit.so.0.1.0
b4ace000-b4ad3000 r--p 0005b000 08:01 6898110
/usr/lib/i386-linux-gnu/libp11-kit.so.0.1.0
b4ad3000-b4ad4000 rw-p 

Bug#818757: [Debian-med-packaging] Bug#818757: orthanc-postgresql: does not start

[Sebastien Jodogne]

In the former case, couldn't it be possible that the
version of some shared library does not match its version of
the headers, maybe because of an unstable Debian?


That is surely possible. That is why I offered to provide
more information if told how to do so.


Yes, sorry about that: As mentioned earlier, I had no time to dig this 
problem further in the past weeks. Orthanc is huge, and I'm alone to 
work on its code.


What I would need would be a full backtrace of the C++ code. This 
requires a fresh build in debug mode, with static linking, of both 
Orthanc 1.1.0 and PostgreSQL plugin 2.0. Static linking allows to become 
independent of a potential instability of your Debian setup.


After downloading the source code of each package, create a "Build" 
directory inside, and run:


# cmake .. -DSTATIC_BUILD=ON -DCMAKE_BUILD_TYPE=Debug
# make

Obviously, make sure that your Orthanc configuration points to the newly 
built PostgreSQL plugin.


Then, please post the full gdb backtrace (command "bt") of the crash 
when you try and run "--upgrade".


Sorry again and thanks for your collaboration,
Sébastien-

Bug#818757: [Debian-med-packaging] Bug#818757: orthanc-postgresql: does not start

[Karsten Hilbert]

On Mon, Jun 27, 2016 at 11:03:36AM +0200, Sébastien Jodogne wrote:

> > - I would not have guessed that adding --upgrade might have made
> >   a difference to a failure that tells me "stack smashing detected"
> >   at the console :-)
> 
> Just to be sure: Is your issue an immediate crash with "stack smashing 
> detected", or is your issue a failure while executing the "--upgrade"?

Both are (were) issues for me.

I have attempted to solve the upgrade issue by
dropping/re-creating the database in PostgreSQL
because the database did not hold production data.

However, Orthanc-on-PG does not start anymore because the PG
plugin now crashes with "stack smashing detected" so I cannot
verify whether it would properly start with the new empty
database ...

> In the former case, couldn't it be possible that the
> version of some shared library does not match its version of
> the headers, maybe because of an unstable Debian?

That is surely possible. That is why I offered to provide
more information if told how to do so.


> > Would you mind adding a line to the log indicating that
> > --upgrade might be needed when Orthanc detects that situation
> > at startup ?
> 
> This is already the case; Here is a sample log of Orthanc 1.0.0 asking an 
> upgrade with the PostgreSQL plugins 2.0 (notice the line starting with "E"):

I figured that out. It didn't have that line in the log when
I first attempted the upgrade.

Thanks,
Karsten
-- 
GPG key ID E4071346 @ eu.pool.sks-keyservers.net
E167 67FD A291 2BEA 73BD  4537 78B9 A9F9 E407 1346

Bug#818757: [Debian-med-packaging] Bug#818757: orthanc-postgresql: does not start

[Sébastien Jodogne]

> - I would not have guessed that adding --upgrade might have made
>   a difference to a failure that tells me "stack smashing detected"
>   at the console :-)

Just to be sure: Is your issue an immediate crash with "stack smashing 
detected", or is your issue a failure while executing the "--upgrade"?

In the former case, couldn't it be possible that the version of some shared 
library does not match its version of the headers, maybe because of an unstable 
Debian?


> Would you mind adding a line to the log indicating that
> --upgrade might be needed when Orthanc detects that situation
> at startup ?

This is already the case; Here is a sample log of Orthanc 1.0.0 asking an 
upgrade with the PostgreSQL plugins 2.0 (notice the line starting with "E"):

>
W0627 10:59:51.840168 main.cpp:1129] Orthanc version: 1.0.0
W0627 10:59:51.842232 OrthancInitialization.cpp:175] Scanning folder "." for 
configuration files
W0627 10:59:51.842363 OrthancInitialization.cpp:128] Reading the configuration 
from: "./postgresql.json"
W0627 10:59:51.842585 OrthancInitialization.cpp:128] Reading the configuration 
from: "./IntegrationTestsConfiguration.json"
W0627 10:59:51.842978 HttpClient.cpp:386] No certificates are provided to 
validate peers, set "HttpsCACertificates" if you need to do HTTPS requests
W0627 10:59:51.843845 FromDcmtkBridge.cpp:148] Loading the external DICOM 
dictionary "/usr/share/libdcmtk2/dicom.dic"
W0627 10:59:51.854365 FromDcmtkBridge.cpp:148] Loading the external DICOM 
dictionary "/usr/share/libdcmtk2/private.dic"
W0627 10:59:51.862899 OrthancInitialization.cpp:420] Registering JPEG Lossless 
codecs
W0627 10:59:51.862938 OrthancInitialization.cpp:425] Registering JPEG codecs
W0627 10:59:51.870631 main.cpp:618] Loading plugin(s) from: ./.
W0627 10:59:51.871377 PluginsManager.cpp:264] Registering plugin 
'postgresql-index' (version 2.0)
W0627 10:59:51.871608 PluginsManager.cpp:167] Using PostgreSQL index
W0627 10:59:51.881027 PluginsManager.cpp:264] Registering plugin 
'postgresql-storage' (version 2.0)
W0627 10:59:51.881324 PluginsManager.cpp:167] Using PostgreSQL storage area
NOTICE:  relation "storagearea" already exists, skipping
W0627 10:59:51.902725 main.cpp:966] Using a custom database from plugins
W0627 10:59:51.902774 main.cpp:977] Using a custom storage area from plugins
E0627 10:59:51.903043 main.cpp:830] The database schema must be upgraded from 
version 5 to 6: Please run Orthanc with the "--upgrade" command-line option
W0627 10:59:51.903187 PluginsManager.cpp:214] Unregistering plugin 
'postgresql-index' (version 2.0)
W0627 10:59:51.903272 PluginsManager.cpp:167] PostgreSQL index is finalizing
W0627 10:59:51.903444 PluginsManager.cpp:214] Unregistering plugin 
'postgresql-storage' (version 2.0)
W0627 10:59:51.903508 PluginsManager.cpp:167] Storage plugin is finalizing
W0627 10:59:51.904071 main.cpp:1202] Orthanc has stopped
<

Sébastien-

Bug#818757: [Debian-med-packaging] Bug#818757: orthanc-postgresql: does not start

[Sébastien Jodogne]

Hello,

I have spent a couple of hours trying to reproduce this issue, but I was unable 
to do so. Here are the tests I made:

* Upgrade from Orthanc 0.9.4 + PosgreSQL plugins 1.3 (database schema v5), to 
Orthanc mainline + PostgreSQL plugins 2.0 (database schema v6).

* Upgrade from Orthanc 0.9.1 + PosgreSQL plugins 1.2 (database schema v5), to 
Orthanc mainline + PostgreSQL plugins 2.0 (database schema v6).

In both cases, I have tried with and without the configuration option 
"StorageCompression". Everything works fine.

Would it be possible for someone to provide me the exact way of reproducing 
this issue?

Regards,
Sébastien-



- Mail original -
> De: "Karsten Hilbert" <karsten.hilb...@gmx.net>
> À: 818...@bugs.debian.org
> Envoyé: Mardi 19 Avril 2016 11:34:17
> Objet: [Debian-med-packaging] Bug#818757: orthanc-postgresql: does not
> start
> 
> On Sun, Mar 27, 2016 at 01:13:03PM +0200, Karsten Hilbert wrote:
> 
> > Is there anything else I can provide to get this bug looked at ?
> 
> Even if I risk to be a pain in the behind - since this
> package is of such importance to responsible Medical Care
> I allow myself to re-inquire about the state of this bug.
> 
> Thanks for looking,
> Karsten

Bug#818757: [Debian-med-packaging] Bug#818757: orthanc-postgresql: does not start

[Sébastien Jodogne]

Hello,

> > Is there anything else I can provide to get this bug looked at ?
> 
> Even if I risk to be a pain in the behind - since this
> package is of such importance to responsible Medical Care
> I allow myself to re-inquire about the state of this bug.

Unfortunately, this item is low-priority wrt. the needs of my employer. I 
cannot personally spend time on it in the next few weeks.

Regards,
Sébastien-

Bug#818757: [Debian-med-packaging] Bug#818757: orthanc-postgresql: does not start

[Karsten Hilbert]

Hi Sébastien,

welcome back.

> I am back from vacations. I will look at the upgrade problem as soon as 
> possible.

I don't think it is an upgrade problem.

> As a quick hack to unblock you,

Fortunately, this is on a testing machine. No production data
at risk - that instance still runs on sqlite.

Just for good measure I have dropped the DB, recreated a new
empty one and restarted with the same crash.

> This upgrade problem should not occur again in the future,
> as I think that the database schema has reached its final
> state since Orthanc 1.0.0.

:-)

Karsten
-- 
GPG key ID E4071346 @ eu.pool.sks-keyservers.net
E167 67FD A291 2BEA 73BD  4537 78B9 A9F9 E407 1346

Bug#818757: [Debian-med-packaging] Bug#818757: orthanc-postgresql: does not start

[Sébastien Jodogne]

Hello Karsten,

> Is there anything else I can provide to get this bug looked at ?

I am back from vacations. I will look at the upgrade problem as soon as 
possible.

As a quick hack to unblock you, I would suggest to apply the "Generic 
replication" procedure [1]. In your case, the procedure would consist in the 
following steps:

* Build from source the old versions of Orthanc and OrthancPostgreSQL that used 
to work against your database (respectively versions 0.8.x and 2.0 according to 
your first mail)
* Uninstall and purge all the orthanc-related packages (make sure to backup 
your configuration files)
* Remove the content of the "/etc/orthanc" folder
* Reinstall the "orthanc" and "orthanc-postgresql" packages from scratch
* Create a new database in PostgreSQL
* Restore your configuration files
* Edit "/etc/orthanc/postgresql.json" so that it points to the newly created 
database
* Restart Orthanc: "sudo /etc/init.d/orthanc restart"
* Run the old version of Orthanc with your old configuration files against your 
old database, making sure to change the DICOM/REST port so that it does not 
collide with the "4242/8042" default ports
* Apply the "Generic replication" procedure [1]

This upgrade problem should not occur again in the future, as I think that the 
database schema has reached its final state since Orthanc 1.0.0.

Sorry for the inconvenience,
Sébastien-


[1] 
https://orthanc.chu.ulg.ac.be/book/users/replication.html#generic-replication

Bug#818757: [Debian-med-packaging] Bug#818757: orthanc-postgresql: does not start

[Karsten Hilbert]

On Sun, Mar 20, 2016 at 05:55:10PM +0100, Sébastien Jodogne wrote:

> > 4) the existing orthanc_db is 0.8 and the jump is to Orthanc 1.0 (orthanc-pg
> > 2.0)
> 
> Have you added the "--upgrade" option so that Orthanc would automatically 
> upgrade the database schema?

I now did run Orthanc with --upgrade and it crashes with the
same error.

Karsten
-- 
GPG key ID E4071346 @ eu.pool.sks-keyservers.net
E167 67FD A291 2BEA 73BD  4537 78B9 A9F9 E407 1346

Bug#818757: [Debian-med-packaging] Bug#818757: orthanc-postgresql: does not start

[Karsten Hilbert]

On Sun, Mar 20, 2016 at 05:55:10PM +0100, Sébastien Jodogne wrote:

> > 4) the existing orthanc_db is 0.8 and the jump is to Orthanc 1.0 (orthanc-pg
> > 2.0)
> 
> Have you added the "--upgrade" option so that Orthanc would automatically 
> upgrade the database schema?

I have not because

- it's not really me calling /usr/bin/Orthanc but /etc/init.d/orthanc
  (I only copied that call manually in order to get more information)

- I didn't know that option existed

- I would not have guessed that adding --upgrade might have made
  a difference to a failure that tells me "stack smashing detected"
  at the console :-)

- I knew from past experience that Orthanc auto-upgraded the schema
  (but that was on SQLite and from major to major w/o a version skip)

However, I will now add this option and report back.

Would you mind adding a line to the log indicating that
--upgrade might be needed when Orthanc detects that situation
at startup ?

Also, would Debian benefit from an

/usr/sbin/orthanc_upgrade

script which calls /usr/sbin/Orthanc --upgrade plus other options ?

Karsten
-- 
GPG key ID E4071346 @ eu.pool.sks-keyservers.net
E167 67FD A291 2BEA 73BD  4537 78B9 A9F9 E407 1346

Bug#818757: [Debian-med-packaging] Bug#818757: orthanc-postgresql: does not start

[Sébastien Jodogne]

Hello,

> 4) the existing orthanc_db is 0.8 and the jump is to Orthanc 1.0 (orthanc-pg
> 2.0)

Have you added the "--upgrade" option so that Orthanc would automatically 
upgrade the database schema?

Sébastien-