Bug#943600: lazarus autopkgtest intermittedly fails due to EAccessViolation: Access violation

[Paul Gevers]

Hi all,

On 02-01-2020 19:29, Graham Inggs wrote:
> As of now, lazarus 2.0.6+dfsg-3 has failed its autopkgtests 2 out of
> 11 times on amd64 [1], and 3 out of 11 times on arm64 [2].
> 
> I am hesitant to re-open this bug as RC, as it potentially removes all
> packages built with lazarus for no good reason.
> 
> 
> [1] https://ci.debian.net/packages/l/lazarus/testing/amd64/
> [2] https://ci.debian.net/packages/l/lazarus/testing/arm64/

I propose we mark the test as flaky until somebody is able to completely
fix it or until the statistics have dramatically improved.

Paul



signature.asc
Description: OpenPGP digital signature

Bug#943600: [Pkg-pascal-devel] Bug#943600: Bug#943600: lazarus autopkgtest intermittedly fails due to EAccessViolation: Access violation

[Graham Inggs]

As of now, lazarus 2.0.6+dfsg-3 has failed its autopkgtests 2 out of
11 times on amd64 [1], and 3 out of 11 times on arm64 [2].

I am hesitant to re-open this bug as RC, as it potentially removes all
packages built with lazarus for no good reason.


[1] https://ci.debian.net/packages/l/lazarus/testing/amd64/
[2] https://ci.debian.net/packages/l/lazarus/testing/arm64/

Bug#943600: [Pkg-pascal-devel] Bug#943600: lazarus autopkgtest intermittedly fails due to EAccessViolation: Access violation

[Abou Al Montacir]

Hi Graham,
Unfortunately I did not find time to dig into it, but I checked today's failure
log and it seems to be related to timeout.
Did not check  on ARM because of lack of time.
-- 
Cheers,
Abou Al Montacir


signature.asc
Description: This is a digitally signed message part

Bug#943600: [Pkg-pascal-devel] Bug#943600: Bug#943600: Bug#943600: lazarus autopkgtest intermittedly fails due to EAccessViolation: Access violation

[Graham Inggs]

Hi Abou

2.0.6+dfsg-3, still fails intermittently on armhf in Ubuntu [1].
Did you try debugging on armhf in Debian?

Regards
Graham


[1] http://autopkgtest.ubuntu.com/packages/l/lazarus/focal/armhf

Processed: Re: [Pkg-pascal-devel] Bug#943600: Bug#943600: lazarus autopkgtest intermittedly fails due to EAccessViolation: Access violation

[Debian Bug Tracking System]

Processing control commands:

> reopen -1
Bug #943600 {Done: Abou Al Montacir } [src:lazarus] 
lazarus autopkgtest intermittedly fails due to EAccessViolation: Access 
violation
'reopen' may be inappropriate when a bug has been closed with a version;
all fixed versions will be cleared, and you may need to re-add them.
Bug reopened
No longer marked as fixed in versions lazarus/2.0.6+dfsg-2.

-- 
943600: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=943600
Debian Bug Tracking System
Contact ow...@bugs.debian.org with problems

Bug#943600: [Pkg-pascal-devel] Bug#943600: Bug#943600: lazarus autopkgtest intermittedly fails due to EAccessViolation: Access violation

[Paul Gevers]

Control: reopen -1

On 20-11-2019 15:43, Graham Inggs wrote:
> On 2019/11/16 14:41, Abou Al Montacir wrote:
>> A patch was applied on 2.0.6+dfsg-2 but it seems not completely fixing
>> this
>> issue.Waiting for upstream to fix it properly.
> 
> FWIW, 2.0.6+dfsg-2 is still failing intermittently on armhf in Ubuntu [1].

And also in Debian on amd64, so reopening the bug. Although, the error
is not always the same. I also see:
(3104) Compiling testcommonsources.pas
/usr/lib/lazarus/2.0.6/components/lazdebuggers/lazdebugtestbase/testcommonsources.pas(128,12)
Note: (5025) Local variable "j" not used
/usr/lib/lazarus/2.0.6/components/lazdebuggers/lazdebugtestbase/testcommonsources.pas(199,3)
Note: (5025) Local variable "i" not used
/usr/lib/lazarus/2.0.6/components/lazdebuggers/lazdebugtestbase/testcommonsources.pas(237,1)
Error: (9021) resource compiler "windres" not found, switching to
external mode
/usr/lib/lazarus/2.0.6/components/lazdebuggers/lazdebugtestbase/testcommonsources.pas(237,1)
Fatal: (10026) There were 1 errors compiling module, stopping
Fatal: (1018) Compilation aborted
Error: /usr/bin/ppcx64 returned an error exitcode
Error: (lazarus) Compile package LazDebugTestBase 0.1: stopped with exit
code 1


Paul

https://ci.debian.net/data/autopkgtest/testing/amd64/l/lazarus/3495295/log.gz
https://ci.debian.net/data/autopkgtest/unstable/amd64/l/lazarus/3533908/log.gz
https://ci.debian.net/data/autopkgtest/unstable/amd64/l/lazarus/3502195/log.gz
https://ci.debian.net/data/autopkgtest/unstable/amd64/l/lazarus/3429618/log.gz
https://ci.debian.net/data/autopkgtest/unstable/amd64/l/lazarus/3423557/log.gz



signature.asc
Description: OpenPGP digital signature

Bug#943600: [Pkg-pascal-devel] Bug#943600: lazarus autopkgtest intermittedly fails due to EAccessViolation: Access violation

[Graham Inggs]

Hi Abou

On 2019/11/16 14:41, Abou Al Montacir wrote:

A patch was applied on 2.0.6+dfsg-2 but it seems not completely fixing this
issue.Waiting for upstream to fix it properly.


FWIW, 2.0.6+dfsg-2 is still failing intermittently on armhf in Ubuntu [1].

Regards
Graham


[1] https://autopkgtest.ubuntu.com/packages/l/lazarus/focal/armhf

Bug#943600: lazarus autopkgtest intermittedly fails due to EAccessViolation: Access violation

[Abou Al Montacir]

A patch was applied on 2.0.6+dfsg-2 but it seems not completely fixing this
issue.Waiting for upstream to fix it properly.
-- 
Cheers,
Abou Al Montacir
Description: lazbuild fails to build some packages during auto tests.
Author: Abou Al Montacir 
Bug-Debian: http://bugs.debian.org/943600
diff --git a/components/ideintf/ideexterntoolintf.pas b/components/ideintf/ideexterntoolintf.pas
index 4654265c..fdf7549f 100644
--- a/components/ideintf/ideexterntoolintf.pas
+++ b/components/ideintf/ideexterntoolintf.pas
@@ -1229,12 +1229,14 @@ end;
 
 procedure TAbstractExternalTool.EnterCriticalSection;
 begin
-  FWorkerMessages.EnterCriticalSection;
+  if Assigned(FWorkerMessages) then
+FWorkerMessages.EnterCriticalSection;
 end;
 
 procedure TAbstractExternalTool.LeaveCriticalSection;
 begin
-  FWorkerMessages.LeaveCriticalSection;
+  if Assigned(FWorkerMessages) then
+FWorkerMessages.LeaveCriticalSection;
 end;
 
 procedure TAbstractExternalTool.ConsistencyCheck;


signature.asc
Description: This is a digitally signed message part

Bug#943600: lazarus autopkgtest intermittedly fails due to EAccessViolation: Access violation

[Abou Al Montacir]

Control: forwarded -1 https://bugs.freepascal.org/view.php?id=36318

I've tried to debug this issue but it seems more difficult than I thought.
It looks like some class instance is freed twice which results in access
violation on the second attempt to free it.

Modified code results in the following logs:

(1008) 3534 lines compiled, 0.6 sec
(1021) 4 warning(s) issued
(1022) 30 hint(s) issued
(1023) 3 note(s) issued
+TAbstractExternalTool.EnterCriticalSection: FWorkerMessages = 77A17940
-TAbstractExternalTool.EnterCriticalSection
+TAbstractExternalTool.LeaveCriticalSection: FWorkerMessages = 77A17940
-TAbstractExternalTool.LeaveCriticalSection
Warning: (lazarus) [TLazPackageGraph.ConvertPackageRSTFiles] skipping read only
directory /usr/lib/lazarus/2.0.6/components/sparta/dockedformeditor/language/
+TAbstractExternalTool.EnterCriticalSection: FWorkerMessages = 77A17940
-TAbstractExternalTool.EnterCriticalSection
+TAbstractExternalTool.EnterCriticalSection: FWorkerMessages = 77A17940
-TAbstractExternalTool.EnterCriticalSection
+TAbstractExternalTool.LeaveCriticalSection: FWorkerMessages = 77A17940
-TAbstractExternalTool.LeaveCriticalSection
+TAbstractExternalTool.LeaveCriticalSection: FWorkerMessages = 77A17940
-TAbstractExternalTool.LeaveCriticalSection
+TAbstractExternalTool.EnterCriticalSection: FWorkerMessages = 77A17940
-TAbstractExternalTool.EnterCriticalSection
+TAbstractExternalTool.EnterCriticalSection: FWorkerMessages = 77A17940
+TAbstractExternalTool.LeaveCriticalSection: FWorkerMessages = 77A17940
-TAbstractExternalTool.LeaveCriticalSection
+TAbstractExternalTool.EnterCriticalSection: FWorkerMessages = 77A17940
-TAbstractExternalTool.EnterCriticalSection
+TAbstractExternalTool.EnterCriticalSection: FWorkerMessages = 77A17940
-TAbstractExternalTool.EnterCriticalSection
+TAbstractExternalTool.LeaveCriticalSection: FWorkerMessages = 77A17940
-TAbstractExternalTool.LeaveCriticalSection
+TAbstractExternalTool.EnterCriticalSection: FWorkerMessages = 77A17940
-TAbstractExternalTool.EnterCriticalSection
+TAbstractExternalTool.LeaveCriticalSection: FWorkerMessages = 77A17940
-TAbstractExternalTool.LeaveCriticalSection
+TAbstractExternalTool.EnterCriticalSection: FWorkerMessages = 77A16DE0
-TAbstractExternalTool.EnterCriticalSection
+TAbstractExternalTool.LeaveCriticalSection: FWorkerMessages = 77A16DE0
-TAbstractExternalTool.LeaveCriticalSection
+TAbstractExternalTool.EnterCriticalSection: FWorkerMessages = 77A13340
-TAbstractExternalTool.EnterCriticalSection
+TAbstractExternalTool.LeaveCriticalSection: FWorkerMessages = 77A13340
-TAbstractExternalTool.LeaveCriticalSection
+TAbstractExternalTool.LeaveCriticalSection: FWorkerMessages = 77A17940
-TAbstractExternalTool.LeaveCriticalSection
-TAbstractExternalTool.EnterCriticalSection
+TAbstractExternalTool.LeaveCriticalSection: FWorkerMessages = 

Thread 4 "lazbuild" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x77408700 (LWP 11030)]
0x0051ad0b in LEAVECRITICALSECTION (this=0x77649880) at
ideexterntoolintf.pas:1240
1240  FWorkerMessages.LeaveCriticalSection;
(gdb) 

I've finally forwarded it to upstream. 
-- 
Cheers,
Abou Al Montacir
$rm -r /tmp/.lazarus ; HOME=/tmp gdb --args ./lazbuild -B --verbose /usr/lib/lazarus/2.0.6/components/sparta/smartformeditor/sparta_smartformeditor.lpk
GNU gdb (Debian 8.2.1-2+b1) 8.2.1
Copyright (C) 2018 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later 
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
.
Find the GDB manual and other documentation resources online at:
.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from ./lazbuild...done.
(gdb) r
Starting program: /mnt/data/mazen/Debian/lazarus/lazbuild -B --verbose /usr/lib/lazarus/2.0.6/components/sparta/smartformeditor/sparta_smartformeditor.lpk
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Parameter: build-all
Hint: (lazarus) primary config path: /tmp/.lazarus
CopySecondaryConfigFile /etc/lazarus/environmentoptions.xml -> /tmp/.lazarus/environmentoptions.xml
Hint: [TPCTargetConfigCache.NeedsUpdate] TargetOS="" TargetCPU="" Options="" compiler file changed "/usr/bin/fpc" FileAge=1548372422 StoredAge=0
Hint: [TPCTargetConfigCache.NeedsUpdate] /usr/bin/fpc TargetOS= TargetCPU= CompilerOptions= ExtraOptions= 

Processed: Re: Bug#943600: lazarus autopkgtest intermittedly fails due to EAccessViolation: Access violation

[Debian Bug Tracking System]

Processing control commands:

> forwarded -1 https://bugs.freepascal.org/view.php?id=36318
Bug #943600 [src:lazarus] lazarus autopkgtest intermittedly fails due to 
EAccessViolation: Access violation
Set Bug forwarded-to-address to 'https://bugs.freepascal.org/view.php?id=36318'.

-- 
943600: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=943600
Debian Bug Tracking System
Contact ow...@bugs.debian.org with problems

Bug#943600: [Pkg-pascal-devel] Bug#943600: lazarus autopkgtest intermittedly fails due to EAccessViolation: Access violation

[Abou Al Montacir]

I finally managed to catch one of these errors on amd64 machine.
The issue is that on some packages that are expected to fail, the failure
results on an exception.However and despite the expected failure is caught, the
test infrastructure detects a message written to stderr and thus triggers the
failure.
I've rebuilt lazuild using debug symbols, and the error seems to be caught in an
object destructor (#3) as shown in the following gdb stack trace:Warning:
(lazarus) [TLazPackageGraph.ConvertPackageRSTFiles] skipping read only directory
/usr/lib/lazarus/2.0.6/components/sparta/dockedformeditor/language/
Thread 4 "lazbuild" received signal SIGSEGV, Segmentation fault.[Switching to
Thread 0x77408700 (LWP 25224)]0x0051ac03 in LEAVECRITICALSECTION
(this=0x77649720) at ideexterntoolintf.pas:12371237   FWorkerMessages.LeaveC
riticalSection;(gdb) bt#0  0x0051ac03 in LEAVECRITICALSECTION
(this=0x77649720) at ideexterntoolintf.pas:1237#1  0x009e9cb5 in
SETTHREAD (this=0x77649720, AVALUE=0x0) at
exttools.pas:411#2  0x009edc57 in SETTOOL (this=0x7761fc40,
AVALUE=0x0) at exttools.pas:1313#3  0x009eeeb1 in DESTROY
(this=0x7761fc40, vmt=0x1) at exttools.pas:1612#4  0x0041cd20 in
SYSTEM$_$TOBJECT_$__$$_FREE ()#5  0x777a9a80 in ??
()#6  0x00440cd7 in CLASSES_$$_THREADFUNC$POINTER$$INT64
()#7  0x7761fc40 in ?? ()#8  0x in ?? ()(gdb) 
To be continued!
-- 
Cheers,
Abou Al Montacir


signature.asc
Description: This is a digitally signed message part

Bug#943600: lazarus autopkgtest intermittedly fails due to EAccessViolation: Access violation

[Abou Al Montacir]

Hi Graham,

On Tue, 2019-11-05 at 10:06 +0200, Graham Inggs wrote:
> Hi Abou
> On Sun, 3 Nov 2019 at 13:42, Abou Al Montacir  wrote:
> > I packaged the new upstream 2.0.6, so maybe this issue will disappear.
> 
> It is still present.
Yes I noticed that, I'll try to have a deeper look this WE. I'll try to
reproduce on the porter box.
-- 
Cheers,
Abou Al Montacir


signature.asc
Description: This is a digitally signed message part

Bug#943600: [Pkg-pascal-devel] Bug#943600: Bug#943600: lazarus autopkgtest intermittedly fails due to EAccessViolation: Access violation

[Graham Inggs]

Hi Abou

On Sun, 3 Nov 2019 at 13:42, Abou Al Montacir  wrote:
> I packaged the new upstream 2.0.6, so maybe this issue will disappear.

It is still present.

A similar failure occurs in Ubuntu on armhf [1].
As of now, there were 6 out of 6 failures with 2.0.2+dfsg-7, and 6 out
of 8 failures with 2.0.6+dfsg-1.

So far, there have been no failures on amd64 [2]; 6 out of 6 passes
with 2.0.2+dfsg-7, and 8 out of 8 passes with 2.0.6+dfsg-1.

Regards
Graham


[1] https://autopkgtest.ubuntu.com/packages/l/lazarus/focal/armhf
[2] https://autopkgtest.ubuntu.com/packages/l/lazarus/focal/amd64

Bug#943600: [Pkg-pascal-devel] Bug#943600: lazarus autopkgtest intermittedly fails due to EAccessViolation: Access violation

[Abou Al Montacir]

Hi Paul,
I failed to reproduce this issue. Maybe we need to enable debug when compiling
for the tests so that we get a comprehensible stack trace.
I packaged the new upstream 2.0.6, so maybe this issue will disappear.
-- 
Cheers,
Abou Al Montacir


signature.asc
Description: This is a digitally signed message part

Bug#943600: lazarus autopkgtest intermittedly fails due to EAccessViolation: Access violation

[Paul Gevers]

Source: lazarus
Version: 2.0.2+dfsg-7
Severity: serious
Tags: sid bullseye
X-Debbugs-CC: debian...@lists.debian.org
User: debian...@lists.debian.org
Usertags: flaky

Dear all,

Version 2.0.2+dfsg-7 of lazarus has failed already at least three times
on the ci.d.n infrastructure. The error is always a logged
EAccessViolation, except the stack isn't always the same length. So far
the offset between the two lowest items is the same.

Paul

https://ci.debian.net/data/autopkgtest/testing/amd64/l/lazarus/3271190/log.gz

An unhandled exception occurred at $:
EAccessViolation: Access violation
  $
  $55EFE24EAB67
  $55EFE24EBDC1

https://ci.debian.net/data/autopkgtest/unstable/amd64/l/lazarus/3250811/log.gz

An unhandled exception occurred at $:
EAccessViolation: Access violation
  $
  $557D05069B67
  $557D0506ADC1

https://ci.debian.net/data/autopkgtest/unstable/amd64/l/lazarus/3250711/log.gz

EAccessViolation: Access violation
  $563EC38CCBF3
  $563EC3D9ABC5
  $563EC3D9EB67
  $563EC3D9FDC1

- -- System Information:
Debian Release: bullseye/sid
  APT prefers testing
  APT policy: (990, 'testing'), (500, 'testing-debug')
Architecture: amd64 (x86_64)

Kernel: Linux 5.2.0-3-amd64 (SMP w/2 CPU cores)
Kernel taint flags: TAINT_PROPRIETARY_MODULE, TAINT_OOT_MODULE,
TAINT_UNSIGNED_MODULE
Locale: LANG=en_US.UTF-8, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8),
LANGUAGE=en_US:en (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash
Init: systemd (via /run/systemd/system)
LSM: AppArmor: enabled



signature.asc
Description: OpenPGP digital signature