subject:"\[jira\] \[Commented\] \(DISPATCH\-1962\) Make LSan suppressions more targeted and specific \(within reason\)"

[jira] [Commented] (DISPATCH-1962) Make LSan suppressions more targeted and specific (within reason)

2021-05-06 Thread ASF subversion and git services (Jira)



[ 
https://issues.apache.org/jira/browse/DISPATCH-1962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17340419#comment-17340419
 ] 

ASF subversion and git services commented on DISPATCH-1962:
---

Commit 5c8b2d4fae2d96bf56f787055cbbd340bdad3477 in qpid-dispatch's branch 
refs/heads/main from Jiri Daněk
[ https://gitbox.apache.org/repos/asf?p=qpid-dispatch.git;h=5c8b2d4 ]

DISPATCH-1962 Update qdr_core_subscribe leak fix so it does not cause 
qdr_delivery_cleanup_t leak (#1191)



> Make LSan suppressions more targeted and specific (within reason)
> -
>
> Key: DISPATCH-1962
> URL: https://issues.apache.org/jira/browse/DISPATCH-1962
> Project: Qpid Dispatch
>  Issue Type: Test
>Affects Versions: 1.15.0
>Reporter: Jiri Daněk
>Assignee: Jiri Daněk
>Priority: Major
> Fix For: 1.16.0
>
> Attachments: dispatch_leaking_agent.png
>
>
> There are some unfortunate suppressions in {{tests/lsan.supp}} at this moment:
> {code}
> leak:*libwebsockets*
> {code}
> This is way too broad. It suppresses leaks in Dispatch files, since it 
> matches e.g. {{src/http-libwebsockets.c}}.
> The stars at the beginning and end are actually assumed implicitly. If you do 
> not want substring match, you have to do ^foo$. LSan suppression format is 
> simplistic, very unlike Valgrind's.
> {code}
> leak:run_unit_tests.c
> {code}
> Same thing, any leaks revealed by running unit_tests get suppressed. This 
> suppression suppresses all leak traces that include run_unit_tests.c anywhere 
> in the stack. What't the point of running such tests under leak detector, 
> then?
> {code}
> leak:run_unit_tests.c
> leak:^libqpid-proton.so$
> {code}
> Same thing. The patterns suppress all leaks that include Python (or Proton) 
> anywhere in the stacktrace. That means there are huge blind spots where 
> dispatch leaks can hide. This is a weakness of the lsan.supp syntax (Valgrind 
> suppressions can be much more targeted and discerning).
> h3. Python leaks
> Leaks are known and there is ongoing effort to fight them: 
> https://bugs.python.org/issue1635741 (https://bugs.python.org/issue25302) and 
> https://www.python.org/dev/peps/pep-3121
> Here's valgrind suppression file from somebody who actually investigated the 
> Python leaks and identified the harmless ones: 
> https://github.com/libgit2/pygit2/blob/master/misc/valgrind-python.supp
> One example of a hidden leak in dispatch, which is revealed by making Python 
> suppressions more targetted:
> {code}
> 9: Direct leak of 56 byte(s) in 1 object(s) allocated from:
> 9: #0 0x7f78a3606e8f in __interceptor_malloc 
> (/nix/store/g40sl3zh3nv52vj0mrl4iki5iphh5ika-gcc-10.2.0-lib/lib/libasan.so.6+0xace8f)
> 9: #1 0x7f78a2d64afb in qd_malloc ../include/qpid/dispatch/ctools.h:229
> 9: #2 0x7f78a2d657da in qdr_core_subscribe 
> ../src/router_core/route_tables.c:149
> 9: #3 0x7f78a2c83072 in IoAdapter_init ../src/python_embedded.c:711
> 9: #4 0x7f78a2353a6c in type_call 
> (/nix/store/r85nxfnwiv45nbmf5yb60jj8ajim4m7w-python3-3.8.5/lib/libpython3.8.so.1.0+0x165a6c)
> {code}
> The problem is in
> {code}
> class Agent:
> ...
> def activate(self, address):
> ...
> self.io = IoAdapter(self.receive, address, 'L', '0', 
> TREATMENT_ANYCAST_CLOSEST)
> {code}
> IoAdapter refers to Agent (through the bound method reference self.receive) 
> and Agent refers to IoAdapter (through property self.io). Since IoAdapter is 
> implemented in C and does not implement support for Python's cyclic GC, there 
> is no way to break the cycle.
> Heap dump in attachment. The bound method is at the top of the picture. 
> (Ignore the Mock objects, I was trying to simplify the picture while not 
> getting crashes due to too much meddling).
> h3. Random observations
> It is possible to build special Debug build of Python, which has tools to 
> detect leaks, asserts to prevent negative refcounts, etc. 
> https://pythonextensionpatterns.readthedocs.io/en/latest/debugging/debug_python.html#debug-version-of-python-memory-alloc-label
> Use the following to detect python leaks (instead of valgrind)
> https://docs.python.org/3/library/tracemalloc.html
> Use https://pypi.org/project/objgraph (with graphviz) to view heap object 
> trees. The following renders the picture as a png under /tmp and prints the 
> path to stdout.
> {code}
> int ret = PyRun_SimpleString("import objgraph; 
> objgraph.show_backrefs(config.global_agent, max_depth=10)\n\n");
> PyErr_PrintEx(0);
> assert(ret == 0);
> {code}
> h2. Tools
> h3. CPyChecker
> https://dmalcolm.fedorapeople.org/presentations/PyCon-US-2013/PyCon-US-2013-dmalcolm-StaticAnalysis.html#(38)
> https://emptysqua.re/blog/analyzing-python-c-extensions-with-cpychecker/
>

[jira] [Commented] (DISPATCH-1962) Make LSan suppressions more targeted and specific (within reason)

2021-05-06 Thread ASF GitHub Bot (Jira)



[ 
https://issues.apache.org/jira/browse/DISPATCH-1962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17340418#comment-17340418
 ] 

ASF GitHub Bot commented on DISPATCH-1962:
--

jiridanek merged pull request #1191:
URL: https://github.com/apache/qpid-dispatch/pull/1191


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Make LSan suppressions more targeted and specific (within reason)
> -
>
> Key: DISPATCH-1962
> URL: https://issues.apache.org/jira/browse/DISPATCH-1962
> Project: Qpid Dispatch
>  Issue Type: Test
>Affects Versions: 1.15.0
>Reporter: Jiri Daněk
>Assignee: Jiri Daněk
>Priority: Major
> Fix For: 1.16.0
>
> Attachments: dispatch_leaking_agent.png
>
>
> There are some unfortunate suppressions in {{tests/lsan.supp}} at this moment:
> {code}
> leak:*libwebsockets*
> {code}
> This is way too broad. It suppresses leaks in Dispatch files, since it 
> matches e.g. {{src/http-libwebsockets.c}}.
> The stars at the beginning and end are actually assumed implicitly. If you do 
> not want substring match, you have to do ^foo$. LSan suppression format is 
> simplistic, very unlike Valgrind's.
> {code}
> leak:run_unit_tests.c
> {code}
> Same thing, any leaks revealed by running unit_tests get suppressed. This 
> suppression suppresses all leak traces that include run_unit_tests.c anywhere 
> in the stack. What't the point of running such tests under leak detector, 
> then?
> {code}
> leak:run_unit_tests.c
> leak:^libqpid-proton.so$
> {code}
> Same thing. The patterns suppress all leaks that include Python (or Proton) 
> anywhere in the stacktrace. That means there are huge blind spots where 
> dispatch leaks can hide. This is a weakness of the lsan.supp syntax (Valgrind 
> suppressions can be much more targeted and discerning).
> h3. Python leaks
> Leaks are known and there is ongoing effort to fight them: 
> https://bugs.python.org/issue1635741 (https://bugs.python.org/issue25302) and 
> https://www.python.org/dev/peps/pep-3121
> Here's valgrind suppression file from somebody who actually investigated the 
> Python leaks and identified the harmless ones: 
> https://github.com/libgit2/pygit2/blob/master/misc/valgrind-python.supp
> One example of a hidden leak in dispatch, which is revealed by making Python 
> suppressions more targetted:
> {code}
> 9: Direct leak of 56 byte(s) in 1 object(s) allocated from:
> 9: #0 0x7f78a3606e8f in __interceptor_malloc 
> (/nix/store/g40sl3zh3nv52vj0mrl4iki5iphh5ika-gcc-10.2.0-lib/lib/libasan.so.6+0xace8f)
> 9: #1 0x7f78a2d64afb in qd_malloc ../include/qpid/dispatch/ctools.h:229
> 9: #2 0x7f78a2d657da in qdr_core_subscribe 
> ../src/router_core/route_tables.c:149
> 9: #3 0x7f78a2c83072 in IoAdapter_init ../src/python_embedded.c:711
> 9: #4 0x7f78a2353a6c in type_call 
> (/nix/store/r85nxfnwiv45nbmf5yb60jj8ajim4m7w-python3-3.8.5/lib/libpython3.8.so.1.0+0x165a6c)
> {code}
> The problem is in
> {code}
> class Agent:
> ...
> def activate(self, address):
> ...
> self.io = IoAdapter(self.receive, address, 'L', '0', 
> TREATMENT_ANYCAST_CLOSEST)
> {code}
> IoAdapter refers to Agent (through the bound method reference self.receive) 
> and Agent refers to IoAdapter (through property self.io). Since IoAdapter is 
> implemented in C and does not implement support for Python's cyclic GC, there 
> is no way to break the cycle.
> Heap dump in attachment. The bound method is at the top of the picture. 
> (Ignore the Mock objects, I was trying to simplify the picture while not 
> getting crashes due to too much meddling).
> h3. Random observations
> It is possible to build special Debug build of Python, which has tools to 
> detect leaks, asserts to prevent negative refcounts, etc. 
> https://pythonextensionpatterns.readthedocs.io/en/latest/debugging/debug_python.html#debug-version-of-python-memory-alloc-label
> Use the following to detect python leaks (instead of valgrind)
> https://docs.python.org/3/library/tracemalloc.html
> Use https://pypi.org/project/objgraph (with graphviz) to view heap object 
> trees. The following renders the picture as a png under /tmp and prints the 
> path to stdout.
> {code}
> int ret = PyRun_SimpleString("import objgraph; 
> objgraph.show_backrefs(config.global_agent, max_depth=10)\n\n");
> PyErr_PrintEx(0);
> assert(ret == 0);
> {code}
> h2. Tools
> h3. CPyChecker
> https://dmalcolm.fedorapeople.org/presentations/PyCon-US-2013/PyCon-US-2013-dmalcolm-StaticAnalysis.html#(38)
> https://emptysqua.re/blog/analyzing-python-c-extensions-with-cpychecker/
>

[jira] [Commented] (DISPATCH-1962) Make LSan suppressions more targeted and specific (within reason)

2021-05-06 Thread ASF GitHub Bot (Jira)



[ 
https://issues.apache.org/jira/browse/DISPATCH-1962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17340417#comment-17340417
 ] 

ASF GitHub Bot commented on DISPATCH-1962:
--

jiridanek commented on a change in pull request #1191:
URL: https://github.com/apache/qpid-dispatch/pull/1191#discussion_r627738190



##
File path: src/router_core/router_core.c
##
@@ -236,6 +236,33 @@ void qdr_core_free(qdr_core_t *core)
 link = DEQ_HEAD(core->open_links);
 }
 
+// finalize modules while we can still submit new actions
+// this must happen after qdrc_endpoint_do_cleanup_CT calls
+qdr_modules_finalize(core);
+
+// discard any left over actions
+
+qdr_action_list_t  action_list;
+DEQ_MOVE(core->action_list, action_list);
+DEQ_APPEND(action_list, core->action_list_background);
+qdr_action_t *action = DEQ_HEAD(action_list);
+while (action) {
+DEQ_REMOVE_HEAD(action_list);
+action->action_handler(core, action, true);
+free_qdr_action_t(action);
+action = DEQ_HEAD(action_list);
+}
+
+// Drain the general work lists
+qdr_general_handler(core);
+
+sys_thread_free(core->thread);
+sys_cond_free(core->action_cond);
+sys_mutex_free(core->action_lock);
+sys_mutex_free(core->work_lock);
+sys_mutex_free(core->id_lock);
+qd_timer_free(core->work_timer);
+
 //
 // Clean up any qdr_delivery_cleanup_t's that are still left in the 
core->delivery_cleanup_list
 //

Review comment:
   thanks for the explanation. next time I'll be sure to rebase and rerun 
all of CI before merging anything, to avoid fixup commits like this




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Make LSan suppressions more targeted and specific (within reason)
> -
>
> Key: DISPATCH-1962
> URL: https://issues.apache.org/jira/browse/DISPATCH-1962
> Project: Qpid Dispatch
>  Issue Type: Test
>Affects Versions: 1.15.0
>Reporter: Jiri Daněk
>Assignee: Jiri Daněk
>Priority: Major
> Fix For: 1.16.0
>
> Attachments: dispatch_leaking_agent.png
>
>
> There are some unfortunate suppressions in {{tests/lsan.supp}} at this moment:
> {code}
> leak:*libwebsockets*
> {code}
> This is way too broad. It suppresses leaks in Dispatch files, since it 
> matches e.g. {{src/http-libwebsockets.c}}.
> The stars at the beginning and end are actually assumed implicitly. If you do 
> not want substring match, you have to do ^foo$. LSan suppression format is 
> simplistic, very unlike Valgrind's.
> {code}
> leak:run_unit_tests.c
> {code}
> Same thing, any leaks revealed by running unit_tests get suppressed. This 
> suppression suppresses all leak traces that include run_unit_tests.c anywhere 
> in the stack. What't the point of running such tests under leak detector, 
> then?
> {code}
> leak:run_unit_tests.c
> leak:^libqpid-proton.so$
> {code}
> Same thing. The patterns suppress all leaks that include Python (or Proton) 
> anywhere in the stacktrace. That means there are huge blind spots where 
> dispatch leaks can hide. This is a weakness of the lsan.supp syntax (Valgrind 
> suppressions can be much more targeted and discerning).
> h3. Python leaks
> Leaks are known and there is ongoing effort to fight them: 
> https://bugs.python.org/issue1635741 (https://bugs.python.org/issue25302) and 
> https://www.python.org/dev/peps/pep-3121
> Here's valgrind suppression file from somebody who actually investigated the 
> Python leaks and identified the harmless ones: 
> https://github.com/libgit2/pygit2/blob/master/misc/valgrind-python.supp
> One example of a hidden leak in dispatch, which is revealed by making Python 
> suppressions more targetted:
> {code}
> 9: Direct leak of 56 byte(s) in 1 object(s) allocated from:
> 9: #0 0x7f78a3606e8f in __interceptor_malloc 
> (/nix/store/g40sl3zh3nv52vj0mrl4iki5iphh5ika-gcc-10.2.0-lib/lib/libasan.so.6+0xace8f)
> 9: #1 0x7f78a2d64afb in qd_malloc ../include/qpid/dispatch/ctools.h:229
> 9: #2 0x7f78a2d657da in qdr_core_subscribe 
> ../src/router_core/route_tables.c:149
> 9: #3 0x7f78a2c83072 in IoAdapter_init ../src/python_embedded.c:711
> 9: #4 0x7f78a2353a6c in type_call 
> (/nix/store/r85nxfnwiv45nbmf5yb60jj8ajim4m7w-python3-3.8.5/lib/libpython3.8.so.1.0+0x165a6c)
> {code}
> The problem is in
> {code}
> class Agent:
> ...
> def activate(self, address):
> ...
> self.io = IoAdapter(self.receive, address, 'L', '0', 
> TREATMENT_ANYCAST_CLOSEST)
> {code}
> IoAdapter refers to Agent (through the bound method

[jira] [Commented] (DISPATCH-1962) Make LSan suppressions more targeted and specific (within reason)

2021-05-06 Thread ASF GitHub Bot (Jira)



[ 
https://issues.apache.org/jira/browse/DISPATCH-1962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17340413#comment-17340413
 ] 

ASF GitHub Bot commented on DISPATCH-1962:
--

ganeshmurthy commented on a change in pull request #1191:
URL: https://github.com/apache/qpid-dispatch/pull/1191#discussion_r627736644



##
File path: src/router_core/router_core.c
##
@@ -236,6 +236,33 @@ void qdr_core_free(qdr_core_t *core)
 link = DEQ_HEAD(core->open_links);
 }
 
+// finalize modules while we can still submit new actions
+// this must happen after qdrc_endpoint_do_cleanup_CT calls
+qdr_modules_finalize(core);
+
+// discard any left over actions
+
+qdr_action_list_t  action_list;
+DEQ_MOVE(core->action_list, action_list);
+DEQ_APPEND(action_list, core->action_list_background);
+qdr_action_t *action = DEQ_HEAD(action_list);
+while (action) {
+DEQ_REMOVE_HEAD(action_list);
+action->action_handler(core, action, true);
+free_qdr_action_t(action);
+action = DEQ_HEAD(action_list);
+}
+
+// Drain the general work lists
+qdr_general_handler(core);
+
+sys_thread_free(core->thread);
+sys_cond_free(core->action_cond);
+sys_mutex_free(core->action_lock);
+sys_mutex_free(core->work_lock);
+sys_mutex_free(core->id_lock);
+qd_timer_free(core->work_timer);
+
 //
 // Clean up any qdr_delivery_cleanup_t's that are still left in the 
core->delivery_cleanup_list
 //

Review comment:
   Ok, I just realized that you are undoing part of your previous commit to 
main. But, yes, the reason I suggested what I suggested earlier is to make the 
diff clean and easy to understand.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Make LSan suppressions more targeted and specific (within reason)
> -
>
> Key: DISPATCH-1962
> URL: https://issues.apache.org/jira/browse/DISPATCH-1962
> Project: Qpid Dispatch
>  Issue Type: Test
>Affects Versions: 1.15.0
>Reporter: Jiri Daněk
>Assignee: Jiri Daněk
>Priority: Major
> Fix For: 1.16.0
>
> Attachments: dispatch_leaking_agent.png
>
>
> There are some unfortunate suppressions in {{tests/lsan.supp}} at this moment:
> {code}
> leak:*libwebsockets*
> {code}
> This is way too broad. It suppresses leaks in Dispatch files, since it 
> matches e.g. {{src/http-libwebsockets.c}}.
> The stars at the beginning and end are actually assumed implicitly. If you do 
> not want substring match, you have to do ^foo$. LSan suppression format is 
> simplistic, very unlike Valgrind's.
> {code}
> leak:run_unit_tests.c
> {code}
> Same thing, any leaks revealed by running unit_tests get suppressed. This 
> suppression suppresses all leak traces that include run_unit_tests.c anywhere 
> in the stack. What't the point of running such tests under leak detector, 
> then?
> {code}
> leak:run_unit_tests.c
> leak:^libqpid-proton.so$
> {code}
> Same thing. The patterns suppress all leaks that include Python (or Proton) 
> anywhere in the stacktrace. That means there are huge blind spots where 
> dispatch leaks can hide. This is a weakness of the lsan.supp syntax (Valgrind 
> suppressions can be much more targeted and discerning).
> h3. Python leaks
> Leaks are known and there is ongoing effort to fight them: 
> https://bugs.python.org/issue1635741 (https://bugs.python.org/issue25302) and 
> https://www.python.org/dev/peps/pep-3121
> Here's valgrind suppression file from somebody who actually investigated the 
> Python leaks and identified the harmless ones: 
> https://github.com/libgit2/pygit2/blob/master/misc/valgrind-python.supp
> One example of a hidden leak in dispatch, which is revealed by making Python 
> suppressions more targetted:
> {code}
> 9: Direct leak of 56 byte(s) in 1 object(s) allocated from:
> 9: #0 0x7f78a3606e8f in __interceptor_malloc 
> (/nix/store/g40sl3zh3nv52vj0mrl4iki5iphh5ika-gcc-10.2.0-lib/lib/libasan.so.6+0xace8f)
> 9: #1 0x7f78a2d64afb in qd_malloc ../include/qpid/dispatch/ctools.h:229
> 9: #2 0x7f78a2d657da in qdr_core_subscribe 
> ../src/router_core/route_tables.c:149
> 9: #3 0x7f78a2c83072 in IoAdapter_init ../src/python_embedded.c:711
> 9: #4 0x7f78a2353a6c in type_call 
> (/nix/store/r85nxfnwiv45nbmf5yb60jj8ajim4m7w-python3-3.8.5/lib/libpython3.8.so.1.0+0x165a6c)
> {code}
> The problem is in
> {code}
> class Agent:
> ...
> def activate(self, address):
> ...
> self.io = IoAdapter(self.receive, address, 'L', '0', 
> TREATMENT_ANYCAST_CLOSEST)
> {code}
>

[jira] [Commented] (DISPATCH-1962) Make LSan suppressions more targeted and specific (within reason)

2021-05-06 Thread ASF GitHub Bot (Jira)



[ 
https://issues.apache.org/jira/browse/DISPATCH-1962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17340408#comment-17340408
 ] 

ASF GitHub Bot commented on DISPATCH-1962:
--

jiridanek commented on a change in pull request #1191:
URL: https://github.com/apache/qpid-dispatch/pull/1191#discussion_r627728440



##
File path: src/router_core/router_core.c
##
@@ -236,6 +236,33 @@ void qdr_core_free(qdr_core_t *core)
 link = DEQ_HEAD(core->open_links);
 }
 
+// finalize modules while we can still submit new actions
+// this must happen after qdrc_endpoint_do_cleanup_CT calls
+qdr_modules_finalize(core);
+
+// discard any left over actions
+
+qdr_action_list_t  action_list;
+DEQ_MOVE(core->action_list, action_list);
+DEQ_APPEND(action_list, core->action_list_background);
+qdr_action_t *action = DEQ_HEAD(action_list);
+while (action) {
+DEQ_REMOVE_HEAD(action_list);
+action->action_handler(core, action, true);
+free_qdr_action_t(action);
+action = DEQ_HEAD(action_list);
+}
+
+// Drain the general work lists
+qdr_general_handler(core);
+
+sys_thread_free(core->thread);
+sys_cond_free(core->action_cond);
+sys_mutex_free(core->action_lock);
+sys_mutex_free(core->work_lock);
+sys_mutex_free(core->id_lock);
+qd_timer_free(core->work_timer);
+
 //
 // Clean up any qdr_delivery_cleanup_t's that are still left in the 
core->delivery_cleanup_list
 //

Review comment:
   @ganeshmurthy are you suggesting this to make the diff size smaller? or 
is there more to it?
   
   Here I am instead moving the qdr_modules_finalize (and the action discarding 
loop) in front of qdr_delivery_cleanup_t's cleanup. It made sense to me that 
way too. (And I moved the code that frees the locks back up, because 
previously, that was early on in the body of the function, before I started 
messing with it)




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Make LSan suppressions more targeted and specific (within reason)
> -
>
> Key: DISPATCH-1962
> URL: https://issues.apache.org/jira/browse/DISPATCH-1962
> Project: Qpid Dispatch
>  Issue Type: Test
>Affects Versions: 1.15.0
>Reporter: Jiri Daněk
>Assignee: Jiri Daněk
>Priority: Major
> Fix For: 1.16.0
>
> Attachments: dispatch_leaking_agent.png
>
>
> There are some unfortunate suppressions in {{tests/lsan.supp}} at this moment:
> {code}
> leak:*libwebsockets*
> {code}
> This is way too broad. It suppresses leaks in Dispatch files, since it 
> matches e.g. {{src/http-libwebsockets.c}}.
> The stars at the beginning and end are actually assumed implicitly. If you do 
> not want substring match, you have to do ^foo$. LSan suppression format is 
> simplistic, very unlike Valgrind's.
> {code}
> leak:run_unit_tests.c
> {code}
> Same thing, any leaks revealed by running unit_tests get suppressed. This 
> suppression suppresses all leak traces that include run_unit_tests.c anywhere 
> in the stack. What't the point of running such tests under leak detector, 
> then?
> {code}
> leak:run_unit_tests.c
> leak:^libqpid-proton.so$
> {code}
> Same thing. The patterns suppress all leaks that include Python (or Proton) 
> anywhere in the stacktrace. That means there are huge blind spots where 
> dispatch leaks can hide. This is a weakness of the lsan.supp syntax (Valgrind 
> suppressions can be much more targeted and discerning).
> h3. Python leaks
> Leaks are known and there is ongoing effort to fight them: 
> https://bugs.python.org/issue1635741 (https://bugs.python.org/issue25302) and 
> https://www.python.org/dev/peps/pep-3121
> Here's valgrind suppression file from somebody who actually investigated the 
> Python leaks and identified the harmless ones: 
> https://github.com/libgit2/pygit2/blob/master/misc/valgrind-python.supp
> One example of a hidden leak in dispatch, which is revealed by making Python 
> suppressions more targetted:
> {code}
> 9: Direct leak of 56 byte(s) in 1 object(s) allocated from:
> 9: #0 0x7f78a3606e8f in __interceptor_malloc 
> (/nix/store/g40sl3zh3nv52vj0mrl4iki5iphh5ika-gcc-10.2.0-lib/lib/libasan.so.6+0xace8f)
> 9: #1 0x7f78a2d64afb in qd_malloc ../include/qpid/dispatch/ctools.h:229
> 9: #2 0x7f78a2d657da in qdr_core_subscribe 
> ../src/router_core/route_tables.c:149
> 9: #3 0x7f78a2c83072 in IoAdapter_init ../src/python_embedded.c:711
> 9: #4 0x7f78a2353a6c in type_call 
>

[jira] [Commented] (DISPATCH-1962) Make LSan suppressions more targeted and specific (within reason)

2021-05-06 Thread ASF GitHub Bot (Jira)



[ 
https://issues.apache.org/jira/browse/DISPATCH-1962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17340388#comment-17340388
 ] 

ASF GitHub Bot commented on DISPATCH-1962:
--

ganeshmurthy commented on a change in pull request #1191:
URL: https://github.com/apache/qpid-dispatch/pull/1191#discussion_r627702199



##
File path: src/router_core/router_core.c
##
@@ -236,6 +236,33 @@ void qdr_core_free(qdr_core_t *core)
 link = DEQ_HEAD(core->open_links);
 }
 
+// finalize modules while we can still submit new actions
+// this must happen after qdrc_endpoint_do_cleanup_CT calls
+qdr_modules_finalize(core);
+
+// discard any left over actions
+
+qdr_action_list_t  action_list;
+DEQ_MOVE(core->action_list, action_list);
+DEQ_APPEND(action_list, core->action_list_background);
+qdr_action_t *action = DEQ_HEAD(action_list);
+while (action) {
+DEQ_REMOVE_HEAD(action_list);
+action->action_handler(core, action, true);
+free_qdr_action_t(action);
+action = DEQ_HEAD(action_list);
+}
+
+// Drain the general work lists
+qdr_general_handler(core);
+
+sys_thread_free(core->thread);
+sys_cond_free(core->action_cond);
+sys_mutex_free(core->action_lock);
+sys_mutex_free(core->work_lock);
+sys_mutex_free(core->id_lock);
+qd_timer_free(core->work_timer);
+
 //
 // Clean up any qdr_delivery_cleanup_t's that are still left in the 
core->delivery_cleanup_list
 //

Review comment:
   If qdr_modules_finalize must run before clean up of any 
qdr_delivery_cleanup_t's, why not just move the code that loops over 
delivery_cleanup_list, after the call to qdr_modules_finalize(core); ?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Make LSan suppressions more targeted and specific (within reason)
> -
>
> Key: DISPATCH-1962
> URL: https://issues.apache.org/jira/browse/DISPATCH-1962
> Project: Qpid Dispatch
>  Issue Type: Test
>Affects Versions: 1.15.0
>Reporter: Jiri Daněk
>Assignee: Jiri Daněk
>Priority: Major
> Fix For: 1.16.0
>
> Attachments: dispatch_leaking_agent.png
>
>
> There are some unfortunate suppressions in {{tests/lsan.supp}} at this moment:
> {code}
> leak:*libwebsockets*
> {code}
> This is way too broad. It suppresses leaks in Dispatch files, since it 
> matches e.g. {{src/http-libwebsockets.c}}.
> The stars at the beginning and end are actually assumed implicitly. If you do 
> not want substring match, you have to do ^foo$. LSan suppression format is 
> simplistic, very unlike Valgrind's.
> {code}
> leak:run_unit_tests.c
> {code}
> Same thing, any leaks revealed by running unit_tests get suppressed. This 
> suppression suppresses all leak traces that include run_unit_tests.c anywhere 
> in the stack. What't the point of running such tests under leak detector, 
> then?
> {code}
> leak:run_unit_tests.c
> leak:^libqpid-proton.so$
> {code}
> Same thing. The patterns suppress all leaks that include Python (or Proton) 
> anywhere in the stacktrace. That means there are huge blind spots where 
> dispatch leaks can hide. This is a weakness of the lsan.supp syntax (Valgrind 
> suppressions can be much more targeted and discerning).
> h3. Python leaks
> Leaks are known and there is ongoing effort to fight them: 
> https://bugs.python.org/issue1635741 (https://bugs.python.org/issue25302) and 
> https://www.python.org/dev/peps/pep-3121
> Here's valgrind suppression file from somebody who actually investigated the 
> Python leaks and identified the harmless ones: 
> https://github.com/libgit2/pygit2/blob/master/misc/valgrind-python.supp
> One example of a hidden leak in dispatch, which is revealed by making Python 
> suppressions more targetted:
> {code}
> 9: Direct leak of 56 byte(s) in 1 object(s) allocated from:
> 9: #0 0x7f78a3606e8f in __interceptor_malloc 
> (/nix/store/g40sl3zh3nv52vj0mrl4iki5iphh5ika-gcc-10.2.0-lib/lib/libasan.so.6+0xace8f)
> 9: #1 0x7f78a2d64afb in qd_malloc ../include/qpid/dispatch/ctools.h:229
> 9: #2 0x7f78a2d657da in qdr_core_subscribe 
> ../src/router_core/route_tables.c:149
> 9: #3 0x7f78a2c83072 in IoAdapter_init ../src/python_embedded.c:711
> 9: #4 0x7f78a2353a6c in type_call 
> (/nix/store/r85nxfnwiv45nbmf5yb60jj8ajim4m7w-python3-3.8.5/lib/libpython3.8.so.1.0+0x165a6c)
> {code}
> The problem is in
> {code}
> class Agent:
> ...
> def activate(self, address):
> ...
> self.io = IoAdapter(self.receive, address, 'L', '0', 
> TREATMENT_ANYCAST_CLOSEST)
> {code}
>

[jira] [Commented] (DISPATCH-1962) Make LSan suppressions more targeted and specific (within reason)

2021-05-06 Thread ASF subversion and git services (Jira)



[ 
https://issues.apache.org/jira/browse/DISPATCH-1962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17340288#comment-17340288
 ] 

ASF subversion and git services commented on DISPATCH-1962:
---

Commit c33dc697ad046a26ec92ca51bf091d9ec456c9e3 in qpid-dispatch's branch 
refs/heads/main from Jiri Daněk
[ https://gitbox.apache.org/repos/asf?p=qpid-dispatch.git;h=c33dc69 ]

DISPATCH-1962 Fix leak of qdr_core_subscribe (#1051)



> Make LSan suppressions more targeted and specific (within reason)
> -
>
> Key: DISPATCH-1962
> URL: https://issues.apache.org/jira/browse/DISPATCH-1962
> Project: Qpid Dispatch
>  Issue Type: Test
>Affects Versions: 1.15.0
>Reporter: Jiri Daněk
>Assignee: Jiri Daněk
>Priority: Major
> Fix For: 1.16.0
>
> Attachments: dispatch_leaking_agent.png
>
>
> There are some unfortunate suppressions in {{tests/lsan.supp}} at this moment:
> {code}
> leak:*libwebsockets*
> {code}
> This is way too broad. It suppresses leaks in Dispatch files, since it 
> matches e.g. {{src/http-libwebsockets.c}}.
> The stars at the beginning and end are actually assumed implicitly. If you do 
> not want substring match, you have to do ^foo$. LSan suppression format is 
> simplistic, very unlike Valgrind's.
> {code}
> leak:run_unit_tests.c
> {code}
> Same thing, any leaks revealed by running unit_tests get suppressed. This 
> suppression suppresses all leak traces that include run_unit_tests.c anywhere 
> in the stack. What't the point of running such tests under leak detector, 
> then?
> {code}
> leak:run_unit_tests.c
> leak:^libqpid-proton.so$
> {code}
> Same thing. The patterns suppress all leaks that include Python (or Proton) 
> anywhere in the stacktrace. That means there are huge blind spots where 
> dispatch leaks can hide. This is a weakness of the lsan.supp syntax (Valgrind 
> suppressions can be much more targeted and discerning).
> h3. Python leaks
> Leaks are known and there is ongoing effort to fight them: 
> https://bugs.python.org/issue1635741 (https://bugs.python.org/issue25302) and 
> https://www.python.org/dev/peps/pep-3121
> Here's valgrind suppression file from somebody who actually investigated the 
> Python leaks and identified the harmless ones: 
> https://github.com/libgit2/pygit2/blob/master/misc/valgrind-python.supp
> One example of a hidden leak in dispatch, which is revealed by making Python 
> suppressions more targetted:
> {code}
> 9: Direct leak of 56 byte(s) in 1 object(s) allocated from:
> 9: #0 0x7f78a3606e8f in __interceptor_malloc 
> (/nix/store/g40sl3zh3nv52vj0mrl4iki5iphh5ika-gcc-10.2.0-lib/lib/libasan.so.6+0xace8f)
> 9: #1 0x7f78a2d64afb in qd_malloc ../include/qpid/dispatch/ctools.h:229
> 9: #2 0x7f78a2d657da in qdr_core_subscribe 
> ../src/router_core/route_tables.c:149
> 9: #3 0x7f78a2c83072 in IoAdapter_init ../src/python_embedded.c:711
> 9: #4 0x7f78a2353a6c in type_call 
> (/nix/store/r85nxfnwiv45nbmf5yb60jj8ajim4m7w-python3-3.8.5/lib/libpython3.8.so.1.0+0x165a6c)
> {code}
> The problem is in
> {code}
> class Agent:
> ...
> def activate(self, address):
> ...
> self.io = IoAdapter(self.receive, address, 'L', '0', 
> TREATMENT_ANYCAST_CLOSEST)
> {code}
> IoAdapter refers to Agent (through the bound method reference self.receive) 
> and Agent refers to IoAdapter (through property self.io). Since IoAdapter is 
> implemented in C and does not implement support for Python's cyclic GC, there 
> is no way to break the cycle.
> Heap dump in attachment. The bound method is at the top of the picture. 
> (Ignore the Mock objects, I was trying to simplify the picture while not 
> getting crashes due to too much meddling).
> h3. Random observations
> It is possible to build special Debug build of Python, which has tools to 
> detect leaks, asserts to prevent negative refcounts, etc. 
> https://pythonextensionpatterns.readthedocs.io/en/latest/debugging/debug_python.html#debug-version-of-python-memory-alloc-label
> Use the following to detect python leaks (instead of valgrind)
> https://docs.python.org/3/library/tracemalloc.html
> Use https://pypi.org/project/objgraph (with graphviz) to view heap object 
> trees. The following renders the picture as a png under /tmp and prints the 
> path to stdout.
> {code}
> int ret = PyRun_SimpleString("import objgraph; 
> objgraph.show_backrefs(config.global_agent, max_depth=10)\n\n");
> PyErr_PrintEx(0);
> assert(ret == 0);
> {code}
> h2. Tools
> h3. CPyChecker
> https://dmalcolm.fedorapeople.org/presentations/PyCon-US-2013/PyCon-US-2013-dmalcolm-StaticAnalysis.html#(38)
> https://emptysqua.re/blog/analyzing-python-c-extensions-with-cpychecker/
> https://nedbatchelder.com/blog/201502/cpychecker.html
> h3. Pungi
>

[jira] [Commented] (DISPATCH-1962) Make LSan suppressions more targeted and specific (within reason)

2021-03-30 Thread ASF subversion and git services (Jira)



[ 
https://issues.apache.org/jira/browse/DISPATCH-1962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17311904#comment-17311904
 ] 

ASF subversion and git services commented on DISPATCH-1962:
---

Commit 8530155f394d92015c50bba4001f190bbffcb073 in qpid-dispatch's branch 
refs/heads/main from Jiri Daněk
[ https://gitbox.apache.org/repos/asf?p=qpid-dispatch.git;h=8530155 ]

DISPATCH-1962 Fix leak in run_unit_tests.c (#1049)



> Make LSan suppressions more targeted and specific (within reason)
> -
>
> Key: DISPATCH-1962
> URL: https://issues.apache.org/jira/browse/DISPATCH-1962
> Project: Qpid Dispatch
>  Issue Type: Test
>Affects Versions: 1.15.0
>Reporter: Jiri Daněk
>Priority: Major
> Attachments: dispatch_leaking_agent.png
>
>
> There are some unfortunate suppressions in {{tests/lsan.supp}} at this moment:
> {code}
> leak:*libwebsockets*
> {code}
> This is way too broad. It suppresses leaks in Dispatch files, since it 
> matches e.g. {{src/http-libwebsockets.c}}.
> The stars at the beginning and end are actually assumed implicitly. If you do 
> not want substring match, you have to do ^foo$. LSan suppression format is 
> simplistic, very unlike Valgrind's.
> {code}
> leak:run_unit_tests.c
> {code}
> Same thing, any leaks revealed by running unit_tests get suppressed. This 
> suppression suppresses all leak traces that include run_unit_tests.c anywhere 
> in the stack. What't the point of running such tests under leak detector, 
> then?
> {code}
> leak:run_unit_tests.c
> leak:^libqpid-proton.so$
> {code}
> Same thing. The patterns suppress all leaks that include Python (or Proton) 
> anywhere in the stacktrace. That means there are huge blind spots where 
> dispatch leaks can hide. This is a weakness of the lsan.supp syntax (Valgrind 
> suppressions can be much more targeted and discerning).
> h3. Python leaks
> Leaks are known and there is ongoing effort to fight them: 
> https://bugs.python.org/issue1635741 (https://bugs.python.org/issue25302) and 
> https://www.python.org/dev/peps/pep-3121
> Here's valgrind suppression file from somebody who actually investigated the 
> Python leaks and identified the harmless ones: 
> https://github.com/libgit2/pygit2/blob/master/misc/valgrind-python.supp
> One example of a hidden leak in dispatch, which is revealed by making Python 
> suppressions more targetted:
> {code}
> 9: Direct leak of 56 byte(s) in 1 object(s) allocated from:
> 9: #0 0x7f78a3606e8f in __interceptor_malloc 
> (/nix/store/g40sl3zh3nv52vj0mrl4iki5iphh5ika-gcc-10.2.0-lib/lib/libasan.so.6+0xace8f)
> 9: #1 0x7f78a2d64afb in qd_malloc ../include/qpid/dispatch/ctools.h:229
> 9: #2 0x7f78a2d657da in qdr_core_subscribe 
> ../src/router_core/route_tables.c:149
> 9: #3 0x7f78a2c83072 in IoAdapter_init ../src/python_embedded.c:711
> 9: #4 0x7f78a2353a6c in type_call 
> (/nix/store/r85nxfnwiv45nbmf5yb60jj8ajim4m7w-python3-3.8.5/lib/libpython3.8.so.1.0+0x165a6c)
> {code}
> The problem is in
> {code}
> class Agent:
> ...
> def activate(self, address):
> ...
> self.io = IoAdapter(self.receive, address, 'L', '0', 
> TREATMENT_ANYCAST_CLOSEST)
> {code}
> IoAdapter refers to Agent (through the bound method reference self.receive) 
> and Agent refers to IoAdapter (through property self.io). Since IoAdapter is 
> implemented in C and does not implement support for Python's cyclic GC, there 
> is no way to break the cycle.
> Heap dump in attachment. The bound method is at the top of the picture. 
> (Ignore the Mock objects, I was trying to simplify the picture while not 
> getting crashes due to too much meddling).
> h3. Random observations
> It is possible to build special Debug build of Python, which has tools to 
> detect leaks, asserts to prevent negative refcounts, etc. 
> https://pythonextensionpatterns.readthedocs.io/en/latest/debugging/debug_python.html#debug-version-of-python-memory-alloc-label
> Use the following to detect python leaks (instead of valgrind)
> https://docs.python.org/3/library/tracemalloc.html
> Use https://pypi.org/project/objgraph (with graphviz) to view heap object 
> trees. The following renders the picture as a png under /tmp and prints the 
> path to stdout.
> {code}
> int ret = PyRun_SimpleString("import objgraph; 
> objgraph.show_backrefs(config.global_agent, max_depth=10)\n\n");
> PyErr_PrintEx(0);
> assert(ret == 0);
> {code}
> h2. Tools
> h3. CPyChecker
> https://dmalcolm.fedorapeople.org/presentations/PyCon-US-2013/PyCon-US-2013-dmalcolm-StaticAnalysis.html#(38)
> https://emptysqua.re/blog/analyzing-python-c-extensions-with-cpychecker/
> https://nedbatchelder.com/blog/201502/cpychecker.html
> h3. Pungi
> http://www.cse.psu.edu/~gxt29/papers/refcount.pdf
> h3. Use CFFI or similar, instead of writing a

[jira] [Commented] (DISPATCH-1962) Make LSan suppressions more targeted and specific (within reason)

2021-03-26 Thread ASF subversion and git services (Jira)



[ 
https://issues.apache.org/jira/browse/DISPATCH-1962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17309498#comment-17309498
 ] 

ASF subversion and git services commented on DISPATCH-1962:
---

Commit 8530155f394d92015c50bba4001f190bbffcb073 in qpid-dispatch's branch 
refs/heads/master from Jiri Daněk
[ https://gitbox.apache.org/repos/asf?p=qpid-dispatch.git;h=8530155 ]

DISPATCH-1962 Fix leak in run_unit_tests.c (#1049)



> Make LSan suppressions more targeted and specific (within reason)
> -
>
> Key: DISPATCH-1962
> URL: https://issues.apache.org/jira/browse/DISPATCH-1962
> Project: Qpid Dispatch
>  Issue Type: Test
>Affects Versions: 1.15.0
>Reporter: Jiri Daněk
>Priority: Major
> Attachments: dispatch_leaking_agent.png
>
>
> There are some unfortunate suppressions in {{tests/lsan.supp}} at this moment:
> {code}
> leak:*libwebsockets*
> {code}
> This is way too broad. It suppresses leaks in Dispatch files, since it 
> matches e.g. {{src/http-libwebsockets.c}}.
> The stars at the beginning and end are actually assumed implicitly. If you do 
> not want substring match, you have to do ^foo$. LSan suppression format is 
> simplistic, very unlike Valgrind's.
> {code}
> leak:run_unit_tests.c
> {code}
> Same thing, any leaks revealed by running unit_tests get suppressed. This 
> suppression suppresses all leak traces that include run_unit_tests.c anywhere 
> in the stack. What't the point of running such tests under leak detector, 
> then?
> {code}
> leak:run_unit_tests.c
> leak:^libqpid-proton.so$
> {code}
> Same thing. The patterns suppress all leaks that include Python (or Proton) 
> anywhere in the stacktrace. That means there are huge blind spots where 
> dispatch leaks can hide. This is a weakness of the lsan.supp syntax (Valgrind 
> suppressions can be much more targeted and discerning).
> h3. Python leaks
> Leaks are known and there is ongoing effort to fight them: 
> https://bugs.python.org/issue1635741 (https://bugs.python.org/issue25302) and 
> https://www.python.org/dev/peps/pep-3121
> Here's valgrind suppression file from somebody who actually investigated the 
> Python leaks and identified the harmless ones: 
> https://github.com/libgit2/pygit2/blob/master/misc/valgrind-python.supp
> One example of a hidden leak in dispatch, which is revealed by making Python 
> suppressions more targetted:
> {code}
> 9: Direct leak of 56 byte(s) in 1 object(s) allocated from:
> 9: #0 0x7f78a3606e8f in __interceptor_malloc 
> (/nix/store/g40sl3zh3nv52vj0mrl4iki5iphh5ika-gcc-10.2.0-lib/lib/libasan.so.6+0xace8f)
> 9: #1 0x7f78a2d64afb in qd_malloc ../include/qpid/dispatch/ctools.h:229
> 9: #2 0x7f78a2d657da in qdr_core_subscribe 
> ../src/router_core/route_tables.c:149
> 9: #3 0x7f78a2c83072 in IoAdapter_init ../src/python_embedded.c:711
> 9: #4 0x7f78a2353a6c in type_call 
> (/nix/store/r85nxfnwiv45nbmf5yb60jj8ajim4m7w-python3-3.8.5/lib/libpython3.8.so.1.0+0x165a6c)
> {code}
> The problem is in
> {code}
> class Agent:
> ...
> def activate(self, address):
> ...
> self.io = IoAdapter(self.receive, address, 'L', '0', 
> TREATMENT_ANYCAST_CLOSEST)
> {code}
> IoAdapter refers to Agent (through the bound method reference self.receive) 
> and Agent refers to IoAdapter (through property self.io). Since IoAdapter is 
> implemented in C and does not implement support for Python's cyclic GC, there 
> is no way to break the cycle.
> Heap dump in attachment. The bound method is at the top of the picture. 
> (Ignore the Mock objects, I was trying to simplify the picture while not 
> getting crashes due to too much meddling).
> h3. Random observations
> It is possible to build special Debug build of Python, which has tools to 
> detect leaks, asserts to prevent negative refcounts, etc. 
> https://pythonextensionpatterns.readthedocs.io/en/latest/debugging/debug_python.html#debug-version-of-python-memory-alloc-label
> Use the following to detect python leaks (instead of valgrind)
> https://docs.python.org/3/library/tracemalloc.html
> Use https://pypi.org/project/objgraph (with graphviz) to view heap object 
> trees. The following renders the picture as a png under /tmp and prints the 
> path to stdout.
> {code}
> int ret = PyRun_SimpleString("import objgraph; 
> objgraph.show_backrefs(config.global_agent, max_depth=10)\n\n");
> PyErr_PrintEx(0);
> assert(ret == 0);
> {code}
> h2. Tools
> h3. CPyChecker
> https://dmalcolm.fedorapeople.org/presentations/PyCon-US-2013/PyCon-US-2013-dmalcolm-StaticAnalysis.html#(38)
> https://emptysqua.re/blog/analyzing-python-c-extensions-with-cpychecker/
> https://nedbatchelder.com/blog/201502/cpychecker.html
> h3. Pungi
> http://www.cse.psu.edu/~gxt29/papers/refcount.pdf
> h3. Use CFFI or similar, instead of writing

[jira] [Commented] (DISPATCH-1962) Make LSan suppressions more targeted and specific (within reason)

2021-03-26 Thread ASF GitHub Bot (Jira)



[ 
https://issues.apache.org/jira/browse/DISPATCH-1962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17309499#comment-17309499
 ] 

ASF GitHub Bot commented on DISPATCH-1962:
--

jiridanek merged pull request #1049:
URL: https://github.com/apache/qpid-dispatch/pull/1049


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Make LSan suppressions more targeted and specific (within reason)
> -
>
> Key: DISPATCH-1962
> URL: https://issues.apache.org/jira/browse/DISPATCH-1962
> Project: Qpid Dispatch
>  Issue Type: Test
>Affects Versions: 1.15.0
>Reporter: Jiri Daněk
>Priority: Major
> Attachments: dispatch_leaking_agent.png
>
>
> There are some unfortunate suppressions in {{tests/lsan.supp}} at this moment:
> {code}
> leak:*libwebsockets*
> {code}
> This is way too broad. It suppresses leaks in Dispatch files, since it 
> matches e.g. {{src/http-libwebsockets.c}}.
> The stars at the beginning and end are actually assumed implicitly. If you do 
> not want substring match, you have to do ^foo$. LSan suppression format is 
> simplistic, very unlike Valgrind's.
> {code}
> leak:run_unit_tests.c
> {code}
> Same thing, any leaks revealed by running unit_tests get suppressed. This 
> suppression suppresses all leak traces that include run_unit_tests.c anywhere 
> in the stack. What't the point of running such tests under leak detector, 
> then?
> {code}
> leak:run_unit_tests.c
> leak:^libqpid-proton.so$
> {code}
> Same thing. The patterns suppress all leaks that include Python (or Proton) 
> anywhere in the stacktrace. That means there are huge blind spots where 
> dispatch leaks can hide. This is a weakness of the lsan.supp syntax (Valgrind 
> suppressions can be much more targeted and discerning).
> h3. Python leaks
> Leaks are known and there is ongoing effort to fight them: 
> https://bugs.python.org/issue1635741 (https://bugs.python.org/issue25302) and 
> https://www.python.org/dev/peps/pep-3121
> Here's valgrind suppression file from somebody who actually investigated the 
> Python leaks and identified the harmless ones: 
> https://github.com/libgit2/pygit2/blob/master/misc/valgrind-python.supp
> One example of a hidden leak in dispatch, which is revealed by making Python 
> suppressions more targetted:
> {code}
> 9: Direct leak of 56 byte(s) in 1 object(s) allocated from:
> 9: #0 0x7f78a3606e8f in __interceptor_malloc 
> (/nix/store/g40sl3zh3nv52vj0mrl4iki5iphh5ika-gcc-10.2.0-lib/lib/libasan.so.6+0xace8f)
> 9: #1 0x7f78a2d64afb in qd_malloc ../include/qpid/dispatch/ctools.h:229
> 9: #2 0x7f78a2d657da in qdr_core_subscribe 
> ../src/router_core/route_tables.c:149
> 9: #3 0x7f78a2c83072 in IoAdapter_init ../src/python_embedded.c:711
> 9: #4 0x7f78a2353a6c in type_call 
> (/nix/store/r85nxfnwiv45nbmf5yb60jj8ajim4m7w-python3-3.8.5/lib/libpython3.8.so.1.0+0x165a6c)
> {code}
> The problem is in
> {code}
> class Agent:
> ...
> def activate(self, address):
> ...
> self.io = IoAdapter(self.receive, address, 'L', '0', 
> TREATMENT_ANYCAST_CLOSEST)
> {code}
> IoAdapter refers to Agent (through the bound method reference self.receive) 
> and Agent refers to IoAdapter (through property self.io). Since IoAdapter is 
> implemented in C and does not implement support for Python's cyclic GC, there 
> is no way to break the cycle.
> Heap dump in attachment. The bound method is at the top of the picture. 
> (Ignore the Mock objects, I was trying to simplify the picture while not 
> getting crashes due to too much meddling).
> h3. Random observations
> It is possible to build special Debug build of Python, which has tools to 
> detect leaks, asserts to prevent negative refcounts, etc. 
> https://pythonextensionpatterns.readthedocs.io/en/latest/debugging/debug_python.html#debug-version-of-python-memory-alloc-label
> Use the following to detect python leaks (instead of valgrind)
> https://docs.python.org/3/library/tracemalloc.html
> Use https://pypi.org/project/objgraph (with graphviz) to view heap object 
> trees. The following renders the picture as a png under /tmp and prints the 
> path to stdout.
> {code}
> int ret = PyRun_SimpleString("import objgraph; 
> objgraph.show_backrefs(config.global_agent, max_depth=10)\n\n");
> PyErr_PrintEx(0);
> assert(ret == 0);
> {code}
> h2. Tools
> h3. CPyChecker
> https://dmalcolm.fedorapeople.org/presentations/PyCon-US-2013/PyCon-US-2013-dmalcolm-StaticAnalysis.html#(38)
> https://emptysqua.re/blog/analyzing-python-c-extensions-with-cpychecker/
> https://nedbatchelder.com/blog/201502/cpychecker.html
> h3. Pungi
>

[jira] [Commented] (DISPATCH-1962) Make LSan suppressions more targeted and specific (within reason)

2021-03-22 Thread ASF subversion and git services (Jira)



[ 
https://issues.apache.org/jira/browse/DISPATCH-1962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17306593#comment-17306593
 ] 

ASF subversion and git services commented on DISPATCH-1962:
---

Commit 9262f7a801c493dacf220b2d52e9da4a2d59e53c in qpid-dispatch's branch 
refs/heads/master from Jiri Daněk
[ https://gitbox.apache.org/repos/asf?p=qpid-dispatch.git;h=9262f7a ]

DISPATCH-1962 Fix leak from libwebsockets.so in unit_tests (#1080)



> Make LSan suppressions more targeted and specific (within reason)
> -
>
> Key: DISPATCH-1962
> URL: https://issues.apache.org/jira/browse/DISPATCH-1962
> Project: Qpid Dispatch
>  Issue Type: Test
>Affects Versions: 1.15.0
>Reporter: Jiri Daněk
>Priority: Major
> Attachments: dispatch_leaking_agent.png
>
>
> There are some unfortunate suppressions in {{tests/lsan.supp}} at this moment:
> {code}
> leak:*libwebsockets*
> {code}
> This is way too broad. It suppresses leaks in Dispatch files, since it 
> matches e.g. {{src/http-libwebsockets.c}}.
> The stars at the beginning and end are actually assumed implicitly. If you do 
> not want substring match, you have to do ^foo$. LSan suppression format is 
> simplistic, very unlike Valgrind's.
> {code}
> leak:run_unit_tests.c
> {code}
> Same thing, any leaks revealed by running unit_tests get suppressed. This 
> suppression suppresses all leak traces that include run_unit_tests.c anywhere 
> in the stack. What't the point of running such tests under leak detector, 
> then?
> {code}
> leak:run_unit_tests.c
> leak:^libqpid-proton.so$
> {code}
> Same thing. The patterns suppress all leaks that include Python (or Proton) 
> anywhere in the stacktrace. That means there are huge blind spots where 
> dispatch leaks can hide. This is a weakness of the lsan.supp syntax (Valgrind 
> suppressions can be much more targeted and discerning).
> h3. Python leaks
> Leaks are known and there is ongoing effort to fight them: 
> https://bugs.python.org/issue1635741 (https://bugs.python.org/issue25302) and 
> https://www.python.org/dev/peps/pep-3121
> Here's valgrind suppression file from somebody who actually investigated the 
> Python leaks and identified the harmless ones: 
> https://github.com/libgit2/pygit2/blob/master/misc/valgrind-python.supp
> One example of a hidden leak in dispatch, which is revealed by making Python 
> suppressions more targetted:
> {code}
> 9: Direct leak of 56 byte(s) in 1 object(s) allocated from:
> 9: #0 0x7f78a3606e8f in __interceptor_malloc 
> (/nix/store/g40sl3zh3nv52vj0mrl4iki5iphh5ika-gcc-10.2.0-lib/lib/libasan.so.6+0xace8f)
> 9: #1 0x7f78a2d64afb in qd_malloc ../include/qpid/dispatch/ctools.h:229
> 9: #2 0x7f78a2d657da in qdr_core_subscribe 
> ../src/router_core/route_tables.c:149
> 9: #3 0x7f78a2c83072 in IoAdapter_init ../src/python_embedded.c:711
> 9: #4 0x7f78a2353a6c in type_call 
> (/nix/store/r85nxfnwiv45nbmf5yb60jj8ajim4m7w-python3-3.8.5/lib/libpython3.8.so.1.0+0x165a6c)
> {code}
> The problem is in
> {code}
> class Agent:
> ...
> def activate(self, address):
> ...
> self.io = IoAdapter(self.receive, address, 'L', '0', 
> TREATMENT_ANYCAST_CLOSEST)
> {code}
> IoAdapter refers to Agent (through the bound method reference self.receive) 
> and Agent refers to IoAdapter (through property self.io). Since IoAdapter is 
> implemented in C and does not implement support for Python's cyclic GC, there 
> is no way to break the cycle.
> Heap dump in attachment. The bound method is at the top of the picture. 
> (Ignore the Mock objects, I was trying to simplify the picture while not 
> getting crashes due to too much meddling).
> h3. Random observations
> It is possible to build special Debug build of Python, which has tools to 
> detect leaks, asserts to prevent negative refcounts, etc. 
> https://pythonextensionpatterns.readthedocs.io/en/latest/debugging/debug_python.html#debug-version-of-python-memory-alloc-label
> Use the following to detect python leaks (instead of valgrind)
> https://docs.python.org/3/library/tracemalloc.html
> Use https://pypi.org/project/objgraph (with graphviz) to view heap object 
> trees. The following renders the picture as a png under /tmp and prints the 
> path to stdout.
> {code}
> int ret = PyRun_SimpleString("import objgraph; 
> objgraph.show_backrefs(config.global_agent, max_depth=10)\n\n");
> PyErr_PrintEx(0);
> assert(ret == 0);
> {code}
> h2. Tools
> h3. CPyChecker
> https://dmalcolm.fedorapeople.org/presentations/PyCon-US-2013/PyCon-US-2013-dmalcolm-StaticAnalysis.html#(38)
> https://emptysqua.re/blog/analyzing-python-c-extensions-with-cpychecker/
> https://nedbatchelder.com/blog/201502/cpychecker.html
> h3. Pungi
> http://www.cse.psu.edu/~gxt29/papers/refcount.pdf
> h3. Use CFFI or similar,

[jira] [Commented] (DISPATCH-1962) Make LSan suppressions more targeted and specific (within reason)

2021-03-22 Thread ASF GitHub Bot (Jira)



[ 
https://issues.apache.org/jira/browse/DISPATCH-1962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17306594#comment-17306594
 ] 

ASF GitHub Bot commented on DISPATCH-1962:
--

jiridanek merged pull request #1080:
URL: https://github.com/apache/qpid-dispatch/pull/1080


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Make LSan suppressions more targeted and specific (within reason)
> -
>
> Key: DISPATCH-1962
> URL: https://issues.apache.org/jira/browse/DISPATCH-1962
> Project: Qpid Dispatch
>  Issue Type: Test
>Affects Versions: 1.15.0
>Reporter: Jiri Daněk
>Priority: Major
> Attachments: dispatch_leaking_agent.png
>
>
> There are some unfortunate suppressions in {{tests/lsan.supp}} at this moment:
> {code}
> leak:*libwebsockets*
> {code}
> This is way too broad. It suppresses leaks in Dispatch files, since it 
> matches e.g. {{src/http-libwebsockets.c}}.
> The stars at the beginning and end are actually assumed implicitly. If you do 
> not want substring match, you have to do ^foo$. LSan suppression format is 
> simplistic, very unlike Valgrind's.
> {code}
> leak:run_unit_tests.c
> {code}
> Same thing, any leaks revealed by running unit_tests get suppressed. This 
> suppression suppresses all leak traces that include run_unit_tests.c anywhere 
> in the stack. What't the point of running such tests under leak detector, 
> then?
> {code}
> leak:run_unit_tests.c
> leak:^libqpid-proton.so$
> {code}
> Same thing. The patterns suppress all leaks that include Python (or Proton) 
> anywhere in the stacktrace. That means there are huge blind spots where 
> dispatch leaks can hide. This is a weakness of the lsan.supp syntax (Valgrind 
> suppressions can be much more targeted and discerning).
> h3. Python leaks
> Leaks are known and there is ongoing effort to fight them: 
> https://bugs.python.org/issue1635741 (https://bugs.python.org/issue25302) and 
> https://www.python.org/dev/peps/pep-3121
> Here's valgrind suppression file from somebody who actually investigated the 
> Python leaks and identified the harmless ones: 
> https://github.com/libgit2/pygit2/blob/master/misc/valgrind-python.supp
> One example of a hidden leak in dispatch, which is revealed by making Python 
> suppressions more targetted:
> {code}
> 9: Direct leak of 56 byte(s) in 1 object(s) allocated from:
> 9: #0 0x7f78a3606e8f in __interceptor_malloc 
> (/nix/store/g40sl3zh3nv52vj0mrl4iki5iphh5ika-gcc-10.2.0-lib/lib/libasan.so.6+0xace8f)
> 9: #1 0x7f78a2d64afb in qd_malloc ../include/qpid/dispatch/ctools.h:229
> 9: #2 0x7f78a2d657da in qdr_core_subscribe 
> ../src/router_core/route_tables.c:149
> 9: #3 0x7f78a2c83072 in IoAdapter_init ../src/python_embedded.c:711
> 9: #4 0x7f78a2353a6c in type_call 
> (/nix/store/r85nxfnwiv45nbmf5yb60jj8ajim4m7w-python3-3.8.5/lib/libpython3.8.so.1.0+0x165a6c)
> {code}
> The problem is in
> {code}
> class Agent:
> ...
> def activate(self, address):
> ...
> self.io = IoAdapter(self.receive, address, 'L', '0', 
> TREATMENT_ANYCAST_CLOSEST)
> {code}
> IoAdapter refers to Agent (through the bound method reference self.receive) 
> and Agent refers to IoAdapter (through property self.io). Since IoAdapter is 
> implemented in C and does not implement support for Python's cyclic GC, there 
> is no way to break the cycle.
> Heap dump in attachment. The bound method is at the top of the picture. 
> (Ignore the Mock objects, I was trying to simplify the picture while not 
> getting crashes due to too much meddling).
> h3. Random observations
> It is possible to build special Debug build of Python, which has tools to 
> detect leaks, asserts to prevent negative refcounts, etc. 
> https://pythonextensionpatterns.readthedocs.io/en/latest/debugging/debug_python.html#debug-version-of-python-memory-alloc-label
> Use the following to detect python leaks (instead of valgrind)
> https://docs.python.org/3/library/tracemalloc.html
> Use https://pypi.org/project/objgraph (with graphviz) to view heap object 
> trees. The following renders the picture as a png under /tmp and prints the 
> path to stdout.
> {code}
> int ret = PyRun_SimpleString("import objgraph; 
> objgraph.show_backrefs(config.global_agent, max_depth=10)\n\n");
> PyErr_PrintEx(0);
> assert(ret == 0);
> {code}
> h2. Tools
> h3. CPyChecker
> https://dmalcolm.fedorapeople.org/presentations/PyCon-US-2013/PyCon-US-2013-dmalcolm-StaticAnalysis.html#(38)
> https://emptysqua.re/blog/analyzing-python-c-extensions-with-cpychecker/
> https://nedbatchelder.com/blog/201502/cpychecker.html
> h3. Pungi
>

[jira] [Commented] (DISPATCH-1962) Make LSan suppressions more targeted and specific (within reason)

2021-03-22 Thread ASF GitHub Bot (Jira)



[ 
https://issues.apache.org/jira/browse/DISPATCH-1962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17306570#comment-17306570
 ] 

ASF GitHub Bot commented on DISPATCH-1962:
--

jiridanek commented on pull request #1080:
URL: https://github.com/apache/qpid-dispatch/pull/1080#issuecomment-804388302


   It is possible to get https://issues.apache.org/jira/browse/DISPATCH-2013 
_Http1AdaptorManagementTest fails with AssertionError: 0 != 1 : HTTP connection 
not deleted_ with this PR. It looks to me that it's a flaky failure not caused 
by the PR. I am rerunning GHA tests, just to get another datapoint.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Make LSan suppressions more targeted and specific (within reason)
> -
>
> Key: DISPATCH-1962
> URL: https://issues.apache.org/jira/browse/DISPATCH-1962
> Project: Qpid Dispatch
>  Issue Type: Test
>Affects Versions: 1.15.0
>Reporter: Jiri Daněk
>Priority: Major
> Attachments: dispatch_leaking_agent.png
>
>
> There are some unfortunate suppressions in {{tests/lsan.supp}} at this moment:
> {code}
> leak:*libwebsockets*
> {code}
> This is way too broad. It suppresses leaks in Dispatch files, since it 
> matches e.g. {{src/http-libwebsockets.c}}.
> The stars at the beginning and end are actually assumed implicitly. If you do 
> not want substring match, you have to do ^foo$. LSan suppression format is 
> simplistic, very unlike Valgrind's.
> {code}
> leak:run_unit_tests.c
> {code}
> Same thing, any leaks revealed by running unit_tests get suppressed. This 
> suppression suppresses all leak traces that include run_unit_tests.c anywhere 
> in the stack. What't the point of running such tests under leak detector, 
> then?
> {code}
> leak:run_unit_tests.c
> leak:^libqpid-proton.so$
> {code}
> Same thing. The patterns suppress all leaks that include Python (or Proton) 
> anywhere in the stacktrace. That means there are huge blind spots where 
> dispatch leaks can hide. This is a weakness of the lsan.supp syntax (Valgrind 
> suppressions can be much more targeted and discerning).
> h3. Python leaks
> Leaks are known and there is ongoing effort to fight them: 
> https://bugs.python.org/issue1635741 (https://bugs.python.org/issue25302) and 
> https://www.python.org/dev/peps/pep-3121
> Here's valgrind suppression file from somebody who actually investigated the 
> Python leaks and identified the harmless ones: 
> https://github.com/libgit2/pygit2/blob/master/misc/valgrind-python.supp
> One example of a hidden leak in dispatch, which is revealed by making Python 
> suppressions more targetted:
> {code}
> 9: Direct leak of 56 byte(s) in 1 object(s) allocated from:
> 9: #0 0x7f78a3606e8f in __interceptor_malloc 
> (/nix/store/g40sl3zh3nv52vj0mrl4iki5iphh5ika-gcc-10.2.0-lib/lib/libasan.so.6+0xace8f)
> 9: #1 0x7f78a2d64afb in qd_malloc ../include/qpid/dispatch/ctools.h:229
> 9: #2 0x7f78a2d657da in qdr_core_subscribe 
> ../src/router_core/route_tables.c:149
> 9: #3 0x7f78a2c83072 in IoAdapter_init ../src/python_embedded.c:711
> 9: #4 0x7f78a2353a6c in type_call 
> (/nix/store/r85nxfnwiv45nbmf5yb60jj8ajim4m7w-python3-3.8.5/lib/libpython3.8.so.1.0+0x165a6c)
> {code}
> The problem is in
> {code}
> class Agent:
> ...
> def activate(self, address):
> ...
> self.io = IoAdapter(self.receive, address, 'L', '0', 
> TREATMENT_ANYCAST_CLOSEST)
> {code}
> IoAdapter refers to Agent (through the bound method reference self.receive) 
> and Agent refers to IoAdapter (through property self.io). Since IoAdapter is 
> implemented in C and does not implement support for Python's cyclic GC, there 
> is no way to break the cycle.
> Heap dump in attachment. The bound method is at the top of the picture. 
> (Ignore the Mock objects, I was trying to simplify the picture while not 
> getting crashes due to too much meddling).
> h3. Random observations
> It is possible to build special Debug build of Python, which has tools to 
> detect leaks, asserts to prevent negative refcounts, etc. 
> https://pythonextensionpatterns.readthedocs.io/en/latest/debugging/debug_python.html#debug-version-of-python-memory-alloc-label
> Use the following to detect python leaks (instead of valgrind)
> https://docs.python.org/3/library/tracemalloc.html
> Use https://pypi.org/project/objgraph (with graphviz) to view heap object 
> trees. The following renders the picture as a png under /tmp and prints the 
> path to stdout.
> {code}
> int ret = PyRun_SimpleString("import objgraph; 
> objgraph.show_backrefs(config.global_agent, max_depth=10)\n\n");
>

[jira] [Commented] (DISPATCH-1962) Make LSan suppressions more targeted and specific (within reason)

2021-03-22 Thread ASF GitHub Bot (Jira)



[ 
https://issues.apache.org/jira/browse/DISPATCH-1962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17306539#comment-17306539
 ] 

ASF GitHub Bot commented on DISPATCH-1962:
--

jiridanek commented on a change in pull request #1080:
URL: https://github.com/apache/qpid-dispatch/pull/1080#discussion_r599037627



##
File path: src/server.c
##
@@ -1372,6 +1372,10 @@ qd_server_t *qd_server(qd_dispatch_t *qd, int 
thread_count, const char *containe
 void qd_server_free(qd_server_t *qd_server)
 {
 if (!qd_server) return;
+
+qd_http_server_stop(qd_server->http); /* Stop HTTP threads immediately */
+qd_http_server_free(qd_server->http);

Review comment:
   ```suggestion
   qd_http_server_free(qd_server->http);
   ```




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Make LSan suppressions more targeted and specific (within reason)
> -
>
> Key: DISPATCH-1962
> URL: https://issues.apache.org/jira/browse/DISPATCH-1962
> Project: Qpid Dispatch
>  Issue Type: Test
>Affects Versions: 1.15.0
>Reporter: Jiri Daněk
>Priority: Major
> Attachments: dispatch_leaking_agent.png
>
>
> There are some unfortunate suppressions in {{tests/lsan.supp}} at this moment:
> {code}
> leak:*libwebsockets*
> {code}
> This is way too broad. It suppresses leaks in Dispatch files, since it 
> matches e.g. {{src/http-libwebsockets.c}}.
> The stars at the beginning and end are actually assumed implicitly. If you do 
> not want substring match, you have to do ^foo$. LSan suppression format is 
> simplistic, very unlike Valgrind's.
> {code}
> leak:run_unit_tests.c
> {code}
> Same thing, any leaks revealed by running unit_tests get suppressed. This 
> suppression suppresses all leak traces that include run_unit_tests.c anywhere 
> in the stack. What't the point of running such tests under leak detector, 
> then?
> {code}
> leak:run_unit_tests.c
> leak:^libqpid-proton.so$
> {code}
> Same thing. The patterns suppress all leaks that include Python (or Proton) 
> anywhere in the stacktrace. That means there are huge blind spots where 
> dispatch leaks can hide. This is a weakness of the lsan.supp syntax (Valgrind 
> suppressions can be much more targeted and discerning).
> h3. Python leaks
> Leaks are known and there is ongoing effort to fight them: 
> https://bugs.python.org/issue1635741 (https://bugs.python.org/issue25302) and 
> https://www.python.org/dev/peps/pep-3121
> Here's valgrind suppression file from somebody who actually investigated the 
> Python leaks and identified the harmless ones: 
> https://github.com/libgit2/pygit2/blob/master/misc/valgrind-python.supp
> One example of a hidden leak in dispatch, which is revealed by making Python 
> suppressions more targetted:
> {code}
> 9: Direct leak of 56 byte(s) in 1 object(s) allocated from:
> 9: #0 0x7f78a3606e8f in __interceptor_malloc 
> (/nix/store/g40sl3zh3nv52vj0mrl4iki5iphh5ika-gcc-10.2.0-lib/lib/libasan.so.6+0xace8f)
> 9: #1 0x7f78a2d64afb in qd_malloc ../include/qpid/dispatch/ctools.h:229
> 9: #2 0x7f78a2d657da in qdr_core_subscribe 
> ../src/router_core/route_tables.c:149
> 9: #3 0x7f78a2c83072 in IoAdapter_init ../src/python_embedded.c:711
> 9: #4 0x7f78a2353a6c in type_call 
> (/nix/store/r85nxfnwiv45nbmf5yb60jj8ajim4m7w-python3-3.8.5/lib/libpython3.8.so.1.0+0x165a6c)
> {code}
> The problem is in
> {code}
> class Agent:
> ...
> def activate(self, address):
> ...
> self.io = IoAdapter(self.receive, address, 'L', '0', 
> TREATMENT_ANYCAST_CLOSEST)
> {code}
> IoAdapter refers to Agent (through the bound method reference self.receive) 
> and Agent refers to IoAdapter (through property self.io). Since IoAdapter is 
> implemented in C and does not implement support for Python's cyclic GC, there 
> is no way to break the cycle.
> Heap dump in attachment. The bound method is at the top of the picture. 
> (Ignore the Mock objects, I was trying to simplify the picture while not 
> getting crashes due to too much meddling).
> h3. Random observations
> It is possible to build special Debug build of Python, which has tools to 
> detect leaks, asserts to prevent negative refcounts, etc. 
> https://pythonextensionpatterns.readthedocs.io/en/latest/debugging/debug_python.html#debug-version-of-python-memory-alloc-label
> Use the following to detect python leaks (instead of valgrind)
> https://docs.python.org/3/library/tracemalloc.html
> Use https://pypi.org/project/objgraph (with graphviz) to view heap object 
> trees. The following renders the picture as a png under /tmp and prints the 
>

[jira] [Commented] (DISPATCH-1962) Make LSan suppressions more targeted and specific (within reason)

2021-03-22 Thread ASF GitHub Bot (Jira)



[ 
https://issues.apache.org/jira/browse/DISPATCH-1962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17306529#comment-17306529
 ] 

ASF GitHub Bot commented on DISPATCH-1962:
--

ganeshmurthy commented on a change in pull request #1080:
URL: https://github.com/apache/qpid-dispatch/pull/1080#discussion_r599029573



##
File path: src/server.c
##
@@ -1372,6 +1372,10 @@ qd_server_t *qd_server(qd_dispatch_t *qd, int 
thread_count, const char *containe
 void qd_server_free(qd_server_t *qd_server)
 {
 if (!qd_server) return;
+
+qd_http_server_stop(qd_server->http); /* Stop HTTP threads immediately */
+qd_http_server_free(qd_server->http);

Review comment:
   Yes, let's remove the call to qd_http_server_stop(). Otherwise this PR 
is good to go. Thanks.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Make LSan suppressions more targeted and specific (within reason)
> -
>
> Key: DISPATCH-1962
> URL: https://issues.apache.org/jira/browse/DISPATCH-1962
> Project: Qpid Dispatch
>  Issue Type: Test
>Affects Versions: 1.15.0
>Reporter: Jiri Daněk
>Priority: Major
> Attachments: dispatch_leaking_agent.png
>
>
> There are some unfortunate suppressions in {{tests/lsan.supp}} at this moment:
> {code}
> leak:*libwebsockets*
> {code}
> This is way too broad. It suppresses leaks in Dispatch files, since it 
> matches e.g. {{src/http-libwebsockets.c}}.
> The stars at the beginning and end are actually assumed implicitly. If you do 
> not want substring match, you have to do ^foo$. LSan suppression format is 
> simplistic, very unlike Valgrind's.
> {code}
> leak:run_unit_tests.c
> {code}
> Same thing, any leaks revealed by running unit_tests get suppressed. This 
> suppression suppresses all leak traces that include run_unit_tests.c anywhere 
> in the stack. What't the point of running such tests under leak detector, 
> then?
> {code}
> leak:run_unit_tests.c
> leak:^libqpid-proton.so$
> {code}
> Same thing. The patterns suppress all leaks that include Python (or Proton) 
> anywhere in the stacktrace. That means there are huge blind spots where 
> dispatch leaks can hide. This is a weakness of the lsan.supp syntax (Valgrind 
> suppressions can be much more targeted and discerning).
> h3. Python leaks
> Leaks are known and there is ongoing effort to fight them: 
> https://bugs.python.org/issue1635741 (https://bugs.python.org/issue25302) and 
> https://www.python.org/dev/peps/pep-3121
> Here's valgrind suppression file from somebody who actually investigated the 
> Python leaks and identified the harmless ones: 
> https://github.com/libgit2/pygit2/blob/master/misc/valgrind-python.supp
> One example of a hidden leak in dispatch, which is revealed by making Python 
> suppressions more targetted:
> {code}
> 9: Direct leak of 56 byte(s) in 1 object(s) allocated from:
> 9: #0 0x7f78a3606e8f in __interceptor_malloc 
> (/nix/store/g40sl3zh3nv52vj0mrl4iki5iphh5ika-gcc-10.2.0-lib/lib/libasan.so.6+0xace8f)
> 9: #1 0x7f78a2d64afb in qd_malloc ../include/qpid/dispatch/ctools.h:229
> 9: #2 0x7f78a2d657da in qdr_core_subscribe 
> ../src/router_core/route_tables.c:149
> 9: #3 0x7f78a2c83072 in IoAdapter_init ../src/python_embedded.c:711
> 9: #4 0x7f78a2353a6c in type_call 
> (/nix/store/r85nxfnwiv45nbmf5yb60jj8ajim4m7w-python3-3.8.5/lib/libpython3.8.so.1.0+0x165a6c)
> {code}
> The problem is in
> {code}
> class Agent:
> ...
> def activate(self, address):
> ...
> self.io = IoAdapter(self.receive, address, 'L', '0', 
> TREATMENT_ANYCAST_CLOSEST)
> {code}
> IoAdapter refers to Agent (through the bound method reference self.receive) 
> and Agent refers to IoAdapter (through property self.io). Since IoAdapter is 
> implemented in C and does not implement support for Python's cyclic GC, there 
> is no way to break the cycle.
> Heap dump in attachment. The bound method is at the top of the picture. 
> (Ignore the Mock objects, I was trying to simplify the picture while not 
> getting crashes due to too much meddling).
> h3. Random observations
> It is possible to build special Debug build of Python, which has tools to 
> detect leaks, asserts to prevent negative refcounts, etc. 
> https://pythonextensionpatterns.readthedocs.io/en/latest/debugging/debug_python.html#debug-version-of-python-memory-alloc-label
> Use the following to detect python leaks (instead of valgrind)
> https://docs.python.org/3/library/tracemalloc.html
> Use https://pypi.org/project/objgraph (with graphviz) to view heap object 
> trees. The following renders the picture as a png

[jira] [Commented] (DISPATCH-1962) Make LSan suppressions more targeted and specific (within reason)

2021-03-22 Thread ASF GitHub Bot (Jira)



[ 
https://issues.apache.org/jira/browse/DISPATCH-1962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17306225#comment-17306225
 ] 

ASF GitHub Bot commented on DISPATCH-1962:
--

kgiusti commented on pull request #1051:
URL: https://github.com/apache/qpid-dispatch/pull/1051#issuecomment-804096568


   > @kgiusti I take it that this leak is extremely low priority and maybe it 
makes sense to instead simply make the suppression permanent. Is that a correct 
understanding?
   
   @jiridanek Yes that would be my opinion.  We should continue suppressing it 
until we (someday) refactor the router shutdown process to deterministically 
release all resources.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Make LSan suppressions more targeted and specific (within reason)
> -
>
> Key: DISPATCH-1962
> URL: https://issues.apache.org/jira/browse/DISPATCH-1962
> Project: Qpid Dispatch
>  Issue Type: Test
>Affects Versions: 1.15.0
>Reporter: Jiri Daněk
>Priority: Major
> Attachments: dispatch_leaking_agent.png
>
>
> There are some unfortunate suppressions in {{tests/lsan.supp}} at this moment:
> {code}
> leak:*libwebsockets*
> {code}
> This is way too broad. It suppresses leaks in Dispatch files, since it 
> matches e.g. {{src/http-libwebsockets.c}}.
> The stars at the beginning and end are actually assumed implicitly. If you do 
> not want substring match, you have to do ^foo$. LSan suppression format is 
> simplistic, very unlike Valgrind's.
> {code}
> leak:run_unit_tests.c
> {code}
> Same thing, any leaks revealed by running unit_tests get suppressed. This 
> suppression suppresses all leak traces that include run_unit_tests.c anywhere 
> in the stack. What't the point of running such tests under leak detector, 
> then?
> {code}
> leak:run_unit_tests.c
> leak:^libqpid-proton.so$
> {code}
> Same thing. The patterns suppress all leaks that include Python (or Proton) 
> anywhere in the stacktrace. That means there are huge blind spots where 
> dispatch leaks can hide. This is a weakness of the lsan.supp syntax (Valgrind 
> suppressions can be much more targeted and discerning).
> h3. Python leaks
> Leaks are known and there is ongoing effort to fight them: 
> https://bugs.python.org/issue1635741 (https://bugs.python.org/issue25302) and 
> https://www.python.org/dev/peps/pep-3121
> Here's valgrind suppression file from somebody who actually investigated the 
> Python leaks and identified the harmless ones: 
> https://github.com/libgit2/pygit2/blob/master/misc/valgrind-python.supp
> One example of a hidden leak in dispatch, which is revealed by making Python 
> suppressions more targetted:
> {code}
> 9: Direct leak of 56 byte(s) in 1 object(s) allocated from:
> 9: #0 0x7f78a3606e8f in __interceptor_malloc 
> (/nix/store/g40sl3zh3nv52vj0mrl4iki5iphh5ika-gcc-10.2.0-lib/lib/libasan.so.6+0xace8f)
> 9: #1 0x7f78a2d64afb in qd_malloc ../include/qpid/dispatch/ctools.h:229
> 9: #2 0x7f78a2d657da in qdr_core_subscribe 
> ../src/router_core/route_tables.c:149
> 9: #3 0x7f78a2c83072 in IoAdapter_init ../src/python_embedded.c:711
> 9: #4 0x7f78a2353a6c in type_call 
> (/nix/store/r85nxfnwiv45nbmf5yb60jj8ajim4m7w-python3-3.8.5/lib/libpython3.8.so.1.0+0x165a6c)
> {code}
> The problem is in
> {code}
> class Agent:
> ...
> def activate(self, address):
> ...
> self.io = IoAdapter(self.receive, address, 'L', '0', 
> TREATMENT_ANYCAST_CLOSEST)
> {code}
> IoAdapter refers to Agent (through the bound method reference self.receive) 
> and Agent refers to IoAdapter (through property self.io). Since IoAdapter is 
> implemented in C and does not implement support for Python's cyclic GC, there 
> is no way to break the cycle.
> Heap dump in attachment. The bound method is at the top of the picture. 
> (Ignore the Mock objects, I was trying to simplify the picture while not 
> getting crashes due to too much meddling).
> h3. Random observations
> It is possible to build special Debug build of Python, which has tools to 
> detect leaks, asserts to prevent negative refcounts, etc. 
> https://pythonextensionpatterns.readthedocs.io/en/latest/debugging/debug_python.html#debug-version-of-python-memory-alloc-label
> Use the following to detect python leaks (instead of valgrind)
> https://docs.python.org/3/library/tracemalloc.html
> Use https://pypi.org/project/objgraph (with graphviz) to view heap object 
> trees. The following renders the picture as a png under /tmp and prints the 
> path to stdout.
> {code}
> int ret = PyRun_SimpleString("import objgraph; 
>

[jira] [Commented] (DISPATCH-1962) Make LSan suppressions more targeted and specific (within reason)

2021-03-22 Thread ASF GitHub Bot (Jira)



[ 
https://issues.apache.org/jira/browse/DISPATCH-1962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17305980#comment-17305980
 ] 

ASF GitHub Bot commented on DISPATCH-1962:
--

jiridanek commented on a change in pull request #1080:
URL: https://github.com/apache/qpid-dispatch/pull/1080#discussion_r598504299



##
File path: src/server.c
##
@@ -1372,6 +1372,10 @@ qd_server_t *qd_server(qd_dispatch_t *qd, int 
thread_count, const char *containe
 void qd_server_free(qd_server_t *qd_server)
 {
 if (!qd_server) return;
+
+qd_http_server_stop(qd_server->http); /* Stop HTTP threads immediately */
+qd_http_server_free(qd_server->http);

Review comment:
   It appears that calling only `qd_http_server_free` is indeed sufficient. 
I was thinking that maybe calling _stop before _free is some sort of dispatch 
convention... Do you want me to remove the call to _stop here?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Make LSan suppressions more targeted and specific (within reason)
> -
>
> Key: DISPATCH-1962
> URL: https://issues.apache.org/jira/browse/DISPATCH-1962
> Project: Qpid Dispatch
>  Issue Type: Test
>Affects Versions: 1.15.0
>Reporter: Jiri Daněk
>Priority: Major
> Attachments: dispatch_leaking_agent.png
>
>
> There are some unfortunate suppressions in {{tests/lsan.supp}} at this moment:
> {code}
> leak:*libwebsockets*
> {code}
> This is way too broad. It suppresses leaks in Dispatch files, since it 
> matches e.g. {{src/http-libwebsockets.c}}.
> The stars at the beginning and end are actually assumed implicitly. If you do 
> not want substring match, you have to do ^foo$. LSan suppression format is 
> simplistic, very unlike Valgrind's.
> {code}
> leak:run_unit_tests.c
> {code}
> Same thing, any leaks revealed by running unit_tests get suppressed. This 
> suppression suppresses all leak traces that include run_unit_tests.c anywhere 
> in the stack. What't the point of running such tests under leak detector, 
> then?
> {code}
> leak:run_unit_tests.c
> leak:^libqpid-proton.so$
> {code}
> Same thing. The patterns suppress all leaks that include Python (or Proton) 
> anywhere in the stacktrace. That means there are huge blind spots where 
> dispatch leaks can hide. This is a weakness of the lsan.supp syntax (Valgrind 
> suppressions can be much more targeted and discerning).
> h3. Python leaks
> Leaks are known and there is ongoing effort to fight them: 
> https://bugs.python.org/issue1635741 (https://bugs.python.org/issue25302) and 
> https://www.python.org/dev/peps/pep-3121
> Here's valgrind suppression file from somebody who actually investigated the 
> Python leaks and identified the harmless ones: 
> https://github.com/libgit2/pygit2/blob/master/misc/valgrind-python.supp
> One example of a hidden leak in dispatch, which is revealed by making Python 
> suppressions more targetted:
> {code}
> 9: Direct leak of 56 byte(s) in 1 object(s) allocated from:
> 9: #0 0x7f78a3606e8f in __interceptor_malloc 
> (/nix/store/g40sl3zh3nv52vj0mrl4iki5iphh5ika-gcc-10.2.0-lib/lib/libasan.so.6+0xace8f)
> 9: #1 0x7f78a2d64afb in qd_malloc ../include/qpid/dispatch/ctools.h:229
> 9: #2 0x7f78a2d657da in qdr_core_subscribe 
> ../src/router_core/route_tables.c:149
> 9: #3 0x7f78a2c83072 in IoAdapter_init ../src/python_embedded.c:711
> 9: #4 0x7f78a2353a6c in type_call 
> (/nix/store/r85nxfnwiv45nbmf5yb60jj8ajim4m7w-python3-3.8.5/lib/libpython3.8.so.1.0+0x165a6c)
> {code}
> The problem is in
> {code}
> class Agent:
> ...
> def activate(self, address):
> ...
> self.io = IoAdapter(self.receive, address, 'L', '0', 
> TREATMENT_ANYCAST_CLOSEST)
> {code}
> IoAdapter refers to Agent (through the bound method reference self.receive) 
> and Agent refers to IoAdapter (through property self.io). Since IoAdapter is 
> implemented in C and does not implement support for Python's cyclic GC, there 
> is no way to break the cycle.
> Heap dump in attachment. The bound method is at the top of the picture. 
> (Ignore the Mock objects, I was trying to simplify the picture while not 
> getting crashes due to too much meddling).
> h3. Random observations
> It is possible to build special Debug build of Python, which has tools to 
> detect leaks, asserts to prevent negative refcounts, etc. 
> https://pythonextensionpatterns.readthedocs.io/en/latest/debugging/debug_python.html#debug-version-of-python-memory-alloc-label
> Use the following to detect python leaks (instead of valgrind)
> https://docs.python.org/3/library/tracemalloc.html
> Use

[jira] [Commented] (DISPATCH-1962) Make LSan suppressions more targeted and specific (within reason)

2021-03-21 Thread ASF GitHub Bot (Jira)



[ 
https://issues.apache.org/jira/browse/DISPATCH-1962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17305811#comment-17305811
 ] 

ASF GitHub Bot commented on DISPATCH-1962:
--

ganeshmurthy commented on a change in pull request #1080:
URL: https://github.com/apache/qpid-dispatch/pull/1080#discussion_r598368463



##
File path: src/server.c
##
@@ -1372,6 +1372,10 @@ qd_server_t *qd_server(qd_dispatch_t *qd, int 
thread_count, const char *containe
 void qd_server_free(qd_server_t *qd_server)
 {
 if (!qd_server) return;
+
+qd_http_server_stop(qd_server->http); /* Stop HTTP threads immediately */
+qd_http_server_free(qd_server->http);

Review comment:
   the function qd_http_server_free() in http-libwebsockets.c seems to be 
calling qd_http_server_stop(), so just calling qd_http_server_free() is enough ?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Make LSan suppressions more targeted and specific (within reason)
> -
>
> Key: DISPATCH-1962
> URL: https://issues.apache.org/jira/browse/DISPATCH-1962
> Project: Qpid Dispatch
>  Issue Type: Test
>Affects Versions: 1.15.0
>Reporter: Jiri Daněk
>Priority: Major
> Attachments: dispatch_leaking_agent.png
>
>
> There are some unfortunate suppressions in {{tests/lsan.supp}} at this moment:
> {code}
> leak:*libwebsockets*
> {code}
> This is way too broad. It suppresses leaks in Dispatch files, since it 
> matches e.g. {{src/http-libwebsockets.c}}.
> The stars at the beginning and end are actually assumed implicitly. If you do 
> not want substring match, you have to do ^foo$. LSan suppression format is 
> simplistic, very unlike Valgrind's.
> {code}
> leak:run_unit_tests.c
> {code}
> Same thing, any leaks revealed by running unit_tests get suppressed. This 
> suppression suppresses all leak traces that include run_unit_tests.c anywhere 
> in the stack. What't the point of running such tests under leak detector, 
> then?
> {code}
> leak:run_unit_tests.c
> leak:^libqpid-proton.so$
> {code}
> Same thing. The patterns suppress all leaks that include Python (or Proton) 
> anywhere in the stacktrace. That means there are huge blind spots where 
> dispatch leaks can hide. This is a weakness of the lsan.supp syntax (Valgrind 
> suppressions can be much more targeted and discerning).
> h3. Python leaks
> Leaks are known and there is ongoing effort to fight them: 
> https://bugs.python.org/issue1635741 (https://bugs.python.org/issue25302) and 
> https://www.python.org/dev/peps/pep-3121
> Here's valgrind suppression file from somebody who actually investigated the 
> Python leaks and identified the harmless ones: 
> https://github.com/libgit2/pygit2/blob/master/misc/valgrind-python.supp
> One example of a hidden leak in dispatch, which is revealed by making Python 
> suppressions more targetted:
> {code}
> 9: Direct leak of 56 byte(s) in 1 object(s) allocated from:
> 9: #0 0x7f78a3606e8f in __interceptor_malloc 
> (/nix/store/g40sl3zh3nv52vj0mrl4iki5iphh5ika-gcc-10.2.0-lib/lib/libasan.so.6+0xace8f)
> 9: #1 0x7f78a2d64afb in qd_malloc ../include/qpid/dispatch/ctools.h:229
> 9: #2 0x7f78a2d657da in qdr_core_subscribe 
> ../src/router_core/route_tables.c:149
> 9: #3 0x7f78a2c83072 in IoAdapter_init ../src/python_embedded.c:711
> 9: #4 0x7f78a2353a6c in type_call 
> (/nix/store/r85nxfnwiv45nbmf5yb60jj8ajim4m7w-python3-3.8.5/lib/libpython3.8.so.1.0+0x165a6c)
> {code}
> The problem is in
> {code}
> class Agent:
> ...
> def activate(self, address):
> ...
> self.io = IoAdapter(self.receive, address, 'L', '0', 
> TREATMENT_ANYCAST_CLOSEST)
> {code}
> IoAdapter refers to Agent (through the bound method reference self.receive) 
> and Agent refers to IoAdapter (through property self.io). Since IoAdapter is 
> implemented in C and does not implement support for Python's cyclic GC, there 
> is no way to break the cycle.
> Heap dump in attachment. The bound method is at the top of the picture. 
> (Ignore the Mock objects, I was trying to simplify the picture while not 
> getting crashes due to too much meddling).
> h3. Random observations
> It is possible to build special Debug build of Python, which has tools to 
> detect leaks, asserts to prevent negative refcounts, etc. 
> https://pythonextensionpatterns.readthedocs.io/en/latest/debugging/debug_python.html#debug-version-of-python-memory-alloc-label
> Use the following to detect python leaks (instead of valgrind)
> https://docs.python.org/3/library/tracemalloc.html
> Use https://pypi.org/project/objgraph (with graphviz) to view heap

[jira] [Commented] (DISPATCH-1962) Make LSan suppressions more targeted and specific (within reason)

2021-03-20 Thread ASF GitHub Bot (Jira)



[ 
https://issues.apache.org/jira/browse/DISPATCH-1962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17305500#comment-17305500
 ] 

ASF GitHub Bot commented on DISPATCH-1962:
--

jiridanek merged pull request #1081:
URL: https://github.com/apache/qpid-dispatch/pull/1081


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Make LSan suppressions more targeted and specific (within reason)
> -
>
> Key: DISPATCH-1962
> URL: https://issues.apache.org/jira/browse/DISPATCH-1962
> Project: Qpid Dispatch
>  Issue Type: Test
>Affects Versions: 1.15.0
>Reporter: Jiri Daněk
>Priority: Major
> Attachments: dispatch_leaking_agent.png
>
>
> There are some unfortunate suppressions in {{tests/lsan.supp}} at this moment:
> {code}
> leak:*libwebsockets*
> {code}
> This is way too broad. It suppresses leaks in Dispatch files, since it 
> matches e.g. {{src/http-libwebsockets.c}}.
> The stars at the beginning and end are actually assumed implicitly. If you do 
> not want substring match, you have to do ^foo$. LSan suppression format is 
> simplistic, very unlike Valgrind's.
> {code}
> leak:run_unit_tests.c
> {code}
> Same thing, any leaks revealed by running unit_tests get suppressed. This 
> suppression suppresses all leak traces that include run_unit_tests.c anywhere 
> in the stack. What't the point of running such tests under leak detector, 
> then?
> {code}
> leak:run_unit_tests.c
> leak:^libqpid-proton.so$
> {code}
> Same thing. The patterns suppress all leaks that include Python (or Proton) 
> anywhere in the stacktrace. That means there are huge blind spots where 
> dispatch leaks can hide. This is a weakness of the lsan.supp syntax (Valgrind 
> suppressions can be much more targeted and discerning).
> h3. Python leaks
> Leaks are known and there is ongoing effort to fight them: 
> https://bugs.python.org/issue1635741 (https://bugs.python.org/issue25302) and 
> https://www.python.org/dev/peps/pep-3121
> Here's valgrind suppression file from somebody who actually investigated the 
> Python leaks and identified the harmless ones: 
> https://github.com/libgit2/pygit2/blob/master/misc/valgrind-python.supp
> One example of a hidden leak in dispatch, which is revealed by making Python 
> suppressions more targetted:
> {code}
> 9: Direct leak of 56 byte(s) in 1 object(s) allocated from:
> 9: #0 0x7f78a3606e8f in __interceptor_malloc 
> (/nix/store/g40sl3zh3nv52vj0mrl4iki5iphh5ika-gcc-10.2.0-lib/lib/libasan.so.6+0xace8f)
> 9: #1 0x7f78a2d64afb in qd_malloc ../include/qpid/dispatch/ctools.h:229
> 9: #2 0x7f78a2d657da in qdr_core_subscribe 
> ../src/router_core/route_tables.c:149
> 9: #3 0x7f78a2c83072 in IoAdapter_init ../src/python_embedded.c:711
> 9: #4 0x7f78a2353a6c in type_call 
> (/nix/store/r85nxfnwiv45nbmf5yb60jj8ajim4m7w-python3-3.8.5/lib/libpython3.8.so.1.0+0x165a6c)
> {code}
> The problem is in
> {code}
> class Agent:
> ...
> def activate(self, address):
> ...
> self.io = IoAdapter(self.receive, address, 'L', '0', 
> TREATMENT_ANYCAST_CLOSEST)
> {code}
> IoAdapter refers to Agent (through the bound method reference self.receive) 
> and Agent refers to IoAdapter (through property self.io). Since IoAdapter is 
> implemented in C and does not implement support for Python's cyclic GC, there 
> is no way to break the cycle.
> Heap dump in attachment. The bound method is at the top of the picture. 
> (Ignore the Mock objects, I was trying to simplify the picture while not 
> getting crashes due to too much meddling).
> h3. Random observations
> It is possible to build special Debug build of Python, which has tools to 
> detect leaks, asserts to prevent negative refcounts, etc. 
> https://pythonextensionpatterns.readthedocs.io/en/latest/debugging/debug_python.html#debug-version-of-python-memory-alloc-label
> Use the following to detect python leaks (instead of valgrind)
> https://docs.python.org/3/library/tracemalloc.html
> Use https://pypi.org/project/objgraph (with graphviz) to view heap object 
> trees. The following renders the picture as a png under /tmp and prints the 
> path to stdout.
> {code}
> int ret = PyRun_SimpleString("import objgraph; 
> objgraph.show_backrefs(config.global_agent, max_depth=10)\n\n");
> PyErr_PrintEx(0);
> assert(ret == 0);
> {code}
> h2. Tools
> h3. CPyChecker
> https://dmalcolm.fedorapeople.org/presentations/PyCon-US-2013/PyCon-US-2013-dmalcolm-StaticAnalysis.html#(38)
> https://emptysqua.re/blog/analyzing-python-c-extensions-with-cpychecker/
> https://nedbatchelder.com/blog/201502/cpychecker.html
> h3. Pungi
>

[jira] [Commented] (DISPATCH-1962) Make LSan suppressions more targeted and specific (within reason)

2021-03-20 Thread ASF subversion and git services (Jira)



[ 
https://issues.apache.org/jira/browse/DISPATCH-1962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17305499#comment-17305499
 ] 

ASF subversion and git services commented on DISPATCH-1962:
---

Commit b835eb7e4ef9af821aa8d0c03df5d332f7e32630 in qpid-dispatch's branch 
refs/heads/master from Jiri Daněk
[ https://gitbox.apache.org/repos/asf?p=qpid-dispatch.git;h=b835eb7 ]

DISPATCH-1962 Remove outdated entries in lsan.supp (#1081)



> Make LSan suppressions more targeted and specific (within reason)
> -
>
> Key: DISPATCH-1962
> URL: https://issues.apache.org/jira/browse/DISPATCH-1962
> Project: Qpid Dispatch
>  Issue Type: Test
>Affects Versions: 1.15.0
>Reporter: Jiri Daněk
>Priority: Major
> Attachments: dispatch_leaking_agent.png
>
>
> There are some unfortunate suppressions in {{tests/lsan.supp}} at this moment:
> {code}
> leak:*libwebsockets*
> {code}
> This is way too broad. It suppresses leaks in Dispatch files, since it 
> matches e.g. {{src/http-libwebsockets.c}}.
> The stars at the beginning and end are actually assumed implicitly. If you do 
> not want substring match, you have to do ^foo$. LSan suppression format is 
> simplistic, very unlike Valgrind's.
> {code}
> leak:run_unit_tests.c
> {code}
> Same thing, any leaks revealed by running unit_tests get suppressed. This 
> suppression suppresses all leak traces that include run_unit_tests.c anywhere 
> in the stack. What't the point of running such tests under leak detector, 
> then?
> {code}
> leak:run_unit_tests.c
> leak:^libqpid-proton.so$
> {code}
> Same thing. The patterns suppress all leaks that include Python (or Proton) 
> anywhere in the stacktrace. That means there are huge blind spots where 
> dispatch leaks can hide. This is a weakness of the lsan.supp syntax (Valgrind 
> suppressions can be much more targeted and discerning).
> h3. Python leaks
> Leaks are known and there is ongoing effort to fight them: 
> https://bugs.python.org/issue1635741 (https://bugs.python.org/issue25302) and 
> https://www.python.org/dev/peps/pep-3121
> Here's valgrind suppression file from somebody who actually investigated the 
> Python leaks and identified the harmless ones: 
> https://github.com/libgit2/pygit2/blob/master/misc/valgrind-python.supp
> One example of a hidden leak in dispatch, which is revealed by making Python 
> suppressions more targetted:
> {code}
> 9: Direct leak of 56 byte(s) in 1 object(s) allocated from:
> 9: #0 0x7f78a3606e8f in __interceptor_malloc 
> (/nix/store/g40sl3zh3nv52vj0mrl4iki5iphh5ika-gcc-10.2.0-lib/lib/libasan.so.6+0xace8f)
> 9: #1 0x7f78a2d64afb in qd_malloc ../include/qpid/dispatch/ctools.h:229
> 9: #2 0x7f78a2d657da in qdr_core_subscribe 
> ../src/router_core/route_tables.c:149
> 9: #3 0x7f78a2c83072 in IoAdapter_init ../src/python_embedded.c:711
> 9: #4 0x7f78a2353a6c in type_call 
> (/nix/store/r85nxfnwiv45nbmf5yb60jj8ajim4m7w-python3-3.8.5/lib/libpython3.8.so.1.0+0x165a6c)
> {code}
> The problem is in
> {code}
> class Agent:
> ...
> def activate(self, address):
> ...
> self.io = IoAdapter(self.receive, address, 'L', '0', 
> TREATMENT_ANYCAST_CLOSEST)
> {code}
> IoAdapter refers to Agent (through the bound method reference self.receive) 
> and Agent refers to IoAdapter (through property self.io). Since IoAdapter is 
> implemented in C and does not implement support for Python's cyclic GC, there 
> is no way to break the cycle.
> Heap dump in attachment. The bound method is at the top of the picture. 
> (Ignore the Mock objects, I was trying to simplify the picture while not 
> getting crashes due to too much meddling).
> h3. Random observations
> It is possible to build special Debug build of Python, which has tools to 
> detect leaks, asserts to prevent negative refcounts, etc. 
> https://pythonextensionpatterns.readthedocs.io/en/latest/debugging/debug_python.html#debug-version-of-python-memory-alloc-label
> Use the following to detect python leaks (instead of valgrind)
> https://docs.python.org/3/library/tracemalloc.html
> Use https://pypi.org/project/objgraph (with graphviz) to view heap object 
> trees. The following renders the picture as a png under /tmp and prints the 
> path to stdout.
> {code}
> int ret = PyRun_SimpleString("import objgraph; 
> objgraph.show_backrefs(config.global_agent, max_depth=10)\n\n");
> PyErr_PrintEx(0);
> assert(ret == 0);
> {code}
> h2. Tools
> h3. CPyChecker
> https://dmalcolm.fedorapeople.org/presentations/PyCon-US-2013/PyCon-US-2013-dmalcolm-StaticAnalysis.html#(38)
> https://emptysqua.re/blog/analyzing-python-c-extensions-with-cpychecker/
> https://nedbatchelder.com/blog/201502/cpychecker.html
> h3. Pungi
> http://www.cse.psu.edu/~gxt29/papers/refcount.pdf
> h3. Use CFFI or similar, instead of

[jira] [Commented] (DISPATCH-1962) Make LSan suppressions more targeted and specific (within reason)

2021-03-20 Thread ASF GitHub Bot (Jira)



[ 
https://issues.apache.org/jira/browse/DISPATCH-1962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17305495#comment-17305495
 ] 

ASF GitHub Bot commented on DISPATCH-1962:
--

jiridanek opened a new pull request #1081:
URL: https://github.com/apache/qpid-dispatch/pull/1081


   leak:qd_message_receive and leak:_flush_send_queue_CT can be now removed


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Make LSan suppressions more targeted and specific (within reason)
> -
>
> Key: DISPATCH-1962
> URL: https://issues.apache.org/jira/browse/DISPATCH-1962
> Project: Qpid Dispatch
>  Issue Type: Test
>Affects Versions: 1.15.0
>Reporter: Jiri Daněk
>Priority: Major
> Attachments: dispatch_leaking_agent.png
>
>
> There are some unfortunate suppressions in {{tests/lsan.supp}} at this moment:
> {code}
> leak:*libwebsockets*
> {code}
> This is way too broad. It suppresses leaks in Dispatch files, since it 
> matches e.g. {{src/http-libwebsockets.c}}.
> The stars at the beginning and end are actually assumed implicitly. If you do 
> not want substring match, you have to do ^foo$. LSan suppression format is 
> simplistic, very unlike Valgrind's.
> {code}
> leak:run_unit_tests.c
> {code}
> Same thing, any leaks revealed by running unit_tests get suppressed. This 
> suppression suppresses all leak traces that include run_unit_tests.c anywhere 
> in the stack. What't the point of running such tests under leak detector, 
> then?
> {code}
> leak:run_unit_tests.c
> leak:^libqpid-proton.so$
> {code}
> Same thing. The patterns suppress all leaks that include Python (or Proton) 
> anywhere in the stacktrace. That means there are huge blind spots where 
> dispatch leaks can hide. This is a weakness of the lsan.supp syntax (Valgrind 
> suppressions can be much more targeted and discerning).
> h3. Python leaks
> Leaks are known and there is ongoing effort to fight them: 
> https://bugs.python.org/issue1635741 (https://bugs.python.org/issue25302) and 
> https://www.python.org/dev/peps/pep-3121
> Here's valgrind suppression file from somebody who actually investigated the 
> Python leaks and identified the harmless ones: 
> https://github.com/libgit2/pygit2/blob/master/misc/valgrind-python.supp
> One example of a hidden leak in dispatch, which is revealed by making Python 
> suppressions more targetted:
> {code}
> 9: Direct leak of 56 byte(s) in 1 object(s) allocated from:
> 9: #0 0x7f78a3606e8f in __interceptor_malloc 
> (/nix/store/g40sl3zh3nv52vj0mrl4iki5iphh5ika-gcc-10.2.0-lib/lib/libasan.so.6+0xace8f)
> 9: #1 0x7f78a2d64afb in qd_malloc ../include/qpid/dispatch/ctools.h:229
> 9: #2 0x7f78a2d657da in qdr_core_subscribe 
> ../src/router_core/route_tables.c:149
> 9: #3 0x7f78a2c83072 in IoAdapter_init ../src/python_embedded.c:711
> 9: #4 0x7f78a2353a6c in type_call 
> (/nix/store/r85nxfnwiv45nbmf5yb60jj8ajim4m7w-python3-3.8.5/lib/libpython3.8.so.1.0+0x165a6c)
> {code}
> The problem is in
> {code}
> class Agent:
> ...
> def activate(self, address):
> ...
> self.io = IoAdapter(self.receive, address, 'L', '0', 
> TREATMENT_ANYCAST_CLOSEST)
> {code}
> IoAdapter refers to Agent (through the bound method reference self.receive) 
> and Agent refers to IoAdapter (through property self.io). Since IoAdapter is 
> implemented in C and does not implement support for Python's cyclic GC, there 
> is no way to break the cycle.
> Heap dump in attachment. The bound method is at the top of the picture. 
> (Ignore the Mock objects, I was trying to simplify the picture while not 
> getting crashes due to too much meddling).
> h3. Random observations
> It is possible to build special Debug build of Python, which has tools to 
> detect leaks, asserts to prevent negative refcounts, etc. 
> https://pythonextensionpatterns.readthedocs.io/en/latest/debugging/debug_python.html#debug-version-of-python-memory-alloc-label
> Use the following to detect python leaks (instead of valgrind)
> https://docs.python.org/3/library/tracemalloc.html
> Use https://pypi.org/project/objgraph (with graphviz) to view heap object 
> trees. The following renders the picture as a png under /tmp and prints the 
> path to stdout.
> {code}
> int ret = PyRun_SimpleString("import objgraph; 
> objgraph.show_backrefs(config.global_agent, max_depth=10)\n\n");
> PyErr_PrintEx(0);
> assert(ret == 0);
> {code}
> h2. Tools
> h3. CPyChecker
> https://dmalcolm.fedorapeople.org/presentations/PyCon-US-2013/PyCon-US-2013-dmalcolm-StaticAnalysis.html#(38)
>

[jira] [Commented] (DISPATCH-1962) Make LSan suppressions more targeted and specific (within reason)

2021-03-20 Thread ASF GitHub Bot (Jira)



[ 
https://issues.apache.org/jira/browse/DISPATCH-1962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17305463#comment-17305463
 ] 

ASF GitHub Bot commented on DISPATCH-1962:
--

jiridanek commented on a change in pull request #1048:
URL: https://github.com/apache/qpid-dispatch/pull/1048#discussion_r598119866



##
File path: tests/lsan.supp
##
@@ -10,6 +10,27 @@ leak:qd_policy_c_counts_alloc
 leak:qd_policy_open_fetch_settings
 leak:qdr_error_description
 
+# to be triaged; unit_tests
+leak:http-libwebsockets.c

Review comment:
   https://github.com/apache/qpid-dispatch/pull/1080




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Make LSan suppressions more targeted and specific (within reason)
> -
>
> Key: DISPATCH-1962
> URL: https://issues.apache.org/jira/browse/DISPATCH-1962
> Project: Qpid Dispatch
>  Issue Type: Test
>Affects Versions: 1.15.0
>Reporter: Jiri Daněk
>Priority: Major
> Attachments: dispatch_leaking_agent.png
>
>
> There are some unfortunate suppressions in {{tests/lsan.supp}} at this moment:
> {code}
> leak:*libwebsockets*
> {code}
> This is way too broad. It suppresses leaks in Dispatch files, since it 
> matches e.g. {{src/http-libwebsockets.c}}.
> The stars at the beginning and end are actually assumed implicitly. If you do 
> not want substring match, you have to do ^foo$. LSan suppression format is 
> simplistic, very unlike Valgrind's.
> {code}
> leak:run_unit_tests.c
> {code}
> Same thing, any leaks revealed by running unit_tests get suppressed. This 
> suppression suppresses all leak traces that include run_unit_tests.c anywhere 
> in the stack. What't the point of running such tests under leak detector, 
> then?
> {code}
> leak:run_unit_tests.c
> leak:^libqpid-proton.so$
> {code}
> Same thing. The patterns suppress all leaks that include Python (or Proton) 
> anywhere in the stacktrace. That means there are huge blind spots where 
> dispatch leaks can hide. This is a weakness of the lsan.supp syntax (Valgrind 
> suppressions can be much more targeted and discerning).
> h3. Python leaks
> Leaks are known and there is ongoing effort to fight them: 
> https://bugs.python.org/issue1635741 (https://bugs.python.org/issue25302) and 
> https://www.python.org/dev/peps/pep-3121
> Here's valgrind suppression file from somebody who actually investigated the 
> Python leaks and identified the harmless ones: 
> https://github.com/libgit2/pygit2/blob/master/misc/valgrind-python.supp
> One example of a hidden leak in dispatch, which is revealed by making Python 
> suppressions more targetted:
> {code}
> 9: Direct leak of 56 byte(s) in 1 object(s) allocated from:
> 9: #0 0x7f78a3606e8f in __interceptor_malloc 
> (/nix/store/g40sl3zh3nv52vj0mrl4iki5iphh5ika-gcc-10.2.0-lib/lib/libasan.so.6+0xace8f)
> 9: #1 0x7f78a2d64afb in qd_malloc ../include/qpid/dispatch/ctools.h:229
> 9: #2 0x7f78a2d657da in qdr_core_subscribe 
> ../src/router_core/route_tables.c:149
> 9: #3 0x7f78a2c83072 in IoAdapter_init ../src/python_embedded.c:711
> 9: #4 0x7f78a2353a6c in type_call 
> (/nix/store/r85nxfnwiv45nbmf5yb60jj8ajim4m7w-python3-3.8.5/lib/libpython3.8.so.1.0+0x165a6c)
> {code}
> The problem is in
> {code}
> class Agent:
> ...
> def activate(self, address):
> ...
> self.io = IoAdapter(self.receive, address, 'L', '0', 
> TREATMENT_ANYCAST_CLOSEST)
> {code}
> IoAdapter refers to Agent (through the bound method reference self.receive) 
> and Agent refers to IoAdapter (through property self.io). Since IoAdapter is 
> implemented in C and does not implement support for Python's cyclic GC, there 
> is no way to break the cycle.
> Heap dump in attachment. The bound method is at the top of the picture. 
> (Ignore the Mock objects, I was trying to simplify the picture while not 
> getting crashes due to too much meddling).
> h3. Random observations
> It is possible to build special Debug build of Python, which has tools to 
> detect leaks, asserts to prevent negative refcounts, etc. 
> https://pythonextensionpatterns.readthedocs.io/en/latest/debugging/debug_python.html#debug-version-of-python-memory-alloc-label
> Use the following to detect python leaks (instead of valgrind)
> https://docs.python.org/3/library/tracemalloc.html
> Use https://pypi.org/project/objgraph (with graphviz) to view heap object 
> trees. The following renders the picture as a png under /tmp and prints the 
> path to stdout.
> {code}
> int ret = PyRun_SimpleString("import objgraph; 
> objgraph.show_backrefs(config.global_agent, max_depth=10)\n\n");
>

[jira] [Commented] (DISPATCH-1962) Make LSan suppressions more targeted and specific (within reason)

2021-03-20 Thread ASF GitHub Bot (Jira)



[ 
https://issues.apache.org/jira/browse/DISPATCH-1962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17305462#comment-17305462
 ] 

ASF GitHub Bot commented on DISPATCH-1962:
--

jiridanek commented on pull request #1080:
URL: https://github.com/apache/qpid-dispatch/pull/1080#issuecomment-803367740


   This also takes care of the related leaks
   
   ```
   9: =
   9: ==2292==ERROR: LeakSanitizer: detected memory leaks
   9: 
   9: Indirect leak of 216 byte(s) in 1 object(s) allocated from:
   9: #0 0x7fa3b6a8bdc6 in calloc 
(/lib/x86_64-linux-gnu/libasan.so.5+0x10ddc6)
   9: #1 0x7fa3b63ce928 in qd_http_server ../src/http-libwebsockets.c:963
   9: #2 0x7fa3b63b5ec7 in qd_server ../src/server.c:1364
   9: #3 0x7fa3b6207fff in qd_dispatch_prepare ../src/dispatch.c:330
   9: #4 0x7fa3b1208ff4  (/lib/x86_64-linux-gnu/libffi.so.7+0x6ff4)
   9: #5 0x7ffce6d1b37f  ([stack]+0x1f37f)
   9: 
   9: Indirect leak of 64 byte(s) in 1 object(s) allocated from:
   9: #0 0x7fa3b6a8caa5 in posix_memalign 
(/lib/x86_64-linux-gnu/libasan.so.5+0x10eaa5)
   9: #1 0x7fa3b6270a94 in sys_cond ../src/posix/threading.c:77
   9: #2 0x7fa3b63ce96d in work_queue_init ../src/http-libwebsockets.c:231
   9: #3 0x7fa3b63ce96d in qd_http_server ../src/http-libwebsockets.c:965
   9: #4 0x7fa3b63b5ec7 in qd_server ../src/server.c:1364
   9: #5 0x7fa3b6207fff in qd_dispatch_prepare ../src/dispatch.c:330
   9: #6 0x7fa3b1208ff4  (/lib/x86_64-linux-gnu/libffi.so.7+0x6ff4)
   9: #7 0x7ffce6d1b37f  ([stack]+0x1f37f)
   9: 
   9: -
   9: Suppressions used:
   9:   count  bytes template
   9:   1   1160 run_unit_tests.c
   9:   1 64 sys_mutex
   9:   1 56 qdr_core_subscribe
   9:   9   5080 ^_PyObject_Realloc
   9: 540 862547 ^PyObject_Malloc$
   9:   1 32 ^PyThread_allocate_lock$
   9:   4   9897 ^PyMem_Malloc$
   9:   1856 ^_PyObject_GC_Resize$
   9:   1295 ^_PyBytes_Resize$
   9:   51058064 /libwebsockets.so
   9: -
   9: 
   9: SUMMARY: AddressSanitizer: 280 byte(s) leaked in 2 allocation(s).
7/37 Test  #9: unit_tests .***Failed
2.03 sec
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Make LSan suppressions more targeted and specific (within reason)
> -
>
> Key: DISPATCH-1962
> URL: https://issues.apache.org/jira/browse/DISPATCH-1962
> Project: Qpid Dispatch
>  Issue Type: Test
>Affects Versions: 1.15.0
>Reporter: Jiri Daněk
>Priority: Major
> Attachments: dispatch_leaking_agent.png
>
>
> There are some unfortunate suppressions in {{tests/lsan.supp}} at this moment:
> {code}
> leak:*libwebsockets*
> {code}
> This is way too broad. It suppresses leaks in Dispatch files, since it 
> matches e.g. {{src/http-libwebsockets.c}}.
> The stars at the beginning and end are actually assumed implicitly. If you do 
> not want substring match, you have to do ^foo$. LSan suppression format is 
> simplistic, very unlike Valgrind's.
> {code}
> leak:run_unit_tests.c
> {code}
> Same thing, any leaks revealed by running unit_tests get suppressed. This 
> suppression suppresses all leak traces that include run_unit_tests.c anywhere 
> in the stack. What't the point of running such tests under leak detector, 
> then?
> {code}
> leak:run_unit_tests.c
> leak:^libqpid-proton.so$
> {code}
> Same thing. The patterns suppress all leaks that include Python (or Proton) 
> anywhere in the stacktrace. That means there are huge blind spots where 
> dispatch leaks can hide. This is a weakness of the lsan.supp syntax (Valgrind 
> suppressions can be much more targeted and discerning).
> h3. Python leaks
> Leaks are known and there is ongoing effort to fight them: 
> https://bugs.python.org/issue1635741 (https://bugs.python.org/issue25302) and 
> https://www.python.org/dev/peps/pep-3121
> Here's valgrind suppression file from somebody who actually investigated the 
> Python leaks and identified the harmless ones: 
> https://github.com/libgit2/pygit2/blob/master/misc/valgrind-python.supp
> One example of a hidden leak in dispatch, which is revealed by making Python 
> suppressions more targetted:
> {code}
> 9: Direct leak of 56 byte(s) in 1 object(s) allocated from:
> 9: #0 0x7f78a3606e8f in __interceptor_malloc 
>

[jira] [Commented] (DISPATCH-1962) Make LSan suppressions more targeted and specific (within reason)

2021-03-20 Thread ASF GitHub Bot (Jira)



[ 
https://issues.apache.org/jira/browse/DISPATCH-1962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17305451#comment-17305451
 ] 

ASF GitHub Bot commented on DISPATCH-1962:
--

jiridanek opened a new pull request #1080:
URL: https://github.com/apache/qpid-dispatch/pull/1080


   This was a silly test issue. There was no good way for the test to stop the 
libwebsockets server, because of how the shutdown code was structured 
previously. All that's needed is to stop it, and the leak disappears.
   
   This may not be the best of fixes for it, I am open to better suggestions.
   
   ```
   9: Test Case thread_tests.test_condition: PASS
   9: 
   9: =
   9: ==10641==ERROR: LeakSanitizer: detected memory leaks
   9: 
   9: Indirect leak of 8397696 byte(s) in 5 object(s) allocated from:
   9: #0 0x7fb6364ee1f8 in __interceptor_realloc 
(/nix/store/g40sl3zh3nv52vj0mrl4iki5iphh5ika-gcc-10.2.0-lib/lib/libasan.so.6+0xad1f8)
   9: #1 0x7fb6353ebde5 in _realloc 
(/nix/store/yf2b40nmyhyyhlzjjniwg9a6g4w99k4s-libwebsockets-3.2.2/lib/libwebsockets.so.15+0x10de5)
   9: 
   9: -
   9: Suppressions used:
   9:   count  bytes template
   9:   2280 http-libwebsockets.c
   9:   1   1160 run_unit_tests.c
   9:   1 64 sys_mutex
   9:   1 56 qdr_core_subscribe
   9:   5   3096 ^_PyObject_Realloc
   9: 580 908616 ^PyObject_Malloc$
   9:   1 32 ^PyThread_allocate_lock$
   9:   5  11945 ^PyMem_Malloc$
   9:   1840 ^_PyObject_GC_Resize$
   9:   1319 ^_PyBytes_Resize$
   9: -
   9: 
   9: SUMMARY: AddressSanitizer: 8397696 byte(s) leaked in 5 allocation(s).
9/73 Test  #9: unit_tests ***Failed 
   1.47 sec
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Make LSan suppressions more targeted and specific (within reason)
> -
>
> Key: DISPATCH-1962
> URL: https://issues.apache.org/jira/browse/DISPATCH-1962
> Project: Qpid Dispatch
>  Issue Type: Test
>Affects Versions: 1.15.0
>Reporter: Jiri Daněk
>Priority: Major
> Attachments: dispatch_leaking_agent.png
>
>
> There are some unfortunate suppressions in {{tests/lsan.supp}} at this moment:
> {code}
> leak:*libwebsockets*
> {code}
> This is way too broad. It suppresses leaks in Dispatch files, since it 
> matches e.g. {{src/http-libwebsockets.c}}.
> The stars at the beginning and end are actually assumed implicitly. If you do 
> not want substring match, you have to do ^foo$. LSan suppression format is 
> simplistic, very unlike Valgrind's.
> {code}
> leak:run_unit_tests.c
> {code}
> Same thing, any leaks revealed by running unit_tests get suppressed. This 
> suppression suppresses all leak traces that include run_unit_tests.c anywhere 
> in the stack. What't the point of running such tests under leak detector, 
> then?
> {code}
> leak:run_unit_tests.c
> leak:^libqpid-proton.so$
> {code}
> Same thing. The patterns suppress all leaks that include Python (or Proton) 
> anywhere in the stacktrace. That means there are huge blind spots where 
> dispatch leaks can hide. This is a weakness of the lsan.supp syntax (Valgrind 
> suppressions can be much more targeted and discerning).
> h3. Python leaks
> Leaks are known and there is ongoing effort to fight them: 
> https://bugs.python.org/issue1635741 (https://bugs.python.org/issue25302) and 
> https://www.python.org/dev/peps/pep-3121
> Here's valgrind suppression file from somebody who actually investigated the 
> Python leaks and identified the harmless ones: 
> https://github.com/libgit2/pygit2/blob/master/misc/valgrind-python.supp
> One example of a hidden leak in dispatch, which is revealed by making Python 
> suppressions more targetted:
> {code}
> 9: Direct leak of 56 byte(s) in 1 object(s) allocated from:
> 9: #0 0x7f78a3606e8f in __interceptor_malloc 
> (/nix/store/g40sl3zh3nv52vj0mrl4iki5iphh5ika-gcc-10.2.0-lib/lib/libasan.so.6+0xace8f)
> 9: #1 0x7f78a2d64afb in qd_malloc ../include/qpid/dispatch/ctools.h:229
> 9: #2 0x7f78a2d657da in qdr_core_subscribe 
> ../src/router_core/route_tables.c:149
> 9: #3 0x7f78a2c83072 in IoAdapter_init ../src/python_embedded.c:711
> 9: #4 0x7f78a2353a6c in type_call 
> (/nix/store/r85nxfnwiv45nbmf5yb60jj8ajim4m7w-python3-3.8.5/lib/libpython3.8.so.1.0+0x165a6c)
> {code}
> The problem is in
> {code}
> class Agent:
> ...
> def

[jira] [Commented] (DISPATCH-1962) Make LSan suppressions more targeted and specific (within reason)

2021-03-20 Thread ASF GitHub Bot (Jira)



[ 
https://issues.apache.org/jira/browse/DISPATCH-1962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17305404#comment-17305404
 ] 

ASF GitHub Bot commented on DISPATCH-1962:
--

jiridanek commented on pull request #1049:
URL: https://github.com/apache/qpid-dispatch/pull/1049#issuecomment-803294379


   Ready for review. Just a missing free() at the end of a test.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Make LSan suppressions more targeted and specific (within reason)
> -
>
> Key: DISPATCH-1962
> URL: https://issues.apache.org/jira/browse/DISPATCH-1962
> Project: Qpid Dispatch
>  Issue Type: Test
>Affects Versions: 1.15.0
>Reporter: Jiri Daněk
>Priority: Major
> Attachments: dispatch_leaking_agent.png
>
>
> There are some unfortunate suppressions in {{tests/lsan.supp}} at this moment:
> {code}
> leak:*libwebsockets*
> {code}
> This is way too broad. It suppresses leaks in Dispatch files, since it 
> matches e.g. {{src/http-libwebsockets.c}}.
> The stars at the beginning and end are actually assumed implicitly. If you do 
> not want substring match, you have to do ^foo$. LSan suppression format is 
> simplistic, very unlike Valgrind's.
> {code}
> leak:run_unit_tests.c
> {code}
> Same thing, any leaks revealed by running unit_tests get suppressed. This 
> suppression suppresses all leak traces that include run_unit_tests.c anywhere 
> in the stack. What't the point of running such tests under leak detector, 
> then?
> {code}
> leak:run_unit_tests.c
> leak:^libqpid-proton.so$
> {code}
> Same thing. The patterns suppress all leaks that include Python (or Proton) 
> anywhere in the stacktrace. That means there are huge blind spots where 
> dispatch leaks can hide. This is a weakness of the lsan.supp syntax (Valgrind 
> suppressions can be much more targeted and discerning).
> h3. Python leaks
> Leaks are known and there is ongoing effort to fight them: 
> https://bugs.python.org/issue1635741 (https://bugs.python.org/issue25302) and 
> https://www.python.org/dev/peps/pep-3121
> Here's valgrind suppression file from somebody who actually investigated the 
> Python leaks and identified the harmless ones: 
> https://github.com/libgit2/pygit2/blob/master/misc/valgrind-python.supp
> One example of a hidden leak in dispatch, which is revealed by making Python 
> suppressions more targetted:
> {code}
> 9: Direct leak of 56 byte(s) in 1 object(s) allocated from:
> 9: #0 0x7f78a3606e8f in __interceptor_malloc 
> (/nix/store/g40sl3zh3nv52vj0mrl4iki5iphh5ika-gcc-10.2.0-lib/lib/libasan.so.6+0xace8f)
> 9: #1 0x7f78a2d64afb in qd_malloc ../include/qpid/dispatch/ctools.h:229
> 9: #2 0x7f78a2d657da in qdr_core_subscribe 
> ../src/router_core/route_tables.c:149
> 9: #3 0x7f78a2c83072 in IoAdapter_init ../src/python_embedded.c:711
> 9: #4 0x7f78a2353a6c in type_call 
> (/nix/store/r85nxfnwiv45nbmf5yb60jj8ajim4m7w-python3-3.8.5/lib/libpython3.8.so.1.0+0x165a6c)
> {code}
> The problem is in
> {code}
> class Agent:
> ...
> def activate(self, address):
> ...
> self.io = IoAdapter(self.receive, address, 'L', '0', 
> TREATMENT_ANYCAST_CLOSEST)
> {code}
> IoAdapter refers to Agent (through the bound method reference self.receive) 
> and Agent refers to IoAdapter (through property self.io). Since IoAdapter is 
> implemented in C and does not implement support for Python's cyclic GC, there 
> is no way to break the cycle.
> Heap dump in attachment. The bound method is at the top of the picture. 
> (Ignore the Mock objects, I was trying to simplify the picture while not 
> getting crashes due to too much meddling).
> h3. Random observations
> It is possible to build special Debug build of Python, which has tools to 
> detect leaks, asserts to prevent negative refcounts, etc. 
> https://pythonextensionpatterns.readthedocs.io/en/latest/debugging/debug_python.html#debug-version-of-python-memory-alloc-label
> Use the following to detect python leaks (instead of valgrind)
> https://docs.python.org/3/library/tracemalloc.html
> Use https://pypi.org/project/objgraph (with graphviz) to view heap object 
> trees. The following renders the picture as a png under /tmp and prints the 
> path to stdout.
> {code}
> int ret = PyRun_SimpleString("import objgraph; 
> objgraph.show_backrefs(config.global_agent, max_depth=10)\n\n");
> PyErr_PrintEx(0);
> assert(ret == 0);
> {code}
> h2. Tools
> h3. CPyChecker
> https://dmalcolm.fedorapeople.org/presentations/PyCon-US-2013/PyCon-US-2013-dmalcolm-StaticAnalysis.html#(38)
>

[jira] [Commented] (DISPATCH-1962) Make LSan suppressions more targeted and specific (within reason)

2021-03-19 Thread ASF GitHub Bot (Jira)



[ 
https://issues.apache.org/jira/browse/DISPATCH-1962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17305165#comment-17305165
 ] 

ASF GitHub Bot commented on DISPATCH-1962:
--

jiridanek commented on pull request #1051:
URL: https://github.com/apache/qpid-dispatch/pull/1051#issuecomment-803085984


   @kgiusti I take it that this leak is extremely low priority and maybe it 
makes sense to instead simply make the suppression permanent. Is that a correct 
understanding?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Make LSan suppressions more targeted and specific (within reason)
> -
>
> Key: DISPATCH-1962
> URL: https://issues.apache.org/jira/browse/DISPATCH-1962
> Project: Qpid Dispatch
>  Issue Type: Test
>Affects Versions: 1.15.0
>Reporter: Jiri Daněk
>Priority: Major
> Attachments: dispatch_leaking_agent.png
>
>
> There are some unfortunate suppressions in {{tests/lsan.supp}} at this moment:
> {code}
> leak:*libwebsockets*
> {code}
> This is way too broad. It suppresses leaks in Dispatch files, since it 
> matches e.g. {{src/http-libwebsockets.c}}.
> The stars at the beginning and end are actually assumed implicitly. If you do 
> not want substring match, you have to do ^foo$. LSan suppression format is 
> simplistic, very unlike Valgrind's.
> {code}
> leak:run_unit_tests.c
> {code}
> Same thing, any leaks revealed by running unit_tests get suppressed. This 
> suppression suppresses all leak traces that include run_unit_tests.c anywhere 
> in the stack. What't the point of running such tests under leak detector, 
> then?
> {code}
> leak:run_unit_tests.c
> leak:^libqpid-proton.so$
> {code}
> Same thing. The patterns suppress all leaks that include Python (or Proton) 
> anywhere in the stacktrace. That means there are huge blind spots where 
> dispatch leaks can hide. This is a weakness of the lsan.supp syntax (Valgrind 
> suppressions can be much more targeted and discerning).
> h3. Python leaks
> Leaks are known and there is ongoing effort to fight them: 
> https://bugs.python.org/issue1635741 (https://bugs.python.org/issue25302) and 
> https://www.python.org/dev/peps/pep-3121
> Here's valgrind suppression file from somebody who actually investigated the 
> Python leaks and identified the harmless ones: 
> https://github.com/libgit2/pygit2/blob/master/misc/valgrind-python.supp
> One example of a hidden leak in dispatch, which is revealed by making Python 
> suppressions more targetted:
> {code}
> 9: Direct leak of 56 byte(s) in 1 object(s) allocated from:
> 9: #0 0x7f78a3606e8f in __interceptor_malloc 
> (/nix/store/g40sl3zh3nv52vj0mrl4iki5iphh5ika-gcc-10.2.0-lib/lib/libasan.so.6+0xace8f)
> 9: #1 0x7f78a2d64afb in qd_malloc ../include/qpid/dispatch/ctools.h:229
> 9: #2 0x7f78a2d657da in qdr_core_subscribe 
> ../src/router_core/route_tables.c:149
> 9: #3 0x7f78a2c83072 in IoAdapter_init ../src/python_embedded.c:711
> 9: #4 0x7f78a2353a6c in type_call 
> (/nix/store/r85nxfnwiv45nbmf5yb60jj8ajim4m7w-python3-3.8.5/lib/libpython3.8.so.1.0+0x165a6c)
> {code}
> The problem is in
> {code}
> class Agent:
> ...
> def activate(self, address):
> ...
> self.io = IoAdapter(self.receive, address, 'L', '0', 
> TREATMENT_ANYCAST_CLOSEST)
> {code}
> IoAdapter refers to Agent (through the bound method reference self.receive) 
> and Agent refers to IoAdapter (through property self.io). Since IoAdapter is 
> implemented in C and does not implement support for Python's cyclic GC, there 
> is no way to break the cycle.
> Heap dump in attachment. The bound method is at the top of the picture. 
> (Ignore the Mock objects, I was trying to simplify the picture while not 
> getting crashes due to too much meddling).
> h3. Random observations
> It is possible to build special Debug build of Python, which has tools to 
> detect leaks, asserts to prevent negative refcounts, etc. 
> https://pythonextensionpatterns.readthedocs.io/en/latest/debugging/debug_python.html#debug-version-of-python-memory-alloc-label
> Use the following to detect python leaks (instead of valgrind)
> https://docs.python.org/3/library/tracemalloc.html
> Use https://pypi.org/project/objgraph (with graphviz) to view heap object 
> trees. The following renders the picture as a png under /tmp and prints the 
> path to stdout.
> {code}
> int ret = PyRun_SimpleString("import objgraph; 
> objgraph.show_backrefs(config.global_agent, max_depth=10)\n\n");
> PyErr_PrintEx(0);
> assert(ret == 0);
> {code}
> h2. Tools
> h3. CPyChecker
>

[jira] [Commented] (DISPATCH-1962) Make LSan suppressions more targeted and specific (within reason)

2021-03-19 Thread ASF GitHub Bot (Jira)



[ 
https://issues.apache.org/jira/browse/DISPATCH-1962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17305121#comment-17305121
 ] 

ASF GitHub Bot commented on DISPATCH-1962:
--

jiridanek merged pull request #1048:
URL: https://github.com/apache/qpid-dispatch/pull/1048


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Make LSan suppressions more targeted and specific (within reason)
> -
>
> Key: DISPATCH-1962
> URL: https://issues.apache.org/jira/browse/DISPATCH-1962
> Project: Qpid Dispatch
>  Issue Type: Test
>Affects Versions: 1.15.0
>Reporter: Jiri Daněk
>Priority: Major
> Attachments: dispatch_leaking_agent.png
>
>
> There are some unfortunate suppressions in {{tests/lsan.supp}} at this moment:
> {code}
> leak:*libwebsockets*
> {code}
> This is way too broad. It suppresses leaks in Dispatch files, since it 
> matches e.g. {{src/http-libwebsockets.c}}.
> The stars at the beginning and end are actually assumed implicitly. If you do 
> not want substring match, you have to do ^foo$. LSan suppression format is 
> simplistic, very unlike Valgrind's.
> {code}
> leak:run_unit_tests.c
> {code}
> Same thing, any leaks revealed by running unit_tests get suppressed. This 
> suppression suppresses all leak traces that include run_unit_tests.c anywhere 
> in the stack. What't the point of running such tests under leak detector, 
> then?
> {code}
> leak:run_unit_tests.c
> leak:^libqpid-proton.so$
> {code}
> Same thing. The patterns suppress all leaks that include Python (or Proton) 
> anywhere in the stacktrace. That means there are huge blind spots where 
> dispatch leaks can hide. This is a weakness of the lsan.supp syntax (Valgrind 
> suppressions can be much more targeted and discerning).
> h3. Python leaks
> Leaks are known and there is ongoing effort to fight them: 
> https://bugs.python.org/issue1635741 (https://bugs.python.org/issue25302) and 
> https://www.python.org/dev/peps/pep-3121
> Here's valgrind suppression file from somebody who actually investigated the 
> Python leaks and identified the harmless ones: 
> https://github.com/libgit2/pygit2/blob/master/misc/valgrind-python.supp
> One example of a hidden leak in dispatch, which is revealed by making Python 
> suppressions more targetted:
> {code}
> 9: Direct leak of 56 byte(s) in 1 object(s) allocated from:
> 9: #0 0x7f78a3606e8f in __interceptor_malloc 
> (/nix/store/g40sl3zh3nv52vj0mrl4iki5iphh5ika-gcc-10.2.0-lib/lib/libasan.so.6+0xace8f)
> 9: #1 0x7f78a2d64afb in qd_malloc ../include/qpid/dispatch/ctools.h:229
> 9: #2 0x7f78a2d657da in qdr_core_subscribe 
> ../src/router_core/route_tables.c:149
> 9: #3 0x7f78a2c83072 in IoAdapter_init ../src/python_embedded.c:711
> 9: #4 0x7f78a2353a6c in type_call 
> (/nix/store/r85nxfnwiv45nbmf5yb60jj8ajim4m7w-python3-3.8.5/lib/libpython3.8.so.1.0+0x165a6c)
> {code}
> The problem is in
> {code}
> class Agent:
> ...
> def activate(self, address):
> ...
> self.io = IoAdapter(self.receive, address, 'L', '0', 
> TREATMENT_ANYCAST_CLOSEST)
> {code}
> IoAdapter refers to Agent (through the bound method reference self.receive) 
> and Agent refers to IoAdapter (through property self.io). Since IoAdapter is 
> implemented in C and does not implement support for Python's cyclic GC, there 
> is no way to break the cycle.
> Heap dump in attachment. The bound method is at the top of the picture. 
> (Ignore the Mock objects, I was trying to simplify the picture while not 
> getting crashes due to too much meddling).
> h3. Random observations
> It is possible to build special Debug build of Python, which has tools to 
> detect leaks, asserts to prevent negative refcounts, etc. 
> https://pythonextensionpatterns.readthedocs.io/en/latest/debugging/debug_python.html#debug-version-of-python-memory-alloc-label
> Use the following to detect python leaks (instead of valgrind)
> https://docs.python.org/3/library/tracemalloc.html
> Use https://pypi.org/project/objgraph (with graphviz) to view heap object 
> trees. The following renders the picture as a png under /tmp and prints the 
> path to stdout.
> {code}
> int ret = PyRun_SimpleString("import objgraph; 
> objgraph.show_backrefs(config.global_agent, max_depth=10)\n\n");
> PyErr_PrintEx(0);
> assert(ret == 0);
> {code}
> h2. Tools
> h3. CPyChecker
> https://dmalcolm.fedorapeople.org/presentations/PyCon-US-2013/PyCon-US-2013-dmalcolm-StaticAnalysis.html#(38)
> https://emptysqua.re/blog/analyzing-python-c-extensions-with-cpychecker/
> https://nedbatchelder.com/blog/201502/cpychecker.html
> h3. Pungi
>

[jira] [Commented] (DISPATCH-1962) Make LSan suppressions more targeted and specific (within reason)

2021-03-19 Thread ASF subversion and git services (Jira)



[ 
https://issues.apache.org/jira/browse/DISPATCH-1962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17305120#comment-17305120
 ] 

ASF subversion and git services commented on DISPATCH-1962:
---

Commit c6ea12e4d0831526b1a7a27958a33de5641d9cdb in qpid-dispatch's branch 
refs/heads/master from Jiri Daněk
[ https://gitbox.apache.org/repos/asf?p=qpid-dispatch.git;h=c6ea12e ]

DISPATCH-1962 Make LSan suppressions more targeted and specific (#1048)



> Make LSan suppressions more targeted and specific (within reason)
> -
>
> Key: DISPATCH-1962
> URL: https://issues.apache.org/jira/browse/DISPATCH-1962
> Project: Qpid Dispatch
>  Issue Type: Test
>Affects Versions: 1.15.0
>Reporter: Jiri Daněk
>Priority: Major
> Attachments: dispatch_leaking_agent.png
>
>
> There are some unfortunate suppressions in {{tests/lsan.supp}} at this moment:
> {code}
> leak:*libwebsockets*
> {code}
> This is way too broad. It suppresses leaks in Dispatch files, since it 
> matches e.g. {{src/http-libwebsockets.c}}.
> The stars at the beginning and end are actually assumed implicitly. If you do 
> not want substring match, you have to do ^foo$. LSan suppression format is 
> simplistic, very unlike Valgrind's.
> {code}
> leak:run_unit_tests.c
> {code}
> Same thing, any leaks revealed by running unit_tests get suppressed. This 
> suppression suppresses all leak traces that include run_unit_tests.c anywhere 
> in the stack. What't the point of running such tests under leak detector, 
> then?
> {code}
> leak:run_unit_tests.c
> leak:^libqpid-proton.so$
> {code}
> Same thing. The patterns suppress all leaks that include Python (or Proton) 
> anywhere in the stacktrace. That means there are huge blind spots where 
> dispatch leaks can hide. This is a weakness of the lsan.supp syntax (Valgrind 
> suppressions can be much more targeted and discerning).
> h3. Python leaks
> Leaks are known and there is ongoing effort to fight them: 
> https://bugs.python.org/issue1635741 (https://bugs.python.org/issue25302) and 
> https://www.python.org/dev/peps/pep-3121
> Here's valgrind suppression file from somebody who actually investigated the 
> Python leaks and identified the harmless ones: 
> https://github.com/libgit2/pygit2/blob/master/misc/valgrind-python.supp
> One example of a hidden leak in dispatch, which is revealed by making Python 
> suppressions more targetted:
> {code}
> 9: Direct leak of 56 byte(s) in 1 object(s) allocated from:
> 9: #0 0x7f78a3606e8f in __interceptor_malloc 
> (/nix/store/g40sl3zh3nv52vj0mrl4iki5iphh5ika-gcc-10.2.0-lib/lib/libasan.so.6+0xace8f)
> 9: #1 0x7f78a2d64afb in qd_malloc ../include/qpid/dispatch/ctools.h:229
> 9: #2 0x7f78a2d657da in qdr_core_subscribe 
> ../src/router_core/route_tables.c:149
> 9: #3 0x7f78a2c83072 in IoAdapter_init ../src/python_embedded.c:711
> 9: #4 0x7f78a2353a6c in type_call 
> (/nix/store/r85nxfnwiv45nbmf5yb60jj8ajim4m7w-python3-3.8.5/lib/libpython3.8.so.1.0+0x165a6c)
> {code}
> The problem is in
> {code}
> class Agent:
> ...
> def activate(self, address):
> ...
> self.io = IoAdapter(self.receive, address, 'L', '0', 
> TREATMENT_ANYCAST_CLOSEST)
> {code}
> IoAdapter refers to Agent (through the bound method reference self.receive) 
> and Agent refers to IoAdapter (through property self.io). Since IoAdapter is 
> implemented in C and does not implement support for Python's cyclic GC, there 
> is no way to break the cycle.
> Heap dump in attachment. The bound method is at the top of the picture. 
> (Ignore the Mock objects, I was trying to simplify the picture while not 
> getting crashes due to too much meddling).
> h3. Random observations
> It is possible to build special Debug build of Python, which has tools to 
> detect leaks, asserts to prevent negative refcounts, etc. 
> https://pythonextensionpatterns.readthedocs.io/en/latest/debugging/debug_python.html#debug-version-of-python-memory-alloc-label
> Use the following to detect python leaks (instead of valgrind)
> https://docs.python.org/3/library/tracemalloc.html
> Use https://pypi.org/project/objgraph (with graphviz) to view heap object 
> trees. The following renders the picture as a png under /tmp and prints the 
> path to stdout.
> {code}
> int ret = PyRun_SimpleString("import objgraph; 
> objgraph.show_backrefs(config.global_agent, max_depth=10)\n\n");
> PyErr_PrintEx(0);
> assert(ret == 0);
> {code}
> h2. Tools
> h3. CPyChecker
> https://dmalcolm.fedorapeople.org/presentations/PyCon-US-2013/PyCon-US-2013-dmalcolm-StaticAnalysis.html#(38)
> https://emptysqua.re/blog/analyzing-python-c-extensions-with-cpychecker/
> https://nedbatchelder.com/blog/201502/cpychecker.html
> h3. Pungi
> http://www.cse.psu.edu/~gxt29/papers/refcount.pdf
> h3. Use CFFI or

[jira] [Commented] (DISPATCH-1962) Make LSan suppressions more targeted and specific (within reason)

2021-03-19 Thread ASF GitHub Bot (Jira)



[ 
https://issues.apache.org/jira/browse/DISPATCH-1962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17305015#comment-17305015
 ] 

ASF GitHub Bot commented on DISPATCH-1962:
--

jiridanek commented on a change in pull request #1048:
URL: https://github.com/apache/qpid-dispatch/pull/1048#discussion_r597822956



##
File path: cmake/RuntimeChecks.cmake
##
@@ -106,9 +106,17 @@ elseif(RUNTIME_CHECK STREQUAL "asan")
 message(FATAL_ERROR "libubsan not installed - address sanitizer not 
available")
   endif(UBSAN_LIBRARY-NOTFOUND)
   message(STATUS "Runtime memory checker: gcc/clang address sanitizers")
+  option(SANITIZE_3RD_PARTY "Detect leaks in 3rd party libraries used by 
Dispatch while running tests" OFF)
+  file (COPY "${CMAKE_SOURCE_DIR}/tests/lsan.supp" DESTINATION 
"${CMAKE_BINARY_DIR}/tests")

Review comment:
   `configure_file` does not work all that well for my purposes; I want to 
essentially call a `cat` to concatenate suppression lists together, so I used a 
custom command




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Make LSan suppressions more targeted and specific (within reason)
> -
>
> Key: DISPATCH-1962
> URL: https://issues.apache.org/jira/browse/DISPATCH-1962
> Project: Qpid Dispatch
>  Issue Type: Test
>Affects Versions: 1.15.0
>Reporter: Jiri Daněk
>Priority: Major
> Attachments: dispatch_leaking_agent.png
>
>
> There are some unfortunate suppressions in {{tests/lsan.supp}} at this moment:
> {code}
> leak:*libwebsockets*
> {code}
> This is way too broad. It suppresses leaks in Dispatch files, since it 
> matches e.g. {{src/http-libwebsockets.c}}.
> The stars at the beginning and end are actually assumed implicitly. If you do 
> not want substring match, you have to do ^foo$. LSan suppression format is 
> simplistic, very unlike Valgrind's.
> {code}
> leak:run_unit_tests.c
> {code}
> Same thing, any leaks revealed by running unit_tests get suppressed. This 
> suppression suppresses all leak traces that include run_unit_tests.c anywhere 
> in the stack. What't the point of running such tests under leak detector, 
> then?
> {code}
> leak:run_unit_tests.c
> leak:^libqpid-proton.so$
> {code}
> Same thing. The patterns suppress all leaks that include Python (or Proton) 
> anywhere in the stacktrace. That means there are huge blind spots where 
> dispatch leaks can hide. This is a weakness of the lsan.supp syntax (Valgrind 
> suppressions can be much more targeted and discerning).
> h3. Python leaks
> Leaks are known and there is ongoing effort to fight them: 
> https://bugs.python.org/issue1635741 (https://bugs.python.org/issue25302) and 
> https://www.python.org/dev/peps/pep-3121
> Here's valgrind suppression file from somebody who actually investigated the 
> Python leaks and identified the harmless ones: 
> https://github.com/libgit2/pygit2/blob/master/misc/valgrind-python.supp
> One example of a hidden leak in dispatch, which is revealed by making Python 
> suppressions more targetted:
> {code}
> 9: Direct leak of 56 byte(s) in 1 object(s) allocated from:
> 9: #0 0x7f78a3606e8f in __interceptor_malloc 
> (/nix/store/g40sl3zh3nv52vj0mrl4iki5iphh5ika-gcc-10.2.0-lib/lib/libasan.so.6+0xace8f)
> 9: #1 0x7f78a2d64afb in qd_malloc ../include/qpid/dispatch/ctools.h:229
> 9: #2 0x7f78a2d657da in qdr_core_subscribe 
> ../src/router_core/route_tables.c:149
> 9: #3 0x7f78a2c83072 in IoAdapter_init ../src/python_embedded.c:711
> 9: #4 0x7f78a2353a6c in type_call 
> (/nix/store/r85nxfnwiv45nbmf5yb60jj8ajim4m7w-python3-3.8.5/lib/libpython3.8.so.1.0+0x165a6c)
> {code}
> The problem is in
> {code}
> class Agent:
> ...
> def activate(self, address):
> ...
> self.io = IoAdapter(self.receive, address, 'L', '0', 
> TREATMENT_ANYCAST_CLOSEST)
> {code}
> IoAdapter refers to Agent (through the bound method reference self.receive) 
> and Agent refers to IoAdapter (through property self.io). Since IoAdapter is 
> implemented in C and does not implement support for Python's cyclic GC, there 
> is no way to break the cycle.
> Heap dump in attachment. The bound method is at the top of the picture. 
> (Ignore the Mock objects, I was trying to simplify the picture while not 
> getting crashes due to too much meddling).
> h3. Random observations
> It is possible to build special Debug build of Python, which has tools to 
> detect leaks, asserts to prevent negative refcounts, etc. 
> https://pythonextensionpatterns.readthedocs.io/en/latest/debugging/debug_python.html#debug-version-of-python-memory-alloc-label
> Use the

[jira] [Commented] (DISPATCH-1962) Make LSan suppressions more targeted and specific (within reason)

2021-02-24 Thread ASF GitHub Bot (Jira)



[ 
https://issues.apache.org/jira/browse/DISPATCH-1962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17290755#comment-17290755
 ] 

ASF GitHub Bot commented on DISPATCH-1962:
--

jiridanek commented on a change in pull request #1048:
URL: https://github.com/apache/qpid-dispatch/pull/1048#discussion_r581731556



##
File path: tests/lsan.supp
##
@@ -23,15 +44,56 @@ leak:_ctypes_alloc_format_string
 leak:__strdup
 
 
- Miscellaneous 3rd party libraries, test code, etc:
+ Miscellaneous 3rd party libraries:
 
 
-leak:*libpython*
-leak:*libwebsockets*
-leak:*python2*
+### Python
 
-# We should be able to uncomment these once all known dispatch leaks have been 
fixed
-leak:*libqpid-proton*
+# these Python leaks happen even after simple Py_Initialize(); Py_Finalize();
+#  https://bugs.python.org/issue1635741
+leak:^_PyObject_Realloc
+leak:^PyObject_Malloc$
+leak:^PyThread_allocate_lock$
 
-# Ignore test code
-leak:run_unit_tests.c
+# the PyMalloc mechanism is incompatible with Valgrind, it must be disabled or 
reported "leaks" must be suppressed

Review comment:
   It isn't really overridden. The two .so suppressions simply cover all 
the Python suppressions that went before. I am thinking I could remove the 
conditional CMake logic and put the .so suppressions for python there always.
   
   The only leak I flushed out by doing suppressions in this more complicated 
way is the `IoAdapter` one, and that is fairly insignificant. There is about 4 
or 5 instances of the class, only. In the whole qdr run. So that leak is hardly 
the most significant one.





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Make LSan suppressions more targeted and specific (within reason)
> -
>
> Key: DISPATCH-1962
> URL: https://issues.apache.org/jira/browse/DISPATCH-1962
> Project: Qpid Dispatch
>  Issue Type: Test
>Affects Versions: 1.15.0
>Reporter: Jiri Daněk
>Priority: Major
> Attachments: dispatch_leaking_agent.png
>
>
> There are some unfortunate suppressions in {{tests/lsan.supp}} at this moment:
> {code}
> leak:*libwebsockets*
> {code}
> This is way too broad. It suppresses leaks in Dispatch files, since it 
> matches e.g. {{src/http-libwebsockets.c}}.
> The stars at the beginning and end are actually assumed implicitly. If you do 
> not want substring match, you have to do ^foo$. LSan suppression format is 
> simplistic, very unlike Valgrind's.
> {code}
> leak:run_unit_tests.c
> {code}
> Same thing, any leaks revealed by running unit_tests get suppressed. This 
> suppression suppresses all leak traces that include run_unit_tests.c anywhere 
> in the stack. What't the point of running such tests under leak detector, 
> then?
> {code}
> leak:run_unit_tests.c
> leak:^libqpid-proton.so$
> {code}
> Same thing. The patterns suppress all leaks that include Python (or Proton) 
> anywhere in the stacktrace. That means there are huge blind spots where 
> dispatch leaks can hide. This is a weakness of the lsan.supp syntax (Valgrind 
> suppressions can be much more targeted and discerning).
> h3. Python leaks
> Leaks are known and there is ongoing effort to fight them: 
> https://bugs.python.org/issue1635741 (https://bugs.python.org/issue25302) and 
> https://www.python.org/dev/peps/pep-3121
> Here's valgrind suppression file from somebody who actually investigated the 
> Python leaks and identified the harmless ones: 
> https://github.com/libgit2/pygit2/blob/master/misc/valgrind-python.supp
> One example of a hidden leak in dispatch, which is revealed by making Python 
> suppressions more targetted:
> {code}
> 9: Direct leak of 56 byte(s) in 1 object(s) allocated from:
> 9: #0 0x7f78a3606e8f in __interceptor_malloc 
> (/nix/store/g40sl3zh3nv52vj0mrl4iki5iphh5ika-gcc-10.2.0-lib/lib/libasan.so.6+0xace8f)
> 9: #1 0x7f78a2d64afb in qd_malloc ../include/qpid/dispatch/ctools.h:229
> 9: #2 0x7f78a2d657da in qdr_core_subscribe 
> ../src/router_core/route_tables.c:149
> 9: #3 0x7f78a2c83072 in IoAdapter_init ../src/python_embedded.c:711
> 9: #4 0x7f78a2353a6c in type_call 
> (/nix/store/r85nxfnwiv45nbmf5yb60jj8ajim4m7w-python3-3.8.5/lib/libpython3.8.so.1.0+0x165a6c)
> {code}
> The problem is in
> {code}
> class Agent:
> ...
> def activate(self, address):
> ...
> self.io = IoAdapter(self.receive, address, 'L', '0', 
> TREATMENT_ANYCAST_CLOSEST)
> {code}
> IoAdapter refers to Agent (through the bound method reference self.receive) 
> and Agent refers to IoAdapter (through property

[jira] [Commented] (DISPATCH-1962) Make LSan suppressions more targeted and specific (within reason)

2021-02-24 Thread ASF GitHub Bot (Jira)



[ 
https://issues.apache.org/jira/browse/DISPATCH-1962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17289748#comment-17289748
 ] 

ASF GitHub Bot commented on DISPATCH-1962:
--

jiridanek commented on a change in pull request #1048:
URL: https://github.com/apache/qpid-dispatch/pull/1048#discussion_r581731556



##
File path: tests/lsan.supp
##
@@ -23,15 +44,56 @@ leak:_ctypes_alloc_format_string
 leak:__strdup
 
 
- Miscellaneous 3rd party libraries, test code, etc:
+ Miscellaneous 3rd party libraries:
 
 
-leak:*libpython*
-leak:*libwebsockets*
-leak:*python2*
+### Python
 
-# We should be able to uncomment these once all known dispatch leaks have been 
fixed
-leak:*libqpid-proton*
+# these Python leaks happen even after simple Py_Initialize(); Py_Finalize();
+#  https://bugs.python.org/issue1635741
+leak:^_PyObject_Realloc
+leak:^PyObject_Malloc$
+leak:^PyThread_allocate_lock$
 
-# Ignore test code
-leak:run_unit_tests.c
+# the PyMalloc mechanism is incompatible with Valgrind, it must be disabled or 
reported "leaks" must be suppressed

Review comment:
   It isn't really overridden. The two .so suppressions simply cover all 
the Python suppressions that went before. I am thinking I could remove the 
conditional CMake logic and put the .so suppressions for python there always.
   
   The only leak I flushed out by doing this in the more complicated way is th 
`IoAdapter` one, and that is fairly insignificant. There is about 4 or 5 
instances of the class, only. In the whole qdr run. So that leak is hardly the 
most significant one.





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Make LSan suppressions more targeted and specific (within reason)
> -
>
> Key: DISPATCH-1962
> URL: https://issues.apache.org/jira/browse/DISPATCH-1962
> Project: Qpid Dispatch
>  Issue Type: Test
>Affects Versions: 1.15.0
>Reporter: Jiri Daněk
>Priority: Major
> Attachments: dispatch_leaking_agent.png
>
>
> There are some unfortunate suppressions in {{tests/lsan.supp}} at this moment:
> {code}
> leak:*libwebsockets*
> {code}
> This is way too broad. It suppresses leaks in Dispatch files, since it 
> matches e.g. {{src/http-libwebsockets.c}}.
> The stars at the beginning and end are actually assumed implicitly. If you do 
> not want substring match, you have to do ^foo$. LSan suppression format is 
> simplistic, very unlike Valgrind's.
> {code}
> leak:run_unit_tests.c
> {code}
> Same thing, any leaks revealed by running unit_tests get suppressed. This 
> suppression suppresses all leak traces that include run_unit_tests.c anywhere 
> in the stack. What't the point of running such tests under leak detector, 
> then?
> {code}
> leak:run_unit_tests.c
> leak:^libqpid-proton.so$
> {code}
> Same thing. The patterns suppress all leaks that include Python (or Proton) 
> anywhere in the stacktrace. That means there are huge blind spots where 
> dispatch leaks can hide. This is a weakness of the lsan.supp syntax (Valgrind 
> suppressions can be much more targeted and discerning).
> h3. Python leaks
> Leaks are known and there is ongoing effort to fight them: 
> https://bugs.python.org/issue1635741 (https://bugs.python.org/issue25302) and 
> https://www.python.org/dev/peps/pep-3121
> Here's valgrind suppression file from somebody who actually investigated the 
> Python leaks and identified the harmless ones: 
> https://github.com/libgit2/pygit2/blob/master/misc/valgrind-python.supp
> One example of a hidden leak in dispatch, which is revealed by making Python 
> suppressions more targetted:
> {code}
> 9: Direct leak of 56 byte(s) in 1 object(s) allocated from:
> 9: #0 0x7f78a3606e8f in __interceptor_malloc 
> (/nix/store/g40sl3zh3nv52vj0mrl4iki5iphh5ika-gcc-10.2.0-lib/lib/libasan.so.6+0xace8f)
> 9: #1 0x7f78a2d64afb in qd_malloc ../include/qpid/dispatch/ctools.h:229
> 9: #2 0x7f78a2d657da in qdr_core_subscribe 
> ../src/router_core/route_tables.c:149
> 9: #3 0x7f78a2c83072 in IoAdapter_init ../src/python_embedded.c:711
> 9: #4 0x7f78a2353a6c in type_call 
> (/nix/store/r85nxfnwiv45nbmf5yb60jj8ajim4m7w-python3-3.8.5/lib/libpython3.8.so.1.0+0x165a6c)
> {code}
> The problem is in
> {code}
> class Agent:
> ...
> def activate(self, address):
> ...
> self.io = IoAdapter(self.receive, address, 'L', '0', 
> TREATMENT_ANYCAST_CLOSEST)
> {code}
> IoAdapter refers to Agent (through the bound method reference self.receive) 
> and Agent refers to IoAdapter (through property self.io).

[jira] [Commented] (DISPATCH-1962) Make LSan suppressions more targeted and specific (within reason)

2021-02-23 Thread ASF GitHub Bot (Jira)



[ 
https://issues.apache.org/jira/browse/DISPATCH-1962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17289359#comment-17289359
 ] 

ASF GitHub Bot commented on DISPATCH-1962:
--

ganeshmurthy commented on a change in pull request #1048:
URL: https://github.com/apache/qpid-dispatch/pull/1048#discussion_r581408056



##
File path: tests/lsan.supp
##
@@ -23,15 +44,56 @@ leak:_ctypes_alloc_format_string
 leak:__strdup
 
 
- Miscellaneous 3rd party libraries, test code, etc:
+ Miscellaneous 3rd party libraries:
 
 
-leak:*libpython*
-leak:*libwebsockets*
-leak:*python2*
+### Python
 
-# We should be able to uncomment these once all known dispatch leaks have been 
fixed
-leak:*libqpid-proton*
+# these Python leaks happen even after simple Py_Initialize(); Py_Finalize();
+#  https://bugs.python.org/issue1635741
+leak:^_PyObject_Realloc
+leak:^PyObject_Malloc$
+leak:^PyThread_allocate_lock$
 
-# Ignore test code
-leak:run_unit_tests.c
+# the PyMalloc mechanism is incompatible with Valgrind, it must be disabled or 
reported "leaks" must be suppressed

Review comment:
   In case you end up doing the configure_file from the first comment, you 
could add this as well? It is deceiving to see a  suppression list being 
overridden by two lines at the bottom.





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Make LSan suppressions more targeted and specific (within reason)
> -
>
> Key: DISPATCH-1962
> URL: https://issues.apache.org/jira/browse/DISPATCH-1962
> Project: Qpid Dispatch
>  Issue Type: Test
>Affects Versions: 1.15.0
>Reporter: Jiri Daněk
>Priority: Major
> Attachments: dispatch_leaking_agent.png
>
>
> There are some unfortunate suppressions in {{tests/lsan.supp}} at this moment:
> {code}
> leak:*libwebsockets*
> {code}
> This is way too broad. It suppresses leaks in Dispatch files, since it 
> matches e.g. {{src/http-libwebsockets.c}}.
> The stars at the beginning and end are actually assumed implicitly. If you do 
> not want substring match, you have to do ^foo$. LSan suppression format is 
> simplistic, very unlike Valgrind's.
> {code}
> leak:run_unit_tests.c
> {code}
> Same thing, any leaks revealed by running unit_tests get suppressed. This 
> suppression suppresses all leak traces that include run_unit_tests.c anywhere 
> in the stack. What't the point of running such tests under leak detector, 
> then?
> {code}
> leak:run_unit_tests.c
> leak:^libqpid-proton.so$
> {code}
> Same thing. The patterns suppress all leaks that include Python (or Proton) 
> anywhere in the stacktrace. That means there are huge blind spots where 
> dispatch leaks can hide. This is a weakness of the lsan.supp syntax (Valgrind 
> suppressions can be much more targeted and discerning).
> h3. Python leaks
> Leaks are known and there is ongoing effort to fight them: 
> https://bugs.python.org/issue1635741 (https://bugs.python.org/issue25302) and 
> https://www.python.org/dev/peps/pep-3121
> Here's valgrind suppression file from somebody who actually investigated the 
> Python leaks and identified the harmless ones: 
> https://github.com/libgit2/pygit2/blob/master/misc/valgrind-python.supp
> One example of a hidden leak in dispatch, which is revealed by making Python 
> suppressions more targetted:
> {code}
> 9: Direct leak of 56 byte(s) in 1 object(s) allocated from:
> 9: #0 0x7f78a3606e8f in __interceptor_malloc 
> (/nix/store/g40sl3zh3nv52vj0mrl4iki5iphh5ika-gcc-10.2.0-lib/lib/libasan.so.6+0xace8f)
> 9: #1 0x7f78a2d64afb in qd_malloc ../include/qpid/dispatch/ctools.h:229
> 9: #2 0x7f78a2d657da in qdr_core_subscribe 
> ../src/router_core/route_tables.c:149
> 9: #3 0x7f78a2c83072 in IoAdapter_init ../src/python_embedded.c:711
> 9: #4 0x7f78a2353a6c in type_call 
> (/nix/store/r85nxfnwiv45nbmf5yb60jj8ajim4m7w-python3-3.8.5/lib/libpython3.8.so.1.0+0x165a6c)
> {code}
> The problem is in
> {code}
> class Agent:
> ...
> def activate(self, address):
> ...
> self.io = IoAdapter(self.receive, address, 'L', '0', 
> TREATMENT_ANYCAST_CLOSEST)
> {code}
> IoAdapter refers to Agent (through the bound method reference self.receive) 
> and Agent refers to IoAdapter (through property self.io). Since IoAdapter is 
> implemented in C and does not implement support for Python's cyclic GC, there 
> is no way to break the cycle.
> Heap dump in attachment. The bound method is at the top of the picture. 
> (Ignore the Mock objects, I was trying to simplify the picture while not 
> getting

[jira] [Commented] (DISPATCH-1962) Make LSan suppressions more targeted and specific (within reason)

2021-02-23 Thread ASF GitHub Bot (Jira)



[ 
https://issues.apache.org/jira/browse/DISPATCH-1962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17289337#comment-17289337
 ] 

ASF GitHub Bot commented on DISPATCH-1962:
--

jiridanek commented on a change in pull request #1048:
URL: https://github.com/apache/qpid-dispatch/pull/1048#discussion_r581392560



##
File path: tests/lsan.supp
##
@@ -23,15 +44,56 @@ leak:_ctypes_alloc_format_string
 leak:__strdup
 
 
- Miscellaneous 3rd party libraries, test code, etc:
+ Miscellaneous 3rd party libraries:
 
 
-leak:*libpython*
-leak:*libwebsockets*
-leak:*python2*
+### Python
 
-# We should be able to uncomment these once all known dispatch leaks have been 
fixed
-leak:*libqpid-proton*
+# these Python leaks happen even after simple Py_Initialize(); Py_Finalize();
+#  https://bugs.python.org/issue1635741
+leak:^_PyObject_Realloc
+leak:^PyObject_Malloc$
+leak:^PyThread_allocate_lock$
 
-# Ignore test code
-leak:run_unit_tests.c
+# the PyMalloc mechanism is incompatible with Valgrind, it must be disabled or 
reported "leaks" must be suppressed

Review comment:
   Why?
   
   The way it is now (in the PR), the `lsan.supp` (or `lsan.supp.in` if I make 
that change) is a valid and usable LSan suppression file by itself, without any 
preprocessing. If I to do what you suggest, this will no longer be the case: 
(it will either contain CMake variable and won't be valid because of it, or it 
will not contain the individual Python function suppressions and will be 
incomplete).





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Make LSan suppressions more targeted and specific (within reason)
> -
>
> Key: DISPATCH-1962
> URL: https://issues.apache.org/jira/browse/DISPATCH-1962
> Project: Qpid Dispatch
>  Issue Type: Test
>Affects Versions: 1.15.0
>Reporter: Jiri Daněk
>Priority: Major
> Attachments: dispatch_leaking_agent.png
>
>
> There are some unfortunate suppressions in {{tests/lsan.supp}} at this moment:
> {code}
> leak:*libwebsockets*
> {code}
> This is way too broad. It suppresses leaks in Dispatch files, since it 
> matches e.g. {{src/http-libwebsockets.c}}.
> The stars at the beginning and end are actually assumed implicitly. If you do 
> not want substring match, you have to do ^foo$. LSan suppression format is 
> simplistic, very unlike Valgrind's.
> {code}
> leak:run_unit_tests.c
> {code}
> Same thing, any leaks revealed by running unit_tests get suppressed. This 
> suppression suppresses all leak traces that include run_unit_tests.c anywhere 
> in the stack. What't the point of running such tests under leak detector, 
> then?
> {code}
> leak:run_unit_tests.c
> leak:^libqpid-proton.so$
> {code}
> Same thing. The patterns suppress all leaks that include Python (or Proton) 
> anywhere in the stacktrace. That means there are huge blind spots where 
> dispatch leaks can hide. This is a weakness of the lsan.supp syntax (Valgrind 
> suppressions can be much more targeted and discerning).
> h3. Python leaks
> Leaks are known and there is ongoing effort to fight them: 
> https://bugs.python.org/issue1635741 (https://bugs.python.org/issue25302) and 
> https://www.python.org/dev/peps/pep-3121
> Here's valgrind suppression file from somebody who actually investigated the 
> Python leaks and identified the harmless ones: 
> https://github.com/libgit2/pygit2/blob/master/misc/valgrind-python.supp
> One example of a hidden leak in dispatch, which is revealed by making Python 
> suppressions more targetted:
> {code}
> 9: Direct leak of 56 byte(s) in 1 object(s) allocated from:
> 9: #0 0x7f78a3606e8f in __interceptor_malloc 
> (/nix/store/g40sl3zh3nv52vj0mrl4iki5iphh5ika-gcc-10.2.0-lib/lib/libasan.so.6+0xace8f)
> 9: #1 0x7f78a2d64afb in qd_malloc ../include/qpid/dispatch/ctools.h:229
> 9: #2 0x7f78a2d657da in qdr_core_subscribe 
> ../src/router_core/route_tables.c:149
> 9: #3 0x7f78a2c83072 in IoAdapter_init ../src/python_embedded.c:711
> 9: #4 0x7f78a2353a6c in type_call 
> (/nix/store/r85nxfnwiv45nbmf5yb60jj8ajim4m7w-python3-3.8.5/lib/libpython3.8.so.1.0+0x165a6c)
> {code}
> The problem is in
> {code}
> class Agent:
> ...
> def activate(self, address):
> ...
> self.io = IoAdapter(self.receive, address, 'L', '0', 
> TREATMENT_ANYCAST_CLOSEST)
> {code}
> IoAdapter refers to Agent (through the bound method reference self.receive) 
> and Agent refers to IoAdapter (through property self.io). Since IoAdapter is 
> implemented in C and does not implement support

[jira] [Commented] (DISPATCH-1962) Make LSan suppressions more targeted and specific (within reason)

2021-02-23 Thread ASF GitHub Bot (Jira)



[ 
https://issues.apache.org/jira/browse/DISPATCH-1962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17289333#comment-17289333
 ] 

ASF GitHub Bot commented on DISPATCH-1962:
--

ganeshmurthy commented on a change in pull request #1048:
URL: https://github.com/apache/qpid-dispatch/pull/1048#discussion_r581386934



##
File path: tests/lsan.supp
##
@@ -23,15 +44,56 @@ leak:_ctypes_alloc_format_string
 leak:__strdup
 
 
- Miscellaneous 3rd party libraries, test code, etc:
+ Miscellaneous 3rd party libraries:
 
 
-leak:*libpython*
-leak:*libwebsockets*
-leak:*python2*
+### Python
 
-# We should be able to uncomment these once all known dispatch leaks have been 
fixed
-leak:*libqpid-proton*
+# these Python leaks happen even after simple Py_Initialize(); Py_Finalize();
+#  https://bugs.python.org/issue1635741
+leak:^_PyObject_Realloc
+leak:^PyObject_Malloc$
+leak:^PyThread_allocate_lock$
 
-# Ignore test code
-leak:run_unit_tests.c
+# the PyMalloc mechanism is incompatible with Valgrind, it must be disabled or 
reported "leaks" must be suppressed

Review comment:
   It would be great if the following list *and* 
leak:/libpython2..so\nleak:/libpython3..so\n don't appear at the same time in 
the file.





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Make LSan suppressions more targeted and specific (within reason)
> -
>
> Key: DISPATCH-1962
> URL: https://issues.apache.org/jira/browse/DISPATCH-1962
> Project: Qpid Dispatch
>  Issue Type: Test
>Affects Versions: 1.15.0
>Reporter: Jiri Daněk
>Priority: Major
> Attachments: dispatch_leaking_agent.png
>
>
> There are some unfortunate suppressions in {{tests/lsan.supp}} at this moment:
> {code}
> leak:*libwebsockets*
> {code}
> This is way too broad. It suppresses leaks in Dispatch files, since it 
> matches e.g. {{src/http-libwebsockets.c}}.
> The stars at the beginning and end are actually assumed implicitly. If you do 
> not want substring match, you have to do ^foo$. LSan suppression format is 
> simplistic, very unlike Valgrind's.
> {code}
> leak:run_unit_tests.c
> {code}
> Same thing, any leaks revealed by running unit_tests get suppressed. This 
> suppression suppresses all leak traces that include run_unit_tests.c anywhere 
> in the stack. What't the point of running such tests under leak detector, 
> then?
> {code}
> leak:run_unit_tests.c
> leak:^libqpid-proton.so$
> {code}
> Same thing. The patterns suppress all leaks that include Python (or Proton) 
> anywhere in the stacktrace. That means there are huge blind spots where 
> dispatch leaks can hide. This is a weakness of the lsan.supp syntax (Valgrind 
> suppressions can be much more targeted and discerning).
> h3. Python leaks
> Leaks are known and there is ongoing effort to fight them: 
> https://bugs.python.org/issue1635741 (https://bugs.python.org/issue25302) and 
> https://www.python.org/dev/peps/pep-3121
> Here's valgrind suppression file from somebody who actually investigated the 
> Python leaks and identified the harmless ones: 
> https://github.com/libgit2/pygit2/blob/master/misc/valgrind-python.supp
> One example of a hidden leak in dispatch, which is revealed by making Python 
> suppressions more targetted:
> {code}
> 9: Direct leak of 56 byte(s) in 1 object(s) allocated from:
> 9: #0 0x7f78a3606e8f in __interceptor_malloc 
> (/nix/store/g40sl3zh3nv52vj0mrl4iki5iphh5ika-gcc-10.2.0-lib/lib/libasan.so.6+0xace8f)
> 9: #1 0x7f78a2d64afb in qd_malloc ../include/qpid/dispatch/ctools.h:229
> 9: #2 0x7f78a2d657da in qdr_core_subscribe 
> ../src/router_core/route_tables.c:149
> 9: #3 0x7f78a2c83072 in IoAdapter_init ../src/python_embedded.c:711
> 9: #4 0x7f78a2353a6c in type_call 
> (/nix/store/r85nxfnwiv45nbmf5yb60jj8ajim4m7w-python3-3.8.5/lib/libpython3.8.so.1.0+0x165a6c)
> {code}
> The problem is in
> {code}
> class Agent:
> ...
> def activate(self, address):
> ...
> self.io = IoAdapter(self.receive, address, 'L', '0', 
> TREATMENT_ANYCAST_CLOSEST)
> {code}
> IoAdapter refers to Agent (through the bound method reference self.receive) 
> and Agent refers to IoAdapter (through property self.io). Since IoAdapter is 
> implemented in C and does not implement support for Python's cyclic GC, there 
> is no way to break the cycle.
> Heap dump in attachment. The bound method is at the top of the picture. 
> (Ignore the Mock objects, I was trying to simplify the picture while not 
> getting crashes due to too much meddling).
> h3. Random

[jira] [Commented] (DISPATCH-1962) Make LSan suppressions more targeted and specific (within reason)

2021-02-23 Thread ASF GitHub Bot (Jira)



[ 
https://issues.apache.org/jira/browse/DISPATCH-1962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17289331#comment-17289331
 ] 

ASF GitHub Bot commented on DISPATCH-1962:
--

ganeshmurthy commented on a change in pull request #1048:
URL: https://github.com/apache/qpid-dispatch/pull/1048#discussion_r581385008



##
File path: tests/lsan.supp
##
@@ -10,6 +10,27 @@ leak:qd_policy_c_counts_alloc
 leak:qd_policy_open_fetch_settings
 leak:qdr_error_description
 
+# to be triaged; unit_tests
+leak:http-libwebsockets.c

Review comment:
   ok, let's fix this in a subsequent step. Thx





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Make LSan suppressions more targeted and specific (within reason)
> -
>
> Key: DISPATCH-1962
> URL: https://issues.apache.org/jira/browse/DISPATCH-1962
> Project: Qpid Dispatch
>  Issue Type: Test
>Affects Versions: 1.15.0
>Reporter: Jiri Daněk
>Priority: Major
> Attachments: dispatch_leaking_agent.png
>
>
> There are some unfortunate suppressions in {{tests/lsan.supp}} at this moment:
> {code}
> leak:*libwebsockets*
> {code}
> This is way too broad. It suppresses leaks in Dispatch files, since it 
> matches e.g. {{src/http-libwebsockets.c}}.
> The stars at the beginning and end are actually assumed implicitly. If you do 
> not want substring match, you have to do ^foo$. LSan suppression format is 
> simplistic, very unlike Valgrind's.
> {code}
> leak:run_unit_tests.c
> {code}
> Same thing, any leaks revealed by running unit_tests get suppressed. This 
> suppression suppresses all leak traces that include run_unit_tests.c anywhere 
> in the stack. What't the point of running such tests under leak detector, 
> then?
> {code}
> leak:run_unit_tests.c
> leak:^libqpid-proton.so$
> {code}
> Same thing. The patterns suppress all leaks that include Python (or Proton) 
> anywhere in the stacktrace. That means there are huge blind spots where 
> dispatch leaks can hide. This is a weakness of the lsan.supp syntax (Valgrind 
> suppressions can be much more targeted and discerning).
> h3. Python leaks
> Leaks are known and there is ongoing effort to fight them: 
> https://bugs.python.org/issue1635741 (https://bugs.python.org/issue25302) and 
> https://www.python.org/dev/peps/pep-3121
> Here's valgrind suppression file from somebody who actually investigated the 
> Python leaks and identified the harmless ones: 
> https://github.com/libgit2/pygit2/blob/master/misc/valgrind-python.supp
> One example of a hidden leak in dispatch, which is revealed by making Python 
> suppressions more targetted:
> {code}
> 9: Direct leak of 56 byte(s) in 1 object(s) allocated from:
> 9: #0 0x7f78a3606e8f in __interceptor_malloc 
> (/nix/store/g40sl3zh3nv52vj0mrl4iki5iphh5ika-gcc-10.2.0-lib/lib/libasan.so.6+0xace8f)
> 9: #1 0x7f78a2d64afb in qd_malloc ../include/qpid/dispatch/ctools.h:229
> 9: #2 0x7f78a2d657da in qdr_core_subscribe 
> ../src/router_core/route_tables.c:149
> 9: #3 0x7f78a2c83072 in IoAdapter_init ../src/python_embedded.c:711
> 9: #4 0x7f78a2353a6c in type_call 
> (/nix/store/r85nxfnwiv45nbmf5yb60jj8ajim4m7w-python3-3.8.5/lib/libpython3.8.so.1.0+0x165a6c)
> {code}
> The problem is in
> {code}
> class Agent:
> ...
> def activate(self, address):
> ...
> self.io = IoAdapter(self.receive, address, 'L', '0', 
> TREATMENT_ANYCAST_CLOSEST)
> {code}
> IoAdapter refers to Agent (through the bound method reference self.receive) 
> and Agent refers to IoAdapter (through property self.io). Since IoAdapter is 
> implemented in C and does not implement support for Python's cyclic GC, there 
> is no way to break the cycle.
> Heap dump in attachment. The bound method is at the top of the picture. 
> (Ignore the Mock objects, I was trying to simplify the picture while not 
> getting crashes due to too much meddling).
> h3. Random observations
> It is possible to build special Debug build of Python, which has tools to 
> detect leaks, asserts to prevent negative refcounts, etc. 
> https://pythonextensionpatterns.readthedocs.io/en/latest/debugging/debug_python.html#debug-version-of-python-memory-alloc-label
> Use the following to detect python leaks (instead of valgrind)
> https://docs.python.org/3/library/tracemalloc.html
> Use https://pypi.org/project/objgraph (with graphviz) to view heap object 
> trees. The following renders the picture as a png under /tmp and prints the 
> path to stdout.
> {code}
> int ret = PyRun_SimpleString("import objgraph; 
>

[jira] [Commented] (DISPATCH-1962) Make LSan suppressions more targeted and specific (within reason)

2021-02-23 Thread ASF GitHub Bot (Jira)



[ 
https://issues.apache.org/jira/browse/DISPATCH-1962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17289278#comment-17289278
 ] 

ASF GitHub Bot commented on DISPATCH-1962:
--

jiridanek commented on a change in pull request #1048:
URL: https://github.com/apache/qpid-dispatch/pull/1048#discussion_r581316679



##
File path: tests/lsan.supp
##
@@ -10,6 +10,27 @@ leak:qd_policy_c_counts_alloc
 leak:qd_policy_open_fetch_settings
 leak:qdr_error_description
 
+# to be triaged; unit_tests
+leak:http-libwebsockets.c

Review comment:
   (IIRC that leak ^^^ is essentially a test issue. There are subsequent 
suppressions `leak:^callback_healthz$` and `leak:^callback_metrics$` which 
looked to me more real, but it's hard to tell. And there is little point in 
investigating/fixing things when I'm no longer dealing with upstream master 
branch, but with my own patched version, so any PR I might produce would end up 
rebased few times down the road...)





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Make LSan suppressions more targeted and specific (within reason)
> -
>
> Key: DISPATCH-1962
> URL: https://issues.apache.org/jira/browse/DISPATCH-1962
> Project: Qpid Dispatch
>  Issue Type: Test
>Affects Versions: 1.15.0
>Reporter: Jiri Daněk
>Priority: Major
> Attachments: dispatch_leaking_agent.png
>
>
> There are some unfortunate suppressions in {{tests/lsan.supp}} at this moment:
> {code}
> leak:*libwebsockets*
> {code}
> This is way too broad. It suppresses leaks in Dispatch files, since it 
> matches e.g. {{src/http-libwebsockets.c}}.
> The stars at the beginning and end are actually assumed implicitly. If you do 
> not want substring match, you have to do ^foo$. LSan suppression format is 
> simplistic, very unlike Valgrind's.
> {code}
> leak:run_unit_tests.c
> {code}
> Same thing, any leaks revealed by running unit_tests get suppressed. This 
> suppression suppresses all leak traces that include run_unit_tests.c anywhere 
> in the stack. What't the point of running such tests under leak detector, 
> then?
> {code}
> leak:run_unit_tests.c
> leak:^libqpid-proton.so$
> {code}
> Same thing. The patterns suppress all leaks that include Python (or Proton) 
> anywhere in the stacktrace. That means there are huge blind spots where 
> dispatch leaks can hide. This is a weakness of the lsan.supp syntax (Valgrind 
> suppressions can be much more targeted and discerning).
> h3. Python leaks
> Leaks are known and there is ongoing effort to fight them: 
> https://bugs.python.org/issue1635741 (https://bugs.python.org/issue25302) and 
> https://www.python.org/dev/peps/pep-3121
> Here's valgrind suppression file from somebody who actually investigated the 
> Python leaks and identified the harmless ones: 
> https://github.com/libgit2/pygit2/blob/master/misc/valgrind-python.supp
> One example of a hidden leak in dispatch, which is revealed by making Python 
> suppressions more targetted:
> {code}
> 9: Direct leak of 56 byte(s) in 1 object(s) allocated from:
> 9: #0 0x7f78a3606e8f in __interceptor_malloc 
> (/nix/store/g40sl3zh3nv52vj0mrl4iki5iphh5ika-gcc-10.2.0-lib/lib/libasan.so.6+0xace8f)
> 9: #1 0x7f78a2d64afb in qd_malloc ../include/qpid/dispatch/ctools.h:229
> 9: #2 0x7f78a2d657da in qdr_core_subscribe 
> ../src/router_core/route_tables.c:149
> 9: #3 0x7f78a2c83072 in IoAdapter_init ../src/python_embedded.c:711
> 9: #4 0x7f78a2353a6c in type_call 
> (/nix/store/r85nxfnwiv45nbmf5yb60jj8ajim4m7w-python3-3.8.5/lib/libpython3.8.so.1.0+0x165a6c)
> {code}
> The problem is in
> {code}
> class Agent:
> ...
> def activate(self, address):
> ...
> self.io = IoAdapter(self.receive, address, 'L', '0', 
> TREATMENT_ANYCAST_CLOSEST)
> {code}
> IoAdapter refers to Agent (through the bound method reference self.receive) 
> and Agent refers to IoAdapter (through property self.io). Since IoAdapter is 
> implemented in C and does not implement support for Python's cyclic GC, there 
> is no way to break the cycle.
> Heap dump in attachment. The bound method is at the top of the picture. 
> (Ignore the Mock objects, I was trying to simplify the picture while not 
> getting crashes due to too much meddling).
> h3. Random observations
> It is possible to build special Debug build of Python, which has tools to 
> detect leaks, asserts to prevent negative refcounts, etc. 
>

[jira] [Commented] (DISPATCH-1962) Make LSan suppressions more targeted and specific (within reason)

2021-02-23 Thread ASF GitHub Bot (Jira)



[ 
https://issues.apache.org/jira/browse/DISPATCH-1962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17289275#comment-17289275
 ] 

ASF GitHub Bot commented on DISPATCH-1962:
--

jiridanek commented on a change in pull request #1048:
URL: https://github.com/apache/qpid-dispatch/pull/1048#discussion_r581312963



##
File path: tests/lsan.supp
##
@@ -10,6 +10,27 @@ leak:qd_policy_c_counts_alloc
 leak:qd_policy_open_fetch_settings
 leak:qdr_error_description
 
+# to be triaged; unit_tests
+leak:http-libwebsockets.c

Review comment:
   I'd be happy to fix this in a subsequent step, or at least make an 
attempt at it, but for now, currently there _are_ leaks coming from 
`http-libwebsockets.c` so the suppression needs to stay here:
   
   
https://github.com/jiridanek/qpid-dispatch/runs/1964063810?check_suite_focus=true#step:9:593
   
   ```
   9: =
   9: ==2292==ERROR: LeakSanitizer: detected memory leaks
   9: 
   9: Indirect leak of 216 byte(s) in 1 object(s) allocated from:
   9: #0 0x7fa3b6a8bdc6 in calloc 
(/lib/x86_64-linux-gnu/libasan.so.5+0x10ddc6)
   9: #1 0x7fa3b63ce928 in qd_http_server ../src/http-libwebsockets.c:963
   9: #2 0x7fa3b63b5ec7 in qd_server ../src/server.c:1364
   9: #3 0x7fa3b6207fff in qd_dispatch_prepare ../src/dispatch.c:330
   9: #4 0x7fa3b1208ff4  (/lib/x86_64-linux-gnu/libffi.so.7+0x6ff4)
   9: #5 0x7ffce6d1b37f  ([stack]+0x1f37f)
   9: 
   9: Indirect leak of 64 byte(s) in 1 object(s) allocated from:
   9: #0 0x7fa3b6a8caa5 in posix_memalign 
(/lib/x86_64-linux-gnu/libasan.so.5+0x10eaa5)
   9: #1 0x7fa3b6270a94 in sys_cond ../src/posix/threading.c:77
   9: #2 0x7fa3b63ce96d in work_queue_init ../src/http-libwebsockets.c:231
   9: #3 0x7fa3b63ce96d in qd_http_server ../src/http-libwebsockets.c:965
   9: #4 0x7fa3b63b5ec7 in qd_server ../src/server.c:1364
   9: #5 0x7fa3b6207fff in qd_dispatch_prepare ../src/dispatch.c:330
   9: #6 0x7fa3b1208ff4  (/lib/x86_64-linux-gnu/libffi.so.7+0x6ff4)
   9: #7 0x7ffce6d1b37f  ([stack]+0x1f37f)
   9: 
   9: -
   9: Suppressions used:
   9:   count  bytes template
   9:   1   1160 run_unit_tests.c
   9:   1 64 sys_mutex
   9:   1 56 qdr_core_subscribe
   9:   9   5080 ^_PyObject_Realloc
   9: 540 862547 ^PyObject_Malloc$
   9:   1 32 ^PyThread_allocate_lock$
   9:   4   9897 ^PyMem_Malloc$
   9:   1856 ^_PyObject_GC_Resize$
   9:   1295 ^_PyBytes_Resize$
   9:   51058064 /libwebsockets.so
   9: -
   9: 
   9: SUMMARY: AddressSanitizer: 280 byte(s) leaked in 2 allocation(s).
7/37 Test  #9: unit_tests .***Failed
2.03 sec
   ```





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Make LSan suppressions more targeted and specific (within reason)
> -
>
> Key: DISPATCH-1962
> URL: https://issues.apache.org/jira/browse/DISPATCH-1962
> Project: Qpid Dispatch
>  Issue Type: Test
>Affects Versions: 1.15.0
>Reporter: Jiri Daněk
>Priority: Major
> Attachments: dispatch_leaking_agent.png
>
>
> There are some unfortunate suppressions in {{tests/lsan.supp}} at this moment:
> {code}
> leak:*libwebsockets*
> {code}
> This is way too broad. It suppresses leaks in Dispatch files, since it 
> matches e.g. {{src/http-libwebsockets.c}}.
> The stars at the beginning and end are actually assumed implicitly. If you do 
> not want substring match, you have to do ^foo$. LSan suppression format is 
> simplistic, very unlike Valgrind's.
> {code}
> leak:run_unit_tests.c
> {code}
> Same thing, any leaks revealed by running unit_tests get suppressed. This 
> suppression suppresses all leak traces that include run_unit_tests.c anywhere 
> in the stack. What't the point of running such tests under leak detector, 
> then?
> {code}
> leak:run_unit_tests.c
> leak:^libqpid-proton.so$
> {code}
> Same thing. The patterns suppress all leaks that include Python (or Proton) 
> anywhere in the stacktrace. That means there are huge blind spots where 
> dispatch leaks can hide. This is a weakness of the lsan.supp syntax (Valgrind 
> suppressions can be much more targeted and discerning).
> h3. Python leaks
> Leaks are known and there is ongoing effort to fight them: 
> https://bugs.python.org/issue1635741

[jira] [Commented] (DISPATCH-1962) Make LSan suppressions more targeted and specific (within reason)

2021-02-23 Thread ASF GitHub Bot (Jira)



[ 
https://issues.apache.org/jira/browse/DISPATCH-1962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17289273#comment-17289273
 ] 

ASF GitHub Bot commented on DISPATCH-1962:
--

jiridanek commented on a change in pull request #1048:
URL: https://github.com/apache/qpid-dispatch/pull/1048#discussion_r581308755



##
File path: tests/lsan.supp
##
@@ -23,15 +44,56 @@ leak:_ctypes_alloc_format_string
 leak:__strdup
 
 
- Miscellaneous 3rd party libraries, test code, etc:
+ Miscellaneous 3rd party libraries:
 
 
-leak:*libpython*
-leak:*libwebsockets*
-leak:*python2*
+### Python
 
-# We should be able to uncomment these once all known dispatch leaks have been 
fixed
-leak:*libqpid-proton*
+# these Python leaks happen even after simple Py_Initialize(); Py_Finalize();
+#  https://bugs.python.org/issue1635741
+leak:^_PyObject_Realloc
+leak:^PyObject_Malloc$
+leak:^PyThread_allocate_lock$
 
-# Ignore test code
-leak:run_unit_tests.c
+# the PyMalloc mechanism is incompatible with Valgrind, it must be disabled or 
reported "leaks" must be suppressed

Review comment:
   I do not append the `leak:/libpython3..so` always, only when the 
SANITIZE_3RD_PARTY option is OFF. In GitHub Actions, the Python debug info is 
installed and works ok, so there SANITIZE_3RD_PARTY=ON and the more granular 
suppressions are used. (And therefore e.g. the leak in IoAdapter is picked up). 
Same thing on my laptop.
   
   On Travis, and on other people's computers, the debug symbols might not be 
present. In that case, the whole-library suppressions are needed, otherwise 
they would see test failures.





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Make LSan suppressions more targeted and specific (within reason)
> -
>
> Key: DISPATCH-1962
> URL: https://issues.apache.org/jira/browse/DISPATCH-1962
> Project: Qpid Dispatch
>  Issue Type: Test
>Affects Versions: 1.15.0
>Reporter: Jiri Daněk
>Priority: Major
> Attachments: dispatch_leaking_agent.png
>
>
> There are some unfortunate suppressions in {{tests/lsan.supp}} at this moment:
> {code}
> leak:*libwebsockets*
> {code}
> This is way too broad. It suppresses leaks in Dispatch files, since it 
> matches e.g. {{src/http-libwebsockets.c}}.
> The stars at the beginning and end are actually assumed implicitly. If you do 
> not want substring match, you have to do ^foo$. LSan suppression format is 
> simplistic, very unlike Valgrind's.
> {code}
> leak:run_unit_tests.c
> {code}
> Same thing, any leaks revealed by running unit_tests get suppressed. This 
> suppression suppresses all leak traces that include run_unit_tests.c anywhere 
> in the stack. What't the point of running such tests under leak detector, 
> then?
> {code}
> leak:run_unit_tests.c
> leak:^libqpid-proton.so$
> {code}
> Same thing. The patterns suppress all leaks that include Python (or Proton) 
> anywhere in the stacktrace. That means there are huge blind spots where 
> dispatch leaks can hide. This is a weakness of the lsan.supp syntax (Valgrind 
> suppressions can be much more targeted and discerning).
> h3. Python leaks
> Leaks are known and there is ongoing effort to fight them: 
> https://bugs.python.org/issue1635741 (https://bugs.python.org/issue25302) and 
> https://www.python.org/dev/peps/pep-3121
> Here's valgrind suppression file from somebody who actually investigated the 
> Python leaks and identified the harmless ones: 
> https://github.com/libgit2/pygit2/blob/master/misc/valgrind-python.supp
> One example of a hidden leak in dispatch, which is revealed by making Python 
> suppressions more targetted:
> {code}
> 9: Direct leak of 56 byte(s) in 1 object(s) allocated from:
> 9: #0 0x7f78a3606e8f in __interceptor_malloc 
> (/nix/store/g40sl3zh3nv52vj0mrl4iki5iphh5ika-gcc-10.2.0-lib/lib/libasan.so.6+0xace8f)
> 9: #1 0x7f78a2d64afb in qd_malloc ../include/qpid/dispatch/ctools.h:229
> 9: #2 0x7f78a2d657da in qdr_core_subscribe 
> ../src/router_core/route_tables.c:149
> 9: #3 0x7f78a2c83072 in IoAdapter_init ../src/python_embedded.c:711
> 9: #4 0x7f78a2353a6c in type_call 
> (/nix/store/r85nxfnwiv45nbmf5yb60jj8ajim4m7w-python3-3.8.5/lib/libpython3.8.so.1.0+0x165a6c)
> {code}
> The problem is in
> {code}
> class Agent:
> ...
> def activate(self, address):
> ...
> self.io = IoAdapter(self.receive, address, 'L', '0', 
> TREATMENT_ANYCAST_CLOSEST)
> {code}
> IoAdapter refers to Agent (through the bound method reference self.receive) 
> and Agent refers to

[jira] [Commented] (DISPATCH-1962) Make LSan suppressions more targeted and specific (within reason)

2021-02-23 Thread ASF GitHub Bot (Jira)



[ 
https://issues.apache.org/jira/browse/DISPATCH-1962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17289271#comment-17289271
 ] 

ASF GitHub Bot commented on DISPATCH-1962:
--

jiridanek commented on a change in pull request #1048:
URL: https://github.com/apache/qpid-dispatch/pull/1048#discussion_r581308755



##
File path: tests/lsan.supp
##
@@ -23,15 +44,56 @@ leak:_ctypes_alloc_format_string
 leak:__strdup
 
 
- Miscellaneous 3rd party libraries, test code, etc:
+ Miscellaneous 3rd party libraries:
 
 
-leak:*libpython*
-leak:*libwebsockets*
-leak:*python2*
+### Python
 
-# We should be able to uncomment these once all known dispatch leaks have been 
fixed
-leak:*libqpid-proton*
+# these Python leaks happen even after simple Py_Initialize(); Py_Finalize();
+#  https://bugs.python.org/issue1635741
+leak:^_PyObject_Realloc
+leak:^PyObject_Malloc$
+leak:^PyThread_allocate_lock$
 
-# Ignore test code
-leak:run_unit_tests.c
+# the PyMalloc mechanism is incompatible with Valgrind, it must be disabled or 
reported "leaks" must be suppressed

Review comment:
   We do not append the `leak:/libpython3..so` always, only when the 
SANITIZE_3RD_PARTY option is OFF. In GitHub Actions, the Python debug info is 
installed and works ok, so there SANITIZE_3RD_PARTY=ON and the more granular 
suppressions are used. (And therefore e.g. the leak in IoAdapter is picked up). 
Same thing on my laptop.
   
   On Travis, and on other people's computers, the debug symbols might not be 
present. In that case, the whole-library suppressions are needed, otherwise 
they would see test failures.





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Make LSan suppressions more targeted and specific (within reason)
> -
>
> Key: DISPATCH-1962
> URL: https://issues.apache.org/jira/browse/DISPATCH-1962
> Project: Qpid Dispatch
>  Issue Type: Test
>Affects Versions: 1.15.0
>Reporter: Jiri Daněk
>Priority: Major
> Attachments: dispatch_leaking_agent.png
>
>
> There are some unfortunate suppressions in {{tests/lsan.supp}} at this moment:
> {code}
> leak:*libwebsockets*
> {code}
> This is way too broad. It suppresses leaks in Dispatch files, since it 
> matches e.g. {{src/http-libwebsockets.c}}.
> The stars at the beginning and end are actually assumed implicitly. If you do 
> not want substring match, you have to do ^foo$. LSan suppression format is 
> simplistic, very unlike Valgrind's.
> {code}
> leak:run_unit_tests.c
> {code}
> Same thing, any leaks revealed by running unit_tests get suppressed. This 
> suppression suppresses all leak traces that include run_unit_tests.c anywhere 
> in the stack. What't the point of running such tests under leak detector, 
> then?
> {code}
> leak:run_unit_tests.c
> leak:^libqpid-proton.so$
> {code}
> Same thing. The patterns suppress all leaks that include Python (or Proton) 
> anywhere in the stacktrace. That means there are huge blind spots where 
> dispatch leaks can hide. This is a weakness of the lsan.supp syntax (Valgrind 
> suppressions can be much more targeted and discerning).
> h3. Python leaks
> Leaks are known and there is ongoing effort to fight them: 
> https://bugs.python.org/issue1635741 (https://bugs.python.org/issue25302) and 
> https://www.python.org/dev/peps/pep-3121
> Here's valgrind suppression file from somebody who actually investigated the 
> Python leaks and identified the harmless ones: 
> https://github.com/libgit2/pygit2/blob/master/misc/valgrind-python.supp
> One example of a hidden leak in dispatch, which is revealed by making Python 
> suppressions more targetted:
> {code}
> 9: Direct leak of 56 byte(s) in 1 object(s) allocated from:
> 9: #0 0x7f78a3606e8f in __interceptor_malloc 
> (/nix/store/g40sl3zh3nv52vj0mrl4iki5iphh5ika-gcc-10.2.0-lib/lib/libasan.so.6+0xace8f)
> 9: #1 0x7f78a2d64afb in qd_malloc ../include/qpid/dispatch/ctools.h:229
> 9: #2 0x7f78a2d657da in qdr_core_subscribe 
> ../src/router_core/route_tables.c:149
> 9: #3 0x7f78a2c83072 in IoAdapter_init ../src/python_embedded.c:711
> 9: #4 0x7f78a2353a6c in type_call 
> (/nix/store/r85nxfnwiv45nbmf5yb60jj8ajim4m7w-python3-3.8.5/lib/libpython3.8.so.1.0+0x165a6c)
> {code}
> The problem is in
> {code}
> class Agent:
> ...
> def activate(self, address):
> ...
> self.io = IoAdapter(self.receive, address, 'L', '0', 
> TREATMENT_ANYCAST_CLOSEST)
> {code}
> IoAdapter refers to Agent (through the bound method reference self.receive) 
> and Agent refers to

[jira] [Commented] (DISPATCH-1962) Make LSan suppressions more targeted and specific (within reason)

2021-02-23 Thread ASF GitHub Bot (Jira)



[ 
https://issues.apache.org/jira/browse/DISPATCH-1962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17289265#comment-17289265
 ] 

ASF GitHub Bot commented on DISPATCH-1962:
--

ganeshmurthy commented on a change in pull request #1048:
URL: https://github.com/apache/qpid-dispatch/pull/1048#discussion_r581301934



##
File path: cmake/RuntimeChecks.cmake
##
@@ -106,9 +106,17 @@ elseif(RUNTIME_CHECK STREQUAL "asan")
 message(FATAL_ERROR "libubsan not installed - address sanitizer not 
available")
   endif(UBSAN_LIBRARY-NOTFOUND)
   message(STATUS "Runtime memory checker: gcc/clang address sanitizers")
+  option(SANITIZE_3RD_PARTY "Detect leaks in 3rd party libraries used by 
Dispatch while running tests" OFF)
+  file (COPY "${CMAKE_SOURCE_DIR}/tests/lsan.supp" DESTINATION 
"${CMAKE_BINARY_DIR}/tests")

Review comment:
   Good point about not having generated files outside of CMAKE_BINARY_DIR. 
I remember following that rule when working in Java projects using Maven. We 
don't seem to follow that rule here which I have not questioned so far. 
   And, yes, I was talking about using the configure_file in cmake.
   But you should wait on this to see what @kgiusti 's take on this is. Thx





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Make LSan suppressions more targeted and specific (within reason)
> -
>
> Key: DISPATCH-1962
> URL: https://issues.apache.org/jira/browse/DISPATCH-1962
> Project: Qpid Dispatch
>  Issue Type: Test
>Affects Versions: 1.15.0
>Reporter: Jiri Daněk
>Priority: Major
> Attachments: dispatch_leaking_agent.png
>
>
> There are some unfortunate suppressions in {{tests/lsan.supp}} at this moment:
> {code}
> leak:*libwebsockets*
> {code}
> This is way too broad. It suppresses leaks in Dispatch files, since it 
> matches e.g. {{src/http-libwebsockets.c}}.
> The stars at the beginning and end are actually assumed implicitly. If you do 
> not want substring match, you have to do ^foo$. LSan suppression format is 
> simplistic, very unlike Valgrind's.
> {code}
> leak:run_unit_tests.c
> {code}
> Same thing, any leaks revealed by running unit_tests get suppressed. This 
> suppression suppresses all leak traces that include run_unit_tests.c anywhere 
> in the stack. What't the point of running such tests under leak detector, 
> then?
> {code}
> leak:run_unit_tests.c
> leak:^libqpid-proton.so$
> {code}
> Same thing. The patterns suppress all leaks that include Python (or Proton) 
> anywhere in the stacktrace. That means there are huge blind spots where 
> dispatch leaks can hide. This is a weakness of the lsan.supp syntax (Valgrind 
> suppressions can be much more targeted and discerning).
> h3. Python leaks
> Leaks are known and there is ongoing effort to fight them: 
> https://bugs.python.org/issue1635741 (https://bugs.python.org/issue25302) and 
> https://www.python.org/dev/peps/pep-3121
> Here's valgrind suppression file from somebody who actually investigated the 
> Python leaks and identified the harmless ones: 
> https://github.com/libgit2/pygit2/blob/master/misc/valgrind-python.supp
> One example of a hidden leak in dispatch, which is revealed by making Python 
> suppressions more targetted:
> {code}
> 9: Direct leak of 56 byte(s) in 1 object(s) allocated from:
> 9: #0 0x7f78a3606e8f in __interceptor_malloc 
> (/nix/store/g40sl3zh3nv52vj0mrl4iki5iphh5ika-gcc-10.2.0-lib/lib/libasan.so.6+0xace8f)
> 9: #1 0x7f78a2d64afb in qd_malloc ../include/qpid/dispatch/ctools.h:229
> 9: #2 0x7f78a2d657da in qdr_core_subscribe 
> ../src/router_core/route_tables.c:149
> 9: #3 0x7f78a2c83072 in IoAdapter_init ../src/python_embedded.c:711
> 9: #4 0x7f78a2353a6c in type_call 
> (/nix/store/r85nxfnwiv45nbmf5yb60jj8ajim4m7w-python3-3.8.5/lib/libpython3.8.so.1.0+0x165a6c)
> {code}
> The problem is in
> {code}
> class Agent:
> ...
> def activate(self, address):
> ...
> self.io = IoAdapter(self.receive, address, 'L', '0', 
> TREATMENT_ANYCAST_CLOSEST)
> {code}
> IoAdapter refers to Agent (through the bound method reference self.receive) 
> and Agent refers to IoAdapter (through property self.io). Since IoAdapter is 
> implemented in C and does not implement support for Python's cyclic GC, there 
> is no way to break the cycle.
> Heap dump in attachment. The bound method is at the top of the picture. 
> (Ignore the Mock objects, I was trying to simplify the picture while not 
> getting crashes due to too much meddling).
> h3. Random observations
> It is possible

[jira] [Commented] (DISPATCH-1962) Make LSan suppressions more targeted and specific (within reason)

2021-02-23 Thread ASF GitHub Bot (Jira)



[ 
https://issues.apache.org/jira/browse/DISPATCH-1962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17289268#comment-17289268
 ] 

ASF GitHub Bot commented on DISPATCH-1962:
--

ganeshmurthy commented on a change in pull request #1048:
URL: https://github.com/apache/qpid-dispatch/pull/1048#discussion_r581304925



##
File path: tests/lsan.supp
##
@@ -23,15 +44,56 @@ leak:_ctypes_alloc_format_string
 leak:__strdup
 
 
- Miscellaneous 3rd party libraries, test code, etc:
+ Miscellaneous 3rd party libraries:
 
 
-leak:*libpython*
-leak:*libwebsockets*
-leak:*python2*
+### Python
 
-# We should be able to uncomment these once all known dispatch leaks have been 
fixed
-leak:*libqpid-proton*
+# these Python leaks happen even after simple Py_Initialize(); Py_Finalize();
+#  https://bugs.python.org/issue1635741
+leak:^_PyObject_Realloc
+leak:^PyObject_Malloc$
+leak:^PyThread_allocate_lock$
 
-# Ignore test code
-leak:run_unit_tests.c
+# the PyMalloc mechanism is incompatible with Valgrind, it must be disabled or 
reported "leaks" must be suppressed

Review comment:
   So, if I understand correctly, no matter what you have in this list 
below, having the leak:/libpython2..so\nleak:/libpython3..so\n at the end of 
the file totally ignores this list. So, does it make sense to not even have 
this list if we are going to have the above two lines at the end of the file ?





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Make LSan suppressions more targeted and specific (within reason)
> -
>
> Key: DISPATCH-1962
> URL: https://issues.apache.org/jira/browse/DISPATCH-1962
> Project: Qpid Dispatch
>  Issue Type: Test
>Affects Versions: 1.15.0
>Reporter: Jiri Daněk
>Priority: Major
> Attachments: dispatch_leaking_agent.png
>
>
> There are some unfortunate suppressions in {{tests/lsan.supp}} at this moment:
> {code}
> leak:*libwebsockets*
> {code}
> This is way too broad. It suppresses leaks in Dispatch files, since it 
> matches e.g. {{src/http-libwebsockets.c}}.
> The stars at the beginning and end are actually assumed implicitly. If you do 
> not want substring match, you have to do ^foo$. LSan suppression format is 
> simplistic, very unlike Valgrind's.
> {code}
> leak:run_unit_tests.c
> {code}
> Same thing, any leaks revealed by running unit_tests get suppressed. This 
> suppression suppresses all leak traces that include run_unit_tests.c anywhere 
> in the stack. What't the point of running such tests under leak detector, 
> then?
> {code}
> leak:run_unit_tests.c
> leak:^libqpid-proton.so$
> {code}
> Same thing. The patterns suppress all leaks that include Python (or Proton) 
> anywhere in the stacktrace. That means there are huge blind spots where 
> dispatch leaks can hide. This is a weakness of the lsan.supp syntax (Valgrind 
> suppressions can be much more targeted and discerning).
> h3. Python leaks
> Leaks are known and there is ongoing effort to fight them: 
> https://bugs.python.org/issue1635741 (https://bugs.python.org/issue25302) and 
> https://www.python.org/dev/peps/pep-3121
> Here's valgrind suppression file from somebody who actually investigated the 
> Python leaks and identified the harmless ones: 
> https://github.com/libgit2/pygit2/blob/master/misc/valgrind-python.supp
> One example of a hidden leak in dispatch, which is revealed by making Python 
> suppressions more targetted:
> {code}
> 9: Direct leak of 56 byte(s) in 1 object(s) allocated from:
> 9: #0 0x7f78a3606e8f in __interceptor_malloc 
> (/nix/store/g40sl3zh3nv52vj0mrl4iki5iphh5ika-gcc-10.2.0-lib/lib/libasan.so.6+0xace8f)
> 9: #1 0x7f78a2d64afb in qd_malloc ../include/qpid/dispatch/ctools.h:229
> 9: #2 0x7f78a2d657da in qdr_core_subscribe 
> ../src/router_core/route_tables.c:149
> 9: #3 0x7f78a2c83072 in IoAdapter_init ../src/python_embedded.c:711
> 9: #4 0x7f78a2353a6c in type_call 
> (/nix/store/r85nxfnwiv45nbmf5yb60jj8ajim4m7w-python3-3.8.5/lib/libpython3.8.so.1.0+0x165a6c)
> {code}
> The problem is in
> {code}
> class Agent:
> ...
> def activate(self, address):
> ...
> self.io = IoAdapter(self.receive, address, 'L', '0', 
> TREATMENT_ANYCAST_CLOSEST)
> {code}
> IoAdapter refers to Agent (through the bound method reference self.receive) 
> and Agent refers to IoAdapter (through property self.io). Since IoAdapter is 
> implemented in C and does not implement support for Python's cyclic GC, there 
> is no way to break the cycle.
> Heap dump in attachment. The bound method is

[jira] [Commented] (DISPATCH-1962) Make LSan suppressions more targeted and specific (within reason)

2021-02-23 Thread ASF GitHub Bot (Jira)



[ 
https://issues.apache.org/jira/browse/DISPATCH-1962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17289260#comment-17289260
 ] 

ASF GitHub Bot commented on DISPATCH-1962:
--

ganeshmurthy commented on a change in pull request #1048:
URL: https://github.com/apache/qpid-dispatch/pull/1048#discussion_r581298744



##
File path: tests/lsan.supp
##
@@ -10,6 +10,27 @@ leak:qd_policy_c_counts_alloc
 leak:qd_policy_open_fetch_settings
 leak:qdr_error_description
 
+# to be triaged; unit_tests
+leak:http-libwebsockets.c

Review comment:
   Ok, what you are saying makes sense to me. What I would like to happen 
is for any leaks in our code in the http-libwebsockets.c to be not ignored. 
Like so, there is a bunch of code we wrote relating to stats and metrics and if 
there are any leaks in our code, I would like those to be not suppressed. I do 
understand that we previously had leak:*libwebsockets* 





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Make LSan suppressions more targeted and specific (within reason)
> -
>
> Key: DISPATCH-1962
> URL: https://issues.apache.org/jira/browse/DISPATCH-1962
> Project: Qpid Dispatch
>  Issue Type: Test
>Affects Versions: 1.15.0
>Reporter: Jiri Daněk
>Priority: Major
> Attachments: dispatch_leaking_agent.png
>
>
> There are some unfortunate suppressions in {{tests/lsan.supp}} at this moment:
> {code}
> leak:*libwebsockets*
> {code}
> This is way too broad. It suppresses leaks in Dispatch files, since it 
> matches e.g. {{src/http-libwebsockets.c}}.
> The stars at the beginning and end are actually assumed implicitly. If you do 
> not want substring match, you have to do ^foo$. LSan suppression format is 
> simplistic, very unlike Valgrind's.
> {code}
> leak:run_unit_tests.c
> {code}
> Same thing, any leaks revealed by running unit_tests get suppressed. This 
> suppression suppresses all leak traces that include run_unit_tests.c anywhere 
> in the stack. What't the point of running such tests under leak detector, 
> then?
> {code}
> leak:run_unit_tests.c
> leak:^libqpid-proton.so$
> {code}
> Same thing. The patterns suppress all leaks that include Python (or Proton) 
> anywhere in the stacktrace. That means there are huge blind spots where 
> dispatch leaks can hide. This is a weakness of the lsan.supp syntax (Valgrind 
> suppressions can be much more targeted and discerning).
> h3. Python leaks
> Leaks are known and there is ongoing effort to fight them: 
> https://bugs.python.org/issue1635741 (https://bugs.python.org/issue25302) and 
> https://www.python.org/dev/peps/pep-3121
> Here's valgrind suppression file from somebody who actually investigated the 
> Python leaks and identified the harmless ones: 
> https://github.com/libgit2/pygit2/blob/master/misc/valgrind-python.supp
> One example of a hidden leak in dispatch, which is revealed by making Python 
> suppressions more targetted:
> {code}
> 9: Direct leak of 56 byte(s) in 1 object(s) allocated from:
> 9: #0 0x7f78a3606e8f in __interceptor_malloc 
> (/nix/store/g40sl3zh3nv52vj0mrl4iki5iphh5ika-gcc-10.2.0-lib/lib/libasan.so.6+0xace8f)
> 9: #1 0x7f78a2d64afb in qd_malloc ../include/qpid/dispatch/ctools.h:229
> 9: #2 0x7f78a2d657da in qdr_core_subscribe 
> ../src/router_core/route_tables.c:149
> 9: #3 0x7f78a2c83072 in IoAdapter_init ../src/python_embedded.c:711
> 9: #4 0x7f78a2353a6c in type_call 
> (/nix/store/r85nxfnwiv45nbmf5yb60jj8ajim4m7w-python3-3.8.5/lib/libpython3.8.so.1.0+0x165a6c)
> {code}
> The problem is in
> {code}
> class Agent:
> ...
> def activate(self, address):
> ...
> self.io = IoAdapter(self.receive, address, 'L', '0', 
> TREATMENT_ANYCAST_CLOSEST)
> {code}
> IoAdapter refers to Agent (through the bound method reference self.receive) 
> and Agent refers to IoAdapter (through property self.io). Since IoAdapter is 
> implemented in C and does not implement support for Python's cyclic GC, there 
> is no way to break the cycle.
> Heap dump in attachment. The bound method is at the top of the picture. 
> (Ignore the Mock objects, I was trying to simplify the picture while not 
> getting crashes due to too much meddling).
> h3. Random observations
> It is possible to build special Debug build of Python, which has tools to 
> detect leaks, asserts to prevent negative refcounts, etc. 
> https://pythonextensionpatterns.readthedocs.io/en/latest/debugging/debug_python.html#debug-version-of-python-memory-alloc-label
> Use the following to detect python leaks

[jira] [Commented] (DISPATCH-1962) Make LSan suppressions more targeted and specific (within reason)

2021-02-23 Thread ASF GitHub Bot (Jira)



[ 
https://issues.apache.org/jira/browse/DISPATCH-1962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17289252#comment-17289252
 ] 

ASF GitHub Bot commented on DISPATCH-1962:
--

jiridanek commented on a change in pull request #1048:
URL: https://github.com/apache/qpid-dispatch/pull/1048#discussion_r581287358



##
File path: tests/lsan.supp
##
@@ -10,6 +10,27 @@ leak:qd_policy_c_counts_alloc
 leak:qd_policy_open_fetch_settings
 leak:qdr_error_description
 
+# to be triaged; unit_tests
+leak:http-libwebsockets.c

Review comment:
   `leak:http-libwebsockets.c` is a narrower version of 
`leak:*libwebsockets*` which was there previously. (The suppression patterns 
are matched as substrings, as I mentioned on the Jira. The stars are actually 
implied, even if they weren't present.) I made the suppression more targeted 
here.





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Make LSan suppressions more targeted and specific (within reason)
> -
>
> Key: DISPATCH-1962
> URL: https://issues.apache.org/jira/browse/DISPATCH-1962
> Project: Qpid Dispatch
>  Issue Type: Test
>Affects Versions: 1.15.0
>Reporter: Jiri Daněk
>Priority: Major
> Attachments: dispatch_leaking_agent.png
>
>
> There are some unfortunate suppressions in {{tests/lsan.supp}} at this moment:
> {code}
> leak:*libwebsockets*
> {code}
> This is way too broad. It suppresses leaks in Dispatch files, since it 
> matches e.g. {{src/http-libwebsockets.c}}.
> The stars at the beginning and end are actually assumed implicitly. If you do 
> not want substring match, you have to do ^foo$. LSan suppression format is 
> simplistic, very unlike Valgrind's.
> {code}
> leak:run_unit_tests.c
> {code}
> Same thing, any leaks revealed by running unit_tests get suppressed. This 
> suppression suppresses all leak traces that include run_unit_tests.c anywhere 
> in the stack. What't the point of running such tests under leak detector, 
> then?
> {code}
> leak:run_unit_tests.c
> leak:^libqpid-proton.so$
> {code}
> Same thing. The patterns suppress all leaks that include Python (or Proton) 
> anywhere in the stacktrace. That means there are huge blind spots where 
> dispatch leaks can hide. This is a weakness of the lsan.supp syntax (Valgrind 
> suppressions can be much more targeted and discerning).
> h3. Python leaks
> Leaks are known and there is ongoing effort to fight them: 
> https://bugs.python.org/issue1635741 (https://bugs.python.org/issue25302) and 
> https://www.python.org/dev/peps/pep-3121
> Here's valgrind suppression file from somebody who actually investigated the 
> Python leaks and identified the harmless ones: 
> https://github.com/libgit2/pygit2/blob/master/misc/valgrind-python.supp
> One example of a hidden leak in dispatch, which is revealed by making Python 
> suppressions more targetted:
> {code}
> 9: Direct leak of 56 byte(s) in 1 object(s) allocated from:
> 9: #0 0x7f78a3606e8f in __interceptor_malloc 
> (/nix/store/g40sl3zh3nv52vj0mrl4iki5iphh5ika-gcc-10.2.0-lib/lib/libasan.so.6+0xace8f)
> 9: #1 0x7f78a2d64afb in qd_malloc ../include/qpid/dispatch/ctools.h:229
> 9: #2 0x7f78a2d657da in qdr_core_subscribe 
> ../src/router_core/route_tables.c:149
> 9: #3 0x7f78a2c83072 in IoAdapter_init ../src/python_embedded.c:711
> 9: #4 0x7f78a2353a6c in type_call 
> (/nix/store/r85nxfnwiv45nbmf5yb60jj8ajim4m7w-python3-3.8.5/lib/libpython3.8.so.1.0+0x165a6c)
> {code}
> The problem is in
> {code}
> class Agent:
> ...
> def activate(self, address):
> ...
> self.io = IoAdapter(self.receive, address, 'L', '0', 
> TREATMENT_ANYCAST_CLOSEST)
> {code}
> IoAdapter refers to Agent (through the bound method reference self.receive) 
> and Agent refers to IoAdapter (through property self.io). Since IoAdapter is 
> implemented in C and does not implement support for Python's cyclic GC, there 
> is no way to break the cycle.
> Heap dump in attachment. The bound method is at the top of the picture. 
> (Ignore the Mock objects, I was trying to simplify the picture while not 
> getting crashes due to too much meddling).
> h3. Random observations
> It is possible to build special Debug build of Python, which has tools to 
> detect leaks, asserts to prevent negative refcounts, etc. 
> https://pythonextensionpatterns.readthedocs.io/en/latest/debugging/debug_python.html#debug-version-of-python-memory-alloc-label
> Use the following to detect python leaks (instead of valgrind)
> https://docs.python.org/3/library/tracemalloc.html
>

[jira] [Commented] (DISPATCH-1962) Make LSan suppressions more targeted and specific (within reason)

2021-02-23 Thread ASF GitHub Bot (Jira)



[ 
https://issues.apache.org/jira/browse/DISPATCH-1962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17289250#comment-17289250
 ] 

ASF GitHub Bot commented on DISPATCH-1962:
--

jiridanek commented on a change in pull request #1048:
URL: https://github.com/apache/qpid-dispatch/pull/1048#discussion_r581287358



##
File path: tests/lsan.supp
##
@@ -10,6 +10,27 @@ leak:qd_policy_c_counts_alloc
 leak:qd_policy_open_fetch_settings
 leak:qdr_error_description
 
+# to be triaged; unit_tests
+leak:http-libwebsockets.c

Review comment:
   `leak:http-libwebsockets.c` is a narrower version of 
`leak:*libwebsockets*` which was there previously. (The suppression patterns 
are matched a substrings, as I mentioned on the Jira. The stars are actually 
implied, even if they weren't present) I made the suppression more targeted 
here.





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Make LSan suppressions more targeted and specific (within reason)
> -
>
> Key: DISPATCH-1962
> URL: https://issues.apache.org/jira/browse/DISPATCH-1962
> Project: Qpid Dispatch
>  Issue Type: Test
>Affects Versions: 1.15.0
>Reporter: Jiri Daněk
>Priority: Major
> Attachments: dispatch_leaking_agent.png
>
>
> There are some unfortunate suppressions in {{tests/lsan.supp}} at this moment:
> {code}
> leak:*libwebsockets*
> {code}
> This is way too broad. It suppresses leaks in Dispatch files, since it 
> matches e.g. {{src/http-libwebsockets.c}}.
> The stars at the beginning and end are actually assumed implicitly. If you do 
> not want substring match, you have to do ^foo$. LSan suppression format is 
> simplistic, very unlike Valgrind's.
> {code}
> leak:run_unit_tests.c
> {code}
> Same thing, any leaks revealed by running unit_tests get suppressed. This 
> suppression suppresses all leak traces that include run_unit_tests.c anywhere 
> in the stack. What't the point of running such tests under leak detector, 
> then?
> {code}
> leak:run_unit_tests.c
> leak:^libqpid-proton.so$
> {code}
> Same thing. The patterns suppress all leaks that include Python (or Proton) 
> anywhere in the stacktrace. That means there are huge blind spots where 
> dispatch leaks can hide. This is a weakness of the lsan.supp syntax (Valgrind 
> suppressions can be much more targeted and discerning).
> h3. Python leaks
> Leaks are known and there is ongoing effort to fight them: 
> https://bugs.python.org/issue1635741 (https://bugs.python.org/issue25302) and 
> https://www.python.org/dev/peps/pep-3121
> Here's valgrind suppression file from somebody who actually investigated the 
> Python leaks and identified the harmless ones: 
> https://github.com/libgit2/pygit2/blob/master/misc/valgrind-python.supp
> One example of a hidden leak in dispatch, which is revealed by making Python 
> suppressions more targetted:
> {code}
> 9: Direct leak of 56 byte(s) in 1 object(s) allocated from:
> 9: #0 0x7f78a3606e8f in __interceptor_malloc 
> (/nix/store/g40sl3zh3nv52vj0mrl4iki5iphh5ika-gcc-10.2.0-lib/lib/libasan.so.6+0xace8f)
> 9: #1 0x7f78a2d64afb in qd_malloc ../include/qpid/dispatch/ctools.h:229
> 9: #2 0x7f78a2d657da in qdr_core_subscribe 
> ../src/router_core/route_tables.c:149
> 9: #3 0x7f78a2c83072 in IoAdapter_init ../src/python_embedded.c:711
> 9: #4 0x7f78a2353a6c in type_call 
> (/nix/store/r85nxfnwiv45nbmf5yb60jj8ajim4m7w-python3-3.8.5/lib/libpython3.8.so.1.0+0x165a6c)
> {code}
> The problem is in
> {code}
> class Agent:
> ...
> def activate(self, address):
> ...
> self.io = IoAdapter(self.receive, address, 'L', '0', 
> TREATMENT_ANYCAST_CLOSEST)
> {code}
> IoAdapter refers to Agent (through the bound method reference self.receive) 
> and Agent refers to IoAdapter (through property self.io). Since IoAdapter is 
> implemented in C and does not implement support for Python's cyclic GC, there 
> is no way to break the cycle.
> Heap dump in attachment. The bound method is at the top of the picture. 
> (Ignore the Mock objects, I was trying to simplify the picture while not 
> getting crashes due to too much meddling).
> h3. Random observations
> It is possible to build special Debug build of Python, which has tools to 
> detect leaks, asserts to prevent negative refcounts, etc. 
> https://pythonextensionpatterns.readthedocs.io/en/latest/debugging/debug_python.html#debug-version-of-python-memory-alloc-label
> Use the following to detect python leaks (instead of valgrind)
> https://docs.python.org/3/library/tracemalloc.html
>

[jira] [Commented] (DISPATCH-1962) Make LSan suppressions more targeted and specific (within reason)

2021-02-23 Thread ASF GitHub Bot (Jira)



[ 
https://issues.apache.org/jira/browse/DISPATCH-1962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17289251#comment-17289251
 ] 

ASF GitHub Bot commented on DISPATCH-1962:
--

jiridanek commented on a change in pull request #1048:
URL: https://github.com/apache/qpid-dispatch/pull/1048#discussion_r581287358



##
File path: tests/lsan.supp
##
@@ -10,6 +10,27 @@ leak:qd_policy_c_counts_alloc
 leak:qd_policy_open_fetch_settings
 leak:qdr_error_description
 
+# to be triaged; unit_tests
+leak:http-libwebsockets.c

Review comment:
   `leak:http-libwebsockets.c` is a narrower version of 
`leak:*libwebsockets*` which was there previously. (The suppression patterns 
are matched a substrings, as I mentioned on the Jira. The stars are actually 
implied, even if they weren't present.) I made the suppression more targeted 
here.





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Make LSan suppressions more targeted and specific (within reason)
> -
>
> Key: DISPATCH-1962
> URL: https://issues.apache.org/jira/browse/DISPATCH-1962
> Project: Qpid Dispatch
>  Issue Type: Test
>Affects Versions: 1.15.0
>Reporter: Jiri Daněk
>Priority: Major
> Attachments: dispatch_leaking_agent.png
>
>
> There are some unfortunate suppressions in {{tests/lsan.supp}} at this moment:
> {code}
> leak:*libwebsockets*
> {code}
> This is way too broad. It suppresses leaks in Dispatch files, since it 
> matches e.g. {{src/http-libwebsockets.c}}.
> The stars at the beginning and end are actually assumed implicitly. If you do 
> not want substring match, you have to do ^foo$. LSan suppression format is 
> simplistic, very unlike Valgrind's.
> {code}
> leak:run_unit_tests.c
> {code}
> Same thing, any leaks revealed by running unit_tests get suppressed. This 
> suppression suppresses all leak traces that include run_unit_tests.c anywhere 
> in the stack. What't the point of running such tests under leak detector, 
> then?
> {code}
> leak:run_unit_tests.c
> leak:^libqpid-proton.so$
> {code}
> Same thing. The patterns suppress all leaks that include Python (or Proton) 
> anywhere in the stacktrace. That means there are huge blind spots where 
> dispatch leaks can hide. This is a weakness of the lsan.supp syntax (Valgrind 
> suppressions can be much more targeted and discerning).
> h3. Python leaks
> Leaks are known and there is ongoing effort to fight them: 
> https://bugs.python.org/issue1635741 (https://bugs.python.org/issue25302) and 
> https://www.python.org/dev/peps/pep-3121
> Here's valgrind suppression file from somebody who actually investigated the 
> Python leaks and identified the harmless ones: 
> https://github.com/libgit2/pygit2/blob/master/misc/valgrind-python.supp
> One example of a hidden leak in dispatch, which is revealed by making Python 
> suppressions more targetted:
> {code}
> 9: Direct leak of 56 byte(s) in 1 object(s) allocated from:
> 9: #0 0x7f78a3606e8f in __interceptor_malloc 
> (/nix/store/g40sl3zh3nv52vj0mrl4iki5iphh5ika-gcc-10.2.0-lib/lib/libasan.so.6+0xace8f)
> 9: #1 0x7f78a2d64afb in qd_malloc ../include/qpid/dispatch/ctools.h:229
> 9: #2 0x7f78a2d657da in qdr_core_subscribe 
> ../src/router_core/route_tables.c:149
> 9: #3 0x7f78a2c83072 in IoAdapter_init ../src/python_embedded.c:711
> 9: #4 0x7f78a2353a6c in type_call 
> (/nix/store/r85nxfnwiv45nbmf5yb60jj8ajim4m7w-python3-3.8.5/lib/libpython3.8.so.1.0+0x165a6c)
> {code}
> The problem is in
> {code}
> class Agent:
> ...
> def activate(self, address):
> ...
> self.io = IoAdapter(self.receive, address, 'L', '0', 
> TREATMENT_ANYCAST_CLOSEST)
> {code}
> IoAdapter refers to Agent (through the bound method reference self.receive) 
> and Agent refers to IoAdapter (through property self.io). Since IoAdapter is 
> implemented in C and does not implement support for Python's cyclic GC, there 
> is no way to break the cycle.
> Heap dump in attachment. The bound method is at the top of the picture. 
> (Ignore the Mock objects, I was trying to simplify the picture while not 
> getting crashes due to too much meddling).
> h3. Random observations
> It is possible to build special Debug build of Python, which has tools to 
> detect leaks, asserts to prevent negative refcounts, etc. 
> https://pythonextensionpatterns.readthedocs.io/en/latest/debugging/debug_python.html#debug-version-of-python-memory-alloc-label
> Use the following to detect python leaks (instead of valgrind)
> https://docs.python.org/3/library/tracemalloc.html
>

[jira] [Commented] (DISPATCH-1962) Make LSan suppressions more targeted and specific (within reason)

2021-02-23 Thread ASF GitHub Bot (Jira)



[ 
https://issues.apache.org/jira/browse/DISPATCH-1962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17289248#comment-17289248
 ] 

ASF GitHub Bot commented on DISPATCH-1962:
--

jiridanek commented on a change in pull request #1048:
URL: https://github.com/apache/qpid-dispatch/pull/1048#discussion_r581287358



##
File path: tests/lsan.supp
##
@@ -10,6 +10,27 @@ leak:qd_policy_c_counts_alloc
 leak:qd_policy_open_fetch_settings
 leak:qdr_error_description
 
+# to be triaged; unit_tests
+leak:http-libwebsockets.c

Review comment:
   `leak:http-libwebsockets.c` is a narrower version of 
`leak:*libwebsockets*` which was there previously. (The suppression patterns 
are matched a substrings, as I mentioned on the Jira) I made the suppression 
more targeted here.





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Make LSan suppressions more targeted and specific (within reason)
> -
>
> Key: DISPATCH-1962
> URL: https://issues.apache.org/jira/browse/DISPATCH-1962
> Project: Qpid Dispatch
>  Issue Type: Test
>Affects Versions: 1.15.0
>Reporter: Jiri Daněk
>Priority: Major
> Attachments: dispatch_leaking_agent.png
>
>
> There are some unfortunate suppressions in {{tests/lsan.supp}} at this moment:
> {code}
> leak:*libwebsockets*
> {code}
> This is way too broad. It suppresses leaks in Dispatch files, since it 
> matches e.g. {{src/http-libwebsockets.c}}.
> The stars at the beginning and end are actually assumed implicitly. If you do 
> not want substring match, you have to do ^foo$. LSan suppression format is 
> simplistic, very unlike Valgrind's.
> {code}
> leak:run_unit_tests.c
> {code}
> Same thing, any leaks revealed by running unit_tests get suppressed. This 
> suppression suppresses all leak traces that include run_unit_tests.c anywhere 
> in the stack. What't the point of running such tests under leak detector, 
> then?
> {code}
> leak:run_unit_tests.c
> leak:^libqpid-proton.so$
> {code}
> Same thing. The patterns suppress all leaks that include Python (or Proton) 
> anywhere in the stacktrace. That means there are huge blind spots where 
> dispatch leaks can hide. This is a weakness of the lsan.supp syntax (Valgrind 
> suppressions can be much more targeted and discerning).
> h3. Python leaks
> Leaks are known and there is ongoing effort to fight them: 
> https://bugs.python.org/issue1635741 (https://bugs.python.org/issue25302) and 
> https://www.python.org/dev/peps/pep-3121
> Here's valgrind suppression file from somebody who actually investigated the 
> Python leaks and identified the harmless ones: 
> https://github.com/libgit2/pygit2/blob/master/misc/valgrind-python.supp
> One example of a hidden leak in dispatch, which is revealed by making Python 
> suppressions more targetted:
> {code}
> 9: Direct leak of 56 byte(s) in 1 object(s) allocated from:
> 9: #0 0x7f78a3606e8f in __interceptor_malloc 
> (/nix/store/g40sl3zh3nv52vj0mrl4iki5iphh5ika-gcc-10.2.0-lib/lib/libasan.so.6+0xace8f)
> 9: #1 0x7f78a2d64afb in qd_malloc ../include/qpid/dispatch/ctools.h:229
> 9: #2 0x7f78a2d657da in qdr_core_subscribe 
> ../src/router_core/route_tables.c:149
> 9: #3 0x7f78a2c83072 in IoAdapter_init ../src/python_embedded.c:711
> 9: #4 0x7f78a2353a6c in type_call 
> (/nix/store/r85nxfnwiv45nbmf5yb60jj8ajim4m7w-python3-3.8.5/lib/libpython3.8.so.1.0+0x165a6c)
> {code}
> The problem is in
> {code}
> class Agent:
> ...
> def activate(self, address):
> ...
> self.io = IoAdapter(self.receive, address, 'L', '0', 
> TREATMENT_ANYCAST_CLOSEST)
> {code}
> IoAdapter refers to Agent (through the bound method reference self.receive) 
> and Agent refers to IoAdapter (through property self.io). Since IoAdapter is 
> implemented in C and does not implement support for Python's cyclic GC, there 
> is no way to break the cycle.
> Heap dump in attachment. The bound method is at the top of the picture. 
> (Ignore the Mock objects, I was trying to simplify the picture while not 
> getting crashes due to too much meddling).
> h3. Random observations
> It is possible to build special Debug build of Python, which has tools to 
> detect leaks, asserts to prevent negative refcounts, etc. 
> https://pythonextensionpatterns.readthedocs.io/en/latest/debugging/debug_python.html#debug-version-of-python-memory-alloc-label
> Use the following to detect python leaks (instead of valgrind)
> https://docs.python.org/3/library/tracemalloc.html
> Use https://pypi.org/project/objgraph (with graphviz) to view

[jira] [Commented] (DISPATCH-1962) Make LSan suppressions more targeted and specific (within reason)

2021-02-23 Thread ASF GitHub Bot (Jira)



[ 
https://issues.apache.org/jira/browse/DISPATCH-1962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17289247#comment-17289247
 ] 

ASF GitHub Bot commented on DISPATCH-1962:
--

jiridanek commented on a change in pull request #1048:
URL: https://github.com/apache/qpid-dispatch/pull/1048#discussion_r581285691



##
File path: tests/lsan.supp
##
@@ -23,15 +44,56 @@ leak:_ctypes_alloc_format_string
 leak:__strdup
 
 
- Miscellaneous 3rd party libraries, test code, etc:
+ Miscellaneous 3rd party libraries:
 
 
-leak:*libpython*
-leak:*libwebsockets*
-leak:*python2*
+### Python
 
-# We should be able to uncomment these once all known dispatch leaks have been 
fixed
-leak:*libqpid-proton*
+# these Python leaks happen even after simple Py_Initialize(); Py_Finalize();
+#  https://bugs.python.org/issue1635741
+leak:^_PyObject_Realloc
+leak:^PyObject_Malloc$
+leak:^PyThread_allocate_lock$
 
-# Ignore test code
-leak:run_unit_tests.c
+# the PyMalloc mechanism is incompatible with Valgrind, it must be disabled or 
reported "leaks" must be suppressed

Review comment:
   Yes, exactly. These following lines require Python debug symbols to be 
installed, so the function names can be matched.





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Make LSan suppressions more targeted and specific (within reason)
> -
>
> Key: DISPATCH-1962
> URL: https://issues.apache.org/jira/browse/DISPATCH-1962
> Project: Qpid Dispatch
>  Issue Type: Test
>Affects Versions: 1.15.0
>Reporter: Jiri Daněk
>Priority: Major
> Attachments: dispatch_leaking_agent.png
>
>
> There are some unfortunate suppressions in {{tests/lsan.supp}} at this moment:
> {code}
> leak:*libwebsockets*
> {code}
> This is way too broad. It suppresses leaks in Dispatch files, since it 
> matches e.g. {{src/http-libwebsockets.c}}.
> The stars at the beginning and end are actually assumed implicitly. If you do 
> not want substring match, you have to do ^foo$. LSan suppression format is 
> simplistic, very unlike Valgrind's.
> {code}
> leak:run_unit_tests.c
> {code}
> Same thing, any leaks revealed by running unit_tests get suppressed. This 
> suppression suppresses all leak traces that include run_unit_tests.c anywhere 
> in the stack. What't the point of running such tests under leak detector, 
> then?
> {code}
> leak:run_unit_tests.c
> leak:^libqpid-proton.so$
> {code}
> Same thing. The patterns suppress all leaks that include Python (or Proton) 
> anywhere in the stacktrace. That means there are huge blind spots where 
> dispatch leaks can hide. This is a weakness of the lsan.supp syntax (Valgrind 
> suppressions can be much more targeted and discerning).
> h3. Python leaks
> Leaks are known and there is ongoing effort to fight them: 
> https://bugs.python.org/issue1635741 (https://bugs.python.org/issue25302) and 
> https://www.python.org/dev/peps/pep-3121
> Here's valgrind suppression file from somebody who actually investigated the 
> Python leaks and identified the harmless ones: 
> https://github.com/libgit2/pygit2/blob/master/misc/valgrind-python.supp
> One example of a hidden leak in dispatch, which is revealed by making Python 
> suppressions more targetted:
> {code}
> 9: Direct leak of 56 byte(s) in 1 object(s) allocated from:
> 9: #0 0x7f78a3606e8f in __interceptor_malloc 
> (/nix/store/g40sl3zh3nv52vj0mrl4iki5iphh5ika-gcc-10.2.0-lib/lib/libasan.so.6+0xace8f)
> 9: #1 0x7f78a2d64afb in qd_malloc ../include/qpid/dispatch/ctools.h:229
> 9: #2 0x7f78a2d657da in qdr_core_subscribe 
> ../src/router_core/route_tables.c:149
> 9: #3 0x7f78a2c83072 in IoAdapter_init ../src/python_embedded.c:711
> 9: #4 0x7f78a2353a6c in type_call 
> (/nix/store/r85nxfnwiv45nbmf5yb60jj8ajim4m7w-python3-3.8.5/lib/libpython3.8.so.1.0+0x165a6c)
> {code}
> The problem is in
> {code}
> class Agent:
> ...
> def activate(self, address):
> ...
> self.io = IoAdapter(self.receive, address, 'L', '0', 
> TREATMENT_ANYCAST_CLOSEST)
> {code}
> IoAdapter refers to Agent (through the bound method reference self.receive) 
> and Agent refers to IoAdapter (through property self.io). Since IoAdapter is 
> implemented in C and does not implement support for Python's cyclic GC, there 
> is no way to break the cycle.
> Heap dump in attachment. The bound method is at the top of the picture. 
> (Ignore the Mock objects, I was trying to simplify the picture while not 
> getting crashes due to too much meddling).
> h3. Random observations
> It is

[jira] [Commented] (DISPATCH-1962) Make LSan suppressions more targeted and specific (within reason)

2021-02-23 Thread ASF GitHub Bot (Jira)



[ 
https://issues.apache.org/jira/browse/DISPATCH-1962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17289246#comment-17289246
 ] 

ASF GitHub Bot commented on DISPATCH-1962:
--

jiridanek commented on a change in pull request #1048:
URL: https://github.com/apache/qpid-dispatch/pull/1048#discussion_r581284840



##
File path: cmake/RuntimeChecks.cmake
##
@@ -106,9 +106,17 @@ elseif(RUNTIME_CHECK STREQUAL "asan")
 message(FATAL_ERROR "libubsan not installed - address sanitizer not 
available")
   endif(UBSAN_LIBRARY-NOTFOUND)
   message(STATUS "Runtime memory checker: gcc/clang address sanitizers")
+  option(SANITIZE_3RD_PARTY "Detect leaks in 3rd party libraries used by 
Dispatch while running tests" OFF)
+  file (COPY "${CMAKE_SOURCE_DIR}/tests/lsan.supp" DESTINATION 
"${CMAKE_BINARY_DIR}/tests")

Review comment:
   I saw that pattern used elsewhere in dispatch. I thought that having 
generated files outside of `CMAKE_BINARY_DIR` is discouraged.
   
   One problem with that is when you have multiple `build` folders, e.g. debug 
and release build. Then running cmake for both at approx the same time (my IDE 
does that) can cause weird issues.
   
   I guess I can do as you ask. Also, I should probably try to use 
`configure_file` in cmake.





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Make LSan suppressions more targeted and specific (within reason)
> -
>
> Key: DISPATCH-1962
> URL: https://issues.apache.org/jira/browse/DISPATCH-1962
> Project: Qpid Dispatch
>  Issue Type: Test
>Affects Versions: 1.15.0
>Reporter: Jiri Daněk
>Priority: Major
> Attachments: dispatch_leaking_agent.png
>
>
> There are some unfortunate suppressions in {{tests/lsan.supp}} at this moment:
> {code}
> leak:*libwebsockets*
> {code}
> This is way too broad. It suppresses leaks in Dispatch files, since it 
> matches e.g. {{src/http-libwebsockets.c}}.
> The stars at the beginning and end are actually assumed implicitly. If you do 
> not want substring match, you have to do ^foo$. LSan suppression format is 
> simplistic, very unlike Valgrind's.
> {code}
> leak:run_unit_tests.c
> {code}
> Same thing, any leaks revealed by running unit_tests get suppressed. This 
> suppression suppresses all leak traces that include run_unit_tests.c anywhere 
> in the stack. What't the point of running such tests under leak detector, 
> then?
> {code}
> leak:run_unit_tests.c
> leak:^libqpid-proton.so$
> {code}
> Same thing. The patterns suppress all leaks that include Python (or Proton) 
> anywhere in the stacktrace. That means there are huge blind spots where 
> dispatch leaks can hide. This is a weakness of the lsan.supp syntax (Valgrind 
> suppressions can be much more targeted and discerning).
> h3. Python leaks
> Leaks are known and there is ongoing effort to fight them: 
> https://bugs.python.org/issue1635741 (https://bugs.python.org/issue25302) and 
> https://www.python.org/dev/peps/pep-3121
> Here's valgrind suppression file from somebody who actually investigated the 
> Python leaks and identified the harmless ones: 
> https://github.com/libgit2/pygit2/blob/master/misc/valgrind-python.supp
> One example of a hidden leak in dispatch, which is revealed by making Python 
> suppressions more targetted:
> {code}
> 9: Direct leak of 56 byte(s) in 1 object(s) allocated from:
> 9: #0 0x7f78a3606e8f in __interceptor_malloc 
> (/nix/store/g40sl3zh3nv52vj0mrl4iki5iphh5ika-gcc-10.2.0-lib/lib/libasan.so.6+0xace8f)
> 9: #1 0x7f78a2d64afb in qd_malloc ../include/qpid/dispatch/ctools.h:229
> 9: #2 0x7f78a2d657da in qdr_core_subscribe 
> ../src/router_core/route_tables.c:149
> 9: #3 0x7f78a2c83072 in IoAdapter_init ../src/python_embedded.c:711
> 9: #4 0x7f78a2353a6c in type_call 
> (/nix/store/r85nxfnwiv45nbmf5yb60jj8ajim4m7w-python3-3.8.5/lib/libpython3.8.so.1.0+0x165a6c)
> {code}
> The problem is in
> {code}
> class Agent:
> ...
> def activate(self, address):
> ...
> self.io = IoAdapter(self.receive, address, 'L', '0', 
> TREATMENT_ANYCAST_CLOSEST)
> {code}
> IoAdapter refers to Agent (through the bound method reference self.receive) 
> and Agent refers to IoAdapter (through property self.io). Since IoAdapter is 
> implemented in C and does not implement support for Python's cyclic GC, there 
> is no way to break the cycle.
> Heap dump in attachment. The bound method is at the top of the picture. 
> (Ignore the Mock objects, I was trying to simplify the picture while not 
> getting crashes due to too

[jira] [Commented] (DISPATCH-1962) Make LSan suppressions more targeted and specific (within reason)

2021-02-23 Thread ASF GitHub Bot (Jira)



[ 
https://issues.apache.org/jira/browse/DISPATCH-1962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17289241#comment-17289241
 ] 

ASF GitHub Bot commented on DISPATCH-1962:
--

jiridanek commented on a change in pull request #1048:
URL: https://github.com/apache/qpid-dispatch/pull/1048#discussion_r581281555



##
File path: cmake/RuntimeChecks.cmake
##
@@ -106,9 +106,17 @@ elseif(RUNTIME_CHECK STREQUAL "asan")
 message(FATAL_ERROR "libubsan not installed - address sanitizer not 
available")
   endif(UBSAN_LIBRARY-NOTFOUND)
   message(STATUS "Runtime memory checker: gcc/clang address sanitizers")
+  option(SANITIZE_3RD_PARTY "Detect leaks in 3rd party libraries used by 
Dispatch while running tests" OFF)
+  file (COPY "${CMAKE_SOURCE_DIR}/tests/lsan.supp" DESTINATION 
"${CMAKE_BINARY_DIR}/tests")
+  if (NOT SANITIZE_3RD_PARTY)

Review comment:
   There used to be python, libwebsockets and proton in the .so ignores. 
Proton is I'd say 2nd party and it is not supposed to leak, so the so should 
not be suppressed wholesale. The libwebsockets ignore will be unnecessary, so I 
pulled it out into the lsan.supp directly and I have a plan to fix the leak 
there. Now only Python remains.
   
   I think I prefer having an option with a generic name. imagine if there were 
any leaks in cyrus sasl or openssl, those would go under this option. It is 
possible that by fixing other suppressions, something like this will crop up.





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Make LSan suppressions more targeted and specific (within reason)
> -
>
> Key: DISPATCH-1962
> URL: https://issues.apache.org/jira/browse/DISPATCH-1962
> Project: Qpid Dispatch
>  Issue Type: Test
>Affects Versions: 1.15.0
>Reporter: Jiri Daněk
>Priority: Major
> Attachments: dispatch_leaking_agent.png
>
>
> There are some unfortunate suppressions in {{tests/lsan.supp}} at this moment:
> {code}
> leak:*libwebsockets*
> {code}
> This is way too broad. It suppresses leaks in Dispatch files, since it 
> matches e.g. {{src/http-libwebsockets.c}}.
> The stars at the beginning and end are actually assumed implicitly. If you do 
> not want substring match, you have to do ^foo$. LSan suppression format is 
> simplistic, very unlike Valgrind's.
> {code}
> leak:run_unit_tests.c
> {code}
> Same thing, any leaks revealed by running unit_tests get suppressed. This 
> suppression suppresses all leak traces that include run_unit_tests.c anywhere 
> in the stack. What't the point of running such tests under leak detector, 
> then?
> {code}
> leak:run_unit_tests.c
> leak:^libqpid-proton.so$
> {code}
> Same thing. The patterns suppress all leaks that include Python (or Proton) 
> anywhere in the stacktrace. That means there are huge blind spots where 
> dispatch leaks can hide. This is a weakness of the lsan.supp syntax (Valgrind 
> suppressions can be much more targeted and discerning).
> h3. Python leaks
> Leaks are known and there is ongoing effort to fight them: 
> https://bugs.python.org/issue1635741 (https://bugs.python.org/issue25302) and 
> https://www.python.org/dev/peps/pep-3121
> Here's valgrind suppression file from somebody who actually investigated the 
> Python leaks and identified the harmless ones: 
> https://github.com/libgit2/pygit2/blob/master/misc/valgrind-python.supp
> One example of a hidden leak in dispatch, which is revealed by making Python 
> suppressions more targetted:
> {code}
> 9: Direct leak of 56 byte(s) in 1 object(s) allocated from:
> 9: #0 0x7f78a3606e8f in __interceptor_malloc 
> (/nix/store/g40sl3zh3nv52vj0mrl4iki5iphh5ika-gcc-10.2.0-lib/lib/libasan.so.6+0xace8f)
> 9: #1 0x7f78a2d64afb in qd_malloc ../include/qpid/dispatch/ctools.h:229
> 9: #2 0x7f78a2d657da in qdr_core_subscribe 
> ../src/router_core/route_tables.c:149
> 9: #3 0x7f78a2c83072 in IoAdapter_init ../src/python_embedded.c:711
> 9: #4 0x7f78a2353a6c in type_call 
> (/nix/store/r85nxfnwiv45nbmf5yb60jj8ajim4m7w-python3-3.8.5/lib/libpython3.8.so.1.0+0x165a6c)
> {code}
> The problem is in
> {code}
> class Agent:
> ...
> def activate(self, address):
> ...
> self.io = IoAdapter(self.receive, address, 'L', '0', 
> TREATMENT_ANYCAST_CLOSEST)
> {code}
> IoAdapter refers to Agent (through the bound method reference self.receive) 
> and Agent refers to IoAdapter (through property self.io). Since IoAdapter is 
> implemented in C and does not implement support for Python's cyclic GC, there 
> is no way to break the cycle.

[jira] [Commented] (DISPATCH-1962) Make LSan suppressions more targeted and specific (within reason)

2021-02-23 Thread ASF GitHub Bot (Jira)



[ 
https://issues.apache.org/jira/browse/DISPATCH-1962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17289237#comment-17289237
 ] 

ASF GitHub Bot commented on DISPATCH-1962:
--

ganeshmurthy commented on a change in pull request #1048:
URL: https://github.com/apache/qpid-dispatch/pull/1048#discussion_r581277212



##
File path: tests/lsan.supp
##
@@ -10,6 +10,27 @@ leak:qd_policy_c_counts_alloc
 leak:qd_policy_open_fetch_settings
 leak:qdr_error_description
 
+# to be triaged; unit_tests
+leak:http-libwebsockets.c

Review comment:
   Just trying to understand why leak:http-libwebsockets.c was added. Can 
you please elaborate ?





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Make LSan suppressions more targeted and specific (within reason)
> -
>
> Key: DISPATCH-1962
> URL: https://issues.apache.org/jira/browse/DISPATCH-1962
> Project: Qpid Dispatch
>  Issue Type: Test
>Affects Versions: 1.15.0
>Reporter: Jiri Daněk
>Priority: Major
> Attachments: dispatch_leaking_agent.png
>
>
> There are some unfortunate suppressions in {{tests/lsan.supp}} at this moment:
> {code}
> leak:*libwebsockets*
> {code}
> This is way too broad. It suppresses leaks in Dispatch files, since it 
> matches e.g. {{src/http-libwebsockets.c}}.
> The stars at the beginning and end are actually assumed implicitly. If you do 
> not want substring match, you have to do ^foo$. LSan suppression format is 
> simplistic, very unlike Valgrind's.
> {code}
> leak:run_unit_tests.c
> {code}
> Same thing, any leaks revealed by running unit_tests get suppressed. This 
> suppression suppresses all leak traces that include run_unit_tests.c anywhere 
> in the stack. What't the point of running such tests under leak detector, 
> then?
> {code}
> leak:run_unit_tests.c
> leak:^libqpid-proton.so$
> {code}
> Same thing. The patterns suppress all leaks that include Python (or Proton) 
> anywhere in the stacktrace. That means there are huge blind spots where 
> dispatch leaks can hide. This is a weakness of the lsan.supp syntax (Valgrind 
> suppressions can be much more targeted and discerning).
> h3. Python leaks
> Leaks are known and there is ongoing effort to fight them: 
> https://bugs.python.org/issue1635741 (https://bugs.python.org/issue25302) and 
> https://www.python.org/dev/peps/pep-3121
> Here's valgrind suppression file from somebody who actually investigated the 
> Python leaks and identified the harmless ones: 
> https://github.com/libgit2/pygit2/blob/master/misc/valgrind-python.supp
> One example of a hidden leak in dispatch, which is revealed by making Python 
> suppressions more targetted:
> {code}
> 9: Direct leak of 56 byte(s) in 1 object(s) allocated from:
> 9: #0 0x7f78a3606e8f in __interceptor_malloc 
> (/nix/store/g40sl3zh3nv52vj0mrl4iki5iphh5ika-gcc-10.2.0-lib/lib/libasan.so.6+0xace8f)
> 9: #1 0x7f78a2d64afb in qd_malloc ../include/qpid/dispatch/ctools.h:229
> 9: #2 0x7f78a2d657da in qdr_core_subscribe 
> ../src/router_core/route_tables.c:149
> 9: #3 0x7f78a2c83072 in IoAdapter_init ../src/python_embedded.c:711
> 9: #4 0x7f78a2353a6c in type_call 
> (/nix/store/r85nxfnwiv45nbmf5yb60jj8ajim4m7w-python3-3.8.5/lib/libpython3.8.so.1.0+0x165a6c)
> {code}
> The problem is in
> {code}
> class Agent:
> ...
> def activate(self, address):
> ...
> self.io = IoAdapter(self.receive, address, 'L', '0', 
> TREATMENT_ANYCAST_CLOSEST)
> {code}
> IoAdapter refers to Agent (through the bound method reference self.receive) 
> and Agent refers to IoAdapter (through property self.io). Since IoAdapter is 
> implemented in C and does not implement support for Python's cyclic GC, there 
> is no way to break the cycle.
> Heap dump in attachment. The bound method is at the top of the picture. 
> (Ignore the Mock objects, I was trying to simplify the picture while not 
> getting crashes due to too much meddling).
> h3. Random observations
> It is possible to build special Debug build of Python, which has tools to 
> detect leaks, asserts to prevent negative refcounts, etc. 
> https://pythonextensionpatterns.readthedocs.io/en/latest/debugging/debug_python.html#debug-version-of-python-memory-alloc-label
> Use the following to detect python leaks (instead of valgrind)
> https://docs.python.org/3/library/tracemalloc.html
> Use https://pypi.org/project/objgraph (with graphviz) to view heap object 
> trees. The following renders the picture as a png under /tmp and prints the 
> path to stdout.
> {code}
> int ret =

[jira] [Commented] (DISPATCH-1962) Make LSan suppressions more targeted and specific (within reason)

2021-02-23 Thread ASF GitHub Bot (Jira)



[ 
https://issues.apache.org/jira/browse/DISPATCH-1962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17289232#comment-17289232
 ] 

ASF GitHub Bot commented on DISPATCH-1962:
--

ganeshmurthy commented on a change in pull request #1048:
URL: https://github.com/apache/qpid-dispatch/pull/1048#discussion_r581269285



##
File path: tests/lsan.supp
##
@@ -23,15 +44,56 @@ leak:_ctypes_alloc_format_string
 leak:__strdup
 
 
- Miscellaneous 3rd party libraries, test code, etc:
+ Miscellaneous 3rd party libraries:
 
 
-leak:*libpython*
-leak:*libwebsockets*
-leak:*python2*
+### Python
 
-# We should be able to uncomment these once all known dispatch leaks have been 
fixed
-leak:*libqpid-proton*
+# these Python leaks happen even after simple Py_Initialize(); Py_Finalize();
+#  https://bugs.python.org/issue1635741
+leak:^_PyObject_Realloc
+leak:^PyObject_Malloc$
+leak:^PyThread_allocate_lock$
 
-# Ignore test code
-leak:run_unit_tests.c
+# the PyMalloc mechanism is incompatible with Valgrind, it must be disabled or 
reported "leaks" must be suppressed

Review comment:
   If SANITIZE_3RD_PARTY is OFF, these following Python leak lines are 
overridden by leak:/libpython2.*.so\nleak:/libpython3.*.so\n ??





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Make LSan suppressions more targeted and specific (within reason)
> -
>
> Key: DISPATCH-1962
> URL: https://issues.apache.org/jira/browse/DISPATCH-1962
> Project: Qpid Dispatch
>  Issue Type: Test
>Affects Versions: 1.15.0
>Reporter: Jiri Daněk
>Priority: Major
> Attachments: dispatch_leaking_agent.png
>
>
> There are some unfortunate suppressions in {{tests/lsan.supp}} at this moment:
> {code}
> leak:*libwebsockets*
> {code}
> This is way too broad. It suppresses leaks in Dispatch files, since it 
> matches e.g. {{src/http-libwebsockets.c}}.
> The stars at the beginning and end are actually assumed implicitly. If you do 
> not want substring match, you have to do ^foo$. LSan suppression format is 
> simplistic, very unlike Valgrind's.
> {code}
> leak:run_unit_tests.c
> {code}
> Same thing, any leaks revealed by running unit_tests get suppressed. This 
> suppression suppresses all leak traces that include run_unit_tests.c anywhere 
> in the stack. What't the point of running such tests under leak detector, 
> then?
> {code}
> leak:run_unit_tests.c
> leak:^libqpid-proton.so$
> {code}
> Same thing. The patterns suppress all leaks that include Python (or Proton) 
> anywhere in the stacktrace. That means there are huge blind spots where 
> dispatch leaks can hide. This is a weakness of the lsan.supp syntax (Valgrind 
> suppressions can be much more targeted and discerning).
> h3. Python leaks
> Leaks are known and there is ongoing effort to fight them: 
> https://bugs.python.org/issue1635741 (https://bugs.python.org/issue25302) and 
> https://www.python.org/dev/peps/pep-3121
> Here's valgrind suppression file from somebody who actually investigated the 
> Python leaks and identified the harmless ones: 
> https://github.com/libgit2/pygit2/blob/master/misc/valgrind-python.supp
> One example of a hidden leak in dispatch, which is revealed by making Python 
> suppressions more targetted:
> {code}
> 9: Direct leak of 56 byte(s) in 1 object(s) allocated from:
> 9: #0 0x7f78a3606e8f in __interceptor_malloc 
> (/nix/store/g40sl3zh3nv52vj0mrl4iki5iphh5ika-gcc-10.2.0-lib/lib/libasan.so.6+0xace8f)
> 9: #1 0x7f78a2d64afb in qd_malloc ../include/qpid/dispatch/ctools.h:229
> 9: #2 0x7f78a2d657da in qdr_core_subscribe 
> ../src/router_core/route_tables.c:149
> 9: #3 0x7f78a2c83072 in IoAdapter_init ../src/python_embedded.c:711
> 9: #4 0x7f78a2353a6c in type_call 
> (/nix/store/r85nxfnwiv45nbmf5yb60jj8ajim4m7w-python3-3.8.5/lib/libpython3.8.so.1.0+0x165a6c)
> {code}
> The problem is in
> {code}
> class Agent:
> ...
> def activate(self, address):
> ...
> self.io = IoAdapter(self.receive, address, 'L', '0', 
> TREATMENT_ANYCAST_CLOSEST)
> {code}
> IoAdapter refers to Agent (through the bound method reference self.receive) 
> and Agent refers to IoAdapter (through property self.io). Since IoAdapter is 
> implemented in C and does not implement support for Python's cyclic GC, there 
> is no way to break the cycle.
> Heap dump in attachment. The bound method is at the top of the picture. 
> (Ignore the Mock objects, I was trying to simplify the picture while not 
> getting crashes due to too much meddling).
> h3. Random

[jira] [Commented] (DISPATCH-1962) Make LSan suppressions more targeted and specific (within reason)

2021-02-23 Thread ASF GitHub Bot (Jira)



[ 
https://issues.apache.org/jira/browse/DISPATCH-1962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17289231#comment-17289231
 ] 

ASF GitHub Bot commented on DISPATCH-1962:
--

ganeshmurthy commented on a change in pull request #1048:
URL: https://github.com/apache/qpid-dispatch/pull/1048#discussion_r581267252



##
File path: cmake/RuntimeChecks.cmake
##
@@ -106,9 +106,17 @@ elseif(RUNTIME_CHECK STREQUAL "asan")
 message(FATAL_ERROR "libubsan not installed - address sanitizer not 
available")
   endif(UBSAN_LIBRARY-NOTFOUND)
   message(STATUS "Runtime memory checker: gcc/clang address sanitizers")
+  option(SANITIZE_3RD_PARTY "Detect leaks in 3rd party libraries used by 
Dispatch while running tests" OFF)
+  file (COPY "${CMAKE_SOURCE_DIR}/tests/lsan.supp" DESTINATION 
"${CMAKE_BINARY_DIR}/tests")

Review comment:
   Instead of copying to the build folder can we keep the lsan.supp in the 
source folder but add it to the .gitignore ? Then add a file called 
lsan.supp.in file to the tests folder and add that to git.





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Make LSan suppressions more targeted and specific (within reason)
> -
>
> Key: DISPATCH-1962
> URL: https://issues.apache.org/jira/browse/DISPATCH-1962
> Project: Qpid Dispatch
>  Issue Type: Test
>Affects Versions: 1.15.0
>Reporter: Jiri Daněk
>Priority: Major
> Attachments: dispatch_leaking_agent.png
>
>
> There are some unfortunate suppressions in {{tests/lsan.supp}} at this moment:
> {code}
> leak:*libwebsockets*
> {code}
> This is way too broad. It suppresses leaks in Dispatch files, since it 
> matches e.g. {{src/http-libwebsockets.c}}.
> The stars at the beginning and end are actually assumed implicitly. If you do 
> not want substring match, you have to do ^foo$. LSan suppression format is 
> simplistic, very unlike Valgrind's.
> {code}
> leak:run_unit_tests.c
> {code}
> Same thing, any leaks revealed by running unit_tests get suppressed. This 
> suppression suppresses all leak traces that include run_unit_tests.c anywhere 
> in the stack. What't the point of running such tests under leak detector, 
> then?
> {code}
> leak:run_unit_tests.c
> leak:^libqpid-proton.so$
> {code}
> Same thing. The patterns suppress all leaks that include Python (or Proton) 
> anywhere in the stacktrace. That means there are huge blind spots where 
> dispatch leaks can hide. This is a weakness of the lsan.supp syntax (Valgrind 
> suppressions can be much more targeted and discerning).
> h3. Python leaks
> Leaks are known and there is ongoing effort to fight them: 
> https://bugs.python.org/issue1635741 (https://bugs.python.org/issue25302) and 
> https://www.python.org/dev/peps/pep-3121
> Here's valgrind suppression file from somebody who actually investigated the 
> Python leaks and identified the harmless ones: 
> https://github.com/libgit2/pygit2/blob/master/misc/valgrind-python.supp
> One example of a hidden leak in dispatch, which is revealed by making Python 
> suppressions more targetted:
> {code}
> 9: Direct leak of 56 byte(s) in 1 object(s) allocated from:
> 9: #0 0x7f78a3606e8f in __interceptor_malloc 
> (/nix/store/g40sl3zh3nv52vj0mrl4iki5iphh5ika-gcc-10.2.0-lib/lib/libasan.so.6+0xace8f)
> 9: #1 0x7f78a2d64afb in qd_malloc ../include/qpid/dispatch/ctools.h:229
> 9: #2 0x7f78a2d657da in qdr_core_subscribe 
> ../src/router_core/route_tables.c:149
> 9: #3 0x7f78a2c83072 in IoAdapter_init ../src/python_embedded.c:711
> 9: #4 0x7f78a2353a6c in type_call 
> (/nix/store/r85nxfnwiv45nbmf5yb60jj8ajim4m7w-python3-3.8.5/lib/libpython3.8.so.1.0+0x165a6c)
> {code}
> The problem is in
> {code}
> class Agent:
> ...
> def activate(self, address):
> ...
> self.io = IoAdapter(self.receive, address, 'L', '0', 
> TREATMENT_ANYCAST_CLOSEST)
> {code}
> IoAdapter refers to Agent (through the bound method reference self.receive) 
> and Agent refers to IoAdapter (through property self.io). Since IoAdapter is 
> implemented in C and does not implement support for Python's cyclic GC, there 
> is no way to break the cycle.
> Heap dump in attachment. The bound method is at the top of the picture. 
> (Ignore the Mock objects, I was trying to simplify the picture while not 
> getting crashes due to too much meddling).
> h3. Random observations
> It is possible to build special Debug build of Python, which has tools to 
> detect leaks, asserts to prevent negative refcounts, etc. 
>

[jira] [Commented] (DISPATCH-1962) Make LSan suppressions more targeted and specific (within reason)

2021-02-23 Thread ASF GitHub Bot (Jira)



[ 
https://issues.apache.org/jira/browse/DISPATCH-1962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17289228#comment-17289228
 ] 

ASF GitHub Bot commented on DISPATCH-1962:
--

ganeshmurthy commented on a change in pull request #1048:
URL: https://github.com/apache/qpid-dispatch/pull/1048#discussion_r581265869



##
File path: cmake/RuntimeChecks.cmake
##
@@ -106,9 +106,17 @@ elseif(RUNTIME_CHECK STREQUAL "asan")
 message(FATAL_ERROR "libubsan not installed - address sanitizer not 
available")
   endif(UBSAN_LIBRARY-NOTFOUND)
   message(STATUS "Runtime memory checker: gcc/clang address sanitizers")
+  option(SANITIZE_3RD_PARTY "Detect leaks in 3rd party libraries used by 
Dispatch while running tests" OFF)
+  file (COPY "${CMAKE_SOURCE_DIR}/tests/lsan.supp" DESTINATION 
"${CMAKE_BINARY_DIR}/tests")
+  if (NOT SANITIZE_3RD_PARTY)

Review comment:
   The flag is called SANITIZE_3RD_PARTY but it seems to be specific to 
libpython2.*.so  and libpython3.*.so
   Is Python really third party ?





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Make LSan suppressions more targeted and specific (within reason)
> -
>
> Key: DISPATCH-1962
> URL: https://issues.apache.org/jira/browse/DISPATCH-1962
> Project: Qpid Dispatch
>  Issue Type: Test
>Affects Versions: 1.15.0
>Reporter: Jiri Daněk
>Priority: Major
> Attachments: dispatch_leaking_agent.png
>
>
> There are some unfortunate suppressions in {{tests/lsan.supp}} at this moment:
> {code}
> leak:*libwebsockets*
> {code}
> This is way too broad. It suppresses leaks in Dispatch files, since it 
> matches e.g. {{src/http-libwebsockets.c}}.
> The stars at the beginning and end are actually assumed implicitly. If you do 
> not want substring match, you have to do ^foo$. LSan suppression format is 
> simplistic, very unlike Valgrind's.
> {code}
> leak:run_unit_tests.c
> {code}
> Same thing, any leaks revealed by running unit_tests get suppressed. This 
> suppression suppresses all leak traces that include run_unit_tests.c anywhere 
> in the stack. What't the point of running such tests under leak detector, 
> then?
> {code}
> leak:run_unit_tests.c
> leak:^libqpid-proton.so$
> {code}
> Same thing. The patterns suppress all leaks that include Python (or Proton) 
> anywhere in the stacktrace. That means there are huge blind spots where 
> dispatch leaks can hide. This is a weakness of the lsan.supp syntax (Valgrind 
> suppressions can be much more targeted and discerning).
> h3. Python leaks
> Leaks are known and there is ongoing effort to fight them: 
> https://bugs.python.org/issue1635741 (https://bugs.python.org/issue25302) and 
> https://www.python.org/dev/peps/pep-3121
> Here's valgrind suppression file from somebody who actually investigated the 
> Python leaks and identified the harmless ones: 
> https://github.com/libgit2/pygit2/blob/master/misc/valgrind-python.supp
> One example of a hidden leak in dispatch, which is revealed by making Python 
> suppressions more targetted:
> {code}
> 9: Direct leak of 56 byte(s) in 1 object(s) allocated from:
> 9: #0 0x7f78a3606e8f in __interceptor_malloc 
> (/nix/store/g40sl3zh3nv52vj0mrl4iki5iphh5ika-gcc-10.2.0-lib/lib/libasan.so.6+0xace8f)
> 9: #1 0x7f78a2d64afb in qd_malloc ../include/qpid/dispatch/ctools.h:229
> 9: #2 0x7f78a2d657da in qdr_core_subscribe 
> ../src/router_core/route_tables.c:149
> 9: #3 0x7f78a2c83072 in IoAdapter_init ../src/python_embedded.c:711
> 9: #4 0x7f78a2353a6c in type_call 
> (/nix/store/r85nxfnwiv45nbmf5yb60jj8ajim4m7w-python3-3.8.5/lib/libpython3.8.so.1.0+0x165a6c)
> {code}
> The problem is in
> {code}
> class Agent:
> ...
> def activate(self, address):
> ...
> self.io = IoAdapter(self.receive, address, 'L', '0', 
> TREATMENT_ANYCAST_CLOSEST)
> {code}
> IoAdapter refers to Agent (through the bound method reference self.receive) 
> and Agent refers to IoAdapter (through property self.io). Since IoAdapter is 
> implemented in C and does not implement support for Python's cyclic GC, there 
> is no way to break the cycle.
> Heap dump in attachment. The bound method is at the top of the picture. 
> (Ignore the Mock objects, I was trying to simplify the picture while not 
> getting crashes due to too much meddling).
> h3. Random observations
> It is possible to build special Debug build of Python, which has tools to 
> detect leaks, asserts to prevent negative refcounts, etc. 
>

[jira] [Commented] (DISPATCH-1962) Make LSan suppressions more targeted and specific (within reason)

2021-02-21 Thread ASF GitHub Bot (Jira)



[ 
https://issues.apache.org/jira/browse/DISPATCH-1962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17287932#comment-17287932
 ] 

ASF GitHub Bot commented on DISPATCH-1962:
--

codecov-io edited a comment on pull request #1048:
URL: https://github.com/apache/qpid-dispatch/pull/1048#issuecomment-782837844


   # 
[Codecov](https://codecov.io/gh/apache/qpid-dispatch/pull/1048?src=pr=h1) 
Report
   > Merging 
[#1048](https://codecov.io/gh/apache/qpid-dispatch/pull/1048?src=pr=desc) 
(3367dea) into 
[master](https://codecov.io/gh/apache/qpid-dispatch/commit/844f995a580ca630afe951ea7c5071bb02bc33f6?el=desc)
 (844f995) will **increase** coverage by `0.08%`.
   > The diff coverage is `n/a`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/qpid-dispatch/pull/1048/graphs/tree.svg?width=650=150=pr=rk2Cgd27pP)](https://codecov.io/gh/apache/qpid-dispatch/pull/1048?src=pr=tree)
   
   ```diff
   @@Coverage Diff @@
   ##   master#1048  +/-   ##
   ==
   + Coverage   82.48%   82.56%   +0.08% 
   ==
 Files 111  111  
 Lines   2742527425  
   ==
   + Hits2262122644  +23 
   + Misses   4804 4781  -23 
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/qpid-dispatch/pull/1048?src=pr=tree) | 
Coverage Δ | |
   |---|---|---|
   | 
[...router\_core/modules/edge\_router/link\_route\_proxy.c](https://codecov.io/gh/apache/qpid-dispatch/pull/1048/diff?src=pr=tree#diff-c3JjL3JvdXRlcl9jb3JlL21vZHVsZXMvZWRnZV9yb3V0ZXIvbGlua19yb3V0ZV9wcm94eS5j)
 | `78.69% <0.00%> (-4.15%)` | :arrow_down: |
   | 
[src/router\_core/modules/edge\_router/edge\_mgmt.c](https://codecov.io/gh/apache/qpid-dispatch/pull/1048/diff?src=pr=tree#diff-c3JjL3JvdXRlcl9jb3JlL21vZHVsZXMvZWRnZV9yb3V0ZXIvZWRnZV9tZ210LmM=)
 | `84.15% <0.00%> (-1.00%)` | :arrow_down: |
   | 
[src/router\_core/delivery.c](https://codecov.io/gh/apache/qpid-dispatch/pull/1048/diff?src=pr=tree#diff-c3JjL3JvdXRlcl9jb3JlL2RlbGl2ZXJ5LmM=)
 | `93.12% <0.00%> (-0.39%)` | :arrow_down: |
   | 
[src/message.c](https://codecov.io/gh/apache/qpid-dispatch/pull/1048/diff?src=pr=tree#diff-c3JjL21lc3NhZ2UuYw==)
 | `87.07% <0.00%> (-0.08%)` | :arrow_down: |
   | 
[src/router\_node.c](https://codecov.io/gh/apache/qpid-dispatch/pull/1048/diff?src=pr=tree#diff-c3JjL3JvdXRlcl9ub2RlLmM=)
 | `94.13% <0.00%> (+0.10%)` | :arrow_up: |
   | 
[src/adaptors/http1/http1\_codec.c](https://codecov.io/gh/apache/qpid-dispatch/pull/1048/diff?src=pr=tree#diff-c3JjL2FkYXB0b3JzL2h0dHAxL2h0dHAxX2NvZGVjLmM=)
 | `85.28% <0.00%> (+0.12%)` | :arrow_up: |
   | 
[src/adaptors/http1/http1\_server.c](https://codecov.io/gh/apache/qpid-dispatch/pull/1048/diff?src=pr=tree#diff-c3JjL2FkYXB0b3JzL2h0dHAxL2h0dHAxX3NlcnZlci5j)
 | `84.79% <0.00%> (+0.29%)` | :arrow_up: |
   | 
[src/router\_core/forwarder.c](https://codecov.io/gh/apache/qpid-dispatch/pull/1048/diff?src=pr=tree#diff-c3JjL3JvdXRlcl9jb3JlL2ZvcndhcmRlci5j)
 | `93.04% <0.00%> (+0.39%)` | :arrow_up: |
   | 
[src/router\_core/transfer.c](https://codecov.io/gh/apache/qpid-dispatch/pull/1048/diff?src=pr=tree#diff-c3JjL3JvdXRlcl9jb3JlL3RyYW5zZmVyLmM=)
 | `93.10% <0.00%> (+0.86%)` | :arrow_up: |
   | 
[...c/router\_core/modules/test\_hooks/core\_test\_hooks.c](https://codecov.io/gh/apache/qpid-dispatch/pull/1048/diff?src=pr=tree#diff-c3JjL3JvdXRlcl9jb3JlL21vZHVsZXMvdGVzdF9ob29rcy9jb3JlX3Rlc3RfaG9va3MuYw==)
 | `93.31% <0.00%> (+1.27%)` | :arrow_up: |
   | ... and [1 
more](https://codecov.io/gh/apache/qpid-dispatch/pull/1048/diff?src=pr=tree-more)
 | |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/qpid-dispatch/pull/1048?src=pr=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/qpid-dispatch/pull/1048?src=pr=footer).
 Last update 
[844f995...3367dea](https://codecov.io/gh/apache/qpid-dispatch/pull/1048?src=pr=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Make LSan suppressions more targeted and specific (within reason)
> -
>
> Key: DISPATCH-1962
> URL: https://issues.apache.org/jira/browse/DISPATCH-1962
> Project: Qpid Dispatch
>  Issue Type:

[jira] [Commented] (DISPATCH-1962) Make LSan suppressions more targeted and specific (within reason)

2021-02-21 Thread ASF GitHub Bot (Jira)



[ 
https://issues.apache.org/jira/browse/DISPATCH-1962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17287931#comment-17287931
 ] 

ASF GitHub Bot commented on DISPATCH-1962:
--

codecov-io edited a comment on pull request #1048:
URL: https://github.com/apache/qpid-dispatch/pull/1048#issuecomment-782837844


   # 
[Codecov](https://codecov.io/gh/apache/qpid-dispatch/pull/1048?src=pr=h1) 
Report
   > Merging 
[#1048](https://codecov.io/gh/apache/qpid-dispatch/pull/1048?src=pr=desc) 
(95a9447) into 
[master](https://codecov.io/gh/apache/qpid-dispatch/commit/844f995a580ca630afe951ea7c5071bb02bc33f6?el=desc)
 (844f995) will **increase** coverage by `0.02%`.
   > The diff coverage is `n/a`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/qpid-dispatch/pull/1048/graphs/tree.svg?width=650=150=pr=rk2Cgd27pP)](https://codecov.io/gh/apache/qpid-dispatch/pull/1048?src=pr=tree)
   
   ```diff
   @@Coverage Diff @@
   ##   master#1048  +/-   ##
   ==
   + Coverage   82.48%   82.51%   +0.02% 
   ==
 Files 111  111  
 Lines   2742527424   -1 
   ==
   + Hits2262122628   +7 
   + Misses   4804 4796   -8 
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/qpid-dispatch/pull/1048?src=pr=tree) | 
Coverage Δ | |
   |---|---|---|
   | 
[...router\_core/modules/edge\_router/link\_route\_proxy.c](https://codecov.io/gh/apache/qpid-dispatch/pull/1048/diff?src=pr=tree#diff-c3JjL3JvdXRlcl9jb3JlL21vZHVsZXMvZWRnZV9yb3V0ZXIvbGlua19yb3V0ZV9wcm94eS5j)
 | `78.69% <0.00%> (-4.15%)` | :arrow_down: |
   | 
[src/router\_core/modules/edge\_router/edge\_mgmt.c](https://codecov.io/gh/apache/qpid-dispatch/pull/1048/diff?src=pr=tree#diff-c3JjL3JvdXRlcl9jb3JlL21vZHVsZXMvZWRnZV9yb3V0ZXIvZWRnZV9tZ210LmM=)
 | `84.15% <0.00%> (-1.00%)` | :arrow_down: |
   | 
[src/iterator.c](https://codecov.io/gh/apache/qpid-dispatch/pull/1048/diff?src=pr=tree#diff-c3JjL2l0ZXJhdG9yLmM=)
 | `89.29% <0.00%> (-0.19%)` | :arrow_down: |
   | 
[src/message.c](https://codecov.io/gh/apache/qpid-dispatch/pull/1048/diff?src=pr=tree#diff-c3JjL21lc3NhZ2UuYw==)
 | `87.22% <0.00%> (+0.07%)` | :arrow_up: |
   | 
[src/router\_node.c](https://codecov.io/gh/apache/qpid-dispatch/pull/1048/diff?src=pr=tree#diff-c3JjL3JvdXRlcl9ub2RlLmM=)
 | `94.12% <0.00%> (+0.09%)` | :arrow_up: |
   | 
[src/parse.c](https://codecov.io/gh/apache/qpid-dispatch/pull/1048/diff?src=pr=tree#diff-c3JjL3BhcnNlLmM=)
 | `87.06% <0.00%> (+0.21%)` | :arrow_up: |
   | 
[src/router\_core/connections.c](https://codecov.io/gh/apache/qpid-dispatch/pull/1048/diff?src=pr=tree#diff-c3JjL3JvdXRlcl9jb3JlL2Nvbm5lY3Rpb25zLmM=)
 | `89.11% <0.00%> (+0.29%)` | :arrow_up: |
   | 
[src/router\_core/forwarder.c](https://codecov.io/gh/apache/qpid-dispatch/pull/1048/diff?src=pr=tree#diff-c3JjL3JvdXRlcl9jb3JlL2ZvcndhcmRlci5j)
 | `93.04% <0.00%> (+0.39%)` | :arrow_up: |
   | 
[src/router\_core/transfer.c](https://codecov.io/gh/apache/qpid-dispatch/pull/1048/diff?src=pr=tree#diff-c3JjL3JvdXRlcl9jb3JlL3RyYW5zZmVyLmM=)
 | `92.88% <0.00%> (+0.64%)` | :arrow_up: |
   | 
[...re/modules/edge\_addr\_tracking/edge\_addr\_tracking.c](https://codecov.io/gh/apache/qpid-dispatch/pull/1048/diff?src=pr=tree#diff-c3JjL3JvdXRlcl9jb3JlL21vZHVsZXMvZWRnZV9hZGRyX3RyYWNraW5nL2VkZ2VfYWRkcl90cmFja2luZy5j)
 | `91.39% <0.00%> (+1.07%)` | :arrow_up: |
   | ... and [1 
more](https://codecov.io/gh/apache/qpid-dispatch/pull/1048/diff?src=pr=tree-more)
 | |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/qpid-dispatch/pull/1048?src=pr=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/qpid-dispatch/pull/1048?src=pr=footer).
 Last update 
[844f995...3367dea](https://codecov.io/gh/apache/qpid-dispatch/pull/1048?src=pr=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Make LSan suppressions more targeted and specific (within reason)
> -
>
> Key: DISPATCH-1962
> URL: https://issues.apache.org/jira/browse/DISPATCH-1962
> Project: Qpid Dispatch
>  Issue Type: Test
>Affects Versions: 1.15.0
>Reporter: Jiri Daněk
>

[jira] [Commented] (DISPATCH-1962) Make LSan suppressions more targeted and specific (within reason)

2021-02-21 Thread ASF GitHub Bot (Jira)



[ 
https://issues.apache.org/jira/browse/DISPATCH-1962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17287921#comment-17287921
 ] 

ASF GitHub Bot commented on DISPATCH-1962:
--

jiridanek commented on pull request #1048:
URL: https://github.com/apache/qpid-dispatch/pull/1048#issuecomment-782841264


   One more leak in system_tests_http1_adaptor, which appears only sometimes, 
that needs to be suppressed
   
   ```
   68: ==15712==ERROR: LeakSanitizer: detected memory leaks
   68: 
   68: Indirect leak of 64 byte(s) in 1 object(s) allocated from:
   68: #0 0x7fc7780f9602 in malloc 
(/usr/lib/x86_64-linux-gnu/libasan.so.2+0x98602)
   68: #1 0x7fc776378396  (/lib/x86_64-linux-gnu/libc.so.6+0xeb396)
   68: #2 0x7fc77637bd5d in getaddrinfo 
(/lib/x86_64-linux-gnu/libc.so.6+0xeed5d)
   68: #3 0x7fc7780d268f  (/usr/lib/x86_64-linux-gnu/libasan.so.2+0x7168f)
   68: #4 0x7fc77572568e in pgetaddrinfo 
/home/travis/build/apache/qpid-dispatch/qpid-proton/c/src/proactor/epoll.c:1368
   68: #5 0x7fc77572568e in pn_proactor_raw_connect 
/home/travis/build/apache/qpid-dispatch/qpid-proton/c/src/proactor/epoll_raw_connection.c:196
   68: #6 0x7fc7778ce0ad in _do_reconnect 
/home/travis/build/apache/qpid-dispatch/src/adaptors/http1/http1_server.c:449
   68: #7 0x7fc777a86632 in qd_timer_visit 
/home/travis/build/apache/qpid-dispatch/src/timer.c:201
   68: #8 0x7fc777a773b6 in handle 
/home/travis/build/apache/qpid-dispatch/src/server.c:1008
   68: #9 0x7fc777a7d9b7 in thread_run 
/home/travis/build/apache/qpid-dispatch/src/server.c:1122
   68: #10 0x7fc777a7f7f6 in qd_server_run 
/home/travis/build/apache/qpid-dispatch/src/server.c:1484
   68: #11 0x402111 in main_process 
/home/travis/build/apache/qpid-dispatch/router/src/main.c:113
   68: #12 0x401d68 in main 
/home/travis/build/apache/qpid-dispatch/router/src/main.c:367
   68: #13 0x7fc7762ad82f in __libc_start_main 
(/lib/x86_64-linux-gnu/libc.so.6+0x2082f)
   68: 
   68: -
   68: Suppressions used:
   68:   count  bytes template
   68:   1 24 ^pn_condition$
   68:   1   1536 ^pn_raw_connection$
   68:   1 56 qdr_core_subscribe
   68:  37120 _ctypes_alloc_format_string
   68:   3144 ^pn_object_new$
   68:   1128 ^pn_list$
   68:   1 24 ^pni_record_create$
   68:50023527472 /libpython2.*.so
   68: -
   68: 
   68: SUMMARY: AddressSanitizer: 64 byte(s) leaked in 1 allocation(s).
   ```
   
   ```
   68: Indirect leak of 64 byte(s) in 1 object(s) allocated from:
   68: #0 0x7fa6cfbf7602 in malloc 
(/usr/lib/x86_64-linux-gnu/libasan.so.2+0x98602)
   68: #1 0x7fa6cdda0396  (/lib/x86_64-linux-gnu/libc.so.6+0xeb396)
   68: #2 0x7fa6cdda3d5d in getaddrinfo 
(/lib/x86_64-linux-gnu/libc.so.6+0xeed5d)
   68: #3 0x7fa6cfbd068f  (/usr/lib/x86_64-linux-gnu/libasan.so.2+0x7168f)
   68: #4 0x7fa6cd15198e in pgetaddrinfo 
/home/travis/build/apache/qpid-dispatch/qpid-proton/c/src/proactor/epoll.c:1369
   68: #5 0x7fa6cd15198e in pn_proactor_raw_connect 
/home/travis/build/apache/qpid-dispatch/qpid-proton/c/src/proactor/epoll_raw_connection.c:193
   68: #6 0x7fa6cf30ccd5 in _do_reconnect 
/home/travis/build/apache/qpid-dispatch/src/adaptors/http1/http1_server.c:449
   68: #7 0x7fa6cf4f35b3 in qd_timer_visit 
/home/travis/build/apache/qpid-dispatch/src/timer.c:201
   68: #8 0x7fa6cf4e621e in handle 
/home/travis/build/apache/qpid-dispatch/src/server.c:1008
   68: #9 0x7fa6cf4e771a in thread_run 
/home/travis/build/apache/qpid-dispatch/src/server.c:1122
   68: #10 0x7fa6cf3c6d91 in _thread_init 
/home/travis/build/apache/qpid-dispatch/src/posix/threading.c:172
   68: #11 0x7fa6ced936b9 in start_thread 
(/lib/x86_64-linux-gnu/libpthread.so.0+0x76b9)
   68: 
   68: -
   68: Suppressions used:
   68:   count  bytes template
   68:   4 96 ^pn_condition$
   68:   4   6144 ^pn_raw_connection$
   68:   1 56 qdr_core_subscribe
   68:  37120 _ctypes_alloc_format_string
   68:   6336 ^pn_string_grow$
   68:  24   1128 ^pn_object_new$
   68:   4512 ^pn_list$
   68:   7168 ^pni_record_create$
   68:49993525904 /libpython2.*.so
   68: -
   68: 
   68: SUMMARY: AddressSanitizer: 64 byte(s) leaked in 1 allocation(s).
   ```



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[jira] [Commented] (DISPATCH-1962) Make LSan suppressions more targeted and specific (within reason)

2021-02-21 Thread ASF GitHub Bot (Jira)



[ 
https://issues.apache.org/jira/browse/DISPATCH-1962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17287914#comment-17287914
 ] 

ASF GitHub Bot commented on DISPATCH-1962:
--

codecov-io commented on pull request #1048:
URL: https://github.com/apache/qpid-dispatch/pull/1048#issuecomment-782837844


   # 
[Codecov](https://codecov.io/gh/apache/qpid-dispatch/pull/1048?src=pr=h1) 
Report
   > Merging 
[#1048](https://codecov.io/gh/apache/qpid-dispatch/pull/1048?src=pr=desc) 
(4c8b55c) into 
[master](https://codecov.io/gh/apache/qpid-dispatch/commit/f07975ee4bde676438d226d885e0eb78937f99b4?el=desc)
 (f07975e) will **decrease** coverage by `0.03%`.
   > The diff coverage is `n/a`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/qpid-dispatch/pull/1048/graphs/tree.svg?width=650=150=pr=rk2Cgd27pP)](https://codecov.io/gh/apache/qpid-dispatch/pull/1048?src=pr=tree)
   
   ```diff
   @@Coverage Diff @@
   ##   master#1048  +/-   ##
   ==
   - Coverage   82.48%   82.45%   -0.04% 
   ==
 Files 111  111  
 Lines   2742527425  
   ==
   - Hits2262122612   -9 
   - Misses   4804 4813   +9 
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/qpid-dispatch/pull/1048?src=pr=tree) | 
Coverage Δ | |
   |---|---|---|
   | 
[src/router\_core/transfer.c](https://codecov.io/gh/apache/qpid-dispatch/pull/1048/diff?src=pr=tree#diff-c3JjL3JvdXRlcl9jb3JlL3RyYW5zZmVyLmM=)
 | `91.81% <0.00%> (-0.65%)` | :arrow_down: |
   | 
[src/router\_core/connections.c](https://codecov.io/gh/apache/qpid-dispatch/pull/1048/diff?src=pr=tree#diff-c3JjL3JvdXRlcl9jb3JlL2Nvbm5lY3Rpb25zLmM=)
 | `88.81% <0.00%> (-0.30%)` | :arrow_down: |
   | 
[src/adaptors/http1/http1\_server.c](https://codecov.io/gh/apache/qpid-dispatch/pull/1048/diff?src=pr=tree#diff-c3JjL2FkYXB0b3JzL2h0dHAxL2h0dHAxX3NlcnZlci5j)
 | `84.50% <0.00%> (-0.30%)` | :arrow_down: |
   | 
[src/router\_node.c](https://codecov.io/gh/apache/qpid-dispatch/pull/1048/diff?src=pr=tree#diff-c3JjL3JvdXRlcl9ub2RlLmM=)
 | `94.03% <0.00%> (-0.11%)` | :arrow_down: |
   | 
[src/message.c](https://codecov.io/gh/apache/qpid-dispatch/pull/1048/diff?src=pr=tree#diff-c3JjL21lc3NhZ2UuYw==)
 | `87.07% <0.00%> (-0.08%)` | :arrow_down: |
   | 
[src/adaptors/http1/http1\_codec.c](https://codecov.io/gh/apache/qpid-dispatch/pull/1048/diff?src=pr=tree#diff-c3JjL2FkYXB0b3JzL2h0dHAxL2h0dHAxX2NvZGVjLmM=)
 | `85.28% <0.00%> (+0.12%)` | :arrow_up: |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/qpid-dispatch/pull/1048?src=pr=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/qpid-dispatch/pull/1048?src=pr=footer).
 Last update 
[f07975e...4c8b55c](https://codecov.io/gh/apache/qpid-dispatch/pull/1048?src=pr=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Make LSan suppressions more targeted and specific (within reason)
> -
>
> Key: DISPATCH-1962
> URL: https://issues.apache.org/jira/browse/DISPATCH-1962
> Project: Qpid Dispatch
>  Issue Type: Test
>Affects Versions: 1.15.0
>Reporter: Jiri Daněk
>Priority: Major
> Attachments: dispatch_leaking_agent.png
>
>
> There are some unfortunate suppressions in {{tests/lsan.supp}} at this moment:
> {code}
> leak:*libwebsockets*
> {code}
> This is way too broad. It suppresses leaks in Dispatch files, since it 
> matches e.g. {{src/http-libwebsockets.c}}.
> The stars at the beginning and end are actually assumed implicitly. If you do 
> not want substring match, you have to do ^foo$. LSan suppression format is 
> simplistic, very unlike Valgrind's.
> {code}
> leak:run_unit_tests.c
> {code}
> Same thing, any leaks revealed by running unit_tests get suppressed. This 
> suppression suppresses all leak traces that include run_unit_tests.c anywhere 
> in the stack. What't the point of running such tests under leak detector, 
> then?
> {code}
> leak:run_unit_tests.c
> leak:^libqpid-proton.so$
> {code}
> Same thing. The patterns suppress all leaks that include Python (or Proton) 
> anywhere in the stacktrace. That

[jira] [Commented] (DISPATCH-1962) Make LSan suppressions more targeted and specific (within reason)

2021-02-21 Thread ASF GitHub Bot (Jira)



[ 
https://issues.apache.org/jira/browse/DISPATCH-1962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17287904#comment-17287904
 ] 

ASF GitHub Bot commented on DISPATCH-1962:
--

jiridanek commented on pull request #1048:
URL: https://github.com/apache/qpid-dispatch/pull/1048#issuecomment-782834418


   Needed to suppress more:
   
   ```
   67: Indirect leak of 32 byte(s) in 1 object(s) allocated from:
   67: #0 0x7f538ada2602 in malloc 
(/usr/lib/x86_64-linux-gnu/libasan.so.2+0x98602)
   67: #1 0x7f5388512802 in pni_mem_allocate 
/home/travis/build/apache/qpid-dispatch/qpid-proton/c/src/core/memory.c:270
   67: #2 0x7f5388512802 in pn_buffer 
/home/travis/build/apache/qpid-dispatch/qpid-proton/c/src/core/buffer.c:47
   ```
   
   ```
   67: Indirect leak of 32 byte(s) in 1 object(s) allocated from:
   67: #0 0x4964dd in malloc 
(/home/travis/build/apache/qpid-dispatch/build/router/qdrouterd+0x4964dd)
   67: #1 0x7f997fa1362d in pni_mem_allocate 
/home/travis/build/apache/qpid-dispatch/qpid-proton/c/src/core/memory.c:270:71
   67: #2 0x7f997fa1362d in pn_buffer 
/home/travis/build/apache/qpid-dispatch/qpid-proton/c/src/core/buffer.c:47:38
   67: #3 0x7f997fa1362d in pni_data_intern_node.llvm.909393496252145 
/home/travis/build/apache/qpid-dispatch/qpid-proton/c/src/core/codec.c:475:17
   ```
   
   ```
   63: Indirect leak of 16 byte(s) in 1 object(s) allocated from:
   63: #0 0x4976ed in malloc 
(/home/travis/build/apache/qpid-dispatch/build/tests/test-sender+0x4976ed)
   63: #1 0x7f3b9dbf942a in pni_mem_suballocate 
/home/travis/build/apache/qpid-dispatch/qpid-proton/c/src/core/memory.c:274:88
   63: #2 0x7f3b9dbf942a in pn_stringn 
/home/travis/build/apache/qpid-dispatch/qpid-proton/c/src/core/object/string.c:114:28
   63: #3 0x7f3b9dbf942a in pn_string 
/home/travis/build/apache/qpid-dispatch/qpid-proton/c/src/core/object/string.c:103:10
   63: #4 0x7f3b9dbf942a in pni_init_default_logger 
/home/travis/build/apache/qpid-dispatch/qpid-proton/c/src/core/logger.c:119:32
   ```
   
   Lets see if its clean result now.



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Make LSan suppressions more targeted and specific (within reason)
> -
>
> Key: DISPATCH-1962
> URL: https://issues.apache.org/jira/browse/DISPATCH-1962
> Project: Qpid Dispatch
>  Issue Type: Test
>Affects Versions: 1.15.0
>Reporter: Jiri Daněk
>Priority: Major
> Attachments: dispatch_leaking_agent.png
>
>
> There are some unfortunate suppressions in {{tests/lsan.supp}} at this moment:
> {code}
> leak:*libwebsockets*
> {code}
> This is way too broad. It suppresses leaks in Dispatch files, since it 
> matches e.g. {{src/http-libwebsockets.c}}.
> The stars at the beginning and end are actually assumed implicitly. If you do 
> not want substring match, you have to do ^foo$. LSan suppression format is 
> simplistic, very unlike Valgrind's.
> {code}
> leak:run_unit_tests.c
> {code}
> Same thing, any leaks revealed by running unit_tests get suppressed. This 
> suppression suppresses all leak traces that include run_unit_tests.c anywhere 
> in the stack. What't the point of running such tests under leak detector, 
> then?
> {code}
> leak:run_unit_tests.c
> leak:^libqpid-proton.so$
> {code}
> Same thing. The patterns suppress all leaks that include Python (or Proton) 
> anywhere in the stacktrace. That means there are huge blind spots where 
> dispatch leaks can hide. This is a weakness of the lsan.supp syntax (Valgrind 
> suppressions can be much more targeted and discerning).
> h3. Python leaks
> Leaks are known and there is ongoing effort to fight them: 
> https://bugs.python.org/issue1635741 (https://bugs.python.org/issue25302) and 
> https://www.python.org/dev/peps/pep-3121
> Here's valgrind suppression file from somebody who actually investigated the 
> Python leaks and identified the harmless ones: 
> https://github.com/libgit2/pygit2/blob/master/misc/valgrind-python.supp
> One example of a hidden leak in dispatch, which is revealed by making Python 
> suppressions more targetted:
> {code}
> 9: Direct leak of 56 byte(s) in 1 object(s) allocated from:
> 9: #0 0x7f78a3606e8f in __interceptor_malloc 
> (/nix/store/g40sl3zh3nv52vj0mrl4iki5iphh5ika-gcc-10.2.0-lib/lib/libasan.so.6+0xace8f)
> 9: #1 0x7f78a2d64afb in qd_malloc ../include/qpid/dispatch/ctools.h:229
> 9: #2 0x7f78a2d657da in qdr_core_subscribe 
> ../src/router_core/route_tables.c:149
> 9: #3 0x7f78a2c83072 in IoAdapter_init

[jira] [Commented] (DISPATCH-1962) Make LSan suppressions more targeted and specific (within reason)

2021-02-21 Thread ASF GitHub Bot (Jira)



[ 
https://issues.apache.org/jira/browse/DISPATCH-1962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17287902#comment-17287902
 ] 

ASF GitHub Bot commented on DISPATCH-1962:
--

jiridanek commented on pull request #1052:
URL: https://github.com/apache/qpid-dispatch/pull/1052#issuecomment-782832205


   Here's a GHA run with PR #1051 merged in. There are some leaks from Proton 
then, but the IoAdapter leak is fixed. 
https://github.com/jiridanek/qpid-dispatch/actions/runs/585158010



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Make LSan suppressions more targeted and specific (within reason)
> -
>
> Key: DISPATCH-1962
> URL: https://issues.apache.org/jira/browse/DISPATCH-1962
> Project: Qpid Dispatch
>  Issue Type: Test
>Affects Versions: 1.15.0
>Reporter: Jiri Daněk
>Priority: Major
> Attachments: dispatch_leaking_agent.png
>
>
> There are some unfortunate suppressions in {{tests/lsan.supp}} at this moment:
> {code}
> leak:*libwebsockets*
> {code}
> This is way too broad. It suppresses leaks in Dispatch files, since it 
> matches e.g. {{src/http-libwebsockets.c}}.
> The stars at the beginning and end are actually assumed implicitly. If you do 
> not want substring match, you have to do ^foo$. LSan suppression format is 
> simplistic, very unlike Valgrind's.
> {code}
> leak:run_unit_tests.c
> {code}
> Same thing, any leaks revealed by running unit_tests get suppressed. This 
> suppression suppresses all leak traces that include run_unit_tests.c anywhere 
> in the stack. What't the point of running such tests under leak detector, 
> then?
> {code}
> leak:run_unit_tests.c
> leak:^libqpid-proton.so$
> {code}
> Same thing. The patterns suppress all leaks that include Python (or Proton) 
> anywhere in the stacktrace. That means there are huge blind spots where 
> dispatch leaks can hide. This is a weakness of the lsan.supp syntax (Valgrind 
> suppressions can be much more targeted and discerning).
> h3. Python leaks
> Leaks are known and there is ongoing effort to fight them: 
> https://bugs.python.org/issue1635741 (https://bugs.python.org/issue25302) and 
> https://www.python.org/dev/peps/pep-3121
> Here's valgrind suppression file from somebody who actually investigated the 
> Python leaks and identified the harmless ones: 
> https://github.com/libgit2/pygit2/blob/master/misc/valgrind-python.supp
> One example of a hidden leak in dispatch, which is revealed by making Python 
> suppressions more targetted:
> {code}
> 9: Direct leak of 56 byte(s) in 1 object(s) allocated from:
> 9: #0 0x7f78a3606e8f in __interceptor_malloc 
> (/nix/store/g40sl3zh3nv52vj0mrl4iki5iphh5ika-gcc-10.2.0-lib/lib/libasan.so.6+0xace8f)
> 9: #1 0x7f78a2d64afb in qd_malloc ../include/qpid/dispatch/ctools.h:229
> 9: #2 0x7f78a2d657da in qdr_core_subscribe 
> ../src/router_core/route_tables.c:149
> 9: #3 0x7f78a2c83072 in IoAdapter_init ../src/python_embedded.c:711
> 9: #4 0x7f78a2353a6c in type_call 
> (/nix/store/r85nxfnwiv45nbmf5yb60jj8ajim4m7w-python3-3.8.5/lib/libpython3.8.so.1.0+0x165a6c)
> {code}
> The problem is in
> {code}
> class Agent:
> ...
> def activate(self, address):
> ...
> self.io = IoAdapter(self.receive, address, 'L', '0', 
> TREATMENT_ANYCAST_CLOSEST)
> {code}
> IoAdapter refers to Agent (through the bound method reference self.receive) 
> and Agent refers to IoAdapter (through property self.io). Since IoAdapter is 
> implemented in C and does not implement support for Python's cyclic GC, there 
> is no way to break the cycle.
> Heap dump in attachment. The bound method is at the top of the picture. 
> (Ignore the Mock objects, I was trying to simplify the picture while not 
> getting crashes due to too much meddling).
> h3. Random observations
> It is possible to build special Debug build of Python, which has tools to 
> detect leaks, asserts to prevent negative refcounts, etc. 
> https://pythonextensionpatterns.readthedocs.io/en/latest/debugging/debug_python.html#debug-version-of-python-memory-alloc-label
> Use the following to detect python leaks (instead of valgrind)
> https://docs.python.org/3/library/tracemalloc.html
> Use https://pypi.org/project/objgraph (with graphviz) to view heap object 
> trees. The following renders the picture as a png under /tmp and prints the 
> path to stdout.
> {code}
> int ret = PyRun_SimpleString("import objgraph; 
> objgraph.show_backrefs(config.global_agent, max_depth=10)\n\n");
> PyErr_PrintEx(0);
> assert(ret == 0);
> {code}



--
This message

[jira] [Commented] (DISPATCH-1962) Make LSan suppressions more targeted and specific (within reason)

2021-02-20 Thread ASF GitHub Bot (Jira)



[ 
https://issues.apache.org/jira/browse/DISPATCH-1962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17287822#comment-17287822
 ] 

ASF GitHub Bot commented on DISPATCH-1962:
--

codecov-io commented on pull request #1052:
URL: https://github.com/apache/qpid-dispatch/pull/1052#issuecomment-782762950


   # 
[Codecov](https://codecov.io/gh/apache/qpid-dispatch/pull/1052?src=pr=h1) 
Report
   > Merging 
[#1052](https://codecov.io/gh/apache/qpid-dispatch/pull/1052?src=pr=desc) 
(78fb8d1) into 
[master](https://codecov.io/gh/apache/qpid-dispatch/commit/844f995a580ca630afe951ea7c5071bb02bc33f6?el=desc)
 (844f995) will **increase** coverage by `0.14%`.
   > The diff coverage is `100.00%`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/qpid-dispatch/pull/1052/graphs/tree.svg?width=650=150=pr=rk2Cgd27pP)](https://codecov.io/gh/apache/qpid-dispatch/pull/1052?src=pr=tree)
   
   ```diff
   @@Coverage Diff @@
   ##   master#1052  +/-   ##
   ==
   + Coverage   82.48%   82.63%   +0.14% 
   ==
 Files 111  111  
 Lines   2742527439  +14 
   ==
   + Hits2262122673  +52 
   + Misses   4804 4766  -38 
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/qpid-dispatch/pull/1052?src=pr=tree) | 
Coverage Δ | |
   |---|---|---|
   | 
[src/dispatch.c](https://codecov.io/gh/apache/qpid-dispatch/pull/1052/diff?src=pr=tree#diff-c3JjL2Rpc3BhdGNoLmM=)
 | `83.80% <100.00%> (+0.07%)` | :arrow_up: |
   | 
[src/python\_embedded.c](https://codecov.io/gh/apache/qpid-dispatch/pull/1052/diff?src=pr=tree#diff-c3JjL3B5dGhvbl9lbWJlZGRlZC5j)
 | `69.55% <100.00%> (+1.12%)` | :arrow_up: |
   | 
[src/router\_node.c](https://codecov.io/gh/apache/qpid-dispatch/pull/1052/diff?src=pr=tree#diff-c3JjL3JvdXRlcl9ub2RlLmM=)
 | `94.03% <100.00%> (ø)` | |
   | 
[src/router\_pynode.c](https://codecov.io/gh/apache/qpid-dispatch/pull/1052/diff?src=pr=tree#diff-c3JjL3JvdXRlcl9weW5vZGUuYw==)
 | `88.34% <100.00%> (+0.37%)` | :arrow_up: |
   | 
[tests/core\_timer\_test.c](https://codecov.io/gh/apache/qpid-dispatch/pull/1052/diff?src=pr=tree#diff-dGVzdHMvY29yZV90aW1lcl90ZXN0LmM=)
 | `90.47% <100.00%> (+0.09%)` | :arrow_up: |
   | 
[src/router\_core/transfer.c](https://codecov.io/gh/apache/qpid-dispatch/pull/1052/diff?src=pr=tree#diff-c3JjL3JvdXRlcl9jb3JlL3RyYW5zZmVyLmM=)
 | `91.81% <0.00%> (-0.44%)` | :arrow_down: |
   | 
[src/router\_core/delivery.c](https://codecov.io/gh/apache/qpid-dispatch/pull/1052/diff?src=pr=tree#diff-c3JjL3JvdXRlcl9jb3JlL2RlbGl2ZXJ5LmM=)
 | `93.12% <0.00%> (-0.39%)` | :arrow_down: |
   | 
[src/router\_core/connections.c](https://codecov.io/gh/apache/qpid-dispatch/pull/1052/diff?src=pr=tree#diff-c3JjL3JvdXRlcl9jb3JlL2Nvbm5lY3Rpb25zLmM=)
 | `88.61% <0.00%> (-0.20%)` | :arrow_down: |
   | 
[src/message.c](https://codecov.io/gh/apache/qpid-dispatch/pull/1052/diff?src=pr=tree#diff-c3JjL21lc3NhZ2UuYw==)
 | `87.07% <0.00%> (-0.08%)` | :arrow_down: |
   | ... and [6 
more](https://codecov.io/gh/apache/qpid-dispatch/pull/1052/diff?src=pr=tree-more)
 | |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/qpid-dispatch/pull/1052?src=pr=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/qpid-dispatch/pull/1052?src=pr=footer).
 Last update 
[844f995...78fb8d1](https://codecov.io/gh/apache/qpid-dispatch/pull/1052?src=pr=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Make LSan suppressions more targeted and specific (within reason)
> -
>
> Key: DISPATCH-1962
> URL: https://issues.apache.org/jira/browse/DISPATCH-1962
> Project: Qpid Dispatch
>  Issue Type: Test
>Affects Versions: 1.15.0
>Reporter: Jiri Daněk
>Priority: Major
> Attachments: dispatch_leaking_agent.png
>
>
> There are some unfortunate suppressions in {{tests/lsan.supp}} at this moment:
> {code}
> leak:*libwebsockets*
> {code}
> This is way too broad. It suppresses leaks in Dispatch files, since it 
> matches e.g. {{src/http-libwebsockets.c}}.
> The stars at the beginning and end are actually assumed implicitly. If you

[jira] [Commented] (DISPATCH-1962) Make LSan suppressions more targeted and specific (within reason)

2021-02-20 Thread ASF GitHub Bot (Jira)



[ 
https://issues.apache.org/jira/browse/DISPATCH-1962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17287815#comment-17287815
 ] 

ASF GitHub Bot commented on DISPATCH-1962:
--

jiridanek opened a new pull request #1052:
URL: https://github.com/apache/qpid-dispatch/pull/1052


   This PR depends on PRs to fix leaks in `qdr_subscription_t` 
(https://github.com/apache/qpid-dispatch/pull/1051) and `qdr_core_t` 
(https://github.com/apache/qpid-dispatch/pull/1049). The former is still WIP, 
so I won't carry the commit here, because it would make review harder.
   
   The original problem is
   
   ```
   9: Direct leak of 56 byte(s) in 1 object(s) allocated from:
   9: #0 0x7f78a3606e8f in __interceptor_malloc 
(/nix/store/g40sl3zh3nv52vj0mrl4iki5iphh5ika-gcc-10.2.0-lib/lib/libasan.so.6+0xace8f)
   9: #1 0x7f78a2d64afb in qd_malloc ../include/qpid/dispatch/ctools.h:229
   9: #2 0x7f78a2d657da in qdr_core_subscribe 
../src/router_core/route_tables.c:149
   9: #3 0x7f78a2c83072 in IoAdapter_init ../src/python_embedded.c:711
   9: #4 0x7f78a2353a6c in type_call 
(/nix/store/r85nxfnwiv45nbmf5yb60jj8ajim4m7w-python3-3.8.5/lib/libpython3.8.so.1.0+0x165a6c)
   ```
   
   The solution consists of
   1) implementing support for GC in IoAdapter so that Python can free the 
cycle described in the Jira.
   2) Fixing Decreffing so that Python does free
   3) Calling PyGC_Collect() at an opportune place where finalizers can be 
called without accessing previously already freed memory.



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Make LSan suppressions more targeted and specific (within reason)
> -
>
> Key: DISPATCH-1962
> URL: https://issues.apache.org/jira/browse/DISPATCH-1962
> Project: Qpid Dispatch
>  Issue Type: Test
>Affects Versions: 1.15.0
>Reporter: Jiri Daněk
>Priority: Major
> Attachments: dispatch_leaking_agent.png
>
>
> There are some unfortunate suppressions in {{tests/lsan.supp}} at this moment:
> {code}
> leak:*libwebsockets*
> {code}
> This is way too broad. It suppresses leaks in Dispatch files, since it 
> matches e.g. {{src/http-libwebsockets.c}}.
> The stars at the beginning and end are actually assumed implicitly. If you do 
> not want substring match, you have to do ^foo$. LSan suppression format is 
> simplistic, very unlike Valgrind's.
> {code}
> leak:run_unit_tests.c
> {code}
> Same thing, any leaks revealed by running unit_tests get suppressed. This 
> suppression suppresses all leak traces that include run_unit_tests.c anywhere 
> in the stack. What't the point of running such tests under leak detector, 
> then?
> {code}
> leak:run_unit_tests.c
> leak:^libqpid-proton.so$
> {code}
> Same thing. The patterns suppress all leaks that include Python (or Proton) 
> anywhere in the stacktrace. That means there are huge blind spots where 
> dispatch leaks can hide. This is a weakness of the lsan.supp syntax (Valgrind 
> suppressions can be much more targeted and discerning).
> h3. Python leaks
> Leaks are known and there is ongoing effort to fight them: 
> https://bugs.python.org/issue1635741 (https://bugs.python.org/issue25302) and 
> https://www.python.org/dev/peps/pep-3121
> Here's valgrind suppression file from somebody who actually investigated the 
> Python leaks and identified the harmless ones: 
> https://github.com/libgit2/pygit2/blob/master/misc/valgrind-python.supp
> One example of a hidden leak in dispatch, which is revealed by making Python 
> suppressions more targetted:
> {code}
> 9: Direct leak of 56 byte(s) in 1 object(s) allocated from:
> 9: #0 0x7f78a3606e8f in __interceptor_malloc 
> (/nix/store/g40sl3zh3nv52vj0mrl4iki5iphh5ika-gcc-10.2.0-lib/lib/libasan.so.6+0xace8f)
> 9: #1 0x7f78a2d64afb in qd_malloc ../include/qpid/dispatch/ctools.h:229
> 9: #2 0x7f78a2d657da in qdr_core_subscribe 
> ../src/router_core/route_tables.c:149
> 9: #3 0x7f78a2c83072 in IoAdapter_init ../src/python_embedded.c:711
> 9: #4 0x7f78a2353a6c in type_call 
> (/nix/store/r85nxfnwiv45nbmf5yb60jj8ajim4m7w-python3-3.8.5/lib/libpython3.8.so.1.0+0x165a6c)
> {code}
> The problem is in
> {code}
> class Agent:
> ...
> def activate(self, address):
> ...
> self.io = IoAdapter(self.receive, address, 'L', '0', 
> TREATMENT_ANYCAST_CLOSEST)
> {code}
> IoAdapter refers to Agent (through the bound method reference self.receive) 
> and Agent refers to IoAdapter (through property self.io). Since IoAdapter is 
> implemented in C and does not implement support for Python's

[jira] [Commented] (DISPATCH-1962) Make LSan suppressions more targeted and specific (within reason)

2021-02-20 Thread ASF GitHub Bot (Jira)



[ 
https://issues.apache.org/jira/browse/DISPATCH-1962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17287808#comment-17287808
 ] 

ASF GitHub Bot commented on DISPATCH-1962:
--

codecov-io commented on pull request #1051:
URL: https://github.com/apache/qpid-dispatch/pull/1051#issuecomment-782757013


   # 
[Codecov](https://codecov.io/gh/apache/qpid-dispatch/pull/1051?src=pr=h1) 
Report
   > Merging 
[#1051](https://codecov.io/gh/apache/qpid-dispatch/pull/1051?src=pr=desc) 
(6b51140) into 
[master](https://codecov.io/gh/apache/qpid-dispatch/commit/844f995a580ca630afe951ea7c5071bb02bc33f6?el=desc)
 (844f995) will **increase** coverage by `0.02%`.
   > The diff coverage is `100.00%`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/qpid-dispatch/pull/1051/graphs/tree.svg?width=650=150=pr=rk2Cgd27pP)](https://codecov.io/gh/apache/qpid-dispatch/pull/1051?src=pr=tree)
   
   ```diff
   @@Coverage Diff @@
   ##   master#1051  +/-   ##
   ==
   + Coverage   82.48%   82.50%   +0.02% 
   ==
 Files 111  111  
 Lines   2742527427   +2 
   ==
   + Hits2262122629   +8 
   + Misses   4804 4798   -6 
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/qpid-dispatch/pull/1051?src=pr=tree) | 
Coverage Δ | |
   |---|---|---|
   | 
[src/router\_core/modules/mobile\_sync/mobile.c](https://codecov.io/gh/apache/qpid-dispatch/pull/1051/diff?src=pr=tree#diff-c3JjL3JvdXRlcl9jb3JlL21vZHVsZXMvbW9iaWxlX3N5bmMvbW9iaWxlLmM=)
 | `92.21% <100.00%> (+0.03%)` | :arrow_up: |
   | 
[src/router\_core/router\_core.c](https://codecov.io/gh/apache/qpid-dispatch/pull/1051/diff?src=pr=tree#diff-c3JjL3JvdXRlcl9jb3JlL3JvdXRlcl9jb3JlLmM=)
 | `86.20% <100.00%> (ø)` | |
   | 
[...router\_core/modules/edge\_router/link\_route\_proxy.c](https://codecov.io/gh/apache/qpid-dispatch/pull/1051/diff?src=pr=tree#diff-c3JjL3JvdXRlcl9jb3JlL21vZHVsZXMvZWRnZV9yb3V0ZXIvbGlua19yb3V0ZV9wcm94eS5j)
 | `78.69% <0.00%> (-4.15%)` | :arrow_down: |
   | 
[src/router\_core/modules/edge\_router/edge\_mgmt.c](https://codecov.io/gh/apache/qpid-dispatch/pull/1051/diff?src=pr=tree#diff-c3JjL3JvdXRlcl9jb3JlL21vZHVsZXMvZWRnZV9yb3V0ZXIvZWRnZV9tZ210LmM=)
 | `84.15% <0.00%> (-1.00%)` | :arrow_down: |
   | 
[src/router\_core/delivery.c](https://codecov.io/gh/apache/qpid-dispatch/pull/1051/diff?src=pr=tree#diff-c3JjL3JvdXRlcl9jb3JlL2RlbGl2ZXJ5LmM=)
 | `93.12% <0.00%> (-0.39%)` | :arrow_down: |
   | 
[src/iterator.c](https://codecov.io/gh/apache/qpid-dispatch/pull/1051/diff?src=pr=tree#diff-c3JjL2l0ZXJhdG9yLmM=)
 | `89.29% <0.00%> (-0.19%)` | :arrow_down: |
   | 
[src/message.c](https://codecov.io/gh/apache/qpid-dispatch/pull/1051/diff?src=pr=tree#diff-c3JjL21lc3NhZ2UuYw==)
 | `87.07% <0.00%> (-0.08%)` | :arrow_down: |
   | 
[src/router\_node.c](https://codecov.io/gh/apache/qpid-dispatch/pull/1051/diff?src=pr=tree#diff-c3JjL3JvdXRlcl9ub2RlLmM=)
 | `94.13% <0.00%> (+0.10%)` | :arrow_up: |
   | 
[src/router\_core/forwarder.c](https://codecov.io/gh/apache/qpid-dispatch/pull/1051/diff?src=pr=tree#diff-c3JjL3JvdXRlcl9jb3JlL2ZvcndhcmRlci5j)
 | `93.04% <0.00%> (+0.39%)` | :arrow_up: |
   | 
[src/router\_core/transfer.c](https://codecov.io/gh/apache/qpid-dispatch/pull/1051/diff?src=pr=tree#diff-c3JjL3JvdXRlcl9jb3JlL3RyYW5zZmVyLmM=)
 | `92.67% <0.00%> (+0.43%)` | :arrow_up: |
   | ... and [2 
more](https://codecov.io/gh/apache/qpid-dispatch/pull/1051/diff?src=pr=tree-more)
 | |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/qpid-dispatch/pull/1051?src=pr=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/qpid-dispatch/pull/1051?src=pr=footer).
 Last update 
[844f995...6b51140](https://codecov.io/gh/apache/qpid-dispatch/pull/1051?src=pr=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Make LSan suppressions more targeted and specific (within reason)
> -
>
> Key: DISPATCH-1962
> URL: https://issues.apache.org/jira/browse/DISPATCH-1962
> Project: Qpid Dispatch
>  Issue Type: Test
>Affects Versions: 1.15.0
>Reporter: Jiri Daněk
>

[jira] [Commented] (DISPATCH-1962) Make LSan suppressions more targeted and specific (within reason)

2021-02-20 Thread ASF GitHub Bot (Jira)



[ 
https://issues.apache.org/jira/browse/DISPATCH-1962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17287804#comment-17287804
 ] 

ASF GitHub Bot commented on DISPATCH-1962:
--

jiridanek opened a new pull request #1051:
URL: https://github.com/apache/qpid-dispatch/pull/1051


   This PR is simply to show that "a solution exists". I have no idea whether I 
am on the right track here, or not.
   
   The original problem is
   
   ```
   14: 2021-02-20 20:54:08.332325 +0100 SERVER (info) [C2] Closing connection 
on shutdown (../src/server.c:1377)
   14: 
   14: 
   14: 
   14: Router QDR.A output file:
   14: 
   14: 
   14: =
   14: ==28365==ERROR: LeakSanitizer: detected memory leaks
   14: 
   14: Direct leak of 56 byte(s) in 1 object(s) allocated from:
   14: #0 0x7f8503a7ce8f in __interceptor_malloc 
(/nix/store/g40sl3zh3nv52vj0mrl4iki5iphh5ika-gcc-10.2.0-lib/lib/libasan.so.6+0xace8f)
   14: #1 0x7f85034afd42 in qd_malloc ../include/qpid/dispatch/ctools.h:229
   14: #2 0x7f85034afd42 in qdr_core_subscribe 
../src/router_core/route_tables.c:149
   14: #3 0x7f85034f4962 in qcm_mobile_sync_init_CT 
../src/router_core/modules/mobile_sync/mobile.c:919
   14: #4 0x7f85034a79d1 in qdr_modules_init 
../src/router_core/router_core_thread.c:120
   14: #5 0x7f850348e889 in qdr_core_setup_init 
../src/router_core/router_core.c:60
   14: #6 0x7f8503491971 in qdr_core ../src/router_core/router_core.c:116
   14: #7 0x7f8503505949 in qd_router_setup_late ../src/router_node.c:2071
   14: #8 0x7f84fee08abc in ffi_call_unix64 
(/nix/store/m8y5mz1f0al3rg3b56rq5bza49jjxnc0-libffi-3.3/lib/libffi.so.7+0x7abc)
   14: #9 0x7fff297a166f  ([stack]+0x1e66f)
   14: 
   14: Direct leak of 56 byte(s) in 1 object(s) allocated from:
   14: #0 0x7f8503a7ce8f in __interceptor_malloc 
(/nix/store/g40sl3zh3nv52vj0mrl4iki5iphh5ika-gcc-10.2.0-lib/lib/libasan.so.6+0xace8f)
   14: #1 0x7f85034afd42 in qd_malloc ../include/qpid/dispatch/ctools.h:229
   14: #2 0x7f85034afd42 in qdr_core_subscribe 
../src/router_core/route_tables.c:149
   14: #3 0x7f85034f49d3 in qcm_mobile_sync_init_CT 
../src/router_core/modules/mobile_sync/mobile.c:921
   14: #4 0x7f85034a79d1 in qdr_modules_init 
../src/router_core/router_core_thread.c:120
   14: #5 0x7f850348e889 in qdr_core_setup_init 
../src/router_core/router_core.c:60
   14: #6 0x7f8503491971 in qdr_core ../src/router_core/router_core.c:116
   14: #7 0x7f8503505949 in qd_router_setup_late ../src/router_node.c:2071
   14: #8 0x7f84fee08abc in ffi_call_unix64 
(/nix/store/m8y5mz1f0al3rg3b56rq5bza49jjxnc0-libffi-3.3/lib/libffi.so.7+0x7abc)
   14: #9 0x7fff297a166f  ([stack]+0x1e66f)
   14: 
   14: -
   14: Suppressions used:
   14:   count  bytes template
   14:   4224 ^IoAdapter_init$
   14:   4   2560 ^_PyObject_Realloc
   14: 558 883628 ^PyObject_Malloc$
   14:   1 32 ^PyThread_allocate_lock$
   14:   4   9897 ^PyMem_Malloc$
   14:   1840 ^_PyObject_GC_Resize$
   14:   1   1184 ^list_append$
   14: -
   14: 
   14: SUMMARY: AddressSanitizer: 112 byte(s) leaked in 2 allocation(s).
   ```
   
   If I want to free this, I have to reorder things in `qdr_core_free` so that 
the finalizers don't try to use something that was already freed:
   
   If I don't move `qdr_modules_finalize` before the `discard any left over 
actions` block, I get the following, because the lock is already freed
   
   ```
   16: qdrouterd: ../src/posix/threading.c:58: sys_mutex_lock: Assertion 
`result == 0' failed.
   ```
   
   And if I run `qcm_edge_router_final_CT` 
(`src/router_core/modules/edge_router/module.c:59`} before 
`qdrc_endpoint_do_cleanup_CT` (`src/router_core/core_link_endpoint.c:241`), I 
get
   
   ```
   16: ==16076==ERROR: AddressSanitizer: heap-use-after-free on address 
0x60c035f0 at pc 0x7ff5117d60bc bp 0x7fffd457e1f0 sp 0x7fffd457e1e8
   16: READ of size 8 at 0x60c035f0 thread T0
   16: #0 0x7ff5117d60bb in qdrc_endpoint_do_cleanup_CT 
../src/router_core/core_link_endpoint.c:241
   16: #1 0x7ff51182e3ea in qdr_core_free 
../src/router_core/router_core.c:230
   16: #2 0x7ff511891ecb in qd_router_free ../src/router_node.c:2108
   16: #3 0x7ff511732229 in qd_dispatch_free ../src/dispatch.c:368
   16: #4 0x402625 in main_process ../router/src/main.c:117
   16: #5 0x403f4b in main ../router/src/main.c:367
   16: #6 0x7ff510382cbc in __libc_start_main 
(/nix/store/q53f5birhik4dxg3q3r2g5f324n7r5mc-glibc-2.31-74/lib/libc.so.6+0x23cbc)
   16: #7 0x402419 in _start 
(/home/jdanek/repos/qpid/qpid-dispatch/cmake-build-debug/router/qdrouterd+0x402419)
   16: 
   16: 0x60c035f0 is

[jira] [Commented] (DISPATCH-1962) Make LSan suppressions more targeted and specific (within reason)

2021-02-20 Thread ASF GitHub Bot (Jira)



[ 
https://issues.apache.org/jira/browse/DISPATCH-1962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17287790#comment-17287790
 ] 

ASF GitHub Bot commented on DISPATCH-1962:
--

codecov-io commented on pull request #1049:
URL: https://github.com/apache/qpid-dispatch/pull/1049#issuecomment-782744008


   # 
[Codecov](https://codecov.io/gh/apache/qpid-dispatch/pull/1049?src=pr=h1) 
Report
   > Merging 
[#1049](https://codecov.io/gh/apache/qpid-dispatch/pull/1049?src=pr=desc) 
(933aba6) into 
[master](https://codecov.io/gh/apache/qpid-dispatch/commit/844f995a580ca630afe951ea7c5071bb02bc33f6?el=desc)
 (844f995) will **decrease** coverage by `0.00%`.
   > The diff coverage is `100.00%`.
   
   [![Impacted file tree 
graph](https://codecov.io/gh/apache/qpid-dispatch/pull/1049/graphs/tree.svg?width=650=150=pr=rk2Cgd27pP)](https://codecov.io/gh/apache/qpid-dispatch/pull/1049?src=pr=tree)
   
   ```diff
   @@Coverage Diff @@
   ##   master#1049  +/-   ##
   ==
   - Coverage   82.48%   82.47%   -0.01% 
   ==
 Files 111  111  
 Lines   2742527426   +1 
   ==
   - Hits2262122620   -1 
   - Misses   4804 4806   +2 
   ```
   
   
   | [Impacted 
Files](https://codecov.io/gh/apache/qpid-dispatch/pull/1049?src=pr=tree) | 
Coverage Δ | |
   |---|---|---|
   | 
[tests/core\_timer\_test.c](https://codecov.io/gh/apache/qpid-dispatch/pull/1049/diff?src=pr=tree#diff-dGVzdHMvY29yZV90aW1lcl90ZXN0LmM=)
 | `90.47% <100.00%> (+0.09%)` | :arrow_up: |
   | 
[...router\_core/modules/edge\_router/link\_route\_proxy.c](https://codecov.io/gh/apache/qpid-dispatch/pull/1049/diff?src=pr=tree#diff-c3JjL3JvdXRlcl9jb3JlL21vZHVsZXMvZWRnZV9yb3V0ZXIvbGlua19yb3V0ZV9wcm94eS5j)
 | `78.69% <0.00%> (-4.15%)` | :arrow_down: |
   | 
[src/router\_core/modules/edge\_router/edge\_mgmt.c](https://codecov.io/gh/apache/qpid-dispatch/pull/1049/diff?src=pr=tree#diff-c3JjL3JvdXRlcl9jb3JlL21vZHVsZXMvZWRnZV9yb3V0ZXIvZWRnZV9tZ210LmM=)
 | `84.15% <0.00%> (-1.00%)` | :arrow_down: |
   | 
[src/router\_core/transfer.c](https://codecov.io/gh/apache/qpid-dispatch/pull/1049/diff?src=pr=tree#diff-c3JjL3JvdXRlcl9jb3JlL3RyYW5zZmVyLmM=)
 | `91.81% <0.00%> (-0.44%)` | :arrow_down: |
   | 
[src/router\_core/delivery.c](https://codecov.io/gh/apache/qpid-dispatch/pull/1049/diff?src=pr=tree#diff-c3JjL3JvdXRlcl9jb3JlL2RlbGl2ZXJ5LmM=)
 | `93.12% <0.00%> (-0.39%)` | :arrow_down: |
   | 
[src/router\_node.c](https://codecov.io/gh/apache/qpid-dispatch/pull/1049/diff?src=pr=tree#diff-c3JjL3JvdXRlcl9ub2RlLmM=)
 | `94.03% <0.00%> (ø)` | |
   | 
[src/server.c](https://codecov.io/gh/apache/qpid-dispatch/pull/1049/diff?src=pr=tree#diff-c3JjL3NlcnZlci5j)
 | `86.78% <0.00%> (+0.10%)` | :arrow_up: |
   | 
[src/parse.c](https://codecov.io/gh/apache/qpid-dispatch/pull/1049/diff?src=pr=tree#diff-c3JjL3BhcnNlLmM=)
 | `87.06% <0.00%> (+0.21%)` | :arrow_up: |
   | 
[src/adaptors/http1/http1\_server.c](https://codecov.io/gh/apache/qpid-dispatch/pull/1049/diff?src=pr=tree#diff-c3JjL2FkYXB0b3JzL2h0dHAxL2h0dHAxX3NlcnZlci5j)
 | `84.79% <0.00%> (+0.29%)` | :arrow_up: |
   | 
[src/router\_core/forwarder.c](https://codecov.io/gh/apache/qpid-dispatch/pull/1049/diff?src=pr=tree#diff-c3JjL3JvdXRlcl9jb3JlL2ZvcndhcmRlci5j)
 | `93.04% <0.00%> (+0.39%)` | :arrow_up: |
   | ... and [1 
more](https://codecov.io/gh/apache/qpid-dispatch/pull/1049/diff?src=pr=tree-more)
 | |
   
   --
   
   [Continue to review full report at 
Codecov](https://codecov.io/gh/apache/qpid-dispatch/pull/1049?src=pr=continue).
   > **Legend** - [Click here to learn 
more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute  (impact)`, `ø = not affected`, `? = missing data`
   > Powered by 
[Codecov](https://codecov.io/gh/apache/qpid-dispatch/pull/1049?src=pr=footer).
 Last update 
[844f995...933aba6](https://codecov.io/gh/apache/qpid-dispatch/pull/1049?src=pr=lastupdated).
 Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Make LSan suppressions more targeted and specific (within reason)
> -
>
> Key: DISPATCH-1962
> URL: https://issues.apache.org/jira/browse/DISPATCH-1962
> Project: Qpid Dispatch
>  Issue Type: Test
>Affects Versions: 1.15.0
>Reporter: Jiri Daněk
>Priority: Major
> Attachments:

[jira] [Commented] (DISPATCH-1962) Make LSan suppressions more targeted and specific (within reason)

2021-02-20 Thread ASF GitHub Bot (Jira)



[ 
https://issues.apache.org/jira/browse/DISPATCH-1962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17287785#comment-17287785
 ] 

ASF GitHub Bot commented on DISPATCH-1962:
--

jiridanek opened a new pull request #1049:
URL: https://github.com/apache/qpid-dispatch/pull/1049


   As explained on the Jira, suppressing all leaks from unittests is kind of 
silly, because it can hide leaks in production code which would be otherwise 
easy to diagnose, if unittests could reveal them. The leak here is only in test 
code and it is easy to fix.
   
   ```
   9: Test Case thread_tests.test_condition: PASS
   9: 
   9: =
   9: ==19457==ERROR: LeakSanitizer: detected memory leaks
   9: 
   9: Direct leak of 1160 byte(s) in 1 object(s) allocated from:
   9: #0 0x7fe6d5751e8f in __interceptor_malloc 
(/nix/store/g40sl3zh3nv52vj0mrl4iki5iphh5ika-gcc-10.2.0-lib/lib/libasan.so.6+0xace8f)
   9: #1 0x40d9fe in qd_malloc ../include/qpid/dispatch/ctools.h:229
   9: #2 0x40d9fe in test_core_timer ../tests/core_timer_test.c:37
   9: #3 0x40e4b2 in core_timer_tests ../tests/core_timer_test.c:183
   9: #4 0x406c83 in main ../tests/run_unit_tests.c:69
   9: #5 0x7fe6d3ccbcbc in __libc_start_main 
(/nix/store/q53f5birhik4dxg3q3r2g5f324n7r5mc-glibc-2.31-74/lib/libc.so.6+0x23cbc)
   9: 
   9: -
   9: Suppressions used:
   9:   count  bytes template
   9:   1 56 qdr_core_subscribe
   9: 572 916269 ^_PyMem_DebugRawAlloc$
   9:   5   3296 ^_PyMem_DebugRawRealloc$
   9: -
   9: 
   9: SUMMARY: AddressSanitizer: 1160 byte(s) leaked in 1 allocation(s).
   ```



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Make LSan suppressions more targeted and specific (within reason)
> -
>
> Key: DISPATCH-1962
> URL: https://issues.apache.org/jira/browse/DISPATCH-1962
> Project: Qpid Dispatch
>  Issue Type: Test
>Affects Versions: 1.15.0
>Reporter: Jiri Daněk
>Priority: Major
> Attachments: dispatch_leaking_agent.png
>
>
> There are some unfortunate suppressions in {{tests/lsan.supp}} at this moment:
> {code}
> leak:*libwebsockets*
> {code}
> This is way too broad. It suppresses leaks in Dispatch files, since it 
> matches e.g. {{src/http-libwebsockets.c}}.
> The stars at the beginning and end are actually assumed implicitly. If you do 
> not want substring match, you have to do ^foo$. LSan suppression format is 
> simplistic, very unlike Valgrind's.
> {code}
> leak:run_unit_tests.c
> {code}
> Same thing, any leaks revealed by running unit_tests get suppressed. This 
> suppression suppresses all leak traces that include run_unit_tests.c anywhere 
> in the stack. What't the point of running such tests under leak detector, 
> then?
> {code}
> leak:run_unit_tests.c
> leak:^libqpid-proton.so$
> {code}
> Same thing. The patterns suppress all leaks that include Python (or Proton) 
> anywhere in the stacktrace. That means there are huge blind spots where 
> dispatch leaks can hide. This is a weakness of the lsan.supp syntax (Valgrind 
> suppressions can be much more targeted and discerning).
> h3. Python leaks
> Leaks are known and there is ongoing effort to fight them: 
> https://bugs.python.org/issue1635741 (https://bugs.python.org/issue25302) and 
> https://www.python.org/dev/peps/pep-3121
> Here's valgrind suppression file from somebody who actually investigated the 
> Python leaks and identified the harmless ones: 
> https://github.com/libgit2/pygit2/blob/master/misc/valgrind-python.supp
> One example of a hidden leak in dispatch, which is revealed by making Python 
> suppressions more targetted:
> {code}
> 9: Direct leak of 56 byte(s) in 1 object(s) allocated from:
> 9: #0 0x7f78a3606e8f in __interceptor_malloc 
> (/nix/store/g40sl3zh3nv52vj0mrl4iki5iphh5ika-gcc-10.2.0-lib/lib/libasan.so.6+0xace8f)
> 9: #1 0x7f78a2d64afb in qd_malloc ../include/qpid/dispatch/ctools.h:229
> 9: #2 0x7f78a2d657da in qdr_core_subscribe 
> ../src/router_core/route_tables.c:149
> 9: #3 0x7f78a2c83072 in IoAdapter_init ../src/python_embedded.c:711
> 9: #4 0x7f78a2353a6c in type_call 
> (/nix/store/r85nxfnwiv45nbmf5yb60jj8ajim4m7w-python3-3.8.5/lib/libpython3.8.so.1.0+0x165a6c)
> {code}
> The problem is in
> {code}
> class Agent:
> ...
> def activate(self, address):
> ...
> self.io = IoAdapter(self.receive, address, 'L', '0',

[jira] [Commented] (DISPATCH-1962) Make LSan suppressions more targeted and specific (within reason)

2021-02-20 Thread ASF GitHub Bot (Jira)



[ 
https://issues.apache.org/jira/browse/DISPATCH-1962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17287731#comment-17287731
 ] 

ASF GitHub Bot commented on DISPATCH-1962:
--

jiridanek opened a new pull request #1048:
URL: https://github.com/apache/qpid-dispatch/pull/1048


   This introduces build option -DSANITIZE_3RD_PARTY, defaulted to OFF. The 
reason is that targeted suppressions only work when the .so libraries are 
installed with debug symbols, which is not always the case.
   
   Leaks are suppressed more granularly. I have already ideas about fixing some 
of them, but I wanted to make their suppressions explicit in this way first.



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Make LSan suppressions more targeted and specific (within reason)
> -
>
> Key: DISPATCH-1962
> URL: https://issues.apache.org/jira/browse/DISPATCH-1962
> Project: Qpid Dispatch
>  Issue Type: Test
>Affects Versions: 1.15.0
>Reporter: Jiri Daněk
>Priority: Major
> Attachments: dispatch_leaking_agent.png
>
>
> There are some unfortunate suppressions in {{tests/lsan.supp}} at this moment:
> {code}
> leak:*libwebsockets*
> {code}
> This is way too broad. It suppresses leaks in Dispatch files, since it 
> matches e.g. {{src/http-libwebsockets.c}}.
> The stars at the beginning and end are actually assumed implicitly. If you do 
> not want substring match, you have to do ^foo$. LSan suppression format is 
> simplistic, very unlike Valgrind's.
> {code}
> leak:run_unit_tests.c
> {code}
> Same thing, any leaks revealed by running unit_tests get suppressed. This 
> suppression suppresses all leak traces that include run_unit_tests.c anywhere 
> in the stack. What't the point of running such tests under leak detector, 
> then?
> {code}
> leak:run_unit_tests.c
> leak:^libqpid-proton.so$
> {code}
> Same thing. The patterns suppress all leaks that include Python (or Proton) 
> anywhere in the stacktrace. That means there are huge blind spots where 
> dispatch leaks can hide. This is a weakness of the lsan.supp syntax (Valgrind 
> suppressions can be much more targeted and discerning).
> h3. Python leaks
> Leaks are known and there is ongoing effort to fight them: 
> https://bugs.python.org/issue1635741 (https://bugs.python.org/issue25302) and 
> https://www.python.org/dev/peps/pep-3121
> Here's valgrind suppression file from somebody who actually investigated the 
> Python leaks and identified the harmless ones: 
> https://github.com/libgit2/pygit2/blob/master/misc/valgrind-python.supp
> One example of a hidden leak in dispatch, which is revealed by making Python 
> suppressions more targetted:
> {code}
> 9: Direct leak of 56 byte(s) in 1 object(s) allocated from:
> 9: #0 0x7f78a3606e8f in __interceptor_malloc 
> (/nix/store/g40sl3zh3nv52vj0mrl4iki5iphh5ika-gcc-10.2.0-lib/lib/libasan.so.6+0xace8f)
> 9: #1 0x7f78a2d64afb in qd_malloc ../include/qpid/dispatch/ctools.h:229
> 9: #2 0x7f78a2d657da in qdr_core_subscribe 
> ../src/router_core/route_tables.c:149
> 9: #3 0x7f78a2c83072 in IoAdapter_init ../src/python_embedded.c:711
> 9: #4 0x7f78a2353a6c in type_call 
> (/nix/store/r85nxfnwiv45nbmf5yb60jj8ajim4m7w-python3-3.8.5/lib/libpython3.8.so.1.0+0x165a6c)
> {code}
> The problem is in
> {code}
> class Agent:
> ...
> def activate(self, address):
> ...
> self.io = IoAdapter(self.receive, address, 'L', '0', 
> TREATMENT_ANYCAST_CLOSEST)
> {code}
> IoAdapter refers to Agent (through the bound method reference self.receive) 
> and Agent refers to IoAdapter (through property self.io). Since IoAdapter is 
> implemented in C and does not implement support for Python's cyclic GC, there 
> is no way to break the cycle.
> Heap dump in attachment. The bound method is at the top of the picture. 
> (Ignore the Mock objects, I was trying to simplify the picture while not 
> getting crashes due to too much meddling).
> h3. Random observations
> It is possible to build special Debug build of Python, which has tools to 
> detect leaks, asserts to prevent negative refcounts, etc. 
> https://pythonextensionpatterns.readthedocs.io/en/latest/debugging/debug_python.html#debug-version-of-python-memory-alloc-label
> Use the following to detect python leaks (instead of valgrind)
> https://docs.python.org/3/library/tracemalloc.html
> Use https://pypi.org/project/objgraph (with graphviz) to view heap object 
> trees. The following renders the picture as a png under /tmp and prints the 
> path to stdout.
> {code}
> int ret =

66 matches

Mail list logo