CORPORATE
Hi Andi
It's just a Segmentation fault error. We are calling the wrapped function in
lch_calc_api.py. I cannot really decode much from the stacktrace to be honest.
Fatal Python error: Segmentation fault
Thread 0x00007fad78686700 (most recent call first):
File
"/home/runner/code/ref/simm/.venv/lib/python3.9/site-packages/execnet/gateway_base.py",
line 534 in read
File
"/home/runner/code/ref/simm/.venv/lib/python3.9/site-packages/execnet/gateway_base.py",
line 567 in from_io
File
"/home/runner/code/ref/simm/.venv/lib/python3.9/site-packages/execnet/gateway_base.py",
line 1160 in _thread_receiver
File
"/home/runner/code/ref/simm/.venv/lib/python3.9/site-packages/execnet/gateway_base.py",
line 341 in run
File
"/home/runner/code/ref/simm/.venv/lib/python3.9/site-packages/execnet/gateway_base.py",
line 411 in _perform_spawn
Current thread 0x00007fad78d9b740 (most recent call first):
File
"/home/runner/code/ref/simm/src/cpty_risk/common/external_apis/lch_calc_api.py",
line 126 in smart_vm
File
"/home/runner/code/ref/simm/src/cpty_risk/common/external_apis/lch_calc_api.py",
line 133 in _get_smart_factory
File
"/home/runner/code/ref/simm/src/cpty_risk/common/external_apis/lch_calc_api.py",
line 118 in smart_factory
File
"/home/runner/code/ref/simm/src/cpty_risk/common/external_apis/lch_calc_api.py",
line 175 in get_forex_smart_factory
File
"/home/runner/code/ref/simm/src/cpty_risk/common/external_apis/calc_api_data.py",
line 592 in get_raw_calc_api_data
File
"/home/runner/code/ref/simm/tests/functional/cpty_risk_tools/external_api/calc/test_calc_im.py",
line 393 in test_get_raw_calc_api_data_frep
File
"/home/runner/code/ref/simm/.venv/lib/python3.9/site-packages/_pytest/python.py",
line 183 in pytest_pyfunc_call
File
"/home/runner/code/ref/simm/.venv/lib/python3.9/site-packages/pluggy/_callers.py",
line 103 in _multicall
File
"/home/runner/code/ref/simm/.venv/lib/python3.9/site-packages/pluggy/_manager.py",
line 120 in _hookexec
File
"/home/runner/code/ref/simm/.venv/lib/python3.9/site-packages/pluggy/_hooks.py",
line 513 in __call__
File
"/home/runner/code/ref/simm/.venv/lib/python3.9/site-packages/_pytest/python.py",
line 1641 in runtest
File
"/home/runner/code/ref/simm/.venv/lib/python3.9/site-packages/_pytest/runner.py",
line 162 in pytest_runtest_call
File
"/home/runner/code/ref/simm/.venv/lib/python3.9/site-packages/pluggy/_callers.py",
line 103 in _multicall
File
"/home/runner/code/ref/simm/.venv/lib/python3.9/site-packages/pluggy/_manager.py",
line 120 in _hookexec
File
"/home/runner/code/ref/simm/.venv/lib/python3.9/site-packages/pluggy/_hooks.py",
line 513 in __call__
File
"/home/runner/code/ref/simm/.venv/lib/python3.9/site-packages/_pytest/runner.py",
line 255 in <lambda>
File
"/home/runner/code/ref/simm/.venv/lib/python3.9/site-packages/_pytest/runner.py",
line 311 in from_call
File
"/home/runner/code/ref/simm/.venv/lib/python3.9/site-packages/_pytest/runner.py",
line 254 in call_runtest_hook
File
"/home/runner/code/ref/simm/.venv/lib/python3.9/site-packages/_pytest/runner.py",
line 215 in call_and_report
File
"/home/runner/code/ref/simm/.venv/lib/python3.9/site-packages/_pytest/runner.py",
line 126 in runtestprotocol
File
"/home/runner/code/ref/simm/.venv/lib/python3.9/site-packages/_pytest/runner.py",
line 109 in pytest_runtest_protocol
File
"/home/runner/code/ref/simm/.venv/lib/python3.9/site-packages/pluggy/_callers.py",
line 103 in _multicall
File
"/home/runner/code/ref/simm/.venv/lib/python3.9/site-packages/pluggy/_manager.py",
line 120 in _hookexec
File
"/home/runner/code/ref/simm/.venv/lib/python3.9/site-packages/pluggy/_hooks.py",
line 513 in __call__
File
"/home/runner/code/ref/simm/.venv/lib/python3.9/site-packages/xdist/remote.py",
line 110 in run_one_test
File
"/home/runner/code/ref/simm/.venv/lib/python3.9/site-packages/xdist/remote.py",
line 91 in pytest_runtestloop
File
"/home/runner/code/ref/simm/.venv/lib/python3.9/site-packages/pluggy/_callers.py",
line 103 in _multicall
File
"/home/runner/code/ref/simm/.venv/lib/python3.9/site-packages/pluggy/_manager.py",
line 120 in _hookexec
File
"/home/runner/code/ref/simm/.venv/lib/python3.9/site-packages/pluggy/_hooks.py",
line 513 in __call__
File
"/home/runner/code/ref/simm/.venv/lib/python3.9/site-packages/_pytest/main.py",
line 323 in _main
File
"/home/runner/code/ref/simm/.venv/lib/python3.9/site-packages/_pytest/main.py",
line 269 in wrap_session
File
"/home/runner/code/ref/simm/.venv/lib/python3.9/site-packages/_pytest/main.py",
line 316 in pytest_cmdline_main
File
"/home/runner/code/ref/simm/.venv/lib/python3.9/site-packages/pluggy/_callers.py",
line 103 in _multicall
File
"/home/runner/code/ref/simm/.venv/lib/python3.9/site-packages/pluggy/_manager.py",
line 120 in _hookexec
File
"/home/runner/code/ref/simm/.venv/lib/python3.9/site-packages/pluggy/_hooks.py",
line 513 in __call__
File
"/home/runner/code/ref/simm/.venv/lib/python3.9/site-packages/xdist/remote.py",
line 291 in <module>
File
"/home/runner/code/ref/simm/.venv/lib/python3.9/site-packages/execnet/gateway_base.py",
line 1291 in executetask
File
"/home/runner/code/ref/simm/.venv/lib/python3.9/site-packages/execnet/gateway_base.py",
line 341 in run
File
"/home/runner/code/ref/simm/.venv/lib/python3.9/site-packages/execnet/gateway_base.py",
line 411 in _perform_spawn
File
"/home/runner/code/ref/simm/.venv/lib/python3.9/site-packages/execnet/gateway_base.py",
line 389 in integrate_as_primary_thread
File
"/home/runner/code/ref/simm/.venv/lib/python3.9/site-packages/execnet/gateway_base.py",
line 1273 in serve
File
"/home/runner/code/ref/simm/.venv/lib/python3.9/site-packages/execnet/gateway_base.py",
line 1806 in serve
File "<string>", line 8 in <module>
File "<string>", line 1 in <module>
From: Andi Vajda <va...@apache.org>
Sent: 12 December 2024 23:40
To: Teh, Chee Yong <cheeyong....@lseg.com>
Cc: Andi Vajda <va...@apache.org>; pylucene-dev@lucene.apache.org
Subject: Re: Multithreading/Multiprocess of JCC
*** EXTERNAL EMAIL ***
On Thu, 12 Dec 2024, Teh, Chee Yong wrote:
> CORPORATE
>
> Hi Andi,
>
> I have question related to running a wrapped function in a
> multithreading/multiprocess environment in python.
>
> We have a java library and we wrapped it using JCC so we can call it from
> python.
>
> We have a list of data that we need to call a wrapped function for each of
> the data in the list.
>
> Everything works fine when we call it in a single threaded env but the
> process is slow as it's sequential. Now we are trying to speed up the
> process by calling this function on multiple thread.
>
> I tried to call the initVM on the "main" thread then on each of new thread
> I will call attachCurrentThread (begin) and detachCurrentThread (end).
> This doesn't seem to work properly as we getting some weird VM issue.
Please explain what "some weird VM issue" is. How can I help you otherwise ?
> What is the best way to do multithread using JCC?
What you're doing seems correct, iirc.
Andi..
>
> Thanks,
>
> Chee Yong
> ------------------------------------------------------------------------------------------------------------
>
> Please read these warnings and restrictions:
> This e-mail transmission is strictly confidential and intended solely for the
> ordinary user of the e-mail address to which it was addressed. It may contain
> legally privileged and/or CONFIDENTIAL information.
> The unauthorised use, disclosure, distribution and/or copying of this e-mail
> or any information it contains is prohibited and could, in certain
> circumstances, constitute a criminal offence.
> If you have received this e-mail in error or are not an intended recipient
> please inform London Stock Exchange Group ("LSEG") immediately by return
> e-mail or telephone 020 7797 1000.
> LSEG may collect, process and retain your personal information for its
> business purposes. For more information please see our Privacy Policy.
> We advise that in keeping with good computing practice the recipient of this
> e-mail should ensure that it is virus free. We do not accept responsibility
> for any virus that may be transferred by way of this e-mail.
> E-mail may be susceptible to data corruption, interception and unauthorised
> amendment, and we do not accept liability for any such corruption,
> interception or amendment or any consequences thereof.
> Calls to London Stock Exchange Group may be recorded to enable LSEG to carry
> out its regulatory responsibilities.
> For more details on the LSEG group of companies click here
> London Stock Exchange Group plc
> 10 Paternoster Square
> London
> EC4M 7LS
> Registered in England and Wales No 05369106
>
> ------------------------------------------------------------------------------------------------------------
>
------------------------------------------------------------------------------------------------------------
Please read these warnings and restrictions:
This e-mail transmission is strictly confidential and intended solely for the
ordinary user of the e-mail address to which it was addressed. It may contain
legally privileged and/or CONFIDENTIAL information.
The unauthorised use, disclosure, distribution and/or copying of this e-mail or
any information it contains is prohibited and could, in certain circumstances,
constitute a criminal offence.
If you have received this e-mail in error or are not an intended recipient
please inform London Stock Exchange Group (“LSEG”) immediately by return e-mail
or telephone 020 7797 1000.
LSEG may collect, process and retain your personal information for its business
purposes. For more information please see our Privacy Policy.
We advise that in keeping with good computing practice the recipient of this
e-mail should ensure that it is virus free. We do not accept responsibility for
any virus that may be transferred by way of this e-mail.
E-mail may be susceptible to data corruption, interception and unauthorised
amendment, and we do not accept liability for any such corruption, interception
or amendment or any consequences thereof.
Calls to London Stock Exchange Group may be recorded to enable LSEG to carry
out its regulatory responsibilities.
For more details on the LSEG group of companies click here
London Stock Exchange Group plc
10 Paternoster Square
London
EC4M 7LS
Registered in England and Wales No 05369106
------------------------------------------------------------------------------------------------------------
