Re: Apache2::Request and other modules

[Mithun Bhattacharya]

Apache2::Request is the base and you would at least need that to consume a
request.

The rest are optional and expand on what $r provides. You include them only
if you are using something they are providing.

On Tue, May 13, 2025, 8:37 PM  wrote:

> Hello
>
> In my modperl handler, I only use Apache2::Request module.
> What relations are there between Apache2::Request and the following
> modules?
>
> use Apache2::RequestRec ();
> use Apache2::RequestIO ();
> use Apache2::Connection ();
> use APR::Table ();
>
>
> Should I include them in startup.pl as well as Apache2::Request?
>
> Thanks.
>

Re: Apache server returns early before process is complete

[Joseph He]

Many thanks to you all.

I am still trying to figure out the issue. Let me re-explain the problem I
experienced with some details.

The environment is Ubuntu 22.04, Apache2, ModPerl.
I run a Http::request with LWP::UserAgent, the server receives the request
and starts to process it.
But it takes much longer due to a stalled SFTP call to the remote server,
the Apache server timeout and sends back failure, meanwhile,* the server
actually is still trying to process this request*.
On the calling side, after receiving the failure status, it initiates
another http::request and the load balancer redirects this call to another
server for processing.
It turns out this same http::request is processed twice.

On my production server the timeout happens at 300 seconds mark. On my QA
and Dev server, the timeout happens at 600 seconds. I have not changed
anything on my production server yet.
But on my QA and DEV servers, I have tried to change Timeout in
apache2.conf, have tried to add Timeout to the virtualhost config, also
have tried to add SetPerlEnv MOD_PERL_TIMEOUT to the virtualhost config,
none of them change the timeout behavior of my QA and DEV servers.

So what exactly controls the Timeout? I am totally lost.

Cheers,
Joe


On Wed, Apr 23, 2025 at 5:17 PM Mithun Bhattacharya 
wrote:

> Okay agreed that is a valid time out basically it is saying that a client
> has established tcp/ip connection but has not put its request either a get
> put or a post
>
> On Wed, Apr 23, 2025, 3:38 PM Joseph He  wrote:
>
>> On Apache2 doc, I found this. How does this timeout work? It looks like
>> it can only wait for 300 seconds before failing a request.
>>
>> https://httpd.apache.org/docs/2.0/mod/core.html#timeout
>> Description:
>>  
>> Amount
>> of time the server will wait for certain events before failing a request
>> Syntax:
>> 
>> TimeOut seconds
>> Default:
>>  TimeOut
>> 300
>> Context:
>>  server
>> config, virtual host
>> Status:
>>  Core
>> Module:
>>  core
>>
>> The TimeOut directive currently defines the amount of time Apache will
>> wait for three things:
>>
>>1. The total amount of time it takes to receive a GET request.
>>2. The amount of time between receipt of TCP packets on a POST or PUT
>>request.
>>3. The amount of time between ACKs on transmissions of TCP packets in
>>responses.
>>
>> We plan on making these separately configurable at some point down the
>> road. The timer used to default to 1200 before 1.2, but has been lowered to
>> 300 which is still far more than necessary in most situations. It is not
>> set any lower by default because there may still be odd places in the code
>> where the timer is not reset when a packet is sent.
>>
>> On Wed, Apr 23, 2025 at 3:07 PM Mithun Bhattacharya 
>> wrote:
>>
>>> You configure timeout at the client side. Apache is at the server side.
>>> Server doesn't have a concept of time it could take days to run and not
>>> care.
>>>
>>> mod_perl code is where you are sending the http return status to make
>>> sure the client doesn't timeout waiting for the server to respond.
>>>
>>> On Wed, Apr 23, 2025, 2:19 PM Joseph He 
>>> wrote:
>>>
 Thanks, all.
 Is that Apache timeout controlled by its configuration "Timeout"?
 I don't think it has anything to do with modPerl. Am I missing
 something?
 Thanks.

 On Wed, Apr 23, 2025 at 1:41 PM Mithun Bhattacharya 
 wrote:

> Timeout happens because of how we handle the request. Timeout is
> basically no response came back. Why that happens is because we think we
> want to have a correct response. Unfortunately for long running requests
> the correct response shouldn't be via http response code or we face
> situations like this. Instead reply with a 200 OK immediately and then
> provide correct status in the message body. Once a response code/header 
> has
> been sent timeout won't trigger and you could potentially hold the
> connection for hours without a problem.
>
> On Wed, Apr 23, 2025, 9:32 AM Andreas Mock 
> wrote:
>
>> Hi Joseph,
>>
>> your description is very vague, so can only answer on some
>> assumptions:
>>
>> It sounds like a timeout is fired somewhere.
>>
>> Best advice in these situations: Log as many steps as you can. Keep
>> your
>> eyes open on TCP/IP and higher level timeouts.
>>
>> Declare only ONE instance responsible for a retry: Either the app
>> server
>> calling the dispatcher with several tries or the dispatcher trying
>> for
>> himself. N

Re: Apache server returns early before process is complete

[Joseph He]

Andreas,

I add sleep 300 to the server code to simulate the stall SFTP and I set the
server Timeout to be 150.
Then I run curl command with different timeout 25, 50, 100, 150, 200, 250,
the curl command always timeout accordingly. The behavior of curl is
exactly the same as that of changing LWP::UserAgent timeout, which verifies
the timeout option on the client side indeed works as desired.
But no matter how I treak the server side Apache config, it just does not
do anything.

Cheers,
Joe

On Tue, May 13, 2025 at 10:49 AM Andreas Mock  wrote:

> Hallo Joe,
>
> try to reduce the problem.
>
> Make the call to your SFTP-Service via curl or some other http(s) client
> to see whether you get the same timeout. If yes, than the server side is
> closing the connection. If not then you have to investigate the
> LWP::UserAgent part.
>
> Another hint in combination with SSL:
> https://stackoverflow.com/questions/9400068/make-timeout-work-for-lwpuseragent-https
>
> Best regards
> Andreas
>
>
> Am 13.05.2025 um 17:22 schrieb Joseph He:
>
> Andreas, thank you.
>
> On the client side, I set the timeout at LWP::UserAgent request to 600,
> and I can verify that it indeed works on my QA and DEV environment. If I
> change it to 120, then it can timeout at 120.
> So on my production server, the client side receives a timeout from the
> server after 5 minutes, so I still think the server Timeout plays a role
> here. I just don't know what config I can change to test it out.
>
> Joe
>
> On Tue, May 13, 2025 at 10:07 AM Andreas Mock  wrote:
>
>> Hi Joe,
>>
>> when you send a request via LWP::UserAgent to the Server which does the
>> long lasting SFTP calls, then I'm pretty sure that you get a timout in the
>> LWP::UserAgent code.
>>
>> I'm pretty sure the client (LWP::UserAgent) is not waiting long enough
>> for the answer: https://metacpan.org/pod/LWP::UserAgent#timeout
>>
>> After having here a long timeout you have to be sure that the very first
>> client which sent the very first request also waits long enough to let the
>> application server make severals tries, therefore n * timeout.
>>
>> Best wishes
>> Andreas
>>
>>
>> Am 13.05.2025 um 16:46 schrieb Joseph He:
>>
>> Many thanks to you all.
>>
>> I am still trying to figure out the issue. Let me re-explain the problem
>> I experienced with some details.
>>
>> The environment is Ubuntu 22.04, Apache2, ModPerl.
>> I run a Http::request with LWP::UserAgent, the server receives the
>> request and starts to process it.
>> But it takes much longer due to a stalled SFTP call to the remote server,
>> the Apache server timeout and sends back failure, meanwhile,* the server
>> actually is still trying to process this request*.
>> On the calling side, after receiving the failure status, it initiates
>> another http::request and the load balancer redirects this call to another
>> server for processing.
>> It turns out this same http::request is processed twice.
>>
>> On my production server the timeout happens at 300 seconds mark. On my QA
>> and Dev server, the timeout happens at 600 seconds. I have not changed
>> anything on my production server yet.
>> But on my QA and DEV servers, I have tried to change Timeout in
>> apache2.conf, have tried to add Timeout to the virtualhost config, also
>> have tried to add SetPerlEnv MOD_PERL_TIMEOUT to the virtualhost config,
>> none of them change the timeout behavior of my QA and DEV servers.
>>
>> So what exactly controls the Timeout? I am totally lost.
>>
>> Cheers,
>> Joe
>>
>>
>> On Wed, Apr 23, 2025 at 5:17 PM Mithun Bhattacharya 
>> wrote:
>>
>>> Okay agreed that is a valid time out basically it is saying that a
>>> client has established tcp/ip connection but has not put its request either
>>> a get put or a post
>>>
>>> On Wed, Apr 23, 2025, 3:38 PM Joseph He 
>>> wrote:
>>>
 On Apache2 doc, I found this. How does this timeout work? It looks like
 it can only wait for 300 seconds before failing a request.

 https://httpd.apache.org/docs/2.0/mod/core.html#timeout
 Description:
  
 Amount
 of time the server will wait for certain events before failing a request
 Syntax:
 
 TimeOut seconds
 Default:
  TimeOut
 300
 Context:
  server
 config, virtual host
 Status:
  Core
 Module:
  core

 The TimeOut directive currently defines the amount of time Apache will
 wait for three things:

1. The total amount of time it takes to receive a GET request.
2. The amount of time between receipt of TCP packets on a POST or
PUT reques

Re: Apache server returns early before process is complete

[Joseph He]

Andreas, thank you.

On the client side, I set the timeout at LWP::UserAgent request to 600, and
I can verify that it indeed works on my QA and DEV environment. If I
change it to 120, then it can timeout at 120.
So on my production server, the client side receives a timeout from the
server after 5 minutes, so I still think the server Timeout plays a role
here. I just don't know what config I can change to test it out.

Joe

On Tue, May 13, 2025 at 10:07 AM Andreas Mock  wrote:

> Hi Joe,
>
> when you send a request via LWP::UserAgent to the Server which does the
> long lasting SFTP calls, then I'm pretty sure that you get a timout in the
> LWP::UserAgent code.
>
> I'm pretty sure the client (LWP::UserAgent) is not waiting long enough for
> the answer: https://metacpan.org/pod/LWP::UserAgent#timeout
>
> After having here a long timeout you have to be sure that the very first
> client which sent the very first request also waits long enough to let the
> application server make severals tries, therefore n * timeout.
>
> Best wishes
> Andreas
>
>
> Am 13.05.2025 um 16:46 schrieb Joseph He:
>
> Many thanks to you all.
>
> I am still trying to figure out the issue. Let me re-explain the problem I
> experienced with some details.
>
> The environment is Ubuntu 22.04, Apache2, ModPerl.
> I run a Http::request with LWP::UserAgent, the server receives the request
> and starts to process it.
> But it takes much longer due to a stalled SFTP call to the remote server,
> the Apache server timeout and sends back failure, meanwhile,* the server
> actually is still trying to process this request*.
> On the calling side, after receiving the failure status, it initiates
> another http::request and the load balancer redirects this call to another
> server for processing.
> It turns out this same http::request is processed twice.
>
> On my production server the timeout happens at 300 seconds mark. On my QA
> and Dev server, the timeout happens at 600 seconds. I have not changed
> anything on my production server yet.
> But on my QA and DEV servers, I have tried to change Timeout in
> apache2.conf, have tried to add Timeout to the virtualhost config, also
> have tried to add SetPerlEnv MOD_PERL_TIMEOUT to the virtualhost config,
> none of them change the timeout behavior of my QA and DEV servers.
>
> So what exactly controls the Timeout? I am totally lost.
>
> Cheers,
> Joe
>
>
> On Wed, Apr 23, 2025 at 5:17 PM Mithun Bhattacharya 
> wrote:
>
>> Okay agreed that is a valid time out basically it is saying that a client
>> has established tcp/ip connection but has not put its request either a get
>> put or a post
>>
>> On Wed, Apr 23, 2025, 3:38 PM Joseph He  wrote:
>>
>>> On Apache2 doc, I found this. How does this timeout work? It looks like
>>> it can only wait for 300 seconds before failing a request.
>>>
>>> https://httpd.apache.org/docs/2.0/mod/core.html#timeout
>>> Description:
>>>  
>>> Amount
>>> of time the server will wait for certain events before failing a request
>>> Syntax:
>>> 
>>> TimeOut seconds
>>> Default:
>>>  TimeOut
>>> 300
>>> Context:
>>>  server
>>> config, virtual host
>>> Status:
>>>  Core
>>> Module:
>>>  core
>>>
>>> The TimeOut directive currently defines the amount of time Apache will
>>> wait for three things:
>>>
>>>1. The total amount of time it takes to receive a GET request.
>>>2. The amount of time between receipt of TCP packets on a POST or
>>>PUT request.
>>>3. The amount of time between ACKs on transmissions of TCP packets
>>>in responses.
>>>
>>> We plan on making these separately configurable at some point down the
>>> road. The timer used to default to 1200 before 1.2, but has been lowered to
>>> 300 which is still far more than necessary in most situations. It is not
>>> set any lower by default because there may still be odd places in the code
>>> where the timer is not reset when a packet is sent.
>>>
>>> On Wed, Apr 23, 2025 at 3:07 PM Mithun Bhattacharya 
>>> wrote:
>>>
 You configure timeout at the client side. Apache is at the server side.
 Server doesn't have a concept of time it could take days to run and not
 care.

 mod_perl code is where you are sending the http return status to make
 sure the client doesn't timeout waiting for the server to respond.

 On Wed, Apr 23, 2025, 2:19 PM Joseph He 
 wrote:

> Thanks, all.
> Is that Apache timeout controlled by its configuration "Timeout"?
> I don't think it has anything to do with modPerl. Am I missing
> something?
> Thanks.
>
> On Wed, Apr 23, 2025 

Re: Apache server returns early before process is complete

[Andreas Mock]

Hallo Joe,

try to reduce the problem.

Make the call to your SFTP-Service via curl or some other http(s) client 
to see whether you get the same timeout. If yes, than the server side is 
closing the connection. If not then you have to investigate the 
LWP::UserAgent part.


Another hint in combination with SSL: 
https://stackoverflow.com/questions/9400068/make-timeout-work-for-lwpuseragent-https


Best regards
Andreas


Am 13.05.2025 um 17:22 schrieb Joseph He:

Andreas, thank you.

On the client side, I set the timeout at LWP::UserAgent request to 
600, and I can verify that it indeed works on my QA and DEV 
environment. If I change it to 120, then it can timeout at 120.
So on my production server, the client side receives a timeout from 
the server after 5 minutes, so I still think the server Timeout plays 
a role here. I just don't know what config I can change to test it out.


Joe

On Tue, May 13, 2025 at 10:07 AM Andreas Mock  wrote:

Hi Joe,

when you send a request via LWP::UserAgent to the Server which
does the long lasting SFTP calls, then I'm pretty sure that you
get a timout in the LWP::UserAgent code.

I'm pretty sure the client (LWP::UserAgent) is not waiting long
enough for the answer: https://metacpan.org/pod/LWP::UserAgent#timeout

After having here a long timeout you have to be sure that the very
first client which sent the very first request also waits long
enough to let the application server make severals tries,
therefore n * timeout.

Best wishes
Andreas


Am 13.05.2025 um 16:46 schrieb Joseph He:

Many thanks to you all.

I am still trying to figure out the issue. Let me re-explain the
problem I experienced with some details.

The environment is Ubuntu 22.04, Apache2, ModPerl.
I run a Http::request with LWP::UserAgent, the server receives
the request and starts to process it.
But it takes much longer due to a stalled SFTP call to the remote
server, the Apache server timeout and sends back failure,
meanwhile,*the server actually is still trying to process this
request*.
On the calling side, after receiving the failure status, it
initiates another http::request and the load balancer redirects
this call to another server for processing.
It turns out this same http::request is processed twice.

On my production server the timeout happens at 300 seconds mark.
On my QA and Dev server, the timeout happens at 600 seconds. I
have not changed anything on my production server yet.
But on my QA and DEV servers, I have tried to change Timeout in
apache2.conf, have tried to add Timeout to the virtualhost
config, also have tried to add SetPerlEnv MOD_PERL_TIMEOUT to the
virtualhost config, none of them change the timeout behavior of
my QA and DEV servers.

So what exactly controls the Timeout? I am totally lost.

Cheers,
Joe


On Wed, Apr 23, 2025 at 5:17 PM Mithun Bhattacharya
 wrote:

Okay agreed that is a valid time out basically it is saying
that a client has established tcp/ip connection but has not
put its request either a get put or a post

On Wed, Apr 23, 2025, 3:38 PM Joseph He
 wrote:

On Apache2 doc, I found this. How does this timeout work?
It looks like it can only wait for 300 seconds before
failing a request.

https://httpd.apache.org/docs/2.0/mod/core.html#timeout
Description:


Amount of time the server will wait for certain events
before failing a request
Syntax:

|TimeOut seconds|
Default:

|TimeOut 300|
Context:

server config, virtual host
Status:

Core
Module:

core

The |TimeOut| directive currently defines the amount of
time Apache will wait for three things:

 1. The total amount of time it takes to receive a GET
request.
 2. The amount of time between receipt of TCP packets on
a POST or PUT request.
 3. The amount of time between ACKs on transmissions of
TCP packets in responses.

We plan on making these separately configurable at some
point down the road. The timer used to default to 1200
before 1.2, but has been lowered to 300 which is still

Re: Return a 200 0K status and send the content of the request later

[Vincent Veyron]

On Tue, 13 May 2025 14:57:15 -0500
Mithun Bhattacharya  wrote:

Hi Mithun,

> Hence if you are writing a mod_perl handler for such an endpoint you return
> success immediately and then start processing the request. In most cases
> this is simply a matter of doing $r->print on the headers followed by print
> of two new lines.
> 

Nice!

Your comment made me revisit mod_perl's documentation, and I found this :

https://perl.apache.org/docs/2.0/user/coding/coding.html#Forcing_HTTP_Response_Headers_Out

The following works, where exportation($r) is the long running sub that builds 
a tar file and returns a page with a link to the file, for the user to click. 
'Building tar file' does get sent first, then the rest of the response 
after completion of the sub.


$r->content_type('text/html; charset=utf-8') ;

$r->print('Building tar file') ;

$r->rflush; # send the headers out

$r->print( exportation($r) ) ;

return Apache2::Const::OK ;


This is great, I can thus send a message to the user, maybe with an estimated 
time of arrival. Thanks for the help.

I have an another question, if I may.

Is there a way I could do :

$r->print('Building tar file') ;

every second or other until the tar file is built? I can't seem to think of one.

My handler calls an sql script that dumps the database with system() :

@args = ('psql', '-f', '/path/to/script/to/export_raw_data.sql', '-v', 
'id_client=' . $r->pnotes('session')->{id_client}, '-v',  'database=' . 
$database, 'postgres') ;

system(@args) == 0 or warn "system @args failed: $!" ;

then waits for the resulting file before proceeding to compress and tar it, 
also with a system() call. 

How can I wrap this in a loop that re-sends the 'Building tar file' message 
until the file exists?



-- 
Bien à vous, Vincent Veyron 

https://marica.fr/ 
Logiciel de suivi des contentieux juridiques, des sinistres d'assurance et des 
contrats

Re: Send a 200 0K return status without the content

[Vincent Veyron]

Sorry, I did not mean to hijack this thread, re-posting.




On Tue, 13 May 2025 19:37:07 +0200
Vincent Veyron  wrote:

> On Wed, 23 Apr 2025 15:06:44 -0500
> Mithun Bhattacharya  wrote:
> 
> Hi Mithun,
> 
> In a recent thread, you wrote :
> 
> > 
> > mod_perl code is where you are sending the http return status to make sure
> > the client doesn't timeout waiting for the server to respond.
> > 
> 
> Could you explain how this is done, I thought one had to send the content 
> with the headers?
> 
> I have a function in my web application which allows users to download the 
> entirety of their data in the form of a postgresql dump, wrapped in a zip 
> file. The files are getting bigger, and take more time to generate and 
> download, so I'm going to hit the timeout one day. 
> 
> 
> -- 
>   Bien à vous, Vincent Veyron
> 
> https://marica.fr
> Logiciel de gestion des contentieux juridiques, des contrats et des sinistres 
> d'assurance


-- 
vv.lists 

Return a 200 0K status and send the content of the request later

[Vincent Veyron]

On Wed, 23 Apr 2025 15:06:44 -0500
Mithun Bhattacharya  wrote:

Hi Mithun,

In a recent thread, you wrote :

> 
> mod_perl code is where you are sending the http return status to make sure
> the client doesn't timeout waiting for the server to respond.
> 

Could you explain how this is done, I thought one had to send the content with 
the headers?

I have a function in my web application which allows users to download the 
entirety of their data in the form of a postgresql dump, wrapped in a zip file. 
The files are getting bigger, and take more time to generate and download, so 
I'm going to hit the timeout one day. 


-- 
Bien à vous, Vincent Veyron

https://marica.fr/
Logiciel de gestion des sinistres assurances, des dossiers contentieux et des 
contrats pour le service juridique

Re: Apache server returns early before process is complete

[Andreas Mock]

Hi Joe,

when you send a request via LWP::UserAgent to the Server which does the 
long lasting SFTP calls, then I'm pretty sure that you get a timout in 
the LWP::UserAgent code.


I'm pretty sure the client (LWP::UserAgent) is not waiting long enough 
for the answer: https://metacpan.org/pod/LWP::UserAgent#timeout


After having here a long timeout you have to be sure that the very first 
client which sent the very first request also waits long enough to let 
the application server make severals tries, therefore n * timeout.


Best wishes
Andreas


Am 13.05.2025 um 16:46 schrieb Joseph He:

Many thanks to you all.

I am still trying to figure out the issue. Let me re-explain the 
problem I experienced with some details.


The environment is Ubuntu 22.04, Apache2, ModPerl.
I run a Http::request with LWP::UserAgent, the server receives the 
request and starts to process it.
But it takes much longer due to a stalled SFTP call to the remote 
server, the Apache server timeout and sends back failure, 
meanwhile,*the server actually is still trying to process this request*.
On the calling side, after receiving the failure status, it initiates 
another http::request and the load balancer redirects this call to 
another server for processing.

It turns out this same http::request is processed twice.

On my production server the timeout happens at 300 seconds mark. On my 
QA and Dev server, the timeout happens at 600 seconds. I have not 
changed anything on my production server yet.
But on my QA and DEV servers, I have tried to change Timeout in 
apache2.conf, have tried to add Timeout to the virtualhost config, 
also have tried to add SetPerlEnv MOD_PERL_TIMEOUT to the virtualhost 
config, none of them change the timeout behavior of my QA and DEV servers.


So what exactly controls the Timeout? I am totally lost.

Cheers,
Joe


On Wed, Apr 23, 2025 at 5:17 PM Mithun Bhattacharya  
wrote:


Okay agreed that is a valid time out basically it is saying that a
client has established tcp/ip connection but has not put its
request either a get put or a post

On Wed, Apr 23, 2025, 3:38 PM Joseph He 
wrote:

On Apache2 doc, I found this. How does this timeout work? It
looks like it can only wait for 300 seconds before failing a
request.

https://httpd.apache.org/docs/2.0/mod/core.html#timeout
Description:

Amount of time the server will wait for certain events before
failing a request
Syntax:

|TimeOut seconds|
Default:

|TimeOut 300|
Context:

server config, virtual host
Status:

Core
Module:

core

The |TimeOut| directive currently defines the amount of time
Apache will wait for three things:

 1. The total amount of time it takes to receive a GET request.
 2. The amount of time between receipt of TCP packets on a
POST or PUT request.
 3. The amount of time between ACKs on transmissions of TCP
packets in responses.

We plan on making these separately configurable at some point
down the road. The timer used to default to 1200 before 1.2,
but has been lowered to 300 which is still far more than
necessary in most situations. It is not set any lower by
default because there may still be odd places in the code
where the timer is not reset when a packet is sent.


On Wed, Apr 23, 2025 at 3:07 PM Mithun Bhattacharya
 wrote:

You configure timeout at the client side. Apache is at the
server side. Server doesn't have a concept of time it
could take days to run and not care.

mod_perl code is where you are sending the http return
status to make sure the client doesn't timeout waiting for
the server to respond.


On Wed, Apr 23, 2025, 2:19 PM Joseph He
 wrote:

Thanks, all.
Is that Apache timeout controlled by its
configuration "Timeout"?
I don't think it has anything to do with modPerl. Am
I missing something?
Thanks.

On Wed, Apr 23, 2025 at 1:41 PM Mithun Bhattacharya
 wrote:

Timeout happens because of how we handle the
request. Timeout is basically no response came
back. Why that 

Send a 200 0K return status without the content

[Vincent Veyron]

On Wed, 23 Apr 2025 15:06:44 -0500
Mithun Bhattacharya  wrote:

Hi Mithun,

In a recent thread, you wrote :

> 
> mod_perl code is where you are sending the http return status to make sure
> the client doesn't timeout waiting for the server to respond.
> 

Could you explain how this is done, I thought one had to send the content with 
the headers?

I have a function in my web application which allows users to download the 
entirety of their data in the form of a postgresql dump, wrapped in a zip file. 
The files are getting bigger, and take more time to generate and download, so 
I'm going to hit the timeout one day. 


-- 
Bien à vous, Vincent Veyron

https://marica.fr
Logiciel de gestion des contentieux juridiques, des contrats et des sinistres 
d'assurance

Re: Apache server returns early before process is complete

[Ian B]

I'm wondering if you were to do an strace -s 2048 -f -p on
the dev server where there's less happening if you may be able to spot
something going on (-f is follow children -s size of line output before
truncated).

Re: Return a 200 0K status and send the content of the request later

[Mithun Bhattacharya]

Http return codes are relevant only for request with fast response time. It
helps the requestor get a quick status 200 is all good, 3xx something
temporary, 4xx client has made some mistakes and 5xx server faulted.

Unfortunately this design doesn't map well to requests which take
significant execution time.

The easiest way to solve this is respond with 200 OK and then the body
tells you whether it succeeded or not. This is not something which is
defined in any place nor is it considered bad.

Hence if you are writing a mod_perl handler for such an endpoint you return
success immediately and then start processing the request. In most cases
this is simply a matter of doing $r->print on the headers followed by print
of two new lines.

On Tue, May 13, 2025, 2:33 PM Vincent Veyron  wrote:

> On Wed, 23 Apr 2025 15:06:44 -0500
> Mithun Bhattacharya  wrote:
>
> Hi Mithun,
>
> In a recent thread, you wrote :
>
> >
> > mod_perl code is where you are sending the http return status to make
> sure
> > the client doesn't timeout waiting for the server to respond.
> >
>
> Could you explain how this is done, I thought one had to send the content
> with the headers?
>
> I have a function in my web application which allows users to download the
> entirety of their data in the form of a postgresql dump, wrapped in a zip
> file. The files are getting bigger, and take more time to generate and
> download, so I'm going to hit the timeout one day.
>
>
> --
> Bien à vous, Vincent Veyron
>
> https://marica.fr/
> Logiciel de gestion des sinistres assurances, des dossiers contentieux et
> des contrats pour le service juridique
>

Apache2::Request and other modules

[ypeng]

Hello

In my modperl handler, I only use Apache2::Request module.
What relations are there between Apache2::Request and the following 
modules?


use Apache2::RequestRec ();
use Apache2::RequestIO ();
use Apache2::Connection ();
use APR::Table ();


Should I include them in startup.pl as well as Apache2::Request?

Thanks.

Re: Return a 200 0K status and send the content of the request later

[Mithun Bhattacharya]

You should do the flush after the ContentType - content_type is header the
H3 is body of the response or at least it should be.

As for your second question, you are looking at a javascript based solution
- this isn't something a server side code can do. Look into a sleep
followed by a window.location.href. This wont help your long running
process though since the refresh will trigger another dump and tar. What
you need to do is send the header and flush followed by the system commands
- once rflush has happened the client side will wait forever without a
timeout.

On Tue, May 13, 2025 at 7:24 PM Vincent Veyron  wrote:

> On Tue, 13 May 2025 14:57:15 -0500
> Mithun Bhattacharya  wrote:
>
> Hi Mithun,
>
> > Hence if you are writing a mod_perl handler for such an endpoint you
> return
> > success immediately and then start processing the request. In most cases
> > this is simply a matter of doing $r->print on the headers followed by
> print
> > of two new lines.
> >
>
> Nice!
>
> Your comment made me revisit mod_perl's documentation, and I found this :
>
>
> https://perl.apache.org/docs/2.0/user/coding/coding.html#Forcing_HTTP_Response_Headers_Out
>
> The following works, where exportation($r) is the long running sub that
> builds a tar file and returns a page with a link to the file, for the user
> to click. 'Building tar file' does get sent first, then the rest
> of the response after completion of the sub.
>
>
> $r->content_type('text/html; charset=utf-8') ;
>
> $r->print('Building tar file') ;
>
> $r->rflush; # send the headers out
>
> $r->print( exportation($r) ) ;
>
> return Apache2::Const::OK ;
>
>
> This is great, I can thus send a message to the user, maybe with an
> estimated time of arrival. Thanks for the help.
>
> I have an another question, if I may.
>
> Is there a way I could do :
>
> $r->print('Building tar file') ;
>
> every second or other until the tar file is built? I can't seem to think
> of one.
>
> My handler calls an sql script that dumps the database with system() :
>
> @args = ('psql', '-f', '/path/to/script/to/export_raw_data.sql', '-v',
> 'id_client=' . $r->pnotes('session')->{id_client}, '-v',  'database=' .
> $database, 'postgres') ;
>
> system(@args) == 0 or warn "system @args failed: $!" ;
>
> then waits for the resulting file before proceeding to compress and tar
> it, also with a system() call.
>
> How can I wrap this in a loop that re-sends the 'Building tar file'
> message until the file exists?
>
>
>
> --
> Bien à vous, Vincent Veyron
>
> https://marica.fr/
> Logiciel de suivi des contentieux juridiques, des sinistres d'assurance et
> des contrats
>

Re: Return a 200 0K status and send the content of the request later

[Ed Sabol]

On May 13, 2025, at 8:23 PM, Vincent Veyron  wrote:
> Is there a way I could do :
> 
>   $r->print('Building tar file') ;
> 
> every second or other until the tar file is built? I can't seem to think of 
> one.
> 
> My handler calls an sql script that dumps the database with system() :
> 
>@args = ('psql', '-f', '/path/to/script/to/export_raw_data.sql', '-v', 
> 'id_client=' . $r->pnotes('session')->{id_client}, '-v',  'database=' . 
> $database, 'postgres') ;
> 
>system(@args) == 0 or warn "system @args failed: $!" ;
> 
> then waits for the resulting file before proceeding to compress and tar it, 
> also with a system() call. 
> 
> How can I wrap this in a loop that re-sends the 'Building tar file' message 
> until the file exists?

Well, the standard way of doing that is you fork a child process to run the 
system command while the parent process loops and prints a period (or whatever 
you prefer) every 1 second (make sure you call flush() after printing!) while 
waiting for the child process to finish. Lots of good examples of this can be 
found here:

https://stackoverflow.com/questions/3193091/showing-progress-whilst-running-a-system-command-in-perl/

The problem with this is that forking from a mod_perl process is more 
complicated than it is in a normal Perl process. The usual advice is to use a 
module called Apache2::SubProcess instead:

https://metacpan.org/pod/Apache2::SubProcess

But I don't know that that will work for what you want to do. I've never 
actually used that module, and I can't find an example in my Internet searches 
that shows the parent process doing something while the subprocess executes. 
Maybe try the example using detach_script.pl and then modify that 
detach_script.pl to build the tar file or execute psql or do whatever you want.

If that doesn't work, you can still fork from mod_perl, but you just need to 
know how to do it properly. Make sure you read this:

https://stackoverflow.com/questions/471681/how-do-i-fork-properly-with-mod-perl2/

Since I'm not convinced Apache2::SubProcess would actually work here, I would 
probably go with the "fork properly with mod_perl2" method, personally.

I also recommend that you still put an upper time limit on your subprocess and 
not just loop infinitely. 30 minutes to an hour seems like a reasonable choice. 
Depends on how big your tar files can be, of course.

When/if you get something along these lines working the way you want, please 
share your solution here on the mailing list for others to learn from.

Hope this helps,
Ed