Re: very long cache_open_read_end - timeout

[Bryan Call]

That is what I set ATS to do in production for us.  Funny that we both came up 
with 4.  It was mainly do to lock contention issues and people benchmarking or 
running ATS at high request rates.

$ tail -4 volume.config
volume=1 scheme=http size=25%
volume=2 scheme=http size=25%
volume=3 scheme=http size=25%
volume=4 scheme=http size=25%

-Bryan

> On Feb 26, 2018, at 12:15 AM, Veiko Kukk  wrote:
> 
> Hi Bryan,
> 
> We decided to create 4 volumes. No strong reasoning behind it, just 2 seemed 
> too little to avoid those blocking timeouts and 4 was next comfortable to set 
> using 25%.
> 
> Veiko
> 
> 
> 2018-02-23 21:21 GMT+02:00 Bryan Call  >:
> Veiko,
> 
> How many volumes did you create?
> 
> -Bryan
> 
> 
>> On Feb 23, 2018, at 3:14 AM, Veiko Kukk > > wrote:
>> 
>> Hi, Mateusz
>> 
>> I wonder if you might experience same issue as me.
>> https://github.com/apache/trafficserver/issues/3057 
>> 
>> 
>> For me, solution, or I would rather call it workaround was to create volumes.
>> 
>> Unfortunately, documentation regarding to storage performance issues/tuning 
>> is missing.
>> 
>> Best regards,
>> Veiko
>> 
>> 
>> 2018-02-22 13:59 GMT+02:00 Mateusz Zajakala > >:
>> Hi,
>> 
>> I have a problem - from time to time my ATS 7.2.1 has problems with serving 
>> static files from origin, client transactions last over 10s and I see the 
>> following diagnostic (I have enabled logging "slow requests")
>> 
>>  cache_open_read_begin: 0.000 cache_open_read_end: 10.640
>> 
>> Any requests to the same URL are blocked for over 10s. Restarting ATS solves 
>> the problem.
>> 
>> What does it mean? Does it have to do with open_read feature?  but the 
>> timeouts don't add...
>> 
>> Any suggestions appreciated!
>> 
>> Here's my records.config with configs related to avoiding thundering herd...
>> 
>> CONFIG proxy.config.cache.enable_read_while_writer INT 1
>> CONFIG proxy.config.http.background_fill_active_timeout INT 0
>> CONFIG proxy.config.http.background_fill_completed_threshold FLOAT 0.00
>> CONFIG proxy.config.cache.read_while_writer.max_retries INT 5
>> CONFIG proxy.config.cache.read_while_writer_retry.delay INT 200
>> CONFIG proxy.config.http.cache.max_open_read_retries INT 5
>> CONFIG proxy.config.http.cache.open_read_retry_time INT 200
>> 
>> CONFIG proxy.config.http.cache.open_write_fail_action INT 2
>> 
>> 
> 
> 

Re: very long cache_open_read_end - timeout

[Veiko Kukk]

Hi Bryan,

We decided to create 4 volumes. No strong reasoning behind it, just 2
seemed too little to avoid those blocking timeouts and 4 was next
comfortable to set using 25%.

Veiko


2018-02-23 21:21 GMT+02:00 Bryan Call :

> Veiko,
>
> How many volumes did you create?
>
> -Bryan
>
>
> On Feb 23, 2018, at 3:14 AM, Veiko Kukk  wrote:
>
> Hi, Mateusz
>
> I wonder if you might experience same issue as me.
> https://github.com/apache/trafficserver/issues/3057
>
> For me, solution, or I would rather call it workaround was to create
> volumes.
>
> Unfortunately, documentation regarding to storage performance
> issues/tuning is missing.
>
> Best regards,
> Veiko
>
>
> 2018-02-22 13:59 GMT+02:00 Mateusz Zajakala :
>
>> Hi,
>>
>> I have a problem - from time to time my ATS 7.2.1 has problems with
>> serving static files from origin, client transactions last over 10s and I
>> see the following diagnostic (I have enabled logging "slow requests")
>>
>>  cache_open_read_begin: 0.000 cache_open_read_end: 10.640
>>
>> Any requests to the same URL are blocked for over 10s. Restarting ATS
>> solves the problem.
>>
>> What does it mean? Does it have to do with open_read feature?  but the
>> timeouts don't add...
>>
>> Any suggestions appreciated!
>>
>> Here's my records.config with configs related to avoiding thundering
>> herd...
>>
>> CONFIG proxy.config.cache.enable_read_while_writer INT 1
>> CONFIG proxy.config.http.background_fill_active_timeout INT 0
>> CONFIG proxy.config.http.background_fill_completed_threshold FLOAT
>> 0.00
>> CONFIG proxy.config.cache.read_while_writer.max_retries INT 5
>> CONFIG proxy.config.cache.read_while_writer_retry.delay INT 200
>> CONFIG proxy.config.http.cache.max_open_read_retries INT 5
>> CONFIG proxy.config.http.cache.open_read_retry_time INT 200
>>
>> CONFIG proxy.config.http.cache.open_write_fail_action INT 2
>>
>>
>
>

Re: very long cache_open_read_end - timeout

[Bryan Call]

Veiko,

How many volumes did you create?

-Bryan

> On Feb 23, 2018, at 3:14 AM, Veiko Kukk  wrote:
> 
> Hi, Mateusz
> 
> I wonder if you might experience same issue as me.
> https://github.com/apache/trafficserver/issues/3057 
> 
> 
> For me, solution, or I would rather call it workaround was to create volumes.
> 
> Unfortunately, documentation regarding to storage performance issues/tuning 
> is missing.
> 
> Best regards,
> Veiko
> 
> 
> 2018-02-22 13:59 GMT+02:00 Mateusz Zajakala  >:
> Hi,
> 
> I have a problem - from time to time my ATS 7.2.1 has problems with serving 
> static files from origin, client transactions last over 10s and I see the 
> following diagnostic (I have enabled logging "slow requests")
> 
>  cache_open_read_begin: 0.000 cache_open_read_end: 10.640
> 
> Any requests to the same URL are blocked for over 10s. Restarting ATS solves 
> the problem.
> 
> What does it mean? Does it have to do with open_read feature?  but the 
> timeouts don't add...
> 
> Any suggestions appreciated!
> 
> Here's my records.config with configs related to avoiding thundering herd...
> 
> CONFIG proxy.config.cache.enable_read_while_writer INT 1
> CONFIG proxy.config.http.background_fill_active_timeout INT 0
> CONFIG proxy.config.http.background_fill_completed_threshold FLOAT 0.00
> CONFIG proxy.config.cache.read_while_writer.max_retries INT 5
> CONFIG proxy.config.cache.read_while_writer_retry.delay INT 200
> CONFIG proxy.config.http.cache.max_open_read_retries INT 5
> CONFIG proxy.config.http.cache.open_read_retry_time INT 200
> 
> CONFIG proxy.config.http.cache.open_write_fail_action INT 2
> 
> 

Re: very long cache_open_read_end - timeout

[Veiko Kukk]

Hi, Mateusz

I wonder if you might experience same issue as me.
https://github.com/apache/trafficserver/issues/3057

For me, solution, or I would rather call it workaround was to create
volumes.

Unfortunately, documentation regarding to storage performance issues/tuning
is missing.

Best regards,
Veiko


2018-02-22 13:59 GMT+02:00 Mateusz Zajakala :

> Hi,
>
> I have a problem - from time to time my ATS 7.2.1 has problems with
> serving static files from origin, client transactions last over 10s and I
> see the following diagnostic (I have enabled logging "slow requests")
>
>  cache_open_read_begin: 0.000 cache_open_read_end: 10.640
>
> Any requests to the same URL are blocked for over 10s. Restarting ATS
> solves the problem.
>
> What does it mean? Does it have to do with open_read feature?  but the
> timeouts don't add...
>
> Any suggestions appreciated!
>
> Here's my records.config with configs related to avoiding thundering
> herd...
>
> CONFIG proxy.config.cache.enable_read_while_writer INT 1
> CONFIG proxy.config.http.background_fill_active_timeout INT 0
> CONFIG proxy.config.http.background_fill_completed_threshold FLOAT
> 0.00
> CONFIG proxy.config.cache.read_while_writer.max_retries INT 5
> CONFIG proxy.config.cache.read_while_writer_retry.delay INT 200
> CONFIG proxy.config.http.cache.max_open_read_retries INT 5
> CONFIG proxy.config.http.cache.open_read_retry_time INT 200
>
> CONFIG proxy.config.http.cache.open_write_fail_action INT 2
>
>

very long cache_open_read_end - timeout

[Mateusz Zajakala]

Hi,

I have a problem - from time to time my ATS 7.2.1 has problems with serving
static files from origin, client transactions last over 10s and I see the
following diagnostic (I have enabled logging "slow requests")

 cache_open_read_begin: 0.000 cache_open_read_end: 10.640

Any requests to the same URL are blocked for over 10s. Restarting ATS
solves the problem.

What does it mean? Does it have to do with open_read feature?  but the
timeouts don't add...

Any suggestions appreciated!

Here's my records.config with configs related to avoiding thundering herd...

CONFIG proxy.config.cache.enable_read_while_writer INT 1
CONFIG proxy.config.http.background_fill_active_timeout INT 0
CONFIG proxy.config.http.background_fill_completed_threshold FLOAT 0.00
CONFIG proxy.config.cache.read_while_writer.max_retries INT 5
CONFIG proxy.config.cache.read_while_writer_retry.delay INT 200
CONFIG proxy.config.http.cache.max_open_read_retries INT 5
CONFIG proxy.config.http.cache.open_read_retry_time INT 200

CONFIG proxy.config.http.cache.open_write_fail_action INT 2