Re: [yocto] [Yocto] Checking sstate mirror is hanging at 100%

2017-11-01 Thread Vineeth Chowdary Karumanchi 



On 11/1/2017 12:11 AM, Andre McCurdy wrote:


The fetcher explicitly tries to avoid that by retrying each file only
once. See retry logic in:

   
http://git.openembedded.org/bitbake/commit/?id=54b1961551511948e0cbd2ac39f19b39b9cee568

However, there is a later commit which may also be related:

   
http://git.openembedded.org/bitbake/commit/?id=6fa07752bbd3ac345cd8617da49a70e0b2dd565f
I didnt have this patch, presently i am on morty branch. I will check 
with master branch


Thanks for the info
Vineeth


Does the version of bitbake which you are using contain both commits?


--
___
yocto mailing list
yocto@yoctoproject.org
https://lists.yoctoproject.org/listinfo/yocto

Re: [yocto] [Yocto] Checking sstate mirror is hanging at 100%

2017-10-31 Thread Vineeth Karumanchi 



On 11/1/2017 12:11 AM, Andre McCurdy wrote:

It looks like the logic in the wget fetcher retries once, so if it's getting 
stuck
then that seems like a genuine bug.

Vineeth, are you sure it's really continuously retrying?

Yes, I see continuous retry of wget of same sstate files.

The fetcher explicitly tries to avoid that by retrying each file only
once. See retry logic in:

   
http://git.openembedded.org/bitbake/commit/?id=54b1961551511948e0cbd2ac39f19b39b9cee568

However, there is a later commit which may also be related:

   
http://git.openembedded.org/bitbake/commit/?id=6fa07752bbd3ac345cd8617da49a70e0b2dd565f

Does the version of bitbake which you are using contain both commits?


Thanks Andre
I am not having later patch. I will check the issue with master branch


--
___
yocto mailing list
yocto@yoctoproject.org
https://lists.yoctoproject.org/listinfo/yocto

Re: [yocto] [Yocto] Checking sstate mirror is hanging at 100%

2017-10-31 Thread Andre McCurdy 
>>
>> It looks like the logic in the wget fetcher retries once, so if it's getting 
>> stuck
>> then that seems like a genuine bug.
>>
>> Vineeth, are you sure it's really continuously retrying?
> Yes, I see continuous retry of wget of same sstate files.

The fetcher explicitly tries to avoid that by retrying each file only
once. See retry logic in:

  
http://git.openembedded.org/bitbake/commit/?id=54b1961551511948e0cbd2ac39f19b39b9cee568

However, there is a later commit which may also be related:

  
http://git.openembedded.org/bitbake/commit/?id=6fa07752bbd3ac345cd8617da49a70e0b2dd565f

Does the version of bitbake which you are using contain both commits?
-- 
___
yocto mailing list
yocto@yoctoproject.org
https://lists.yoctoproject.org/listinfo/yocto

Re: [yocto] [Yocto] Checking sstate mirror is hanging at 100%

2017-10-30 Thread Vineeth Karumanchi 

> From: Andre McCurdy [mailto:armccu...@gmail.com]
> Sent: Tuesday, October 31, 2017 1:19 AM
> To: Alexander Kanavin <alexander.kana...@linux.intel.com>
> Cc: Vineeth Karumanchi <vinee...@xilinx.com>; Yocto discussion list
> <yocto@yoctoproject.org>
> Subject: Re: [yocto] [Yocto] Checking sstate mirror is hanging at 100%
> 
> On Mon, Oct 30, 2017 at 2:37 AM, Alexander Kanavin
> <alexander.kana...@linux.intel.com> wrote:
> > On 10/30/2017 07:54 AM, Vineeth Karumanchi wrote:
> >
> >> In this case i would expect bitbake to "check for sstate mirrors" for
> >> some time and rebuild from scratch. Is there any approach to over come
> this ?
> >
> > The approach is to fix your infrastructure so that fetching from
> > network doesn't fail. We've gotten a similar request recently for
> > enabling 'soft sstate cache failure', and RP is firmly against it:
> >
> > http://lists.openembedded.org/pipermail/openembedded-core/2017-
> August/
> > 141593.html
> 
> It looks like the logic in the wget fetcher retries once, so if it's getting 
> stuck
> then that seems like a genuine bug.
> 
> Vineeth, are you sure it's really continuously retrying? 
Yes, I see continuous retry of wget of same sstate files.
With -DD I see the below messages  for every try:

DEBUG: checkstatus: trying again
DEBUG: checkstatus: trying again
DEBUG: checkstatus: trying again
DEBUG: checkstatus: trying again
DEBUG: checkstatus: trying again

>Or are you just seeing
> timeouts for many fetches (which will eventually complete)? Note that you
> can reduce the wget timeout by setting your own value for
> "FETCHCMD_wget". See bitbake/lib/bb/fetch2/wget.py for the default value
> - use that as a starting point for your modification.

No fetch failure error, Just hanging at 100%.

I did some more analysis:
My network needs proxy to access any site.

To validate a URL, bitbake/lib/bb/fetch2/__init__.py:  
urllib.parse.urlparse(uri) and checking for netloc.
This is returning true without proxy being set, but the same uri is not 
establishing connection without proxies.

bitbake is thinking it as a valid  and keep retrying without any errors.

Thanks
Vineeth





-- 
___
yocto mailing list
yocto@yoctoproject.org
https://lists.yoctoproject.org/listinfo/yocto

Re: [yocto] [Yocto] Checking sstate mirror is hanging at 100%

2017-10-30 Thread Andre McCurdy 
On Mon, Oct 30, 2017 at 2:37 AM, Alexander Kanavin
 wrote:
> On 10/30/2017 07:54 AM, Vineeth Karumanchi wrote:
>
>> In this case i would expect bitbake to "check for sstate mirrors" for some
>> time and rebuild from scratch. Is there any approach to over come this ?
>
> The approach is to fix your infrastructure so that fetching from network
> doesn't fail. We've gotten a similar request recently for enabling 'soft
> sstate cache failure', and RP is firmly against it:
>
> http://lists.openembedded.org/pipermail/openembedded-core/2017-August/141593.html

It looks like the logic in the wget fetcher retries once, so if it's
getting stuck then that seems like a genuine bug.

Vineeth, are you sure it's really continuously retrying? Or are you
just seeing timeouts for many fetches (which will eventually
complete)? Note that you can reduce the wget timeout by setting your
own value for "FETCHCMD_wget". See bitbake/lib/bb/fetch2/wget.py for
the default value - use that as a starting point for your
modification.
-- 
___
yocto mailing list
yocto@yoctoproject.org
https://lists.yoctoproject.org/listinfo/yocto

Re: [yocto] [Yocto] Checking sstate mirror is hanging at 100%

2017-10-30 Thread Alexander Kanavin 

On 10/30/2017 07:54 AM, Vineeth Karumanchi wrote:

In this case i would expect bitbake to "check for sstate mirrors" for 
some time and rebuild from scratch. Is there any approach to over come 
this ?


The approach is to fix your infrastructure so that fetching from network 
doesn't fail. We've gotten a similar request recently for enabling 'soft 
sstate cache failure', and RP is firmly against it:


http://lists.openembedded.org/pipermail/openembedded-core/2017-August/141593.html

Alex
--
___
yocto mailing list
yocto@yoctoproject.org
https://lists.yoctoproject.org/listinfo/yocto

[yocto] [Yocto] Checking sstate mirror is hanging at 100%

2017-10-30 Thread Vineeth Karumanchi 

Hi All,

I am running sstate mirrors on a web server. My network has proxy, 
sstate is sharing well with proxies, .


But, without proxies, bitbake is hanging at "/Checking sstate mirror 
object ###100%/".  Is there a way to change timeout of sstate mirror 
checking.


I did some debugging with "-DD" and I see bitbake is trying to fetch the 
sstate objects, failing and keep on re-trying.



/DEBUG: checkstatus: trying again/


In this case i would expect bitbake to "check for sstate mirrors" for 
some time and rebuild from scratch. Is there any approach to over come 
this ?



Thanks in Advance

Vineeth

-- 
___
yocto mailing list
yocto@yoctoproject.org
https://lists.yoctoproject.org/listinfo/yocto