Re: [OE-core] [PATCH v2 1/3] ccache.bbclass: enable max size setup for ccache dir

[Yannick GICQUEL]

2017-01-24 0:20 GMT+01:00 Andre McCurdy :
> On Fri, Jan 20, 2017 at 8:47 AM, Yannick GICQUEL
>  wrote:
>> 2017-01-20 17:23 GMT+01:00 Yannick GICQUEL :
>>> 2017-01-19 20:02 GMT+01:00 Andre McCurdy :
 On Thu, Jan 19, 2017 at 7:01 AM, Yannick Gicquel
  wrote:
> ccache directories are limited to 1G by default.

 1G has been the default for a long time, so maybe it's time to propose
 an upstream change to increase, e.g. to 10G, giving the size of the
 webkitgtk build as justification?
>>>
>>> You are right, 1GB was the default cache size limit in old versions
>>> and it's up to 5GB since release 3.2 [1].
>>>
 Since we build ccache-native within oe-core, any such patch could be
 applied to our version of ccache right now (ie we don't need to wait
 for it to be accepted and merged upstream).
>>>
>>> Current ccache.bbclass implementation is using the host ccache, so the
>>> default limit can be different depending on the version deployed on
>>> host distro. The point is that even if we use ccache-native, their
>>> still will be a default size limit set.
>>> Even if 5GB (or 10GB) limit should fit the webkitgtk cache size
>>> requirements, some other recipes can require more space to gain full
>>> benefit of the ccache (e.g. chromium from meta-browser) and for those
>>> it can be useful to tune the size.
>>>
>>> Does it make sense?
>
> Yes, it does. If 5GB isn't enough, then enabling compression of files
> in ccache ( https://ccache.samba.org/manual.html#_cache_compression )
> may be a better solution than removing the 5GB limit, but I do see the
> argument that since we don't place arbitrary limits on downloads or
> sstate directory sizes, why limit ccache?
>
> However, calling "ccache -M XXX" after every recipe's do_configure()
> still doesn't seem ideal. Firstly because the command will run many
> more times than necessary but also because, for users who over-ride

This add a call to ccache binary once per recipe build. In term of
execution time, IMHO it looks quite reasonable compared to the
drawback of a rebuild with limited cache size, but...

> CCACHE_DIR to try to share ccache files between builds, it will
> silently and permanently over-write any existing "max_size" option
> which they may have manually configured.

... I agree : the default behavior proposed can silently change some
user's custom setup, which may not be a good thing. (Same remark can
be done on the activation of compression, which may be kept to the
user's preferences)

>
> As an alternative, does simply exporting CCACHE_MAXSIZE=0 from
> ccache.bbclass work?

With recent ccache release, yes it does (v3.3.3 at least). But
depending on the host distro, it may not.
I had some experiment about using "ccache-native" binary, and I also
agree it can provides more flexibility : most (all?) of the
configuration variable can be tuned by exporting relevant environment
variable.

Updating ccache.bbclass to use ccache-native is not an easy task :
basically we could add a DEPENDS on "ccache-native" to all recipe
which inherit this class, but it can bring some dependency loops on
ccache recipe itself, which I do not know how to fix in a generic way.

About this patch serie, I propose to split it and keep this "max_size"
tune feature on a separate scope.

>
> (In general, I'm all in favour of improving ccache support in oe-core
> so it's good to see these changes being proposed and discussed).

You are welcome, thanks for your feedbacks.

>
>>> Yannick
>>>
>>> [1]: https://ccache.samba.org/releasenotes.html#_ccache_3_2
>>>

> This patch enables the configuration of their limits, and as default
> location is TMPDIR, it proposes a size limit to "0" (unlimited).
>
> The setup can be overloaded in local.conf by setting
> CCACHE_MAX_SIZE to a custom value if needed.
>
> Signed-off-by: Yannick Gicquel 
> ---
>  meta/classes/ccache.bbclass | 8 
>  1 file changed, 8 insertions(+)
>
> diff --git a/meta/classes/ccache.bbclass b/meta/classes/ccache.bbclass
> index 93fcaca..b6643a1 100644
> --- a/meta/classes/ccache.bbclass
> +++ b/meta/classes/ccache.bbclass
> @@ -1,6 +1,14 @@
>  CCACHE = "${@bb.utils.which(d.getVar('PATH'), 'ccache') and 'ccache '}"
>  export CCACHE_DIR ?= "${TMPDIR}/ccache/${MULTIMACH_HOST_SYS}/${PN}"
>  CCACHE_DISABLE[unexport] = "1"
> +CCACHE_MAX_SIZE ?= "0"
>
>  do_configure[dirs] =+ "${CCACHE_DIR}"
>  do_kernel_configme[dirs] =+ "${CCACHE_DIR}"
> +
> +ccache_init() {
> +if [ -n "${CCACHE}" ]; then
> +${CCACHE} -M ${CCACHE_MAX_SIZE}
> +fi
> +}
> +do_configure[postfuncs] += "ccache_init"
> --
> 1.9.1
>
-- 
___
Openembedded-core mailing list

Re: [OE-core] [PATCH v2 1/3] ccache.bbclass: enable max size setup for ccache dir

[Andre McCurdy]

On Fri, Jan 20, 2017 at 8:47 AM, Yannick GICQUEL
 wrote:
> 2017-01-20 17:23 GMT+01:00 Yannick GICQUEL :
>> 2017-01-19 20:02 GMT+01:00 Andre McCurdy :
>>> On Thu, Jan 19, 2017 at 7:01 AM, Yannick Gicquel
>>>  wrote:
 ccache directories are limited to 1G by default.
>>>
>>> 1G has been the default for a long time, so maybe it's time to propose
>>> an upstream change to increase, e.g. to 10G, giving the size of the
>>> webkitgtk build as justification?
>>
>> You are right, 1GB was the default cache size limit in old versions
>> and it's up to 5GB since release 3.2 [1].
>>
>>> Since we build ccache-native within oe-core, any such patch could be
>>> applied to our version of ccache right now (ie we don't need to wait
>>> for it to be accepted and merged upstream).
>>
>> Current ccache.bbclass implementation is using the host ccache, so the
>> default limit can be different depending on the version deployed on
>> host distro. The point is that even if we use ccache-native, their
>> still will be a default size limit set.
>> Even if 5GB (or 10GB) limit should fit the webkitgtk cache size
>> requirements, some other recipes can require more space to gain full
>> benefit of the ccache (e.g. chromium from meta-browser) and for those
>> it can be useful to tune the size.
>>
>> Does it make sense?

Yes, it does. If 5GB isn't enough, then enabling compression of files
in ccache ( https://ccache.samba.org/manual.html#_cache_compression )
may be a better solution than removing the 5GB limit, but I do see the
argument that since we don't place arbitrary limits on downloads or
sstate directory sizes, why limit ccache?

However, calling "ccache -M XXX" after every recipe's do_configure()
still doesn't seem ideal. Firstly because the command will run many
more times than necessary but also because, for users who over-ride
CCACHE_DIR to try to share ccache files between builds, it will
silently and permanently over-write any existing "max_size" option
which they may have manually configured.

As an alternative, does simply exporting CCACHE_MAXSIZE=0 from
ccache.bbclass work?

(In general, I'm all in favour of improving ccache support in oe-core
so it's good to see these changes being proposed and discussed).

>> Yannick
>>
>> [1]: https://ccache.samba.org/releasenotes.html#_ccache_3_2
>>
>>>
 This patch enables the configuration of their limits, and as default
 location is TMPDIR, it proposes a size limit to "0" (unlimited).

 The setup can be overloaded in local.conf by setting
 CCACHE_MAX_SIZE to a custom value if needed.

 Signed-off-by: Yannick Gicquel 
 ---
  meta/classes/ccache.bbclass | 8 
  1 file changed, 8 insertions(+)

 diff --git a/meta/classes/ccache.bbclass b/meta/classes/ccache.bbclass
 index 93fcaca..b6643a1 100644
 --- a/meta/classes/ccache.bbclass
 +++ b/meta/classes/ccache.bbclass
 @@ -1,6 +1,14 @@
  CCACHE = "${@bb.utils.which(d.getVar('PATH'), 'ccache') and 'ccache '}"
  export CCACHE_DIR ?= "${TMPDIR}/ccache/${MULTIMACH_HOST_SYS}/${PN}"
  CCACHE_DISABLE[unexport] = "1"
 +CCACHE_MAX_SIZE ?= "0"

  do_configure[dirs] =+ "${CCACHE_DIR}"
  do_kernel_configme[dirs] =+ "${CCACHE_DIR}"
 +
 +ccache_init() {
 +if [ -n "${CCACHE}" ]; then
 +${CCACHE} -M ${CCACHE_MAX_SIZE}
 +fi
 +}
 +do_configure[postfuncs] += "ccache_init"
 --
 1.9.1

-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH v2 1/3] ccache.bbclass: enable max size setup for ccache dir

[Yannick GICQUEL]

Hi,

Some comments inline below (with oe-list this time),

Regards,

2017-01-20 17:23 GMT+01:00 Yannick GICQUEL :
> 2017-01-19 20:02 GMT+01:00 Andre McCurdy :
>> On Thu, Jan 19, 2017 at 7:01 AM, Yannick Gicquel
>>  wrote:
>>> ccache directories are limited to 1G by default.
>>
>> 1G has been the default for a long time, so maybe it's time to propose
>> an upstream change to increase, e.g. to 10G, giving the size of the
>> webkitgtk build as justification?
>
> You are right, 1GB was the default cache size limit in old versions
> and it's up to 5GB since release 3.2 [1].
>
>>
>> Since we build ccache-native within oe-core, any such patch could be
>> applied to our version of ccache right now (ie we don't need to wait
>> for it to be accepted and merged upstream).
>
> Current ccache.bbclass implementation is using the host ccache, so the
> default limit can be different depending on the version deployed on
> host distro. The point is that even if we use ccache-native, their
> still will be a default size limit set.
> Even if 5GB (or 10GB) limit should fit the webkitgtk cache size
> requirements, some other recipes can require more space to gain full
> benefit of the ccache (e.g. chromium from meta-browser) and for those
> it can be useful to tune the size.
>
> Does it make sense?
>
> Yannick
>
> [1]: https://ccache.samba.org/releasenotes.html#_ccache_3_2
>
>>
>>> This patch enables the configuration of their limits, and as default
>>> location is TMPDIR, it proposes a size limit to "0" (unlimited).
>>>
>>> The setup can be overloaded in local.conf by setting
>>> CCACHE_MAX_SIZE to a custom value if needed.
>>>
>>> Signed-off-by: Yannick Gicquel 
>>> ---
>>>  meta/classes/ccache.bbclass | 8 
>>>  1 file changed, 8 insertions(+)
>>>
>>> diff --git a/meta/classes/ccache.bbclass b/meta/classes/ccache.bbclass
>>> index 93fcaca..b6643a1 100644
>>> --- a/meta/classes/ccache.bbclass
>>> +++ b/meta/classes/ccache.bbclass
>>> @@ -1,6 +1,14 @@
>>>  CCACHE = "${@bb.utils.which(d.getVar('PATH'), 'ccache') and 'ccache '}"
>>>  export CCACHE_DIR ?= "${TMPDIR}/ccache/${MULTIMACH_HOST_SYS}/${PN}"
>>>  CCACHE_DISABLE[unexport] = "1"
>>> +CCACHE_MAX_SIZE ?= "0"
>>>
>>>  do_configure[dirs] =+ "${CCACHE_DIR}"
>>>  do_kernel_configme[dirs] =+ "${CCACHE_DIR}"
>>> +
>>> +ccache_init() {
>>> +if [ -n "${CCACHE}" ]; then
>>> +${CCACHE} -M ${CCACHE_MAX_SIZE}
>>> +fi
>>> +}
>>> +do_configure[postfuncs] += "ccache_init"
>>> --
>>> 1.9.1
>>>
>>> --
>>> ___
>>> Openembedded-core mailing list
>>> Openembedded-core@lists.openembedded.org
>>> http://lists.openembedded.org/mailman/listinfo/openembedded-core
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [OE-core] [PATCH v2 1/3] ccache.bbclass: enable max size setup for ccache dir

[Andre McCurdy]

On Thu, Jan 19, 2017 at 7:01 AM, Yannick Gicquel
 wrote:
> ccache directories are limited to 1G by default.

1G has been the default for a long time, so maybe it's time to propose
an upstream change to increase, e.g. to 10G, giving the size of the
webkitgtk build as justification?

Since we build ccache-native within oe-core, any such patch could be
applied to our version of ccache right now (ie we don't need to wait
for it to be accepted and merged upstream).

> This patch enables the configuration of their limits, and as default
> location is TMPDIR, it proposes a size limit to "0" (unlimited).
>
> The setup can be overloaded in local.conf by setting
> CCACHE_MAX_SIZE to a custom value if needed.
>
> Signed-off-by: Yannick Gicquel 
> ---
>  meta/classes/ccache.bbclass | 8 
>  1 file changed, 8 insertions(+)
>
> diff --git a/meta/classes/ccache.bbclass b/meta/classes/ccache.bbclass
> index 93fcaca..b6643a1 100644
> --- a/meta/classes/ccache.bbclass
> +++ b/meta/classes/ccache.bbclass
> @@ -1,6 +1,14 @@
>  CCACHE = "${@bb.utils.which(d.getVar('PATH'), 'ccache') and 'ccache '}"
>  export CCACHE_DIR ?= "${TMPDIR}/ccache/${MULTIMACH_HOST_SYS}/${PN}"
>  CCACHE_DISABLE[unexport] = "1"
> +CCACHE_MAX_SIZE ?= "0"
>
>  do_configure[dirs] =+ "${CCACHE_DIR}"
>  do_kernel_configme[dirs] =+ "${CCACHE_DIR}"
> +
> +ccache_init() {
> +if [ -n "${CCACHE}" ]; then
> +${CCACHE} -M ${CCACHE_MAX_SIZE}
> +fi
> +}
> +do_configure[postfuncs] += "ccache_init"
> --
> 1.9.1
>
> --
> ___
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/openembedded-core
-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core

[OE-core] [PATCH v2 1/3] ccache.bbclass: enable max size setup for ccache dir

[Yannick Gicquel]

ccache directories are limited to 1G by default.
This patch enables the configuration of their limits, and as default
location is TMPDIR, it proposes a size limit to "0" (unlimited).

The setup can be overloaded in local.conf by setting
CCACHE_MAX_SIZE to a custom value if needed.

Signed-off-by: Yannick Gicquel 
---
 meta/classes/ccache.bbclass | 8 
 1 file changed, 8 insertions(+)

diff --git a/meta/classes/ccache.bbclass b/meta/classes/ccache.bbclass
index 93fcaca..b6643a1 100644
--- a/meta/classes/ccache.bbclass
+++ b/meta/classes/ccache.bbclass
@@ -1,6 +1,14 @@
 CCACHE = "${@bb.utils.which(d.getVar('PATH'), 'ccache') and 'ccache '}"
 export CCACHE_DIR ?= "${TMPDIR}/ccache/${MULTIMACH_HOST_SYS}/${PN}"
 CCACHE_DISABLE[unexport] = "1"
+CCACHE_MAX_SIZE ?= "0"
 
 do_configure[dirs] =+ "${CCACHE_DIR}"
 do_kernel_configme[dirs] =+ "${CCACHE_DIR}"
+
+ccache_init() {
+if [ -n "${CCACHE}" ]; then
+${CCACHE} -M ${CCACHE_MAX_SIZE}
+fi
+}
+do_configure[postfuncs] += "ccache_init"
-- 
1.9.1

-- 
___
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core