Re: Action return value too long for commandline

[Mark Shuttleworth]

Right. I think what you're describing really is an object exchange
mechanism / store with appropriate permissioning so other peers can see
an object a unit places there.

That's basically fine because it keeps the internal implementation of
the charm internal - only peer units with the same charm get to see the
objects, so the code will definitely know what to expect.

If we also allow operators to see those objects, we might have scripts
written to expect them, and we need to think if that's good. And that's
DOUBLY true if we want to think about exposing those objects across
relations, because it becomes yet another way to add friction to charm
substitutions - the new charm must also export all the same expected
objects.

But the first part sounds definitely +1

Mark

On 05/04/16 22:53, Merlijn Sebrechts wrote:
> Hi Mark
>
>
> What's your opinion to adding outflowing and peer-flowing capabilities to
> resources? Outflowing: from charm to operator, peer-flowing: from charm to
> charm. I'm not sure if there is such a big conceptual difference between
> inflowing, outflowing and peer-flowing. I have the tendency to see an
> operator as just another Charm. This might become reality when the "stacks"
> functionality gets introduced because then there will be a charm-like
> entity that manages a bunch of charms... This might make that transition
> easier...
>
>
>
> Kind regards
> Merlijn
>
> 2016-04-05 2:42 GMT-07:00 Mark Shuttleworth :
>
>> We should definitely have a mechanism for large-file output from actions
>> and other hooks. I think resources are "inward flowing", they come from the
>> charmer or they are provided by the operator. You're looking for "outward
>> flowing" blobs.
>>
>> As I recall, the intention was that actions could drop results into the
>> controller in a way that was friendly to large BLOB results. That's why we
>> have an async action output and result mechanism. I reckon it would be fine
>> to use resources for input to actions (there would just be a specific
>> user-provided resource name and there would be only one version of that at
>> any given time), but we want to make sure that output can also be
>> efficiently handled regardless of size, so if we need to improve that,
>> let's crack on.
>>
>> Mark
>>
>>
>> On 04/04/16 16:56, Merlijn Sebrechts wrote:
>>
>> Hi Stuart
>>
>>
>> Thanks for this explanation. 16MB is definitely a step in the right
>> direction. However, in the future I'll need to transfer even bigger files.
>> Is there a possibility for using resources for this in the future? It would
>> be great if an action could upload a file to the resources server so the
>> admin can download it from there.
>>
>>
>>
>> Kind regards
>> Merlijn Sebrechts
>>
>> 2016-04-03 19:34 GMT-07:00 Stuart Bishop  
>> :
>>
>>
>> On 4 April 2016 at 02:00, Merlijn Sebrechts  
>> 
>> wrote:
>>
>> Hi all
>>
>>
>> The apache-kafka charm has an action "read-topic" that can return a lot
>>
>> of
>>
>> data. Sometimes, this data is too long to be passed to `action-set` by
>> commandline. You get the following error:
>>
>> Traceback (most recent call last):
>>   File "actions/read-topic", line 36, in 
>> hookenv.action_set({'output': output})
>>   File
>> "/usr/local/lib/python2.7/dist-packages/charmhelpers/core/hookenv.py",
>>
>> line
>>
>> 615, in action_set
>> subprocess.check_call(cmd)
>>   File "/usr/lib/python2.7/subprocess.py", line 535, in check_call
>> retcode = call(*popenargs, **kwargs)
>>   File "/usr/lib/python2.7/subprocess.py", line 522, in call
>> return Popen(*popenargs, **kwargs).wait()
>>   File "/usr/lib/python2.7/subprocess.py", line 710, in __init__
>> errread, errwrite)
>>   File "/usr/lib/python2.7/subprocess.py", line 1327, in _execute_child
>> raise child_exception
>> OSError: [Errno 7] Argument list too long
>>
>>
>> Is there any way to pass a file to action-set?
>>
>> This bug affects many of the Juju tools causing charms to fail at
>> scale. https://bugs.launchpad.net/juju-core/+bug/1437366
>> (relation-set) is the only one I know of that has been 
>> fixed.https://bugs.launchpad.net/juju-core/+bug/1274460 (juju-log) is still
>> open. leader-set also fails now I think of it, but I haven't tripped
>> over that one (it limits scalability the way I'm using leadershp
>> settings, but I should be able to squeeze out several hundred units).
>> Maybe we can use the opportunity to fix quoting and encoding issues.
>>
>> I suspect if you can get past command line length limitations, I
>> suspect the next glass ceiling is a 16MB document size limit in
>> MongoDB (which is large enough to not need fixing?)
>>
>> --
>> Stuart Bishop  
>>
>>
>>
>>
>>



-- 
Juju mailing list
Juju@lists.ubuntu.com
Modify settings or unsubscribe at: 

Re: Action return value too long for commandline

[Merlijn Sebrechts]

Hi Mark


What's your opinion to adding outflowing and peer-flowing capabilities to
resources? Outflowing: from charm to operator, peer-flowing: from charm to
charm. I'm not sure if there is such a big conceptual difference between
inflowing, outflowing and peer-flowing. I have the tendency to see an
operator as just another Charm. This might become reality when the "stacks"
functionality gets introduced because then there will be a charm-like
entity that manages a bunch of charms... This might make that transition
easier...



Kind regards
Merlijn

2016-04-05 2:42 GMT-07:00 Mark Shuttleworth :

>
> We should definitely have a mechanism for large-file output from actions
> and other hooks. I think resources are "inward flowing", they come from the
> charmer or they are provided by the operator. You're looking for "outward
> flowing" blobs.
>
> As I recall, the intention was that actions could drop results into the
> controller in a way that was friendly to large BLOB results. That's why we
> have an async action output and result mechanism. I reckon it would be fine
> to use resources for input to actions (there would just be a specific
> user-provided resource name and there would be only one version of that at
> any given time), but we want to make sure that output can also be
> efficiently handled regardless of size, so if we need to improve that,
> let's crack on.
>
> Mark
>
>
> On 04/04/16 16:56, Merlijn Sebrechts wrote:
>
> Hi Stuart
>
>
> Thanks for this explanation. 16MB is definitely a step in the right
> direction. However, in the future I'll need to transfer even bigger files.
> Is there a possibility for using resources for this in the future? It would
> be great if an action could upload a file to the resources server so the
> admin can download it from there.
>
>
>
> Kind regards
> Merlijn Sebrechts
>
> 2016-04-03 19:34 GMT-07:00 Stuart Bishop  
> :
>
>
> On 4 April 2016 at 02:00, Merlijn Sebrechts  
> 
> wrote:
>
> Hi all
>
>
> The apache-kafka charm has an action "read-topic" that can return a lot
>
> of
>
> data. Sometimes, this data is too long to be passed to `action-set` by
> commandline. You get the following error:
>
> Traceback (most recent call last):
>   File "actions/read-topic", line 36, in 
> hookenv.action_set({'output': output})
>   File
> "/usr/local/lib/python2.7/dist-packages/charmhelpers/core/hookenv.py",
>
> line
>
> 615, in action_set
> subprocess.check_call(cmd)
>   File "/usr/lib/python2.7/subprocess.py", line 535, in check_call
> retcode = call(*popenargs, **kwargs)
>   File "/usr/lib/python2.7/subprocess.py", line 522, in call
> return Popen(*popenargs, **kwargs).wait()
>   File "/usr/lib/python2.7/subprocess.py", line 710, in __init__
> errread, errwrite)
>   File "/usr/lib/python2.7/subprocess.py", line 1327, in _execute_child
> raise child_exception
> OSError: [Errno 7] Argument list too long
>
>
> Is there any way to pass a file to action-set?
>
> This bug affects many of the Juju tools causing charms to fail at
> scale. https://bugs.launchpad.net/juju-core/+bug/1437366
> (relation-set) is the only one I know of that has been 
> fixed.https://bugs.launchpad.net/juju-core/+bug/1274460 (juju-log) is still
> open. leader-set also fails now I think of it, but I haven't tripped
> over that one (it limits scalability the way I'm using leadershp
> settings, but I should be able to squeeze out several hundred units).
> Maybe we can use the opportunity to fix quoting and encoding issues.
>
> I suspect if you can get past command line length limitations, I
> suspect the next glass ceiling is a 16MB document size limit in
> MongoDB (which is large enough to not need fixing?)
>
> --
> Stuart Bishop  
>
>
>
>
>
-- 
Juju mailing list
Juju@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju

Re: Action return value too long for commandline

[Mark Shuttleworth]

We should definitely have a mechanism for large-file output from actions
and other hooks. I think resources are "inward flowing", they come from
the charmer or they are provided by the operator. You're looking for
"outward flowing" blobs.

As I recall, the intention was that actions could drop results into the
controller in a way that was friendly to large BLOB results. That's why
we have an async action output and result mechanism. I reckon it would
be fine to use resources for input to actions (there would just be a
specific user-provided resource name and there would be only one version
of that at any given time), but we want to make sure that output can
also be efficiently handled regardless of size, so if we need to improve
that, let's crack on.

Mark

On 04/04/16 16:56, Merlijn Sebrechts wrote:
> Hi Stuart
>
>
> Thanks for this explanation. 16MB is definitely a step in the right
> direction. However, in the future I'll need to transfer even bigger files.
> Is there a possibility for using resources for this in the future? It would
> be great if an action could upload a file to the resources server so the
> admin can download it from there.
>
>
>
> Kind regards
> Merlijn Sebrechts
>
> 2016-04-03 19:34 GMT-07:00 Stuart Bishop :
>
>> On 4 April 2016 at 02:00, Merlijn Sebrechts 
>> wrote:
>>> Hi all
>>>
>>>
>>> The apache-kafka charm has an action "read-topic" that can return a lot
>> of
>>> data. Sometimes, this data is too long to be passed to `action-set` by
>>> commandline. You get the following error:
>>>
>>> Traceback (most recent call last):
>>>   File "actions/read-topic", line 36, in 
>>> hookenv.action_set({'output': output})
>>>   File
>>> "/usr/local/lib/python2.7/dist-packages/charmhelpers/core/hookenv.py",
>> line
>>> 615, in action_set
>>> subprocess.check_call(cmd)
>>>   File "/usr/lib/python2.7/subprocess.py", line 535, in check_call
>>> retcode = call(*popenargs, **kwargs)
>>>   File "/usr/lib/python2.7/subprocess.py", line 522, in call
>>> return Popen(*popenargs, **kwargs).wait()
>>>   File "/usr/lib/python2.7/subprocess.py", line 710, in __init__
>>> errread, errwrite)
>>>   File "/usr/lib/python2.7/subprocess.py", line 1327, in _execute_child
>>> raise child_exception
>>> OSError: [Errno 7] Argument list too long
>>>
>>>
>>> Is there any way to pass a file to action-set?
>> This bug affects many of the Juju tools causing charms to fail at
>> scale. https://bugs.launchpad.net/juju-core/+bug/1437366
>> (relation-set) is the only one I know of that has been fixed.
>> https://bugs.launchpad.net/juju-core/+bug/1274460 (juju-log) is still
>> open. leader-set also fails now I think of it, but I haven't tripped
>> over that one (it limits scalability the way I'm using leadershp
>> settings, but I should be able to squeeze out several hundred units).
>> Maybe we can use the opportunity to fix quoting and encoding issues.
>>
>> I suspect if you can get past command line length limitations, I
>> suspect the next glass ceiling is a 16MB document size limit in
>> MongoDB (which is large enough to not need fixing?)
>>
>> --
>> Stuart Bishop 
>>
>
>

-- 
Juju mailing list
Juju@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju

Re: Action return value too long for commandline

[Merlijn Sebrechts]

Hi Stuart


Thanks for this explanation. 16MB is definitely a step in the right
direction. However, in the future I'll need to transfer even bigger files.
Is there a possibility for using resources for this in the future? It would
be great if an action could upload a file to the resources server so the
admin can download it from there.



Kind regards
Merlijn Sebrechts

2016-04-03 19:34 GMT-07:00 Stuart Bishop :

> On 4 April 2016 at 02:00, Merlijn Sebrechts 
> wrote:
> > Hi all
> >
> >
> > The apache-kafka charm has an action "read-topic" that can return a lot
> of
> > data. Sometimes, this data is too long to be passed to `action-set` by
> > commandline. You get the following error:
> >
> > Traceback (most recent call last):
> >   File "actions/read-topic", line 36, in 
> > hookenv.action_set({'output': output})
> >   File
> > "/usr/local/lib/python2.7/dist-packages/charmhelpers/core/hookenv.py",
> line
> > 615, in action_set
> > subprocess.check_call(cmd)
> >   File "/usr/lib/python2.7/subprocess.py", line 535, in check_call
> > retcode = call(*popenargs, **kwargs)
> >   File "/usr/lib/python2.7/subprocess.py", line 522, in call
> > return Popen(*popenargs, **kwargs).wait()
> >   File "/usr/lib/python2.7/subprocess.py", line 710, in __init__
> > errread, errwrite)
> >   File "/usr/lib/python2.7/subprocess.py", line 1327, in _execute_child
> > raise child_exception
> > OSError: [Errno 7] Argument list too long
> >
> >
> > Is there any way to pass a file to action-set?
>
> This bug affects many of the Juju tools causing charms to fail at
> scale. https://bugs.launchpad.net/juju-core/+bug/1437366
> (relation-set) is the only one I know of that has been fixed.
> https://bugs.launchpad.net/juju-core/+bug/1274460 (juju-log) is still
> open. leader-set also fails now I think of it, but I haven't tripped
> over that one (it limits scalability the way I'm using leadershp
> settings, but I should be able to squeeze out several hundred units).
> Maybe we can use the opportunity to fix quoting and encoding issues.
>
> I suspect if you can get past command line length limitations, I
> suspect the next glass ceiling is a 16MB document size limit in
> MongoDB (which is large enough to not need fixing?)
>
> --
> Stuart Bishop 
>
-- 
Juju mailing list
Juju@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju

Re: Action return value too long for commandline

[Rick Harding]

Thanks, this is interesting to know. The fact that we've fixed it in
relation set means we have a pattern to move forward with. The team's
slammed on 2.0 work right now, but I've set this up to be something we try
to address across the board after the fact. We'll make sure to look for
other -set operations to make sure we standardize it across the board.

On Sun, Apr 3, 2016 at 10:34 PM Stuart Bishop 
wrote:

> On 4 April 2016 at 02:00, Merlijn Sebrechts 
> wrote:
> > Hi all
> >
> >
> > The apache-kafka charm has an action "read-topic" that can return a lot
> of
> > data. Sometimes, this data is too long to be passed to `action-set` by
> > commandline. You get the following error:
> >
> > Traceback (most recent call last):
> >   File "actions/read-topic", line 36, in 
> > hookenv.action_set({'output': output})
> >   File
> > "/usr/local/lib/python2.7/dist-packages/charmhelpers/core/hookenv.py",
> line
> > 615, in action_set
> > subprocess.check_call(cmd)
> >   File "/usr/lib/python2.7/subprocess.py", line 535, in check_call
> > retcode = call(*popenargs, **kwargs)
> >   File "/usr/lib/python2.7/subprocess.py", line 522, in call
> > return Popen(*popenargs, **kwargs).wait()
> >   File "/usr/lib/python2.7/subprocess.py", line 710, in __init__
> > errread, errwrite)
> >   File "/usr/lib/python2.7/subprocess.py", line 1327, in _execute_child
> > raise child_exception
> > OSError: [Errno 7] Argument list too long
> >
> >
> > Is there any way to pass a file to action-set?
>
> This bug affects many of the Juju tools causing charms to fail at
> scale. https://bugs.launchpad.net/juju-core/+bug/1437366
> (relation-set) is the only one I know of that has been fixed.
> https://bugs.launchpad.net/juju-core/+bug/1274460 (juju-log) is still
> open. leader-set also fails now I think of it, but I haven't tripped
> over that one (it limits scalability the way I'm using leadershp
> settings, but I should be able to squeeze out several hundred units).
> Maybe we can use the opportunity to fix quoting and encoding issues.
>
> I suspect if you can get past command line length limitations, I
> suspect the next glass ceiling is a 16MB document size limit in
> MongoDB (which is large enough to not need fixing?)
>
> --
> Stuart Bishop 
>
> --
> Juju mailing list
> Juju@lists.ubuntu.com
> Modify settings or unsubscribe at:
> https://lists.ubuntu.com/mailman/listinfo/juju
>
-- 
Juju mailing list
Juju@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju

Re: Action return value too long for commandline

[Stuart Bishop]

On 4 April 2016 at 02:00, Merlijn Sebrechts  wrote:
> Hi all
>
>
> The apache-kafka charm has an action "read-topic" that can return a lot of
> data. Sometimes, this data is too long to be passed to `action-set` by
> commandline. You get the following error:
>
> Traceback (most recent call last):
>   File "actions/read-topic", line 36, in 
> hookenv.action_set({'output': output})
>   File
> "/usr/local/lib/python2.7/dist-packages/charmhelpers/core/hookenv.py", line
> 615, in action_set
> subprocess.check_call(cmd)
>   File "/usr/lib/python2.7/subprocess.py", line 535, in check_call
> retcode = call(*popenargs, **kwargs)
>   File "/usr/lib/python2.7/subprocess.py", line 522, in call
> return Popen(*popenargs, **kwargs).wait()
>   File "/usr/lib/python2.7/subprocess.py", line 710, in __init__
> errread, errwrite)
>   File "/usr/lib/python2.7/subprocess.py", line 1327, in _execute_child
> raise child_exception
> OSError: [Errno 7] Argument list too long
>
>
> Is there any way to pass a file to action-set?

This bug affects many of the Juju tools causing charms to fail at
scale. https://bugs.launchpad.net/juju-core/+bug/1437366
(relation-set) is the only one I know of that has been fixed.
https://bugs.launchpad.net/juju-core/+bug/1274460 (juju-log) is still
open. leader-set also fails now I think of it, but I haven't tripped
over that one (it limits scalability the way I'm using leadershp
settings, but I should be able to squeeze out several hundred units).
Maybe we can use the opportunity to fix quoting and encoding issues.

I suspect if you can get past command line length limitations, I
suspect the next glass ceiling is a 16MB document size limit in
MongoDB (which is large enough to not need fixing?)

-- 
Stuart Bishop 

-- 
Juju mailing list
Juju@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju

Action return value too long for commandline

[Merlijn Sebrechts]

Hi all


The apache-kafka charm has an action "read-topic" that can return a lot of
data. Sometimes, this data is too long to be passed to `action-set` by
commandline. You get the following error:

Traceback (most recent call last):
  File "actions/read-topic", line 36, in 
hookenv.action_set({'output': output})
  File
"/usr/local/lib/python2.7/dist-packages/charmhelpers/core/hookenv.py", line
615, in action_set
subprocess.check_call(cmd)
  File "/usr/lib/python2.7/subprocess.py", line 535, in check_call
retcode = call(*popenargs, **kwargs)
  File "/usr/lib/python2.7/subprocess.py", line 522, in call
return Popen(*popenargs, **kwargs).wait()
  File "/usr/lib/python2.7/subprocess.py", line 710, in __init__
errread, errwrite)
  File "/usr/lib/python2.7/subprocess.py", line 1327, in _execute_child
raise child_exception
OSError: [Errno 7] Argument list too long


Is there any way to pass a file to action-set?



Kind regards
Merlijn Sebrechts
-- 
Juju mailing list
Juju@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju