Re: [Bug 1592822] Re: Charm push crashes when pushing 'big' charm.

[Uros Jovanovic]

That's all true. We've increased the timeout for now, however, the proper
fix requires quite some effort. We need to move the GridFS blobs to Swift
and then storage management will become simpler (production wise). This
also covers multi-charm store syncing at the same time. I'm also going to
propose that we introduce quotas for charm store on user namespace. You can
ask for more space but that's how we can prevent from some CI to upload
gigabytes of data per day by mistake. Log analysis will help as well, but
that's also on ToDo ATM :-/


On Sun, Jul 10, 2016 at 9:28 AM, John A Meinel 
wrote:

> Is this a case that we are successfully sending data to squid at a
> reasonable but slow rate, such that it takes more than 5 minutes to upload
> the whole content and squid doesn't send anything to Apache until all of it
> has been transferred.
> It sounds like we'll have the same problem with resources. While it might
> allow us to break up a large charm into a few smaller resources, AIUI some
> of the charms have a single tarball that is more than 1GB in size.
> Arbitrarily limiting the size is going to just cause people to work around
> the system (self hosting, etc) Which breaks many of the benefits of having
> resources (demure transfers, offline mirroring, etx)
>
> --
> You received this bug notification because you are a bug assignee.
> https://bugs.launchpad.net/bugs/1592822
>
> Title:
>   Charm push crashes when pushing 'big' charm.
>
> Status in charm package in Ubuntu:
>   Confirmed
>
> Bug description:
>   running `charm push` for a very large charm causes a crash. Removing
>   the largest files from the charm fixes the issue. If large charms are
>   not supported, it might be better to throw a helpful error message.
>
>
>   ```
>   ERROR cannot post archive: cannot unmarshal error response " html PUBLIC \"-//W3C//DTD HTML 4.01//EN\" \"
> http://www.w3.org/TR/html4/strict.dtd\;>\n\n http-equiv=\"Content-Type\" content=\"text/html;
> charset=utf-8\">\nERROR: The requested URL could not be
> retrieved\n\n id=ERR_ZERO_SIZE_OBJECT>\n\nERROR\nThe
> requested URL could not be retrieved\n\n\n\n id=\"content\">\nThe following error was encountered while trying to
> retrieve the URL:  http://api.jujucharms.com/v5/~tengu-bot/trusty/rest2jfed/archive?\;>
> http://api.jujucharms.com/v5/~tengu-bot/trusty/rest2jfed/archive?\n\n id=\"error\">\nZero Sized Reply\n\n\nSquid
> did not receive any data for this request.\n\nYour cache
> administrator is mailto:webmaster?subjec ... [3528 bytes
> omitted]": invalid character '<' looking for beginning of value
>   ```
>
>   Putting the charm in attachment causes a launchpad crash. Please let
>   me know if I can upload the charm somewhere else.
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/ubuntu/+source/charm/+bug/1592822/+subscriptions
>

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1592822

Title:
  Charm push crashes when pushing 'big' charm.

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/charm/+bug/1592822/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1592822] Re: Charm push crashes when pushing 'big' charm.

[John A Meinel]

Is this a case that we are successfully sending data to squid at a reasonable 
but slow rate, such that it takes more than 5 minutes to upload the whole 
content and squid doesn't send anything to Apache until all of it has been 
transferred.
It sounds like we'll have the same problem with resources. While it might allow 
us to break up a large charm into a few smaller resources, AIUI some of the 
charms have a single tarball that is more than 1GB in size. Arbitrarily 
limiting the size is going to just cause people to work around the system (self 
hosting, etc) Which breaks many of the benefits of having resources (demure 
transfers, offline mirroring, etx)

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1592822

Title:
  Charm push crashes when pushing 'big' charm.

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/charm/+bug/1592822/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1592822] Re: Charm push crashes when pushing 'big' charm.

[Jay R. Wren]

I replied on email list:

Hello Merlijn,

I can replicate the problem and I can work around it by using a faster
internet connection.

At some point, tcp connections have to time out. I can only replicate
the issue when that timeout is reached. If you have the means to
relocate to a faster internet connection temporarily for pushing to
charmstore, please do. You might also try recompressing any items in the
charm using a higher compression level. xz -9 instead of gzip -3 or
whatever things may be using now.

We are aware this is a poor longterm solution. We are investigating
better solutions for uploads. As you've mentioned, resources will also
help the situation.

I am sorry that I do not have a better solution.

---

The timeout is apache setting `Timeout` which defaults to 300. It is
likely squid and/or haproxy also have a similar timeout. We cannot
support arbitrary large file uploads at arbitrary low speeds without
exhausting TCP connections. It seems we may wish to state 200MB as a
practical charm size limit.

** Changed in: charm (Ubuntu)
   Status: New => Confirmed

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1592822

Title:
  Charm push crashes when pushing 'big' charm.

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/charm/+bug/1592822/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

[Bug 1592822] Re: Charm push crashes when pushing 'big' charm.

[Uros Jovanovic]

The new charm publishing is in the charm package.

** Changed in: juju
 Assignee: Uros Jovanovic (uros-jovanovic) => (unassigned)

** Project changed: juju => charm (Ubuntu)

** Changed in: charm (Ubuntu)
 Assignee: (unassigned) => Uros Jovanovic (uros-jovanovic)

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1592822

Title:
  Charm push crashes when pushing 'big' charm.

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/charm/+bug/1592822/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs