Re: [openstack-dev] [javascript] [infra] NPM Mirrors (IMPORTANT)

[Michael Krotscheck]

We appear to be encountering an intermittent server issue - that is to say,
some of the time the npm request for either a json file or a tarball will
terminate prematurely, which we believe is due to a request timeout at the
client side. We'll be temporarily switching back to the public registry
while we investigate. The patch in question is this one:
https://review.openstack.org/#/c/318875/

Michael

On Thu, May 19, 2016 at 6:21 AM Michael Krotscheck 
wrote:

> Well, my current theory is that npm's default timeout and retries are set
> too permissively for cold AFS caches (2 retries, 10 second mintimeout, 1
> minute maxtimeout). We _just_ landed this patch (
> https://review.openstack.org/#/c/318279/) which increases both the # of
> retries and the timeouts, which would address that issue.
>
> We've got two choices- revert the mirror configuration, or wait for this
> patch to be propagated to our slaves. Both of them will require a dib run,
> which happens once a day, so neither fix will be available until tomorrow.
> Frankly, I'd prefer to move forward than back, so I'm going to check in the
> infra channel to see if we can trigger that run manually.
>
> Michael
>
> On Thu, May 19, 2016 at 5:41 AM Vitaly Kramskikh 
> wrote:
>
>> It seems this issue started to affect us on OpenStack infra about 13
>> hours ago. The main 2 reasons of failing tests are "Unexpected end of
>> input"/"registry error parsing json" (example
>> )
>> and "Error: socket hang up" (example
>> ).
>> See https://review.openstack.org/#/c/315119/ for a long list of similar
>> failures (though almost every gate-fuel-ui-npm-run-lint run now fails).
>>
>> What should we do?
>>
>> 2016-05-16 16:58 GMT+03:00 Michael Krotscheck :
>>
>>> Hey there, Vitaly-
>>>
>>> I suspect that the issue you're encountering is actually a
>>> cross-atlantic lag, combined with the Mirror's AFS cache warming up. As of
>>> this morning, fuel-ui seems to be installing fine from dfw.rax, though
>>> you may run into similar issues with other mirrors until those caches warm
>>> up.
>>>
>>> Michael
>>>
>>> On Thu, May 12, 2016 at 4:10 AM Vitaly Kramskikh <
>>> vkramsk...@mirantis.com> wrote:
>>>
 Hi, Michael,

 I randomly get "error parsing json" for fuel-ui
  project:
 http://paste.openstack.org/show/496871/. Got such errors 2 times out
 of 5.

 2016-05-11 22:07 GMT+03:00 Michael Krotscheck :

> Hello everyone!
>
> We've recently added NPM mirrors to our infrastructure, and are about
> to turn them on. Before that happens, however, we'd like to get a sanity
> check from impacted projects to make sure that we don't wedge your gate.
>
> If you are in charge of a project that invokes `npm install` during
> any of its gate jobs, then please invoke the following commands at your
> project root.
>
> echo "registry=http://mirror.dfw.rax.openstack.org/npm/; >> .npmrc
> rm -rf ./node_modules/
> rm -rf ~/.npm/
> npm install
>
> If you encounter an error, put it in paste.openstack.org and reply to
> this thread. If not, great! Delete the .npmrc file and go on your merry 
> way.
>
> Have a great day!
>
> Michael
>
>
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe:
> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>


 --
 Vitaly Kramskikh,
 Fuel UI Tech Lead,
 Mirantis, Inc.

 __
 OpenStack Development Mailing List (not for usage questions)
 Unsubscribe:
 openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

>>>
>>>
>>> __
>>> OpenStack Development Mailing List (not for usage questions)
>>> Unsubscribe:
>>> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>
>>>
>>
>>
>> --
>> Vitaly Kramskikh,
>> Fuel UI Tech Lead,
>> Mirantis, Inc.
>> __
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe:
>> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>

Re: [openstack-dev] [javascript] [infra] NPM Mirrors (IMPORTANT)

[Michael Krotscheck]

Well, my current theory is that npm's default timeout and retries are set
too permissively for cold AFS caches (2 retries, 10 second mintimeout, 1
minute maxtimeout). We _just_ landed this patch (
https://review.openstack.org/#/c/318279/) which increases both the # of
retries and the timeouts, which would address that issue.

We've got two choices- revert the mirror configuration, or wait for this
patch to be propagated to our slaves. Both of them will require a dib run,
which happens once a day, so neither fix will be available until tomorrow.
Frankly, I'd prefer to move forward than back, so I'm going to check in the
infra channel to see if we can trigger that run manually.

Michael

On Thu, May 19, 2016 at 5:41 AM Vitaly Kramskikh 
wrote:

> It seems this issue started to affect us on OpenStack infra about 13 hours
> ago. The main 2 reasons of failing tests are "Unexpected end of
> input"/"registry error parsing json" (example
> )
> and "Error: socket hang up" (example
> ).
> See https://review.openstack.org/#/c/315119/ for a long list of similar
> failures (though almost every gate-fuel-ui-npm-run-lint run now fails).
>
> What should we do?
>
> 2016-05-16 16:58 GMT+03:00 Michael Krotscheck :
>
>> Hey there, Vitaly-
>>
>> I suspect that the issue you're encountering is actually a cross-atlantic
>> lag, combined with the Mirror's AFS cache warming up. As of this morning,
>> fuel-ui seems to be installing fine from dfw.rax, though you may run
>> into similar issues with other mirrors until those caches warm up.
>>
>> Michael
>>
>> On Thu, May 12, 2016 at 4:10 AM Vitaly Kramskikh 
>> wrote:
>>
>>> Hi, Michael,
>>>
>>> I randomly get "error parsing json" for fuel-ui
>>>  project:
>>> http://paste.openstack.org/show/496871/. Got such errors 2 times out of
>>> 5.
>>>
>>> 2016-05-11 22:07 GMT+03:00 Michael Krotscheck :
>>>
 Hello everyone!

 We've recently added NPM mirrors to our infrastructure, and are about
 to turn them on. Before that happens, however, we'd like to get a sanity
 check from impacted projects to make sure that we don't wedge your gate.

 If you are in charge of a project that invokes `npm install` during any
 of its gate jobs, then please invoke the following commands at your project
 root.

 echo "registry=http://mirror.dfw.rax.openstack.org/npm/; >> .npmrc
 rm -rf ./node_modules/
 rm -rf ~/.npm/
 npm install

 If you encounter an error, put it in paste.openstack.org and reply to
 this thread. If not, great! Delete the .npmrc file and go on your merry 
 way.

 Have a great day!

 Michael



 __
 OpenStack Development Mailing List (not for usage questions)
 Unsubscribe:
 openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


>>>
>>>
>>> --
>>> Vitaly Kramskikh,
>>> Fuel UI Tech Lead,
>>> Mirantis, Inc.
>>>
>>> __
>>> OpenStack Development Mailing List (not for usage questions)
>>> Unsubscribe:
>>> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>
>>
>> __
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe:
>> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>>
>
>
> --
> Vitaly Kramskikh,
> Fuel UI Tech Lead,
> Mirantis, Inc.
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [javascript] [infra] NPM Mirrors (IMPORTANT)

[Vitaly Kramskikh]

It seems this issue started to affect us on OpenStack infra about 13 hours
ago. The main 2 reasons of failing tests are "Unexpected end of
input"/"registry error parsing json" (example
)
and "Error: socket hang up" (example
).
See https://review.openstack.org/#/c/315119/ for a long list of similar
failures (though almost every gate-fuel-ui-npm-run-lint run now fails).

What should we do?

2016-05-16 16:58 GMT+03:00 Michael Krotscheck :

> Hey there, Vitaly-
>
> I suspect that the issue you're encountering is actually a cross-atlantic
> lag, combined with the Mirror's AFS cache warming up. As of this morning,
> fuel-ui seems to be installing fine from dfw.rax, though you may run into
> similar issues with other mirrors until those caches warm up.
>
> Michael
>
> On Thu, May 12, 2016 at 4:10 AM Vitaly Kramskikh 
> wrote:
>
>> Hi, Michael,
>>
>> I randomly get "error parsing json" for fuel-ui
>>  project:
>> http://paste.openstack.org/show/496871/. Got such errors 2 times out of
>> 5.
>>
>> 2016-05-11 22:07 GMT+03:00 Michael Krotscheck :
>>
>>> Hello everyone!
>>>
>>> We've recently added NPM mirrors to our infrastructure, and are about to
>>> turn them on. Before that happens, however, we'd like to get a sanity check
>>> from impacted projects to make sure that we don't wedge your gate.
>>>
>>> If you are in charge of a project that invokes `npm install` during any
>>> of its gate jobs, then please invoke the following commands at your project
>>> root.
>>>
>>> echo "registry=http://mirror.dfw.rax.openstack.org/npm/; >> .npmrc
>>> rm -rf ./node_modules/
>>> rm -rf ~/.npm/
>>> npm install
>>>
>>> If you encounter an error, put it in paste.openstack.org and reply to
>>> this thread. If not, great! Delete the .npmrc file and go on your merry way.
>>>
>>> Have a great day!
>>>
>>> Michael
>>>
>>>
>>>
>>> __
>>> OpenStack Development Mailing List (not for usage questions)
>>> Unsubscribe:
>>> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>
>>>
>>
>>
>> --
>> Vitaly Kramskikh,
>> Fuel UI Tech Lead,
>> Mirantis, Inc.
>> __
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe:
>> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>


-- 
Vitaly Kramskikh,
Fuel UI Tech Lead,
Mirantis, Inc.
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [javascript] [infra] NPM Mirrors (IMPORTANT)

[Michael Krotscheck]

Hey there, Vitaly-

I suspect that the issue you're encountering is actually a cross-atlantic
lag, combined with the Mirror's AFS cache warming up. As of this morning,
fuel-ui seems to be installing fine from dfw.rax, though you may run into
similar issues with other mirrors until those caches warm up.

Michael

On Thu, May 12, 2016 at 4:10 AM Vitaly Kramskikh 
wrote:

> Hi, Michael,
>
> I randomly get "error parsing json" for fuel-ui
>  project:
> http://paste.openstack.org/show/496871/. Got such errors 2 times out of
> 5.
>
> 2016-05-11 22:07 GMT+03:00 Michael Krotscheck :
>
>> Hello everyone!
>>
>> We've recently added NPM mirrors to our infrastructure, and are about to
>> turn them on. Before that happens, however, we'd like to get a sanity check
>> from impacted projects to make sure that we don't wedge your gate.
>>
>> If you are in charge of a project that invokes `npm install` during any
>> of its gate jobs, then please invoke the following commands at your project
>> root.
>>
>> echo "registry=http://mirror.dfw.rax.openstack.org/npm/; >> .npmrc
>> rm -rf ./node_modules/
>> rm -rf ~/.npm/
>> npm install
>>
>> If you encounter an error, put it in paste.openstack.org and reply to
>> this thread. If not, great! Delete the .npmrc file and go on your merry way.
>>
>> Have a great day!
>>
>> Michael
>>
>>
>> __
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe:
>> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>>
>
>
> --
> Vitaly Kramskikh,
> Fuel UI Tech Lead,
> Mirantis, Inc.
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [javascript] [infra] NPM Mirrors (IMPORTANT)

[Michael Krotscheck]

Thanks, Vitaly- I'll add it to the list of things to investigate. I suspect
it might be a cold cache, however trusting my gut when it comes to
javascript related things has burned me in the past.

Michael

On Thu, May 12, 2016 at 4:10 AM Vitaly Kramskikh 
wrote:

> Hi, Michael,
>
> I randomly get "error parsing json" for fuel-ui
>  project:
> http://paste.openstack.org/show/496871/. Got such errors 2 times out of
> 5.
>
> 2016-05-11 22:07 GMT+03:00 Michael Krotscheck :
>
>> Hello everyone!
>>
>> We've recently added NPM mirrors to our infrastructure, and are about to
>> turn them on. Before that happens, however, we'd like to get a sanity check
>> from impacted projects to make sure that we don't wedge your gate.
>>
>> If you are in charge of a project that invokes `npm install` during any
>> of its gate jobs, then please invoke the following commands at your project
>> root.
>>
>> echo "registry=http://mirror.dfw.rax.openstack.org/npm/; >> .npmrc
>> rm -rf ./node_modules/
>> rm -rf ~/.npm/
>> npm install
>>
>> If you encounter an error, put it in paste.openstack.org and reply to
>> this thread. If not, great! Delete the .npmrc file and go on your merry way.
>>
>> Have a great day!
>>
>> Michael
>>
>>
>> __
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe:
>> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>>
>
>
> --
> Vitaly Kramskikh,
> Fuel UI Tech Lead,
> Mirantis, Inc.
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [javascript] [infra] NPM Mirrors (IMPORTANT)

[Vitaly Kramskikh]

Hi, Michael,

I randomly get "error parsing json" for fuel-ui
 project:
http://paste.openstack.org/show/496871/. Got such errors 2 times out of 5.

2016-05-11 22:07 GMT+03:00 Michael Krotscheck :

> Hello everyone!
>
> We've recently added NPM mirrors to our infrastructure, and are about to
> turn them on. Before that happens, however, we'd like to get a sanity check
> from impacted projects to make sure that we don't wedge your gate.
>
> If you are in charge of a project that invokes `npm install` during any of
> its gate jobs, then please invoke the following commands at your project
> root.
>
> echo "registry=http://mirror.dfw.rax.openstack.org/npm/; >> .npmrc
> rm -rf ./node_modules/
> rm -rf ~/.npm/
> npm install
>
> If you encounter an error, put it in paste.openstack.org and reply to
> this thread. If not, great! Delete the .npmrc file and go on your merry way.
>
> Have a great day!
>
> Michael
>
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>


-- 
Vitaly Kramskikh,
Fuel UI Tech Lead,
Mirantis, Inc.
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [javascript] [infra] NPM Mirrors (IMPORTANT)

[Michael Krotscheck]

Hello everyone!

We've recently added NPM mirrors to our infrastructure, and are about to
turn them on. Before that happens, however, we'd like to get a sanity check
from impacted projects to make sure that we don't wedge your gate.

If you are in charge of a project that invokes `npm install` during any of
its gate jobs, then please invoke the following commands at your project
root.

echo "registry=http://mirror.dfw.rax.openstack.org/npm/; >> .npmrc
rm -rf ./node_modules/
rm -rf ~/.npm/
npm install

If you encounter an error, put it in paste.openstack.org and reply to this
thread. If not, great! Delete the .npmrc file and go on your merry way.

Have a great day!

Michael
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev