Re: [openstack-dev] [Solum] [ Supporting swift downloads for operator languagepacks
A bit of a tangent, but it seems like the url would be to a public Swift system. I am unclear if a source git repo would be relevant but, assuming Swift would be optional, perhaps users could host catalog LP's in git or some other distribution mechanism and have a method by which solum could import them from the catalog into that deployment's object store. On Jun 17, 2015, at 1:58 PM, Fox, Kevin M kevin@pnnl.gov wrote: This question may be off on a tangent, or may be related. As part of the application catalog project, (http://apps.openstack.org/) we're trying to provide globally accessible resources that can be easily consumed in OpenStack Clouds. How would these global Language Packs fit in? Would the url record in the app catalog be required to point to an Internet facing public Swift system then? Or, would it point to the source git repo that Solum would use to generate the LP still? Thanks, Kevin From: Randall Burt [randall.b...@rackspace.com] Sent: Wednesday, June 17, 2015 11:38 AM To: OpenStack Development Mailing List (not for usage questions) Subject: Re: [openstack-dev] [Solum] Supporting swift downloads for operatorlanguagepacks Yes. If an operator wants to make their LP publicly available outside of Solum, I was thinking they could just make GET's on the container public. That being said, I'm unsure if this is realistically do-able if you still have to have an authenticated tenant to access the objects. Scratch that; http://blog.fsquat.net/?p=40 may be helpful. On Jun 17, 2015, at 1:27 PM, Adrian Otto adrian.o...@rackspace.com wrote: To be clear, Randall is referring to a swift container (directory). Murali has a good idea of attempting to use swift client first, as it has performance optimizations that can speed up the process more than naive file transfer tools. I did mention to him that wget does have a retiree feature, and that we could see about using curl instead to allow for chunked encoding as additional optimizations. Randall, are you suggesting that we could use swift client for both private and public LP uses? That sounds like a good suggestion to me. Adrian On Jun 17, 2015, at 11:10 AM, Randall Burt randall.b...@rackspace.com wrote: Can't an operator make the target container public therefore removing the need for multiple access strategies? Original message From: Murali Allada Date:06/17/2015 11:41 AM (GMT-06:00) To: OpenStack Development Mailing List (not for usage questions) Subject: [openstack-dev] [Solum] Supporting swift downloads for operator languagepacks Hello Solum Developers, When we were designing the operator languagepack feature for Solum, we wanted to make use of public urls to download operator LPs, such as those available for CDN backed swift containers we have at Rackspace, or any publicly accessible url. This would mean that when a user chooses to build applications on top of a languagepack provided by the operator, we use a url to 'wget' the LP image. Recently, we have started noticing a number of failures because of corrupted docker images downloaded using 'wget'. The docker images work fine when we download them manually with a swift client and use them. The corruption seem to be happening when we try to download a large image using 'wget' and there are dropped packets or intermittent network issues. My thinking is to start using the swift client to download operator LPs by default instead of wget. The swift client already implements retry logic, downloading large images in chunks, etc. This means we would not get the niceties of using publicly accessible urls. However, the feature will be more reliable and robust. The implementation would be as follows: • We'll use the existing service tenant configuration available in the solum config file to authenticate and store operator languagepacks using the swift client. We were using a different tenant to build and host LPs, but now that we require the tenants credentials in the config file, it's best to reuse the existing service tenant creds. Note: If we don't, we'll have 3 separate tenants to maintain. • Service tenant • Operator languagepack tenant • Global admin tenant • I'll keep the option to download the operator languagepacks from a publicly available url. I'll allow operators to choose which method they want to use by changing a setting in the solum config file. FYI: In my tests, I've noticed that downloading an image using the swift client is twice as fast as downloading the same image using 'wget' from a CDN url. Thanks, Murali __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org
Re: [openstack-dev] [Solum] [ Supporting swift downloads for operator languagepacks
This question may be off on a tangent, or may be related. As part of the application catalog project, (http://apps.openstack.org/) we're trying to provide globally accessible resources that can be easily consumed in OpenStack Clouds. How would these global Language Packs fit in? Would the url record in the app catalog be required to point to an Internet facing public Swift system then? Or, would it point to the source git repo that Solum would use to generate the LP still? Thanks, Kevin From: Randall Burt [randall.b...@rackspace.com] Sent: Wednesday, June 17, 2015 11:38 AM To: OpenStack Development Mailing List (not for usage questions) Subject: Re: [openstack-dev] [Solum] Supporting swift downloads for operatorlanguagepacks Yes. If an operator wants to make their LP publicly available outside of Solum, I was thinking they could just make GET's on the container public. That being said, I'm unsure if this is realistically do-able if you still have to have an authenticated tenant to access the objects. Scratch that; http://blog.fsquat.net/?p=40 may be helpful. On Jun 17, 2015, at 1:27 PM, Adrian Otto adrian.o...@rackspace.com wrote: To be clear, Randall is referring to a swift container (directory). Murali has a good idea of attempting to use swift client first, as it has performance optimizations that can speed up the process more than naive file transfer tools. I did mention to him that wget does have a retiree feature, and that we could see about using curl instead to allow for chunked encoding as additional optimizations. Randall, are you suggesting that we could use swift client for both private and public LP uses? That sounds like a good suggestion to me. Adrian On Jun 17, 2015, at 11:10 AM, Randall Burt randall.b...@rackspace.com wrote: Can't an operator make the target container public therefore removing the need for multiple access strategies? Original message From: Murali Allada Date:06/17/2015 11:41 AM (GMT-06:00) To: OpenStack Development Mailing List (not for usage questions) Subject: [openstack-dev] [Solum] Supporting swift downloads for operator languagepacks Hello Solum Developers, When we were designing the operator languagepack feature for Solum, we wanted to make use of public urls to download operator LPs, such as those available for CDN backed swift containers we have at Rackspace, or any publicly accessible url. This would mean that when a user chooses to build applications on top of a languagepack provided by the operator, we use a url to 'wget' the LP image. Recently, we have started noticing a number of failures because of corrupted docker images downloaded using 'wget'. The docker images work fine when we download them manually with a swift client and use them. The corruption seem to be happening when we try to download a large image using 'wget' and there are dropped packets or intermittent network issues. My thinking is to start using the swift client to download operator LPs by default instead of wget. The swift client already implements retry logic, downloading large images in chunks, etc. This means we would not get the niceties of using publicly accessible urls. However, the feature will be more reliable and robust. The implementation would be as follows: • We'll use the existing service tenant configuration available in the solum config file to authenticate and store operator languagepacks using the swift client. We were using a different tenant to build and host LPs, but now that we require the tenants credentials in the config file, it's best to reuse the existing service tenant creds. Note: If we don't, we'll have 3 separate tenants to maintain. • Service tenant • Operator languagepack tenant • Global admin tenant • I'll keep the option to download the operator languagepacks from a publicly available url. I'll allow operators to choose which method they want to use by changing a setting in the solum config file. FYI: In my tests, I've noticed that downloading an image using the swift client is twice as fast as downloading the same image using 'wget' from a CDN url. Thanks, Murali __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development
Re: [openstack-dev] [Solum] Supporting swift downloads for operator languagepacks
To be clear, Randall is referring to a swift container (directory). Murali has a good idea of attempting to use swift client first, as it has performance optimizations that can speed up the process more than naive file transfer tools. I did mention to him that wget does have a retiree feature, and that we could see about using curl instead to allow for chunked encoding as additional optimizations. Randall, are you suggesting that we could use swift client for both private and public LP uses? That sounds like a good suggestion to me. Adrian On Jun 17, 2015, at 11:10 AM, Randall Burt randall.b...@rackspace.commailto:randall.b...@rackspace.com wrote: Can't an operator make the target container public therefore removing the need for multiple access strategies? Original message From: Murali Allada Date:06/17/2015 11:41 AM (GMT-06:00) To: OpenStack Development Mailing List (not for usage questions) Subject: [openstack-dev] [Solum] Supporting swift downloads for operator languagepacks Hello Solum Developers, When we were designing the operator languagepack feature for Solum, we wanted to make use of public urls to download operator LPs, such as those available for CDN backed swift containers we have at Rackspace, or any publicly accessible url. This would mean that when a user chooses to build applications on top of a languagepack provided by the operator, we use a url to 'wget' the LP image. Recently, we have started noticing a number of failures because of corrupted docker images downloaded using 'wget'. The docker images work fine when we download them manually with a swift client and use them. The corruption seem to be happening when we try to download a large image using 'wget' and there are dropped packets or intermittent network issues. My thinking is to start using the swift client to download operator LPs by default instead of wget. The swift client already implements retry logic, downloading large images in chunks, etc. This means we would not get the niceties of using publicly accessible urls. However, the feature will be more reliable and robust. The implementation would be as follows: * We'll use the existing service tenant configuration available in the solum config file to authenticate and store operator languagepacks using the swift client. We were using a different tenant to build and host LPs, but now that we require the tenants credentials in the config file, it's best to reuse the existing service tenant creds. Note: If we don't, we'll have 3 separate tenants to maintain. * Service tenant * Operator languagepack tenant * Global admin tenant * I'll keep the option to download the operator languagepacks from a publicly available url. I'll allow operators to choose which method they want to use by changing a setting in the solum config file. FYI: In my tests, I've noticed that downloading an image using the swift client is twice as fast as downloading the same image using 'wget' from a CDN url. Thanks, Murali __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.orgmailto:openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Solum] Supporting swift downloads for operator languagepacks
Can't an operator make the target container public therefore removing the need for multiple access strategies? Original message From: Murali Allada Date:06/17/2015 11:41 AM (GMT-06:00) To: OpenStack Development Mailing List (not for usage questions) Subject: [openstack-dev] [Solum] Supporting swift downloads for operator languagepacks Hello Solum Developers, When we were designing the operator languagepack feature for Solum, we wanted to make use of public urls to download operator LPs, such as those available for CDN backed swift containers we have at Rackspace, or any publicly accessible url. This would mean that when a user chooses to build applications on to??p of a languagepack provided by the operator, we use a url to 'wget' the LP image. Recently, we have started noticing a number of failures because of corrupted docker images downloaded using 'wget'. The docker images work fine when we download them manually with a swift client and use them. The corruption seem to be happening when we try to download a large image using 'wget' and there are dropped packets or intermittent network issues. My thinking is to start using the swift client to download operator LPs by default instead of wget. The swift client already implements retry logic, downloading large images in chunks, etc. This means we would not get the niceties of using publicly accessible urls. However, the feature will be more reliable and robust. The implementation would be as follows: * ?We'll use the existing service tenant configuration available in the solum config file to authenticate and store operator languagepacks using the swift client. We were using a different tenant to build and host LPs, but now that we require the tenants credentials in the config file, it's best to reuse the existing service tenant creds. Note: If we don't, we'll have 3 separate tenants to maintain. * ?Service tenant * Operator languagepack tenant * Global admin tenant * I'll keep the option to download the operator languagepacks from a publicly available url. I'll allow operators to choose which method they want to use by changing a setting in the solum config file. FYI: In my tests, I've noticed that downloading an image using the swift client is twice as fast as downloading the same image using 'wget' from a CDN url. Thanks, Murali __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Solum] Supporting swift downloads for operator languagepacks
Yes. If an operator wants to make their LP publicly available outside of Solum, I was thinking they could just make GET's on the container public. That being said, I'm unsure if this is realistically do-able if you still have to have an authenticated tenant to access the objects. Scratch that; http://blog.fsquat.net/?p=40 may be helpful. On Jun 17, 2015, at 1:27 PM, Adrian Otto adrian.o...@rackspace.com wrote: To be clear, Randall is referring to a swift container (directory). Murali has a good idea of attempting to use swift client first, as it has performance optimizations that can speed up the process more than naive file transfer tools. I did mention to him that wget does have a retiree feature, and that we could see about using curl instead to allow for chunked encoding as additional optimizations. Randall, are you suggesting that we could use swift client for both private and public LP uses? That sounds like a good suggestion to me. Adrian On Jun 17, 2015, at 11:10 AM, Randall Burt randall.b...@rackspace.com wrote: Can't an operator make the target container public therefore removing the need for multiple access strategies? Original message From: Murali Allada Date:06/17/2015 11:41 AM (GMT-06:00) To: OpenStack Development Mailing List (not for usage questions) Subject: [openstack-dev] [Solum] Supporting swift downloads for operator languagepacks Hello Solum Developers, When we were designing the operator languagepack feature for Solum, we wanted to make use of public urls to download operator LPs, such as those available for CDN backed swift containers we have at Rackspace, or any publicly accessible url. This would mean that when a user chooses to build applications on top of a languagepack provided by the operator, we use a url to 'wget' the LP image. Recently, we have started noticing a number of failures because of corrupted docker images downloaded using 'wget'. The docker images work fine when we download them manually with a swift client and use them. The corruption seem to be happening when we try to download a large image using 'wget' and there are dropped packets or intermittent network issues. My thinking is to start using the swift client to download operator LPs by default instead of wget. The swift client already implements retry logic, downloading large images in chunks, etc. This means we would not get the niceties of using publicly accessible urls. However, the feature will be more reliable and robust. The implementation would be as follows: • We'll use the existing service tenant configuration available in the solum config file to authenticate and store operator languagepacks using the swift client. We were using a different tenant to build and host LPs, but now that we require the tenants credentials in the config file, it's best to reuse the existing service tenant creds. Note: If we don't, we'll have 3 separate tenants to maintain. • Service tenant • Operator languagepack tenant • Global admin tenant • I'll keep the option to download the operator languagepacks from a publicly available url. I'll allow operators to choose which method they want to use by changing a setting in the solum config file. FYI: In my tests, I've noticed that downloading an image using the swift client is twice as fast as downloading the same image using 'wget' from a CDN url. Thanks, Murali __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [Solum] Supporting swift downloads for operator languagepacks
Hello Solum Developers, When we were designing the operator languagepack feature for Solum, we wanted to make use of public urls to download operator LPs, such as those available for CDN backed swift containers we have at Rackspace, or any publicly accessible url. This would mean that when a user chooses to build applications on to??p of a languagepack provided by the operator, we use a url to 'wget' the LP image. Recently, we have started noticing a number of failures because of corrupted docker images downloaded using 'wget'. The docker images work fine when we download them manually with a swift client and use them. The corruption seem to be happening when we try to download a large image using 'wget' and there are dropped packets or intermittent network issues. My thinking is to start using the swift client to download operator LPs by default instead of wget. The swift client already implements retry logic, downloading large images in chunks, etc. This means we would not get the niceties of using publicly accessible urls. However, the feature will be more reliable and robust. The implementation would be as follows: * ?We'll use the existing service tenant configuration available in the solum config file to authenticate and store operator languagepacks using the swift client. We were using a different tenant to build and host LPs, but now that we require the tenants credentials in the config file, it's best to reuse the existing service tenant creds. Note: If we don't, we'll have 3 separate tenants to maintain. * ?Service tenant * Operator languagepack tenant * Global admin tenant * I'll keep the option to download the operator languagepacks from a publicly available url. I'll allow operators to choose which method they want to use by changing a setting in the solum config file. FYI: In my tests, I've noticed that downloading an image using the swift client is twice as fast as downloading the same image using 'wget' from a CDN url. Thanks, Murali __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
