Re: [openstack-dev] [networking-odl][networking-bgpvpn][Telemetry] all requirement updates are currently blocked
On Mon, Sep 10 2018, Tony Breeds wrote:
> It looks like in August this was already setup
> https://review.openstack.org/#/c/591682/
> So releases going forward will be on pypi.
>
> Julien, Do you mind me arranging for at least the following versions to
> be published to pypi?
>
> [tony@thor ceilometer]$ for branch in origin/stable/{ocata,pike,queens,rocky}
> ; do printf "%-25s: %s\n" $branch "$(git describe --abbrev=0 $branch)" ; done
> origin/stable/ocata : 8.1.5
> origin/stable/pike : 9.0.6
> origin/stable/queens : 10.0.1
> origin/stable/rocky : 11.0.0
Sure, go ahead!
--
Julien Danjou
;; Free Software hacker
;; https://julien.danjou.info
signature.asc
Description: PGP signature
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [networking-odl][networking-bgpvpn][Telemetry] all requirement updates are currently blocked
On Fri, Sep 07 2018, Tony Breeds wrote: > On Thu, Sep 06, 2018 at 01:33:12PM +0300, Michel Peterson wrote: > >> I remember that before landing the problematic patch [1] there was some >> discussion around it. Basically the problem was not n-odl but ceilometer >> not being in pypi, but we never foresaw this problem. >> >> Now that the problem is so critical, the question is how can we, from the >> n-odl team, help in fixing this? I am open to help in any effort that >> involves n-odl or any other project. > > As other have pointed out we can just ask the Telemetry team (PTL on CC) > why we can't publish ceilometer to pypi? You can, I've already said +1 on a review a few weeks ago. :) > https://pypi.org/project/ceilometer/ certainly seems to be the right > project. > > The crux of the code issue is: > from ceilometer.network.statistics import driver > > in networking_odl/ceilometer/network/statistics/opendaylight_v2/driver.py > > If this is supposed to be used they way you are how are prjects supposed > to get the ceilometer code? > > > > Yours Tony. > -- Julien Danjou // Free Software hacker // https://julien.danjou.info signature.asc Description: PGP signature __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [telemetry][ceilometer][monasca] Monasca publisher for Ceilometer
On Fri, Jun 22 2018, Bedyk, Witold wrote: > You've said lacking manpower is currently the main issue in Ceilometer which > stops you from accepting new publishers and that you don't want to add > maintenance overhead. > > We're offering help on maintaining the project. Oh cool, I misunderstood. I though you were offering only help for maintaining your publisher. That sounds great. We never worked with each other yet, so it'd be hard for us to add you right away to the core team — but we'd be more than happy to see how you can help and bring you in ASAP! -- Julien Danjou ;; Free Software hacker ;; https://julien.danjou.info signature.asc Description: PGP signature __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [telemetry][ceilometer][monasca] Monasca publisher for Ceilometer
On Fri, Jun 22 2018, Bedyk, Witold wrote: > I know it's not how it normally works, but in that case it seems to be a clear > win-win situation. I'm sorry, I really don't see what's the win for Ceilometer or its developer. Could you elaborate? -- Julien Danjou ;; Free Software hacker ;; https://julien.danjou.info signature.asc Description: PGP signature __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [telemetry][ceilometer][monasca] Monasca publisher for Ceilometer
On Wed, Jun 20 2018, Bedyk, Witold wrote: Hi Witek, It's not a transparency issue. It's a manpower issue. We are only 2 developers active on Ceilometer: me and Mehdi. Neither me nor Mehdi wants to maintain Monasca stuff; meaning, we don't want to spend time reviewing patches, having bug opened, or whatever. There's no interest for us in that. THe Prometheus publisher you mention has been written by Mehdi and we've approved it because it fits the roadmap of the Ceilometer developers that we are — and, again we're just two. We have other projects — such as Panko — that provide Ceilometer publishers and their code is in Panko, not in Ceilometer. It's totally possible and sane. Now, if you really, really, care that much about Ceilometer and its integration with Monasca, and if you have an amazing roadmap that'll make Ceilometer better and awesome, please, do start with that. Right now it just looks like more work for us with no gain. :( > could you please add some transparency to the decision process on which > publishers are acceptable and which not? Just two months ago you have added > new > Prometheus publisher. That's around the same time when our code was submitted > to review. > > We have delivered tested code and offer its maintenance. The code is > self-contained and does not touch Ceilometer core. If it's broken, just > Monasca > interface won't work. > > Please reconsider it again. > > Greetings > Witek > > >> -Original Message- >> From: Julien Danjou >> Sent: Mittwoch, 20. Juni 2018 14:07 >> To: Bedyk, Witold >> Cc: OpenStack Development Mailing List (not for usage questions) >> >> Subject: Re: [openstack-dev] [telemetry][ceilometer][monasca] Monasca >> publisher for Ceilometer >> >> On Wed, Jun 20 2018, Bedyk, Witold wrote: >> >> Same as Gordon. You should maintain that in your own repo. >> There's just no bandwidth in Ceilometer right now for things like that. >> :( >> >> > Hello Telemetry Team, >> > >> > any opinion on this? >> > >> > Best greetings >> > Witek >> > >> > >> >> -Original Message- >> >> From: Bedyk, Witold >> >> Sent: Mittwoch, 13. Juni 2018 10:28 >> >> To: OpenStack Development Mailing List (not for usage questions) >> >> >> >> Subject: [openstack-dev] [telemetry][ceilometer][monasca] Monasca >> >> publisher for Ceilometer >> >> >> >> Hello Telemetry Team, >> >> >> >> We would like to contribute a Monasca publisher to Ceilometer project >> >> [1] and add it to the list of currently supported transports [2]. >> >> The goal of the plugin is to send Ceilometer samples to Monasca API. >> >> >> >> I understand Gordon's concerns about adding maintenance overhead for >> >> Ceilometer team which he expressed in review but the code is pretty >> >> much self-contained and does not affect Ceilometer core. It's not our >> >> intention to shift the maintenance effort and Monasca team should >> >> still be responsible for this code. >> >> >> >> Adding this plugin will help in terms of interoperability of both >> >> projects and can be useful for wider parts of the OpenStack community. >> >> >> >> Please let me know your thoughts. I hope we can get this code merged. >> >> >> >> Cheers >> >> Witek >> >> >> >> >> >> [1] https://review.openstack.org/562400 >> >> [2] >> >> https://docs.openstack.org/ceilometer/latest/contributor/architecture >> >> .html >> >> #processing-the-data >> > >> >> -- >> Julien Danjou >> /* Free Software hacker >>https://julien.danjou.info */ > > -- Julien Danjou ;; Free Software hacker ;; https://julien.danjou.info signature.asc Description: PGP signature __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [telemetry][ceilometer][monasca] Monasca publisher for Ceilometer
On Wed, Jun 20 2018, Bedyk, Witold wrote: Same as Gordon. You should maintain that in your own repo. There's just no bandwidth in Ceilometer right now for things like that. :( > Hello Telemetry Team, > > any opinion on this? > > Best greetings > Witek > > >> -Original Message- >> From: Bedyk, Witold >> Sent: Mittwoch, 13. Juni 2018 10:28 >> To: OpenStack Development Mailing List (not for usage questions) >> >> Subject: [openstack-dev] [telemetry][ceilometer][monasca] Monasca >> publisher for Ceilometer >> >> Hello Telemetry Team, >> >> We would like to contribute a Monasca publisher to Ceilometer project [1] >> and add it to the list of currently supported transports [2]. >> The goal of the plugin is to send Ceilometer samples to Monasca API. >> >> I understand Gordon's concerns about adding maintenance overhead for >> Ceilometer team which he expressed in review but the code is pretty much >> self-contained and does not affect Ceilometer core. It's not our intention to >> shift the maintenance effort and Monasca team should still be responsible >> for this code. >> >> Adding this plugin will help in terms of interoperability of both projects >> and >> can be useful for wider parts of the OpenStack community. >> >> Please let me know your thoughts. I hope we can get this code merged. >> >> Cheers >> Witek >> >> >> [1] https://review.openstack.org/562400 >> [2] >> https://docs.openstack.org/ceilometer/latest/contributor/architecture.html >> #processing-the-data > -- Julien Danjou /* Free Software hacker https://julien.danjou.info */ signature.asc Description: PGP signature __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [stable] [tooz] [ceilometer] cmd2 without upper constraints causes errors in tox-py27
On Wed, May 30 2018, Elõd Illés wrote: > cmd2 says that: > > "Python 2.7 support is EOL > > Support for adding new features to the Python 2.7 release of |cmd2| was > discontinued on April 15, 2018. Bug fixes will be supported for Python 2.7 via > 0.8.x until August 31, 2018. > > Supporting Python 2 was an increasing burden on our limited resources. > Switching to support only Python 3 will allow us to clean up the codebase, > remove some cruft, and focus on developing new features." Erf. :( So the problem is that cmd2 is not a requirements in Ceilometer. It's pulled by cliff. As far as I can see, cliff latest version (2.11.0) announces it supports Python 2.7 (obviously) while also depends on cmd2>=0.6.7. Cliff needs to upper cap cmd2 if it wants to support Python 2. I see cliff has already set that requirements in its master branch, but no new version has been released. Releasing a new cliff version should fix that issue. -- Julien Danjou // Free Software hacker // https://julien.danjou.info signature.asc Description: PGP signature __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [stable] [tooz] [ceilometer] cmd2 without upper constraints causes errors in tox-py27
On Wed, May 30 2018, Elõd Illés wrote: > In the last two days the ceilometer [1] [2] [3] and tooz [4] [5] [6] tox-py27 > periodic stable jobs are failing. The root cause is the following: > * cmd2 released version 0.9.0, which requires python >=3.4 from now on. > These projects have comment in their tox.ini that they do not consume > upper-constraints.txt (in which there is an upper constraints for cmd2). > > My question is: could we use upper-constraints.txt on these projects as well, > or is there any reason why it isn't the case? > Of course an entry could be added to test-requirements.txt with "cmd2<0.9.0", > but wouldn't it be better to use the upper-constraints.txt? The question is: why cmd2 0.9.0 does not work and how do we fix that? -- Julien Danjou ;; Free Software hacker ;; https://julien.danjou.info signature.asc Description: PGP signature __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [oslo.db] innodb OPTIMIZE TABLE ?
On Tue, Apr 03 2018, Jay Pipes wrote: > Possibly, though it's questionable to use MySQL/InnoDB for storing transient > data that is deleted often like ceilometer samples and keystone tokens. A much > better solution is to use RDBMS partitioning so you can simply ALTER TABLE .. > DROP PARTITION those partitions that are no longer relevant (and don't even > bother DELETEing individual rows) or, in the case of Ceilometer samples, don't > use a traditional RDBMS for timeseries data at all... For the record, and because I imagine not everyone follows Ceilometer, this codes does not exist anymore in Queens. Ceilometer storage (and API) has been deprecated for 2 cycles already and removed last release. Feel free to continue discussing the problem, but you can ignore Ceilometer. :) -- Julien Danjou /* Free Software hacker https://julien.danjou.info */ signature.asc Description: PGP signature __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] 回复： [ceilometer] [gnocchi] keystone verification failed.
On Tue, Mar 20 2018, __ mango. wrote:
> I have configured the following
> export OS_PROJECT_DOMAIN_NAME=Default
> export OS_USER_DOMAIN_NAME=Default
> export OS_PROJECT_NAME=admin
> export OS_USERNAME=admin
> export OS_PASSWORD=admin
> export OS_AUTH_URL=http://controller:35357/v3
> export OS_IDENTITY_API_VERSION=3
> export OS_IMAGE_API_VERSION=2
>
> /etc/gnocchi/gnocchi.conf
> [DEFAULT]
> [api]
> auth_mode = keystone
You said in your mail that you were using basic as auth_mode and your
request here:
>> # gnocchi status --debug
>> REQ: curl -g -i -X GET http://localhost:8041/v1/status?details=False -H
>> "Authorization: {SHA1}d4daf1cf567f14f32dbc762154b3a281b4ea4c62" -H "Accept:
>> application/json, */*" -H "User-Agent: gnocchi keystoneauth1/3.1.0
>> python-requests/2.18.1 CPython/2.7.12"
>> Starting new HTTP connection (1): localhost
>> http://localhost:8041 "GET /v1/status?details=False HTTP/1.1" 401 114
>> RESP: [401] Content-Type: application/json Content-Length: 114
>> WWW-Authenticate: Keystone uri='http://controller:5000/v3' Connection:
>> Keep-Alive
>> RESP BODY: {"error": {"message": "The request you have made requires
>> authentication.", "code": 401, "title": "Unauthorized"}}
>> The request you have made requires authentication. (HTTP 401)
Indicates that the client is using basic auth mode. As Gordon already
replied:
https://gnocchi.xyz/gnocchiclient/shell.html#openstack-keystone-authentication
you're missing OS_AUTH_TYPE
Sigh.
--
Julien Danjou
/* Free Software hacker
https://julien.danjou.info */
signature.asc
Description: PGP signature
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [ceilometer] [gnocchi] keystone verification failed.
On Tue, Mar 20 2018, __ mango. wrote:
> hi，
> I have a question about the validation of gnocchi keystone.
> I run the following command, but it is not successful.(api.auth_mode :basic,
> basic mode can be
> # gnocchi status --debug
> REQ: curl -g -i -X GET http://localhost:8041/v1/status?details=False -H
> "Authorization: {SHA1}d4daf1cf567f14f32dbc762154b3a281b4ea4c62" -H "Accept:
> application/json, */*" -H "User-Agent: gnocchi keystoneauth1/3.1.0
> python-requests/2.18.1 CPython/2.7.12"
> Starting new HTTP connection (1): localhost
> http://localhost:8041 "GET /v1/status?details=False HTTP/1.1" 401 114
> RESP: [401] Content-Type: application/json Content-Length: 114
> WWW-Authenticate: Keystone uri='http://controller:5000/v3' Connection:
> Keep-Alive
> RESP BODY: {"error": {"message": "The request you have made requires
> authentication.", "code": 401, "title": "Unauthorized"}}
> The request you have made requires authentication. (HTTP 401)
You need to be authed as "admin" to get the status.
--
Julien Danjou
/* Free Software hacker
https://julien.danjou.info */
signature.asc
Description: PGP signature
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [gnocchi] gnocchi-keystone verification failed.
On Thu, Mar 15 2018, __ mango. wrote:
> I have a question about the validation of gnocchi keystone.
There's no question in your message.
> I run the following command, but it is not successful.(api.auth_mode :basic,
> basic mode can be successful)
>
> # gnocchi status --debug
> REQ: curl -g -i -X GET http://localhost:8041/v1/status?details=False
> -H "Authorization: {SHA1}d4daf1cf567f14f32dbc762154b3a281b4ea4c62" -H
> "Accept: application/json, */*" -H "User-Agent: gnocchi
> keystoneauth1/3.1.0 python-requests/2.18.1 CPython/2.7.12"
There's no token in this request so Keystone auth won't work. You did
not set the environment variable OS_* correctly.
--
Julien Danjou
# Free Software hacker
# https://julien.danjou.info
signature.asc
Description: PGP signature
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] OpenStack - Install gnocchi error.
On Wed, Mar 14 2018, __ mango. wrote: > After the installation (apt-get install gnocchi-api gnocchi- gnocchiclient), > When I tried to launch the gnocc-api, I got the message. >Failed to start gnocchi-api. Service: Unit gnocchi-api. Service not found. > I checked /etc/init.d and there is no script gnocchi-api (although > gnocchi-metricd is, and it's working properly). > > Why is that? Please help me to solve it. Thank you. Sounds like a Ubuntu packaging issue potentially. Did you try reading the documentation? https://gnocchi.xyz/operating.html#running-api-as-a-wsgi-application -- Julien Danjou ;; Free Software hacker ;; https://julien.danjou.info signature.asc Description: PGP signature __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] OpenStack - Install gnocchi error.
On Tue, Mar 13 2018, __ mango. wrote: > hi, > I refer to: > https://docs.openstack.org/ceilometer/pike/install/install-base-ubuntu.html > installation gnocchi, > unable to find gnocchi - API related services is this why? > Please help answer my question, thank you! Can you provide more details as what's your problem? thank you! -- Julien Danjou // Free Software hacker // https://julien.danjou.info signature.asc Description: PGP signature __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] Switching to longer development cycles
On Thu, Dec 14 2017, Thierry Carrez wrote: > It takes time to get a feature merged (or any significant work done) in > OpenStack. It takes time to get reviews, we need to be careful about not > breaking all our users, etc. If you are a 20% time person, it's just > impossible to get something significant done within the timeframe of a > cycle, which leads to frustration as you have to get your stuff > re-discussed and re-prioritized at the start of the next cycle. > > I'm open to other ways to solve what is perceived by many as an > untenable cadence -- but so far the only suggestion I got on this thread > was that 20% people should really ask their manager to become 80% > people. And I don't see that happening. Yeah that does not seem realistic. I like the way you state the problem actually: "it's just impossible to get something significant done within the timeframe of a cycle, which leads to frustration as you have to get your stuff re-discussed and re-prioritized at the start of the next cycle". So the first thing that comes to mind is that having a longer release cycle will fix it. However, has anyone tried to understand the reasons why it is hard to impossible to do anything useful in a cycle, other than "it is too short"? -- Julien Danjou ;; Free Software hacker ;; https://julien.danjou.info signature.asc Description: PGP signature __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] Switching to longer development cycles
On Thu, Dec 14 2017, Thierry Carrez wrote: > - People who would like to get more involved (and become core > developers) but can't keep up with what's happening in their project or > reach the review activity necessary to become core developers. A number > of projects are struggling to recruit new core reviewers, so I think > reducing expectations and slowing down the pace could help there. Can you detail more how having a longer cycle will make people that spends 20% of their time on OpenStack "catch up" with the people that spends 80% of their time on OpenStack? I understand the problem but I don't see how the proposed solution solves it. -- Julien Danjou // Free Software hacker // https://julien.danjou.info signature.asc Description: PGP signature __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [ceilometer] about add transformer into libvirt
On Thu, Dec 14 2017, Jaze Lee wrote: > Oh, sorry. I found libvirt can not get cpu.util, cpu.delta, and > network/disk rate info directly > from inspector. So i mean, we can move this into compute pollster. > Before it sends to notifications.sample, > transform it. Then we can scale notification agent and do not have > flapping sample problems The plan is to actually make the TSDB (Gnocchi) do that job for us. -- Julien Danjou /* Free Software hacker https://julien.danjou.info */ signature.asc Description: PGP signature __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [ceilometer][gnocchi] some code not very understood
On Thu, Dec 07 2017, Jaze Lee wrote: > Yes. I misunderstand here. If there any exception in gnocchi api, it > won't store any measures. > Just after all resource, and all metrics are there, gnocchi will store > those measures. Am i right? Yes. -- Julien Danjou # Free Software hacker # https://julien.danjou.info signature.asc Description: PGP signature __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [ceilometer][gnocchi] some code not very understood
On Thu, Dec 07 2017, Jaze Lee wrote: >I think at line 416, we should remove some measures that already post at > 390. >Or may be i miss something? > > > https://github.com/openstack/ceilometer/blob/master/ceilometer/publisher/gnocchi.py#L416 The code at L416 is only executed if L390 raises an exception. So if L390 raises an exception, it has not complete. That's why it's done again here. -- Julien Danjou # Free Software hacker # https://julien.danjou.info signature.asc Description: PGP signature __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [ceilometer][cinder]about cinder volume usage
On Mon, Dec 04 2017, gordon chung wrote: > but at the end of the day, are you building for reality or for some > personal ideal scenario? :p Haha, you know that if I was building for an ideal scenario, most of Ceilometer would probably have never existed in this way. Half of Ceilometer is already hacking around OpenStack project limitation for the last 5 years. > is there a service that offers a stats endpoint? Outside of OpenStack? Plenty. :) -- Julien Danjou /* Free Software hacker https://julien.danjou.info */ signature.asc Description: PGP signature __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [ceilometer][cinder]about cinder volume usage
On Mon, Dec 04 2017, gordon chung wrote: > On 2017-12-04 05:30 AM, Jaze Lee wrote: >> Right now,we can get volume from central pollester. Then i think >> may be we can drop cinder volume usage. > > which metrics are you referring to? just for the record, except for > libvirt stats, the polling done by ceilometer is against the API which > may create additional load. Which is usually a problem because of the APIs, not Ceilometer. They're sometimes utterly slow for simple things and other times don't offer a way to retrieve what Ceilometer needs in a batch-y way. > in general, there's a preference that stats be pushed to ceilometer > rather than pulled. Yes, but for the wrong reasons. Having a component sending a batch of notification every hour without any configuration knob and granularity choice is a PITA for the users. Plus it ususally comes from a single point (of failure?). Ceilometer would do a better job than $project at getting this info, if said project(s) would offer a proper and performant API to retrieve them efficiently. -- Julien Danjou -- Free Software hacker -- https://julien.danjou.info signature.asc Description: PGP signature __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [ceilometer] remove gnocchi http interface?
On Mon, Dec 04 2017, Jaze Lee wrote: > Right now, the dispatch will cost much time in http request. And > keystone auth token cost much time. Although we can configure it to > noauth, then anyone can update gnocchi database which is not we want. > So what about to remove gnocchi http, and add a simple tcp socket, which > will send to gnocchi tcp server(which will also be created.). At > first, we think udp, … and then "anyone can update gnocchi database which is not you want." I'm not saying it's not a good idea, but your arguments don't sound right. :) There's already an issue here https://github.com/gnocchixyz/gnocchi/issues/496 that you opened. At that point, you need more than "suggesting", you need to provide plan and code if you want that to happen "soon". :) -- Julien Danjou /* Free Software hacker https://julien.danjou.info */ signature.asc Description: PGP signature __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [ceilometer] Retiring ceilometerclient
Hi, Now that the Ceilometer API is gone, we really don't need ceilometerclient anymore. I've proposed a set of patches to retire it: https://review.openstack.org/#/c/522183/ -- Julien Danjou ;; Free Software hacker ;; https://julien.danjou.info signature.asc Description: PGP signature __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [gnocchi]can we put the api on udp?
On Tue, Nov 21 2017, 李田清 wrote: > right now, ceilometer notification agent can send samples by udp > publisher.. > But gnocchi can only accept by rest api. Is there a way to use udp to > accept notification agent's samples that sending by udp? I'd suggest tracking this here since an issue just has been opened about it: https://github.com/gnocchixyz/gnocchi/issues/496 -- Julien Danjou # Free Software hacker # https://julien.danjou.info signature.asc Description: PGP signature __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [ceilometer] Time for API removal?
Hey there, We deprecated the Ceilometer API last year (Ocata) and our latest user survery shows than more than 50% of our users are now using Gnocchi or something else than the old deprecated storage methods. I'm starting to think it's time to stop carrying this dead code around. The biggest reason, other than cleaning our code base, is to stop having confusing options such as, e.g., "time-to-live" that make users being confused about where to configure the expiration of their metrics. The question, is there any compelling reason to keep this deprecated stuffs for one more cycle? -- Julien Danjou # Free Software hacker # https://julien.danjou.info signature.asc Description: PGP signature __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [gnocchi] Redis for storage backend
On Fri, Oct 13 2017, Yaguang Tang wrote: > I see the latest Gnocchi support using Redis as a storage backend, I am > testing performance of Redis and Ceph, it seems using Redis as storage > backend we can achieve more realtime metric > data, gnocchi status shows there is always few metric to process. > > Is Redis a recommend storage backend ? Redis is recommended as an incoming measures backend, not really as a storage – though it really depends on the size of your installation. Up until 4.0 version, Gnocchi process metrics every $metricd.metric_processing_delay seconds. With version 4.1 (to be released), the Redis incoming driver has a more realtime processing delay which avoids having to poll for incoming data. -- Julien Danjou // Free Software hacker // https://julien.danjou.info signature.asc Description: PGP signature __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Regarding Horizon panel of Gnoochi/Adoh Services
On Thu, Oct 12 2017, gordon chung wrote: > Sudheer, the other gnocchi devs can correct me if i'm wrong, but > supporting gnocchi in horizon is not on our roadmap. we will go to > grafana route since it's a common monitoring/metric interface. you are > welcome to try to incorporate grafana into horizon's plugin system if > it's possible (and you really need horizon+gnocchi) There are also plugins like: https://github.com/b3lab/safir_monitor_dashboard that supports Gnocchi IIRC. -- Julien Danjou // Free Software hacker // https://julien.danjou.info signature.asc Description: PGP signature __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all][oslo] Retiring openstack/pylockfile
On Mon, Oct 02 2017, Doug Hellmann wrote: > Or https://github.com/jazzband <https://github.com/jazzband> > > Now we need a project to list all of the organizations full of unmaintained > software... Who's up for maintaining that list? -- Julien Danjou ;; Free Software hacker ;; https://julien.danjou.info signature.asc Description: PGP signature __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] patches for simple typo fixes
On Mon, Sep 25 2017, Doug Hellmann wrote: >> I was trying to ignore the thread in the hopes it would die out quick. >> But torches and pitchforks all came out from the far corners, so I'm >> going to push back on that a bit. >> >> I'm not super clear why there is always so much outrage about these >> patches. They are fixing real things. When I encounter them, I just >> approve them to get them merged quickly and not backing up the review >> queue, using more CI later if they need rebasing. They are fixing real >> things. Maybe there is a CI cost, but the faster they are merged the >> less likely someone else is to propose it in the future, which keeps >> down the CI cost. And if we have a culture of just fixing typos later, >> then we spend less CI time on patches the first time around with 2 or 3 >> iterations catching typos. >> >> I think the concern is the ascribed motive for why people are putting >> these up. That's fine to feel that people are stat padding (and that too >> many things are driven off metrics). But, honestly, that's only >> important if we make it important. Contributor stats are always going to >> be pretty much junk stats. They are counting things to be the same which >> are wildly variable in meaning (number of patches, number of Lines of >> Code). >> >> My personal view is just merge things that fix things that are wrong, >> don't care why people are doing it. If it gets someone a discounted >> ticket somewhere, so be it. It's really not any skin off our back in the >> process. >> >> If people are deeply concerned about CI resources, step one is to get >> some better accounting into the existing system to see where resources >> are currently spent, and how we could ensure that time is fairly spread >> around to ensure maximum productivity by all developers. Well said, agreed. What I hate about this thread is that it's not the first one to pop up, and it does not help to get the real problem getting acknowledge, which IMHO is, that sometimes people spams Gerrit with small *wrong* patches. But as you said, if they are real fixes, just blame yourself for merging mistakes or typos in the first place. > I'm less concerned with the motivation of someone submitting the > patches than I am with their effect. Just like the situation we had > with the bug squash days a year or so ago, if we had a poorly timed > set of these trivial patches coming in at our feature freeze deadline, > it would be extremely disruptive. So to me the fact that we're > seeing them in large batches means we have people who are not fully > engaged with the community and don't understand the impact they're > having. My goal is to reach out and try to improve that engagement, > and try to help them become more fully constructive contributors. I think it's a good idea but I'm also pretty sure somebody already said that (and probably did that) last time we had this discussion. While it might or not (from my experience) improve this batch of contributors, the next ones are already waiting to spam next cycle. ;) -- Julien Danjou ;; Free Software hacker ;; https://julien.danjou.info signature.asc Description: PGP signature __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [horizon] A Monitoring panel plugin for Horizon
On Fri, Sep 15 2017, Esra Celik wrote: Hi Esra, > We published a new release of our Monitor Panel which uses Gnocchi api as the > backend [1]. > As Gokhan said, our cloud platform is still on Mitaka so we tested this new > release on Devstack, That sounds cool. > It seems quite faster than mongodb, so we also added the weekly and monthly > monitoring options. It is way faster than MongoDB indeed, which was a starting point for the project back then. :) > Do you have any other suggestions? I guess if you found everything you wanted in the Gnocchi API that's awesome. If there are feature you think you miss, please let us know! -- Julien Danjou # Free Software hacker # https://julien.danjou.info signature.asc Description: PGP signature __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Monasca] GridDB repository support
On Mon, Aug 28 2017, witold.be...@est.fujitsu.com wrote: > Can you give some reference to Gnocchi's performance measurements? > I have seen you presentation from the last Summit [1] but I cannot find > information about Gnocchi's write throughput. In the issues slide you note the > slow post API. The API responsiveness test shows that the API can handle > correctly up to 2500 instances. You collect around 60 metrics per instance and > use polling interval of 10 minutes. That would translate to: > > 2500 instances * 60 metrics/instance / 600 s = 250 metrics/s > > Are these assumptions correct? It really depends on the number of measures you include in each of your API requests. Ceilometer does very little batching. If you batch those measurements, you could achieve up to 120k measures/s in Gnocchi already 2 years ago¹. Considering the various improvement we did over that timeframe, you can likely go higher. And since it scales horizontally, you can double the request handling by adding another node. :) I'll probably try to do some benchmark one of these days, but I don't have proper hardware to do so for now. :) ¹ https://julien.danjou.info/blog/2015/gnocchi-benchmarks -- Julien Danjou -- Free Software hacker -- https://julien.danjou.info signature.asc Description: PGP signature __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Monasca] GridDB repository support
On Tue, Aug 22 2017, witold.be...@est.fujitsu.com wrote: Hi Witek, > thank you for your contribution. We indeed urgently need a scalable open > source TSDB. Did you look at Gnocchi? It's exactly that. http://gnocchi.xyz/ > In the recent weeks James Gu and his team have re-evaluated using Cassandra as > Monasca backend [1]. They have measured write throughput of 64 kmetrics/s when > storing 200 mil unique metric definitions and 50 bil measurements on the two > nodes cluster. It's way below what InfluxDB can on the single node, but that's > a number we can live with, I guess. Can GridDB perform better? > > What do the others think? Should we experiment and add support for different > backend drivers or rather choose one or two and concentrate on these? We don't have any recent comparison with InfluxDB but last time we tried, Gnocchi was already getting faster than it. And scalable horizontally, so… You might want to give it a try. -- Julien Danjou ;; Free Software hacker ;; https://julien.danjou.info signature.asc Description: PGP signature __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [release] [telemetry] Ceilometer stable/pike branch outlook
On Wed, Aug 16 2017, Eric S Berglund wrote: Hi Eric, > Is there an outlook for cutting a pike branch for ceilometer? > We currently can't run our 3rd party CI against pike without a pike > release branch and are deciding whether it's worth the time to > implement a workaround. AFAIU it's impossible to cut a branch for our projects and release a rc1 because of the release model we use. The release team does not allow us to do that. We need to release directly a stable version and cut a branch. I guess we'll do that in a couple of week, at release time. -- Julien Danjou // Free Software hacker // https://julien.danjou.info signature.asc Description: PGP signature __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [horizon] A Monitoring panel plugin for Horizon
On Tue, Aug 01 2017, Gökhan IŞIK (BİLGEM BTE) wrote: Hi, > We are happy to share with you that we developed an AngularJS based usage > monitoring panel for Horizon. > The panel visualizes the collected utilization data by Ceilometer. > > Source code is now available on github [1]. > > The plugin consists of 3 panels; a monitoring panel for users, a project > monitoring panel and a hypervisor monitoring panel for admins. > > A user can display utilization charts for the current project's instances, > export the metrics in csv format and launch alarms using the dashboard. > We have another service to capture these alarms and send notification e-mails > to the cloud users which is also available on github.com/b3lab. > An admin can display projects' utilization charts, export the metrics in csv > format in Project Monitor panel. And the Hypervisor Monitor panel is to > visualize SNMP based metrics collected from hypervisors. > > We are currently improving some features. > > Some screenshots published at our website [2]. We also uploaded two videos > showing some features of the plugin [3][4]. > > All comments and ideas are welcome! Did you build all of this against the deprecated Ceilometer API rather than Gnocchi? :( -- Julien Danjou ;; Free Software hacker ;; https://julien.danjou.info signature.asc Description: PGP signature __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [gnocchi][ceilometer] gnocchi-metricd error using redis as coordination
On Tue, Aug 08 2017, Yaguang Tang wrote:
Yes, this fix is included in Gnocchi 4.0.1 already.
> Thanks Along, finally I figure out that this is a bug and fixed by this
> commit
>
> commit e749b60f49a4a3b48cc5da67a797f717dd8cd01d
> Author: Julien Danjou <jul...@danjou.info>
> Date: Tue Jun 20 16:36:14 2017 +0200
>
> utils: use ASCII bytes as member id
>
> Tooz actually wants ASCII bytes and not random bytes.
>
> Fixes #130
>
> diff --git a/gnocchi/utils.py b/gnocchi/utils.py
> index f81d93e..7666711 100644
> --- a/gnocchi/utils.py
> +++ b/gnocchi/utils.py
> @@ -90,7 +90,7 @@ def _enable_coordination(coord):
>
>
> def get_coordinator_and_start(url):
> -my_id = uuid.uuid4().bytes
> +my_id = str(uuid.uuid4()).encode()
> coord = coordination.get_coordinator(url, my_id)
> _enable_coordination(coord)
> return coord, my_id
>
>
> On Mon, Aug 7, 2017 at 10:10 PM, Along Meng <alongm...@gmail.com> wrote:
>
>> From the log info,It shows that your 'node' maybe is not the valid str.
>> You can show the node name via 'print node' and try to call
>> str(node).encode('utf-8')
>> , identify does it can goes well.
>>
>> 2017-08-05 18:14:19.100 1329257 ERROR cotyledon._utils key =
>> str(node).encode('utf-8')
>> 2017-08-05 18:14:19.100 1329257 ERROR cotyledon._utils
>> UnicodeDecodeError: 'ascii' codec can't decode byte 0xde in position 4:
>> ordinal not in range(128)
>> 2017-08-05 18:14:19.100 1329257 ERROR cotyledon._utils
>>
>>
>>
>> On Sat, Aug 5, 2017 at 7:16 PM, Yaguang Tang <heut2...@gmail.com> wrote:
>>
>>> Hi gnocchi devs,
>>>
>>> I have an issue when using gnocchi 4.0, the storage backend is ceph, and
>>> tooz coordination is redis. currently gnocchi api in apache wsgi mode,
>>> only one controller node running gnocchi-metricd & gnocchi-statsd daemon.
>>> the error log of gnocchi-metricd is as follow.
>>>
>>>
>>>
>>> 2017-08-05 18:14:18.643 1329257 INFO gnocchi.storage.common.ceph [-] Ceph
>>> storage backend use 'cradox' python library
>>> 2017-08-05 18:14:18.654 1329257 INFO gnocchi.storage.common.ceph [-] Ceph
>>> storage backend use 'cradox' python library
>>> 2017-08-05 18:14:19.100 1329257 ERROR cotyledon._utils [-] Unhandled
>>> exception
>>> 2017-08-05 18:14:19.100 1329257 ERROR cotyledon._utils Traceback (most
>>> recent call last):
>>> 2017-08-05 18:14:19.100 1329257 ERROR cotyledon._utils File
>>> "/usr/lib/python2.7/site-packages/cotyledon/_utils.py", line 84, in
>>> exit_on_exception
>>> 2017-08-05 18:14:19.100 1329257 ERROR cotyledon._utils yield
>>> 2017-08-05 18:14:19.100 1329257 ERROR cotyledon._utils File
>>> "/usr/lib/python2.7/site-packages/cotyledon/_service.py", line 139, in
>>> _run
>>> 2017-08-05 18:14:19.100 1329257 ERROR cotyledon._utils self.run()
>>> 2017-08-05 18:14:19.100 1329257 ERROR cotyledon._utils File
>>> "/usr/lib/python2.7/site-packages/gnocchi/cli.py", line 120, in run
>>> 2017-08-05 18:14:19.100 1329257 ERROR cotyledon._utils
>>> self._configure()
>>> 2017-08-05 18:14:19.100 1329257 ERROR cotyledon._utils File
>>> "/usr/lib/python2.7/site-packages/tenacity/__init__.py", line 87, in
>>> wrapped_f
>>> 2017-08-05 18:14:19.100 1329257 ERROR cotyledon._utils return
>>> r.call(f, *args, **kw)
>>> 2017-08-05 18:14:19.100 1329257 ERROR cotyledon._utils File
>>> "/usr/lib/python2.7/site-packages/tenacity/__init__.py", line 177, in
>>> call
>>> 2017-08-05 18:14:19.100 1329257 ERROR cotyledon._utils return
>>> fut.result()
>>> 2017-08-05 18:14:19.100 1329257 ERROR cotyledon._utils File
>>> "/usr/lib/python2.7/site-packages/concurrent/futures/_base.py", line
>>> 396, in result
>>> 2017-08-05 18:14:19.100 1329257 ERROR cotyledon._utils return
>>> self.__get_result()
>>> 2017-08-05 18:14:19.100 1329257 ERROR cotyledon._utils File
>>> "/usr/lib/python2.7/site-packages/tenacity/__init__.py", line 159, in
>>> call
>>> 2017-08-05 18:14:19.100 1329257 ERROR cotyledon._utils result =
>>> fn(*args, **kwargs)
>>> 2017-08-05 18:14:19.100 1329257 ERROR cotyledon._utils File
>>> "/usr/lib/python2.7/site-packages/gnocchi/cli.py", line 193, in
>>> _configure
>>> 2017-08-05 18:14:19.100 1329257 ERROR cotyledon._utils self.GROUP_ID,
>>> partitions=200)
>
[openstack-dev] [oslo] Stepping down from oslo-core
Hi folks, I've not been reviewing or contributing to oslo.* stuff for a while and I don't intend to change that in the near future. It seems only fair to step down. As I'm currently the only maintainer of tooz, I'd still suggest to leave me on tooz-core though. :) Cheers, -- Julien Danjou # Free Software hacker # https://julien.danjou.info signature.asc Description: PGP signature __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [gnocchi] gnocchi-metricd couldn't start up with ceph backend
On Thu, Jul 06 2017, Hui Xiang wrote:
> The index storage mysql has gnocchi database and get two arhive related
> table.
> In addition, I am checking the gnocchi-upgrade code, didn't see it will
> create 'measure' object to metric storage anywhere, could it be somewhere
> wrong?
It's tested several times a day in our CI, so it really do work. :)
The error raised is PermissionDenied, are you sure that the cap listed
in the gnocchi Ceph user are enough?
(I'm no Ceph expert so it might not be related at all, just thinking out
loud)
> if not conf.skip_storage:
> s = storage.get_driver(conf)
> LOG.info("Upgrading storage %s", s)
> s.upgrade(index)
FYI this is what handles the storage initial creations of Ceph objects
or whatever is needed.
--
Julien Danjou
-- Free Software hacker
-- https://julien.danjou.info
signature.asc
Description: PGP signature
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [gnocchi] gnocchi-metricd couldn't start up with ceph backend
On Thu, Jul 06 2017, Hui Xiang wrote: > I am setting up OpenStack with Gnocchi with ceph as the storage backend, > after service gnocchi-metricd start, it always reported "PermissionError: > Failed to operate read op for oid measure", seems at the time metricd > start, it tried to list the measure object, however, there is none such > object, so does the permission show error? I didn't find any bug related, > could anyone help to tell me what am I missing? Thanks. Did you run gnocchi-upgrade? -- Julien Danjou ;; Free Software hacker ;; https://julien.danjou.info signature.asc Description: PGP signature __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [telemetry][ceilometer][gnocchi] RBAC for attributes in resource
On Fri, Jun 23 2017, Deepthi V V wrote: > Current gnocchi code supports RBAC at operation level > [gnocchi/gnocchi/rest/policy.json]. > Is it possible to add RBAC for attributes in a resource? > For eg: Restrict resource search/show should display specific attributes only > when query is performed by resource creator or admin. oslo.policy does not have such a capability, so this is done by auth helpers: https://github.com/gnocchixyz/gnocchi/blob/master/gnocchi/rest/auth_helper.py They are picked by the `api.auth_mode' setting. Feel free to send patches or write a new one if you prefer. -- Julien Danjou /* Free Software hacker https://julien.danjou.info */ signature.asc Description: PGP signature __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all][tc] Moving away from "big tent" terminology
On Fri, Jun 16 2017, Thierry Carrez wrote: > I should have made it clearer in my original post that this discussion > actually originated at the Board+TC+UC workshop in Boston, as part of > the "better communicating what is openstack" subgroup. This is still such a vague problem statement that it's hard to see how any interesting outcome will emerge in this thread. -- Julien Danjou # Free Software hacker # https://julien.danjou.info signature.asc Description: PGP signature __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all][tc] Moving away from "big tent" terminology
On Thu, Jun 15 2017, Doug Hellmann wrote: > One of the *most* common complaints the TC gets from outside the > contributor community is that people do not understand what projects > are part of OpenStack and what parts are not. We have a clear > definition of that in our minds (the projects that have said they > want to be part of OpenStack, and agreed to put themselves under > TC governance, with all of the policies that implies). That definition > is so trivial to say, that it seems like a tautology. However, > looking in from the outside of the community, that definition isn't > helpful. I still wonder why they care. Who care, really? Can we have some people that care on this thread so they explain directly what we're trying to solve here? Everything is just a bunch of free software projects to me. The governance made zero difference in my contributions or direction of the projects I PTL'ed. (There's so much effort put around trivial things like that. Why do I read only 20 emails in the "Glance is dying" thread and already 55 in that "let's rename big tent". Sigh. -- Julien Danjou /* Free Software hacker https://julien.danjou.info */ signature.asc Description: PGP signature __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all][tc] Moving away from "big tent" terminology
On Fri, Jun 16 2017, gordon chung wrote: > *sigh* so this is why we can't have nice things :p > > as an aside, in telemetry project, we did something somewhat similar > when we renamed/rebranded to telemetry from ceilometer. we wrote several > notes to the ML, had a few blog posts, fixed the docs, mentioned the new > project structure in our presentations... 2 years on, we still > occasionally get asked "what's ceilometer", "is xyz not ceilometer?", or > "so ceilometer is deprecated?". to a certain extent i think we'll have > to be prepared to do some hand holding and say "hey, that's not what the > "big tent/." Yeah, even the Foundation kept talking about Ceilometer instead of Telemetry in some internal/summit branding. I just gave up on that. They say time helps. -- Julien Danjou -- Free Software hacker -- https://julien.danjou.info signature.asc Description: PGP signature __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] etcd3 as base service - update
On Thu, Jun 08 2017, Mike Bayer wrote: > So far I've seen a proposal of etcd3 as a replacement for memcached in > keystone, and a new dogpile connector was added to oslo.cache to handle > referring to etcd3 as a cache backend. This is a really simplistic / minimal > kind of use case for a key-store. etcd3 is not meant to be a cache. Synchronizing caching value using the Raft protocol sounds a bit overkill. A cluster of memcached would be probably of a better use. > But, keeping in mind I don't know anything about etcd3 other than "it's > another > key-store", it's the only database used by Kubernetes as a whole, which > suggests it's doing a better job than Redis in terms of "durable". Not sure about that. And Redis has much more data structure than etcd, that is can be faster/more efficient than etcd. But it does not have Raft and a synchronisation protocol. Its clustering is rather poor in comparison of etcd. > So I wouldn't be surprised if new / existing openstack applications > express some gravitational pull towards using it as their own > datastore as well. I'll be trying to hang onto the etcd3 track as much > as possible so that if/when that happens I still have a job :). Sounds like a recipe for disaster. :) -- Julien Danjou /* Free Software hacker https://julien.danjou.info */ signature.asc Description: PGP signature __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [reno] we need help reviewing patches to reno
On Tue, Jun 06 2017, Doug Hellmann wrote: > I am looking for one or two people interested in learning about how reno > works to help with reviews. If you like graph traversal algorithms > and/or text processing, have a look at the code in the openstack/reno > repository and let me know if you're interested in helping out. I've added it to my list of watched projects. I'll try to review patches as they come in. I use reno and I like it. :) -- Julien Danjou ;; Free Software hacker ;; https://julien.danjou.info signature.asc Description: PGP signature __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [gnocchi] regional incoming storage targets
On Wed, May 31 2017, gordon chung wrote: […] > i'm not entirely sure this is an issue, just thought i'd raise it to > discuss. It's a really interesting point you raise. I never thought we could do that but indeed, we could. Maybe we built a great architecture after all. ;-) Easy solution: disable refresh. Problem solved. Also that means you could not push measures to the central API endpoint, or that'd be a problem. There might be a lot of little problem like that we need to solve. > regardless, thoughts on maybe writing up deployment strategies like > this? or make everyone who reads this to erase their minds and use this > for 'consulting' fees :P Yes, write doc or log an issue at least. It's best way to keep a public track now on ideas and what's going on since it's what people are going to read and search into. -- Julien Danjou -- Free Software hacker -- https://julien.danjou.info signature.asc Description: PGP signature __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [gnocchi][collectd-ceilometer-plugin][ceilometer][rally] Gates with gnocchi+devstack will break
Hi, If you're consuming Gnocchi via its devstack in the gate, you'll need to change that soon. As the repository has been moved to GitHub and the infra team not wanting to depend on GitHub (external) repositories, you'll need to set up Gnocchi via pip. I've started doing the work for Ceilometer here: https://review.openstack.org/#/c/468844/ https://review.openstack.org/#/c/468876/ If it can inspire you! As soon as https://review.openstack.org/#/c/466317/ is merged, your jobs will break. Cheers, -- Julien Danjou /* Free Software hacker https://julien.danjou.info */ signature.asc Description: PGP signature __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [panko] dropping hbase driver support
On Fri, May 26 2017, gordon chung wrote: > as all of you know, we moved all storage out of ceilometer so it is > handles only data generation and normalisation. there seems to be very > little contribution to panko which handles metadata indexing, event > storage so given how little it's being adopted and how little resources > are being put on supporting it, i'd like to proposed to drop hbase > support as a first step in making the project more manageable for > whatever resource chooses to support it. No objection from me. -- Julien Danjou # Free Software hacker # https://julien.danjou.info signature.asc Description: PGP signature __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [gnocchi] Running tests
On Tue, May 23 2017, aalvarez wrote: > Ok so started the tests using: > > tox -e py27-postgresql-file > > The suite starts running fine, but then I get a failing test: Can you reproduce it each time? That's weird, I don't think we ever saw that. -- Julien Danjou ;; Free Software hacker ;; https://julien.danjou.info signature.asc Description: PGP signature __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [telemetry] Room during the next PTG
On Tue, May 23 2017, Hanxi Liu wrote: > Can we restore weekly meeting or discuss in some centralized way just like > other TC > projects so that some developers can get timely responses and brainstorm > for Telemetry? There's a weekly IRC meeting for Telemetry, but it is only run if there's anything on the agenda. Last time we had a meeting was March 2016, since then nobody wrote anything on the agenda. The meeting page is here: https://wiki.openstack.org/wiki/Meetings/Telemetry It seems everybody is more comfortable with mailing lists and their asynchronicity. :) -- Julien Danjou -- Free Software hacker -- https://julien.danjou.info signature.asc Description: PGP signature __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [telemetry] Room during the next PTG
On Tue, May 23 2017, gordon chung wrote: > i just want to add, i am ok to meet at PTG but if we consider the turn > out at forum sessions, we aren't getting any more developers, just > random requirement requests. > > we can discuss on irc and mailing list? in fact we encourage it, which > is why you might see random short emails from me on list about whether i > should proceed on work item a certain way. i understand how meetings and > presence create this false narrative that work is being done. > regardless, i do hope we leverage ML and irc more. if you have a comment > or question, don't be afraid to use this medium. i shouldn't have to > wait 6 months to here problems :) I completely agree with gord, obviously. The question is: if we were to request a room, what we'd talk about or do that we couldn't achieve remotely? I'm not sure of what TBH. As the deadline was last Sunday to request a room, it seems like we're not gonna join this time neither. :) -- Julien Danjou /* Free Software hacker https://julien.danjou.info */ signature.asc Description: PGP signature __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [gnocchi] Running tests
On Tue, May 23 2017, Andres Alvarez wrote: > Hello everyone > > I am having a hard time in understanding the correct way to run the tests > in Gnocchi. I have already read about tox and testr, but it seems I still > can't get to run the tests. tox -e py27-postgresql-file That'll run the test with PostgreSQL and the file storage backend. tox -e py35-mysql-ceph For MySQL and Ceph, etc. That's all you need. -- Julien Danjou -- Free Software hacker -- https://julien.danjou.info signature.asc Description: PGP signature __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [oslo] Can we stop global requirements update?
On Fri, May 19 2017, Mike Bayer wrote: > IMO that's a bug for them. Of course it's a bug. IIRC Mehdi tried to fix it without much success. > I'm inspired to see that Keystone, Nova etc. are > able to move between and eventlet backend and a mod_wsgi backend.IMO > eventlet is really not needed for those services that present a REST > interface. > Although for a message queue with lots of long-running connections that > receive > events, that's a place where I *would* want to use a polling / non-blocking > model. But I'd use it explicitly, not with monkeypatching. +1 > I'd ask why not oslo.cotyledon but it seems there's a faction here that is > overall moving out of the Openstack umbrella in any case. Not oslo because it can be used by other projects than just OpenStack. And it's a condition of success. As Mehdi said, Oslo has been deserted in the recent cycles, so putting a lib there as very little chance of seeing its community and maintenance help grow. Whereas trying to reach the whole Python ecosystem is more likely to get traction. As a maintainer of SQLAlchemy I'm surprised you even suggest that. Or do you plan on doing oslo.sqlalchemy? ;) > Basically I think openstack should be getting off eventlet in a big way so I > guess my sentiment here is that the Gnocchi / Cotyledon /etc. faction is just > splitting off rather than serving as any kind of direction for the rest of > Openstack to start looking. But that's only an impression, maybe projects > will > use Cotyledon anyway. If every project goes off and uses something > completely > different though, then I think we're losing. The point of oslo was to > prevent > that. I understand your concern and opinion. I think you, me and Mehdi don't have the experience as contributors in OpenStack. I invite you to try moving any major OpenStack project to something like oslo.service2 or Cotyledon or to achieve any technical debt resolution in OpenStack to have a view on hard it is to tackle. Then you'll see where we stand. :) Especially when your job is not doing that, but e.g. working on Telemetry. :) -- Julien Danjou -- Free Software hacker -- https://julien.danjou.info signature.asc Description: PGP signature __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [gnocchi] Migration to GitHub
On Thu, May 18 2017, Julien Danjou wrote: > I've started to migrate Gnocchi itself to GitHub. The Launchpad bugs > have been re-created at https://github.com/gnocchixyz/gnocchi/issues and > I'll move the repository as soon as all opened reviews are merged. Everything has been merged today so the repository is now live at GitHub. The rest of the deprecation patches are up for review: https://review.openstack.org/#/q/status:open+project:openstack-infra/project-config+branch:master+topic:jd/move-gnocchi-out -- Julien Danjou /* Free Software hacker https://julien.danjou.info */ signature.asc Description: PGP signature __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] Onboarding rooms postmortem, what did you do, what worked, lessons learned
On Fri, May 19 2017, Sean Dague wrote: > If you ran a room, please post the project, what you did in the room, > what you think worked, what you would have done differently. If you > attended a room you didn't run, please provide feedback about which one > it was, and what you thought worked / didn't work from the other side of > the table. We shared a room for Telemetry and CloudKitty for 90 minutes. I was there with Gordon Chung for Telemetry. Christophe Sauthier was there for CloudKitty. We only had 3 people showing up in the session. One wanted to read his emails in a quiet room, the two others had a couple of question on Telemetry – though it was not really related to contribution as far as I can recall. I had to leave after 45 minutes because they was an overlap with a talk I was doing and rescheduling did not seem possible. And everybody left a few minutes after I left apparently. -- Julien Danjou -- Free Software hacker -- https://julien.danjou.info signature.asc Description: PGP signature __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [oslo] Can we stop global requirements update?
On Fri, May 19 2017, Mehdi Abaakouk wrote: > Not really, I just put some comments on reviews and discus this on IRC. > Since nobody except Telemetry have expressed/try to get rid of eventlet. TBH Keystone get rid of it too. But they only provide WSGI servers. They don't build any daemon so they don't need and user either Cotyledon nor oslo.service. :) > Because the internal of cotyledon and oslo.service are so different. > Having the code in oslo or not doesn't help for maintenance anymore. > Cotyledon is a lib, code and bugs :) can already be shared between > projects that doesn't want eventlet. Cotyledon is explicitly better just by being out of Oslo, because it's usable by the whole Python ecosystem. :) -- Julien Danjou -- Free Software hacker -- https://julien.danjou.info signature.asc Description: PGP signature __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [gnocchi] Running Gnocchi API in specific interface
On Thu, May 18 2017, aalvarez wrote: > Yes but doesn't Pecan allow to use a development server (pecan serve) that > can accept interface and port options? I thought this would be the > test/development server Gnocchi would use. We could but there's no need and it's just one line to rely on pbr's WSGI server. I'm sure adding the feature to pbr and be framework agnostic would not be a big deal. :) -- Julien Danjou # Free Software hacker # https://julien.danjou.info signature.asc Description: PGP signature __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [telemetry] Room during the next PTG
Hi team, It's time for us to request a room (or share one) for the next PTG in September if we want to meet. Last time we did not. Do we want one this time? -- Julien Danjou /* Free Software hacker https://julien.danjou.info */ signature.asc Description: PGP signature __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [oslo] Can we stop global requirements update?
On Thu, May 18 2017, Mike Bayer wrote: > replaces oslo.service with a multiprocessing approach that doesn't use > eventlet. great! any openstack service that rides on oslo.service would like > to be able to transparently switch from eventlet to multiprocessing the same > way they can more or less switch to mod_wsgi at the moment.IMO this should > be part of oslo.service itself. Docs state: "oslo.service being impossible > to > fix and bringing an heavy dependency on eventlet, " is there a discussion > thread on that? Yes, and many reviews around that. I'll let Mehdi comments if he feels like it. :) > I'm finding it hard to believe that only a few years ago, everyone saw the > wisdom of not re-implementing everything in their own projects and using a > common layer like oslo, and already that whole situation is becoming forgotten > - not just for consistency, but also when a bug is found, if fixed in oslo it > gets fixed for everyone. I guess it depends what you mean by everyone. FTR, one of the two first projects in OpenStack, Swift, never used anything from Oslo for anything and always refused to depends on any of its library. -- Julien Danjou /* Free Software hacker https://julien.danjou.info */ signature.asc Description: PGP signature __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [oslo] Can we stop global requirements update?
On Thu, May 18 2017, Mike Bayer wrote: > I'm not understanding this? do you mean this? In the long run, yes. Unfortunately, we're not happy with the way Oslo libraries are managed and too OpenStack centric. I've tried for the last couple of years to move things on, but it's barely possible to deprecate anything and contribute, so I feel it's safer to start fresh and better alternative. Cotyledon by Mehdi is a good example of what can be achieved. Though to comment on your example, oslo.db is probably the most useful Oslo library that Gnocchi depends on and that won't go away in a snap. :-( -- Julien Danjou /* Free Software hacker https://julien.danjou.info */ signature.asc Description: PGP signature __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [gnocchi] Migration to GitHub
Hi, I've started to migrate Gnocchi itself to GitHub. The Launchpad bugs have been re-created at https://github.com/gnocchixyz/gnocchi/issues and I'll move the repository as soon as all opened reviews are merged. Cheers, -- Julien Danjou // Free Software hacker // https://julien.danjou.info signature.asc Description: PGP signature __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [gnocchi] Running Gnocchi API in specific interface
On Thu, May 18 2017, Hanxi Liu wrote: > Ceilometer, Gnocchi, Aodh all use pbr, so the port is 8000 by default. > > I guess we also should hardcode Gnocchi's port in rdo project, together > with Aodh. > i proposed patchs for Aodh and Gnocchi: > > https://review.rdoproject.org/r/#/c/5848/ > https://review.rdoproject.org/r/#/c/5847/ > > But hguemar suggest not to hardcode port. > > How do you think about this? Port for HTTP is 80. The rest, unless assigned by IANA, is fantasy. :-) You can make all your (OpenStack) services run under http://example.com/openstack, e.g. http://example.com/openstack/metric for Gnocchi, if you want. So there's no good reason to assigne a port to a service more than there is to assigne a hardcoded URL prefix. So I think I'd agree with Haikel here. But ultimately, in the RDO case, the packaging files should leverage a real WSGI Web server like uwsgi if they want to start the service, rather than defaulting *all* packages to the same pbr default. Which will conflict and is bad user experience. Nobody asks what the default port of e.g. phpmyadmin. The same should go with OpenStack services ultimately. Unfortunately, the bad habit has spread from the early day of Nova and Swift using a port and not providing a WSGI file. -- Julien Danjou /* Free Software hacker https://julien.danjou.info */ signature.asc Description: PGP signature __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [gnocchi] Running Gnocchi API in specific interface
On Thu, May 18 2017, aalvarez wrote: > I thought the API was based on and mounted by Pecan? Isn't there a way to > pass these options to Pecan? Pecan is an API framework, not a HTTP server. -- Julien Danjou # Free Software hacker # https://julien.danjou.info signature.asc Description: PGP signature __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [gnocchi] Running Gnocchi API in specific interface
On Wed, May 17 2017, aalvarez wrote: > I do not need this functionality for production, but for testing. I think it > would be nice if we can specify the interface for the gnocchi-api even for > test purposes, just like the port. Feel free to send a patch. This is provided by pbr so that's where you should sent the patch: https://docs.openstack.org/developer/pbr/ -- Julien Danjou // Free Software hacker // https://julien.danjou.info signature.asc Description: PGP signature __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all][release] sphinx 1.6.1 behavior changes triggering job failures
On Wed, May 17 2017, Julien Danjou wrote: >> There's a patch in review for the reno issue. It would be great if >> someone had time to look into a fix for pbr to make it work with >> older and newer Sphinx. > > I'll look into it. Here's the pbr fix: https://review.openstack.org/#/c/465489/ If we can merge and release that ASAP that'd be great. -- Julien Danjou -- Free Software hacker -- https://julien.danjou.info signature.asc Description: PGP signature __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all][release] sphinx 1.6.1 behavior changes triggering job failures
On Tue, May 16 2017, Doug Hellmann wrote: > https://bugs.launchpad.net/pbr/+bug/1691129 describes a traceback > produced when building the developer documentation through pbr. > > https://bugs.launchpad.net/reno/+bug/1691224 describes a change where > Sphinx now treats log messages at WARNING or ERROR level as reasons to > abort the build when strict mode is enabled. > > I have a patch up to the global requirements list to block 1.6.1 for > builds following g-r and constraints: > https://review.openstack.org/#/c/465135/ > > Many of our doc builds do not use constraints, so if your doc build > fails you will want to apply the same change locally. > > There's a patch in review for the reno issue. It would be great if > someone had time to look into a fix for pbr to make it work with > older and newer Sphinx. I'll look into it. -- Julien Danjou /* Free Software hacker https://julien.danjou.info */ signature.asc Description: PGP signature __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [oslo] Can we stop global requirements update?
On Tue, May 16 2017, Andreas Jaeger wrote: > It is needed to generate the translations, but can't we move it for > oslo-i18n into test-requirements? I've pushed this and it seems to work, pretty sure it's safe. https://review.openstack.org/#/c/465014/ If we can merge this today and then release quickly after that'd be a great help -_- > But os-testr does not need Babel at all - let's remove it, > https://review.openstack.org/465023 Arf, sure! I can only +1 though :( -- Julien Danjou /* Free Software hacker https://julien.danjou.info */ signature.asc Description: PGP signature __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [oslo] Can we stop global requirements update?
On Tue, May 16 2017, Andreas Jaeger wrote:
> what exactly happened with Babel?
>
> I see in global-requirements the following:
> Babel>=2.3.4,!=2.4.0 # BSD
>
> that shouldn't case a problem - or does it? Or what's the problem?
Damn, at the moment I pressed the `Sent' button I thought "You just
complained without including much detail idiot". Sorry about that!
One of the log that fails:
http://logs.openstack.org/13/464713/2/check/gate-gnocchi-tox-py27-mysql-ceph-upgrade-from-3.1-ubuntu-xenial/db61bdf/console.html
Basically oslo.policy pulls oslo.i18n which pulls Babel!=2.4.0
But Babel is already pulled by os-testr which depends on >=2.3.4.
So pip does not solve that (unfortunately) and then the failure is:
2017-05-16 05:08:43.629772 | 2017-05-16 05:08:43.503 10699 ERROR gnocchi
ContextualVersionConflict: (Babel 2.4.0
(/home/jenkins/workspace/gate-gnocchi-tox-py27-mysql-ceph-upgrade-from-3.1-ubuntu-xenial/upgrade/lib/python2.7/site-packages),
Requirement.parse('Babel!=2.4.0,>=2.3.4'), set(['oslo.i18n']))
I'm pretty sure Babel should not even be in the requirements list of
oslo.i18n since it's not a runtime dependency AFAIU.
--
Julien Danjou
/* Free Software hacker
https://julien.danjou.info */
signature.asc
Description: PGP signature
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [oslo] Can we stop global requirements update?
On Wed, Apr 19 2017, Julien Danjou wrote: > So Gnocchi gate is all broken (agan) because it depends on "pbr" and > some new release of oslo.* depends on pbr!=2.1.0. Same things happened today with Babel. As far as Gnocchi is concerned, we're going to take the easiest route and remove all our oslo dependencies over the next months for sanely maintained alternative at this point. Cheers, -- Julien Danjou -- Free Software hacker -- https://julien.danjou.info signature.asc Description: PGP signature __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [tc][all] Should the Technical Committee meetings be dropped?
On Thu, May 04 2017, Flavio Percoco wrote: > Sending this out to kick off a broader discussion on these topics. Thoughts? > Opinions? Objections? IRC meetings are a barrier of entry for many people because it makes it complicated to contribute as you described (and as I did a while back¹). So I'm completely for their suppression as one of the main medium of conversation – that does not mean IRC has to be banned. :) ¹ https://julien.danjou.info/blog/2016/foss-projects-management-bad-practice -- Julien Danjou /* Free Software hacker https://julien.danjou.info */ signature.asc Description: PGP signature __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [gnocchi] per-sack vs per-metric locking tradeoffs
On Fri, Apr 28 2017, gordon chung wrote: >>> refresh i believe is always disabled by default regardless of what >>> interface you're using. >> >> You gotta to show me where it is 'cause I can't see that and I don't >> recall any option for that. :/ > > https://github.com/openstack/gnocchi/commit/72a2091727431555eba65c6ef8ff89448f3432f0 > > > > although now that i check, i see it's blocking by default... which means > we did guarantee all measures would be return. I guess we misunderstood each other. What I meant originally is that we did not have a flag to "disable its usage completely". Because from an operator point of view it could be an easier way to provoke a DoS. > hmmm. true. i'm still hoping we don't have to lock an entire sack for > one metric and not return an error status just because it can't lock. > doesn't seem like a good experience. If you wait long enough, you always can return. :) -- Julien Danjou /* Free Software hacker https://julien.danjou.info */ signature.asc Description: PGP signature __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [gnocchi] per-sack vs per-metric locking tradeoffs
On Fri, Apr 28 2017, gordon chung wrote: >>> if the sack is unlocked, then it means a processing worker isn't looking >>> at the sack, and when it does lock the sack, it first has to check sack >>> for existing measures to process and then check indexer to validate that >>> they are still active. because it checks indexer later, even if both >>> janitor and processing worker check lock at same time, we can guarantee >>> it will have indexer state processing worker sees is > 00:00:00 since >>> janitor has state before getting lock, while processing worker as state >>> sometime after getting lock. >> >> My brain hurts but that sounds perfect. That even means we potentially >> did not have to lock currently, sack or no sack. >> > > oh darn, i didn't consider multiple janitors... so this only works if we > make janitor completely separate and only allow one janitor ever. back > to square 1 Not sure it's a big deal if you have multiple janitors. Do you have a scenario in mind where we'd have problem? Since it's mainly: 1. get 'deleted' metrics 2. delete all things in storage -> if it fails, whatever, ignore, maybe a janitor is doing the same thing? 3. expunge from indexer I miss something hm? -- Julien Danjou ;; Free Software hacker ;; https://julien.danjou.info signature.asc Description: PGP signature __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [gnocchi] per-sack vs per-metric locking tradeoffs
On Fri, Apr 28 2017, gordon chung wrote: > if the sack is unlocked, then it means a processing worker isn't looking > at the sack, and when it does lock the sack, it first has to check sack > for existing measures to process and then check indexer to validate that > they are still active. because it checks indexer later, even if both > janitor and processing worker check lock at same time, we can guarantee > it will have indexer state processing worker sees is > 00:00:00 since > janitor has state before getting lock, while processing worker as state > sometime after getting lock. My brain hurts but that sounds perfect. That even means we potentially did not have to lock currently, sack or no sack. -- Julien Danjou // Free Software hacker // https://julien.danjou.info signature.asc Description: PGP signature __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [gnocchi] per-sack vs per-metric locking tradeoffs
On Fri, Apr 28 2017, gordon chung wrote: > refresh i believe is always disabled by default regardless of what > interface you're using. You gotta to show me where it is 'cause I can't see that and I don't recall any option for that. :/ > in the case of cross-metric aggregations, this is a timeout for entire > request or per metric timeout? i think it's going to get quite chaotic > in the multiple metric (multiple sacks) refresh. :( Right I did not think about multiple refresh. Well it'll be a nice slalom of lock acquiring. :-) > i'm hoping not to have a timeout because i imagine there will be > scenarios where we block trying to lock sack, and when we finally get > sack lock, we find there is no work for us. this means we just added x > seconds to response to for no reason. Right, I see your point. Though we _could_ enhance refresh to first check if there's any job to do. It's lock-free. Just checking. :) -- Julien Danjou # Free Software hacker # https://julien.danjou.info signature.asc Description: PGP signature __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [tooz] etcd3 driver (was: [oslo][devstack][all] ZooKeeper vs etcd for Tooz/DLM)
On Sun, Mar 19 2017, Jay Pipes wrote: Jay, are you working or planning to work on the group interface soon? I'm planning to take a look at it ASAP now but I'd hate to duplicate any work. :) > Also, I gave a stab at an etcd3 tooz coordination driver: > > https://review.openstack.org/#/c/447223/ > > Can't figure out how to get the tests to run properly, but meh, I'll get to it > in a bit :) > > -jay > > On 03/19/2017 04:34 PM, Davanum Srinivas wrote: >> Quick update: We have 3 options to talk to etcd: >> >> 1) existing V2 HTTP API - still not deprecated in etcd 3.1.x, so >> existing code in tooz still works. >> 2) grpc based V3 API - We can use the etcd3 package, discussion in >> review[1], problem is that this will not work currently with eventlet >> based services >> 3) v3alpha HTTP API - See details in [2], quick prototype requests >> based code [3] >> >> Thanks, >> Dims >> >> [1] https://review.openstack.org/#/c/446983/ >> [2] >> https://github.com/coreos/etcd/blob/master/Documentation/dev-guide/api_grpc_gateway.md >> [3] https://gist.github.com/dims/19ceaf9472ef54aa3011d7a11e809371 >> >> On Sun, Mar 19, 2017 at 9:32 AM, Jay Pipes <jaypi...@gmail.com> wrote: >>> On 03/18/2017 07:48 PM, Mike Perez wrote: >>>> >>>> On 12:35 Mar 14, Jay Pipes wrote: >>>>> >>>>> On 03/14/2017 08:57 AM, Julien Danjou wrote: >>>>>> >>>>>> On Tue, Mar 14 2017, Davanum Srinivas wrote: >>>>>> >>>>>>> Let's do it!! (etcd v2-v3 in tooz) >>>>>> >>>>>> >>>>>> Hehe. I'll move that higher in my priority list, I swear. But anyone is >>>>>> free to beat me to it in the meantime. ;) >>>>> >>>>> >>>>> A weekend experiment: >>>>> >>>>> https://github.com/jaypipes/os-lively >>>>> >>>>> Not tooz, because I'm not interested in a DLM nor leader election library >>>>> (that's what the underlying etcd3 cluster handles for me), only a fast >>>>> service liveness/healthcheck system, but it shows usage of etcd3 and >>>>> Google >>>>> Protocol Buffers implementing a simple API for liveness checking and host >>>>> maintenance reporting. >>>>> >>>>> My plan is to push some proof-of-concept patches that replace Nova's >>>>> servicegroup API with os-lively and eliminate Nova's use of an RDBMS for >>>>> service liveness checking, which should dramatically reduce the amount of >>>>> both DB traffic as well as conductor/MQ service update traffic. >>>> >>>> >>>> As Monty has mentioned, I'd love for us to decide on one thing. As being >>>> a moderator of that discussion I was trying not to be one-sided. >>>> >>>> Whether or not a decision was made 1.5 years ago, the community that was >>>> present at that time of the decision at the summit decided on an >>>> abstraction >>>> layer to have options. Forcing an option on the community without >>>> gathering >>>> feedback of what the community currently looks like is not a good idea. >>>> >>>> I'd recommend if you want to make this base service, start the discussions >>>> in >>>> somewhere other than the dev list, like the Forum. >>> >>> >>> Mike, it was an experiment :) >>> >>> But, yes, happy to participate in a discussion at the forum. >>> >>> >>> Best, >>> -jay >>> >>> __ >>> OpenStack Development Mailing List (not for usage questions) >>> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe >>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >> >> >> > > __ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > -- Julien Danjou # Free Software hacker # https://julien.danjou.info signature.asc Description: PGP signature __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [gnocchi] per-sack vs per-metric locking tradeoffs
On Fri, Apr 28 2017, gordon chung wrote: > what if we just don't lock ever on delete, but if we still check lock to > see if it's processed. at that point, the janitor knows that metric(s) > is deleted, and since no one else is working on sack, any metricd that > follows will also know that the metric(s) are deleted and therefore, > won't work on it. I did not understand your whole idea, can you detail a bit more? Though the "check lock" approach usually does not work and is a source of race condition, but again, I did not get the whole picture. :) -- Julien Danjou -- Free Software hacker -- https://julien.danjou.info signature.asc Description: PGP signature __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [gnocchi] per-sack vs per-metric locking tradeoffs
On Fri, Apr 28 2017, gordon chung wrote: > so the tradeoff here is that now we're doing a lot more calls to > indexer. additionally, we're pulling a lot more unused results from db. > a single janitor currently just grabs all deleted metrics and starts > attempting to clean them up one at a time. if we merge, we will have n > calls to indexer, where n is number of workers, each pulling in all the > deleted metrics, and then checking to see if the metric is in it's sack, > and if not, moving on. that's a lot of extra, wasted work. we could > reduce that work by adding sack information to indexer ;) but that will > still add significantly more calls to indexer (which we could reduce by > not triggering cleanup every job interval) That's not what I meant. You can have the same mechanism as currently, but then you compute the sacks of all metrics and you itertools.groupby() per sack on them before locking the sack and expunging them. > refresh is currently disabled by default so i think we're ok. Well you mean it's disabled by default _in the CLI_, not in the API right? > what's the timeout for? timeout api's attempt to aggregate metric? i > think it's a bad experience if we add any timeout since i assume it will > still return what it can return and then the results become somewhat > ambiguous. No, I meant timeout for grabbing the sack's lock. You wouldn't return a 2xx but a 5xx stating the API is unable to compute stuff right now, so try again without refresh or something. -- Julien Danjou // Free Software hacker // https://julien.danjou.info signature.asc Description: PGP signature __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [gnocchi] Moving python-gnocchiclient to GitHub
On Fri, Apr 28 2017, Javier Pena wrote: > I see none of the tags or branches have been moved over, could they be copied? > It would of great help to packagers. Oh for sure, that's my mistake, I pushed all branches and old tags! Thanks for noting! -- Julien Danjou -- Free Software hacker -- https://julien.danjou.info signature.asc Description: PGP signature __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [gnocchi] Moving python-gnocchiclient to GitHub
On Tue, Apr 25 2017, Julien Danjou wrote: > We're in the process of moving python-gnocchiclient to GitHub. The > patches are up for review: > > https://review.openstack.org/#/c/459748/ > > and its dependencies need to be merged before this happen. As soon as > this patch is merged, the repository will officially be available at: > > https://github.com/gnocchixyz/python-gnocchiclient This happened! The repository has now been moved to GitHub. I've also created copies of the existing opened bugs there so we don't lose that data. If I missed anything, let me know. -- Julien Danjou ;; Free Software hacker ;; https://julien.danjou.info signature.asc Description: PGP signature __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [gnocchi] per-sack vs per-metric locking tradeoffs
On Thu, Apr 27 2017, gordon chung wrote: > so as we transition to the bucket/shard/sack framework for incoming > writes, we've set up locks on the sacks so we only have one process > handling any given sack. this allows us to remove the per-metric locking > we had previously. yay! > the issue i've notice now is that if we only have per-sack locking, > metric based actions can affect sack level processing. for example: > > scenario 1: > 1. delete metric, locks sack to delete single metric, > 2. metricd processor attempts to process entire sack but can't, so skips. Yes, I wrote that in a review somewhere. We need to rework 1. so deletion happens at the same time we lock the sack to process metrics basically. We might want to merge the janitor into the worker I imagine. Currently a janitor can grab metrics and do dumb things like: - metric1 from sackA - metric2 from sackB - metric3 from sackA and do 3 different lock+delete -_- > scenario 2: > 1. API request passes in 'refresh' param so they want all unaggregated > metrics to be processed on demand and returned. > 2. API locks 1 or more sacks to process 1 or more metrics > 3. metricd processor attempts to process entire sack but can't, so > skips. potentially multiple sacks unprocessed in currently cycle. > > scenario 3 > same as scenario 2 but metricd processor locks first, and either blocks > API process OR doesn't allow API to guarantee 'all measures processed'. Yes, I'm even more worried about scenario 3, we should probably add a safe guard timeout parameter set by the admin there. > i imagine these scenarios are not critical unless a very large > processing interval is defined or if for some unfortunate reason, the > metric-based actions are perfectly timed to lock out background processing. > > alternatively, this could be solved by keeping per-metric locks in > addition to per-sack locks. this would effectively double the number of > active locks we have so instead of each metricd worker having a single > per-sack lock, it will also have a per-metric lock for whatever metric > it may be publishing at the time. If we got a timeout set for scenario 3, I'm not that worried. I guess worst thing is that people would be unhappy with the API spending time doing computation anyway so we'd need to rework how refresh work or add an ability to disable it. -- Julien Danjou // Free Software hacker // https://julien.danjou.info signature.asc Description: PGP signature __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [cinder] Is Cinder still maintained?
On Thu, Apr 27 2017, Gorka Eguileor wrote: Hi Gorka, > So, you mention that the feature has been there for a while, but funny > enough documentation has not been updated and it's still reflecting it > should be done how we are doing it in Cinder [1]. As I replied on the review, thanks for the catch, I've sent a patch to update it: https://review.openstack.org/#/c/460615/ > As far as I can tell looking at Tooz's code for the heartbeat it doesn't > do the same thing our code does, because it doesn't reconnect on > failure... Am I missing something? As I replied on the review, the reconnection is handled by tooz drivers directly, so there should not be any need to handle it in the application. Otherwise, it'd be considered as a bug in tooz (which is a possibility, who knows). Cheers, -- Julien Danjou # Free Software hacker # https://julien.danjou.info signature.asc Description: PGP signature __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [cinder] Is Cinder still maintained?
Hi, I've posted a refactoring patch that simplifies tooz (read: remove technical debt) usage more than a month ago, and I got 0 review since then: https://review.openstack.org/#/c/447079 I'm a bit worried to see this zero review on such patches. It seems the most recently merged things are all vendor specific. Is the core of Cinder still maintained? Is there any other reason for such patches to be ignored for so long? -- Julien Danjou // Free Software hacker // https://julien.danjou.info signature.asc Description: PGP signature __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Gnocchi
On Wed, Apr 26 2017, simona marinova wrote: Hi Simona, > I am a student working on a project that involves an OpenStack Newton > platform. > > Currently, we are trying to implement the Data Collection service. We saw that > Gnocchi is recommended for this purpose, and we installed it. > > Now we have problems with the configuration. > > I have tried to configure the basic parameters, but the same errors appear > over and over. > > Until this point, every installation and configuration of the services in > OpenStack is done exactly the same as shown in the official OpenStack > documentation. > > I am sending you a screenshot of the output when I try to run gnocchi. > > > Can you help me with a basic configuration or some advice? It looks like you set your Swift URL to a Keystone URL something like that. Could you join your gnocchi.conf file? -- Julien Danjou /* Free Software hacker https://julien.danjou.info */ signature.asc Description: PGP signature __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] project on-board schedule
On Tue, Apr 25 2017, Tom Fifield wrote: Hi Tom, > It's listed on the main summit schedule, under the Forum :) > > Here's a direct link to the Forum category: > > https://www.openstack.org/summit/boston-2017/summit-schedule/#track=146 I've also just realized the Telemetry onboarding session overlaps with one of my talks: https://www.openstack.org/summit/boston-2017/summit-schedule/events/18698/cloudkittytelemetry-project-onboarding https://www.openstack.org/summit/boston-2017/summit-schedule/events/17530/openstack-telemetry-and-the-1-instances Any chance to reschedule one or the other? -- Julien Danjou // Free Software hacker // https://julien.danjou.info signature.asc Description: PGP signature __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [gnocchi] Moving python-gnocchiclient to GitHub
Hi, We're in the process of moving python-gnocchiclient to GitHub. The patches are up for review: https://review.openstack.org/#/c/459748/ and its dependencies need to be merged before this happen. As soon as this patch is merged, the repository will officially be available at: https://github.com/gnocchixyz/python-gnocchiclient Cheers, -- Julien Danjou // Free Software hacker // https://julien.danjou.info signature.asc Description: PGP signature __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [oslo] Can we stop global requirements update?
On Fri, Apr 21 2017, Doug Hellmann wrote: > My memory of the outcome of that session was that we needed to maintain > co-installability; that we could continue to keep an eye on the > container space as an alternative; and that a new team of maintainers > would take over the requirements list (which was my secret agenda for > proposing that we stop doing it at all). Just to come back to the original topic, co-installability is *not* impacted by my proposal of stopping the sync. On the opposite, last week it was very hard to install both Gnocchi and Oslo because of Oslo… So it would *improve* co-installability, not remove it. My 2c, -- Julien Danjou // Free Software hacker // https://julien.danjou.info signature.asc Description: PGP signature __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [oslo] Can we stop global requirements update?
On Wed, Apr 19 2017, Clark Boylan wrote: > I think we know from experience that just stopping (eg reverting to the > situation we had before requirements and constraints) would lead to > sadness. Installations would frequently be impossible due to some > unresolvable error in dependency resolution. Do you have some > alternative in mind? Perhaps we loosen the in project requirements and > explicitly state that constraints are known to work due to testing and > users should use constraints? That would give users control to manage > their own constraints list too if they wish. Maybe we do this in > libraries while continuing to be more specific in applications? Most of the problem that requirements is trying to solve is related to upper-constraints, blocking new releases. And this upper constraints are used in most jobs, preventing most failure that are seen in gates. It would have "covered" the pbr issue. What I want to stop here, is the automatic push of blacklisting/capping of stuff to *everything* in OpenStack as soon as one project have a problem with something. > I don't have all the answers, but am fairly certain the situation we > have today is better than the one from several years ago. It is just not > perfect. I think we are better served by refining the current setup or > replacing it with something better but not by reverting. Agreed, I'm not suggesting to revert everything. Just the automatic push of random requirements limits and binding to Oslo. And other projects if you like, we don't do it for a good year anymore in Telemetry, and again here we saw 0 breakage due to that change. Just more easyness to install stuff. -- Julien Danjou -- Free Software hacker -- https://julien.danjou.info signature.asc Description: PGP signature __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [oslo] Can we stop global requirements update?
Hoy, So Gnocchi gate is all broken (agan) because it depends on "pbr" and some new release of oslo.* depends on pbr!=2.1.0. Neither Gnocchi nor Oslo cares about whatever bug there is in pbr 2.1.0 that got in banished by requirements Gods. It does not prevent it to be used e.g. to install the software or get version information. But it does break anything that is not in OpenStack because well, pip installs the latest pbr (2.1.0) and then oslo.* is unhappy about it. So I understand the culprit is probably pip installation scheme, and we can blame him until we fix it. I'm also trying to push pbr 2.2.0 to avoid the entire issue. But for the future, could we stop updating the requirements in oslo libs for no good reason? just because some random OpenStack project hit a bug somewhere? For example, I've removed requirements update on tooz¹ for more than a year now, which did not break *anything* in the meantime, proving that this process is giving more problem than solutions. Oslo libs doing that automatic update introduce more pain for all consumers than anything (at least not in OpenStack). So if we care about Oslo users outside OpenStack, I beg us to stop this crazyness. If we don't, we'll just spend time getting rid of Oslo over the long term… My 2c, Cheers, ¹ Unless some API changed in a dep and we needed to raise the dep, obviously. -- Julien Danjou # Free Software hacker # https://julien.danjou.info signature.asc Description: PGP signature __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [gnocchi] new measures backlog scheduling
On Tue, Apr 18 2017, gordon chung wrote: > do we want it configurable? tbh, would anyway one configure it or know > how to configure it? even for us, we're just guessing somewhat.lol i'm > going to leave it static for now. I think we want it to be configurable, though most people would probably not tweak it. But I can imagine some setups where increasing it would make sure that. There's some sense of exposing it anyway, even if it does not change much. For example, we never exposed TASKS_PER_WORKER but in the end it seems the 16 value is not optimal. But since we did not expose it, there's barely no way for tester to try to tweak it and see what value works best. :) > this is not really free. choosing hashring means we will have idle > workers and more complexity of figuring out what each of the other > agents look like in group. it's a trade-off (especially considering how > few keys to nodes we have) which is why i brought up question. You're right, it's a trade-off. I think we can go with the non-hashring approach for now, that you implemented, and see how if/how we need to evolve it. It seems to be better than the current scheduling anyway. > i'll be honest, i'll probably still switch back to hashring... but want > to make sure we're not just thinking hashring only. You're completely right, we needed to discuss that anyway. All your patches version and tries build up our knowledge and expertise on the subject, so it was definitely worth the effort, and kudos go to you for taking that job! What will probably make you go back to hashring? -- Julien Danjou # Free Software hacker # https://julien.danjou.info signature.asc Description: PGP signature __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [gnocchi] new measures backlog scheduling
On Tue, Apr 18 2017, gordon chung wrote: > the issue i see is not with how the sacks will be assigned to metricd > but how metrics (not daemon) are assigned to sacks. i don't think > storing value in storage object solves the issue because when would we > load/read it when the api and metricd processes startup? it seems this > would require: 1) all services to be shut down and 2) have a completely > clean incoming storage path. if any of the two steps aren't done, you > have a corrupt incoming storage. if this is a requirement and both of > these are done successfully, this means, any kind of 'live upgrade' is > impossible in gnocchi. Live upgrade never has been supported in Gnocchi, so I don't see how that's a problem. It'd be cool to support it for sure, but we're far from having been able to implement it at any point in time in the best. So it's not a new issue or anything like that. I really don't see a problem with loading the number of sacks at startup. > i had did test w/ 2 replicas (see: google sheet) and it's still > non-uniform but better than without replicas: ~4%-30% vs ~8%-45%. we > could also minimise the number lock calls by dividing sacks across > workers per agent. > > going to play devils advocate now, using hashring in our use case will > always hurt throughput (even with perfect distribution since the sack > contents themselves are not uniform). returning to original question, is > using hashring worth it? i don't think we're even leveraging the > re-balancing aspect of hashring. I think it's worth it only if you use replicas – and I don't think 2 is enough, I'd try 3 at least, and make it configurable. It'll reduce a lot lock-contention (e.g. by 17x time in my previous example). As far as I'm concerned, since the number of replicas is configurable, you can add a knob that would set replicas=number_of_metricd_worker that would implement the current behaviour you implemented – every worker tries to grab every sack. We're not leveraging the re-balancing aspect of hashring, that's true. We could probably use any dumber system to spread sacks across workers, We could stick to the good ol' "len(sacks) / len(workers in the group)". But I think there's a couple of things down the road that may help us: Using the hashring makes sure worker X does not jump from sacks [A, B, C], to [W, X, Y, Z] but just to [A, B] or [A, B, C, X]. That should minimize lock contention when bringing up/down new workers. I admit it's a very marginal win, but… it comes free with it. Also, I envision a push based approach in the future (to replace the metricd_processing_delay) which will require worker to register to sacks. Making sure the rebalancing does not shake everything but is rather smooth will also reduce workload around that. Again, it comes free. -- Julien Danjou # Free Software hacker # https://julien.danjou.info signature.asc Description: PGP signature __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [gnocchi] new measures backlog scheduling
On Mon, Apr 17 2017, gordon chung wrote: Hi Gordon, > i've started to implement multiple buckets and the initial tests look > promising. here's some things i've done: > > - dropped the scheduler process and allow processing workers to figure > out tasks themselves > - each sack is now handled fully (not counting anything added after > processing worker) > - number of sacks are static > > after the above, i've been testing it and it works pretty well, i'm able > to process 40K metrics, 60 points each, in 8-10mins with 54 workers when > it took significantly longer before. Great! > the issues i've run into: > > - dynamic sack size > making number of sacks dynamic is a concern. previously, we said to have > sack size in conf file. the concern is that changing that option > incorrectly actually 'corrupts' the db to a state that it cannot recover > from. it will have stray unprocessed measures constantly. if we change > the db path incorrectly, we don't actually corrupt anything, we just > lose data. we've said we don't want sack mappings in indexer so it seems > to me, the only safe solution is to make it sack size static and only > changeable by hacking? Not hacking, just we need a proper tool to rebalance it. As I already wrote, I think it's good enough to have this documented and set to a moderated good value by default (e.g. 4096). There's no need to store it in a configuration file, it should be stored in the storage driver itself to avoid any mistake, when the storage is initialized via `gnocchi-upgrade'. > - sack distribution > to distribute sacks across workers, i initially implemented consistent > hashing. the issue i noticed is that because hashring is inherently has > non-uniform distribution[1], i would have workers sitting idle because > it was given less sacks, while other workers were still working. > > i tried also to implement jump hash[2], which improved distribution and > is in theory, less memory intensive as it does not maintain a hash > table. while better at distribution, it still is not completely uniform > and similarly, the less sacks per worker, the worse the distribution. > > lastly, i tried just simple locking where each worker is completely > unaware of any other worker and handles all sacks. it will lock the sack > it is working on, so if another worker tries to work on it, it will just > skip. this will effectively cause an additional requirement on locking > system (in my case redis) as each worker will make x lock requests where > x is number of sacks. so if we have 50 workers and 2048 sacks, it will > be 102K requests per cycle. this is in addition to the n number of lock > requests per metric (10K-1M metrics?). this does guarantee if a worker > is free and there is work to be done, it will do it. > > i guess the question i have is: by using a non-uniform hash, it seems we > gain possibly less load at the expense of efficiency/'speed'. the number > of sacks/tasks we have is stable, it won't really change. the number of > metricd workers may change but not constantly. lastly, the number of > sacks per worker will always be relatively low (10:1, 100:1 assuming max > number of sacks is 2048). given these conditions, do we need > consistent/jump hashing? is it better to just modulo sacks and ensure > 'uniform' distribution and allow for 'larger' set of buckets to be > reshuffled when workers are added? What about using the hashring with replicas (e.g. 3 by default) and a lock per sack? This should reduce largely the number of lock try that you see. If you have 2k sacks divided across 50 workers and each one has a replica, that make each process care about 122 metrics so they might send 122 acquire() try each, which is 50 × 122 = 6100 acquire request, 17 times less than 102k. This also solve the problem of non-uniform distribution, as having replicas make sure every node gets some work. You can then probably remove the per-metric-lock too: this is just used when processing new measures (here the sack lock is enough) and when expunging metrics. You can safely use the same lock sack-lock for expunging metric. We may just need to it out from janitor? Something to think about! Cheers, -- Julien Danjou -- Free Software hacker -- https://julien.danjou.info signature.asc Description: PGP signature __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [ceilometer]
On Mon, Apr 17 2017, Andres Alvarez wrote: > Hello everyone > > I am trying to get acquainted with the Ceilometer code base. I was reading > through the Telemetry Admin Guide > <https://docs.openstack.org/admin-guide/telemetry-data-pipelines.html> > regarding publishers and noticed that there are many deprecated publishers > such as *direct*, *kafka*, and *database*. Yet on the source code's master > branch > <https://github.com/openstack/ceilometer/tree/master/ceilometer/publisher> > can still see some of these publishers there. > > Is there a reason for this? Deprecated does not mean they are not shipped. That mean they should not be configured and used. They'll be removed in a next version. -- Julien Danjou ;; Free Software hacker ;; https://julien.danjou.info signature.asc Description: PGP signature __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [telemetry] [ceilometer] Difference between publishers and dispatchers?
On Mon, Apr 17 2017, Andres Alvarez wrote: Hi Andres, > I am a bit confused on what is the difference between dispatchers and > publishers in Ceilometer. The documentation explains a bit about publishers > in the pipeline, but it does not mention much (if anything) about > dispatchers. Publishers are configured into the pipeline to indicate where to push samples data (e.g. to Gnocchi). One of the publisher is notifier:// which sends the samples to the (now deprecated) ceilometer-collector process. Ceilometer collector stores data into other system via a dispatcher mechanism (e.g. to Gnocchi). It's now deprecated as it's just, with current architecture, a unnecessary step: publishers can do the job directly. -- Julien Danjou # Free Software hacker # https://julien.danjou.info signature.asc Description: PGP signature __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [tc] [elections] Available time and top priority
On Wed, Apr 12 2017, Thierry Carrez wrote: > One idea I've been considering is eliminating the one and only sacred > one-hour weekly TC meeting, and encouraging ad-hoc discussions (on > #openstack-dev for example) between change proposers and smaller groups > of TC members present in the same timezone. That would give us a good > feel of what everyone thinks, reduce noise, and happen at various times > during the day on a public forum, giving an opportunity for more people > to jump in the discussion. The informal nature of those discussions > would make the governance reviews the clear focal point for coordination > and final decision. > > It's clearly not the perfect silver bullet though, so I'd very much like > to hear other ideas :) +1 We ditched meeting a while ago in Telemetry without any regret. Discussions happen when needed on IRC and when people are around, and if they're not or it requires more thinking we just go to email. Requiring people to connect and block a full hour at any time of the day (or night) never have been a good and productive idea. Sometimes I feel like OpenStack stole this idea from "Worst Ideas In Enterprise Management Practice". ;) -- Julien Danjou /* Free Software hacker https://julien.danjou.info */ signature.asc Description: PGP signature __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [oslo][infra][pbr] Nominating Stephen Finucane for pbr-core
On Wed, Apr 12 2017, Monty Taylor wrote: > Recently Stephen Finucane (sfinucan) has stepped up to the plate to help sort > out issues we've been having. He's shown a lack of fear of the codebase and an > understanding of what's going on. He's also following through on patches to > projects themselves when needed, which is a huge part of the game. And most > importantly he knows when to suggest we _not_ do something. +1, he's done a good job. :) -- Julien Danjou # Free Software hacker # https://julien.danjou.info signature.asc Description: PGP signature __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [oslo][devstack][all] ZooKeeper vs etcd for Tooz/DLM
On Thu, Mar 30 2017, Davanum Srinivas wrote: > Julien, > > Yes, Jay is adding this driver for "etcd3://" using the > https://pypi.python.org/pypi/etcd3 library > https://review.openstack.org/#/c/447223/ > > and i'll followup with a "etcd3+http://; driver using the > https://pypi.python.org/pypi/etcd3gw library Ah ok, works for me. I'm not sure I see the value of using etc3gw which is IIUC just a wrapper around requests. For the etcd:// driver in tooz we just went ahead and used requests directly, so I imagine that's what I'd do for etcd3+http:// :-) -- Julien Danjou -- Free Software hacker -- https://julien.danjou.info signature.asc Description: PGP signature __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [oslo][devstack][all] ZooKeeper vs etcd for Tooz/DLM
On Thu, Mar 30 2017, Davanum Srinivas wrote: > Update.. So i just proposed a new library built off the [3] gist to be > added to g-r and u-c. > > https://review.openstack.org/#/c/450967/ > > The alternative was to add support for custom backends in etcd3 > library itself with grpc and v3alpha HTTP API as implementations. > Since tooz is already an abstraction and the work non-trivial we can > start this way. If etcd3 ends up with supporting both, then we can > just eliminate this library at that time. Is it simpler than just writing two tooz drivers, e.g. etcd3 and etcd3+http for example? -- Julien Danjou /* Free Software hacker https://julien.danjou.info */ signature.asc Description: PGP signature __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [telemetry] Moving Gnocchi out
Hi, After a lot of talk within the Gnocchi team, it appeared to us that Gnocchi, which has been wisely tagged as 'independent' since its inception, has a lot of potential usage outside of OpenStack directly. Being part of the big tent helped the project to be built, but it now appears that it restrains its adoption and contribution from users outside of the OpenStack realm. Therefore, the Gnocchi team has decided to move the project outside of the OpenStack Big Tent. As a first step, a patch has been submitted to the governance to delist the project from Telemetry: https://review.openstack.org/447438 As a second step, the project will likely move out of the OpenStack infrastructure in the future. We expect Gnocchi to continue to thrive and be used by OpenStack, such as Ceilometer, which Gnocchi is now its primary storage backend. Gnocchi will also continue to be developed with support for OpenStack base services. And if you have any other question, feel free to ask us! Cheers, -- Julien Danjou ;; Free Software hacker ;; https://julien.danjou.info signature.asc Description: PGP signature __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [ptls] Project On-Boarding Rooms
On Thu, Mar 16 2017, Kendall Nelson wrote: […] > If there are any other projects willing to share a slot together please let > me know! We're ok to share our slot, a 90 minutes slot is plenty enough for us. I'd be (happily) surprised we get people and things going for that long. :) -- Julien Danjou ;; Free Software hacker ;; https://julien.danjou.info signature.asc Description: PGP signature __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [ptls] Project On-Boarding Rooms
On Wed, Mar 15 2017, Kendall Nelson wrote: > As you may have seen in a previous thread [1] the Forum will offer project > on-boarding rooms! This idea is that these rooms will provide a place for > new contributors to a given project to find out more about the project, > people, and code base. The slots will be spread out throughout the whole > Summit and will be 90 min long. > > We have a very limited slots available for interested projects so it will > be a first come first served process. Let me know if you are interested and > I will reserve a slot for you if there are spots left. Pretty sure we could on-board interested people in Telemetry too. :) -- Julien Danjou # Free Software hacker # https://julien.danjou.info signature.asc Description: PGP signature __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [oslo][devstack][all] ZooKeeper vs etcd for Tooz/DLM
On Wed, Mar 15 2017, Monty Taylor wrote: > On 03/15/2017 02:44 PM, Julien Danjou wrote: >> On Wed, Mar 15 2017, Davanum Srinivas wrote: >> >>> Yep, jd__ and i confirmed that things work with 3.x >> >> Though to be clear, what's used in tooz is the v2 HTTP API, not the new >> v3 gRPC API. > > But if it conceptually works with v3 server using v2 http api, then we > should be able to iterate in support for grpc api as patches to tooz and > have everything be happy, right? Definitely! -- Julien Danjou -- Free Software hacker -- https://julien.danjou.info signature.asc Description: PGP signature __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
