Re: [openstack-dev] [sahara] Nominate Telles Mota Vidal Nóbrega for core team

+2 On Fri, 2016-08-12 at 00:56 +0800, lu jander wrote: > +2 from me thx Telles > > 2016-08-12 0:20 GMT+08:00 Sergey Reshetnyak > : > +2 from me > > 2016-08-11 19:15 GMT+03:00 Sergey Lukjanov > : >

Re: [openstack-dev] Sahara Job Binaries Storage

is, > has there been any progress to deprecate sahara jobs -> internal db mechanism? > and/or the config option to disable internal db storage? > > Regards, > > Jerico > > > > > On 18 Mar 2016, at 12:55 AM, Trevor McKay <tmc...@redhat.com> wrote: > > >

Re: [openstack-dev] [sahara] Nominating new members to Sahara Core

+2 for all On Fri, 2016-05-13 at 18:33 +0300, Vitaly Gridnev wrote: > Hello Sahara core folks! > > > I'd like to bring the following folks to Sahara Core: > > > 1. Lu Huichun > 2. Nikita Konovalov > 3. Chad Roberts > > > Let's vote with +2/-2 for additions above. > > > [0]

Re: [openstack-dev] [Sahara][QA] Notes about the move of the Sahara Tempest API test to sahara-tests

Thanks Luigi, sounds good to me. I'll be happy to help with reviews/approvals as needed. Trevor On Mon, 2016-03-21 at 12:05 +0100, Luigi Toscano wrote: > On Monday 21 of March 2016 10:50:30 Evgeny Sikachev wrote: > > Hi, Luigi! > > > > Thanks for this short spec :) > > Changes looking good

Re: [openstack-dev] Sahara Job Binaries Storage

Hi Jerico, Internal db storage for job binaries was added at the start of EDP as an alternative for sites that do not have swift running. Since then, we've also added integration with manila so that job binaries can be stored in manila shares. You are correct, storing lots of binaries in the

Re: [openstack-dev] Sahara Job Binaries Storage

Hi Jerico, Internal db storage for job binaries was added at the start of EDP as an alternative for sites that do not have swift running. Since then, we've also added integration with manila so that job binaries can be stored in manila shares. You are correct, storing lots of binaries in the

Re: [openstack-dev] [sahara]FFE Request for resume EDP job

and approved, I the FFE should be granted. Best, Trev On Mon, 2016-03-07 at 09:07 -0500, Trevor McKay wrote: > For some reason the link below is wrong for me, it goes to a different > review. Here is a good one (I hope!): > > https://review.openstack.org/#/c/285839/ > > Trev >

Re: [openstack-dev] [sahara]FFE Request for resume EDP job

For some reason the link below is wrong for me, it goes to a different review. Here is a good one (I hope!): https://review.openstack.org/#/c/285839/ Trev On Mon, 2016-03-07 at 14:28 +0800, lu jander wrote: > Hi folks, > > I would like to request a FFE for the feature “Resume EDP job”: > >

Re: [openstack-dev] [sahara] shelve/unshelve cluster

And of course, don't forget the sahara specs repo: https://github.com/openstack/sahara-specs This is always a good way to propose a new idea. Create a high-level blueprint, then submit an associated spec through gerrit in the openstack/sahara-specs project. Trev On Mon, 2016-02-01 at 18:59

Re: [openstack-dev] [sahara] Proposing Vitaly Gridnev to core reviewer team

Vitaly, yes, welcome! I would have voted +1 but I was on PTO :) Trev On Thu, 2015-10-15 at 10:51 -0400, michael mccune wrote: > congrats Vitaly! > > On 10/15/2015 10:38 AM, Sergey Lukjanov wrote: > > I think we have a quorum. > > > > Vitaly, congrats! > > > > On Tue, Oct 13, 2015 at 6:39 PM,

Re: [openstack-dev] [Openstack] [Horizon] [Sahara] FFE request for Sahara unified job interface map UI

+1 from me as well. It would be a shame to see this go to the next cycle. On Fri, 2015-09-04 at 10:40 -0400, Ethan Gafford wrote: > Hello all, > > I request a FFE for the change at: https://review.openstack.org/#/c/209683/ > > This change enables a significant improvement to UX in Sahara's

Re: [openstack-dev] [sahara] Proposing Ethan Gafford for the core reviewer team

Hi Telles, you technically don't get a vote, but thanks anyway :) Trev On Fri, 2015-08-14 at 12:14 +, Telles Nobrega wrote: +1 On Fri, Aug 14, 2015 at 7:11 AM Alexander Ignatov aigna...@mirantis.com wrote: +1 Regards, Alexander Ignatov

Re: [openstack-dev] [sahara] Proposing Ethan Gafford for the core reviewer team

Flavio, thanks, bad joke on my part. I work with Telles on Sahara, just poking him in jest. Apologies, didn't mean to create an issue on the list. Trev On Fri, 2015-08-14 at 17:30 +0200, Flavio Percoco wrote: On 14/08/15 09:29 -0400, Trevor McKay wrote: Hi Telles, you technically don't

Re: [openstack-dev] [sahara] Proposing Ethan Gafford for the core reviewer team

+1, welcome addition On Thu, 2015-08-13 at 17:56 +0300, Sergey Lukjanov wrote: Hi folks, I'd like to propose Ethan Gafford as a member of the Sahara core reviewer team. Ethan contributing to Sahara for a long time and doing a great job on reviewing and improving Sahara. Here are the

Re: [openstack-dev] [Sahara] [EDP] about get_job_status in oozie engine

Hi Lu, yes, you're right. Return is a dictionary and for the other EDP engines only status is returned (and we primarily care about status). For Oozie, there is more information. I'm fine with changing the name to get_job_info() throughout the job_manager and EDP. It actually raises the

Re: [openstack-dev] [sahara][CDH] Is it possible to add CDH5.4 into Kilo release now?

Kilo is closed to everything but critical bugs at this point. Going forward, this change probably counts as a new feature and is forbidden as a backport to Kilo too. So Liberty is the earliest opportunity. https://wiki.openstack.org/wiki/StableBranch#Appropriate_Fixes Trev On Wed, 2015-04-29

Re: [openstack-dev] About Sahara EDP New Ideas for Liberty

Hi Ken, responses inline On Wed, 2015-04-22 at 12:36 +, Chen, Ken wrote: Hi Trevor, I saw below items in Proposed Sprint Topics of sahara liberty. https://etherpad.openstack.org/p/sahara-liberty-proposed-sessions. I guess these are the EDP ideas we want to discuss on Vancouver design

Re: [openstack-dev] About Sahara EDP New Ideas for Liberty

On Wed, 2015-04-22 at 12:36 +, Chen, Ken wrote: o more complex workflows (job dependencies, DAGs, etc. Do we rely on Oozie, or something else? Huichun is now figuring this. I am not whether you guys already have some detail ideas about this? If needed we can contribute some

Re: [openstack-dev] Global Cluster Template in Sahara

Thanks, Sergey. I agree -- my first thought was default templates plus new ACLs! Yanchao, note that with the current default template mechanism, the templates can be added per tenant -- you can run the script multiple times and change the tenant id. We thought this provided enough functionality

Re: [openstack-dev] About Sahara EDP New Ideas for Liberty

Weiting, Andrew, Agreed, great ideas! As Andrew noted, we have discussed some of these things before and it would be great to discuss them in Vancouver. I think that a Sahara-side workflow manager is the right approach. Oozie has a lot of capability for job coordination, but it won't work for

Re: [openstack-dev] [Sahara][Horizon] Can't open Data Processing panel after update sahara horizon

Hi Li, I am using a fresh devstack with Horizon deployed as part of devstack. I am running Sahara separately from the command line from the git sources (master branch). I use a little script to register the Sahara endpoint so that Horizon sees it. The only change I had to make was to

[openstack-dev] [sahara] Feedback on default-templates implementation

Hi Sahara folks, please checkout https://review.openstack.org/#/c/159872/ and respond there in comments, or here on the email thread. We have some things to figure out for this new CLI, and I want to make sure that we make sane choices and set a good precedent. Once we have a consensus we

[openstack-dev] [stable] New to project stable maintenance, question on requirements changes

Hi folks, I've just joined the stable maintenance team for Sahara. We have this review here, from OpenStack proposal bot: https://review.openstack.org/158775/ Since it came from the proposal bot, there's no justification in the commit message and no cherry pick. I didn't see this case

Re: [openstack-dev] [stable] New to project stable maintenance, question on requirements changes

Sean, thanks! I feel better already. I'll check out the review. Trevor On Tue, 2015-02-24 at 14:39 -0500, Sean Dague wrote: On 02/24/2015 02:34 PM, Trevor McKay wrote: Hi folks, I've just joined the stable maintenance team for Sahara. We have this review here, from OpenStack

[openstack-dev] [sahara] Shell Action, Re: Running HBase Jobs (was: About Sahara Oozie plan)

job.xml -run Best, Trev On Thu, 2015-02-12 at 08:39 -0500, Trevor McKay wrote: Hi Lu, folks, I've been investigating how to run Java actions in Sahara EDP that depend on HBase libraries (see snippet from the original question by Lu below). In a nutshell, we need to use Oozie sharelibs

Re: [openstack-dev] [sahara] Shell Action, Re: Running HBase Jobs (was: About Sahara Oozie plan)

message[${wf:errorMessage(wf:lastErrorNode())}]/message /kill end name='end' / /workflow-app On Thu, 2015-02-12 at 17:15 -0500, Trevor McKay wrote: Hi folks, Here is another way to do this. Lu had mentioned Oozie shell actions previously. Sahara doesn't support them, but I played

[openstack-dev] [sahara] Running HBase Jobs (was: About Sahara Oozie plan)

Hi Lu, folks, I've been investigating how to run Java actions in Sahara EDP that depend on HBase libraries (see snippet from the original question by Lu below). In a nutshell, we need to use Oozie sharelibs for this. I am working on a spec now, thinking about the best way to support this in

[openstack-dev] [sahara] Spark CDH followup and questions related to DIB

but without other changes and address the above issues in L, or do we push on and try to resolve it all for Kilo? Best regards, Trevor On Wed, 2015-01-28 at 11:57 -0500, Trevor McKay wrote: Daniele, Excellent! I'll have to keep a closer eye on bigfoot activity :) I'll pursue this. Best

Re: [openstack-dev] About Sahara Oozie plan

Hi, Thanks for your patience. I have been consumed with spark-swift, but I can start to address these questions now :) On (1) (a) below, I will try to reproduce and look at how we can better support classpath in EDP. I'll let you know what I find. We may need to add some configuration options

Re: [openstack-dev] About Sahara Oozie plan

Answers to other questions: 2) (first part) Yes, I think Oozie shell actions are a great idea. I can help work on a spec for this. In general, Sahara should be able to support any kind of Oozie action. Each will require a new job type, changes to the Oozie engine, and a UI form to handle

[openstack-dev] [sahara] About Sahara Oozie plan and Spark CDH Issues

Intel folks, Belated welcome to Sahara! Thank you for your recent commits. Moving this thread to openstack-dev so others may contribute, cc'ing Daniele and Pietro who pioneered the Spark plugin. I'll respond with another email about Oozie work, but I want to address the Spark/Swift issue in

Re: [openstack-dev] [sahara] About Sahara Oozie plan and Spark CDH Issues

changes. Feel free to take the code, clean it up and submit for review. Dan On Wed, Jan 28, 2015 at 10:43:30AM -0500, Trevor McKay wrote: Intel folks, Belated welcome to Sahara! Thank you for your recent commits. Moving this thread to openstack-dev so others may contribute, cc'ing

Re: [openstack-dev] [sahara] Asia friendly IRC meeting time

+1 for me, too. But I agree with Mike -- we need input from Saratov and similar time zones. I think though that the folks in Saratov generally try to shift hours anyway, to better align with the rest of us. So I think something that works for the US will likely work for them. Trev On Tue,

[openstack-dev] [sahara] Discussion on cm_api for global requirement

Hello all, at our last Sahara IRC meeting we started discussing whether or not to add a global requirement for cm_api.py https://review.openstack.org/#/c/130153/ One issue (but not the only issue) is that cm_api is not packaged for Fedora, Centos, or Ubuntu currently. The global requirements

Re: [openstack-dev] [sahara] Nominate Michael McCune to sahara-core

+ 2 On 11/11/2014 12:37 PM, Sergey Lukjanov wrote: Hi folks, I'd like to propose Michael McCune to sahara-core. He has a good knowledge of codebase and implemented important features such as Swift auth using trusts. Mike has been consistently giving us very well thought out and constructive

Re: [openstack-dev] [sahara] Nominate Sergey Reshetniak to sahara-core

+2 On 11/11/2014 12:35 PM, Sergey Lukjanov wrote: Hi folks, I'd like to propose Sergey to sahara-core. He's made a lot of work on different parts of Sahara and he has a very good knowledge of codebase, especially in plugins area. Sergey has been consistently giving us very well thought out

Re: [openstack-dev] [sahara] changing host name and /etc/hosts in container

Zhidong, Thanks for your question. I personally don't have an answer, but I think we definitely should bring up the possibility of dockerization for Sahara at the design summit next week. It may be something we want to formalize for Kilo. Will you be at the summit? Just to be clear, are you

Re: [openstack-dev] [Sahara][FFE] Requesting exception for Swift trust authentication blueprint

Not sure how this is done, but I'm a core member for Sahara, and I hereby sponsor it. On Fri, 2014-09-05 at 09:57 -0400, Michael McCune wrote: hey folks, I am requesting an exception for the Swift trust authentication blueprint[1]. This blueprint addresses a security bug in Sahara and

Re: [openstack-dev] [sahara] integration tests in python-saharaclient

Yes, I wrote them. I use them all the time -- no typo that I know of. They are great for spinning up a cluster and running EDP jobs. They may need some polish, but the point is to test the whole chain of operations from the CLI. This is contrary to what most OpenStack projects traditionally do

Re: [openstack-dev] [sahara] integration tests in python-saharaclient

by the way, what typo? Trev On Wed, 2014-09-03 at 14:58 -0700, Andrew Lazarev wrote: Hi team, Today I've realized that we have some tests called 'integration' in python-saharaclient. Also I've found out that Jenkins doesn't use them and they can't be run starting from April because of

[openstack-dev] [sahara] Notes on developing Sahara Spark EDP to work with swift:// paths

Hi folks, I've updated this etherpad with notes from an investigation of Spark/Swift and the hadoop-openstack plugin carried in the sahara-extra repo. Following the notes there, I was able to access swift:// paths from Spark jobs on a Spark standalone cluster launched from Sahara and then

Re: [openstack-dev] [sahara] Notes on developing Sahara Spark EDP to work with swift:// paths

information about this patch to this mailing list about two months ago. All the best, Gil. From:Trevor McKay tmc...@redhat.com To:OpenStack Development Mailing List openstack-dev@lists.openstack.org Date:28/08/2014 06:22 PM Subject:[openstack-dev

Re: [openstack-dev] [Sahara] Swift authentication and backwards compatibility

Thoughts, rapidfire :) In short, I think we should plan on backward compat unless some stubborn technical problem gets in our way I think backward compatibility is a good idea. We can make the user/pass inputs for data objects optional (they are required currently), maybe even gray them

Re: [openstack-dev] [Sahara] Error config?

Hi, I believe if you look in the sahara log output you may see more messages about this. There should be some indicator of what the validation error was. Trevor On Wed, 2014-07-09 at 23:44 +0700, Dat Tran wrote: Thank matt, It's worked. Can i ask you a question, please? When i

Re: [openstack-dev] [Sahara] Spark plugin: EDP and Spark jobs

-server could be a good place to integrate a better job scheduler. Trevor McKay opened a pad here: https://etherpad.openstack.org/p/sahara_spark_edp to gather ideas and feedback. This email is based on the very preliminary discussion that happened yesterday via IRC, email and the above

[openstack-dev] [sahara] Re: Spark plugin: EDP and Spark jobs

to the cluster. Hence, the spark job-server could be a good place to integrate a better job scheduler. Trevor McKay opened a pad here: https://etherpad.openstack.org/p/sahara_spark_edp to gather ideas and feedback. This email is based on the very preliminary discussion that happened

[openstack-dev] [sahara] Etherpad for discussion of spark EDP implementation

Hi folks, We've just started an investigation of spark EDP for Sahara. Please visit the etherpad and share your insights. Spark experts especially welcome! https://etherpad.openstack.org/p/sahara_spark_edp There are some design/roadmap decisions we need to make -- how are we going to do

Re: [openstack-dev] [sahara] summit wrap-up: subprojects

below, sahara-extra On Wed, 2014-05-28 at 20:02 +0400, Sergey Lukjanov wrote: Hey folks, it's a small wrap-up for the topic Sahara subprojects releasing and versioning that was discussed partially on summit and requires some more discussions. You can find details in [0]. common We'll

Re: [openstack-dev] [sahara] Nominate Trevor McKay for sahara-core

: Trevor, congrats! welcome to the sahara-core. On Thu, May 15, 2014 at 11:41 AM, Matthew Farrellee m...@redhat.com wrote: On 05/12/2014 05:31 PM, Sergey Lukjanov wrote: Hey folks, I'd like to nominate Trevor McKay (tmckay) for sahara-core. He is among the top reviewers

Re: [openstack-dev] [Nova] Tox issues on a clean environment

I am having a very similar issue with horizon, just today. I cloned the repo and started from scratch on master. tools/install_venv.py is trying to install cffi as a depdendency, ultimately fails with ImportError: cannot import name Feature This is Fedora 19. I know some folks on Fedora 20

Re: [openstack-dev] [savanna] Specific job type for streaming mapreduce? (and someday pipes)

in all Savanna subprojects; * eventually we'd like to use not only Oozie for EDP (for example, if we'll support Twitter Storm) and this new tools could require additional 'subtypes'. Thanks for catching this. On Tue, Feb 4, 2014 at 10:47 PM, Trevor McKay tmc...@redhat.com wrote

Re: [openstack-dev] savann-ci, Re: [savanna] Alembic migrations and absence of DROP column in sqlite

On 05 Feb 2014, at 01:01, Trevor McKay tmc...@redhat.com wrote: This brings up an interesting problem

Re: [openstack-dev] [savanna] Specific job type for streaming mapreduce? (and someday pipes)

it optional will not break backward compatibility. Adding database migration script is also pretty straightforward. Summarizing, my vote is on subtype field. Thanks, Andrew. On Mon, Feb 3, 2014 at 2:10 PM, Trevor McKay tmc...@redhat.com wrote: I was trying my

[openstack-dev] savann-ci, Re: [savanna] Alembic migrations and absence of DROP column in sqlite

on alembic side too. Good opportunity for contribution. Andrew. On Fri, Jan 31, 2014 at 6:12 AM, Trevor McKay tmc...@redhat.com wrote: Okay, I can accept that migrations shouldn't be supported on sqlite. However, if that's the case then we need to fix up savanna-db-manage so

[openstack-dev] [savanna] Specific job type for streaming mapreduce? (and someday pipes)

I was trying my best to avoid adding extra job types to support mapreduce variants like streaming or mapreduce with pipes, but it seems that adding the types is the simplest solution. On the API side, Savanna can live without a specific job type by examining the data in the job record.

Re: [openstack-dev] [savanna] Choosing provisioning engine during cluster launch

My mistake, it's already there. I missed the distinction between set on startup and set per cluster. Trev On Thu, 2014-01-30 at 10:50 -0500, Trevor McKay wrote: +1 How about an undocumented config? Trev On Thu, 2014-01-30 at 09:24 -0500, Matthew Farrellee wrote: i imagine

Re: [openstack-dev] [savanna] Choosing provisioning engine during cluster launch

+1 How about an undocumented config? Trev On Thu, 2014-01-30 at 09:24 -0500, Matthew Farrellee wrote: i imagine this is something that can be useful in a development and testing environment, especially during the transition period from direct to heat. so having the ability is not

[openstack-dev] [savanna] Alembic migrations and absence of DROP column in sqlite

I was playing with alembic migration and discovered that op.drop_column() doesn't work with sqlite. This is because sqlite doesn't support dropping a column (broken imho, but that's another discussion). Sqlite throws a syntax error. To make this work with sqlite, you have to copy the table to

Re: [openstack-dev] [savanna] How to handle diverging EDP job configuration settings

So, assuming we go forward with this, the followup question is whether or not to move main_class and java_opts for Java actions into edp.java.main_class and edp.java.java_opts configs. I think yes. Best, Trevor On Wed, 2014-01-29 at 09:15 -0500, Trevor McKay wrote: On Wed, 2014-01-29 at 14:35

[openstack-dev] [savanna] Undoing a change in the alembic migrations

Hi Sergey, In https://review.openstack.org/#/c/69982/1 we are moving the 'main_class' and 'java_opts' fields for a job execution into the job_configs['configs'] dictionary. This means that 'main_class' and 'java_opts' don't need to be in the database anymore. These fields were just added in

[openstack-dev] [savanna] How to handle diverging EDP job configuration settings

Hello all, In our first pass at EDP, the model for job settings was very consistent across all of our job types. The execution-time settings fit into this (superset) structure: job_configs = {'configs': {}, # config settings for oozie and hadoop 'params': {}, # substitution

Re: [openstack-dev] [savanna] savannaclient v2 api

We should consider turning mains into a string instead of a list for v2. Hive and Pig Oozie actions use mains, and each may only specify a single script element. There is no utility akaik for having multiple mains associated with a job. Workflows with multiple actions might need multiple mains,

Re: [openstack-dev] [savanna] why swift-internal:// ?

Matt et al, Yes, swift-internal was meant as a marker to distinguish it from swift-external someday. I agree, this could be indicated by setting other fields. Little bit of implementation detail for scope: In the current EDP implementation, SWIFT_INTERNAL_PREFIX shows up in essentially two

[openstack-dev] Savanna EDP sequence diagrams added for discussion...

Hi all, Here is a page to hold sequence diagrams for Savanna EDP, based on current launchpad blueprints. We thought it might be helpful to create some diagrams for discussion as the component specs are written and the API is worked out:

Re: [openstack-dev] [Savanna-all] Savanna EDP sequence diagrams added for discussion...

fyi, updates to the diagram based on feedback On Thu, 2013-07-18 at 13:49 -0400, Trevor McKay wrote: Hi all, Here is a page to hold sequence diagrams for Savanna EDP, based on current launchpad blueprints. We thought it might be helpful to create some diagrams for discussion