[openstack-dev] [sahara] integration tests in python-saharaclient

2014-09-03 Thread Andrew Lazarev
Hi team, Today I've realized that we have some tests called 'integration' in python-saharaclient. Also I've found out that Jenkins doesn't use them and they can't be run starting from April because of typo in tox.ini. Does anyone know what these tests are? Does anyone mind if I delete them since

Re: [openstack-dev] [sahara] integration tests in python-saharaclient

2014-09-04 Thread Andrew Lazarev
Trevor, by the way, what typo? https://review.openstack.org/#/c/118903/ Andrew. On Thu, Sep 4, 2014 at 7:58 AM, Trevor McKay tmc...@redhat.com wrote: by the way, what typo? Trev On Wed, 2014-09-03 at 14:58 -0700, Andrew Lazarev wrote: Hi team, Today I've realized that we have

Re: [openstack-dev] [Sahara] Working on DOC bug

2014-09-12 Thread Andrew Lazarev
Hi Sharan, Sahara uses python clients to communicate with other components. So, you can start with inspecting requirements.txt file on python-* dependancies. Next, you need to understand what is really used and how. Also, some components could be used indirectly via nova client (e.g. security

Re: [openstack-dev] [Sahara] Networking Service for Sahara

2014-09-17 Thread Andrew Lazarev
Hi Sharan, Sahara works with either network service installed in OpenStack. If OpenStack uses neutron - sahara will use neutron too. If nova network is used, Sahara supports that as well. Thanks, Andrew. On Wed, Sep 17, 2014 at 1:38 PM, Sharan Kumar M sharan.monikan...@gmail.com wrote: Hi

Re: [openstack-dev] [sahara] Juno release images

2014-09-23 Thread Andrew Lazarev
No fedora? Andrew. On Tue, Sep 23, 2014 at 1:16 PM, Sergey Reshetnyak sreshetn...@mirantis.com wrote: Hi sahara folks, I've prepared all new images for Sahara (using diskimage-builder) for Juno release. Vanilla plugin:

Re: [openstack-dev] [Sahara][Doc] Filing a bug for modifying the image

2014-09-24 Thread Andrew Lazarev
Hi Sharan, Yes, file bug, commit new image to review. Thanks, Andrew. On Wed, Sep 24, 2014 at 10:16 AM, Sharan Kumar M sharan.monikan...@gmail.com wrote: Hi all, In this documentation http://docs.openstack.org/developer/sahara/overview.html#details, the missing services were added. And I

Re: [openstack-dev] [savanna] why swift-internal:// ?

2014-01-23 Thread Andrew Lazarev
Matt, I can easily imagine situation when job binaries are stored in external HDFS or external SWIFT (like data sources). Internal and external swifts are different since we need additional credentials. Thanks, Andrew. On Thu, Jan 23, 2014 at 5:30 PM, Matthew Farrellee m...@redhat.com wrote:

Re: [openstack-dev] [savanna] why swift-internal:// ?

2014-01-23 Thread Andrew Lazarev
pass swift-external needs user pass ?auth url? best, matt On 01/23/2014 08:43 PM, Andrew Lazarev wrote: Matt, I can easily imagine situation when job binaries are stored in external HDFS or external SWIFT (like data sources). Internal and external swifts are different since we

Re: [openstack-dev] [savanna] why swift-internal:// ?

2014-01-24 Thread Andrew Lazarev
in icehouse. it's a pretty big limitation to only allow a single tenant. best, matt On 01/23/2014 11:15 PM, Andrew Lazarev wrote: Matt, For swift-internal we are using the same keystone (and identity protocol version) as for savanna. Also savanna admin tenant is used. Thanks, Andrew

Re: [openstack-dev] [savanna] How to handle diverging EDP job configuration settings

2014-01-29 Thread Andrew Lazarev
I like idea of edp. prefix. Andrew. On Wed, Jan 29, 2014 at 6:23 AM, Trevor McKay tmc...@redhat.com wrote: So, assuming we go forward with this, the followup question is whether or not to move main_class and java_opts for Java actions into edp.java.main_class and edp.java.java_opts configs.

Re: [openstack-dev] [savanna] Undoing a change in the alembic migrations

2014-01-29 Thread Andrew Lazarev
+1 on new migration script. Just to be consecutive. Andrew. On Wed, Jan 29, 2014 at 2:17 PM, Trevor McKay tmc...@redhat.com wrote: Hi Sergey, In https://review.openstack.org/#/c/69982/1 we are moving the 'main_class' and 'java_opts' fields for a job execution into the

Re: [openstack-dev] [savanna] Choosing provisioning engine during cluster launch

2014-01-29 Thread Andrew Lazarev
Alexander, What is the purpose of exposing this to user side? Both engines must do exactly the same thing and they exist in the same time only for transition period until heat engine is stabilized. I don't see any value in proposed option. Andrew. On Wed, Jan 29, 2014 at 8:44 PM, Alexander

Re: [openstack-dev] [savanna] Alembic migrations and absence of DROP column in sqlite

2014-01-31 Thread Andrew Lazarev
Trevor, Such check could be useful on alembic side too. Good opportunity for contribution. Andrew. On Fri, Jan 31, 2014 at 6:12 AM, Trevor McKay tmc...@redhat.com wrote: Okay, I can accept that migrations shouldn't be supported on sqlite. However, if that's the case then we need to fix up

Re: [openstack-dev] [savanna] Specific job type for streaming mapreduce? (and someday pipes)

2014-02-03 Thread Andrew Lazarev
I see two points: * having Savanna types mapped to Oozie action types is intuitive for hadoop users and this is something we would like to keep * it is hard to distinguish different kinds of one job type Adding 'subtype' field will solve both problems. Having it optional will not break backward

Re: [openstack-dev] savann-ci, Re: [savanna] Alembic migrations and absence of DROP column in sqlite

2014-02-04 Thread Andrew Lazarev
, Boris Pavlovic On Fri, Jan 31, 2014 at 7:31 PM, Andrew Lazarev alaza...@mirantis.com wrote: Trevor, Such check could be useful on alembic side too. Good opportunity for contribution. Andrew. On Fri, Jan 31, 2014 at 6:12 AM, Trevor McKay tmc...@redhat.com wrote

Re: [openstack-dev] [savanna] Mission Statement wording

2014-02-13 Thread Andrew Lazarev
Short version looks good for me. Andrew. On Thu, Feb 13, 2014 at 4:29 AM, Sergey Lukjanov slukja...@mirantis.comwrote: Hi folks, I'm working now on adding Savanna's mission statement to governance docs [0]. There are some comments on our current one to make it simpler and remove marketing

Re: [openstack-dev] [savanna] plugin version or hadoop version?

2014-02-17 Thread Andrew Lazarev
IDH uses version of IDH distro and there is no direct mapping between distro version and hadoop version. E.g. IDH 2.5.1 works with apache hadoop 1.0.3. I suggest to call the field as just 'version' everywhere and assume this version as plugin specific property. Andrew. On Mon, Feb 17, 2014 at

[openstack-dev] [savanna] floating ip pool by name

2014-02-27 Thread Andrew Lazarev
Hi Team, I was always using floating_ip_pool: net04_ext construction and it worked fine. Now it responds with validation error Floating IP pool net04_ext for node group 'manager' not found because https://bugs.launchpad.net/savanna/+bug/1282027 was merged and savanna expects only ID here. Is it

Re: [openstack-dev] [dev-env] Error setting up dev environment on Mac OS X (10.9.2)

2014-03-31 Thread Andrew Lazarev
Hi Bob, The error is because of https://bugs.launchpad.net/sahara/+bug/1283133. Fix is ready, but not merged to master because of FF ( https://review.openstack.org/#/c/75456/). As a workaround I could suggest temporarily removing DROP queries in migration script #3. Or use other sql db (e.g.

[openstack-dev] [sahara] Upgrade of Hadoop components inside released version

2014-06-24 Thread Andrew Lazarev
Hi Team, I want to raise topic about upgrade of components in Hadoop version that is already supported by released Sahara plugin. The question is raised because of several change requests [1] and [2]. Topic was discussed in Atlanta ([3]), but we didn't come to the decision. All of us agreed that

Re: [openstack-dev] [sahara] Nominate Trevor McKay for sahara-core

2014-05-13 Thread Andrew Lazarev
+1 It will be good addition to the core team. Thanks, Andrew. On Mon, May 12, 2014 at 2:31 PM, Sergey Lukjanov slukja...@mirantis.comwrote: Hey folks, I'd like to nominate Trevor McKay (tmckay) for sahara-core. He is among the top reviewers of Sahara subprojects. Trevor is working on

Re: [openstack-dev] [sahara] bug triage day after summit

2014-05-20 Thread Andrew Lazarev
I think May 26 was a random 'day after summit'. I'm Ok with May 27 too. Andrew. On Tue, May 20, 2014 at 10:16 AM, Sergey Lukjanov slukja...@mirantis.comwrote: I'm ok with moving it to May 27. On Tuesday, May 20, 2014, Michael McCune mimcc...@redhat.com wrote: I think in our eagerness to

Re: [openstack-dev] [sahara] summit wrap-up: backward compat

2014-05-28 Thread Andrew Lazarev
for juno we should just have a v1 api (there can still be a v1.1 endpoint, but it should be deprecated), and maybe a v2 api +1 any semantic changes require new major version number +1 api should only have a major number (no 1.1 or 2.1) In this case we will end up with new major number each

Re: [openstack-dev] [sahara] 2014.1.1 preparation

2014-06-03 Thread Andrew Lazarev
correction: https://review.openstack.org/#/c/96621/ (Added validate_edp method to Plugin SPI doc) has no sense without https://review.openstack.org/#/c/87573 (Fix running EDP job on transient cluster) where validate_edp was introduced. Andrew. On Mon, Jun 2, 2014 at 11:34 PM, Andrew Lazarev

Re: [openstack-dev] [sahara] 2014.1.1 preparation

2014-06-03 Thread Andrew Lazarev
https://review.openstack.org/#/c/93564/ has no sense without https://review.openstack.org/#/c/87573 +1 on merging DOC bugs you listed and these 2 EDP bugs Andrew. On Mon, Jun 2, 2014 at 11:08 PM, Sergey Lukjanov slukja...@mirantis.com wrote: /me proposing to backport: Docs:

Re: [openstack-dev] [sahara] team meeting Sept 25 1800 UTC

2014-09-25 Thread Andrew Lazarev
Thanks everyone who have joined Sahara meeting. Here are the logs from the meeting: http://eavesdrop.openstack.org/meetings/sahara/2014/sahara.2014-09-25-18.02.html http://eavesdrop.openstack.org/meetings/sahara/2014/sahara.2014-09-25-18.02.log.html Andrew. On Wed, Sep 24, 2014 at 2:50 PM,

Re: [openstack-dev] [Ceilometer] Adding pylint checking of new ceilometer patches

2014-10-07 Thread Andrew Lazarev
I can't say I'm too deeply versed in the code, but it's enough to make me wonder if we want to go that direction and avoid the issues altogether? It's the nature of python that methods and modules can be added in runtime and pylint can't do full analysis. That's why the best use of it - limited

Re: [openstack-dev] [Ceilometer] Adding pylint checking of new ceilometer patches

2014-10-07 Thread Andrew Lazarev
At first step we won't implement pylint as gate job, but will add it at master to have a possibility to check code with pylint locally, if it is needed. No much sense in running it locally. It has too many false positives. The best use - see new critical errors as a non-voting job. Andrew. On

Re: [openstack-dev] [all][oslo] projects still using obsolete oslo modules

2014-10-13 Thread Andrew Lazarev
Filed https://bugs.launchpad.net/sahara/+bug/1380725 for sahara stuff. Andrew. On Mon, Oct 13, 2014 at 6:20 AM, Doug Hellmann d...@doughellmann.com wrote: I’ve put together a little script to generate a report of the projects using modules that used to be in the oslo-incubator but that have

Re: [openstack-dev] [sahara] Nominate Michael McCune to sahara-core

2014-11-11 Thread Andrew Lazarev
+2 Andrew. On Tue, Nov 11, 2014 at 9:37 AM, Sergey Lukjanov slukja...@mirantis.com wrote: Hi folks, I'd like to propose Michael McCune to sahara-core. He has a good knowledge of codebase and implemented important features such as Swift auth using trusts. Mike has been consistently giving

Re: [openstack-dev] [sahara] Nominate Sergey Reshetniak to sahara-core

2014-11-11 Thread Andrew Lazarev
+2 Andrew. On Tue, Nov 11, 2014 at 9:35 AM, Sergey Lukjanov slukja...@mirantis.com wrote: Hi folks, I'd like to propose Sergey to sahara-core. He's made a lot of work on different parts of Sahara and he has a very good knowledge of codebase, especially in plugins area. Sergey has been

[openstack-dev] [sahara] team meeting Nov 20 1800 UTC

2014-11-20 Thread Andrew Lazarev
Minutes: *http://eavesdrop.openstack.org/meetings/sahara/2014/sahara.2014-11-20-18.01.html http://eavesdrop.openstack.org/meetings/sahara/2014/sahara.2014-11-20-18.01.html* Logs: *http://eavesdrop.openstack.org/meetings/sahara/2014/sahara.2014-11-20-18.01.log.html

[openstack-dev] [sahara] team meeting Feb 12 1400 UTC

2015-02-11 Thread Andrew Lazarev
Hi guys, We'll be having the Sahara team meeting tomorrow at #openstack-meeting-3 channel. Agenda: https://wiki.openstack.org/wiki/Meetings/SaharaAgenda#Next_meetings http://www.timeanddate.com/worldclock/fixedtime.html?msg=Sahara+Meetingiso=20150212T14 Thanks, Andrew

Re: [openstack-dev] [nova] [devstack] configuring https for glance client

2015-02-10 Thread Andrew Lazarev
...@linux.vnet.ibm.com wrote: On 2/9/2015 5:40 PM, Andrew Lazarev wrote: Hi Nova experts, Some time ago I figured out that devstack fails to stack with USE_SSL=True option because it doesn't configure nova to work with secured glace [1]. Support of secured glance was added to nova in Juno

[openstack-dev] [nova] [devstack] configuring https for glance client

2015-02-09 Thread Andrew Lazarev
Hi Nova experts, Some time ago I figured out that devstack fails to stack with USE_SSL=True option because it doesn't configure nova to work with secured glace [1]. Support of secured glance was added to nova in Juno cycle [2], but it looks strange for me. Glance client takes settings form

[openstack-dev] [nova] [devstack] configuring https for glance client

2015-02-09 Thread Andrew Lazarev
Hi Nova experts, Some time ago I figured out that devstack fails to stack with USE_SSL=True option because it doesn't configure nova to work with secured glace [1]. Support of secured glance was added to nova in Juno cycle [2], but it looks strange for me. Glance client takes settings form

Re: [openstack-dev] [Sahara] [Heat] validating properties of Sahara resources in Heat

2015-01-07 Thread Andrew Lazarev
Answers inlined and marked as [AL]. On Mon, Jan 5, 2015 at 5:17 AM, Pavlo Shchelokovskyy pshchelokovs...@mirantis.com wrote: Hi all, I would like to ask Sahara developers' opinion on two bugs raised against Heat's resources - [1] and [2]. Below I am going to repeat some of my comments from

Re: [openstack-dev] About Sahara EDP New Ideas for Liberty

2015-03-20 Thread Andrew Lazarev
Hi Weiting, 1. Add a schedule feature to run the jobs on time: This request comes from the customer, they usually run the job in a specific time every day. So it should be great if there is a scheduler to help arrange the regular job to run. Looks like a great feature. And should be quite easy

Re: [openstack-dev] [Sahara] Is it possible to do instance scale for a node in a living cluster ?

2015-03-23 Thread Andrew Lazarev
Hi Chen, 1. resize the master node. Using command like “nova resize”, to add more cpu memory and other resources for this single instance. Instance reside is not supported on Sahara level. But you can always do that on nova level. Sahara doesn't use any specific information about flavor

Re: [openstack-dev] Sahara HA discussion

2015-05-13 Thread Andrew Lazarev
Hi Lu, Topic in https://etherpad.openstack.org/p/sahara-liberty-proposed-sessions is about HA in Sahara itself. How to recover after openstack controller failure. It doesn't cover HA of clusters created by Sahara. Thanks, Andrew. On Sun, Apr 26, 2015 at 10:44 PM, lu jander

Re: [openstack-dev] [surge] Introducing Surge - rapid deploy/scale stream processing systems on OpenStack

2015-05-18 Thread Andrew Lazarev
As I see Surge is pretty much replicating Sahara functionality running in one process per host mode. Sahara currently supports much more features. Andrew. On Fri, May 15, 2015 at 10:38 AM, Joe Gordon joe.gord...@gmail.com wrote: On Fri, May 15, 2015 at 10:13 AM, Debojyoti Dutta

Re: [openstack-dev] [Sahara] improve oozie engine for common lib management

2015-05-18 Thread Andrew Lazarev
I think it could be useful and pretty easy to implement. Feel free to file blueprint+spec. Andrew. On Mon, May 11, 2015 at 9:49 AM, Sergey Lukjanov slukja...@mirantis.com wrote: Hi, do you think it could be implemented based on job binaries? It sounds like it's a type of job binary that

Re: [openstack-dev] [Sahara] Difference between Sahara and CloudBrak

2015-06-15 Thread Andrew Lazarev
Hi Jay, Cloudbreak is a Hadoop installation tool driven by Hortonworks. The main difference with Sahara is a point of control. In Hortonworks world you have Ambari and different planforms (AWS, OpenStack, etc.) to run Hadoop. Sahara point of view - you have OpenStack cluster and want to control

Re: [openstack-dev] [Sahara] Questions about how Sahara use trust ?

2015-07-10 Thread Andrew Lazarev
Hi Chen, As I remember, proxy users were added for security reasons. When one user creates cluster in Sahara he should not get access to data of other users. Thanks, Andrew. On Thu, Jul 9, 2015 at 11:12 PM, Li, Chen chen...@intel.com wrote: Hi Sahara guys, When sahara create a