Re: [openstack-dev] [sahara] Nominate Telles Mota Vidal Nóbrega for core team

[lu jander]

+2 from me thx Telles

2016-08-12 0:20 GMT+08:00 Sergey Reshetnyak :

> +2 from me
>
> 2016-08-11 19:15 GMT+03:00 Sergey Lukjanov :
>
>> +2
>>
>> On Thu, Aug 11, 2016 at 8:48 AM, Elise Gafford 
>> wrote:
>>
>>> Hearty +2. Telles has been working on Sahara for years, and has been a
>>> consistent and incisive reviewer and code contributor.
>>>
>>> Congratulations Telles; very well deserved!
>>>
>>> - Elise
>>>
>>> On Thu, Aug 11, 2016 at 11:37 AM, Vitaly Gridnev 
>>> wrote:
>>>
 Hello core team,

 I'd like to nominate Telles Mota Vidal Nóbrega for core reviewer team.
 Let's vote with +2/-2 for his candidacy. Review/Commits stats can be found
 at [0].

 [0] http://stackalytics.com/?module=sahara-group_id=tellesmvn

 --
 Best Regards,
 Vitaly Gridnev,
 Project Technical Lead of OpenStack DataProcessing Program (Sahara)
 Mirantis, Inc

 
 __
 OpenStack Development Mailing List (not for usage questions)
 Unsubscribe: openstack-dev-requ...@lists.op
 enstack.org?subject:unsubscribe
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


>>>
>>> 
>>> __
>>> OpenStack Development Mailing List (not for usage questions)
>>> Unsubscribe: openstack-dev-requ...@lists.op
>>> enstack.org?subject:unsubscribe
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>
>>>
>>
>>
>> --
>> Sincerely yours,
>> Sergey Lukjanov
>> Sr. Development Manager
>> Mirantis Inc.
>>
>> 
>> __
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscrib
>> e
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>>
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [sahara]FFE Request for resume EDP job

[lu jander]

Hi folks,

I would like to request a FFE for the feature “Resume EDP job”:



BP:
https://blueprints.launchpad.net/sahara/+spec/add-suspend-resume-ability-for-edp-jobs





Spec has been merged. https://review.openstack.org/#/c/198264/


Suspend EDP patch has been merged.
https://review.openstack.org/#/c/201448/





Code Review: https://review.openstack.org/#/c/285839/




code is ready for review.



The Benefits for this change: after suspend job, we can resume this job.



The Risk: The risk would be low for this patch, since the code of suspend
patch has been long time reviewed.



Thanks,

luhuichun
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [sahara] Scheduler EDP patch request to be merged

[lu jander]

Hi, List



Thx for reminding to add release notes for new features.



I have add the release notes for scheduling EDP jobs
https://review.openstack.org/#/c/268881/





*https://review.openstack.org/#/c/182310/45
 *

This patch has been long time reviewed and passed the integration test for
scheduler EDP jobs,  so request to merge before mitaka-2.



Thx
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [sahara] FFE request for scheduler and suspend EDP job for sahara

[lu jander]

Hi Vitaly,
enable-scheduled-edp-jobs
<https://blueprints.launchpad.net/sahara/+spec/enable-scheduled-edp-jobs>
patch has 34 patch sets review. https://review.openstack.org/#/c/182310/ ,
it has no impact with another working in process patch.

2015-09-07 18:48 GMT+08:00 Vitaly Gridnev <vgrid...@mirantis.com>:

> Hey!
>
> From my point of view, we definetly should not give FFE for
> add-suspend-resume-ability-for-edp-jobs
> <https://blueprints.launchpad.net/sahara/+spec/add-suspend-resume-ability-for-edp-jobs>
>  spec,
> because client side for this change is not included in official liberty
> release.
>
> By the way, I am not sure about FFE for enable-scheduled-edp-jobs
> <https://blueprints.launchpad.net/sahara/+spec/enable-scheduled-edp-jobs>,
> because it's not clear which progress of these blueprint. Implementation of
> that consists with 2 patch-sets, and one of that marked as Work In Progress.
>
>
> On Sun, Sep 6, 2015 at 7:18 PM, lu jander <juvenboy1...@gmail.com> wrote:
>
>> Hi, Guys
>>
>>  I would like to request FFE for scheduler EDP job and suspend EDP job
>> for sahara. these patches has been reviewed for a long time with lots of
>> patch sets.
>>
>> Blueprint:
>>
>> (1)
>> https://blueprints.launchpad.net/sahara/+spec/enable-scheduled-edp-jobs
>> (2)
>> https://blueprints.launchpad.net/sahara/+spec/add-suspend-resume-ability-for-edp-jobs
>>
>>
>> Spec:
>>
>> (1) https://review.openstack.org/#/c/175719/
>> (2) https://review.openstack.org/#/c/198264/
>>
>>
>> Patch:
>>
>> (1) https://review.openstack.org/#/c/182310/
>> (2) https://review.openstack.org/#/c/201448/
>>
>>
>> __
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe:
>> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>>
>
>
> --
> Best Regards,
> Vitaly Gridnev
> Mirantis, Inc
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [sahara] FFE request for scheduler and suspend EDP job for sahara

[lu jander]

Hi, Guys

 I would like to request FFE for scheduler EDP job and suspend EDP job for
sahara. these patches has been reviewed for a long time with lots of patch
sets.

Blueprint:

(1) https://blueprints.launchpad.net/sahara/+spec/enable-scheduled-edp-jobs
(2)
https://blueprints.launchpad.net/sahara/+spec/add-suspend-resume-ability-for-edp-jobs


Spec:

(1) https://review.openstack.org/#/c/175719/
(2) https://review.openstack.org/#/c/198264/


Patch:

(1) https://review.openstack.org/#/c/182310/
(2) https://review.openstack.org/#/c/201448/
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [Sahara] [EDP] about get_job_status in oozie engine

[lu jander]

Hi Trevor

in sahara oozie engine (sahara/service/edp/oozie/engine.py
sahara/service/edp/oozie/oozie.py)

function get_job_status actually returns not only the status of the job,
but it returns all the info about the job, so i think that we should
 rename this function as get_job_info maybe more convenient for us? cause I
want add a function named get_job_info but i find that it already exists
here with a confused name.
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Sahara] [EDP] about get_job_status in oozie engine

[lu jander]

Hi Trevor
I am huichun

I agree with you, currently sahara edp engine and DB mixed with lots of
oozieness info, abstract incompletely.

i find this issue because i want add recurrence edp job in sahara, and with
oozie implementation, i find i need these oozie information not only the
status value.

so i will make a little change with current code with little impact, but in
the long term plan,  I think we should make a disscussion about refactoring
current edp engine and oozie implementation

2015-06-23 21:17 GMT+08:00 Trevor McKay tmc...@redhat.com:

 Hi Lu,

   yes, you're right.  Return is a dictionary and for the other EDP
 engines only status is returned (and we primarily care about
 status).  For Oozie, there is more information.

   I'm fine with changing the name to get_job_info() throughout the
 job_manager and EDP.

   It actually raises the question for me about whether or not in the
 Oozie case we really even need the extra Oozie information in the Sahara
 database.  I don't think we use it anywhere, not even sure the UI
 displays it (but it might) or how much comes through the REST responses.

   Maybe we should have get_job_status() which returns only status, and
 an optional get_job_info() that returns more? But that may be a bigger
 discussion.

 Best,

 Trevor

 On Tue, 2015-06-23 at 15:18 +0800, lu jander wrote:
  Hi Trevor
 
 
  in sahara oozie engine (sahara/service/edp/oozie/engine.py
  sahara/service/edp/oozie/oozie.py)
 
 
  function get_job_status actually returns not only the status of the
  job, but it returns all the info about the job, so i think that we
  should  rename this function as get_job_info maybe more convenient for
  us? cause I want add a function named get_job_info but i find that it
  already exists here with a confused name.



__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [Sahara] improve oozie engine for common lib management

[lu jander]

Hi, All
Currently, oozie share lib is not well known and can be hardly used by the
users, so I think we can  make it less oozieness and more friendly for the
users, it can be used for running jobs which are using third party libs. If
many jobs use the same libs, oozie share lib can make it as a common share
lib.

here is the bp,
https://blueprints.launchpad.net/sahara/+spec/improve-oozie-share-lib
I will write a spec soon after scheduled edp jobs bp.
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [Sahara] improve oozie engine for common lib management

[lu jander]

Hi Sergey
yes, it is based on job binaries. what we should do is provide a user
friendly interface, to let user know how to set oozie.libpath( may be we
can show it as share lib path in the job execution config horizon page)
 there are two scenarios as below:
(1) if Job A and Job B all need lib A, we can upload lib A through job
binary into sahara, then before we run Job A and B, user can set
oozie.libpath to the path where lib A exists.
(2) if Job A and Job B need system lib or other libs already exists in the
OS, user needs not to upload additional job binary

2015-05-12 0:49 GMT+08:00 Sergey Lukjanov slukja...@mirantis.com:

 Hi,

 do you think it could be implemented based on job binaries? It sounds like
 it's a type of job binary that should be always uploaded to Oozie for the
 tenant where it lives. (A bit crazy, but could useful).

 Thanks.

 On Mon, May 11, 2015 at 6:39 PM, lu jander juvenboy1...@gmail.com wrote:

 Hi, All
 Currently, oozie share lib is not well known and can be hardly used by
 the users, so I think we can  make it less oozieness and more friendly for
 the users, it can be used for running jobs which are using third party
 libs. If many jobs use the same libs, oozie share lib can make it as a
 common share lib.

 here is the bp,
 https://blueprints.launchpad.net/sahara/+spec/improve-oozie-share-lib
 I will write a spec soon after scheduled edp jobs bp.

 __
 OpenStack Development Mailing List (not for usage questions)
 Unsubscribe:
 openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev




 --
 Sincerely yours,
 Sergey Lukjanov
 Sahara Technical Lead
 (OpenStack Data Processing)
 Principal Software Engineer
 Mirantis Inc.

 __
 OpenStack Development Mailing List (not for usage questions)
 Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] About Sahara EDP New Ideas for Liberty

[lu jander]

Hi Trevor

you mean adding an abstract layer for basic function like run job, cancel
job etc, the we implement these by specific job engine like oozie, Ooyala
job server etc, right ?

so I will do two things as below:
(1) working on the scheduler edp jobs by oozie coordination so make sure we
can running scheduler edp jobs  (currently working on ,will submit patch
later)
(2) abstract the basic function as the abstract layer, and rewrite the
specific job engine like oozie and ooyala job server. ( after doing first
step, will talk it with you)

so do you think  that I have caught you meanings? or missed something?

2015-04-22 21:59 GMT+08:00 Trevor McKay tmc...@redhat.com:

 On Wed, 2015-04-22 at 12:36 +, Chen, Ken wrote:
  o more complex workflows (job dependencies, DAGs, etc. Do we rely on
 Oozie, or something else?
   Huichun is now figuring this. I am not whether you guys already
 have some detail ideas about this? If needed we can contribute some effort.
 If no details are ready, we can help draw a draft version first.

 I just made a note on the pad

 https://etherpad.openstack.org/p/sahara-liberty-proposed-sessions

 Maybe the right approach here is to develop a mapping notation that can
 be expressed as a JSON object (like the proposed job interface mapping).

 If we can develop an abstract way to describe relationships between
 jobs, then the individual EDP engines can implement it. For the Oozie
 EDP engine, maybe it uses Oozie features in workflows.  For Spark, or
 Storm, maybe it uses some existing opensource coordinator or one is
 written.

 The key idea would be to make job coordination part of the EDP engine,
 with a well defined set of objects to describe the relationships.

 What do you think? Just a rough idea.  Maybe there is a better way.



 __
 OpenStack Development Mailing List (not for usage questions)
 Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Sahara HA discussion

[lu jander]

Hi, Sergey

we are in the same phase like you, I have noticed that there is a topic in
the https://etherpad.openstack.org/p/sahara-liberty-proposed-sessions

currently we decide to do the HA on service level (HDFS etc), here is the
doc
http://www.cloudera.com/content/cloudera/en/documentation/core/latest/topics/admin_ha.html

but I have heard that you will focus on the node level HA? for example.
 like rebuild a node when one node is failed?

2015-04-23 16:33 GMT+08:00 Sergey Reshetnyak sreshetn...@mirantis.com:

 Hi Lu

 I'm going to start working on HA support for Sahara  for HDP and CDH
 plugins. Now I didn't create specs or blueprints about HA. Also I don't
 have code for HA support.
 When are you going to start implement HA for CDH?

 Thanks
 Sergey

 2015-04-20 4:06 GMT+03:00 Lu, Huichun huichun...@intel.com:

  Hi Sergey

 Last IRC meeting, I heard that you are currently working on the HA on CDH
 and HDP, by chance that we just raise a bp about HA, so do you have any bp
 or spec about this topic? I think it’s interesting about this topic, we can
 have some discussion.

 https://blueprints.launchpad.net/sahara/+spec/cdh-ha-support





 thx Sergey ^^



 __
 OpenStack Development Mailing List (not for usage questions)
 Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Global Cluster Template in Sahara

[lu jander]

We have already implement the default template for sahara

https://blueprints.launchpad.net/sahara/+spec/default-templates

2015-04-16 5:22 GMT+08:00 Liang, Yanchao yanli...@ebay.com:

  Dear Openstack Developers,

  My name is Yanchao Liang. I am a software engineer in eBay, working on
 Hadoop as a Service on top of Openstack cloud.

  Right now we are using Sahara, Juno version. We want to stay current and
 introduce global template into sahara.

  In order to simplify the cluster creation process for user, we would
 like to create some cluster templates available for all users. User can
 just go to the horizon webUI, select one of the pre-popluated templates and
 create a hadoop cluster, in just a few clicks.

  Here is how I would implement this feature:

- In the database, Create a new column in “cluster_templates table
called “is_global”, which is a boolean value indicating whether the
template is available for all users or not.
- When user getting the cluster template from database,  add another
function similar to “cluster_template_get”, which query the database for
global templates.
- When creating cluster, put the user’s tenant id in
the “merged_values” config variable, instead of the tenant id from cluster
template.
- Use an admin account create and manage global cluster templates

 Since I don’t know the code base as well as you do, what do you think
 about the global template idea? How would you implement this new feature?

  We would like to contribute this feature back to the Openstack
 community. Any feedback would be greatly appreciated. Thank you.

  Best,
 Yanchao


 __
 OpenStack Development Mailing List (not for usage questions)
 Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [sahara] sahara Integration tests issue

[lu jander]

Hi Sahara dev,

I am working on the integration tests in sahara(with nova network), when I
am using tox -e cdh  to run cdh tests, it failed with error below, then I
check the sahara log, it says heat error.  This error reminds me a bug
which is not merged https://bugs.launchpad.net/sahara/+bug/1392738 (I
checkout this patch, and it seems still does't work)   so I manually set
auto_security_group false in test_cdh_gating.py, but still met this error
and not successfully passed the integration test.

below is Integration tests error and sahara log


Tests Error:

Traceback (most recent call last):
  File
/opt/stack/sahara/sahara/tests/integration/tests/gating/test_cdh_gating.py,
line 305, in test_cdh_plugin_gating
self._create_cluster()
  File sahara/tests/integration/tests/base.py, line 49, in wrapper
ITestCase.print_error_log(message, e)
  File
/opt/stack/sahara/.tox/integration/local/lib/python2.7/site-packages/oslo/utils/excutils.py,
line 82, in __exit__
six.reraise(self.type_, self.value, self.tb)
  File sahara/tests/integration/tests/base.py, line 46, in wrapper
fct(*args, **kwargs)
  File
/opt/stack/sahara/sahara/tests/integration/tests/gating/test_cdh_gating.py,
line 191, in _create_cluster
self.poll_cluster_state(self.cluster_id)
  File sahara/tests/integration/tests/base.py, line 237, in
poll_cluster_state
self.fail('Cluster state == \'Error\'.')
  File
/opt/stack/sahara/.tox/integration/local/lib/python2.7/site-packages/unittest2/case.py,
line 666, in fail
raise self.failureException(msg)
AssertionError: Cluster state == 'Error'.
Ran 1 tests in 147.204s (+21.220s)
FAILED (id=47, failures=1)
error: testr failed (1)
ERROR: InvocationError: '/opt/stack/sahara/.tox/integration/bin/python
setup.py test --slowest --testr-args=cdh'


Sahara Log:

2014-12-01 09:02:44.001 ERROR sahara.service.ops [-] Error during operating
cluster 'test-cluster-cdh' (reason: Heat stack failed with status
CREATE_FAILED)
2014-12-01 09:02:44.001 TRACE sahara.service.ops Traceback (most recent
call last):
2014-12-01 09:02:44.001 TRACE sahara.service.ops   File
/opt/stack/sahara/sahara/service/ops.py, line 141, in wrapper
2014-12-01 09:02:44.001 TRACE sahara.service.ops f(cluster_id, *args,
**kwds)
2014-12-01 09:02:44.001 TRACE sahara.service.ops   File
/opt/stack/sahara/sahara/service/ops.py, line 227, in _provision_cluster
2014-12-01 09:02:44.001 TRACE sahara.service.ops
INFRA.create_cluster(cluster)
2014-12-01 09:02:44.001 TRACE sahara.service.ops   File
/opt/stack/sahara/sahara/service/heat_engine.py, line 57, in
create_cluster
2014-12-01 09:02:44.001 TRACE sahara.service.ops
launcher.launch_instances(cluster, target_count)
2014-12-01 09:02:44.001 TRACE sahara.service.ops   File
/opt/stack/sahara/sahara/service/heat_engine.py, line 209, in
launch_instances
2014-12-01 09:02:44.001 TRACE sahara.service.ops
heat.wait_stack_completion(stack.heat_stack)
2014-12-01 09:02:44.001 TRACE sahara.service.ops   File
/opt/stack/sahara/sahara/utils/openstack/heat.py, line 60, in
wait_stack_completion
2014-12-01 09:02:44.001 TRACE sahara.service.ops
2014-12-01 09:02:44.001 TRACE sahara.service.ops HeatStackException: Heat
stack failed with status CREATE_FAILED
2014-12-01 09:02:44.001 TRACE sahara.service.ops
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev