Re: [Engine-devel] [Users] RHEVM 3.1 DB Schema diagram

2012-12-04 Thread Allon Mureinik
Nice!

How are the names of the blocks generated?
Some names strike me a bit there (e.g., images is in the storage_domains_static 
block).

- Original Message -
> From: "Eli Mesika" 
> To: "users" , "engine-devel" 
> Sent: Tuesday, December 4, 2012 7:12:28 PM
> Subject: [Users] RHEVM 3.1 DB Schema diagram
> 
> Hi
> 
> There were some requests on the users list to get our DB schema
> diagram.
> After looking a while for a tool for generating that , I have found
> the DBSchema tool.
> 
> Please find attached the RHEVM 3.1 DB Schema diagram.
> 
> I intend to get it into the oVirt site but currently it's too large
> and does not fit well.
> 
> Regards
> 
> Eli Mesika
> 
> ___
> Users mailing list
> us...@ovirt.org
> http://lists.ovirt.org/mailman/listinfo/users
> 
___
Engine-devel mailing list
Engine-devel@ovirt.org
http://lists.ovirt.org/mailman/listinfo/engine-devel


Re: [Engine-devel] [vdsm] RFC: New Storage API

2012-12-04 Thread Saggi Mizrahi


- Original Message -
> From: "Adam Litke" 
> To: "Saggi Mizrahi" 
> Cc: "VDSM Project Development" , 
> "engine-devel" 
> Sent: Tuesday, December 4, 2012 6:08:25 PM
> Subject: Re: [vdsm] RFC: New Storage API
> 
> Thanks for sharing this.  It's nice to have something a little more
> concrete to
> think about.  Just a few comments and questions inline to get some
> discussion
> flowing.
> 
> On Tue, Dec 04, 2012 at 04:52:40PM -0500, Saggi Mizrahi wrote:
> > I've been throwing a lot of bits out about the new storage API and
> > I think it's time to talk a bit.
> > I will purposefully try and keep implementation details away and
> > concentrate about how the API looks and how you use it.
> > 
> > First major change is in terminology, there is no long a storage
> > domain but a storage repository.
> > This change is done because so many things are already called
> > domain in the system and this will make things less confusing for
> > new-commers with a libvirt background.
> > 
> > One other changes is that repositories no longer have a UUID.
> > The UUID was only used in the pool members manifest and is no
> > longer needed.
> > 
> > 
> > connectStorageRepository(repoId, repoFormat,
> > connectionParameters={}):
> 
> We should probably add an options/flags parameter for extension of
> all new
> APIs.
Usually I agree but connectionParameters is already generic enough :)
> 
> > repoId - is a transient name that will be used to refer to the
> > connected domain, it is not persisted and doesn't have to be the
> > same across the cluster.
> > repoFormat - Similar to what used to be type (eg. localfs-1.0,
> > nfs-3.4, clvm-1.2).
> > connectionParameters - This is format specific and will used to
> > tell VDSM how to connect to the repo.
> > 
> > disconnectStorageRepository(self, repoId):
> 
> I assume 'self' is a mistake here.  Just want to clarify given all of
> the recent
> talk about instances vs. namespaces.
Yea, it's just pasted from my code
> 
> > In the new API there are only images, some images are mutable and
> > some are not.
> > mutable images are also called VirtualDisks
> > immutable images are also called Snapshots
> 
> By mutable you mean writable right?  Or does the word mutable imply
> more than
> that?
It's a semantic distinction due to implementation details, in general terms, 
yes.
> 
> > There are no explicit templates, you can create as many images as
> > you want from any snapshot.
> > 
> > There are 4 major image operations:
> > 
> > 
> > createVirtualDisk(targetRepoId, size, baseSnapshotId=None,
> >   userData={}, options={}):
> 
> Is userdata a 'StringMap'?
currently it's a json object. We could limit it to a string map and trust the 
client to parse types.
We can have it be a string\blob and trust the user to serialize the data.
It's pass-through object either way.
> 
> I will reopen the argument about an options dict vs a flags
> parameter.  I oppose
> the dict for expansion because I think it causes APIs to devolve into
> a mess
> where lots of arbitrary and not well thought out overrides are packed
> into the
> dict over time.  A flags argument (in json and python it can be an
> enum array)
> limits us to really switching flags on and off instead of passing
> arbitrary
> data.
We already have strategy that we know we want to have several options.
Other stuff that have been suggested is to be able to override the img format 
(qcow2\qed)

The way I envision it is having an class
opts = CommandOptions()
that you add
opts.addStringOption("key", "value")
opts.addIntOption("key", 3)
opt.addBoolOption("key", True)

I know you could just as well have
strategy_space_flag and strategy_performance_flag
and fail the operation if they both exist.
Since it is a matter of personal taste I think it should be decided by a vote.
> 
> > targetRepoId - ID of a connected repo where the disk will be
> > created
> > size - The size of the image you wish to create
> > baseSnapshotId - the ID of the snapshot you want the base the new
> > virtual disk on
> > userData - optional data that will be attached to the new VD, could
> > be anything that the user desires.
> > options - options to modify VDSMs default behavior
> > 
> > returns the id of the new VD
> > 
> > createSnapshot(targetRepoId, baseVirtualDiskId,
> >userData={}, options={}):
> > targetRepoId - The ID of a connected repo where the new sanpshot
> > will be created and the original image exists as well.
> > size - The size of the image you wish to create
> 
> Why is this needed?  Doesn't the size of a snapshot have to be equal
> to its
> base image?
oops, another copy\paste error, you can see this arg doesn't exist in the 
method signature.
My proof reading do need more work.
> 
> > baseVirtualDisk - the ID of a mutable image (Virtual Disk) you want
> > to snapshot
> 
> Can you snapshot a snapshot?  In that case, this parameter should be
> called
> baseImage.
You can't snapshot a sanpshot, it makes no sense as

Re: [Engine-devel] [vdsm] RFC: New Storage API

2012-12-04 Thread Adam Litke
Thanks for sharing this.  It's nice to have something a little more concrete to
think about.  Just a few comments and questions inline to get some discussion
flowing.

On Tue, Dec 04, 2012 at 04:52:40PM -0500, Saggi Mizrahi wrote:
> I've been throwing a lot of bits out about the new storage API and I think 
> it's time to talk a bit.
> I will purposefully try and keep implementation details away and concentrate 
> about how the API looks and how you use it.
> 
> First major change is in terminology, there is no long a storage domain but a 
> storage repository.
> This change is done because so many things are already called domain in the 
> system and this will make things less confusing for new-commers with a 
> libvirt background.
> 
> One other changes is that repositories no longer have a UUID.
> The UUID was only used in the pool members manifest and is no longer needed.
> 
> 
> connectStorageRepository(repoId, repoFormat, connectionParameters={}):

We should probably add an options/flags parameter for extension of all new
APIs.

> repoId - is a transient name that will be used to refer to the connected 
> domain, it is not persisted and doesn't have to be the same across the 
> cluster.
> repoFormat - Similar to what used to be type (eg. localfs-1.0, nfs-3.4, 
> clvm-1.2).
> connectionParameters - This is format specific and will used to tell VDSM how 
> to connect to the repo.
> 
> disconnectStorageRepository(self, repoId):

I assume 'self' is a mistake here.  Just want to clarify given all of the recent
talk about instances vs. namespaces.

> In the new API there are only images, some images are mutable and some are 
> not.
> mutable images are also called VirtualDisks
> immutable images are also called Snapshots

By mutable you mean writable right?  Or does the word mutable imply more than
that?

> There are no explicit templates, you can create as many images as you want 
> from any snapshot.
> 
> There are 4 major image operations:
> 
> 
> createVirtualDisk(targetRepoId, size, baseSnapshotId=None,
>   userData={}, options={}):

Is userdata a 'StringMap'?

I will reopen the argument about an options dict vs a flags parameter.  I oppose
the dict for expansion because I think it causes APIs to devolve into a mess
where lots of arbitrary and not well thought out overrides are packed into the
dict over time.  A flags argument (in json and python it can be an enum array)
limits us to really switching flags on and off instead of passing arbitrary
data.

> targetRepoId - ID of a connected repo where the disk will be created
> size - The size of the image you wish to create
> baseSnapshotId - the ID of the snapshot you want the base the new virtual 
> disk on
> userData - optional data that will be attached to the new VD, could be 
> anything that the user desires.
> options - options to modify VDSMs default behavior
> 
> returns the id of the new VD
> 
> createSnapshot(targetRepoId, baseVirtualDiskId,
>userData={}, options={}):
> targetRepoId - The ID of a connected repo where the new sanpshot will be 
> created and the original image exists as well.
> size - The size of the image you wish to create

Why is this needed?  Doesn't the size of a snapshot have to be equal to its
base image?

> baseVirtualDisk - the ID of a mutable image (Virtual Disk) you want to 
> snapshot

Can you snapshot a snapshot?  In that case, this parameter should be called
baseImage.

> userData - optional data that will be attached to the new Snapshot, could be 
> anything that the user desires.
> options - options to modify VDSMs default behavior
> 
> returns the id of the new Snapshot
> 
> copyImage(targetRepoId, imageId, baseImageId=None, userData={}, options={})
> targetRepoId - The ID of a connected repo where the new image will be created
> imageId - The image you wish to copy

Do we locate the sourceRepoId automatically based on the imageId?

> baseImageId - if specified, the new image will contain only the diff between 
> image and Id.
>   If None the new image will contain all the bits of image Id. 
> This can be used to copy partial parts of images for export.
> userData - optional data that will be attached to the new image, could be 
> anything that the user desires.
> options - options to modify VDSMs default behavior
> 
> return the Id of the new image. In case of copying an immutable image the ID 
> will be identical to the original image as they contain the same data. 
> However the user should not assume that and always use the value returned 
> from the method.
> 
> removeImage(repositoryId, imageId, options={}):
> repositoryId - The ID of a connected repo where the image to delete resides
> imageId - The id of the image you wish to delete.
> 
> 
> getImageStatus(repositoryId, imageId)
> repositoryId - The ID of a connected repo where the image to check resides
> imageId - The id of the image you wish to check.

What is in this return value?  Is it a single enum i

Re: [Engine-devel] VDSM tasks, the future

2012-12-04 Thread Saggi Mizrahi
As the only subsystem to use asynchronous tasks until now is the storage 
subsystem I suggest going over how
I suggest tackling task creation, task stop, task remove and task recovery.
Other subsystem can create similar mechanisms depending on their needs.

There is no way of avoiding it, different types of tasks need different ways of 
tracking\recovering from them.
network should always auto-recover because it can't get a "please fix" command 
if the network is down.
Storage on the other hand should never start operations on it's own because it 
might take up valuable resources from the host.
Tasks that need to be tracked on a single host, 2 hosts, or the entire cluster 
need to have their own APIs.
VM configuration never persist across reboots, networking sometimes persists 
and storage always persists.
This means that recovery procedures (from the managers point of view) need to 
be vastly different.
Add policy, resource allocation, and error flows you see that VDSM doesn't have 
nearly as much information to deal with the tasks.

- Original Message -
> From: "Adam Litke" 
> To: "Saggi Mizrahi" 
> Cc: "VDSM Project Development" , 
> "engine-devel" , "Ayal
> Baron" , "Barak Azulay" , "Shireesh 
> Anjal" 
> Sent: Tuesday, December 4, 2012 3:50:28 PM
> Subject: Re: VDSM tasks, the future
> 
> On Tue, Dec 04, 2012 at 10:35:01AM -0500, Saggi Mizrahi wrote:
> > Because I started hinting about how VDSM tasks are going to look
> > going forward
> > I thought it's better I'll just write everything in an email so we
> > can talk
> > about it in context.  This is not set in stone and I'm still
> > debating things
> > myself but it's very close to being done.
> 
> Don't debate them yourself, debate them here!  Even better, propose
> your idea in
> schema form to show how a command might work exactly.
I don't like throwing ideas in the air
> 
> > - Everything is asynchronous.  The nature of message based
> > communication is
> > that you can't have synchronous operations.  This is not really
> > debatable
> > because it's just how TCP\AMQP\ works.
> 
> Can you show how a traditionally synchronous command might work?
>  Let's take
> Host.getVmList as an example.
The same as it works today, it's all a matter of how you wrap the transport 
layer.
You will send a json-rpc request and wait for a response with the same id.

As for the bindings, there are a lot of way we can tackle that.
Always wait for the response and simulate synchronous behavior.
Make every method return an object to track the task.
task = host.getVmList()
if not task.wait(1):
task.cancel()
else:
res = task.result()

Have it both ways (it's auto generated anyway) and have
list = host.getVmList()
task = host.getVmList_async()

Have a high level and low level interfaces.
host = host()
host.connect("tcp://host:3233")
req = host.sendRequest("123213", "getVmList", [])
if not req.wait(1):
   

shost = SynchHost(host)
shost.getVmList() # Actually wraps a request object
ahost = AsyncHost(host)
task = getVmList() # Actually wraps a request object
> 
> > - Task IDs will be decided by the caller.  This is how json-rpc
> > works and also
> > makes sense because no the engine can track the task without
> > needing to have a
> > stage where we give it the task ID back.  IDs are reusable as long
> > as no one
> > else is using them at the time so they can be used for
> > synchronizing
> > operations between clients (making sure a command is only executed
> > once on a
> > specific host without locking).
> > 
> > - Tasks are transient If VDSM restarts it forgets all the task
> > information.
> > There are 2 ways to have persistent tasks: 1. The task creates an
> > object that
> > you can continue work on in VDSM.  The new storage does that by the
> > fact that
> > copyImage() returns one the target volume has been created but
> > before the data
> > has been fully copied.  From that moment on the stat of the copy
> > can be
> > queried from any host using getImageStatus() and the specific copy
> > operation
> > can be queried with getTaskStatus() on the host performing it.
> >  After VDSM
> > crashes, depending on policy, either VDSM will create a new task to
> > continue
> > the copy or someone else will send a command to continue the
> > operation and
> > that will be a new task.  2. VDSM tasks just start other operations
> > track-able
> > not through the task interface. For example Gluster.
> > gluster.startVolumeRebalance() will return once it has been
> > registered with
> > Gluster.  glster.getOperationStatuses() will return the state of
> > the operation
> > from any host.  Each call is a task in itself.
> 
> I worry about this approach because every command has a different
> semantic for
> checking progress.  For migration, we have to check VM status on the
> src and
> dest hosts.  For image copy we need to use a special status call on
> the dest
> image.  It would be nice if there was a unified method for checking
> on an
> operation.  M

[Engine-devel] RFC: New Storage API

2012-12-04 Thread Saggi Mizrahi
I've been throwing a lot of bits out about the new storage API and I think it's 
time to talk a bit.
I will purposefully try and keep implementation details away and concentrate 
about how the API looks and how you use it.

First major change is in terminology, there is no long a storage domain but a 
storage repository.
This change is done because so many things are already called domain in the 
system and this will make things less confusing for new-commers with a libvirt 
background.

One other changes is that repositories no longer have a UUID.
The UUID was only used in the pool members manifest and is no longer needed.


connectStorageRepository(repoId, repoFormat, connectionParameters={}):
repoId - is a transient name that will be used to refer to the connected 
domain, it is not persisted and doesn't have to be the same across the cluster.
repoFormat - Similar to what used to be type (eg. localfs-1.0, nfs-3.4, 
clvm-1.2).
connectionParameters - This is format specific and will used to tell VDSM how 
to connect to the repo.

disconnectStorageRepository(self, repoId):


In the new API there are only images, some images are mutable and some are not.
mutable images are also called VirtualDisks
immutable images are also called Snapshots

There are no explicit templates, you can create as many images as you want from 
any snapshot.

There are 4 major image operations:


createVirtualDisk(targetRepoId, size, baseSnapshotId=None,
  userData={}, options={}):

targetRepoId - ID of a connected repo where the disk will be created
size - The size of the image you wish to create
baseSnapshotId - the ID of the snapshot you want the base the new virtual disk 
on
userData - optional data that will be attached to the new VD, could be anything 
that the user desires.
options - options to modify VDSMs default behavior

returns the id of the new VD

createSnapshot(targetRepoId, baseVirtualDiskId,
   userData={}, options={}):
targetRepoId - The ID of a connected repo where the new sanpshot will be 
created and the original image exists as well.
size - The size of the image you wish to create
baseVirtualDisk - the ID of a mutable image (Virtual Disk) you want to snapshot
userData - optional data that will be attached to the new Snapshot, could be 
anything that the user desires.
options - options to modify VDSMs default behavior

returns the id of the new Snapshot

copyImage(targetRepoId, imageId, baseImageId=None, userData={}, options={})
targetRepoId - The ID of a connected repo where the new image will be created
imageId - The image you wish to copy
baseImageId - if specified, the new image will contain only the diff between 
image and Id.
  If None the new image will contain all the bits of image Id. This 
can be used to copy partial parts of images for export.
userData - optional data that will be attached to the new image, could be 
anything that the user desires.
options - options to modify VDSMs default behavior

return the Id of the new image. In case of copying an immutable image the ID 
will be identical to the original image as they contain the same data. However 
the user should not assume that and always use the value returned from the 
method.

removeImage(repositoryId, imageId, options={}):
repositoryId - The ID of a connected repo where the image to delete resides
imageId - The id of the image you wish to delete.



getImageStatus(repositoryId, imageId)
repositoryId - The ID of a connected repo where the image to check resides
imageId - The id of the image you wish to check.

All operations return once the operations has been committed to disk NOT when 
the operation actually completes.
This is done so that:
- operation come to a stable state as quickly as possible.
- In case where there is an SDM, only small portion of the operation actually 
needs to be performed on the SDM host.
- No matter how many times the operation fails and on how many hosts, you can 
always resume the operation and choose when to do it.
- You can stop an operation at any time and remove the resulting object making 
a distinction between "stop because the host is overloaded" to "I don't want 
that image"

This means that after calling any operation that creates a new image the user 
must then call getImageStatus() to check what is the status of the image.
The status of the image can be either optimized, degraded, or broken.
"Optimized" means that the image is available and you can run VMs of it.
"Degraded" means that the image is available and will run VMs but it might be a 
better way VDSM can represent the underlying data. 
"Broken" means that the image can't be used at the moment, probably because not 
all the data has been set up on the volume.

Apart from that VDSM will also return the last persisted status information 
which will conatin
hostID - the last host to try and optimize of fix the image
stage - X/Y (eg. 1/10) the last persisted stage of the fix.
percent_complete - -1 or 0

Re: [Engine-devel] engine-config utility

2012-12-04 Thread Alon Bar-Lev


- Original Message -
> From: "Itamar Heim" 
> To: "Shireesh Anjal" 
> Cc: "Alon Bar-Lev" , "engine-devel" 
> 
> Sent: Tuesday, December 4, 2012 10:50:09 PM
> Subject: Re: [Engine-devel] engine-config utility
> 
> On 12/04/2012 04:23 PM, Shireesh Anjal wrote:
> >
> >
> > - Original Message -
> >> From: "Alon Bar-Lev" 
> >> To: "engine-devel" 
> >> Sent: Tuesday, December 4, 2012 4:41:52 PM
> >> Subject: [Engine-devel] engine-config utility
> >>
> >> Hello,
> >>
> >> I had interesting discussion with Eli regarding the engine-config
> >> utility and recent commit.
> >>
> >> Is the following claim correct?
> >>
> >> """Every change of a variable by engine-config cannot break
> >> engine"""
> >>
> >> This issue came up because we used to export the CABaseDirectory
> >> parameter to engine-config. Changing it without attending to copy
> >> pki files may lead to failure. A change was already committed to
> >> remove the ability to update this parameter.
> >>
> >> Most applications I know, do allow to change variable that in
> >> certain
> >> cases may break the application. The assumption that I am usually
> >> familiar with is that the target audience of these kind of tools
> >> are
> >> system admins or integrators, which "should know what they are
> >> doing", so it is OK to expose parameters that may lead to
> >> application breakage, as the ability to customize the application
> >> is
> >> important.
> >
> > I agree with Alon. I don't think it is feasible to prevent the user
> > from every possible update of configuration that can break the
> > system.
> 
> I agree in general, and i don't think we should extend the config
> utility to test such things.
> but some basic sanity should be done (config key is legit, config
> value
> is validated for type safety/regex/enum, etc.

Right.
if Value1+Value2 = crash is not something that should be checked.
if Value1=Path while Path does not exist or contains wrong files is not 
something we should check.

> 
> >
> >>
> >> I am carious what other think in this regard.
> >>
> >> Regards,
> >> Alon Bar-Lev.
> >>
> >> ___
> >> Engine-devel mailing list
> >> Engine-devel@ovirt.org
> >> http://lists.ovirt.org/mailman/listinfo/engine-devel
> >>
> > ___
> > Engine-devel mailing list
> > Engine-devel@ovirt.org
> > http://lists.ovirt.org/mailman/listinfo/engine-devel
> >
> 
> 
> 
___
Engine-devel mailing list
Engine-devel@ovirt.org
http://lists.ovirt.org/mailman/listinfo/engine-devel


Re: [Engine-devel] VDSM tasks, the future

2012-12-04 Thread Adam Litke
On Tue, Dec 04, 2012 at 10:35:01AM -0500, Saggi Mizrahi wrote:
> Because I started hinting about how VDSM tasks are going to look going forward
> I thought it's better I'll just write everything in an email so we can talk
> about it in context.  This is not set in stone and I'm still debating things
> myself but it's very close to being done.

Don't debate them yourself, debate them here!  Even better, propose your idea in
schema form to show how a command might work exactly.

> - Everything is asynchronous.  The nature of message based communication is
> that you can't have synchronous operations.  This is not really debatable
> because it's just how TCP\AMQP\ works.

Can you show how a traditionally synchronous command might work?  Let's take
Host.getVmList as an example.

> - Task IDs will be decided by the caller.  This is how json-rpc works and also
> makes sense because no the engine can track the task without needing to have a
> stage where we give it the task ID back.  IDs are reusable as long as no one
> else is using them at the time so they can be used for synchronizing
> operations between clients (making sure a command is only executed once on a
> specific host without locking).
> 
> - Tasks are transient If VDSM restarts it forgets all the task information.
> There are 2 ways to have persistent tasks: 1. The task creates an object that
> you can continue work on in VDSM.  The new storage does that by the fact that
> copyImage() returns one the target volume has been created but before the data
> has been fully copied.  From that moment on the stat of the copy can be
> queried from any host using getImageStatus() and the specific copy operation
> can be queried with getTaskStatus() on the host performing it.  After VDSM
> crashes, depending on policy, either VDSM will create a new task to continue
> the copy or someone else will send a command to continue the operation and
> that will be a new task.  2. VDSM tasks just start other operations track-able
> not through the task interface. For example Gluster.
> gluster.startVolumeRebalance() will return once it has been registered with
> Gluster.  glster.getOperationStatuses() will return the state of the operation
> from any host.  Each call is a task in itself.

I worry about this approach because every command has a different semantic for
checking progress.  For migration, we have to check VM status on the src and
dest hosts.  For image copy we need to use a special status call on the dest
image.  It would be nice if there was a unified method for checking on an
operation.  Maybe that can be completion events.

Client:   vdsm:
---   -

Image.copy(...)  -->
 <--  Operation Started
Wait for event   ...
 <--  Event: Operation  done 

For an early error:

Client:   vdsm:
---   -

Image.copy(...)  -->
 <--  Error: 


> - No task tags.  They are silly and the caller can mangle whatever in the task
> ID if he really wants to tag tasks.

Yes.  Agreed.

> - No explicit recovery stage.  VDSM will be crash-only, there should be
> efforts to make everything crash-safe.  If that is problematic, in case of
> networking, VDSM will recover on start without having a task for it.

How does this work in practice for something like creating a new image from a
template?

> - No clean Task: Tasks can be started by any number of hosts this means that
> there is no way to own all tasks.  There could be cases where VDSM starts
> tasks on it's own and thus they have no owner at all.  The caller needs to
> continually track the state of VDSM. We will have brodcasted events to
> mitigate polling.

If a disconnected client might have missed a completion event, it will need to
check state.  This means each async operation that changes state must document a
proceedure for checking progress of a potentially ongoing operation.  For
Image.copy, that process would be to lookup the new image and check its state.

> - No revert Impossible to implement safely.

How do the engine folks feel about this?  I am ok with it :)

> - No SPM\HSM tasks SPM\SDM is no longer necessary for all domain types (only
> for type).  What used to be SPM tasks, or tasks that persist and can be
> restarted on other hosts is talked about in previous bullet points.
> 
A nice simplification.


-- 
Adam Litke 
IBM Linux Technology Center

___
Engine-devel mailing list
Engine-devel@ovirt.org
http://lists.ovirt.org/mailman/listinfo/engine-devel


Re: [Engine-devel] engine-config utility

2012-12-04 Thread Itamar Heim

On 12/04/2012 04:23 PM, Shireesh Anjal wrote:



- Original Message -

From: "Alon Bar-Lev" 
To: "engine-devel" 
Sent: Tuesday, December 4, 2012 4:41:52 PM
Subject: [Engine-devel] engine-config utility

Hello,

I had interesting discussion with Eli regarding the engine-config
utility and recent commit.

Is the following claim correct?

"""Every change of a variable by engine-config cannot break engine"""

This issue came up because we used to export the CABaseDirectory
parameter to engine-config. Changing it without attending to copy
pki files may lead to failure. A change was already committed to
remove the ability to update this parameter.

Most applications I know, do allow to change variable that in certain
cases may break the application. The assumption that I am usually
familiar with is that the target audience of these kind of tools are
system admins or integrators, which "should know what they are
doing", so it is OK to expose parameters that may lead to
application breakage, as the ability to customize the application is
important.


I agree with Alon. I don't think it is feasible to prevent the user from every 
possible update of configuration that can break the system.


I agree in general, and i don't think we should extend the config 
utility to test such things.
but some basic sanity should be done (config key is legit, config value 
is validated for type safety/regex/enum, etc.






I am carious what other think in this regard.

Regards,
Alon Bar-Lev.

___
Engine-devel mailing list
Engine-devel@ovirt.org
http://lists.ovirt.org/mailman/listinfo/engine-devel


___
Engine-devel mailing list
Engine-devel@ovirt.org
http://lists.ovirt.org/mailman/listinfo/engine-devel




___
Engine-devel mailing list
Engine-devel@ovirt.org
http://lists.ovirt.org/mailman/listinfo/engine-devel


Re: [Engine-devel] Revisiting Java7

2012-12-04 Thread Asaf Shakarchi
- Original Message -
> On 12/04/2012 03:53 PM, Alon Bar-Lev wrote:
> >
> >
> > - Original Message -
> >> From: "Itamar Heim" 
> >> To: "Allon Mureinik" 
> >> Cc: "engine-devel" 
> >> Sent: Tuesday, December 4, 2012 1:13:49 PM
> >> Subject: Re: [Engine-devel] Revisiting Java7
> >>
> >> On 12/04/2012 11:20 AM, Allon Mureinik wrote:
> >>>
> >>>
> >>> - Original Message -
>  From: "Itamar Heim" 
>  To: "Allon Mureinik" 
>  Cc: "engine-devel" 
>  Sent: Tuesday, December 4, 2012 1:40:10 AM
>  Subject: Re: [Engine-devel] Revisiting Java7
> 
>  On 12/03/2012 04:03 PM, Allon Mureinik wrote:
> > Hi guys,
> >
> > Earlier today, Java6 compatibility was broken
> > (http://gerrit.ovirt.org/#/c/9430/).
> > This was picked up on pretty quickly, and easily fixed
> > (http://gerrit.ovirt.org/#/c/9666/).
> >
> > However, I think this is a good opportunity to revisit our
> > policy
> > towards Java 7.
> > Currently, we have an "odd" setup, where we recommend running a
> > compiling /with/ Java 7 [1] but comply to Java 6 language level
> > [2] and JDK [3].
> > Inevitably, mistakes like the that happened today will happen.
> >
> > I know we're holding back due to GWT issues, but looking
> > forward
> > to
> > oVirt 4.0, is Java 7 on our roadmap?
> 
>  what was the GWT issue?
> >>> GWT 2.3 that we use doesn't support java 7 syntax.
> >>> The latest version, 2.5, doesn't either.
> >>> I'm not sure, however, this is a good reason to enforce the java
> >>> 6
> >>> limitation on the entire project (including backed, rest, etc.)
> >>
> >> i agree. you can limit it to the parts used by GWT for now.
> >
> > The major advantage to move to java7 is the try-with-resources
> > statement.
> >
> > If we can do this for 3.2 it will be great. From what I understand
> > from Allon, the only noisy change is to move the common and compat
> > out of the backend directory into its own top-level directory, so
> > we can apply a different maven policy on these two easily.
> >
> > This way the common/compat and frontend will be built using 1.6 and
> > backend will be built using 1.7.
> 
> and searchbackend.
> asaf, is there no easier way to enable java 7 compilation/enforcement
> except for these 3?

In the long run consider having a project layer such as :
Frontend / Backend / Shared [common/compat/searchbackend] , this will allow us 
to use plugins inheritance cleanly [regarding versions],

Of course inheritance is nicer but it requires a lot of movements (maybe in v4?)


For now its easy - we can just use the maven-compiler to define source/target 
of [v1.7] for the root project, while defining explicitly different versions 
per frontend/shared module [v1.6]
Checkstyles should be upgraded as well.
> 
___
Engine-devel mailing list
Engine-devel@ovirt.org
http://lists.ovirt.org/mailman/listinfo/engine-devel


Re: [Engine-devel] Revisiting Java7

2012-12-04 Thread Itamar Heim

On 12/04/2012 03:53 PM, Alon Bar-Lev wrote:



- Original Message -

From: "Itamar Heim" 
To: "Allon Mureinik" 
Cc: "engine-devel" 
Sent: Tuesday, December 4, 2012 1:13:49 PM
Subject: Re: [Engine-devel] Revisiting Java7

On 12/04/2012 11:20 AM, Allon Mureinik wrote:



- Original Message -

From: "Itamar Heim" 
To: "Allon Mureinik" 
Cc: "engine-devel" 
Sent: Tuesday, December 4, 2012 1:40:10 AM
Subject: Re: [Engine-devel] Revisiting Java7

On 12/03/2012 04:03 PM, Allon Mureinik wrote:

Hi guys,

Earlier today, Java6 compatibility was broken
(http://gerrit.ovirt.org/#/c/9430/).
This was picked up on pretty quickly, and easily fixed
(http://gerrit.ovirt.org/#/c/9666/).

However, I think this is a good opportunity to revisit our policy
towards Java 7.
Currently, we have an "odd" setup, where we recommend running a
compiling /with/ Java 7 [1] but comply to Java 6 language level
[2] and JDK [3].
Inevitably, mistakes like the that happened today will happen.

I know we're holding back due to GWT issues, but looking forward
to
oVirt 4.0, is Java 7 on our roadmap?


what was the GWT issue?

GWT 2.3 that we use doesn't support java 7 syntax.
The latest version, 2.5, doesn't either.
I'm not sure, however, this is a good reason to enforce the java 6
limitation on the entire project (including backed, rest, etc.)


i agree. you can limit it to the parts used by GWT for now.


The major advantage to move to java7 is the try-with-resources statement.

If we can do this for 3.2 it will be great. From what I understand from Allon, 
the only noisy change is to move the common and compat out of the backend 
directory into its own top-level directory, so we can apply a different maven 
policy on these two easily.

This way the common/compat and frontend will be built using 1.6 and backend 
will be built using 1.7.


and searchbackend.
asaf, is there no easier way to enable java 7 compilation/enforcement 
except for these 3?

___
Engine-devel mailing list
Engine-devel@ovirt.org
http://lists.ovirt.org/mailman/listinfo/engine-devel


[Engine-devel] VDSM tasks, the future

2012-12-04 Thread Saggi Mizrahi
Because I started hinting about how VDSM tasks are going to look going forward 
I thought it's better I'll just write everything in an email so we can talk 
about it in context.
This is not set in stone and I'm still debating things myself but it's very 
close to being done.

- Everything is asynchronous.
  The nature of message based communication is that you can't have synchronous 
operations.
  This is not really debatable because it's just how TCP\AMQP\ works.

- Task IDs will be decided by the caller.
  This is how json-rpc works and also makes sense because no the engine can 
track the task without needing to have a stage where we give it the task ID 
back.
  IDs are reusable as long as no one else is using them at the time so they can 
be used for synchronizing operations between clients (making sure a command is
  only executed once on a specific host without locking).

- Tasks are transient
  If VDSM restarts it forgets all the task information.
  There are 2 ways to have persistent tasks:
  1. The task creates an object that you can continue work on in VDSM.
 The new storage does that by the fact that copyImage() returns one the 
target volume has been created but before the data has been fully copied.
 From that moment on the stat of the copy can be queried from any host 
using getImageStatus() and the specific copy operation can be queried with 
getTaskStatus() on the host performing it.
 After VDSM crashes, depending on policy, either VDSM will create a new 
task to continue the copy or someone else will send a command to continue the 
operation and that will be a new task.
  2. VDSM tasks just start other operations track-able not through the task 
interface. For example Gluster.
 gluster.startVolumeRebalance() will return once it has been registered 
with Gluster.
 glster.getOperationStatuses() will return the state of the operation from 
any host.
 Each call is a task in itself.
  
- No task tags.
  They are silly and the caller can mangle whatever in the task ID if he really 
wants to tag tasks.

- No explicit recovery stage.
  VDSM will be crash-only, there should be efforts to make everything 
crash-safe.
  If that is problematic, in case of networking, VDSM will recover on start 
without having a task for it.

- No clean Task:
  Tasks can be started by any number of hosts this means that there is no way 
to own all tasks.
  There could be cases where VDSM starts tasks on it's own and thus they have 
no owner at all.
  The caller needs to continually track the state of VDSM. We will have 
brodcasted events to mitigate polling.

- No revert
  Impossible to implement safely.

- No SPM\HSM tasks
  SPM\SDM is no longer necessary for all domain types (only for type).
  What used to be SPM tasks, or tasks that persist and can be restarted on 
other hosts is talked about in previous bullet points.
___
Engine-devel mailing list
Engine-devel@ovirt.org
http://lists.ovirt.org/mailman/listinfo/engine-devel


Re: [Engine-devel] Task cancelation feature

2012-12-04 Thread Saggi Mizrahi


- Original Message -
> From: "Dan Kenigsberg" 
> To: "Saggi Mizrahi" 
> Cc: "Michael Kublin" , "engine-devel" 
> 
> Sent: Tuesday, December 4, 2012 4:38:13 AM
> Subject: Re: [Engine-devel] Task cancelation feature
> 
> On Mon, Dec 03, 2012 at 12:15:07PM -0500, Saggi Mizrahi wrote:
> > VDSM tasks are changing to something completely different.
> > It's still under discussion but the general direction is that:
> > - TaskIDs will be decided by the caller.
> > - VDSM can start tasks on it's own
> > - There will be no distinction between async tasks and sync tasks.
> > Everything is always async.
> > - There will be no cleanTask() when tasks are done they return
> > result to the caller and disappear immediately.
> 
> I'm not sure I understand the motivation for the latter change. I
> kinda
> like the unix process semantics, were the return code of a process is
> kept with its id after the process ends, until the process parent
> calls
> wait(2). Otherwise, how can the caller tell why its task has failed?
> 
> For example, I'd like to see vmCreate using async tasks like that.
> vmCreate returns immediately, and a vdsm task is tracking the vm
> creation. If something bad happens, the information about the failure
> can be polled by the Engine that created the vm (or a new Engine
> instance, after an Engine crash).
You can have clean task only when there is a clear owner for a task.
This is not the case because it has been decided VDSM can be used by more then 
one client.
I know that people kind of like how convenient it is but it's problematic from 
the same reason that the unix method is problematic.
- If you die you can no longer track the task.
- If the kid dies and restarts itself you no longer can track the task.
- If the task is started by another process you can no longer track it.
- If you happen to die and not restart all your tasks stay there until someone 
clears them and it's impossible to know for sure when that is safe.

We already have issues because this semantic, doesn't hold up for a lot of the 
error cases. It doesn't hold up at all when there are more then one task 
instigators.
Because in TCP and and AMQP you are guaranteed to get the result message as 
long as you are connected to the bus you will always know the result unless you 
or VDSM crashed.
When you restart, you have to write code to sync up in case something happen 
while you where down anyway.
When VDSM restart\errors you need to write code to check how the task ended 
anyway because even "failed" tasks can finish successfully.
Also, in the new storage, VDSM restarts usually don't fail big operations and 
the can continue even on another host.
(We send the commit write and the storage fails just after the request was sent 
but we didn't get the write ack. The task is committed but it failed on 
timeout.)
> 
> Similarily, we may need to make setupNetwork asynchronous, since we
> depend on dhclient, which may take a lot of time to finish.
As I said, everything *is* asynchronous. It's impossible to have synchronous 
calls because you can't have synchronous calls on AMQP\TCP.
Because the engine is responsible for generating the ID there is no need for 
the silly 2 step approach we have now.
> 
> Have these future use cases been debated?
Yes, If you convince everyone to move away from message based communication and 
multiplexing requests on the same TCP connection (or go back to only HTTP) you 
can get synchronous tasks back.
If you can find a way to make "clean task" make sense and work then you can 
have that back too.
I think that after thinking about it you'll realize that considering all the 
possible error flows, sync-up on failure is the most robust and flexible way to 
go.
Further more, I doubt you can find a system where you never need to sync-up on 
the state of VDSM so might as well make that the only flow.
> 
> Dan.
> > 
> > Also, some stuff you consider tasks will no longer be tasks any
> > more.
> > For instance, copying and image will finish successfully once VDSM
> > registers the operation for with the storage subsystem and creates
> > the image handle.
> > After that the status of the copy is bound to the status of the new
> > image and is tracked that way.
> > This means that the thing you track when you do copyImage() is
> > actually the creation of the image handle and the metadata to make
> > it usable.
> > After that is done any host can query the state of the new image by
> > using the image ID and not the task Id which was deprecated.
> > This will be true for all storage operations.
> > 
> > 
> > - Original Message -
> > > From: "Michael Kublin" 
> > > To: "engine-devel" 
> > > Sent: Monday, December 3, 2012 4:19:48 AM
> > > Subject: [Engine-devel] Task cancelation feature
> > > 
> > > Hi, I created a wiki page with design of task cancellation
> > > feature.
> > > The url is : http://www.ovirt.org/Features/TaskManagerCancelTask
> > > I can not call these design, I have not any requirements 

Re: [Engine-devel] Task cancelation feature

2012-12-04 Thread Shireesh Anjal


- Original Message -
> From: "Dan Kenigsberg" 
> To: "Saggi Mizrahi" 
> Cc: "engine-devel" 
> Sent: Tuesday, December 4, 2012 3:08:13 PM
> Subject: Re: [Engine-devel] Task cancelation feature
> 
> On Mon, Dec 03, 2012 at 12:15:07PM -0500, Saggi Mizrahi wrote:
> > VDSM tasks are changing to something completely different.
> > It's still under discussion but the general direction is that:
> > - TaskIDs will be decided by the caller.
> > - VDSM can start tasks on it's own
> > - There will be no distinction between async tasks and sync tasks.
> > Everything is always async.
> > - There will be no cleanTask() when tasks are done they return
> > result to the caller and disappear immediately.
> 
> I'm not sure I understand the motivation for the latter change. I
> kinda
> like the unix process semantics, were the return code of a process is
> kept with its id after the process ends, until the process parent
> calls
> wait(2). Otherwise, how can the caller tell why its task has failed?
> 
> For example, I'd like to see vmCreate using async tasks like that.
> vmCreate returns immediately, and a vdsm task is tracking the vm
> creation. If something bad happens, the information about the failure
> can be polled by the Engine that created the vm (or a new Engine
> instance, after an Engine crash).
> 
> Similarily, we may need to make setupNetwork asynchronous, since we
> depend on dhclient, which may take a lot of time to finish.
> 
> Have these future use cases been debated?

There are also a few requirements from gluster perspective that may need 
enhancements in vdsm as well as engine.
- Tasks are created and managed by glusterfs and not by vdsm.
- Gluster tasks are not bound to a particular host, but are cluster-wide, and 
their status can be checked from any of the hosts of the cluster.
- Concept of SPM does not come into picture in case of gluster clusters/hosts
- Apart from starting, aborting and checking status, some of the gluster tasks 
support additional actions like pause, resume and commit.

Based on some of the telephonic conversations with maintainers of engine and 
vdsm, we were planning to enhance the existing task management as follows:
- Enhance the getAllTasks verb in vdsm to accept one or more tags for filtering 
tasks. (http://gerrit.ovirt.org/7579)
- Currently all tasks created in vdsm through requests from engine have the 
'spm' tag as they are SPM tasks.
- Introduce a new field, say task_target in engine, which indicates what kind 
of a task it is. Possible values:
   - SPM (SPM tasks)
   - CLUSTER (cluster-wide tasks e.g. gluster tasks)
   - HOST (tasks specific to a particular host e.g. format a disk on a host)
- Enhance the async task manager in engine, to fetch details of all types of 
tasks, by sending appropriate tags to the new getAllTasks verb. (At present it 
fetches only SPM tasks)
- Once all the tasks are fetched, rest of the processing (updating status) 
remains same as before

I would like to know whether we should stop working on above approach in case 
the new design is coming up immediately. If yes, we should make sure that the 
new design is capable of handling the gluster tasks as well.

In case it is too far in the future,

1) We will start working with above approach. Any comments/suggestions/concerns 
are welcome.
2) Instead of focusing on just 'cancellation', we should try and come up with a 
more generic approach which can help in easily supporting more actions like 
pause, resume, commit, etc. (I haven't yet gone through the feature page, and 
please pardon me if this is already taken care)


> 
> Dan.
> > 
> > Also, some stuff you consider tasks will no longer be tasks any
> > more.
> > For instance, copying and image will finish successfully once VDSM
> > registers the operation for with the storage subsystem and creates
> > the image handle.
> > After that the status of the copy is bound to the status of the new
> > image and is tracked that way.
> > This means that the thing you track when you do copyImage() is
> > actually the creation of the image handle and the metadata to make
> > it usable.
> > After that is done any host can query the state of the new image by
> > using the image ID and not the task Id which was deprecated.
> > This will be true for all storage operations.
> > 
> > 
> > - Original Message -
> > > From: "Michael Kublin" 
> > > To: "engine-devel" 
> > > Sent: Monday, December 3, 2012 4:19:48 AM
> > > Subject: [Engine-devel] Task cancelation feature
> > > 
> > > Hi, I created a wiki page with design of task cancellation
> > > feature.
> > > The url is : http://www.ovirt.org/Features/TaskManagerCancelTask
> > > I can not call these design, I have not any requirements , except
> > > a
> > > name of the feature,
> > > so my wiki doesn't contains anything except open questions.
> > > Also, I think that it is impossible to make a good feature based
> > > on
> > > very problematic infrastructure,
> > > I think before we should fix al

Re: [Engine-devel] engine-config utility

2012-12-04 Thread Shireesh Anjal


- Original Message -
> From: "Alon Bar-Lev" 
> To: "engine-devel" 
> Sent: Tuesday, December 4, 2012 4:41:52 PM
> Subject: [Engine-devel] engine-config utility
> 
> Hello,
> 
> I had interesting discussion with Eli regarding the engine-config
> utility and recent commit.
> 
> Is the following claim correct?
> 
> """Every change of a variable by engine-config cannot break engine"""
> 
> This issue came up because we used to export the CABaseDirectory
> parameter to engine-config. Changing it without attending to copy
> pki files may lead to failure. A change was already committed to
> remove the ability to update this parameter.
> 
> Most applications I know, do allow to change variable that in certain
> cases may break the application. The assumption that I am usually
> familiar with is that the target audience of these kind of tools are
> system admins or integrators, which "should know what they are
> doing", so it is OK to expose parameters that may lead to
> application breakage, as the ability to customize the application is
> important.

I agree with Alon. I don't think it is feasible to prevent the user from every 
possible update of configuration that can break the system.

> 
> I am carious what other think in this regard.
> 
> Regards,
> Alon Bar-Lev.
> 
> ___
> Engine-devel mailing list
> Engine-devel@ovirt.org
> http://lists.ovirt.org/mailman/listinfo/engine-devel
> 
___
Engine-devel mailing list
Engine-devel@ovirt.org
http://lists.ovirt.org/mailman/listinfo/engine-devel


Re: [Engine-devel] Revisiting Java7

2012-12-04 Thread Alon Bar-Lev


- Original Message -
> From: "Itamar Heim" 
> To: "Allon Mureinik" 
> Cc: "engine-devel" 
> Sent: Tuesday, December 4, 2012 1:13:49 PM
> Subject: Re: [Engine-devel] Revisiting Java7
> 
> On 12/04/2012 11:20 AM, Allon Mureinik wrote:
> >
> >
> > - Original Message -
> >> From: "Itamar Heim" 
> >> To: "Allon Mureinik" 
> >> Cc: "engine-devel" 
> >> Sent: Tuesday, December 4, 2012 1:40:10 AM
> >> Subject: Re: [Engine-devel] Revisiting Java7
> >>
> >> On 12/03/2012 04:03 PM, Allon Mureinik wrote:
> >>> Hi guys,
> >>>
> >>> Earlier today, Java6 compatibility was broken
> >>> (http://gerrit.ovirt.org/#/c/9430/).
> >>> This was picked up on pretty quickly, and easily fixed
> >>> (http://gerrit.ovirt.org/#/c/9666/).
> >>>
> >>> However, I think this is a good opportunity to revisit our policy
> >>> towards Java 7.
> >>> Currently, we have an "odd" setup, where we recommend running a
> >>> compiling /with/ Java 7 [1] but comply to Java 6 language level
> >>> [2] and JDK [3].
> >>> Inevitably, mistakes like the that happened today will happen.
> >>>
> >>> I know we're holding back due to GWT issues, but looking forward
> >>> to
> >>> oVirt 4.0, is Java 7 on our roadmap?
> >>
> >> what was the GWT issue?
> > GWT 2.3 that we use doesn't support java 7 syntax.
> > The latest version, 2.5, doesn't either.
> > I'm not sure, however, this is a good reason to enforce the java 6
> > limitation on the entire project (including backed, rest, etc.)
> 
> i agree. you can limit it to the parts used by GWT for now.

The major advantage to move to java7 is the try-with-resources statement.

If we can do this for 3.2 it will be great. From what I understand from Allon, 
the only noisy change is to move the common and compat out of the backend 
directory into its own top-level directory, so we can apply a different maven 
policy on these two easily.

This way the common/compat and frontend will be built using 1.6 and backend 
will be built using 1.7.

Alon
___
Engine-devel mailing list
Engine-devel@ovirt.org
http://lists.ovirt.org/mailman/listinfo/engine-devel


Re: [Engine-devel] Revisiting Java7

2012-12-04 Thread Itamar Heim

On 12/04/2012 11:20 AM, Allon Mureinik wrote:



- Original Message -

From: "Itamar Heim" 
To: "Allon Mureinik" 
Cc: "engine-devel" 
Sent: Tuesday, December 4, 2012 1:40:10 AM
Subject: Re: [Engine-devel] Revisiting Java7

On 12/03/2012 04:03 PM, Allon Mureinik wrote:

Hi guys,

Earlier today, Java6 compatibility was broken
(http://gerrit.ovirt.org/#/c/9430/).
This was picked up on pretty quickly, and easily fixed
(http://gerrit.ovirt.org/#/c/9666/).

However, I think this is a good opportunity to revisit our policy
towards Java 7.
Currently, we have an "odd" setup, where we recommend running a
compiling /with/ Java 7 [1] but comply to Java 6 language level
[2] and JDK [3].
Inevitably, mistakes like the that happened today will happen.

I know we're holding back due to GWT issues, but looking forward to
oVirt 4.0, is Java 7 on our roadmap?


what was the GWT issue?

GWT 2.3 that we use doesn't support java 7 syntax.
The latest version, 2.5, doesn't either.
I'm not sure, however, this is a good reason to enforce the java 6 limitation 
on the entire project (including backed, rest, etc.)


i agree. you can limit it to the parts used by GWT for now.
___
Engine-devel mailing list
Engine-devel@ovirt.org
http://lists.ovirt.org/mailman/listinfo/engine-devel


[Engine-devel] engine-config utility

2012-12-04 Thread Alon Bar-Lev
Hello,

I had interesting discussion with Eli regarding the engine-config utility and 
recent commit.

Is the following claim correct?

"""Every change of a variable by engine-config cannot break engine"""

This issue came up because we used to export the CABaseDirectory parameter to 
engine-config. Changing it without attending to copy pki files may lead to 
failure. A change was already committed to remove the ability to update this 
parameter.

Most applications I know, do allow to change variable that in certain cases may 
break the application. The assumption that I am usually familiar with is that 
the target audience of these kind of tools are system admins or integrators, 
which "should know what they are doing", so it is OK to expose parameters that 
may lead to application breakage, as the ability to customize the application 
is important.

I am carious what other think in this regard.

Regards,
Alon Bar-Lev.

___
Engine-devel mailing list
Engine-devel@ovirt.org
http://lists.ovirt.org/mailman/listinfo/engine-devel


Re: [Engine-devel] Task cancelation feature

2012-12-04 Thread Dan Kenigsberg
On Mon, Dec 03, 2012 at 12:15:07PM -0500, Saggi Mizrahi wrote:
> VDSM tasks are changing to something completely different.
> It's still under discussion but the general direction is that:
> - TaskIDs will be decided by the caller.
> - VDSM can start tasks on it's own
> - There will be no distinction between async tasks and sync tasks. Everything 
> is always async.
> - There will be no cleanTask() when tasks are done they return result to the 
> caller and disappear immediately.

I'm not sure I understand the motivation for the latter change. I kinda
like the unix process semantics, were the return code of a process is
kept with its id after the process ends, until the process parent calls
wait(2). Otherwise, how can the caller tell why its task has failed?

For example, I'd like to see vmCreate using async tasks like that.
vmCreate returns immediately, and a vdsm task is tracking the vm
creation. If something bad happens, the information about the failure
can be polled by the Engine that created the vm (or a new Engine
instance, after an Engine crash).

Similarily, we may need to make setupNetwork asynchronous, since we
depend on dhclient, which may take a lot of time to finish.

Have these future use cases been debated?

Dan.
> 
> Also, some stuff you consider tasks will no longer be tasks any more.
> For instance, copying and image will finish successfully once VDSM registers 
> the operation for with the storage subsystem and creates the image handle.
> After that the status of the copy is bound to the status of the new image and 
> is tracked that way.
> This means that the thing you track when you do copyImage() is actually the 
> creation of the image handle and the metadata to make it usable.
> After that is done any host can query the state of the new image by using the 
> image ID and not the task Id which was deprecated.
> This will be true for all storage operations.
> 
> 
> - Original Message -
> > From: "Michael Kublin" 
> > To: "engine-devel" 
> > Sent: Monday, December 3, 2012 4:19:48 AM
> > Subject: [Engine-devel] Task cancelation feature
> > 
> > Hi, I created a wiki page with design of task cancellation feature.
> > The url is : http://www.ovirt.org/Features/TaskManagerCancelTask
> > I can not call these design, I have not any requirements , except a
> > name of the feature,
> > so my wiki doesn't contains anything except open questions.
> > Also, I think that it is impossible to make a good feature based on
> > very problematic infrastructure,
> > I think before we should fix all our infrastructure problems, and
> > after that to add any cancellation task
> > feature will be a meter of couple hours of work
> > 
> > Regards Michael
> > ___
> > Engine-devel mailing list
> > Engine-devel@ovirt.org
> > http://lists.ovirt.org/mailman/listinfo/engine-devel
> > 
> ___
> Engine-devel mailing list
> Engine-devel@ovirt.org
> http://lists.ovirt.org/mailman/listinfo/engine-devel
___
Engine-devel mailing list
Engine-devel@ovirt.org
http://lists.ovirt.org/mailman/listinfo/engine-devel


Re: [Engine-devel] Revisiting Java7

2012-12-04 Thread Allon Mureinik


- Original Message -
> From: "Laszlo Hornyak" 
> To: "Allon Mureinik" 
> Cc: "engine-devel" 
> Sent: Monday, December 3, 2012 4:36:42 PM
> Subject: Re: [Engine-devel] Revisiting Java7
> 
> Hi,
> 
> 
> - Original Message -
> > From: "Allon Mureinik" 
> > To: "engine-devel" 
> > Sent: Monday, December 3, 2012 3:03:21 PM
> > Subject: [Engine-devel] Revisiting Java7
> > 
> > Hi guys,
> > 
> > Earlier today, Java6 compatibility was broken
> > (http://gerrit.ovirt.org/#/c/9430/).
> > This was picked up on pretty quickly, and easily fixed
> > (http://gerrit.ovirt.org/#/c/9666/).
> > 
> > However, I think this is a good opportunity to revisit our policy
> > towards Java 7.
> > Currently, we have an "odd" setup, where we recommend running a
> > compiling /with/ Java 7 [1] but comply to Java 6 language level [2]
> > and JDK [3].
> > Inevitably, mistakes like the that happened today will happen.
> > 
> > I know we're holding back due to GWT issues, but looking forward to
> > oVirt 4.0, is Java 7 on our roadmap?
> 
> Last time I checked the latest checkstyle, it did not work with java
> 7 syntax unfortunately...
2.9.1 seems to support it fine.
I had some issues getting Treewalker to work in it (some weird error with 
google collections API incompatibility), but I'm sure it's doable.

> Theoretically checkstyle already supports java 7, probably they only
> forgot about the maven plugin.
> 
> I think this is blocking the way from using java 7 in the code that
> we do not share with GWT.
> 
> > 
> > -Allon
> > 
> > 
> > [1] http://wiki.ovirt.org/Building_oVirt_engine#Installing_OpenJDK
> > [2] maven-compiler-plugin section in ${OVIRT_GIT}/pom.xml
> > [3] http://jenkins.ovirt.org/job/ovirt_engine_animal_sniffer_check/
> > 
> > P.S.
> > If you want to check that you aren't breaking Java/JDK 6
> > compatibility locally without installing java 6, you can run "mvn
> > animal-sniffer:check". Note that animal sniffer analyzes binaries,
> > so this has to be done /after/ the project was built. Of course,
> > you
> > can do this in a single line "mvn install animal-sniffer:check"
> 
> Nice :)
> 
> > ___
> > Engine-devel mailing list
> > Engine-devel@ovirt.org
> > http://lists.ovirt.org/mailman/listinfo/engine-devel
> > 
> 
___
Engine-devel mailing list
Engine-devel@ovirt.org
http://lists.ovirt.org/mailman/listinfo/engine-devel


Re: [Engine-devel] Revisiting Java7

2012-12-04 Thread Allon Mureinik


- Original Message -
> From: "Itamar Heim" 
> To: "Allon Mureinik" 
> Cc: "engine-devel" 
> Sent: Tuesday, December 4, 2012 1:40:10 AM
> Subject: Re: [Engine-devel] Revisiting Java7
> 
> On 12/03/2012 04:03 PM, Allon Mureinik wrote:
> > Hi guys,
> >
> > Earlier today, Java6 compatibility was broken
> > (http://gerrit.ovirt.org/#/c/9430/).
> > This was picked up on pretty quickly, and easily fixed
> > (http://gerrit.ovirt.org/#/c/9666/).
> >
> > However, I think this is a good opportunity to revisit our policy
> > towards Java 7.
> > Currently, we have an "odd" setup, where we recommend running a
> > compiling /with/ Java 7 [1] but comply to Java 6 language level
> > [2] and JDK [3].
> > Inevitably, mistakes like the that happened today will happen.
> >
> > I know we're holding back due to GWT issues, but looking forward to
> > oVirt 4.0, is Java 7 on our roadmap?
> 
> what was the GWT issue?
GWT 2.3 that we use doesn't support java 7 syntax.
The latest version, 2.5, doesn't either.
I'm not sure, however, this is a good reason to enforce the java 6 limitation 
on the entire project (including backed, rest, etc.)
> 
> >
> > -Allon
> >
> >
> > [1] http://wiki.ovirt.org/Building_oVirt_engine#Installing_OpenJDK
> > [2] maven-compiler-plugin section in ${OVIRT_GIT}/pom.xml
> > [3] http://jenkins.ovirt.org/job/ovirt_engine_animal_sniffer_check/
> >
> > P.S.
> > If you want to check that you aren't breaking Java/JDK 6
> > compatibility locally without installing java 6, you can run "mvn
> > animal-sniffer:check". Note that animal sniffer analyzes binaries,
> > so this has to be done /after/ the project was built. Of course,
> > you can do this in a single line "mvn install
> > animal-sniffer:check"
> > ___
> > Engine-devel mailing list
> > Engine-devel@ovirt.org
> > http://lists.ovirt.org/mailman/listinfo/engine-devel
> >
> 
> 
> 
___
Engine-devel mailing list
Engine-devel@ovirt.org
http://lists.ovirt.org/mailman/listinfo/engine-devel


[Engine-devel] Change of engine version number to 3.2.0

2012-12-04 Thread Juan Hernandez
Hello all,

Please note that the change to move engine version number to 3.2.0
(http://gerrit.ovirt.org/8993) has been merged. Please be careful not to
undo it accidentally if you need to push changes to the pom.xml files.

Regards,
Juan Hernandez
-- 
Dirección Comercial: C/Jose Bardasano Baos, 9, Edif. Gorbea 3, planta
3ºD, 28016 Madrid, Spain
Inscrita en el Reg. Mercantil de Madrid – C.I.F. B82657941 - Red Hat S.L.
___
Engine-devel mailing list
Engine-devel@ovirt.org
http://lists.ovirt.org/mailman/listinfo/engine-devel


Re: [Engine-devel] Proposal to add Shireesh Anjal as maintainer to engine-gluster

2012-12-04 Thread Allon Mureinik
+1

- Original Message -
> From: "Itamar Heim" 
> To: engine-devel@ovirt.org
> Sent: Friday, November 30, 2012 1:28:51 PM
> Subject: [Engine-devel] Proposal to add Shireesh Anjal as maintainer to   
> engine-gluster
> 
> Shireesh has been working on the engine gluster support for the past
> year, covering patches and leading reviews of the various engine
> gluster
> patches.
> 
> I'd like to propose a new sub-component of the engine for the gluster
> module, adding shireesh as its maintainer.
> 
> Thanks,
> Itamar
> ___
> Engine-devel mailing list
> Engine-devel@ovirt.org
> http://lists.ovirt.org/mailman/listinfo/engine-devel
> 
___
Engine-devel mailing list
Engine-devel@ovirt.org
http://lists.ovirt.org/mailman/listinfo/engine-devel