Re: [webkit-dev] New feature announcement – Video conferencing and peer-to-peer communication

[Leandro Graciá Gil]

About the modularity proposal, I have no problem creating a new separate
folder in WebCore for this feature. An example of its structure could be the
following.

In WebCore/mediastream:

   - GeneratedStream.h/cpp/idl
   - Stream.h/cpp/idl
   - StreamRecorder.h/cpp/idl
   - CallbackTask.h
   - MediaStreamClient.h
   - MediaStreamController.h/cpp
   - MediaStreamFrameController.h/cpp
   - NavigatorUserMediaError.h/idl
   - NavigatorUserMediaErrorCallback.h/idl
   - NavigatorUserMediaSuccessCallback.h/idl

In platform:

   - MediaStreamRegistry.h/cpp (required for the DOMURL)
   - mock/MediaStreamClientMock.h/cpp

In fileapi:

   - BlobCallback.h/idl  (for the blob-providing callback used in the
   StreamRecorders)


We would need to decide where to place the code for the tracklist objects,
as they are also used by the HTMLMediaElement and enabled by the VIDEO_TRACK
flag.
These are:

   - TrackList.h/cpp/idl
   - MultipleTrackList.h/cpp/idl
   - ExclusiveTrackList.h/cpp/idl


Finally, it requires minor modifications in the following files in WebCore.
Note that this list may be increased in the future.

   - dom/EventTarget.h/cpp
   - dom/ScriptExecutionContext.h/cpp (for the DOMURL)
   - html/DOMURL.h/cpp/idl
   - page/Navigator.h/cpp/idl
   - page/DOMWindow.h/idl  (to enable DOMURL with both the BLOB and
   MEDIA_STREAM flags)
   - page/Frame.h/cpp
   - page/Page.h/cpp


So, with this more detailed information, do you think I should move the code
to a separate folder? Any suggestions on where to put the tracklist object
files? I'd like to have some consensus on this if possible, as this decision
will affect the landing of the remaining patches.

Thanks,
Leandro


On 17 May 2011 00:15, Ian Hickson  wrote:

> On Fri, 13 May 2011, Eric Seidel wrote:
> >
> > On the surface "peer to peer video conferencing" does not seem like
> > something appropriate to add to WebCore/WebKit.  Just like "an API for
> > reading my email" is out of scope for the project.  (But certainly lots
> > of people build such things on top of WebKit.) :)
>
> What people are talking about as "video conferencing and peer-to-peer
> communication" here really are the building blocks for letting people
> build such things on top of the Web platform. The relevant part of the
> HTML spec is basically just these features:
>
>  - A way to get input from the system camera(s) and microphone(s).
>
>  - A way for two peers to connect and exchange data and audio/video
>   streams directly, reusing the widely used ICE, STUN, TURN, and SDP
>   technologies, assuming they have an existing indirect channel.
>
>  - A way to record or display these streams.
>
> Actually deploying a video conferencing system would need far more than
> this. For example, you need identity and presence, which are not part of
> the Web platform but have to be built on top of it. You also need to
> actually implement the "existing indirect channel" mentioned above (e.g.
> using XHR or WebSockets), and you need to provide the STUN and TURN
> servers in the cloud.
>
> The components that are provided by this part of the HTML spec have other
> purposes beyond video conferencing; for example you can use the access to
> local camera and microphones for augmented reality tools, you can use the
> peer-to-peer component for exchanging real-time game state data in a video
> game, and you can use the recording component for recording video blogs.
>
> The parallel to e-mail apps is apt: it is much like how the platform
> provides XHR, input controls, and local storage, all necessary components
> for an e-mail app, but does not provide an "e-mail app" component; and the
> components it does provide can be similarly reused in other contexts.
>
> HTH,
> --
> Ian Hickson   U+1047E)\._.,--,'``.fL
> http://ln.hixie.ch/   U+263A/,   _.. \   _\  ;`._ ,.
> Things that are impossible just take longer.   `._.-(,_..'--(,_..'`-.;.'
>
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev

Re: [webkit-dev] Implementing the element

[Leandro Graciá Gil]

Hi,

El 22 de febrero de 2011 12:20, Adam Bergkvist
escribió:

> Hi,
>
> On 2011-02-15 17:50, Leandro Graciá Gil wrote:
> > Yes, the platform independent code in the device element will
> determine
> > if a selection has been made, or altered, in the list of available
> > devices
> > Provided by the platform. Before passing it to the device element,
> the
> > list can be filtered by the platform to exclude some entries for
> > whatever
> > reason. Selections can also be automatically applied without showing
> the
> > Device selector based on, e.g., stored preferences. The device
> element
> > will examine the list for selection changes to determine if a new
> device
> > handler should be created (and a change event dispatched).
> >
> > Consider the following example:
> > 1. The user clicks the device element button and a list of devices is
> >presented.
> > 2. The user selects a webcam in the list and clicks OK.
> > *A new Stream is created*
> > 3. The user once again clicks the device element button and the list
> >is presented with the webcam selected from before.
> > 4. The user clicks OK.
> > *No new Stream is created*
> > 5. The user once again clicks the device element button and the list
> >is presented with the webcam selected from before.
> > 6. The user selects a microphone in the list and clicks OK.
> > *A new Stream is created*
> >
> > This is the behavior that should be consistent between ports.
> >
> >
> > I agree that this behavior should be consisted between ports. However,
> > we seem to be missing a case here: what happens if the user wants to
> > revoke access to a device in the middle of a streaming session? We need
> > to be able to allow this case for obvious privacy reasons.
> >
>
> I agree that this is definitively something we need to consider, but
> perhaps
> as a (near) future improvement. In fact, we have previously been discussing
> how to revoke device access and came to the conclusion that it must be done
> via the browser chrome since the device element can be removed from the
> page
> as soon as access has been granted. A natural choice would be to combine
> this with the mechanism for indicating that a camera/mic is active
> (discussed in the initial device thread
> http://lists.w3.org/Archives/Public/public-device-apis/2009Dec/0149.html).
> That would however require additions to the WebKit API. We have left this
> out in the first version, simply because it is not mentioned in the spec,
> and prioritized to land an initial implemenation that could be used as a
> basis for improvements.
>

I'm glad we agree on the importance of allowing the user to revoke device
access. However we think that this point is important at the design level
rather than the implementation level. If access revoking is introduced, then
the concept of access lifetime arises and needs to be managed within the
code. This access lifetime, let's call it session for instance, is exactly
the concept we introduced (and misnamed) as 'connections'. These sessions
and their management are the core of our proposed model, causing most of the
design differences and problems between our proposals. That's why we would
like to get this point clear at the design level before submitting any
implementation to be patched later.

I also like your proposal of providing access revocation in the same UI used
to notify active device sessions. However I'm not sure about how do you plan
to do this using the chrome client without explicitly managing the device
access lifetime (sessions) in the common WebCore code. It also sounds to me
more like an event that WebCore receives from the UA than a request from
WebCore to the chrome client. Could you please elaborate on this?

Also about your proposal for access revocation, I would like to point
that it's unintuitive to have to click on one button inside the page to
enable access to camera and to have to click on another button, located in a
different place altogether, to end this access. Furthermore, consider mobile
browsers where the screen real state is extremely expensive. Where would you
place this chrome-based UI? To be realistic, I think we'd have to allow
both: let the  renderer have state and act as a toggle, as well as
allow the possibility to have the device session ended via some UI element
in the browser's chrome.


>
> > We need a way to let the UA maintain the lifetime of Stream objects.
> > These can be killed by the UA at any point (perhaps due to user action
> > or hardware failure), so we need to specify what happens to these

Re: [webkit-dev] Implementing the element

[Leandro Graciá Gil]

Hi,

El 10 de febrero de 2011 15:15, Adam Bergkvist
escribió:

> Hi,
>
> On 2011-02-09 22:10, Leandro Graciá Gil wrote:
> > So, does that mean that a WebCore platform-independent code is going
> > to determine if a new device handler should be created? I think that
> > some UAs may like to, for example, keep a track of trusted pages for
> > specific types of devices to determine if such a handler should be
> > created. And that is not likely to be implemented in the same way by
> > all the different platforms even if they intend to perform a set of
> > basic common security steps. I completely agree with you that the
> > behaviour should be consistent across platforms, especially the
> > security and privacy aspects, but I don't think that forcing the
> > common code to use available device lists is the way to do it. If we
> > want a consistent behaviour we should ask for it in the specification
> > and leave the implementation specific details open, not the other way
> around.
> >
>
> Yes, the platform independent code in the device element will determine
> if a selection has been made, or altered, in the list of available devices
> Provided by the platform. Before passing it to the device element, the
> list can be filtered by the platform to exclude some entries for whatever
> reason. Selections can also be automatically applied without showing the
> Device selector based on, e.g., stored preferences. The device element
> will examine the list for selection changes to determine if a new device
> handler should be created (and a change event dispatched).
>
> Consider the following example:
> 1. The user clicks the device element button and a list of devices is
>   presented.
> 2. The user selects a webcam in the list and clicks OK.
> *A new Stream is created*
> 3. The user once again clicks the device element button and the list
>   is presented with the webcam selected from before.
> 4. The user clicks OK.
> *No new Stream is created*
> 5. The user once again clicks the device element button and the list
>   is presented with the webcam selected from before.
> 6. The user selects a microphone in the list and clicks OK.
> *A new Stream is created*
>
> This is the behavior that should be consistent between ports.
>

I agree that this behavior should be consisted between ports. However, we
seem to be missing a case here: what happens if the user wants to revoke
access to a device in the middle of a streaming session? We need to be able
to allow this case for obvious privacy reasons.

We need a way to let the UA maintain the lifetime of Stream objects. These
can be killed by the UA at any point (perhaps due to user action or hardware
failure), so we need to specify what happens to these objects in such a
case. We also need to design our implementation in a way that can handle
this situation.

As far as I know, and please correct me if I'm wrong, your proposed design
doesn't provide the user a clear way to invalidate a stream if he wants to.
The Stream's lifetime is something completely handled by the
platform-specific client and with no presence at all in the common WebCore
code. We think this should also be an essential part of the common behaviour
across platforms.

Because of this reason we would like to raise this discussion to the
specification level in the whatwg list. We think that the expected behaviour
for the device element is something that still needs to be discussed before
facing any implementation-specific details.


>
> > One important point that a selection-based design misses is that
> > potentially many devices, if not all, will require exclusive access to
> > them. Let me give a example. We have two different pages with device
> > elements. First, a user selects some device (e.g. a microphone) in the
> > first page, but the page makes no use of it for the moment and hence
> > the real connection is not performed. Minutes later and without
> > closing the first page, the same user selects the same device from
> > before on the second device element (if you have only one microphone
> > and an average user this could be very easy to happen) but this page
> > also makes no use for it at that moment. What happens if for example
> > the first page starts making use of the device and causes the
> > connections on the second page to fail? This is almost for sure not
> > the behaviour that the user is expecting or wanting.
> >
>
> It will be up to the device handler API (e.g. Stream) and its potential
> consumers to handle any errors that may occur. Exclusiveness for certain
> types of devices being one type of error. Another error occurs when a
> peripheral device is disconnected while in use. Even if you have

Re: [webkit-dev] Implementing the element

[Leandro Graciá Gil]

Hi,

El 7 de febrero de 2011 18:56, Adam Bergkvist 
 escribió:

> Hi,
>
> On 2011-02-04 19:21, Leandro Graciá Gil wrote:
> > This is good news! Especially for the situations where WebCore can't
> > directly access the hardware. One existing case of this we should keep
> > in mind are the sandboxed environments, where both the probing and the
> > connections must be requested to somewhere outside the sandbox. Usually
> > this will require to communicate with another process, and in this case
> > asynchronous messages are preferred to avoid delays and to make
> > inter-process communication simpler.
> >
> > However I have to disagree in one point. The specification doesn't say
> > anywhere that we should always present a dialog, only that the device
> > element represents a device selector.
>
> The UI has been a central part of the discussions that lead up to the
> device element proposal. The reason for that is that it should enforce
> security.
>
> See the thread "UI for enabling webcam use from untrusted content":
> http://lists.w3.org/Archives/Public/public-device-apis/2009Dec/0149.html
> especially
> http://lists.w3.org/Archives/Public/public-device-apis/2009Dec/0194.html
>
> You're right that we shouldn't limit ourselves by referring to the device
> selector as a dialog. I've renamed ChromeClient::runDeviceDialog to
> runDeviceSelector and the corresponding client interface accordingly.
>

Thanks. I also think that this name, and more importantly this point of
view, is more appropriate.


>
> > Consider this completely valid use
> > case: instead of asking repeatedly the user to select a device, a UA
> > might decide to create some kind of internal device settings
> > configuration panel to select a set of default devices. Later when
> > visiting a page and clicking the device element the UA will
> > automatically use the preferred devices from its internal settings if
> > they are available and the page is trusted. Where is the dialog here?
> >
>
> Couldn't you just see the internal device settings configuration panel,
> you mention, as the device selector that produces a device list that's
> reused several times? In that case you would skip the dialog and simply
> apply the predefined selections (similar to the case where a "remember
> my choices" check box would be available in the device dialog).
>
> Yes, of course. The reason I proposed that use case was to show that we
should not force any dialog, but ask for selection in the way that the UA
decides. This was just an example of what I mentioned in the previous
paragraph and I'm glad that we agree on this.


>
> > I agree that the device should perform selection, as the spec says.
> > However as I've already explained I don't think we need for example a
> > selection dialog for all use cases. Considering that we don't explicitly
> > need a dialog to perform the selection, the only reason to bring lists
> > of available devices back to WebCore is to send them again to a client,
> > probably the same one we asked for probing. Also, if we consider the
> > possibility of sandboxed environments then the device connection
> > operation cannot be a synchronous operation as commented before.
> >
>
> As mentioned above, I see a point in sending the list of available
> devices to WebCore to determine if a new Stream (or other device
> handler) should be created since this behavior should be consistent
> across platforms, regardless if the device is of type "media" or
> "fishtank".
>

So, does that mean that a WebCore platform-independent code is going to
determine if a new device handler should be created? I think that some UAs
may like to, for example, keep a track of trusted pages for specific types
of devices to determine if such a handler should be created. And that is not
likely to be implemented in the same way by all the different platforms even
if they intend to perform a set of basic common security steps. I completely
agree with you that the behaviour should be consistent across platforms,
especially the security and privacy aspects, but I don't think that forcing
the common code to use available device lists is the way to do it. If we
want a consistent behaviour we should ask for it in the specification and
leave the implementation specific details open, not the other way around.


>
> The device "connection operation" is not handled by the device element. The
> device element is used to simply select devices (similar to how you select
> a file with  and get a File object which is just a handle
> to the actual file). The "connection" takes p

Re: [webkit-dev] Implementing the element

[Leandro Graciá Gil]

El 4 de febrero de 2011 18:21, Leandro Graciá Gil <
leandrogra...@chromium.org> escribió:

>
> I have also created a small diagram to explain how our implementation
> works:
>
> https://docs.google.com/a/google.com/drawings/edit?id=1jSW-6MJd8mp2qPvwnvZnBVzll6UtBz3r1viZgTE4XVA&hl=en&authkey=CPLpy5oJ
>

It seems that my previous link to the diagram has permission problems. I've
uploaded it as a pdf file in https://bugs.webkit.org/show_bug.cgi?id=53777 .
Sorry for the inconveniences.
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev

Re: [webkit-dev] Implementing the element

[Leandro Graciá Gil]

El 3 de febrero de 2011 18:59, Adam Bergkvist 
 escribió:

> Hi,
>
> Leandro Graciá Gil wrote:
> > The approach proposed in https://bugs.webkit.org/show_bug.cgi?id=47264
> > brings the device probing and selection to the WebCore level. It does
> > so by first creating a list of available devices using an interface to
> > a platform-dependent client and then it passes this list to a client
> > dialog to perform the actual selection from it. With its current
> > design both operations are performed synchronously, possibly blocking
> > the device element event handler for a long time. I think this is not
> > a suitable design.
>
> The device selector dialog was never intended to block the event loop.
> That was a mistake. In GTK the main loop continues to run when the dialog
> is shown so although the click handler is blocked the event loop is not
> blocked. We should not have expected this behavior on other platforms.
> I have now added a callback from the dialog to the device element when a
> selection has been made, making it possible to implement a completely
> asynchronous dialog.


> The interface towards the platform for getting a list of available
> devices is called createDeviceList; it makes no assumption about how
> the probing is done. The only assumption is that when a device selector
> dialog is about to be presented to the user, a list of available devices
> is needed. Typically, probing involves asking the OS kernel for a list of
> available devices and this operation is usually fast. That's why we in our
> GTK implementation have chosen to simply do the probing synchronously from
> createDeviceList. I have now moved the createDeviceList call to device
> dialog code. This makes it possible for platforms that want to do
> on-demand probing but fear it might block, to call it off the main loop.
>

This is good news! Especially for the situations where WebCore can't
directly access the hardware. One existing case of this we should keep in
mind are the sandboxed environments, where both the probing and the
connections must be requested to somewhere outside the sandbox. Usually this
will require to communicate with another process, and in this case
asynchronous messages are preferred to avoid delays and to make
inter-process communication simpler.

However I have to disagree in one point. The specification doesn't say
anywhere that we should always present a dialog, only that the device
element represents a device selector. Consider this completely valid use
case: instead of asking repeatedly the user to select a device, a UA might
decide to create some kind of internal device settings configuration panel
to select a set of default devices. Later when visiting a page and clicking
the device element the UA will automatically use the preferred devices from
its internal settings if they are available and the page is trusted. Where
is the dialog here?


>
> > Given the drawbacks of the approach above, I would like to propose an
> > alternative design that solves these issues.
> > First of all, we think that is not necessary at all to bring the
> > device probing, available device lists, device selection or connection to
> WebCore.
> > The device element isn't really about actual devices, but about
> > connecting to them. We think that it would be the best for all
> > platforms to actually delegate the actual probing, selection and
> > connecting to them in an asynchronous client-based model and hold only
> > connection and handler information in WebCore.
> >
> The device element is about devices and specifically about selecting
> devices.
>  is inspired by  and so is our implementation.
> When
> the change event should trigger on the device element, for example, is
> something that will eventually be specced out and so the code that handles
> this logic is best put in WebCore in order to preserve consistency between
> ports.
>

I agree that the device should perform selection, as the spec says. However
as I've already explained I don't think we need for example a selection
dialog for all use cases. Considering that we don't explicitly need a dialog
to perform the selection, the only reason to bring lists of available
devices back to WebCore is to send them again to a client, probably the same
one we asked for probing. Also, if we consider the possibility of sandboxed
environments then the device connection operation cannot be a synchronous
operation as commented before.

Reviewing the design with all these factors leaves us the following scheme:
- Request device selection asynchronously to the client (not necessarily
using a dialog).
- Retrieve the available device list.
- Forward the list to a client (probably the same that a moment ago probed
the devices) to con

Re: [webkit-dev] Implementing the element

[Leandro Graciá Gil]

I have created a patch to introduce compilation guards for the device
element implementation. These should be the same independently of the design
approach we decide to take while enabling us to propose more specific
patches later. Here's the link:
https://bugs.webkit.org/show_bug.cgi?id=53595

Thanks,
Leandro

PS: sorry if you received this twice. I used the wrong account before and it
seems that it wasn't properly reported as a thread reply in the list
archive.

El 1 de febrero de 2011 01:43, Leandro Graciá Gil <
leandrogra...@chromium.org> escribió:

> Hi,
>
> We have recently noticed a patch to implement the  element in
> WebKit. Since this is an important new feature, I thought we should have a
> discussion about the best way to implement it. Here's the actual
> specification for it:
> http://www.whatwg.org/specs/web-apps/current-work/complete/commands.html#devices
>  .
>
> The approach proposed in https://bugs.webkit.org/show_bug.cgi?id=47264 brings
> the device probing and selection to the WebCore level. It does so by first
> creating a list of available devices using an interface to a
> platform-dependent client and then it passes this list to a client dialog to
> perform the actual selection from it. With its current design both
> operations are performed synchronously, possibly blocking the device element
> event handler for a long time. I think this is not a suitable design.
>
> Given the drawbacks of the approach above, I would like to propose an
> alternative design that solves these issues.
> First of all, we think that is not necessary at all to bring the device
> probing, available device lists, device selection or connection to WebCore.
> The device element isn't really about actual devices, but about connecting
> to them. We think that it would be the best for all platforms to actually
> delegate the actual probing, selection and connecting to them in an
> asynchronous client-based model and hold only connection and handler
> information in WebCore.
>
> Here's an example of the call flow in our proposal:
>
> 0. The device element is created in a no connection state and with an empty
> device connection descriptor.
> 1. User clicks the device element (a button, for example).
> 2. The device element changes its internal state from no connection to
> connecting, and asks to the device controller to connect to a device of its
> type sending also the document security origin.
> 3. The device controller, which handles and maps all the device requests of
> the page, forwards the connection request to the device page client. This
> client is implemented by the corresponding WebKit platforms.
> 4. The device client receives the request and, in a non-blocking way,
> implements the way it likes the device probing, selection, connecting and
> all the required security operations (the latter also missing in the
> existing patch). This allows every UA vendor to make their own choices about
> the UI for device selection.
>
> 5. When the platform has finished connecting to a device, it sends back to
> the device controller a device connection update message. This message comes
> with a device connection descriptor that contains the actual connection
> status, error codes/messages, device ids/names if connected and so on. In
> case of user cancelation, connection or communication error the same message
> is sent, but with different device connection descriptor values.
> 6. The device controller forwards the message to the original device
> element. The original element sets its connection descriptor to the received
> object and changes its state accordingly. It will change to connected if the
> connection was successful or to not connected in case of error. It will also
> fire the appropriate JS events.
> 7. If the connection was successful, the element will then create valid
> stream objects from its data attribute. These will not contain any real
> streaming data but just a url string. All the actual data is internally
> managed by the platforms, leaving only handler-like objects in WebCore. The
> url can be provided for the moment with the connection descriptor object,
> being later replaced by some type-independent object when device types other
> than media are defined.
>
> Requesting to disconnect the device would work much like requesting a
> connection, but providing the device descriptor object as a parameter
> instead.
>
> With this approach, not only we avoid to block WebCore with potentially
> long operations but we provide a great flexibility to all WebKit platforms
> to implement the nasty device handling details as they want to. This should
> also potentially increase the amount of code that can be reused by the
> different platforms, wh

Re: [webkit-dev] Implementing the element

[Leandro Graciá Gil]

I have created a patch to introduce compilation guards for the device
element implementation. These should be the same independently of the design
approach we decide to take while enabling us to propose more specific
patches later. Here's the link:
https://bugs.webkit.org/show_bug.cgi?id=53595

Thanks,
Leandro

El 1 de febrero de 2011 01:43, Leandro Graciá Gil <
leandrogra...@chromium.org> escribió:

> Hi,
>
> We have recently noticed a patch to implement the  element in
> WebKit. Since this is an important new feature, I thought we should have a
> discussion about the best way to implement it. Here's the actual
> specification for it:
> http://www.whatwg.org/specs/web-apps/current-work/complete/commands.html#devices
>  .
>
> The approach proposed in https://bugs.webkit.org/show_bug.cgi?id=47264 brings
> the device probing and selection to the WebCore level. It does so by first
> creating a list of available devices using an interface to a
> platform-dependent client and then it passes this list to a client dialog to
> perform the actual selection from it. With its current design both
> operations are performed synchronously, possibly blocking the device element
> event handler for a long time. I think this is not a suitable design.
>
> Given the drawbacks of the approach above, I would like to propose an
> alternative design that solves these issues.
> First of all, we think that is not necessary at all to bring the device
> probing, available device lists, device selection or connection to WebCore.
> The device element isn't really about actual devices, but about connecting
> to them. We think that it would be the best for all platforms to actually
> delegate the actual probing, selection and connecting to them in an
> asynchronous client-based model and hold only connection and handler
> information in WebCore.
>
> Here's an example of the call flow in our proposal:
>
> 0. The device element is created in a no connection state and with an empty
> device connection descriptor.
> 1. User clicks the device element (a button, for example).
> 2. The device element changes its internal state from no connection to
> connecting, and asks to the device controller to connect to a device of its
> type sending also the document security origin.
> 3. The device controller, which handles and maps all the device requests of
> the page, forwards the connection request to the device page client. This
> client is implemented by the corresponding WebKit platforms.
> 4. The device client receives the request and, in a non-blocking way,
> implements the way it likes the device probing, selection, connecting and
> all the required security operations (the latter also missing in the
> existing patch). This allows every UA vendor to make their own choices about
> the UI for device selection.
>
> 5. When the platform has finished connecting to a device, it sends back to
> the device controller a device connection update message. This message comes
> with a device connection descriptor that contains the actual connection
> status, error codes/messages, device ids/names if connected and so on. In
> case of user cancelation, connection or communication error the same message
> is sent, but with different device connection descriptor values.
> 6. The device controller forwards the message to the original device
> element. The original element sets its connection descriptor to the received
> object and changes its state accordingly. It will change to connected if the
> connection was successful or to not connected in case of error. It will also
> fire the appropriate JS events.
> 7. If the connection was successful, the element will then create valid
> stream objects from its data attribute. These will not contain any real
> streaming data but just a url string. All the actual data is internally
> managed by the platforms, leaving only handler-like objects in WebCore. The
> url can be provided for the moment with the connection descriptor object,
> being later replaced by some type-independent object when device types other
> than media are defined.
>
> Requesting to disconnect the device would work much like requesting a
> connection, but providing the device descriptor object as a parameter
> instead.
>
> With this approach, not only we avoid to block WebCore with potentially
> long operations but we provide a great flexibility to all WebKit platforms
> to implement the nasty device handling details as they want to. This should
> also potentially increase the amount of code that can be reused by the
> different platforms, while avoiding any list matching code that can be found
> in the existing patch.
>
> I have a patch that implements this design, but I wanted to get som

Re: [webkit-dev] element

[Leandro Graciá Gil]

Hi René,

There's already people working on this in WebKit. I can't speak about Safari
but we're actually discussing different approaches to its
implementation. Please take a look at this thread and feel free to
contribute:
https://lists.webkit.org/pipermail/webkit-dev/2011-January/015822.html

Regards,
Leandro

El 2 de febrero de 2011 14:16, "René Müller" escribió:

> Hello Safari,
>
> Recently I read about the device element in the HTML specification. It
> seems the possibilities of such an element wourld improve a browser´s
> features sigificantly. Since the device element is not implemented in any
> browser yet, is Safari already planning to implement this?
>
> Link to device element:
> http://www.whatwg.org/specs/web-apps/current-work/complete/commands.html#devices
>
> Regards,
> Rene
>
>
> NEU: FreePhone - kostenlos mobil telefonieren und surfen!
> Jetzt informieren: 
> *http://produkte.web.de/go/webdefreephone*
>
> ___
> webkit-dev mailing list
> webkit-dev@lists.webkit.org
> http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
>
>
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev

[webkit-dev] Implementing the element

[Leandro Graciá Gil]

Hi,

We have recently noticed a patch to implement the  element in
WebKit. Since this is an important new feature, I thought we should have a
discussion about the best way to implement it. Here's the actual
specification for it:
http://www.whatwg.org/specs/web-apps/current-work/complete/commands.html#devices
 .

The approach proposed in https://bugs.webkit.org/show_bug.cgi?id=47264 brings
the device probing and selection to the WebCore level. It does so by first
creating a list of available devices using an interface to a
platform-dependent client and then it passes this list to a client dialog to
perform the actual selection from it. With its current design both
operations are performed synchronously, possibly blocking the device element
event handler for a long time. I think this is not a suitable design.

Given the drawbacks of the approach above, I would like to propose an
alternative design that solves these issues.
First of all, we think that is not necessary at all to bring the device
probing, available device lists, device selection or connection to WebCore.
The device element isn't really about actual devices, but about connecting
to them. We think that it would be the best for all platforms to actually
delegate the actual probing, selection and connecting to them in an
asynchronous client-based model and hold only connection and handler
information in WebCore.

Here's an example of the call flow in our proposal:

0. The device element is created in a no connection state and with an empty
device connection descriptor.
1. User clicks the device element (a button, for example).
2. The device element changes its internal state from no connection to
connecting, and asks to the device controller to connect to a device of its
type sending also the document security origin.
3. The device controller, which handles and maps all the device requests of
the page, forwards the connection request to the device page client. This
client is implemented by the corresponding WebKit platforms.
4. The device client receives the request and, in a non-blocking way,
implements the way it likes the device probing, selection, connecting and
all the required security operations (the latter also missing in the
existing patch). This allows every UA vendor to make their own choices about
the UI for device selection.

5. When the platform has finished connecting to a device, it sends back to
the device controller a device connection update message. This message comes
with a device connection descriptor that contains the actual connection
status, error codes/messages, device ids/names if connected and so on. In
case of user cancelation, connection or communication error the same message
is sent, but with different device connection descriptor values.
6. The device controller forwards the message to the original device
element. The original element sets its connection descriptor to the received
object and changes its state accordingly. It will change to connected if the
connection was successful or to not connected in case of error. It will also
fire the appropriate JS events.
7. If the connection was successful, the element will then create valid
stream objects from its data attribute. These will not contain any real
streaming data but just a url string. All the actual data is internally
managed by the platforms, leaving only handler-like objects in WebCore. The
url can be provided for the moment with the connection descriptor object,
being later replaced by some type-independent object when device types other
than media are defined.

Requesting to disconnect the device would work much like requesting a
connection, but providing the device descriptor object as a parameter
instead.

With this approach, not only we avoid to block WebCore with potentially long
operations but we provide a great flexibility to all WebKit platforms to
implement the nasty device handling details as they want to. This should
also potentially increase the amount of code that can be reused by the
different platforms, while avoiding any list matching code that can be found
in the existing patch.

I have a patch that implements this design, but I wanted to get some
feedback on this list before sending it out for review.

Please don't hesitate to make any suggestions that could help to improve
this.

Thanks,
Leandro
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev

Re: [webkit-dev] PSA: Don't try to hold onto temporaries with references

[Leandro Graciá Gil]

>
> In summary, looking at code like this
>
>  B& b = c->foo();
>  ...
>  b.m();
>
> If c->foo() returns a temporary ("return B();"), then it is safe.
>

Maybe I'm wrong, but are you completely sure about this one? I would say
that the temporary object created in return B() will cease to exist as soon
as it returns (just after the constructor finishes). So you will be
returning a reference to a temporary which, I think, no longer is valid. I
made a quick test to be sure and the destructor of B is indeed called. Why
is it safe?
___
webkit-dev mailing list
webkit-dev@lists.webkit.org
http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev