Re: Implementing a Chrome DevTools Protocol server in Firefox

[kenneth]

On Tuesday, September 5, 2017 at 10:57:58 AM UTC-7, Jim Blandy wrote:
> As an offer of help, from a group whose charter covers this work, that's
> very welcome. I felt that I was being shepherded into something on behalf
> of others for whom I cannot speak, which was uncomfortable!
> 
> For my own sake, I am disinclined to participate in a standardization
> effort outside of the usual institutions. I think we could benefit from the
> experience of people who have produced web standards before. I'm not
> well-connected enough yet to anticipate what folks will say, but I'll
> suggest the Browser Testing Tools WG when the opportunity comes up.
> 
> 
> On Tue, Sep 5, 2017 at 5:32 AM, James Graham  wrote:
> 
> > On 04/09/17 23:34, Jim Blandy wrote:
> >
> >> On Mon, Sep 4, 2017 at 7:36 AM, David Burns  wrote:
> >>
> >>> I don't think anyone would disagree with the reasons for doing this. I,
> >>>
> >> like James who brought it up earlier, am concerned that we from the emails
> >> appear to think that implementing the wire protocol would be sufficient to
> >> making sure we have the same semantics.
> >>
> >> LOL, give us a little credit, okay? The authors of the email do not think
> >> that. We want to have a properly written specification and conformance
> >> tests. I think you're reading "we have no interest in established
> >> standardization processes" when what we wrote was "the process is in very
> >> early stages".
> >>
> >> Do you think the Browser Testing Tools WG is the right body to work on a
> >> JS
> >> debugging and console protocol, used by interactive developer tools? That
> >> seems like a surprising choice to me.
> >>
> >
> > It is certainly not the only possible venue, but if you want to do the
> > work at the W3C then it's probably the easiest way to get things going from
> > a Process point of view, since this kind of protocol would be in the
> > general remit of the group, and the rechartering could add it specifically.
> > Certainly the people currently in the group aren't the right ones to do the
> > work, but adding new participants to work specifically on this would be
> > trivial.
> >
> > Also - at least as far as I know -  this is not where the current
> >> participants in the discussion (Kenneth Auchenberg or Christian Bromann,
> >> to
> >> name two) have been working. Is having a previously uninvolved standards
> >> committee take up an area in which current activity is occurring elsewhere
> >> considered friendly and cooperative behavior? It seems unfriendly to me. I
> >> would like to avoid upsetting the people I'm hoping to work closely with.
> >>
> >
> > I think you have misinterpreted the intent here. I don't think anyone is
> > interested in doing a hostile takeover of existing work. But there is
> > concern that the work actually happens. Pointing at remotedebug.org,
> > which has been around since 2013 without producing any specification
> > materials, isn't helping assuage my concerns, and I guess others are having
> > a similar reaction. It is of course entirely possible that there's work
> > going on that we can't see. But my interpretation of David's email is that
> > he is trying to offer you options, not force you down a certain path. The
> > W3C is not always the right venue to work in, but it is sometimes sought
> > out by organisations who would likely participate in this work because of
> > its relatively strong IPR policy.
> >
> > I should stress that irrespective of venue I would expect this
> > standardisation effort to take years; people always underestimate the work
> > and time required for standards work. It will certainly require us to
> > commit resources to make it happen.
> >
> > ___
> > dev-platform mailing list
> > dev-platform@lists.mozilla.org
> > https://lists.mozilla.org/listinfo/dev-platform
> >

Hi everyone,

Kenneth, the original initiator of RemoteDebug here. I just want to provide an 
overview of the state of the RemoteDebug initiative and how I see CDP fit into 
the broader tooling ecosystem.

Status of RemoteDebug

Now several years after I initiated RemoteDebug a lot has happened in the 
tooling ecosystem, as we today have several 3rd party tools build on and around 
CDP by the big players (Debugger.html by Mozilla, Lighthouse+ Puppeteer by 
Google, and Sonar + VS Code debuggers by Microsoft), which has reinforced that 
CDP is the de-facto DevTools "standard" protocol.

At the last Chrome DevSummit (2016) we had a meeting where Firefox DevTools, 
Edge DevTools and Chrome DevTools where present to discuss the latest 
RemoteDebug proposal, the RemoteDebug DevTools Core Specification [1], which 
could serve as an implementation starting point for new vendors who want to 
embrace CDP compatibility but focus on the stable parts of the API. At the 
meeting it was agreed that Google would focus on stabilizing most of the CDP 
API in its CDP 1.2 version (which 

Re: Implementing a Chrome DevTools Protocol server in Firefox

[Jim Blandy]

As an offer of help, from a group whose charter covers this work, that's
very welcome. I felt that I was being shepherded into something on behalf
of others for whom I cannot speak, which was uncomfortable!

For my own sake, I am disinclined to participate in a standardization
effort outside of the usual institutions. I think we could benefit from the
experience of people who have produced web standards before. I'm not
well-connected enough yet to anticipate what folks will say, but I'll
suggest the Browser Testing Tools WG when the opportunity comes up.


On Tue, Sep 5, 2017 at 5:32 AM, James Graham  wrote:

> On 04/09/17 23:34, Jim Blandy wrote:
>
>> On Mon, Sep 4, 2017 at 7:36 AM, David Burns  wrote:
>>
>>> I don't think anyone would disagree with the reasons for doing this. I,
>>>
>> like James who brought it up earlier, am concerned that we from the emails
>> appear to think that implementing the wire protocol would be sufficient to
>> making sure we have the same semantics.
>>
>> LOL, give us a little credit, okay? The authors of the email do not think
>> that. We want to have a properly written specification and conformance
>> tests. I think you're reading "we have no interest in established
>> standardization processes" when what we wrote was "the process is in very
>> early stages".
>>
>> Do you think the Browser Testing Tools WG is the right body to work on a
>> JS
>> debugging and console protocol, used by interactive developer tools? That
>> seems like a surprising choice to me.
>>
>
> It is certainly not the only possible venue, but if you want to do the
> work at the W3C then it's probably the easiest way to get things going from
> a Process point of view, since this kind of protocol would be in the
> general remit of the group, and the rechartering could add it specifically.
> Certainly the people currently in the group aren't the right ones to do the
> work, but adding new participants to work specifically on this would be
> trivial.
>
> Also - at least as far as I know -  this is not where the current
>> participants in the discussion (Kenneth Auchenberg or Christian Bromann,
>> to
>> name two) have been working. Is having a previously uninvolved standards
>> committee take up an area in which current activity is occurring elsewhere
>> considered friendly and cooperative behavior? It seems unfriendly to me. I
>> would like to avoid upsetting the people I'm hoping to work closely with.
>>
>
> I think you have misinterpreted the intent here. I don't think anyone is
> interested in doing a hostile takeover of existing work. But there is
> concern that the work actually happens. Pointing at remotedebug.org,
> which has been around since 2013 without producing any specification
> materials, isn't helping assuage my concerns, and I guess others are having
> a similar reaction. It is of course entirely possible that there's work
> going on that we can't see. But my interpretation of David's email is that
> he is trying to offer you options, not force you down a certain path. The
> W3C is not always the right venue to work in, but it is sometimes sought
> out by organisations who would likely participate in this work because of
> its relatively strong IPR policy.
>
> I should stress that irrespective of venue I would expect this
> standardisation effort to take years; people always underestimate the work
> and time required for standards work. It will certainly require us to
> commit resources to make it happen.
>
> ___
> dev-platform mailing list
> dev-platform@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-platform
>
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Implementing a Chrome DevTools Protocol server in Firefox

[James Graham]

On 04/09/17 23:34, Jim Blandy wrote:

On Mon, Sep 4, 2017 at 7:36 AM, David Burns  wrote:

I don't think anyone would disagree with the reasons for doing this. I,

like James who brought it up earlier, am concerned that we from the emails
appear to think that implementing the wire protocol would be sufficient to
making sure we have the same semantics.

LOL, give us a little credit, okay? The authors of the email do not think
that. We want to have a properly written specification and conformance
tests. I think you're reading "we have no interest in established
standardization processes" when what we wrote was "the process is in very
early stages".

Do you think the Browser Testing Tools WG is the right body to work on a JS
debugging and console protocol, used by interactive developer tools? That
seems like a surprising choice to me.


It is certainly not the only possible venue, but if you want to do the 
work at the W3C then it's probably the easiest way to get things going 
from a Process point of view, since this kind of protocol would be in 
the general remit of the group, and the rechartering could add it 
specifically. Certainly the people currently in the group aren't the 
right ones to do the work, but adding new participants to work 
specifically on this would be trivial.



Also - at least as far as I know -  this is not where the current
participants in the discussion (Kenneth Auchenberg or Christian Bromann, to
name two) have been working. Is having a previously uninvolved standards
committee take up an area in which current activity is occurring elsewhere
considered friendly and cooperative behavior? It seems unfriendly to me. I
would like to avoid upsetting the people I'm hoping to work closely with.


I think you have misinterpreted the intent here. I don't think anyone is 
interested in doing a hostile takeover of existing work. But there is 
concern that the work actually happens. Pointing at remotedebug.org, 
which has been around since 2013 without producing any specification 
materials, isn't helping assuage my concerns, and I guess others are 
having a similar reaction. It is of course entirely possible that 
there's work going on that we can't see. But my interpretation of 
David's email is that he is trying to offer you options, not force you 
down a certain path. The W3C is not always the right venue to work in, 
but it is sometimes sought out by organisations who would likely 
participate in this work because of its relatively strong IPR policy.


I should stress that irrespective of venue I would expect this 
standardisation effort to take years; people always underestimate the 
work and time required for standards work. It will certainly require us 
to commit resources to make it happen.

___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Implementing a Chrome DevTools Protocol server in Firefox

[Jim Blandy]

On Mon, Sep 4, 2017 at 7:36 AM, David Burns  wrote:
> I don't think anyone would disagree with the reasons for doing this. I,
like James who brought it up earlier, am concerned that we from the emails
appear to think that implementing the wire protocol would be sufficient to
making sure we have the same semantics.

LOL, give us a little credit, okay? The authors of the email do not think
that. We want to have a properly written specification and conformance
tests. I think you're reading "we have no interest in established
standardization processes" when what we wrote was "the process is in very
early stages".

Do you think the Browser Testing Tools WG is the right body to work on a JS
debugging and console protocol, used by interactive developer tools? That
seems like a surprising choice to me.

Also - at least as far as I know -  this is not where the current
participants in the discussion (Kenneth Auchenberg or Christian Bromann, to
name two) have been working. Is having a previously uninvolved standards
committee take up an area in which current activity is occurring elsewhere
considered friendly and cooperative behavior? It seems unfriendly to me. I
would like to avoid upsetting the people I'm hoping to work closely with.

I think the people who have actively participating in the work should be
the ones to decide which standards body to collaborate with.
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Implementing a Chrome DevTools Protocol server in Firefox

[David Burns]

I don't think anyone would disagree with the reasons for doing this. I,
like James who brought it up earlier, am concerned that we from the emails
appear to think that implementing the wire protocol would be sufficient to
making sure we have the same semantics.

As mentioned by Karl earlier, it was part of the Browser Testing Tools WG,
but was dropped due to lack of interest from vendors but this thread is now
suggesting that feeling has changed. I am happy, as chair, to see about
adding it back when we recharter, which luckily at the end of September!

I am happy to create a separate thread to get this started.

David

On 31 August 2017 at 21:51, Jim Blandy  wrote:

> Sorry for the premature send. The complete message should read:
>
> The primary goals here are not related to automation and testing.
>
> - We want to migrate the Devtools console and the JS debugger to the CDP,
> to replace an unpopular protocol with a more widely-used one.
>
> - Servo wants its devtools server to use an industry-standard protocol, not
> Firefox's custom thing.
>
> - We'd like to have a devtools server we can share between Gecko and Servo.
>
> - We'd like to move devtools server code out of JS and into a language that
> gives us better control over memory use and performance, because the
> current server, implemented in JS, introduces a lot of noise into
> measurements, affecting the quality of the performance data we're able to
> collect.
>
> - We'd like to share front ends (i.e. clients) between Firefox and Servo.
> devtools.html already implements both the Firefox protocol and the CDP.
>
> Using a protocol that's more familiar to web developers doing automation
> and testing is also good, and we're hoping that will have ancillary
> benefits. For example, it turns out that there is lots of interest in our
> new JS debugger UI, which has hundreds of contributors now. I don't know
> why people want a JS debugger UI, but they do. I believe that Firefox's use
> of a bespoke protocol prevents many similar opportunities for
> collaboration.
>
>
> On Thu, Aug 31, 2017 at 1:36 PM, Jim Blandy  wrote:
>
> > Certain bits of the original post are getting more emphasis than I had
> > anticipated. Let me try to clarify why we in Devtools want this change or
> > something like it.
> >
> > The primary goals here are not related to automation and testing. They
> are:
> >
> >- to allow Devtools to migrate the console and the JS debugger to the
> >CDP;
> >- to start a tools server that can be shared between Gecko and Servo;
> >- to replace Gecko's devtools server, implemented in JS, with one
> >implemented in Rust, to reduce memory consumption and introduce less
> noise
> >into performance and memory measurements
> >
> >
> >
> > and to help us share code with Servo. Our user interfaces already work
> > with the CDP.
> >
> ___
> dev-platform mailing list
> dev-platform@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-platform
>
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Implementing a Chrome DevTools Protocol server in Firefox

[James Graham]

On 31/08/17 21:22, Jack Moffitt wrote:
Is there another alternative besides CDP you'd like to propose? 



I don't have an alternate proposal, and I feel like I must have been 
unclear at some point. I'm not saying "this is bad, period". I'm 
certainly not saying "this is bad because it isn't WebDriver". Given 
that people seem to be in agreement that technically the CDP is good, 
I'm saying "this is bad if it remains a vendor-controlled, partially 
documented, pseudo-standard". The fact that there is apparently interest 
in creating a standard is reassuring, but there doesn't seem to be any 
recent activity on remotedebug.org, so it's hard to tell what the real 
status is, or whether people have understood the amount of work that 
entails. I am slightly worried that there have been several replies 
suggesting that poor interoperability above the message layer won't be a 
big problem because the users will be technical and therefore happy to 
absorb the cost of backwards-incompatible changes between releases. 
Experience from WebDriver is that this isn't true.

___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Implementing a Chrome DevTools Protocol server in Firefox

[Harald Kirschner]

On Thursday, August 31, 2017 at 1:10:30 PM UTC-7, James Graham wrote:
> If people are starting to standardise not just the protocol but also the 
> semantics of CDP, that's great. But people tend to vastly underestimate 
> how long standardisation will take, and overestimate the resources that 
> they will find to work on it. So it would be good to see concrete 
> progress before we are actually shipping.

As mentioned in my last email, CDP already provides a stable core. We did 
review existing tools for API dependencies to understand what the minimum 
viable set should include to be useful for developers. We are still reaching 
out to developers and browser-testing services to capture their feedback, but 
preliminary results are captured in [1]. The list so far suggests that there is 
a shortlist of APIs solving most needs.

[1]: 
https://docs.google.com/document/d/19bimjSpwbiNOPFiEWzBsJ9PtB6cvEaBQCY1Y8UValRk/edit#
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Implementing a Chrome DevTools Protocol server in Firefox

[Jim Blandy]

Certain bits of the original post are getting more emphasis than I had
anticipated. Let me try to clarify why we in Devtools want this change or
something like it.

The primary goals here are not related to automation and testing. They are:

   - to allow Devtools to migrate the console and the JS debugger to the
   CDP;
   - to start a tools server that can be shared between Gecko and Servo;
   - to replace Gecko's devtools server, implemented in JS, with one
   implemented in Rust, to reduce memory consumption and introduce less noise
   into performance and memory measurements



and to help us share code with Servo. Our user interfaces already work with
the CDP.
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Implementing a Chrome DevTools Protocol server in Firefox

[Jack Moffitt]

> I entirely agree that the current Firefox protocol is also proprietary.
> However I also assumed that it's considered an internal implementation
> detail rather than something we would expect people to interoperate with. If
> that wasn't the case then I apologise: I should have complained earlier :)

We would like a console and debugger for Servo, and those seemed
pretty common. We did an initial implementation of the Firefox
protocol, but hat protocol changed out from under us, and some of the
work we did broke and wasn't not easily fixable.

At the very least there is a desire to interoperate between Servo and
Firefox and the current protocol isn't very helpful for this. The
Chrome one has a huge advantage of being documented, even if not all
the semantics are. I was also under the impression that the design of
the Chrome protocol is better than the Firefox one because you can
ingest the JSON messages and only handle the things you want. It seems
like that would make it more robust to future changes.

Is there another alternative besides CDP you'd like to propose? I
think there's a strong case that doing something besides the current
Firefox protocol is a good idea, and the CDP seems to have some nice
qualities. Is there an alternative that is also suitable, or is the
preference that we simultaneously get involved in standardizing CDP
(or at least ensuring that all parties are amenable to doing that
work)?

jack.
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Implementing a Chrome DevTools Protocol server in Firefox

[Jim Blandy]

Google has indicated a willingness to participate in standardizing the
protocol.

If we switch from a devtools protocol used only by us to a tooling protocol
used by the rest of the industry, that is strictly an improvement over the
status quo, even if our implementation deviates from the others' to some
degree: developers whose tooling doesn't run into the points of
incompatibility can now include Firefox in their scripting and tests, and
the maintainers of tooling that use the CDP can work around
incompatibilities.

I think it's a mistake to analogize this too closely to content-exposed
APIs. Those are under extreme backwards compatibility pressure, of a sort
that I think doesn't hold here. Unlike a web user, who simply sees a broken
page and can't do anything about it, our audience is more technical, and
can do things like upgrade tools when things don't work.


On Thu, Aug 31, 2017 at 1:09 PM, James Graham 
wrote:

> On 31/08/17 19:42, Jim Blandy wrote:
>
>> Some possibly missing context: Mozilla Devtools wants to see this
>> implemented for our own use. After much discussion last summer in London,
>> the Firefox Devtools team decided to adopt the Chrome Debugging Protocol
>> for the console and the JavaScript debugger. (The cases for converting the
>> other tools like the Inspector are less compelling.)
>>
>> Speaking as the designer of Firefox's protocol, the CDP is a de-facto
>> standard. The Firefox protocol really has not seen much uptake outside
>> Mozilla, whereas the Chrome Debugging Protocol is implemented with varying
>> degrees of fidelity by several different browsers. "Proprietary" is not the
>> right term here, but in the sense of "used nowhere else", one could argue
>> that it is Mozilla that is using the proprietary protocol, not Chrome. In a
>> real sense, it is more consistent with Mozilla's mission for us to join the
>> rest of the community, implement the CDP for the tools where it makes
>> sense, and participate in its standardization, than to continue to push a
>> protocol nobody else uses.
>>
>
> I entirely agree that the current Firefox protocol is also proprietary.
> However I also assumed that it's considered an internal implementation
> detail rather than something we would expect people to interoperate with.
> If that wasn't the case then I apologise: I should have complained earlier
> :)
>
> Going forward, if we implement a "de-facto" standard that is not actually
> standardised, we are assuming a large risk, in addition to the problems
> around our stated values. An obvious concern is that Google are free to
> change the protocol as they like, including in ways that are intentionally
> or accidentally incompatible with other implementations. We also know from
> past experience of implementing "de-facto" standards that implementation
> differences end up hardcoded into third party consumers (i.e. web pages in
> the case of DOM APIs), making it impossible to get interoperability without
> causing intolerable short-term breakage. This has prevented standardisation
> and compatibility of "de-facto" standards like innerText and
> contentEditable, which remain nominally equivalent but actually very
> different in all browsers.
>
> If people are starting to standardise not just the protocol but also the
> semantics of CDP, that's great. But people tend to vastly underestimate how
> long standardisation will take, and overestimate the resources that they
> will find to work on it. So it would be good to see concrete progress
> before we are actually shipping.
>
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Implementing a Chrome DevTools Protocol server in Firefox

[Harald Kirschner]

On Thursday, August 31, 2017 at 2:51:41 AM UTC-7, James Graham wrote:
> On 31/08/17 02:14, Michael Smith wrote:
> > On 8/30/2017 15:56, David Burns wrote:
> >  > Do we know if the other vendors would see value in having this 
> > spec'ed properly so that we have true interop here? Reverse engineering 
> > seems like a "fun" project but what stops people from breaking stuff 
> > without realising?
> > 
> In general it seems unfortunate if we are deciding to implement a 
> proprietary protocol rather than opting to either extend something that 
> is already a standard (e.g. WebDriver) or perform standardisation work 
> alongside the implementation. What alternatives have we considered here? 
> Is it possible to extend existing standards with missing features? Or 
> are the current tools using the protocol so valuable that we don't have 
> any choice but to support them on their terms? If it's the latter, or we 
> just think the Chrome protocol is so technically superior to the other 
> options that we would be foolish to ignore it, can we work with Google 
> to get it standardised? I think some meaningful attempt at 
> standardisation should be a prerequisite to this kind of protocol 
> implementation shipping in Firefox.

For more history on executing on a de-facto standard; remotedebug.org was 
started as an effort to spec the protocol by representatives from Edge, Chrome 
and Firefox. Chrome committed to keeping the stable parts of the API stable, 
which so far worked. For comparison, WebDriver has gone a similar route, 
experimenting in the open while standardized the stable parts of the API.

To the idea of having one tool like Web Driver solving all remote debug needs: 
Having WebDriver focussed on Automation and not on matching CDP's more advanced 
use cases avoid reinventing the wheel.
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Implementing a Chrome DevTools Protocol server in Firefox

[James Graham]

On 31/08/17 19:42, Jim Blandy wrote:
Some possibly missing context: Mozilla Devtools wants to see this 
implemented for our own use. After much discussion last summer in 
London, the Firefox Devtools team decided to adopt the Chrome Debugging 
Protocol for the console and the JavaScript debugger. (The cases for 
converting the other tools like the Inspector are less compelling.)


Speaking as the designer of Firefox's protocol, the CDP is a de-facto 
standard. The Firefox protocol really has not seen much uptake outside 
Mozilla, whereas the Chrome Debugging Protocol is implemented with 
varying degrees of fidelity by several different browsers. "Proprietary" 
is not the right term here, but in the sense of "used nowhere else", one 
could argue that it is Mozilla that is using the proprietary protocol, 
not Chrome. In a real sense, it is more consistent with Mozilla's 
mission for us to join the rest of the community, implement the CDP for 
the tools where it makes sense, and participate in its standardization, 
than to continue to push a protocol nobody else uses.


I entirely agree that the current Firefox protocol is also proprietary. 
However I also assumed that it's considered an internal implementation 
detail rather than something we would expect people to interoperate 
with. If that wasn't the case then I apologise: I should have complained 
earlier :)


Going forward, if we implement a "de-facto" standard that is not 
actually standardised, we are assuming a large risk, in addition to the 
problems around our stated values. An obvious concern is that Google are 
free to change the protocol as they like, including in ways that are 
intentionally or accidentally incompatible with other implementations. 
We also know from past experience of implementing "de-facto" standards 
that implementation differences end up hardcoded into third party 
consumers (i.e. web pages in the case of DOM APIs), making it impossible 
to get interoperability without causing intolerable short-term breakage. 
This has prevented standardisation and compatibility of "de-facto" 
standards like innerText and contentEditable, which remain nominally 
equivalent but actually very different in all browsers.


If people are starting to standardise not just the protocol but also the 
semantics of CDP, that's great. But people tend to vastly underestimate 
how long standardisation will take, and overestimate the resources that 
they will find to work on it. So it would be good to see concrete 
progress before we are actually shipping.

___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Implementing a Chrome DevTools Protocol server in Firefox

[Jim Blandy]

On Thu, Aug 31, 2017 at 2:50 AM, James Graham 
wrote:

> In general it seems unfortunate if we are deciding to implement a
> proprietary protocol rather than opting to either extend something that is
> already a standard (e.g. WebDriver) or perform standardisation work
> alongside the implementation.


That would be unfortunate, I agree. I hope that's not what's happening
here. By adopting the CDP console and JS debugger protocols I think we are
doing exactly what Mozilla should do: encouraging the adoption and
formalization of de-facto standards.

What alternatives have we considered here?


The Devtools team had extensive discussions a year ago in London about
whether to continue to develop Firefox's own protocols, or whether to go
with what Chrome, Safari, and now Edge are using. We considered the
question on a tool-by-tool basis. Harald Kirschner has started to assemble
a list of the third-party tools that use the CDP.
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Implementing a Chrome DevTools Protocol server in Firefox

[Josh Matthews]

On 8/30/17 6:14 PM, Michael Smith wrote:

On 8/30/2017 18:04, Josh Matthews wrote:
 > One clarification - while all of the Servo codebase exists in 
mozilla-central/servo/, only the dependencies used by Stylo are vendored 
in third-party/rust. That means that neither hyper nor tokio are 
vendored in mozilla-central at the moment, or part of any 
mozilla-central build.


Isn't it vendored into third_party/rust/hyper [4] on mozilla-central?


Ah, it turns out that it's brought in by the geckodriver crate rather 
than Servo.


Cheers,
Josh
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Implementing a Chrome DevTools Protocol server in Firefox

[Jim Blandy]

Some possibly missing context: Mozilla Devtools wants to see this
implemented for our own use. After much discussion last summer in London,
the Firefox Devtools team decided to adopt the Chrome Debugging Protocol
for the console and the JavaScript debugger. (The cases for converting the
other tools like the Inspector are less compelling.)

Speaking as the designer of Firefox's protocol, the CDP is a de-facto
standard. The Firefox protocol really has not seen much uptake outside
Mozilla, whereas the Chrome Debugging Protocol is implemented with varying
degrees of fidelity by several different browsers. "Proprietary" is not the
right term here, but in the sense of "used nowhere else", one could argue
that it is Mozilla that is using the proprietary protocol, not Chrome. In a
real sense, it is more consistent with Mozilla's mission for us to join the
rest of the community, implement the CDP for the tools where it makes
sense, and participate in its standardization, than to continue to push a
protocol nobody else uses.

The devtools.html JavaScript debugger already implements the Chrome
protocol. We've implemented adapters like Valence that implement the
Firefox protocol in terms of the Chrome protocol. So while it's true that
not everything is documented at the standards of a WHATWG specification
(yet), in practical terms, there hasn't been much problem getting things
going.


On Thu, Aug 31, 2017 at 2:50 AM, James Graham 
wrote:

> On 31/08/17 02:14, Michael Smith wrote:
>
>> On 8/30/2017 15:56, David Burns wrote:
>>  > Do we know if the other vendors would see value in having this spec'ed
>> properly so that we have true interop here? Reverse engineering seems like
>> a "fun" project but what stops people from breaking stuff without realising?
>>
>> Fortunately we're not reverse engineering here (for the most part), all
>> protocol messages are specified in a machine-readable JSON format which
>> includes inline documentation [0] --- this is what the cdp Rust library
>> consumes. The spec is versioned and the authors do seem to follow a proper
>> process of introducing new features as "experimental", stabilizing mature
>> ones, and deprecating things before they're removed.
>>
>
> I think that the reverse engineering part is not the wire protocol, which
> is usually the most trivial part, but the associated semantics. It doesn't
> seem that useful to support the protocol unless we behave in the same way
> as Chrome in response to the messages. It's the specification of that
> behaviour which is — as far as I can tell — missing, and which seems likely
> to involve reverse engineering.
>
> In general it seems unfortunate if we are deciding to implement a
> proprietary protocol rather than opting to either extend something that is
> already a standard (e.g. WebDriver) or perform standardisation work
> alongside the implementation. What alternatives have we considered here? Is
> it possible to extend existing standards with missing features? Or are the
> current tools using the protocol so valuable that we don't have any choice
> but to support them on their terms? If it's the latter, or we just think
> the Chrome protocol is so technically superior to the other options that we
> would be foolish to ignore it, can we work with Google to get it
> standardised? I think some meaningful attempt at standardisation should be
> a prerequisite to this kind of protocol implementation shipping in Firefox.
>
> ___
> dev-platform mailing list
> dev-platform@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-platform
>
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Implementing a Chrome DevTools Protocol server in Firefox

[Jim Blandy]

Definitely, yes. As Michael said, some core subset of the protocol is
already a de-facto standard. The process of actual specification, with
committees and processes and imprimaturs and all that, is getting started,
with interest from several major players, including Microsoft and Google.

On Wed, Aug 30, 2017 at 3:56 PM, David Burns  wrote:

> Do we know if the other vendors would see value in having this spec'ed
> properly so that we have true interop here? Reverse engineering  seems like
> a "fun" project but what stops people from breaking stuff without
> realising?
>
> David
>
> On 30 August 2017 at 22:55, Michael Smith  wrote:
>
> > Hi everyone,
> >
> > Mozilla DevTools is exploring implementing parts of the Chrome DevTools
> > Protocol ("CDP") [0] in Firefox. This is an HTTP, WebSockets, and JSON
> > based protocol for automating and inspecting running browser pages.
> >
> > Originally built for the Chrome DevTools, it has seen wider adoption with
> > outside developers. In addition to Chrome/Chromium, the CDP is supported
> by
> > WebKit, Safari, Node.js, and soon Edge, and an ecosystem of libraries and
> > tools already exists which plug into it, for debugging, extracting
> > performance data, providing live-preview functionality like the Brackets
> > editor, and so on. We believe it would be beneficial if these could be
> > leveraged with Firefox as well.
> >
> > The initial implementation we have in mind is an alternate target for
> > third-party integrations to connect to, in addition to the existing
> Firefox
> > DevTools Server. The Servo project has also expressed interest in adding
> > CDP support to improve its own devtools story, and a PR is in flight to
> > land a CDP server implementation there [1].
> >
> > I've been working on this project with guidance from Jim Blandy. We've
> > come up with the following approach:
> >
> > - A complete, typed Rust implementation of the CDP protocol messages and
> > (de)serialization lives in the "cdp" crate [2], automatically generated
> > from the protocol's JSON specification [3] using a build script (this
> > happens transparently as part of the normal Cargo compilation process).
> > This comes with Rustdoc API documentation of all messages/types in the
> > protocol [4] including textual descriptions bundled with the
> specification
> > JSON. The cdp crate will likely track the Chrome stable release for which
> > version of the protocol is supported. A maintainers' script exists which
> > can find and fetch the appropriate JSON [5].
> >
> > - The "tokio-cdp" crate [6] builds on the types and (de)serialization
> > implementation in the cdp crate to provide a server implementation built
> on
> > the Tokio asynchronous I/O system. The server side provides traits for
> > consuming incoming CDP RPC commands, executing them concurrently and
> > sending back responses, and simultaneously pushing events to the client.
> > They are generic over the underlying transport, so the same backend
> > implementation could provide support for "remote" clients plugging in
> over
> > HTTP/WebSockets/JSON or, for example, a browser-local client
> communicating
> > over IPDL.
> >
> > - In Servo, a new component plugs into the cdp and tokio-cdp crates and
> > acts on behalf of connected CDP clients in response to their commands,
> > communicating with the rest of the Servo constellation. This server is
> > disabled by default and can be started by passing a "--cdp" flag to the
> > Servo binary, binding a TCP listener to the loopback interface at the
> > standard CDP port 9222 (a different port can be specified as an option to
> > the flag).
> >
> > - The implementation we envision in Firefox/Gecko would act similarly: a
> > new Rust component, disabled by default and switched on via a command
> line
> > flag, which binds to a local port and mediates between Gecko internals
> and
> > clients connected via tokio-cdp.
> >
> > We chose to build this on Rust and the Tokio event loop, along with the
> > hyper HTTP library and rust-websocket which plug into Tokio.
> >
> > Rust and Cargo provide excellent facilities for compile-time code
> > generation which integrate transparently into the normal build process,
> > avoiding the need to invoke scripts by hand to keep generated artifacts
> in
> > sync. The Rust ecosystem provides libraries such as quote [7] and serde
> [8]
> > which allow us to auto-generate an efficient, typed, and self-contained
> > interface for the entire protocol. This moves the complexity of
> ingesting,
> > validating, and extracting information from client messages out of the
> > Servo- and Gecko-specific backend implementations, helps to ensure they
> > conform correctly to the protocol specification, and provides a
> structured
> > way of upgrading to new protocol versions.
> >
> > As for Tokio, the event loop and Futures-based model of concurrency it
> > offers maps well to the Chrome DevTools Protocol. RPC 

Re: Implementing a Chrome DevTools Protocol server in Firefox

[Ted Mielczarek]

On Wed, Aug 30, 2017, at 08:20 PM, Eric Rescorla wrote:
> I assume this is going to involve TLS (generally this is a requirement
> for
> H2). In Firefox, this is done with NSS. Does Tokio/Hyper cleanly separate
> out the TLS stack so that you can do that?

This was mostly answered in another reply, but just to be clear: yes,
Hyper allows plugging in alternate TLS stacks. This is very commonly
used with the `native-tls` crate[1] by way of `hyper-tls`[2], which uses
the native TLS stack on Windows/macOS, and OpenSSL on Linux.

1. https://github.com/sfackler/rust-native-tls
2. https://github.com/hyperium/hyper-tls
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Implementing a Chrome DevTools Protocol server in Firefox

[James Graham]

On 31/08/17 02:14, Michael Smith wrote:

On 8/30/2017 15:56, David Burns wrote:
 > Do we know if the other vendors would see value in having this 
spec'ed properly so that we have true interop here? Reverse engineering 
seems like a "fun" project but what stops people from breaking stuff 
without realising?


Fortunately we're not reverse engineering here (for the most part), all 
protocol messages are specified in a machine-readable JSON format which 
includes inline documentation [0] --- this is what the cdp Rust library 
consumes. The spec is versioned and the authors do seem to follow a 
proper process of introducing new features as "experimental", 
stabilizing mature ones, and deprecating things before they're removed.


I think that the reverse engineering part is not the wire protocol, 
which is usually the most trivial part, but the associated semantics. It 
doesn't seem that useful to support the protocol unless we behave in the 
same way as Chrome in response to the messages. It's the specification 
of that behaviour which is — as far as I can tell — missing, and which 
seems likely to involve reverse engineering.


In general it seems unfortunate if we are deciding to implement a 
proprietary protocol rather than opting to either extend something that 
is already a standard (e.g. WebDriver) or perform standardisation work 
alongside the implementation. What alternatives have we considered here? 
Is it possible to extend existing standards with missing features? Or 
are the current tools using the protocol so valuable that we don't have 
any choice but to support them on their terms? If it's the latter, or we 
just think the Chrome protocol is so technically superior to the other 
options that we would be foolish to ignore it, can we work with Google 
to get it standardised? I think some meaningful attempt at 
standardisation should be a prerequisite to this kind of protocol 
implementation shipping in Firefox.

___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Implementing a Chrome DevTools Protocol server in Firefox

[Michael Smith]

On 8/30/2017 15:56, David Burns wrote:
> Do we know if the other vendors would see value in having this 
spec'ed properly so that we have true interop here? Reverse engineering  
seems like a "fun" project but what stops people from breaking stuff 
without realising?


Fortunately we're not reverse engineering here (for the most part), all 
protocol messages are specified in a machine-readable JSON format which 
includes inline documentation [0] --- this is what the cdp Rust library 
consumes. The spec is versioned and the authors do seem to follow a 
proper process of introducing new features as "experimental", 
stabilizing mature ones, and deprecating things before they're removed.


There's the Chrome DevTools Protocol viewer which provides a 
human-readable view [1], or if you prefer, the Rustdoc generated from 
the cdp crate [2] which carries over all the same information from the 
specification JSON.


Exposing a typed Rust interface is intended to help us avoid "breaking 
stuff without realizing". Other projects like Node.js have also adopted 
the CDP and an ecosystem of applications have grown around it which 
provide a counterweight against random breakages on Chrome's end.


As far as standardization goes, I'm aware of the RemoteDebug.org [3] 
initiative which has been trying to standardize parts of the CDP as a 
cross-browser remote debugging protocol. From what I recall from 
conversations with others I believe Edge is also interested in this, as 
they're introducing their own CDP implementation/compatibility layer.


On 8/30/2017 17:20, Eric Rescorla wrote:
> I assume this is going to involve TLS (generally this is a 
requirement for H2). In Firefox, this is done with NSS. Does Tokio/Hyper 
cleanly separate out the TLS stack so that you can do that?


The Chrome DevTools Protocol actually only runs over HTTP/1.1 and 
WebSockets, without TLS (binding to a port on a local interface, so 
transport security is less of a concern). My reference to HTTP/2 
concerned the possibility of Servo wanting to support that protocol, 
which would necessitate an upgrade to the future hyper version that will 
support it, which in turn would mean pulling Tokio into Servo.


On 8/30/2017 18:04, Josh Matthews wrote:
> One clarification - while all of the Servo codebase exists in 
mozilla-central/servo/, only the dependencies used by Stylo are vendored 
in third-party/rust. That means that neither hyper nor tokio are 
vendored in mozilla-central at the moment, or part of any 
mozilla-central build.


Isn't it vendored into third_party/rust/hyper [4] on mozilla-central?

-Michael

[0] https://github.com/ChromeDevTools/devtools-protocol/tree/master/json
[1] https://chromedevtools.github.io/devtools-protocol/
[2] https://www.spinda.net/files/mozilla/cdp/doc/cdp/tools/index.html
[3] https://remotedebug.org/
[4] http://searchfox.org/mozilla-central/source/third_party/rust/hyper
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Implementing a Chrome DevTools Protocol server in Firefox

[Josh Matthews]

On 8/30/17 2:55 PM, Michael Smith wrote:
Use of Tokio is becoming a standard in the Rust ecosystem---it's worth 
mentioning that Mozilla funds Tokio development [9] and employs some of 
its primary developers. Servo currently depends on an older version of 
the hyper HTTP client/server library, and consequently this is already 
present in the Firefox tree. The current release of hyper is built on 
top of Tokio, so upgrading hyper, either as maintenance or to take 
advantage of the forthcoming HTTP/2 support, would require pulling in 
Tokio anyway. The current release of rust-websocket, from which Servo 
derives its WebSockets implementation, also supports Tokio.


One clarification - while all of the Servo codebase exists in 
mozilla-central/servo/, only the dependencies used by Stylo are vendored 
in third-party/rust. That means that neither hyper nor tokio are 
vendored in mozilla-central at the moment, or part of any 
mozilla-central build.


Cheers,
Josh
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Implementing a Chrome DevTools Protocol server in Firefox

[Michael Smith]

On 8/30/2017 15:25, Jet Villegas wrote:

Can you summarize the desired outcomes? [...] This is potentially a very large 
API surface to support, and I'm skeptical about our ability to emulate 
Chromium's behavior when attached to devtools.


The Chrome DevTools Protocol is split up into separate commands 
(client-to-server RPC) and events (server-initiated push). These can be 
implemented individually, so we can pick and choose which parts of the 
protocol to tackle and in what order.


Supporting an entire devtools suite as a client (i.e. "use Chrome 
devtools with Firefox") would *not* be an initial target. Instead I've 
been looking toward adding enough support so that some of the 
third-party testing and automation tooling which exists for the CDP can 
be used with Firefox.


Sonar [0], for example, is a web linting tool which uses the CDP. The 
surface area it touches is not all that large [1], consisting of 
straightforward operations like opening and closing tabs, navigating to 
URLs, clearing the cache, and observing page load and network events. 
We've also looked at the WebPageTest agent implementation for Chrome 
[2]: its core functionality also requires navigating between URLs and 
observing page load and network events.


I'm aware of other places where the CDP is used to collect performance 
and profiling data as well as console output, as part of internal 
automated testing. This is testing that isn't being done with Firefox, 
but could be if we supported a limited subset of the protocol. There's a 
common set of functionality here which covers much of the use cases 
developers have in the wild, without having to touch potentially-hairier 
parts such as layout inspection.


*** So, desired outcome #1: be compatible with enough of the Chrome 
DevTools Protocol that we're a better target for testing and automation. 
This complements the recent work in implementing a headless mode.


There are also editors and editor plugins which use the CDP to provide 
HTML/CSS live preview and JS debugging support. I mentioned Brackets in 
my original post, and similar plugins exist for Atom, VS Code, Sublime 
(see [3]). The more developers that adopt these, the more that are 
building web pages with Chrome as the primary target (or Edge, which is 
also adding support). But if Firefox had a CDP implementation which 
supported the subset these tools need, we would suddenly become 
compatible with them. The subset that VS Code's plugin touches [4] looks 
very approachable.


*** Desired outcome #2: be compatible with enough of the Chrome DevTools 
Protocol that we're a better target for web development. This 
complements recent additions to the Firefox DevTools (eg. the CSS Grid 
Inspector) which can be used as points of differentiation once we've 
arrived on the playing field.


On 8/30/2017 15:25, Jet Villegas wrote:

For example, we've been exposing new API's for Layout geometry to the
front-end and devtools, and would like to do more of that sort of tight
integration with our Layout engine.


Implementing parts of the Chrome DevTools Protocol wouldn't mean ripping 
out the Firefox one: we can keep doing tight, Firefox-specific 
integrations with our own devtools.


Where it makes sense, we might replace parts used by the Firefox 
devtools with CDP equivalents---perhaps in conjunction with the 
RemoteDebug.org initiative [5], which is trying to standardize a common 
remote debugging protocol from parts of the CDP. The Console and JS 
Debugger seem like good targets for this, and Debugger.html can already 
communicate with CDP-based servers. But this would, I think, be further 
down the line from the goal of integrating with clients outside the browser.


*** Desired outcome #3: Firefox's CDP server implementation supports 
external (remote) clients, but is designed in such a way that it can be 
applied efficiently to internal use-cases where appropriate. This has 
informed the development direction of the cdp/tokio-cdp Rust libraries.


-Michael

[0] https://sonarwhal.com/
[1] https://github.com/sonarwhal/sonar/issues/377#issuecomment-314929118
[2] 
https://github.com/WPO-Foundation/wptagent/blob/master/internal/devtools.py

[3] https://remotedebug.org/integrations/
[4] 
https://github.com/Microsoft/vscode-chrome-debug/blob/c77829dad40c8f2f783a0ae23d61ef79cbbc/test/debugProtocolMocks.ts#L24-L77

[5] https://remotedebug.org/

On 8/30/2017 15:25, Jet Villegas wrote:

Can you summarize the desired outcomes?
e.g.,
1. people using devtools in Chrome can also debug Firefox
2. devtools for Chrome can be ported to Firefox
3. devtools for Firefox can be used with Chrome
4. ...

This is potentially a very large API surface to support, and I'm skeptical
about our ability to emulate Chromium's behavior when attached to devtools.
For example, we've been exposing new API's for Layout geometry to the
front-end and devtools, and would like to do more of that sort of tight
integration with our Layout engine. I'm not 

Re: Implementing a Chrome DevTools Protocol server in Firefox

[Karl Dubost]

Michael,

Le 31 août 2017 à 07:56, David Burns  a écrit :
> Do we know if the other vendors would see value in having this spec'ed
> properly so that we have true interop here? 

Yeah I had the same train of thoughts than David when I read the initial 
message.

There is currently Browser Testing and Tools Working Group Charter which does 
only Web Driver API. David is a co-chair there.
https://www.w3.org/2016/05/browser-testing-tools-charter.html

The 2011 charter was open on more APIs
https://www.w3.org/2011/08/browser-testing-charter.html


But I had a slight memory of a long time ago attempt, which indeed didn't take 
off. 
There was this talk at XTech 2008 by Mike Smith (W3C)
https://www.w3.org/2008/Talks/05-07-smith-xtech/slides.pdf

Also this talk at the 2008 W3C Technical Plenary 
by Chaals (Opera then) 
on standardizing Scope (Opera Dragonfly Inspection API) 
"Standards API and Debuggers"
https://www.w3.org/2008/10/1022-dragonfly-chaals/talk.html
https://www.w3.org/2008/10/22-tp-minutes.html
http://operasoftware.github.io/scope-interface/


But I don't remember if there was a workshop or anything and it just didn't 
build up steam.


-- 
Karl Dubost, mozilla  Webcompat
http://www.la-grange.net/karl/moz





___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Implementing a Chrome DevTools Protocol server in Firefox

[Eric Rescorla]

On Wed, Aug 30, 2017 at 3:55 PM, Michael Smith  wrote:

> Hi everyone,
>
> Mozilla DevTools is exploring implementing parts of the Chrome DevTools
> Protocol ("CDP") [0] in Firefox. This is an HTTP, WebSockets, and JSON
> based protocol for automating and inspecting running browser pages.
>
> Originally built for the Chrome DevTools, it has seen wider adoption with
> outside developers. In addition to Chrome/Chromium, the CDP is supported by
> WebKit, Safari, Node.js, and soon Edge, and an ecosystem of libraries and
> tools already exists which plug into it, for debugging, extracting
> performance data, providing live-preview functionality like the Brackets
> editor, and so on. We believe it would be beneficial if these could be
> leveraged with Firefox as well.
>
> The initial implementation we have in mind is an alternate target for
> third-party integrations to connect to, in addition to the existing Firefox
> DevTools Server. The Servo project has also expressed interest in adding
> CDP support to improve its own devtools story, and a PR is in flight to
> land a CDP server implementation there [1].
>
> I've been working on this project with guidance from Jim Blandy. We've
> come up with the following approach:
>
> - A complete, typed Rust implementation of the CDP protocol messages and
> (de)serialization lives in the "cdp" crate [2], automatically generated
> from the protocol's JSON specification [3] using a build script (this
> happens transparently as part of the normal Cargo compilation process).
> This comes with Rustdoc API documentation of all messages/types in the
> protocol [4] including textual descriptions bundled with the specification
> JSON. The cdp crate will likely track the Chrome stable release for which
> version of the protocol is supported. A maintainers' script exists which
> can find and fetch the appropriate JSON [5].
>
> - The "tokio-cdp" crate [6] builds on the types and (de)serialization
> implementation in the cdp crate to provide a server implementation built on
> the Tokio asynchronous I/O system. The server side provides traits for
> consuming incoming CDP RPC commands, executing them concurrently and
> sending back responses, and simultaneously pushing events to the client.
> They are generic over the underlying transport, so the same backend
> implementation could provide support for "remote" clients plugging in over
> HTTP/WebSockets/JSON or, for example, a browser-local client communicating
> over IPDL.
>
> - In Servo, a new component plugs into the cdp and tokio-cdp crates and
> acts on behalf of connected CDP clients in response to their commands,
> communicating with the rest of the Servo constellation. This server is
> disabled by default and can be started by passing a "--cdp" flag to the
> Servo binary, binding a TCP listener to the loopback interface at the
> standard CDP port 9222 (a different port can be specified as an option to
> the flag).
>
> - The implementation we envision in Firefox/Gecko would act similarly: a
> new Rust component, disabled by default and switched on via a command line
> flag, which binds to a local port and mediates between Gecko internals and
> clients connected via tokio-cdp.
>
> We chose to build this on Rust and the Tokio event loop, along with the
> hyper HTTP library and rust-websocket which plug into Tokio.
>
> Rust and Cargo provide excellent facilities for compile-time code
> generation which integrate transparently into the normal build process,
> avoiding the need to invoke scripts by hand to keep generated artifacts in
> sync. The Rust ecosystem provides libraries such as quote [7] and serde [8]
> which allow us to auto-generate an efficient, typed, and self-contained
> interface for the entire protocol. This moves the complexity of ingesting,
> validating, and extracting information from client messages out of the
> Servo- and Gecko-specific backend implementations, helps to ensure they
> conform correctly to the protocol specification, and provides a structured
> way of upgrading to new protocol versions.
>
> As for Tokio, the event loop and Futures-based model of concurrency it
> offers maps well to the Chrome DevTools Protocol. RPC commands typically
> execute simultaneously, returning responses in order of completion, while
> the server continuously generates events to which the client has
> subscribed. Under Tokio we can spawn multiple lightweight Tasks, dispatch
> messages to them, and multiplex their responses back over the single client
> connection. The Tokio event loop is nicely self-contained to the one or,
> optionally, more threads it is allocated, so the rest of the application
> doesn't need to be aware of it.
>
> Use of Tokio is becoming a standard in the Rust ecosystem---it's worth
> mentioning that Mozilla funds Tokio development [9] and employs some of its
> primary developers. Servo currently depends on an older version of the
> hyper HTTP client/server library, and consequently 

Re: Implementing a Chrome DevTools Protocol server in Firefox

[David Burns]

Do we know if the other vendors would see value in having this spec'ed
properly so that we have true interop here? Reverse engineering  seems like
a "fun" project but what stops people from breaking stuff without realising?

David

On 30 August 2017 at 22:55, Michael Smith  wrote:

> Hi everyone,
>
> Mozilla DevTools is exploring implementing parts of the Chrome DevTools
> Protocol ("CDP") [0] in Firefox. This is an HTTP, WebSockets, and JSON
> based protocol for automating and inspecting running browser pages.
>
> Originally built for the Chrome DevTools, it has seen wider adoption with
> outside developers. In addition to Chrome/Chromium, the CDP is supported by
> WebKit, Safari, Node.js, and soon Edge, and an ecosystem of libraries and
> tools already exists which plug into it, for debugging, extracting
> performance data, providing live-preview functionality like the Brackets
> editor, and so on. We believe it would be beneficial if these could be
> leveraged with Firefox as well.
>
> The initial implementation we have in mind is an alternate target for
> third-party integrations to connect to, in addition to the existing Firefox
> DevTools Server. The Servo project has also expressed interest in adding
> CDP support to improve its own devtools story, and a PR is in flight to
> land a CDP server implementation there [1].
>
> I've been working on this project with guidance from Jim Blandy. We've
> come up with the following approach:
>
> - A complete, typed Rust implementation of the CDP protocol messages and
> (de)serialization lives in the "cdp" crate [2], automatically generated
> from the protocol's JSON specification [3] using a build script (this
> happens transparently as part of the normal Cargo compilation process).
> This comes with Rustdoc API documentation of all messages/types in the
> protocol [4] including textual descriptions bundled with the specification
> JSON. The cdp crate will likely track the Chrome stable release for which
> version of the protocol is supported. A maintainers' script exists which
> can find and fetch the appropriate JSON [5].
>
> - The "tokio-cdp" crate [6] builds on the types and (de)serialization
> implementation in the cdp crate to provide a server implementation built on
> the Tokio asynchronous I/O system. The server side provides traits for
> consuming incoming CDP RPC commands, executing them concurrently and
> sending back responses, and simultaneously pushing events to the client.
> They are generic over the underlying transport, so the same backend
> implementation could provide support for "remote" clients plugging in over
> HTTP/WebSockets/JSON or, for example, a browser-local client communicating
> over IPDL.
>
> - In Servo, a new component plugs into the cdp and tokio-cdp crates and
> acts on behalf of connected CDP clients in response to their commands,
> communicating with the rest of the Servo constellation. This server is
> disabled by default and can be started by passing a "--cdp" flag to the
> Servo binary, binding a TCP listener to the loopback interface at the
> standard CDP port 9222 (a different port can be specified as an option to
> the flag).
>
> - The implementation we envision in Firefox/Gecko would act similarly: a
> new Rust component, disabled by default and switched on via a command line
> flag, which binds to a local port and mediates between Gecko internals and
> clients connected via tokio-cdp.
>
> We chose to build this on Rust and the Tokio event loop, along with the
> hyper HTTP library and rust-websocket which plug into Tokio.
>
> Rust and Cargo provide excellent facilities for compile-time code
> generation which integrate transparently into the normal build process,
> avoiding the need to invoke scripts by hand to keep generated artifacts in
> sync. The Rust ecosystem provides libraries such as quote [7] and serde [8]
> which allow us to auto-generate an efficient, typed, and self-contained
> interface for the entire protocol. This moves the complexity of ingesting,
> validating, and extracting information from client messages out of the
> Servo- and Gecko-specific backend implementations, helps to ensure they
> conform correctly to the protocol specification, and provides a structured
> way of upgrading to new protocol versions.
>
> As for Tokio, the event loop and Futures-based model of concurrency it
> offers maps well to the Chrome DevTools Protocol. RPC commands typically
> execute simultaneously, returning responses in order of completion, while
> the server continuously generates events to which the client has
> subscribed. Under Tokio we can spawn multiple lightweight Tasks, dispatch
> messages to them, and multiplex their responses back over the single client
> connection. The Tokio event loop is nicely self-contained to the one or,
> optionally, more threads it is allocated, so the rest of the application
> doesn't need to be aware of it.
>
> Use of Tokio is becoming a standard in the Rust 

Re: Implementing a Chrome DevTools Protocol server in Firefox

[Jet Villegas]

Can you summarize the desired outcomes?
e.g.,
1. people using devtools in Chrome can also debug Firefox
2. devtools for Chrome can be ported to Firefox
3. devtools for Firefox can be used with Chrome
4. ...

This is potentially a very large API surface to support, and I'm skeptical
about our ability to emulate Chromium's behavior when attached to devtools.
For example, we've been exposing new API's for Layout geometry to the
front-end and devtools, and would like to do more of that sort of tight
integration with our Layout engine. I'm not sure we could guarantee bug
compatibility when a Chrome devtool requires similar-but-not-quite
functionality. In other words, it sounds like your proposed protocol
emulation is feasible, but I'm less certain we can live up to the expected
semantics of things going over that wire.

--Jet





On Wed, Aug 30, 2017 at 2:55 PM, Michael Smith  wrote:

> Hi everyone,
>
> Mozilla DevTools is exploring implementing parts of the Chrome DevTools
> Protocol ("CDP") [0] in Firefox. This is an HTTP, WebSockets, and JSON
> based protocol for automating and inspecting running browser pages.
>
> Originally built for the Chrome DevTools, it has seen wider adoption with
> outside developers. In addition to Chrome/Chromium, the CDP is supported by
> WebKit, Safari, Node.js, and soon Edge, and an ecosystem of libraries and
> tools already exists which plug into it, for debugging, extracting
> performance data, providing live-preview functionality like the Brackets
> editor, and so on. We believe it would be beneficial if these could be
> leveraged with Firefox as well.
>
> The initial implementation we have in mind is an alternate target for
> third-party integrations to connect to, in addition to the existing Firefox
> DevTools Server. The Servo project has also expressed interest in adding
> CDP support to improve its own devtools story, and a PR is in flight to
> land a CDP server implementation there [1].
>
> I've been working on this project with guidance from Jim Blandy. We've
> come up with the following approach:
>
> - A complete, typed Rust implementation of the CDP protocol messages and
> (de)serialization lives in the "cdp" crate [2], automatically generated
> from the protocol's JSON specification [3] using a build script (this
> happens transparently as part of the normal Cargo compilation process).
> This comes with Rustdoc API documentation of all messages/types in the
> protocol [4] including textual descriptions bundled with the specification
> JSON. The cdp crate will likely track the Chrome stable release for which
> version of the protocol is supported. A maintainers' script exists which
> can find and fetch the appropriate JSON [5].
>
> - The "tokio-cdp" crate [6] builds on the types and (de)serialization
> implementation in the cdp crate to provide a server implementation built on
> the Tokio asynchronous I/O system. The server side provides traits for
> consuming incoming CDP RPC commands, executing them concurrently and
> sending back responses, and simultaneously pushing events to the client.
> They are generic over the underlying transport, so the same backend
> implementation could provide support for "remote" clients plugging in over
> HTTP/WebSockets/JSON or, for example, a browser-local client communicating
> over IPDL.
>
> - In Servo, a new component plugs into the cdp and tokio-cdp crates and
> acts on behalf of connected CDP clients in response to their commands,
> communicating with the rest of the Servo constellation. This server is
> disabled by default and can be started by passing a "--cdp" flag to the
> Servo binary, binding a TCP listener to the loopback interface at the
> standard CDP port 9222 (a different port can be specified as an option to
> the flag).
>
> - The implementation we envision in Firefox/Gecko would act similarly: a
> new Rust component, disabled by default and switched on via a command line
> flag, which binds to a local port and mediates between Gecko internals and
> clients connected via tokio-cdp.
>
> We chose to build this on Rust and the Tokio event loop, along with the
> hyper HTTP library and rust-websocket which plug into Tokio.
>
> Rust and Cargo provide excellent facilities for compile-time code
> generation which integrate transparently into the normal build process,
> avoiding the need to invoke scripts by hand to keep generated artifacts in
> sync. The Rust ecosystem provides libraries such as quote [7] and serde [8]
> which allow us to auto-generate an efficient, typed, and self-contained
> interface for the entire protocol. This moves the complexity of ingesting,
> validating, and extracting information from client messages out of the
> Servo- and Gecko-specific backend implementations, helps to ensure they
> conform correctly to the protocol specification, and provides a structured
> way of upgrading to new protocol versions.
>
> As for Tokio, the event loop and Futures-based model of concurrency it

Implementing a Chrome DevTools Protocol server in Firefox

[Michael Smith]

Hi everyone,

Mozilla DevTools is exploring implementing parts of the Chrome DevTools 
Protocol ("CDP") [0] in Firefox. This is an HTTP, WebSockets, and JSON 
based protocol for automating and inspecting running browser pages.


Originally built for the Chrome DevTools, it has seen wider adoption 
with outside developers. In addition to Chrome/Chromium, the CDP is 
supported by WebKit, Safari, Node.js, and soon Edge, and an ecosystem of 
libraries and tools already exists which plug into it, for debugging, 
extracting performance data, providing live-preview functionality like 
the Brackets editor, and so on. We believe it would be beneficial if 
these could be leveraged with Firefox as well.


The initial implementation we have in mind is an alternate target for 
third-party integrations to connect to, in addition to the existing 
Firefox DevTools Server. The Servo project has also expressed interest 
in adding CDP support to improve its own devtools story, and a PR is in 
flight to land a CDP server implementation there [1].


I've been working on this project with guidance from Jim Blandy. We've 
come up with the following approach:


- A complete, typed Rust implementation of the CDP protocol messages and 
(de)serialization lives in the "cdp" crate [2], automatically generated 
from the protocol's JSON specification [3] using a build script (this 
happens transparently as part of the normal Cargo compilation process). 
This comes with Rustdoc API documentation of all messages/types in the 
protocol [4] including textual descriptions bundled with the 
specification JSON. The cdp crate will likely track the Chrome stable 
release for which version of the protocol is supported. A maintainers' 
script exists which can find and fetch the appropriate JSON [5].


- The "tokio-cdp" crate [6] builds on the types and (de)serialization 
implementation in the cdp crate to provide a server implementation built 
on the Tokio asynchronous I/O system. The server side provides traits 
for consuming incoming CDP RPC commands, executing them concurrently and 
sending back responses, and simultaneously pushing events to the client. 
They are generic over the underlying transport, so the same backend 
implementation could provide support for "remote" clients plugging in 
over HTTP/WebSockets/JSON or, for example, a browser-local client 
communicating over IPDL.


- In Servo, a new component plugs into the cdp and tokio-cdp crates and 
acts on behalf of connected CDP clients in response to their commands, 
communicating with the rest of the Servo constellation. This server is 
disabled by default and can be started by passing a "--cdp" flag to the 
Servo binary, binding a TCP listener to the loopback interface at the 
standard CDP port 9222 (a different port can be specified as an option 
to the flag).


- The implementation we envision in Firefox/Gecko would act similarly: a 
new Rust component, disabled by default and switched on via a command 
line flag, which binds to a local port and mediates between Gecko 
internals and clients connected via tokio-cdp.


We chose to build this on Rust and the Tokio event loop, along with the 
hyper HTTP library and rust-websocket which plug into Tokio.


Rust and Cargo provide excellent facilities for compile-time code 
generation which integrate transparently into the normal build process, 
avoiding the need to invoke scripts by hand to keep generated artifacts 
in sync. The Rust ecosystem provides libraries such as quote [7] and 
serde [8] which allow us to auto-generate an efficient, typed, and 
self-contained interface for the entire protocol. This moves the 
complexity of ingesting, validating, and extracting information from 
client messages out of the Servo- and Gecko-specific backend 
implementations, helps to ensure they conform correctly to the protocol 
specification, and provides a structured way of upgrading to new 
protocol versions.


As for Tokio, the event loop and Futures-based model of concurrency it 
offers maps well to the Chrome DevTools Protocol. RPC commands typically 
execute simultaneously, returning responses in order of completion, 
while the server continuously generates events to which the client has 
subscribed. Under Tokio we can spawn multiple lightweight Tasks, 
dispatch messages to them, and multiplex their responses back over the 
single client connection. The Tokio event loop is nicely self-contained 
to the one or, optionally, more threads it is allocated, so the rest of 
the application doesn't need to be aware of it.


Use of Tokio is becoming a standard in the Rust ecosystem---it's worth 
mentioning that Mozilla funds Tokio development [9] and employs some of 
its primary developers. Servo currently depends on an older version of 
the hyper HTTP client/server library, and consequently this is already 
present in the Firefox tree. The current release of hyper is built on 
top of Tokio, so upgrading hyper, either as maintenance or to take