Re: [cmake-developers] cmake daemon mode protocol

[Tobias Hunger]

Hi Stephen,

I think what I suggested so far is covered by the first way of
interaction covered in
http://thread.gmane.org/gmane.comp.lib.qt.creator/11794/focus=15411 .

I am just proposing something that is convenient for me to use.

On Sat, Jan 16, 2016 at 12:26 PM, Stephen Kelly  wrote:
>>> For this case, I suggest that if the user tries to 'open the source
>>> directory', you would use QTemporaryDir to build in a temporary location
>> and
>>> run the daemon there. I believe clion does something equivalent to that.
>>> Is that viable? I suppose you are suggesting that cmake do that instead
>>> of leaving it for clients to do?
>>
>> It is doable. I just do not see why this should be necessary.
>
> I'm aiming to first design as much as possible of a 'minimal viable
> protocol' as a first step. Given the above I think your ideas of running the
> daemon with no build dir are not 'minimal' and I'm convinced they're not
> viable.

I am thinking about the simplest possible use-case I can think of
right now: User gets sources from a project somewhere and wants to
build those.

What is entirely unclear to me with your current implementation is how
do I get from "sources are checked out in $DIR" to "I have a $BUILD
that I can use to start the daemon"?

Then I also do not see how I can support a user switching between two
build directories. Do I need to close down one daemon and start
another?

> Someone would have to prototype the ideas, but I don't think that's a good
> use of time or energy right now (mostly because I think they're not viable).
>
> I recommend focussing on the tasks in my OP:
>
>  http://thread.gmane.org/gmane.comp.lib.qt.creator/11794

That list is so full of cmake internals that it is basically
meaningless to me. Sorry.

>>> Richer information about some semantics like 'task' and 'busy state'
>>> could also be provided in a similar way I expect, assuming those could be
>> defined.
>>
>> This is some basic functionality that we should get right soon, as this
>> can influence how long-running tasks need to be designed.
>
> Yes. Perhaps you can expand on what 'tasks' and 'busy state' you have in
> mind. Particularly if you can relate them to what is already in my github
> branch.

"task" is just some Id to identify what the daemon is currently doing.
"state" is just some indication of whether the daemon thinks it is
busy with something or not. In "idle" state it should accept more
requests, in "busy" state it will just queue them or maybe even ignore
them.

>> So "cmake --build" is out of scope for the daemon?
>
> I listed three ways cmake and an IDE could interface here:
>
>  http://thread.gmane.org/gmane.comp.lib.qt.creator/11794/focus=15411
>
> 'triggering the build' would be a fourth. I don't think it needs to be part
> of the initial discussion of the design.
>
> Let's try to keep scope small so that we can get on common ground.

So let's forget about all the interactive stuff for now. That is IMHO
step 2, after the basic protocol is validated by implementing bare
bones project information:-)

>> That is why I brought up progress information so early: They make things
>> complicated by there suddenly being several responses to a request and you
>> need a way to identify those. At that point simpler solutions tend to blow
>> up:-)
>
> Yes.
>
> BTW: I don't expect to get any part of the design of the protocol 'right' on
> the first iteration. I think we need to start everything that will need to
> be in the protocol, then iterate. Start high level ignoring details, and
> fill in the details as we iterate and can encode needs in unit tests.

I agree and do not expect anything different.

Basic protocol functionality from my point of view needs to include:
* Being able to detect the daemon or IDE becoming unresponsive
* Being able to find which responses are the result of which requests
* Being able to reliably find what kind of object I got as request or response

>From the point of view of an IDE: Being able to have progress
information is not a high level detail to be filled in later.

Best Regards,
Tobias
-- 

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: 
http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more 
information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at 
http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
http://public.kitware.com/mailman/listinfo/cmake-developers

Re: [cmake-developers] cmake daemon mode protocol

[Tobias Hunger]

Hi Stephen,

Am 15.01.2016 00:38 schrieb "Stephen Kelly" :
> > * Start daemon-mode without any additional parameters
> >
> > * Daemon responds with "
> >   {
> > "type": "handShake",
> > "version": "3.5.x",
> > "supportedProtocols": "3.5"
> >   }
>
> As you discovered, the cmake binary might not even support the daemon mode
> at all.

That is really straight forward to detect: It will send malformed JSON on
stdout if daemon-mode is unsupported:-) The error message you get is not
JSON and definitely on of type handShake.

> I think what you are really asking for is 'the capabilities of the cmake
> binary'. In that sense, you might also want to know what generators are
> available, whether daemon mode is available (and with which protocol
> versions) etc.
>
> The issue at
>
>  http://public.kitware.com/Bug/view.php?id=15462
>
> proposes to dump that information to stdout with a different -E command.

I do not really see the need for that.

Daemon-mode is easy to detect by the reply to the attempt to start it.

Everything else I expect to extract from within daemon mode.

> >
> > * Client Handshake:
> >   {
> > "type": "handshake",
> > "requestProtocolVersion": "3.5"
> >   }
> >
> > * Daemon:
> >   {
> > "type:"handshake",
> > "protocolVersion": "3.5"
> >   }
>
> Something like this is possible too. It's not far off what is already done
> in the branch.

I know, that is what this is based on after all. All I want is to be able
to start daemon-mode without having a builddir that is already associated
with a sourcedir.

> > At this point the normal operation should be possible.
> >
> > * Client:
> >   {
> >  "type": "setProjectInformation",
> >  "sourceDirectory": "/some/path",
> >  "cookie": "1"
> >   }
> >
> >   Being able to set the source directory is important so that a fresh
> > checkout can be configured. There is no build directory available at
> > that time...
>
> I understand the desire to inspect a cmake project without asking the user
> to specify a build dir,

You can safely read the CMakelists at this point, give syntax errors, maybe
even extract basic information like project name at this point.

> but cmake has to write build files somewhere in
> order to test the compiler etc.

If this works with a temporary build directory, then these files probably
do not need to litter the build directory in the first place:-)

> For this case, I suggest that if the user tries to 'open the source
> directory', you would use QTemporaryDir to build in a temporary location
and
> run the daemon there. I believe clion does something equivalent to that.

Hey, the idea is to improve on the current situation, not to perpetuate the
workarounds.

And how do I associate an empty directory as a builddir to a sourcedir?

How can I switch between builds when the daemon-mode is set up to work with
one builddir? Do I have to close it down and start a new daemon for the
next build?

That would be suboptimal, as all the built-in dependent state of the
project would need to get regenerated by the new daemon. Associating a
daemon with a sourcedir and being able to switch builddir would be nicer.

> Is that viable? I suppose you are suggesting that cmake do that instead of
> leaving it for clients to do?

It is doable. I just do not see why this should be necessary.

The daemon is not need to persist its state to disk all the time like
command-based cmake currently needs to.

> > * Daemon should do progress handling while reading the CMakeLists.txt
> > or whatever it does:
> >   { "type": "progress", "state": "busy", "task": "someId",
> > "taskDescription": "Reading CMakeLists.txt files", "cookie": "1" }
> >
> > * Daemon sends occasional:
> >   { "type": "progress", "task": "someId", progressTotal: 25,
> > progressCurrent: 12, "cookie": "1" }
>
> Yes. CMake can provide approximate values of how complete the configure
step
> is. See cmGlobalGenerator::AddMakefile. That could be exposed to the
daemon
> 'somehow', such as by defining some virtual callback interface.
>
> Richer information about some semantics like 'task' and 'busy state' could
> also be provided in a similar way I expect, assuming those could be
defined.

This is some basic functionality that we should get right soon, as this can
influence how long-running tasks need to be designed.

> > * Finally Daemon responds sends data as that is available now.
> >   {
> > "type": "projectInformation",
> > "sourceDirectory": "/some/path",
> > "buildDirectory": "",
> > "projectName": "SomeProject",
> > "cookie": "1"
> >   }
>
> At this point in your proposal, cmake has run the configure step in a
> temporary directory, right?

I am currently arguing with complete ignorance of how cmake works
internally:-) This is just what I think makes the most sense from an IDE
perspective.

> So it would be
>
>   "buildDirectory": "/tmp/foo/bar"
>
> right?

No.

The daemon should be associated with a sourcedir, 

Re: [cmake-developers] cmake daemon mode protocol

[Stephen Kelly]

Stephen Kelly wrote:
> I recommend focussing on the tasks in my OP:
> 
>  http://thread.gmane.org/gmane.comp.lib.qt.creator/11794

To be more clear:

The goal I have is to enable debugging, introspection of the buildsystem and 
the state during execution, code completion etc.

Your goal seems to be completely orthogonal to that and requires an entirely 
different set of steps to achieve (perhaps the first being 'demonstrate 
viability'). I don't think I can help with your goal.

Thanks,

Steve.


-- 

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: 
http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more 
information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at 
http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
http://public.kitware.com/mailman/listinfo/cmake-developers

Re: [cmake-developers] cmake daemon mode protocol

[Stephen Kelly]

Tobias Hunger wrote:

>> For this case, I suggest that if the user tries to 'open the source
>> directory', you would use QTemporaryDir to build in a temporary location
> and
>> run the daemon there. I believe clion does something equivalent to that.
>> Is that viable? I suppose you are suggesting that cmake do that instead
>> of leaving it for clients to do?
> 
> It is doable. I just do not see why this should be necessary.

I'm aiming to first design as much as possible of a 'minimal viable 
protocol' as a first step. Given the above I think your ideas of running the 
daemon with no build dir are not 'minimal' and I'm convinced they're not 
viable. 

Someone would have to prototype the ideas, but I don't think that's a good 
use of time or energy right now (mostly because I think they're not viable).

I recommend focussing on the tasks in my OP:

 http://thread.gmane.org/gmane.comp.lib.qt.creator/11794

>> Richer information about some semantics like 'task' and 'busy state'
>> could also be provided in a similar way I expect, assuming those could be
> defined.
> 
> This is some basic functionality that we should get right soon, as this
> can influence how long-running tasks need to be designed.

Yes. Perhaps you can expand on what 'tasks' and 'busy state' you have in 
mind. Particularly if you can relate them to what is already in my github 
branch.

> So "cmake --build" is out of scope for the daemon?

I listed three ways cmake and an IDE could interface here:

 http://thread.gmane.org/gmane.comp.lib.qt.creator/11794/focus=15411

'triggering the build' would be a fourth. I don't think it needs to be part 
of the initial discussion of the design. 

Let's try to keep scope small so that we can get on common ground.

> That is why I brought up progress information so early: They make things
> complicated by there suddenly being several responses to a request and you
> need a way to identify those. At that point simpler solutions tend to blow
> up:-)

Yes.

BTW: I don't expect to get any part of the design of the protocol 'right' on 
the first iteration. I think we need to start everything that will need to 
be in the protocol, then iterate. Start high level ignoring details, and 
fill in the details as we iterate and can encode needs in unit tests.

Thanks,

Steve.


-- 

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: 
http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more 
information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at 
http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
http://public.kitware.com/mailman/listinfo/cmake-developers

Re: [cmake-developers] cmake daemon mode protocol

[Stephen Kelly]

Tobias Hunger wrote:

> Hi Stephen,
> 
> I have successfully build and run your cmake server mode changes and
> the python client script does work as advertised.

Thanks for doing that!

> I do have a couple of remarks about it. This is more intended as a
> starting point for discussion as a real proposal. Would something
> along these lines be possible:


> * Start daemon-mode without any additional parameters
> 
> * Daemon responds with "
>   {
> "type": "handShake",
> "version": "3.5.x",
> "supportedProtocols": "3.5"
>   }

As you discovered, the cmake binary might not even support the daemon mode 
at all.

I think what you are really asking for is 'the capabilities of the cmake 
binary'. In that sense, you might also want to know what generators are 
available, whether daemon mode is available (and with which protocol 
versions) etc.

The issue at

 http://public.kitware.com/Bug/view.php?id=15462

proposes to dump that information to stdout with a different -E command.

> 
> * Client Handshake:
>   {
> "type": "handshake",
> "requestProtocolVersion": "3.5"
>   }
> 
> * Daemon:
>   {
> "type:"handshake",
> "protocolVersion": "3.5"
>   }

Something like this is possible too. It's not far off what is already done 
in the branch.

> 
> At this point the normal operation should be possible.
> 
> * Client:
>   {
>  "type": "setProjectInformation",
>  "sourceDirectory": "/some/path",
>  "cookie": "1"
>   }
> 
>   Being able to set the source directory is important so that a fresh
> checkout can be configured. There is no build directory available at
> that time...

I understand the desire to inspect a cmake project without asking the user 
to specify a build dir, but cmake has to write build files somewhere in 
order to test the compiler etc. 

For this case, I suggest that if the user tries to 'open the source 
directory', you would use QTemporaryDir to build in a temporary location and 
run the daemon there. I believe clion does something equivalent to that. 

Is that viable? I suppose you are suggesting that cmake do that instead of 
leaving it for clients to do?

> * Daemon should do progress handling while reading the CMakeLists.txt
> or whatever it does:
>   { "type": "progress", "state": "busy", "task": "someId",
> "taskDescription": "Reading CMakeLists.txt files", "cookie": "1" }
> 
> * Daemon sends occasional:
>   { "type": "progress", "task": "someId", progressTotal: 25,
> progressCurrent: 12, "cookie": "1" }

Yes. CMake can provide approximate values of how complete the configure step 
is. See cmGlobalGenerator::AddMakefile. That could be exposed to the daemon 
'somehow', such as by defining some virtual callback interface. 

Richer information about some semantics like 'task' and 'busy state' could 
also be provided in a similar way I expect, assuming those could be defined.

> * Finally Daemon responds sends data as that is available now.
>   {
> "type": "projectInformation",
> "sourceDirectory": "/some/path",
> "buildDirectory": "",
> "projectName": "SomeProject",
> "cookie": "1"
>   }

At this point in your proposal, cmake has run the configure step in a 
temporary directory, right? So it would be 

  "buildDirectory": "/tmp/foo/bar"

right?

> * Client:
>   {
> "type": "setProjectInformation",
> "buildDirectory": "/some/other/path",
> "cookie": "2"
>   }

As this sets a new build directory, cmake will have to run the configure 
step from scratch in this new location.

> * Daemon does progress indication as above...
> 
> * Daemon responds with project info:
>   {
> "type": "progressInformation",
> "sourceDirectory": "/some/path",
> "buildDirectory": "/some/other/path",
> "projectName": "SomeProject",
> "cookie": "2"
>   }

I must admit I'm not seduced by this idea as it seems to be that it should 
be easy for an IDE to build in a temporary dir itself. 

However, I think it makes sense to design the protocol to handle specifying 
either a sourceDir or a buildDir (or both if from scratch). Starting daemon 
mode without arguments as you suggest could work (though we would have to 
see how that affects the protocol). Or with a different argument (eg

 cmake -E daemon buildDir
 cmake -E daemon_source sourceDir

) could work. I am also just brainstorming ideas, not proscribing anything.

> We might also want a "ping"/"pong" to detect if the daemon is still
> responding. 

Yes, I had the same thought. I also considered self-terminating the daemon 
if there is no message on the wire for N seconds, effectively requiring the 
IDE to ping every N seconds if it is not doing anything else.  I considered 
this because otherwise the IDE might crash and leave zombie cmake daemons 
running, but I didn't investigate that possibility.

> That would require the daemon to be able to handle
> multiple requests to be in flight... not sure that this is worth it.

Yes, adding a multi-threading requirement to the daemon might 

[cmake-developers] cmake daemon mode protocol

[Tobias Hunger]

Hi Stephen,

I have successfully build and run your cmake server mode changes and
the python client script does work as advertised.

I do have a couple of remarks about it. This is more intended as a
starting point for discussion as a real proposal. Would something
along these lines be possible:

* Start daemon-mode without any additional parameters

* Daemon responds with "
  {
"type": "handShake",
"version": "3.5.x",
"supportedProtocols": "3.5"
  }

* Client Handshake:
  {
"type": "handshake",
"requestProtocolVersion": "3.5"
  }

* Daemon:
  {
"type:"handshake",
"protocolVersion": "3.5"
  }


At this point the normal operation should be possible.

* Client:
  {
 "type": "setProjectInformation",
 "sourceDirectory": "/some/path",
 "cookie": "1"
  }

  Being able to set the source directory is important so that a fresh
checkout can be configured. There is no build directory available at
that time...

* Daemon should do progress handling while reading the CMakeLists.txt
or whatever it does:
  { "type": "progress", "state": "busy", "task": "someId",
"taskDescription": "Reading CMakeLists.txt files", "cookie": "1" }

* Daemon sends occasional:
  { "type": "progress", "task": "someId", progressTotal: 25,
progressCurrent: 12, "cookie": "1" }

* Daemon sends (probably optional...):
  { "type": "progress", "task": "someId", "state": "done", "cookie": "1" }

* Finally Daemon responds sends data as that is available now.
  {
"type": "projectInformation",
"sourceDirectory": "/some/path",
"buildDirectory": "",
"projectName": "SomeProject",
"cookie": "1"
  }

* Client:
  {
"type": "setProjectInformation",
"buildDirectory": "/some/other/path",
"cookie": "2"
  }

* Daemon does progress indication as above...

* Daemon responds with project info:
  {
"type": "progressInformation",
"sourceDirectory": "/some/path",
"buildDirectory": "/some/other/path",
"projectName": "SomeProject",
"cookie": "2"
  }

We might also want a "ping"/"pong" to detect if the daemon is still
responding. That would require the daemon to be able to handle
multiple requests to be in flight... not sure that this is worth it.
But if we can not do that, then we need to provide constant progress
information, which is also not great:-)

Not sure about the need for cookies (text set by the client and
repeated by the server in all its replies to that request). I do think
having them makes sense, even when only one request can be active at
any time: They make it so much easier to sanity-check communication.

Consistently having a "type" in all messages going back and forth is
probably a good idea for the same reason as having cookies.

A final, remark unrelated to the protocol: Ideally the daemon should
delay writing into (and even creating!) the build directory for as
long as possible (or till instructed to actually persist the
configuration). We do have many users that use Qt Creator to browse
foreign code and those tend to get upset when we write stuff to disk.

Best Regards,
Tobias
-- 

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: 
http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more 
information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at 
http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
http://public.kitware.com/mailman/listinfo/cmake-developers