Re: [lldb-dev] bindings as service idea

[Jim Ingham via lldb-dev]

The server is sending back code.  I'd want to know I can trust whoever is 
sending me back code that I plan to build and run locally.

Jim

> On Nov 19, 2015, at 11:40 AM, Zachary Turner via lldb-dev 
>  wrote:
> 
> 
> 
> On Thu, Nov 19, 2015 at 10:28 AM Todd Fiala  wrote:
> Some other points we need to consider on the bindings-as-service idea:
> 
> * The service should be exposed via secure connection (https/ssl/etc.)  This 
> might already be guaranteed on the Google end by virtue of the endpoint, but 
> we'll want to make sure we can have a secure connection.  (This will be a 
> non-issue for standing up a custom server, but the official one should have 
> this taken care of).
>  
> If the only thing we're sending from client -> server is packaged up source 
> code which is already available on the open source repository, and the server 
> doesn't require authentication, is this necessary?
> 
> If so, does this mean everyone needs to generate a cert locally?
> ___
> lldb-dev mailing list
> lldb-dev@lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev

___
lldb-dev mailing list
lldb-dev@lists.llvm.org
http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev

Re: [lldb-dev] bindings as service idea

[Todd Fiala via lldb-dev]

On Thu, Nov 19, 2015 at 9:44 AM, Zachary Turner  wrote:

> Just to re-iterate, if we use the bindings as a service, then I envision
> checking the bindings in.  This addresses a lot of the potential pitfalls
> you point out, such as the "oops, you can't hit the network, no build for
> you" and the issue of production build flows not wanting to hit a third
> party server, etc.
>
> So if we do that, then I don't think falling back to local generation will
> be an issue (or important) in practice.  i.e. it won't matter if you can't
> hit the network.  The reason I say this is that if you can't hit the
> network you can't check in code either.  So, sure, there might be a short
> window where you can't do a local build , but that would only affect you if
> you were actively modifying a swig interface file AND you were actively
> without a network connection.  The service claims 99.95% uptime, and it's
> safe to say we are looking at significantly less than 100% usage of the
> server (given checked in bindings), so I think we're looking at once a year
> -- if that -- that anyone anywhere has an issue with being able to access
> the service.
>
>
That seems fine.


> And, as you said, the option can be provided to change the host that the
> service runs on, so someone could run one internally.
>
> But do note, that if the goal here is to get the SWIG version bumped in
> the upstream, then we will probably take advantage of some of these new
> SWIG features, which may not work in earlier versions of SWIG.  So you
> should consider how useful it will be to be able to run this server
> internally, because if you can't run a new version of swig locally, then
> can you run it internally anywhere?  I don't know, I'll leave that for you
> to figure out.
>
>
That also seems fine.  And yes, we can work it out on our end.

We'd need to make sure that developer flows would pick up the need to
generate the bindings again if binding surface area changed, but that is no
different than now.


> Either way, it will definitely have the ability to use a different host,
> because that's the easiest way to debug theclient and server (i.e. run them
> on the same machine with 127.0.0.1)
>
>
Yep, sounds right.


> On Thu, Nov 19, 2015 at 8:00 AM Todd Fiala  wrote:
>
>> For the benefit of continuity in conversation, here is what you had to
>> say about it before:
>>
>> > One possibility (which I mentioned to you offline, but I'll put it here for
>> others to see) is that we make a swig bot which is hosted in the cloud much
>> like our public build bots.  We provide a Python script that can be run on
>> your machine, which sends requests over to the swig bot to run swig and
>> send back the results.  Availability of the service would be governed by
>> the SLA of Google Compute Engine, viewable 
>> here:https://cloud.google.com/compute/sla?hl=en
>>
>> > If we do something like this, it would allow us to raise the SWIG version
>> in the upstream, and at that point I can see some benefit in checking the
>> bindings in.  Short of that, I still dont' see the value proposition in
>> checking bindings in to the repo.  [bits deleted]
>>
>> > If it means we can get off of SWIG 1.x in the upstream, I will do the work
>> to make remote swig generation service and get it up and running.
>>
>>
>> I'd like feedback from others on this.  Is this something we want to 
>> consider doing?
>>
>> From my perspective, this seems reasonable to look into doing if we:
>>
>> (a) have the service code available, and
>>
>> (b) if we so choose, we can readily have the script hit another server (so 
>> that a consumer can have the entire setup on an internal network), and
>>
>> (c: option 1) be able to fall back to generate with swig locally as we do 
>> now in the event that we can't hit the server
>>
>> (c: option 2) rather than fall back to swig generation, use swig generation 
>> as primary (as it is now) but, if a swig is not found, then do the 
>> get-bindings-as-a-service flow.
>>
>> This does open up multiple ways to do something, but I think we need to 
>> avoid a failure mode that says "Oops, you can't hit the network.  Sorry, no 
>> lldb build for you."
>>
>>
>> Reasoning:
>>
>> For (a): just so we all know what we're using.
>>
>> For (b): I can envision production build flows that will not want to be 
>> hitting a third-party server.  We shouldn't require that.
>>
>> For (c): we don't want to prevent building in scenarios that can't hit a 
>> network during the build.
>>
>>
>> -Todd
>>
>>
>> On Wed, Nov 18, 2015 at 10:58 PM, Todd Fiala 
>> wrote:
>>
>>>
>>>
>>> On Wed, Nov 18, 2015 at 10:06 PM, Todd Fiala 
>>> wrote:
>>>
 Hey Zachary,

 I think the time pressure has gotten the better of me, so I want to
 apologize for getting snippy about the static bindings of late.  I am
 confident we will get to a good solution for removing that dependency, but

Re: [lldb-dev] bindings as service idea

[Todd Fiala via lldb-dev]

Some other points we need to consider on the bindings-as-service idea:

* The service should be exposed via secure connection (https/ssl/etc.)
 This might already be guaranteed on the Google end by virtue of the
endpoint, but we'll want to make sure we can have a secure connection.
 (This will be a non-issue for standing up a custom server, but the
official one should have this taken care of).

* The method behind how/when the service is updated needs to be clear to
everyone.  This is both a transparency item and affects how changes to the
service code get to the online service.

We don't have to work those out immediately, but they are things we need to
consider.

-Todd


On Thu, Nov 19, 2015 at 10:17 AM, Todd Fiala  wrote:

> I'm out next week, but I can help if needed after that.
>
> Related to all this, you have mentioned a few times that there are newer
> swig features you want to use.
>
> Can you enumerate the features not present in 1.x but present in 3.x that
> you want to take advantage of, and what benefits they will bring us?  (I'm
> not referring to bug fixes in bindings, but actual features that bring
> something new that we didn't have before).
>
> Thanks!
>
> -Todd
>
> On Thu, Nov 19, 2015 at 10:14 AM, Zachary Turner 
> wrote:
>
>> I wasn't planning on working on this immediately, but given the outcome
>> of the recent static bindings work, I can re-prioritize.  I don't know how
>> long it will take, because honestly writing this kind of thing in Python is
>> new to me.. to make an understatement.  But I'll get it done.  Give me
>> until mid next week and I'll post an update.
>>
>> On Thu, Nov 19, 2015 at 10:12 AM Todd Fiala  wrote:
>>
>>> On Thu, Nov 19, 2015 at 9:44 AM, Zachary Turner 
>>> wrote:
>>>
 Just to re-iterate, if we use the bindings as a service, then I
 envision checking the bindings in.  This addresses a lot of the potential
 pitfalls you point out, such as the "oops, you can't hit the network, no
 build for you" and the issue of production build flows not wanting to hit a
 third party server, etc.

 So if we do that, then I don't think falling back to local generation
 will be an issue (or important) in practice.  i.e. it won't matter if you
 can't hit the network.  The reason I say this is that if you can't hit the
 network you can't check in code either.  So, sure, there might be a short
 window where you can't do a local build , but that would only affect you if
 you were actively modifying a swig interface file AND you were actively
 without a network connection.  The service claims 99.95% uptime, and it's
 safe to say we are looking at significantly less than 100% usage of the
 server (given checked in bindings), so I think we're looking at once a year
 -- if that -- that anyone anywhere has an issue with being able to access
 the service.


>>> That seems fine.
>>>
>>>
 And, as you said, the option can be provided to change the host that
 the service runs on, so someone could run one internally.

 But do note, that if the goal here is to get the SWIG version bumped in
 the upstream, then we will probably take advantage of some of these new
 SWIG features, which may not work in earlier versions of SWIG.  So you
 should consider how useful it will be to be able to run this server
 internally, because if you can't run a new version of swig locally, then
 can you run it internally anywhere?  I don't know, I'll leave that for you
 to figure out.


>>> That also seems fine.  And yes, we can work it out on our end.
>>>
>>> We'd need to make sure that developer flows would pick up the need to
>>> generate the bindings again if binding surface area changed, but that is no
>>> different than now.
>>>
>>>
 Either way, it will definitely have the ability to use a different
 host, because that's the easiest way to debug theclient and server (i.e.
 run them on the same machine with 127.0.0.1)


>>> Yep, sounds right.
>>>
>>>
 On Thu, Nov 19, 2015 at 8:00 AM Todd Fiala 
 wrote:

> For the benefit of continuity in conversation, here is what you had to
> say about it before:
>
> > One possibility (which I mentioned to you offline, but I'll put it here 
> > for
> others to see) is that we make a swig bot which is hosted in the cloud 
> much
> like our public build bots.  We provide a Python script that can be run on
> your machine, which sends requests over to the swig bot to run swig and
> send back the results.  Availability of the service would be governed by
> the SLA of Google Compute Engine, viewable 
> here:https://cloud.google.com/compute/sla?hl=en
>
> > If we do something like this, it would allow us to raise the SWIG 
> > version
> in the upstream, and at 

Re: [lldb-dev] bindings as service idea

[Zachary Turner via lldb-dev]

On Thu, Nov 19, 2015 at 10:50 AM Zachary Turner  wrote:

> Well some of the bugfixes are actually worth mentioning, because we
> actually have bugs on the C++ side that we can't fix because then SWIG
> won't be able to process the header files.  For example, if SWIG sees this
> in a header file, it errors out and can't even proceed.
>
> enum Foo : unsigned {
> Bar = 0x
> };
>
> So instead we have to write this:
>
> enum Foo {
> Bar = 0x
> };
>
> According to the standard this produces an unsigned enum, but MSVC is
> non-comformant here, and we get a signed enum.  So we have hundreds of
> warnings disabled as a result of this, and it's a legitimate bug on the C++
> side that we would like to fix, irrespective of SWIG.
>
> Feature-wise, here's a potentially incomplete list:
>
> * Typemaps can re-use other typemaps, similar to how functions can call
> other functions
> * A new -debug-tmsearch command line option which helps debugging typemap
> problems.  For example, when you're looking at some generated code and
> trying to figure out which typemap generated it.  I know I've had to do
> this many times during the Python 3 conversion, and this would have helped.
> * Template classes are supported.  We probably don't care about this, but
> it's an interesting thought.
> * -builtin option supports generating higher performance wrapping code.
> Read the SWIG documentation about -builtin
>
Grr, do embedded links not work on the mailing list or something?

http://www.swig.org/Doc3.0/Python.html#Python_builtin_types


>
> JavaScript support has been mentioned as someone someone needs on more
> than one occasion.  The name of the person interested escapes me, but this
> wasn't added until SWIG 3.x.  Other languages like Go and Lua aren't in 1.x
> of SWIG either, I don't believe.
>
> I'll try to think of some more.
>
>
> On Thu, Nov 19, 2015 at 10:17 AM Todd Fiala  wrote:
>
>> I'm out next week, but I can help if needed after that.
>>
>> Related to all this, you have mentioned a few times that there are newer
>> swig features you want to use.
>>
>> Can you enumerate the features not present in 1.x but present in 3.x that
>> you want to take advantage of, and what benefits they will bring us?  (I'm
>> not referring to bug fixes in bindings, but actual features that bring
>> something new that we didn't have before).
>>
>> Thanks!
>>
>> -Todd
>>
>> On Thu, Nov 19, 2015 at 10:14 AM, Zachary Turner 
>> wrote:
>>
>>> I wasn't planning on working on this immediately, but given the outcome
>>> of the recent static bindings work, I can re-prioritize.  I don't know how
>>> long it will take, because honestly writing this kind of thing in Python is
>>> new to me.. to make an understatement.  But I'll get it done.  Give me
>>> until mid next week and I'll post an update.
>>>
>>> On Thu, Nov 19, 2015 at 10:12 AM Todd Fiala 
>>> wrote:
>>>
 On Thu, Nov 19, 2015 at 9:44 AM, Zachary Turner 
 wrote:

> Just to re-iterate, if we use the bindings as a service, then I
> envision checking the bindings in.  This addresses a lot of the potential
> pitfalls you point out, such as the "oops, you can't hit the network, no
> build for you" and the issue of production build flows not wanting to hit 
> a
> third party server, etc.
>
> So if we do that, then I don't think falling back to local generation
> will be an issue (or important) in practice.  i.e. it won't matter if you
> can't hit the network.  The reason I say this is that if you can't hit the
> network you can't check in code either.  So, sure, there might be a short
> window where you can't do a local build , but that would only affect you 
> if
> you were actively modifying a swig interface file AND you were actively
> without a network connection.  The service claims 99.95% uptime, and it's
> safe to say we are looking at significantly less than 100% usage of the
> server (given checked in bindings), so I think we're looking at once a 
> year
> -- if that -- that anyone anywhere has an issue with being able to access
> the service.
>
>
 That seems fine.


> And, as you said, the option can be provided to change the host that
> the service runs on, so someone could run one internally.
>
> But do note, that if the goal here is to get the SWIG version bumped
> in the upstream, then we will probably take advantage of some of these new
> SWIG features, which may not work in earlier versions of SWIG.  So you
> should consider how useful it will be to be able to run this server
> internally, because if you can't run a new version of swig locally, then
> can you run it internally anywhere?  I don't know, I'll leave that for you
> to figure out.
>
>
 That also seems fine.  And yes, we can work it out 

Re: [lldb-dev] bindings as service idea

[Sean Callanan via lldb-dev]

I don’t think so, this was just an embedded link to your hard drive:

file:///C:/tools/swigwin-3.0.7/Doc/Manual/Python.html#Python_builtin_types

Sean

> On Nov 19, 2015, at 10:51 AM, Zachary Turner via lldb-dev 
>  wrote:
> 
> 
> 
> On Thu, Nov 19, 2015 at 10:50 AM Zachary Turner  > wrote:
> Well some of the bugfixes are actually worth mentioning, because we actually 
> have bugs on the C++ side that we can't fix because then SWIG won't be able 
> to process the header files.  For example, if SWIG sees this in a header 
> file, it errors out and can't even proceed.
> 
> enum Foo : unsigned {
> Bar = 0x
> };
> 
> So instead we have to write this:
> 
> enum Foo {
> Bar = 0x
> };
> 
> According to the standard this produces an unsigned enum, but MSVC is 
> non-comformant here, and we get a signed enum.  So we have hundreds of 
> warnings disabled as a result of this, and it's a legitimate bug on the C++ 
> side that we would like to fix, irrespective of SWIG.
> 
> Feature-wise, here's a potentially incomplete list:
> 
> * Typemaps can re-use other typemaps, similar to how functions can call other 
> functions
> * A new -debug-tmsearch command line option which helps debugging typemap 
> problems.  For example, when you're looking at some generated code and trying 
> to figure out which typemap generated it.  I know I've had to do this many 
> times during the Python 3 conversion, and this would have helped.
> * Template classes are supported.  We probably don't care about this, but 
> it's an interesting thought.
> * -builtin option supports generating higher performance wrapping code.  Read 
> the SWIG documentation about -builtin <>
> Grr, do embedded links not work on the mailing list or something?
> 
> http://www.swig.org/Doc3.0/Python.html#Python_builtin_types 
> 
>  
> 
> JavaScript support has been mentioned as someone someone needs on more than 
> one occasion.  The name of the person interested escapes me, but this wasn't 
> added until SWIG 3.x.  Other languages like Go and Lua aren't in 1.x of SWIG 
> either, I don't believe.
> 
> I'll try to think of some more.
> 
> 
> On Thu, Nov 19, 2015 at 10:17 AM Todd Fiala  > wrote:
> I'm out next week, but I can help if needed after that.
> 
> Related to all this, you have mentioned a few times that there are newer swig 
> features you want to use.
> 
> Can you enumerate the features not present in 1.x but present in 3.x that you 
> want to take advantage of, and what benefits they will bring us?  (I'm not 
> referring to bug fixes in bindings, but actual features that bring something 
> new that we didn't have before).
> 
> Thanks!
> 
> -Todd
> 
> On Thu, Nov 19, 2015 at 10:14 AM, Zachary Turner  > wrote:
> I wasn't planning on working on this immediately, but given the outcome of 
> the recent static bindings work, I can re-prioritize.  I don't know how long 
> it will take, because honestly writing this kind of thing in Python is new to 
> me.. to make an understatement.  But I'll get it done.  Give me until mid 
> next week and I'll post an update.
> 
> On Thu, Nov 19, 2015 at 10:12 AM Todd Fiala  > wrote:
> On Thu, Nov 19, 2015 at 9:44 AM, Zachary Turner  > wrote:
> Just to re-iterate, if we use the bindings as a service, then I envision 
> checking the bindings in.  This addresses a lot of the potential pitfalls you 
> point out, such as the "oops, you can't hit the network, no build for you" 
> and the issue of production build flows not wanting to hit a third party 
> server, etc.  
> 
> So if we do that, then I don't think falling back to local generation will be 
> an issue (or important) in practice.  i.e. it won't matter if you can't hit 
> the network.  The reason I say this is that if you can't hit the network you 
> can't check in code either.  So, sure, there might be a short window where 
> you can't do a local build , but that would only affect you if you were 
> actively modifying a swig interface file AND you were actively without a 
> network connection.  The service claims 99.95% uptime, and it's safe to say 
> we are looking at significantly less than 100% usage of the server (given 
> checked in bindings), so I think we're looking at once a year -- if that -- 
> that anyone anywhere has an issue with being able to access the service.
> 
> 
> That seems fine.
>  
> And, as you said, the option can be provided to change the host that the 
> service runs on, so someone could run one internally.
> 
> But do note, that if the goal here is to get the SWIG version bumped in the 
> upstream, then we will probably take advantage of some of these new SWIG 
> features, which may not 

Re: [lldb-dev] bindings as service idea

[Zachary Turner via lldb-dev]

Well some of the bugfixes are actually worth mentioning, because we
actually have bugs on the C++ side that we can't fix because then SWIG
won't be able to process the header files.  For example, if SWIG sees this
in a header file, it errors out and can't even proceed.

enum Foo : unsigned {
Bar = 0x
};

So instead we have to write this:

enum Foo {
Bar = 0x
};

According to the standard this produces an unsigned enum, but MSVC is
non-comformant here, and we get a signed enum.  So we have hundreds of
warnings disabled as a result of this, and it's a legitimate bug on the C++
side that we would like to fix, irrespective of SWIG.

Feature-wise, here's a potentially incomplete list:

* Typemaps can re-use other typemaps, similar to how functions can call
other functions
* A new -debug-tmsearch command line option which helps debugging typemap
problems.  For example, when you're looking at some generated code and
trying to figure out which typemap generated it.  I know I've had to do
this many times during the Python 3 conversion, and this would have helped.
* Template classes are supported.  We probably don't care about this, but
it's an interesting thought.
* -builtin option supports generating higher performance wrapping code.
Read the SWIG documentation about -builtin


JavaScript support has been mentioned as someone someone needs on more than
one occasion.  The name of the person interested escapes me, but this
wasn't added until SWIG 3.x.  Other languages like Go and Lua aren't in 1.x
of SWIG either, I don't believe.

I'll try to think of some more.

On Thu, Nov 19, 2015 at 10:17 AM Todd Fiala  wrote:

> I'm out next week, but I can help if needed after that.
>
> Related to all this, you have mentioned a few times that there are newer
> swig features you want to use.
>
> Can you enumerate the features not present in 1.x but present in 3.x that
> you want to take advantage of, and what benefits they will bring us?  (I'm
> not referring to bug fixes in bindings, but actual features that bring
> something new that we didn't have before).
>
> Thanks!
>
> -Todd
>
> On Thu, Nov 19, 2015 at 10:14 AM, Zachary Turner 
> wrote:
>
>> I wasn't planning on working on this immediately, but given the outcome
>> of the recent static bindings work, I can re-prioritize.  I don't know how
>> long it will take, because honestly writing this kind of thing in Python is
>> new to me.. to make an understatement.  But I'll get it done.  Give me
>> until mid next week and I'll post an update.
>>
>> On Thu, Nov 19, 2015 at 10:12 AM Todd Fiala  wrote:
>>
>>> On Thu, Nov 19, 2015 at 9:44 AM, Zachary Turner 
>>> wrote:
>>>
 Just to re-iterate, if we use the bindings as a service, then I
 envision checking the bindings in.  This addresses a lot of the potential
 pitfalls you point out, such as the "oops, you can't hit the network, no
 build for you" and the issue of production build flows not wanting to hit a
 third party server, etc.

 So if we do that, then I don't think falling back to local generation
 will be an issue (or important) in practice.  i.e. it won't matter if you
 can't hit the network.  The reason I say this is that if you can't hit the
 network you can't check in code either.  So, sure, there might be a short
 window where you can't do a local build , but that would only affect you if
 you were actively modifying a swig interface file AND you were actively
 without a network connection.  The service claims 99.95% uptime, and it's
 safe to say we are looking at significantly less than 100% usage of the
 server (given checked in bindings), so I think we're looking at once a year
 -- if that -- that anyone anywhere has an issue with being able to access
 the service.


>>> That seems fine.
>>>
>>>
 And, as you said, the option can be provided to change the host that
 the service runs on, so someone could run one internally.

 But do note, that if the goal here is to get the SWIG version bumped in
 the upstream, then we will probably take advantage of some of these new
 SWIG features, which may not work in earlier versions of SWIG.  So you
 should consider how useful it will be to be able to run this server
 internally, because if you can't run a new version of swig locally, then
 can you run it internally anywhere?  I don't know, I'll leave that for you
 to figure out.


>>> That also seems fine.  And yes, we can work it out on our end.
>>>
>>> We'd need to make sure that developer flows would pick up the need to
>>> generate the bindings again if binding surface area changed, but that is no
>>> different than now.
>>>
>>>
 Either way, it will definitely have the ability to use a different
 host, because that's the easiest way to debug theclient and server (i.e.
 run them on the 

Re: [lldb-dev] bindings as service idea

[Zachary Turner via lldb-dev]

I wasn't planning on working on this immediately, but given the outcome of
the recent static bindings work, I can re-prioritize.  I don't know how
long it will take, because honestly writing this kind of thing in Python is
new to me.. to make an understatement.  But I'll get it done.  Give me
until mid next week and I'll post an update.

On Thu, Nov 19, 2015 at 10:12 AM Todd Fiala  wrote:

> On Thu, Nov 19, 2015 at 9:44 AM, Zachary Turner 
> wrote:
>
>> Just to re-iterate, if we use the bindings as a service, then I envision
>> checking the bindings in.  This addresses a lot of the potential pitfalls
>> you point out, such as the "oops, you can't hit the network, no build for
>> you" and the issue of production build flows not wanting to hit a third
>> party server, etc.
>>
>> So if we do that, then I don't think falling back to local generation
>> will be an issue (or important) in practice.  i.e. it won't matter if you
>> can't hit the network.  The reason I say this is that if you can't hit the
>> network you can't check in code either.  So, sure, there might be a short
>> window where you can't do a local build , but that would only affect you if
>> you were actively modifying a swig interface file AND you were actively
>> without a network connection.  The service claims 99.95% uptime, and it's
>> safe to say we are looking at significantly less than 100% usage of the
>> server (given checked in bindings), so I think we're looking at once a year
>> -- if that -- that anyone anywhere has an issue with being able to access
>> the service.
>>
>>
> That seems fine.
>
>
>> And, as you said, the option can be provided to change the host that the
>> service runs on, so someone could run one internally.
>>
>> But do note, that if the goal here is to get the SWIG version bumped in
>> the upstream, then we will probably take advantage of some of these new
>> SWIG features, which may not work in earlier versions of SWIG.  So you
>> should consider how useful it will be to be able to run this server
>> internally, because if you can't run a new version of swig locally, then
>> can you run it internally anywhere?  I don't know, I'll leave that for you
>> to figure out.
>>
>>
> That also seems fine.  And yes, we can work it out on our end.
>
> We'd need to make sure that developer flows would pick up the need to
> generate the bindings again if binding surface area changed, but that is no
> different than now.
>
>
>> Either way, it will definitely have the ability to use a different host,
>> because that's the easiest way to debug theclient and server (i.e. run them
>> on the same machine with 127.0.0.1)
>>
>>
> Yep, sounds right.
>
>
>> On Thu, Nov 19, 2015 at 8:00 AM Todd Fiala  wrote:
>>
>>> For the benefit of continuity in conversation, here is what you had to
>>> say about it before:
>>>
>>> > One possibility (which I mentioned to you offline, but I'll put it here 
>>> > for
>>> others to see) is that we make a swig bot which is hosted in the cloud much
>>> like our public build bots.  We provide a Python script that can be run on
>>> your machine, which sends requests over to the swig bot to run swig and
>>> send back the results.  Availability of the service would be governed by
>>> the SLA of Google Compute Engine, viewable 
>>> here:https://cloud.google.com/compute/sla?hl=en
>>>
>>> > If we do something like this, it would allow us to raise the SWIG version
>>> in the upstream, and at that point I can see some benefit in checking the
>>> bindings in.  Short of that, I still dont' see the value proposition in
>>> checking bindings in to the repo.  [bits deleted]
>>>
>>> > If it means we can get off of SWIG 1.x in the upstream, I will do the work
>>> to make remote swig generation service and get it up and running.
>>>
>>>
>>> I'd like feedback from others on this.  Is this something we want to 
>>> consider doing?
>>>
>>> From my perspective, this seems reasonable to look into doing if we:
>>>
>>> (a) have the service code available, and
>>>
>>> (b) if we so choose, we can readily have the script hit another server (so 
>>> that a consumer can have the entire setup on an internal network), and
>>>
>>> (c: option 1) be able to fall back to generate with swig locally as we do 
>>> now in the event that we can't hit the server
>>>
>>> (c: option 2) rather than fall back to swig generation, use swig generation 
>>> as primary (as it is now) but, if a swig is not found, then do the 
>>> get-bindings-as-a-service flow.
>>>
>>> This does open up multiple ways to do something, but I think we need to 
>>> avoid a failure mode that says "Oops, you can't hit the network.  Sorry, no 
>>> lldb build for you."
>>>
>>>
>>> Reasoning:
>>>
>>> For (a): just so we all know what we're using.
>>>
>>> For (b): I can envision production build flows that will not want to be 
>>> hitting a third-party server.  We shouldn't require that.
>>>
>>> For (c): we don't want 

Re: [lldb-dev] bindings as service idea

[Zachary Turner via lldb-dev]

Doh!  lol

On Thu, Nov 19, 2015 at 10:57 AM Sean Callanan  wrote:

> I don’t think so, this was just an embedded link to your hard drive:
>
> file:///C:/tools/swigwin-3.0.7/Doc/Manual/Python.html#Python_builtin_types
>
> Sean
>
> On Nov 19, 2015, at 10:51 AM, Zachary Turner via lldb-dev <
> lldb-dev@lists.llvm.org> wrote:
>
>
>
> On Thu, Nov 19, 2015 at 10:50 AM Zachary Turner 
> wrote:
>
>> Well some of the bugfixes are actually worth mentioning, because we
>> actually have bugs on the C++ side that we can't fix because then SWIG
>> won't be able to process the header files.  For example, if SWIG sees this
>> in a header file, it errors out and can't even proceed.
>>
>> enum Foo : unsigned {
>> Bar = 0x
>> };
>>
>> So instead we have to write this:
>>
>> enum Foo {
>> Bar = 0x
>> };
>>
>> According to the standard this produces an unsigned enum, but MSVC is
>> non-comformant here, and we get a signed enum.  So we have hundreds of
>> warnings disabled as a result of this, and it's a legitimate bug on the C++
>> side that we would like to fix, irrespective of SWIG.
>>
>> Feature-wise, here's a potentially incomplete list:
>>
>> * Typemaps can re-use other typemaps, similar to how functions can call
>> other functions
>> * A new -debug-tmsearch command line option which helps debugging typemap
>> problems.  For example, when you're looking at some generated code and
>> trying to figure out which typemap generated it.  I know I've had to do
>> this many times during the Python 3 conversion, and this would have helped.
>> * Template classes are supported.  We probably don't care about this, but
>> it's an interesting thought.
>> * -builtin option supports generating higher performance wrapping code.
>> Read the SWIG documentation about -builtin
>>
> Grr, do embedded links not work on the mailing list or something?
>
> http://www.swig.org/Doc3.0/Python.html#Python_builtin_types
>
>
>>
>> JavaScript support has been mentioned as someone someone needs on more
>> than one occasion.  The name of the person interested escapes me, but this
>> wasn't added until SWIG 3.x.  Other languages like Go and Lua aren't in 1.x
>> of SWIG either, I don't believe.
>>
>> I'll try to think of some more.
>>
>>
>> On Thu, Nov 19, 2015 at 10:17 AM Todd Fiala  wrote:
>>
>>> I'm out next week, but I can help if needed after that.
>>>
>>> Related to all this, you have mentioned a few times that there are newer
>>> swig features you want to use.
>>>
>>> Can you enumerate the features not present in 1.x but present in 3.x
>>> that you want to take advantage of, and what benefits they will bring us?
>>>  (I'm not referring to bug fixes in bindings, but actual features that
>>> bring something new that we didn't have before).
>>>
>>> Thanks!
>>>
>>> -Todd
>>>
>>> On Thu, Nov 19, 2015 at 10:14 AM, Zachary Turner 
>>> wrote:
>>>
 I wasn't planning on working on this immediately, but given the outcome
 of the recent static bindings work, I can re-prioritize.  I don't know how
 long it will take, because honestly writing this kind of thing in Python is
 new to me.. to make an understatement.  But I'll get it done.  Give me
 until mid next week and I'll post an update.

 On Thu, Nov 19, 2015 at 10:12 AM Todd Fiala 
 wrote:

> On Thu, Nov 19, 2015 at 9:44 AM, Zachary Turner 
> wrote:
>
>> Just to re-iterate, if we use the bindings as a service, then I
>> envision checking the bindings in.  This addresses a lot of the potential
>> pitfalls you point out, such as the "oops, you can't hit the network, no
>> build for you" and the issue of production build flows not wanting to 
>> hit a
>> third party server, etc.
>>
>> So if we do that, then I don't think falling back to local generation
>> will be an issue (or important) in practice.  i.e. it won't matter if you
>> can't hit the network.  The reason I say this is that if you can't hit 
>> the
>> network you can't check in code either.  So, sure, there might be a short
>> window where you can't do a local build , but that would only affect you 
>> if
>> you were actively modifying a swig interface file AND you were actively
>> without a network connection.  The service claims 99.95% uptime, and it's
>> safe to say we are looking at significantly less than 100% usage of the
>> server (given checked in bindings), so I think we're looking at once a 
>> year
>> -- if that -- that anyone anywhere has an issue with being able to access
>> the service.
>>
>>
> That seems fine.
>
>
>> And, as you said, the option can be provided to change the host that
>> the service runs on, so someone could run one internally.
>>
>> But do note, that if the goal here is to get the SWIG version bumped
>> in 

Re: [lldb-dev] bindings as service idea

[Zachary Turner via lldb-dev]

Derp, I forgot C++11 support.  Some of it is not useful to us given the
limited nature of the SB API, but certain things could be useful.  Being
able to transition to enum classes is nice, for example.

Anyway, there's a lot here, so feel free to read through the document.
http://www.swig.org/Doc3.0/CPlusPlus11.html


On Thu, Nov 19, 2015 at 10:50 AM Zachary Turner  wrote:

> Well some of the bugfixes are actually worth mentioning, because we
> actually have bugs on the C++ side that we can't fix because then SWIG
> won't be able to process the header files.  For example, if SWIG sees this
> in a header file, it errors out and can't even proceed.
>
> enum Foo : unsigned {
> Bar = 0x
> };
>
> So instead we have to write this:
>
> enum Foo {
> Bar = 0x
> };
>
> According to the standard this produces an unsigned enum, but MSVC is
> non-comformant here, and we get a signed enum.  So we have hundreds of
> warnings disabled as a result of this, and it's a legitimate bug on the C++
> side that we would like to fix, irrespective of SWIG.
>
> Feature-wise, here's a potentially incomplete list:
>
> * Typemaps can re-use other typemaps, similar to how functions can call
> other functions
> * A new -debug-tmsearch command line option which helps debugging typemap
> problems.  For example, when you're looking at some generated code and
> trying to figure out which typemap generated it.  I know I've had to do
> this many times during the Python 3 conversion, and this would have helped.
> * Template classes are supported.  We probably don't care about this, but
> it's an interesting thought.
> * -builtin option supports generating higher performance wrapping code.
> Read the SWIG documentation about -builtin
>
> JavaScript support has been mentioned as someone someone needs on more
> than one occasion.  The name of the person interested escapes me, but this
> wasn't added until SWIG 3.x.  Other languages like Go and Lua aren't in 1.x
> of SWIG either, I don't believe.
>
> I'll try to think of some more.
>
>
> On Thu, Nov 19, 2015 at 10:17 AM Todd Fiala  wrote:
>
>> I'm out next week, but I can help if needed after that.
>>
>> Related to all this, you have mentioned a few times that there are newer
>> swig features you want to use.
>>
>> Can you enumerate the features not present in 1.x but present in 3.x that
>> you want to take advantage of, and what benefits they will bring us?  (I'm
>> not referring to bug fixes in bindings, but actual features that bring
>> something new that we didn't have before).
>>
>> Thanks!
>>
>> -Todd
>>
>> On Thu, Nov 19, 2015 at 10:14 AM, Zachary Turner 
>> wrote:
>>
>>> I wasn't planning on working on this immediately, but given the outcome
>>> of the recent static bindings work, I can re-prioritize.  I don't know how
>>> long it will take, because honestly writing this kind of thing in Python is
>>> new to me.. to make an understatement.  But I'll get it done.  Give me
>>> until mid next week and I'll post an update.
>>>
>>> On Thu, Nov 19, 2015 at 10:12 AM Todd Fiala 
>>> wrote:
>>>
 On Thu, Nov 19, 2015 at 9:44 AM, Zachary Turner 
 wrote:

> Just to re-iterate, if we use the bindings as a service, then I
> envision checking the bindings in.  This addresses a lot of the potential
> pitfalls you point out, such as the "oops, you can't hit the network, no
> build for you" and the issue of production build flows not wanting to hit 
> a
> third party server, etc.
>
> So if we do that, then I don't think falling back to local generation
> will be an issue (or important) in practice.  i.e. it won't matter if you
> can't hit the network.  The reason I say this is that if you can't hit the
> network you can't check in code either.  So, sure, there might be a short
> window where you can't do a local build , but that would only affect you 
> if
> you were actively modifying a swig interface file AND you were actively
> without a network connection.  The service claims 99.95% uptime, and it's
> safe to say we are looking at significantly less than 100% usage of the
> server (given checked in bindings), so I think we're looking at once a 
> year
> -- if that -- that anyone anywhere has an issue with being able to access
> the service.
>
>
 That seems fine.


> And, as you said, the option can be provided to change the host that
> the service runs on, so someone could run one internally.
>
> But do note, that if the goal here is to get the SWIG version bumped
> in the upstream, then we will probably take advantage of some of these new
> SWIG features, which may not work in earlier versions of SWIG.  So you
> should consider how useful it will be to be able to run this server
> internally, because if you can't run a new version of 

Re: [lldb-dev] bindings as service idea

[Todd Fiala via lldb-dev]

>> If so, does this mean everyone needs to generate a cert locally?

Generally not - as long as the server is dishing out something over https,
the server will be signed with a certificate that is going to be in the
local OS's set of trusted root certificates (particularly if this is
provided by Google).  It is true that a die-hard self OS builder can
totally build their own trusted set of roots, but that's going beyond.  (It
is possible to buy/procure very cheap certificates that come from
Certificate Authorities that are generally not popular or well known enough
to be in the stock set of Microsoft/OS X/Ubuntu trusted roots, but this is
totally avoidable.)



On Thu, Nov 19, 2015 at 11:48 AM, Jim Ingham  wrote:

> The server is sending back code.  I'd want to know I can trust whoever is
> sending me back code that I plan to build and run locally.
>
> Jim
>
> > On Nov 19, 2015, at 11:40 AM, Zachary Turner via lldb-dev <
> lldb-dev@lists.llvm.org> wrote:
> >
> >
> >
> > On Thu, Nov 19, 2015 at 10:28 AM Todd Fiala 
> wrote:
> > Some other points we need to consider on the bindings-as-service idea:
> >
> > * The service should be exposed via secure connection (https/ssl/etc.)
> This might already be guaranteed on the Google end by virtue of the
> endpoint, but we'll want to make sure we can have a secure connection.
> (This will be a non-issue for standing up a custom server, but the official
> one should have this taken care of).
> >
> > If the only thing we're sending from client -> server is packaged up
> source code which is already available on the open source repository, and
> the server doesn't require authentication, is this necessary?
> >
> > If so, does this mean everyone needs to generate a cert locally?
> > ___
> > lldb-dev mailing list
> > lldb-dev@lists.llvm.org
> > http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev
>
>


-- 
-Todd
___
lldb-dev mailing list
lldb-dev@lists.llvm.org
http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev

Re: [lldb-dev] bindings as service idea

[Ted Woodward via lldb-dev]

For our builds at QUIC, we're not interested in hitting an external server to 
get code. So we'd either hit the server when needed and check in the resultant 
bindings, or  (preferably) use bindings from upstream.

 

--

Qualcomm Innovation Center, Inc.

The Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, a Linux 
Foundation Collaborative Project

 

From: lldb-dev [mailto:lldb-dev-boun...@lists.llvm.org] On Behalf Of Zachary 
Turner via lldb-dev
Sent: Thursday, November 19, 2015 11:45 AM
To: Todd Fiala
Cc: LLDB
Subject: Re: [lldb-dev] bindings as service idea

 

Just to re-iterate, if we use the bindings as a service, then I envision 
checking the bindings in.  This addresses a lot of the potential pitfalls you 
point out, such as the "oops, you can't hit the network, no build for you" and 
the issue of production build flows not wanting to hit a third party server, 
etc.  

 

So if we do that, then I don't think falling back to local generation will be 
an issue (or important) in practice.  i.e. it won't matter if you can't hit the 
network.  The reason I say this is that if you can't hit the network you can't 
check in code either.  So, sure, there might be a short window where you can't 
do a local build , but that would only affect you if you were actively 
modifying a swig interface file AND you were actively without a network 
connection.  The service claims 99.95% uptime, and it's safe to say we are 
looking at significantly less than 100% usage of the server (given checked in 
bindings), so I think we're looking at once a year -- if that -- that anyone 
anywhere has an issue with being able to access the service.

 

And, as you said, the option can be provided to change the host that the 
service runs on, so someone could run one internally.

 

But do note, that if the goal here is to get the SWIG version bumped in the 
upstream, then we will probably take advantage of some of these new SWIG 
features, which may not work in earlier versions of SWIG.  So you should 
consider how useful it will be to be able to run this server internally, 
because if you can't run a new version of swig locally, then can you run it 
internally anywhere?  I don't know, I'll leave that for you to figure out.

 

Either way, it will definitely have the ability to use a different host, 
because that's the easiest way to debug theclient and server (i.e. run them on 
the same machine with 127.0.0.1)

 

On Thu, Nov 19, 2015 at 8:00 AM Todd Fiala <todd.fi...@gmail.com 
<mailto:todd.fi...@gmail.com> > wrote:

For the benefit of continuity in conversation, here is what you had to say 
about it before:

 

> One possibility (which I mentioned to you offline, but I'll put it here for
others to see) is that we make a swig bot which is hosted in the cloud much
like our public build bots.  We provide a Python script that can be run on
your machine, which sends requests over to the swig bot to run swig and
send back the results.  Availability of the service would be governed by
the SLA of Google Compute Engine, viewable here:
https://cloud.google.com/compute/sla?hl=en
 
> If we do something like this, it would allow us to raise the SWIG version
in the upstream, and at that point I can see some benefit in checking the
bindings in.  Short of that, I still dont' see the value proposition in
checking bindings in to the repo.  [bits deleted]
 
> If it means we can get off of SWIG 1.x in the upstream, I will do the work
to make remote swig generation service and get it up and running.
 
I'd like feedback from others on this.  Is this something we want to consider 
doing?
>From my perspective, this seems reasonable to look into doing if we:
(a) have the service code available, and
(b) if we so choose, we can readily have the script hit another server (so that 
a consumer can have the entire setup on an internal network), and
(c: option 1) be able to fall back to generate with swig locally as we do now 
in the event that we can't hit the server
(c: option 2) rather than fall back to swig generation, use swig generation as 
primary (as it is now) but, if a swig is not found, then do the 
get-bindings-as-a-service flow.
This does open up multiple ways to do something, but I think we need to avoid a 
failure mode that says "Oops, you can't hit the network.  Sorry, no lldb build 
for you."
 
Reasoning:
For (a): just so we all know what we're using.
For (b): I can envision production build flows that will not want to be hitting 
a third-party server.  We shouldn't require that. 
For (c): we don't want to prevent building in scenarios that can't hit a 
network during the build.
 
-Todd

 

On Wed, Nov 18, 2015 at 10:58 PM, Todd Fiala <todd.fi...@gmail.com 
<mailto:todd.fi...@gmail.com> > wrote:

 

 

On Wed, Nov 18, 2015 at 10:06 PM, Todd Fiala <todd.fi...@gmail.com 
<mailto:todd.fi...@gmail.com> > wrote:

Hey Zachary,

 

I think the time pressure has

Re: [lldb-dev] bindings as service idea

[Zachary Turner via lldb-dev]

On Thu, Nov 19, 2015 at 1:38 PM Ted Woodward 
wrote:

> For our builds at QUIC, we're not interested in hitting an external server
> to get code. So we'd either hit the server when needed and check in the
> resultant bindings, or  (preferably) use bindings from upstream.
>

Right, and this is actually what I'm proposing.  You only hit the service
when you are editing a swig interface file and you need to generate updated
code.  Otherwise the latest version of the bindings is already checked out
in your repository.

And even then, you only *really* need to hit it when you physically cannot
(for whatever reason) or do not want to install swig on your machine.
There can still be a local path, that says "use the swig installed on my
machine".  Of course, if we're going to standardize on a version, it will
be the version of swig that the service runs.   So whatever version you
have on your machine -- if you intend to check in those bindings -- need to
be the same version used by the service.
___
lldb-dev mailing list
lldb-dev@lists.llvm.org
http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev

Re: [lldb-dev] bindings as service idea

[Zachary Turner via lldb-dev]

Just to re-iterate, if we use the bindings as a service, then I envision
checking the bindings in.  This addresses a lot of the potential pitfalls
you point out, such as the "oops, you can't hit the network, no build for
you" and the issue of production build flows not wanting to hit a third
party server, etc.

So if we do that, then I don't think falling back to local generation will
be an issue (or important) in practice.  i.e. it won't matter if you can't
hit the network.  The reason I say this is that if you can't hit the
network you can't check in code either.  So, sure, there might be a short
window where you can't do a local build , but that would only affect you if
you were actively modifying a swig interface file AND you were actively
without a network connection.  The service claims 99.95% uptime, and it's
safe to say we are looking at significantly less than 100% usage of the
server (given checked in bindings), so I think we're looking at once a year
-- if that -- that anyone anywhere has an issue with being able to access
the service.

And, as you said, the option can be provided to change the host that the
service runs on, so someone could run one internally.

But do note, that if the goal here is to get the SWIG version bumped in the
upstream, then we will probably take advantage of some of these new SWIG
features, which may not work in earlier versions of SWIG.  So you should
consider how useful it will be to be able to run this server internally,
because if you can't run a new version of swig locally, then can you run it
internally anywhere?  I don't know, I'll leave that for you to figure out.

Either way, it will definitely have the ability to use a different host,
because that's the easiest way to debug theclient and server (i.e. run them
on the same machine with 127.0.0.1)

On Thu, Nov 19, 2015 at 8:00 AM Todd Fiala  wrote:

> For the benefit of continuity in conversation, here is what you had to say
> about it before:
>
> > One possibility (which I mentioned to you offline, but I'll put it here for
> others to see) is that we make a swig bot which is hosted in the cloud much
> like our public build bots.  We provide a Python script that can be run on
> your machine, which sends requests over to the swig bot to run swig and
> send back the results.  Availability of the service would be governed by
> the SLA of Google Compute Engine, viewable 
> here:https://cloud.google.com/compute/sla?hl=en
>
> > If we do something like this, it would allow us to raise the SWIG version
> in the upstream, and at that point I can see some benefit in checking the
> bindings in.  Short of that, I still dont' see the value proposition in
> checking bindings in to the repo.  [bits deleted]
>
> > If it means we can get off of SWIG 1.x in the upstream, I will do the work
> to make remote swig generation service and get it up and running.
>
>
> I'd like feedback from others on this.  Is this something we want to consider 
> doing?
>
> From my perspective, this seems reasonable to look into doing if we:
>
> (a) have the service code available, and
>
> (b) if we so choose, we can readily have the script hit another server (so 
> that a consumer can have the entire setup on an internal network), and
>
> (c: option 1) be able to fall back to generate with swig locally as we do now 
> in the event that we can't hit the server
>
> (c: option 2) rather than fall back to swig generation, use swig generation 
> as primary (as it is now) but, if a swig is not found, then do the 
> get-bindings-as-a-service flow.
>
> This does open up multiple ways to do something, but I think we need to avoid 
> a failure mode that says "Oops, you can't hit the network.  Sorry, no lldb 
> build for you."
>
>
> Reasoning:
>
> For (a): just so we all know what we're using.
>
> For (b): I can envision production build flows that will not want to be 
> hitting a third-party server.  We shouldn't require that.
>
> For (c): we don't want to prevent building in scenarios that can't hit a 
> network during the build.
>
>
> -Todd
>
>
> On Wed, Nov 18, 2015 at 10:58 PM, Todd Fiala  wrote:
>
>>
>>
>> On Wed, Nov 18, 2015 at 10:06 PM, Todd Fiala 
>> wrote:
>>
>>> Hey Zachary,
>>>
>>> I think the time pressure has gotten the better of me, so I want to
>>> apologize for getting snippy about the static bindings of late.  I am
>>> confident we will get to a good solution for removing that dependency, but
>>> I can certainly wait for a solution (using an alternate approach in our
>>> branch) until we arrive at something more palatable to everyone.
>>>
>>> Regarding the bindings as service idea:
>>>
>>> How quickly do you think you could flesh out the bindings as a service
>>> idea?  With a relatively strong dislike for the static approach I'm taking,
>>> I can back off that and just use my current code here in a downstream
>>> branch for now.  Ultimately I want to remove the requirement