Re: Upgrading minimum Go version?

[John Meinel]

Given how often people still use "--upload-tools" for things like private
clouds (and is definitely the one used for local provider), I'd really
worry about having a jujud on your local machine that wasn't built with the
same toolchain as the one you get from "juju bootstrap" in other cases.
Very easy to end up with hard to understand/reproduce bugs.

John
=:->


On Mon, Nov 30, 2015 at 6:32 PM, Curtis Hovey-Canonical <
cur...@canonical.com> wrote:

> Good discussion. I have one nuance that I think we want to discuss in
> SFO. Namely, controlling the Juju tool chain.
>
> Juju QA uses CI to make many of the things we releases, such as win
> agents. We use Launchpad/Ubuntu to make Ubuntu agents and clients.
> Juju QA qill soon have access to ARM hardware. When we do, we can
> choose to by-pass Lp/Ubuntu for agents. The agents that were built and
> tested by CI are the agents we release in streams. This permits us to
> choose the best tool chain to create each series+arch combination.
> This also introduces drift between what Juju officially releases, and
> and Ubuntu releases.
>
> On Thu, Nov 26, 2015 at 8:27 PM, Andrew Wilkins
>  wrote:
> > On Fri, Nov 27, 2015 at 7:49 AM Michael Hudson-Doyle
> >  wrote:
> >>
> >> On 27 November 2015 at 09:39, Tim Penhey 
> wrote:
> >> > On 27/11/15 08:43, Michael Hudson-Doyle wrote:
> >> >> On 27 November 2015 at 02:24, Martin Packman
> >> >>  wrote:
> >> >>> On 26/11/2015, Andrew Wilkins  wrote:
> >>  Hi (mostly Curtis),
> >> 
> >>  Is there a plan to bump the minimum Go version? Some of our
> >>  dependencies do
> >>  not build with Go 1.2. The LXD provider only builds with Go 1.3 (I
> >>  think?),
> >>  and I've got a PR up that updates the azure-sdk-for-go dependency,
> >>  but it's
> >>  blocked because the newer doesn't build with Go 1.2.
> >> >>
> >> >> Is this something we've done to ourselves or is there a third-party
> >> >> library we're depending on that doesn't work with Go 1.2?
> >> >
> >> > The two main ones I know about are lxd and the new azure go bindings.
> >>
> >> By the azure go bindings you mean something Canonical didn't write,
> >> like https://github.com/Azure/azure-sdk-for-go? That sort of thing
> >> sounds like a good argument for the 1.5-in-trusty SRU thing.
>
>
>
>
> --
> Curtis Hovey
> Canonical Cloud Development and Operations
> http://launchpad.net/~sinzui
>
> --
> Juju-dev mailing list
> Juju-dev@lists.ubuntu.com
> Modify settings or unsubscribe at:
> https://lists.ubuntu.com/mailman/listinfo/juju-dev
>
-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: Juju devel 1.26-alpha2 is available for testing

[Curtis Hovey-Canonical]

There are other options to play with juju+lxd on trusty...

On Fri, Nov 27, 2015 at 2:26 PM, Rick Harding
 wrote:
>
> On Fri, Nov 27, 2015 at 11:35 AM Mark Shuttleworth  wrote:
>>
>> On 27/11/15 16:21, Aaron Bentley wrote:
>>
>> It's dependent on what compiler was used to create the jujud binary.
>> AIUI, the Ubuntu policy is that nothing goes into a distroseries which
>> cannot be compiled with the tools in that distroseries.  Thus the
>> jujud for Trusty is compiled with the version of Go provided by that
>> platform.
>>
>>
>> My understanding is that a Go 1.5 backport to Trusty is part of the
>> current cycle planned work.
>
>
> Yes, the work for Go 1.5 into Trusty moves forward. For this alpha it's not
> yet ready to provide the build so my understanding is that the alpha build
> for Trusty is done with the current outdated tool chain. Once the Go
> toolchain is updated for Trusty the builds released will be in order.
>
> Aaron, please correct me if I'm mistaken there.

The Juju clients and agents built with Go lang are statically
compiled. They are Ubuntu release agnostic. The wily-built Juju runs
fine on Trusty and Precise (and Centos 7). You can install the wily
juju-core and juju-local packages to play with the lxd feature now.

Per the conversation above, the Juju PPAs build with a deps that
provides the Juju teams minimum and preferred Golang. We used this to
get newer Gos for precise without waiting on Ubuntu. We plan to switch
to switch to Go 1.5 soon at a part of our plan to change Juju's
minimum version of Go.


-- 
Curtis Hovey
Canonical Cloud Development and Operations
http://launchpad.net/~sinzui

-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: Upgrading minimum Go version?

[Aaron Bentley]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256

On 2015-11-30 10:36 AM, John Meinel wrote:
> Given how often people still use "--upload-tools" for things like 
> private clouds (and is definitely the one used for local provider),
> I'd really worry about having a jujud on your local machine that
> wasn't built with the same toolchain as the one you get from "juju
> bootstrap" in other cases. Very easy to end up with hard to
> understand/reproduce bugs.

But you understand that we've been doing that for ages, right?  The
- --upload-tools jujud is always compiled with the toolchain from the
local machine's series, and non-upload-tools one is always compiled
with the toolchain from target machine's series.

Aaron
-BEGIN PGP SIGNATURE-
Version: GnuPG v2

iQEcBAEBCAAGBQJWXG6dAAoJEK84cMOcf+9hfiQIAIv4jOSITt/vFXxlb1sY1MiQ
5OTtpdFOzPmY4X2grUPLEW8PYXMDqT53V0dWuHQic6OcDcnvSc5W06sMqLbfDz7V
dIxCfY5Lt6MpjI9UfS3ec9BVuSC3QT9klVf5ELrQ1HzKzkZK6NE3XzA1rlDJ/+0v
Z3rKymKt8M1kNbZiG3WNODpamyqp6Gt9ie28SOFcBIyaiM+vHhPIog7vjjshtUTh
UwSSWqfBPMUIFLJttH1Nl+XnGPeJD/p5fTSt/2CCs4VUT9q2fEIR1Ur3IcC6JPAK
Vw1Dcuuso9fLZkkFCCRBLMwRCQn8mghtsXQ8qpdFUiF87sWYDDdVI8A101YHc30=
=XP/V
-END PGP SIGNATURE-

-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: Upgrading minimum Go version?

[Curtis Hovey-Canonical]

On Mon, Nov 30, 2015 at 10:36 AM, John Meinel  wrote:
> Given how often people still use "--upload-tools" for things like private
> clouds (and is definitely the one used for local provider), I'd really worry
> about having a jujud on your local machine that wasn't built with the same
> toolchain as the one you get from "juju bootstrap" in other cases. Very easy
> to end up with hard to understand/reproduce bugs.

I share your concerns John.

We have seen several bugs marked invalid because the agent uploads is
essentially unknown origin because Juju is more than willing to fake a
version when uploading. There are also reports of Juju attempting to
build agents from source:
https://bugs.launchpad.net/juju-core/+bug/1399606

We do want everyone using the agent in streams, which have a lot more
testing behind them. The agent (jujud) on users of the Juju stable PPA
is the same that was used to make streams. This is not true for users
of Ubuntu's packages, and we know Ubuntu can change its tool chain
between the time we create official agents and it builds its packages.
We do testing to certify they Ubuntu clients and agents are
functionally equivalent to those in the PPA and streams. As Ubuntu
Wily+ prefers system Go dev packages to the embedded go packages we
provide, Ubuntu's packages will contain more divergence than Trusty.
We are obligated to do on-demand testing to certify a change to a
system Go dev package.

Ideally, We would separate the jujud from juju-core package that
provides the client. Users don't get a jujud when they install a
client. Instead, Juju can get the desired agents from streams.

-- 
Curtis Hovey
Canonical Cloud Development and Operations
http://launchpad.net/~sinzui

-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: Reviews for merges _from_ master to feature branches

[Tim Penhey]

On 01/12/15 13:56, Ian Booth wrote:
> 
> 
> On 01/12/15 10:17, David Cheney wrote:
>> Hello,
>>
>> Why are reviewers being created for merges from master to feature
>> branches ? What purpose does this serve ?
>>
> 
> They appear because you create a github PR which triggers a reviewboard review
> to be created. I just self approve and everything gets processed soon enough.
> You still want a PR to interface with the landing bot.

+1 I also just self approve merges from master into feature branches,
and I have told others that they can do the same.

No need to review merges of master into your own feature.

Makes it go through the bot, which I like. Also best practice to just
merge in a blessed revision.

Tim


-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: utils/fslock needs to DIAF

[David Cheney]

fcntl won't work in threaded go applications, it barely works in non
threaded applications.

I'm not interested in "doesn't work on windows" arguments. Yes, we
have to support windows, but that doesn't mean we have to be dragged
down to it's lowest common denominator.

I think it's fine to develop a lock type that does the best available
for each platform using conditional compilation.

On Tue, Dec 1, 2015 at 11:47 AM, Andrew Wilkins
 wrote:
> On Tue, Dec 1, 2015 at 8:03 AM David Cheney 
> wrote:
>>
>> On Tue, Dec 1, 2015 at 11:00 AM, Andrew Wilkins
>>  wrote:
>> > On Tue, Dec 1, 2015 at 7:57 AM David Cheney 
>> > wrote:
>> >>
>> >> Doesn't look like there is windows support, and it uses fcntl (flock)
>> >> under the hood, which is what we have now.
>> >
>> >
>> > flock isn't the problematic thing Tim is talking about. utils/fslock
>> > attempts to create a directory in a known location, and if it succeeds,
>> > it
>> > "holds the lock". Unlocking means removing the directory.
>>
>> The problem is if the process dies/exits/goes mental while holding the
>> lock we get into this existential gridlock where we're not sure if the
>> process _is_ alive, so we shouldn't remove the lock, or the process is
>> dead, so we should remove the lock.
>>
>> abstract unix domain sockets do not have this problem on windows; kill
>> the process, file is removed from the abstract namespace.
>
>
> POSIX advisory file locks (flock) do not have this problem either. See:
> man(2) fcntl, section "Advisory record locking". When the file descriptor is
> closed, the lock is released; file descriptors are closed when the process
> dies.
>
> We could build a mutex on top of a unix domain socket, but then we'll have
> something completely separate for Windows. Shared named mutex? I'm not
> convinced the overall solution would be any more robust, and I'm pretty sure
> it's going to be more complicated. Happy to be proven wrong though.
>
>> >
>> > We would have to contribute Windows support, but it's not hard, I've
>> > done it
>> > before.
>> >
>> >>
>> >> On Tue, Dec 1, 2015 at 10:55 AM, Casey Marshall
>> >>  wrote:
>> >> > How about github.com/camlistore/lock ?
>> >> >
>> >> > On Mon, Nov 30, 2015 at 5:43 PM, Tim Penhey
>> >> > 
>> >> > wrote:
>> >> >>
>> >> >> Hi folks,
>> >> >>
>> >> >> The fslock was a mistake that I added to the codebase some time
>> >> >> back.
>> >> >> It
>> >> >> provided an overly simplistic solution to a more complex problem.
>> >> >>
>> >> >> Really the filesystem shouldn't be used as a locking mechanism.
>> >> >>
>> >> >> Most of the code that exists for the fslock now is working around
>> >> >> its
>> >> >> deficiencies. Instead we should be looking for a better replacement.
>> >> >>
>> >> >> Some "features" that were added to fslock were added to work around
>> >> >> the
>> >> >> issue that the lock did not die with the process that created it, so
>> >> >> some mechanism was needed to determine whether the lock should be
>> >> >> broken
>> >> >> or not.
>> >> >>
>> >> >> What we really need is a good OS agnostic abstraction that provides
>> >> >> the
>> >> >> ability to create a "named" lock, acquire the lock, release the
>> >> >> lock,
>> >> >> and make sure that the lock dies when the process dies, so another
>> >> >> process that is waiting can acquire the lock. This way no
>> >> >> "BreakLock"
>> >> >> functionality is required, nor do we need to try and do think like
>> >> >> remember which process owns the lock.
>> >> >>
>> >> >> So...
>> >> >>
>> >> >> We have three current operating systems we need to support:
>> >> >>
>> >> >> Linux - Ubuntu and CentOS
>> >> >> MacOS - client only - bit the CLI uses a lock for the local cache
>> >> >> Windows
>> >> >>
>> >> >> For Linux, and possibly MacOS, flock is a possibility, but can we do
>> >> >> better? Is there something that is better suited?
>> >> >>
>> >> >> For Windows, while you can create global semaphores or mutex
>> >> >> instances,
>> >> >> I'm not sure of entities that die with the process. Can people
>> >> >> recommend
>> >> >> solutions?
>> >> >>
>> >> >> Cheers,
>> >> >> Tim
>> >> >>
>> >> >> --
>> >> >> Juju-dev mailing list
>> >> >> Juju-dev@lists.ubuntu.com
>> >> >> Modify settings or unsubscribe at:
>> >> >> https://lists.ubuntu.com/mailman/listinfo/juju-dev
>> >> >
>> >> >
>> >> >
>> >> > --
>> >> > Juju-dev mailing list
>> >> > Juju-dev@lists.ubuntu.com
>> >> > Modify settings or unsubscribe at:
>> >> > https://lists.ubuntu.com/mailman/listinfo/juju-dev
>> >> >
>> >>
>> >> --
>> >> Juju-dev mailing list
>> >> Juju-dev@lists.ubuntu.com
>> >> Modify settings or unsubscribe at:
>> >> https://lists.ubuntu.com/mailman/listinfo/juju-dev

-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 

Re: utils/fslock needs to DIAF

[Andrew Wilkins]

On Tue, Dec 1, 2015 at 9:07 AM David Cheney 
wrote:

> http://0pointer.de/blog/projects/locking.html
>
> In short, opening the same file twice and asserting a lock on it will
> succeed.
>

Thanks. The article is a bit exasperated. Yes, there are problems to be
aware of, but it doesn't make them unusable in all cases.
 - Juju agents don't get installed onto NFS file systems, so doesn't matter
for the agents.
 - We're in full control of the files we're locking, we're not locking some
file like /etc/passwd where some other random bit of code in the process is
going to open/close it and release the lock by accident.
 - We don't need byte-range locking.

So only the original uncertainty remains: do we care about clients running
their home directory on an NFS share, where the NFS *server* is too old to
support flock?

Maybe a named mutex on Windows and a domain socket on *NIX is the way to
go. I'm not dead set on flock; I just want something that is simple, robust
and portable.

Cheers,
Andrew

On Tue, Dec 1, 2015 at 12:04 PM, Andrew Wilkins
>  wrote:
> > On Tue, Dec 1, 2015 at 8:57 AM David Cheney 
> > wrote:
> >>
> >> fcntl won't work in threaded go applications, it barely works in non
> >> threaded applications.
> >
> >
> > This is news to me. I've used it plenty in the past, in multi-threaded
> > programs. Please point me at relevant literature that explains where it
> > doesn't work.
> >
> >>
> >> I'm not interested in "doesn't work on windows" arguments. Yes, we
> >> have to support windows, but that doesn't mean we have to be dragged
> >> down to it's lowest common denominator.
> >
> >
> > Agreed, if we're actually crippling anything.
> >
> >>
> >> I think it's fine to develop a lock type that does the best available
> >> for each platform using conditional compilation.
> >
> >
> > Again, agreed, but only if there's something to be gained by doing this.
> I'm
> > still not convinced there is.
> >
> >> On Tue, Dec 1, 2015 at 11:47 AM, Andrew Wilkins
> >>  wrote:
> >> > On Tue, Dec 1, 2015 at 8:03 AM David Cheney <
> david.che...@canonical.com>
> >> > wrote:
> >> >>
> >> >> On Tue, Dec 1, 2015 at 11:00 AM, Andrew Wilkins
> >> >>  wrote:
> >> >> > On Tue, Dec 1, 2015 at 7:57 AM David Cheney
> >> >> > 
> >> >> > wrote:
> >> >> >>
> >> >> >> Doesn't look like there is windows support, and it uses fcntl
> >> >> >> (flock)
> >> >> >> under the hood, which is what we have now.
> >> >> >
> >> >> >
> >> >> > flock isn't the problematic thing Tim is talking about.
> utils/fslock
> >> >> > attempts to create a directory in a known location, and if it
> >> >> > succeeds,
> >> >> > it
> >> >> > "holds the lock". Unlocking means removing the directory.
> >> >>
> >> >> The problem is if the process dies/exits/goes mental while holding
> the
> >> >> lock we get into this existential gridlock where we're not sure if
> the
> >> >> process _is_ alive, so we shouldn't remove the lock, or the process
> is
> >> >> dead, so we should remove the lock.
> >> >>
> >> >> abstract unix domain sockets do not have this problem on windows;
> kill
> >> >> the process, file is removed from the abstract namespace.
> >> >
> >> >
> >> > POSIX advisory file locks (flock) do not have this problem either.
> See:
> >> > man(2) fcntl, section "Advisory record locking". When the file
> >> > descriptor is
> >> > closed, the lock is released; file descriptors are closed when the
> >> > process
> >> > dies.
> >> >
> >> > We could build a mutex on top of a unix domain socket, but then we'll
> >> > have
> >> > something completely separate for Windows. Shared named mutex? I'm not
> >> > convinced the overall solution would be any more robust, and I'm
> pretty
> >> > sure
> >> > it's going to be more complicated. Happy to be proven wrong though.
> >> >
> >> >> >
> >> >> > We would have to contribute Windows support, but it's not hard,
> I've
> >> >> > done it
> >> >> > before.
> >> >> >
> >> >> >>
> >> >> >> On Tue, Dec 1, 2015 at 10:55 AM, Casey Marshall
> >> >> >>  wrote:
> >> >> >> > How about github.com/camlistore/lock ?
> >> >> >> >
> >> >> >> > On Mon, Nov 30, 2015 at 5:43 PM, Tim Penhey
> >> >> >> > 
> >> >> >> > wrote:
> >> >> >> >>
> >> >> >> >> Hi folks,
> >> >> >> >>
> >> >> >> >> The fslock was a mistake that I added to the codebase some time
> >> >> >> >> back.
> >> >> >> >> It
> >> >> >> >> provided an overly simplistic solution to a more complex
> problem.
> >> >> >> >>
> >> >> >> >> Really the filesystem shouldn't be used as a locking mechanism.
> >> >> >> >>
> >> >> >> >> Most of the code that exists for the fslock now is working
> around
> >> >> >> >> its
> >> >> >> >> deficiencies. Instead we should be looking for a better
> >> >> >> >> replacement.
> >> >> >> >>
> >> >> >> >> Some "features" that were added 

Re: utils/fslock needs to DIAF

[David Cheney]

Doesn't look like there is windows support, and it uses fcntl (flock)
under the hood, which is what we have now.

On Tue, Dec 1, 2015 at 10:55 AM, Casey Marshall
 wrote:
> How about github.com/camlistore/lock ?
>
> On Mon, Nov 30, 2015 at 5:43 PM, Tim Penhey 
> wrote:
>>
>> Hi folks,
>>
>> The fslock was a mistake that I added to the codebase some time back. It
>> provided an overly simplistic solution to a more complex problem.
>>
>> Really the filesystem shouldn't be used as a locking mechanism.
>>
>> Most of the code that exists for the fslock now is working around its
>> deficiencies. Instead we should be looking for a better replacement.
>>
>> Some "features" that were added to fslock were added to work around the
>> issue that the lock did not die with the process that created it, so
>> some mechanism was needed to determine whether the lock should be broken
>> or not.
>>
>> What we really need is a good OS agnostic abstraction that provides the
>> ability to create a "named" lock, acquire the lock, release the lock,
>> and make sure that the lock dies when the process dies, so another
>> process that is waiting can acquire the lock. This way no "BreakLock"
>> functionality is required, nor do we need to try and do think like
>> remember which process owns the lock.
>>
>> So...
>>
>> We have three current operating systems we need to support:
>>
>> Linux - Ubuntu and CentOS
>> MacOS - client only - bit the CLI uses a lock for the local cache
>> Windows
>>
>> For Linux, and possibly MacOS, flock is a possibility, but can we do
>> better? Is there something that is better suited?
>>
>> For Windows, while you can create global semaphores or mutex instances,
>> I'm not sure of entities that die with the process. Can people recommend
>> solutions?
>>
>> Cheers,
>> Tim
>>
>> --
>> Juju-dev mailing list
>> Juju-dev@lists.ubuntu.com
>> Modify settings or unsubscribe at:
>> https://lists.ubuntu.com/mailman/listinfo/juju-dev
>
>
>
> --
> Juju-dev mailing list
> Juju-dev@lists.ubuntu.com
> Modify settings or unsubscribe at:
> https://lists.ubuntu.com/mailman/listinfo/juju-dev
>

-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Reviews for merges _from_ master to feature branches

[David Cheney]

Hello,

Why are reviewers being created for merges from master to feature
branches ? What purpose does this serve ?

Thanks

Dave

-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: utils/fslock needs to DIAF

[David Cheney]

On Tue, Dec 1, 2015 at 10:43 AM, Tim Penhey  wrote:
> Hi folks,
>
> The fslock was a mistake that I added to the codebase some time back. It
> provided an overly simplistic solution to a more complex problem.
>
> Really the filesystem shouldn't be used as a locking mechanism.
>
> Most of the code that exists for the fslock now is working around its
> deficiencies. Instead we should be looking for a better replacement.
>
> Some "features" that were added to fslock were added to work around the
> issue that the lock did not die with the process that created it, so
> some mechanism was needed to determine whether the lock should be broken
> or not.
>
> What we really need is a good OS agnostic abstraction that provides the
> ability to create a "named" lock, acquire the lock, release the lock,
> and make sure that the lock dies when the process dies, so another
> process that is waiting can acquire the lock. This way no "BreakLock"
> functionality is required, nor do we need to try and do think like
> remember which process owns the lock.
>
> So...
>
> We have three current operating systems we need to support:
>
> Linux - Ubuntu and CentOS
> MacOS - client only - bit the CLI uses a lock for the local cache
> Windows
>
> For Linux, and possibly MacOS, flock is a possibility, but can we do
> better? Is there something that is better suited?

For linux the best alternative I know of for a per process mutex is to
use a unix domain socket in the abstract namespace. These are
automatically removed when the listening socket is closed, or the
process dies, there is no need to cleanup stale pids.

For other posix systems a pidfile, or on disk unix domain socket is
probably the best we can do.

>
> For Windows, while you can create global semaphores or mutex instances,
> I'm not sure of entities that die with the process. Can people recommend
> solutions?
>
> Cheers,
> Tim
>
> --
> Juju-dev mailing list
> Juju-dev@lists.ubuntu.com
> Modify settings or unsubscribe at: 
> https://lists.ubuntu.com/mailman/listinfo/juju-dev

-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: utils/fslock needs to DIAF

[David Cheney]

Please no. The better way is to use an abstract unix domain socket to
create a mutex.

On Tue, Dec 1, 2015 at 10:56 AM, Andrew Wilkins
 wrote:
> On Tue, Dec 1, 2015 at 7:43 AM Tim Penhey  wrote:
>>
>> Hi folks,
>>
>> The fslock was a mistake that I added to the codebase some time back. It
>> provided an overly simplistic solution to a more complex problem.
>>
>> Really the filesystem shouldn't be used as a locking mechanism.
>>
>> Most of the code that exists for the fslock now is working around its
>> deficiencies. Instead we should be looking for a better replacement.
>>
>> Some "features" that were added to fslock were added to work around the
>> issue that the lock did not die with the process that created it, so
>> some mechanism was needed to determine whether the lock should be broken
>> or not.
>>
>> What we really need is a good OS agnostic abstraction that provides the
>> ability to create a "named" lock, acquire the lock, release the lock,
>> and make sure that the lock dies when the process dies, so another
>> process that is waiting can acquire the lock. This way no "BreakLock"
>> functionality is required, nor do we need to try and do think like
>> remember which process owns the lock.
>>
>> So...
>>
>> We have three current operating systems we need to support:
>>
>> Linux - Ubuntu and CentOS
>> MacOS - client only - bit the CLI uses a lock for the local cache
>> Windows
>>
>> For Linux, and possibly MacOS, flock is a possibility, but can we do
>> better? Is there something that is better suited?
>>
>> For Windows, while you can create global semaphores or mutex instances,
>> I'm not sure of entities that die with the process. Can people recommend
>> solutions?
>
>
> I've been wanting to do this for a long time (I think I've whinged in your
> vicinity about it before), but I've held off because of uncertainty about
> compatibility with NFS (which is probably a rare scenario, and only for the
> client).
>
> I think it was jam that brought up the heritage of directory locking in bzr,
> where flock doesn't work reliably over NFS. I think that's old news though.
> The manpage for flock discusses the NFS/kernel limitations of flock; since
> we don't go back past precise, we should be fine.
>
> I think flock (fcntl F_SETLK/F_GETLK) is an appropriate option for Linux and
> OS X. Windows has FileLock
> (https://msdn.microsoft.com/en-us/library/windows/desktop/aa365202(v=vs.85).aspx),
> and LockFileEx (for more control). Just as with F_SETLK, if the process
> dies, the lock is released.
>
> Cheers,
> Andrew
>
>> Cheers,
>> Tim
>>
>> --
>> Juju-dev mailing list
>> Juju-dev@lists.ubuntu.com
>> Modify settings or unsubscribe at:
>> https://lists.ubuntu.com/mailman/listinfo/juju-dev
>
>
> --
> Juju-dev mailing list
> Juju-dev@lists.ubuntu.com
> Modify settings or unsubscribe at:
> https://lists.ubuntu.com/mailman/listinfo/juju-dev
>

-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: Reviews for merges _from_ master to feature branches

[Andrew Wilkins]

On Tue, Dec 1, 2015 at 8:18 AM David Cheney 
wrote:

> Hello,
>
> Why are reviewers being created for merges from master to feature
> branches ? What purpose does this serve ?
>

I just ignore them. If anyone's not sure if they resolved a conflict
properly, they should ask for advice.


> Thanks
>
> Dave
>
> --
> Juju-dev mailing list
> Juju-dev@lists.ubuntu.com
> Modify settings or unsubscribe at:
> https://lists.ubuntu.com/mailman/listinfo/juju-dev
>
-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

utils/fslock needs to DIAF

[Tim Penhey]

Hi folks,

The fslock was a mistake that I added to the codebase some time back. It
provided an overly simplistic solution to a more complex problem.

Really the filesystem shouldn't be used as a locking mechanism.

Most of the code that exists for the fslock now is working around its
deficiencies. Instead we should be looking for a better replacement.

Some "features" that were added to fslock were added to work around the
issue that the lock did not die with the process that created it, so
some mechanism was needed to determine whether the lock should be broken
or not.

What we really need is a good OS agnostic abstraction that provides the
ability to create a "named" lock, acquire the lock, release the lock,
and make sure that the lock dies when the process dies, so another
process that is waiting can acquire the lock. This way no "BreakLock"
functionality is required, nor do we need to try and do think like
remember which process owns the lock.

So...

We have three current operating systems we need to support:

Linux - Ubuntu and CentOS
MacOS - client only - bit the CLI uses a lock for the local cache
Windows

For Linux, and possibly MacOS, flock is a possibility, but can we do
better? Is there something that is better suited?

For Windows, while you can create global semaphores or mutex instances,
I'm not sure of entities that die with the process. Can people recommend
solutions?

Cheers,
Tim

-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: utils/fslock needs to DIAF

[Andrew Wilkins]

On Tue, Dec 1, 2015 at 7:43 AM Tim Penhey  wrote:

> Hi folks,
>
> The fslock was a mistake that I added to the codebase some time back. It
> provided an overly simplistic solution to a more complex problem.
>
> Really the filesystem shouldn't be used as a locking mechanism.
>
> Most of the code that exists for the fslock now is working around its
> deficiencies. Instead we should be looking for a better replacement.
>
> Some "features" that were added to fslock were added to work around the
> issue that the lock did not die with the process that created it, so
> some mechanism was needed to determine whether the lock should be broken
> or not.
>
> What we really need is a good OS agnostic abstraction that provides the
> ability to create a "named" lock, acquire the lock, release the lock,
> and make sure that the lock dies when the process dies, so another
> process that is waiting can acquire the lock. This way no "BreakLock"
> functionality is required, nor do we need to try and do think like
> remember which process owns the lock.
>
> So...
>
> We have three current operating systems we need to support:
>
> Linux - Ubuntu and CentOS
> MacOS - client only - bit the CLI uses a lock for the local cache
> Windows
>
> For Linux, and possibly MacOS, flock is a possibility, but can we do
> better? Is there something that is better suited?
>
> For Windows, while you can create global semaphores or mutex instances,
> I'm not sure of entities that die with the process. Can people recommend
> solutions?
>

I've been wanting to do this for a long time (I think I've whinged in your
vicinity about it before), but I've held off because of uncertainty about
compatibility with NFS (which is probably a rare scenario, and only for the
client).

I think it was jam that brought up the heritage of directory locking in
bzr, where flock doesn't work reliably over NFS. I think that's old news
though. The manpage for flock discusses the NFS/kernel limitations of
flock; since we don't go back past precise, we should be fine.

I think flock (fcntl F_SETLK/F_GETLK) is an appropriate option for Linux
and OS X. Windows has FileLock (
https://msdn.microsoft.com/en-us/library/windows/desktop/aa365202(v=vs.85).aspx),
and LockFileEx (for more control). Just as with F_SETLK, if the process
dies, the lock is released.

Cheers,
Andrew

Cheers,
> Tim
>
> --
> Juju-dev mailing list
> Juju-dev@lists.ubuntu.com
> Modify settings or unsubscribe at:
> https://lists.ubuntu.com/mailman/listinfo/juju-dev
>
-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: Reviews for merges _from_ master to feature branches

[David Cheney]

It's not gonna get through the merge bot if that's the case. To
clarify, using the landing bot, thumbs up, pointless reviews that
nobody reviews and have poor descriptions, thumbs down.

On Tue, Dec 1, 2015 at 11:20 AM, Rick Harding
 wrote:
> Sanity check on merge conflicts resolved?
>
>
> On Mon, Nov 30, 2015, 7:18 PM David Cheney 
> wrote:
>>
>> Hello,
>>
>> Why are reviewers being created for merges from master to feature
>> branches ? What purpose does this serve ?
>>
>> Thanks
>>
>> Dave
>>
>> --
>> Juju-dev mailing list
>> Juju-dev@lists.ubuntu.com
>> Modify settings or unsubscribe at:
>> https://lists.ubuntu.com/mailman/listinfo/juju-dev

-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: Upgrading minimum Go version?

[David Cheney]

As of this morning, the unit tests pass on xenial (thanks to Andrew
for getting them over the hump) using Go 1.5

On Tue, Dec 1, 2015 at 3:21 AM, Curtis Hovey-Canonical
 wrote:
> On Mon, Nov 30, 2015 at 10:36 AM, John Meinel  wrote:
>> Given how often people still use "--upload-tools" for things like private
>> clouds (and is definitely the one used for local provider), I'd really worry
>> about having a jujud on your local machine that wasn't built with the same
>> toolchain as the one you get from "juju bootstrap" in other cases. Very easy
>> to end up with hard to understand/reproduce bugs.
>
> I share your concerns John.
>
> We have seen several bugs marked invalid because the agent uploads is
> essentially unknown origin because Juju is more than willing to fake a
> version when uploading. There are also reports of Juju attempting to
> build agents from source:
> https://bugs.launchpad.net/juju-core/+bug/1399606
>
> We do want everyone using the agent in streams, which have a lot more
> testing behind them. The agent (jujud) on users of the Juju stable PPA
> is the same that was used to make streams. This is not true for users
> of Ubuntu's packages, and we know Ubuntu can change its tool chain
> between the time we create official agents and it builds its packages.
> We do testing to certify they Ubuntu clients and agents are
> functionally equivalent to those in the PPA and streams. As Ubuntu
> Wily+ prefers system Go dev packages to the embedded go packages we
> provide, Ubuntu's packages will contain more divergence than Trusty.
> We are obligated to do on-demand testing to certify a change to a
> system Go dev package.
>
> Ideally, We would separate the jujud from juju-core package that
> provides the client. Users don't get a jujud when they install a
> client. Instead, Juju can get the desired agents from streams.
>
> --
> Curtis Hovey
> Canonical Cloud Development and Operations
> http://launchpad.net/~sinzui
>
> --
> Juju-dev mailing list
> Juju-dev@lists.ubuntu.com
> Modify settings or unsubscribe at: 
> https://lists.ubuntu.com/mailman/listinfo/juju-dev

-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: utils/fslock needs to DIAF

[Andrew Wilkins]

On Tue, Dec 1, 2015 at 7:57 AM David Cheney 
wrote:

> Doesn't look like there is windows support, and it uses fcntl (flock)
> under the hood, which is what we have now.
>

flock isn't the problematic thing Tim is talking about. utils/fslock
attempts to create a directory in a known location, and if it succeeds, it
"holds the lock". Unlocking means removing the directory.

We would have to contribute Windows support, but it's not hard, I've done
it before.


> On Tue, Dec 1, 2015 at 10:55 AM, Casey Marshall
>  wrote:
> > How about github.com/camlistore/lock ?
> >
> > On Mon, Nov 30, 2015 at 5:43 PM, Tim Penhey 
> > wrote:
> >>
> >> Hi folks,
> >>
> >> The fslock was a mistake that I added to the codebase some time back. It
> >> provided an overly simplistic solution to a more complex problem.
> >>
> >> Really the filesystem shouldn't be used as a locking mechanism.
> >>
> >> Most of the code that exists for the fslock now is working around its
> >> deficiencies. Instead we should be looking for a better replacement.
> >>
> >> Some "features" that were added to fslock were added to work around the
> >> issue that the lock did not die with the process that created it, so
> >> some mechanism was needed to determine whether the lock should be broken
> >> or not.
> >>
> >> What we really need is a good OS agnostic abstraction that provides the
> >> ability to create a "named" lock, acquire the lock, release the lock,
> >> and make sure that the lock dies when the process dies, so another
> >> process that is waiting can acquire the lock. This way no "BreakLock"
> >> functionality is required, nor do we need to try and do think like
> >> remember which process owns the lock.
> >>
> >> So...
> >>
> >> We have three current operating systems we need to support:
> >>
> >> Linux - Ubuntu and CentOS
> >> MacOS - client only - bit the CLI uses a lock for the local cache
> >> Windows
> >>
> >> For Linux, and possibly MacOS, flock is a possibility, but can we do
> >> better? Is there something that is better suited?
> >>
> >> For Windows, while you can create global semaphores or mutex instances,
> >> I'm not sure of entities that die with the process. Can people recommend
> >> solutions?
> >>
> >> Cheers,
> >> Tim
> >>
> >> --
> >> Juju-dev mailing list
> >> Juju-dev@lists.ubuntu.com
> >> Modify settings or unsubscribe at:
> >> https://lists.ubuntu.com/mailman/listinfo/juju-dev
> >
> >
> >
> > --
> > Juju-dev mailing list
> > Juju-dev@lists.ubuntu.com
> > Modify settings or unsubscribe at:
> > https://lists.ubuntu.com/mailman/listinfo/juju-dev
> >
>
> --
> Juju-dev mailing list
> Juju-dev@lists.ubuntu.com
> Modify settings or unsubscribe at:
> https://lists.ubuntu.com/mailman/listinfo/juju-dev
>
-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: utils/fslock needs to DIAF

[Casey Marshall]

How about github.com/camlistore/lock ?

On Mon, Nov 30, 2015 at 5:43 PM, Tim Penhey 
wrote:

> Hi folks,
>
> The fslock was a mistake that I added to the codebase some time back. It
> provided an overly simplistic solution to a more complex problem.
>
> Really the filesystem shouldn't be used as a locking mechanism.
>
> Most of the code that exists for the fslock now is working around its
> deficiencies. Instead we should be looking for a better replacement.
>
> Some "features" that were added to fslock were added to work around the
> issue that the lock did not die with the process that created it, so
> some mechanism was needed to determine whether the lock should be broken
> or not.
>
> What we really need is a good OS agnostic abstraction that provides the
> ability to create a "named" lock, acquire the lock, release the lock,
> and make sure that the lock dies when the process dies, so another
> process that is waiting can acquire the lock. This way no "BreakLock"
> functionality is required, nor do we need to try and do think like
> remember which process owns the lock.
>
> So...
>
> We have three current operating systems we need to support:
>
> Linux - Ubuntu and CentOS
> MacOS - client only - bit the CLI uses a lock for the local cache
> Windows
>
> For Linux, and possibly MacOS, flock is a possibility, but can we do
> better? Is there something that is better suited?
>
> For Windows, while you can create global semaphores or mutex instances,
> I'm not sure of entities that die with the process. Can people recommend
> solutions?
>
> Cheers,
> Tim
>
> --
> Juju-dev mailing list
> Juju-dev@lists.ubuntu.com
> Modify settings or unsubscribe at:
> https://lists.ubuntu.com/mailman/listinfo/juju-dev
>
-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: utils/fslock needs to DIAF

[Andrew Wilkins]

On Tue, Dec 1, 2015 at 7:58 AM David Cheney 
wrote:

> Please no. The better way is to use an abstract unix domain socket to
> create a mutex.
>

I don't have a problem with that, but I'd like to know why it's better.


>
> On Tue, Dec 1, 2015 at 10:56 AM, Andrew Wilkins
>  wrote:
> > On Tue, Dec 1, 2015 at 7:43 AM Tim Penhey 
> wrote:
> >>
> >> Hi folks,
> >>
> >> The fslock was a mistake that I added to the codebase some time back. It
> >> provided an overly simplistic solution to a more complex problem.
> >>
> >> Really the filesystem shouldn't be used as a locking mechanism.
> >>
> >> Most of the code that exists for the fslock now is working around its
> >> deficiencies. Instead we should be looking for a better replacement.
> >>
> >> Some "features" that were added to fslock were added to work around the
> >> issue that the lock did not die with the process that created it, so
> >> some mechanism was needed to determine whether the lock should be broken
> >> or not.
> >>
> >> What we really need is a good OS agnostic abstraction that provides the
> >> ability to create a "named" lock, acquire the lock, release the lock,
> >> and make sure that the lock dies when the process dies, so another
> >> process that is waiting can acquire the lock. This way no "BreakLock"
> >> functionality is required, nor do we need to try and do think like
> >> remember which process owns the lock.
> >>
> >> So...
> >>
> >> We have three current operating systems we need to support:
> >>
> >> Linux - Ubuntu and CentOS
> >> MacOS - client only - bit the CLI uses a lock for the local cache
> >> Windows
> >>
> >> For Linux, and possibly MacOS, flock is a possibility, but can we do
> >> better? Is there something that is better suited?
> >>
> >> For Windows, while you can create global semaphores or mutex instances,
> >> I'm not sure of entities that die with the process. Can people recommend
> >> solutions?
> >
> >
> > I've been wanting to do this for a long time (I think I've whinged in
> your
> > vicinity about it before), but I've held off because of uncertainty about
> > compatibility with NFS (which is probably a rare scenario, and only for
> the
> > client).
> >
> > I think it was jam that brought up the heritage of directory locking in
> bzr,
> > where flock doesn't work reliably over NFS. I think that's old news
> though.
> > The manpage for flock discusses the NFS/kernel limitations of flock;
> since
> > we don't go back past precise, we should be fine.
> >
> > I think flock (fcntl F_SETLK/F_GETLK) is an appropriate option for Linux
> and
> > OS X. Windows has FileLock
> > (
> https://msdn.microsoft.com/en-us/library/windows/desktop/aa365202(v=vs.85).aspx
> ),
> > and LockFileEx (for more control). Just as with F_SETLK, if the process
> > dies, the lock is released.
> >
> > Cheers,
> > Andrew
> >
> >> Cheers,
> >> Tim
> >>
> >> --
> >> Juju-dev mailing list
> >> Juju-dev@lists.ubuntu.com
> >> Modify settings or unsubscribe at:
> >> https://lists.ubuntu.com/mailman/listinfo/juju-dev
> >
> >
> > --
> > Juju-dev mailing list
> > Juju-dev@lists.ubuntu.com
> > Modify settings or unsubscribe at:
> > https://lists.ubuntu.com/mailman/listinfo/juju-dev
> >
>
-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: utils/fslock needs to DIAF

[David Cheney]

On Tue, Dec 1, 2015 at 11:00 AM, Andrew Wilkins
 wrote:
> On Tue, Dec 1, 2015 at 7:57 AM David Cheney 
> wrote:
>>
>> Doesn't look like there is windows support, and it uses fcntl (flock)
>> under the hood, which is what we have now.
>
>
> flock isn't the problematic thing Tim is talking about. utils/fslock
> attempts to create a directory in a known location, and if it succeeds, it
> "holds the lock". Unlocking means removing the directory.

The problem is if the process dies/exits/goes mental while holding the
lock we get into this existential gridlock where we're not sure if the
process _is_ alive, so we shouldn't remove the lock, or the process is
dead, so we should remove the lock.

abstract unix domain sockets do not have this problem on windows; kill
the process, file is removed from the abstract namespace.

>
> We would have to contribute Windows support, but it's not hard, I've done it
> before.
>
>>
>> On Tue, Dec 1, 2015 at 10:55 AM, Casey Marshall
>>  wrote:
>> > How about github.com/camlistore/lock ?
>> >
>> > On Mon, Nov 30, 2015 at 5:43 PM, Tim Penhey 
>> > wrote:
>> >>
>> >> Hi folks,
>> >>
>> >> The fslock was a mistake that I added to the codebase some time back.
>> >> It
>> >> provided an overly simplistic solution to a more complex problem.
>> >>
>> >> Really the filesystem shouldn't be used as a locking mechanism.
>> >>
>> >> Most of the code that exists for the fslock now is working around its
>> >> deficiencies. Instead we should be looking for a better replacement.
>> >>
>> >> Some "features" that were added to fslock were added to work around the
>> >> issue that the lock did not die with the process that created it, so
>> >> some mechanism was needed to determine whether the lock should be
>> >> broken
>> >> or not.
>> >>
>> >> What we really need is a good OS agnostic abstraction that provides the
>> >> ability to create a "named" lock, acquire the lock, release the lock,
>> >> and make sure that the lock dies when the process dies, so another
>> >> process that is waiting can acquire the lock. This way no "BreakLock"
>> >> functionality is required, nor do we need to try and do think like
>> >> remember which process owns the lock.
>> >>
>> >> So...
>> >>
>> >> We have three current operating systems we need to support:
>> >>
>> >> Linux - Ubuntu and CentOS
>> >> MacOS - client only - bit the CLI uses a lock for the local cache
>> >> Windows
>> >>
>> >> For Linux, and possibly MacOS, flock is a possibility, but can we do
>> >> better? Is there something that is better suited?
>> >>
>> >> For Windows, while you can create global semaphores or mutex instances,
>> >> I'm not sure of entities that die with the process. Can people
>> >> recommend
>> >> solutions?
>> >>
>> >> Cheers,
>> >> Tim
>> >>
>> >> --
>> >> Juju-dev mailing list
>> >> Juju-dev@lists.ubuntu.com
>> >> Modify settings or unsubscribe at:
>> >> https://lists.ubuntu.com/mailman/listinfo/juju-dev
>> >
>> >
>> >
>> > --
>> > Juju-dev mailing list
>> > Juju-dev@lists.ubuntu.com
>> > Modify settings or unsubscribe at:
>> > https://lists.ubuntu.com/mailman/listinfo/juju-dev
>> >
>>
>> --
>> Juju-dev mailing list
>> Juju-dev@lists.ubuntu.com
>> Modify settings or unsubscribe at:
>> https://lists.ubuntu.com/mailman/listinfo/juju-dev

-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: utils/fslock needs to DIAF

[Andrew Wilkins]

On Tue, Dec 1, 2015 at 8:03 AM David Cheney 
wrote:

> On Tue, Dec 1, 2015 at 11:00 AM, Andrew Wilkins
>  wrote:
> > On Tue, Dec 1, 2015 at 7:57 AM David Cheney 
> > wrote:
> >>
> >> Doesn't look like there is windows support, and it uses fcntl (flock)
> >> under the hood, which is what we have now.
> >
> >
> > flock isn't the problematic thing Tim is talking about. utils/fslock
> > attempts to create a directory in a known location, and if it succeeds,
> it
> > "holds the lock". Unlocking means removing the directory.
>
> The problem is if the process dies/exits/goes mental while holding the
> lock we get into this existential gridlock where we're not sure if the
> process _is_ alive, so we shouldn't remove the lock, or the process is
> dead, so we should remove the lock.
>
> abstract unix domain sockets do not have this problem on windows; kill
> the process, file is removed from the abstract namespace.
>

POSIX advisory file locks (flock) do not have this problem either. See:
man(2) fcntl, section "Advisory record locking". When the file descriptor
is closed, the lock is released; file descriptors are closed when the
process dies.

We could build a mutex on top of a unix domain socket, but then we'll have
something completely separate for Windows. Shared named mutex? I'm not
convinced the overall solution would be any more robust, and I'm pretty
sure it's going to be more complicated. Happy to be proven wrong though.

>
> > We would have to contribute Windows support, but it's not hard, I've
> done it
> > before.
> >
> >>
> >> On Tue, Dec 1, 2015 at 10:55 AM, Casey Marshall
> >>  wrote:
> >> > How about github.com/camlistore/lock ?
> >> >
> >> > On Mon, Nov 30, 2015 at 5:43 PM, Tim Penhey  >
> >> > wrote:
> >> >>
> >> >> Hi folks,
> >> >>
> >> >> The fslock was a mistake that I added to the codebase some time back.
> >> >> It
> >> >> provided an overly simplistic solution to a more complex problem.
> >> >>
> >> >> Really the filesystem shouldn't be used as a locking mechanism.
> >> >>
> >> >> Most of the code that exists for the fslock now is working around its
> >> >> deficiencies. Instead we should be looking for a better replacement.
> >> >>
> >> >> Some "features" that were added to fslock were added to work around
> the
> >> >> issue that the lock did not die with the process that created it, so
> >> >> some mechanism was needed to determine whether the lock should be
> >> >> broken
> >> >> or not.
> >> >>
> >> >> What we really need is a good OS agnostic abstraction that provides
> the
> >> >> ability to create a "named" lock, acquire the lock, release the lock,
> >> >> and make sure that the lock dies when the process dies, so another
> >> >> process that is waiting can acquire the lock. This way no "BreakLock"
> >> >> functionality is required, nor do we need to try and do think like
> >> >> remember which process owns the lock.
> >> >>
> >> >> So...
> >> >>
> >> >> We have three current operating systems we need to support:
> >> >>
> >> >> Linux - Ubuntu and CentOS
> >> >> MacOS - client only - bit the CLI uses a lock for the local cache
> >> >> Windows
> >> >>
> >> >> For Linux, and possibly MacOS, flock is a possibility, but can we do
> >> >> better? Is there something that is better suited?
> >> >>
> >> >> For Windows, while you can create global semaphores or mutex
> instances,
> >> >> I'm not sure of entities that die with the process. Can people
> >> >> recommend
> >> >> solutions?
> >> >>
> >> >> Cheers,
> >> >> Tim
> >> >>
> >> >> --
> >> >> Juju-dev mailing list
> >> >> Juju-dev@lists.ubuntu.com
> >> >> Modify settings or unsubscribe at:
> >> >> https://lists.ubuntu.com/mailman/listinfo/juju-dev
> >> >
> >> >
> >> >
> >> > --
> >> > Juju-dev mailing list
> >> > Juju-dev@lists.ubuntu.com
> >> > Modify settings or unsubscribe at:
> >> > https://lists.ubuntu.com/mailman/listinfo/juju-dev
> >> >
> >>
> >> --
> >> Juju-dev mailing list
> >> Juju-dev@lists.ubuntu.com
> >> Modify settings or unsubscribe at:
> >> https://lists.ubuntu.com/mailman/listinfo/juju-dev
>
-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: utils/fslock needs to DIAF

[David Cheney]

http://0pointer.de/blog/projects/locking.html

In short, opening the same file twice and asserting a lock on it will succeed.

On Tue, Dec 1, 2015 at 12:04 PM, Andrew Wilkins
 wrote:
> On Tue, Dec 1, 2015 at 8:57 AM David Cheney 
> wrote:
>>
>> fcntl won't work in threaded go applications, it barely works in non
>> threaded applications.
>
>
> This is news to me. I've used it plenty in the past, in multi-threaded
> programs. Please point me at relevant literature that explains where it
> doesn't work.
>
>>
>> I'm not interested in "doesn't work on windows" arguments. Yes, we
>> have to support windows, but that doesn't mean we have to be dragged
>> down to it's lowest common denominator.
>
>
> Agreed, if we're actually crippling anything.
>
>>
>> I think it's fine to develop a lock type that does the best available
>> for each platform using conditional compilation.
>
>
> Again, agreed, but only if there's something to be gained by doing this. I'm
> still not convinced there is.
>
>> On Tue, Dec 1, 2015 at 11:47 AM, Andrew Wilkins
>>  wrote:
>> > On Tue, Dec 1, 2015 at 8:03 AM David Cheney 
>> > wrote:
>> >>
>> >> On Tue, Dec 1, 2015 at 11:00 AM, Andrew Wilkins
>> >>  wrote:
>> >> > On Tue, Dec 1, 2015 at 7:57 AM David Cheney
>> >> > 
>> >> > wrote:
>> >> >>
>> >> >> Doesn't look like there is windows support, and it uses fcntl
>> >> >> (flock)
>> >> >> under the hood, which is what we have now.
>> >> >
>> >> >
>> >> > flock isn't the problematic thing Tim is talking about. utils/fslock
>> >> > attempts to create a directory in a known location, and if it
>> >> > succeeds,
>> >> > it
>> >> > "holds the lock". Unlocking means removing the directory.
>> >>
>> >> The problem is if the process dies/exits/goes mental while holding the
>> >> lock we get into this existential gridlock where we're not sure if the
>> >> process _is_ alive, so we shouldn't remove the lock, or the process is
>> >> dead, so we should remove the lock.
>> >>
>> >> abstract unix domain sockets do not have this problem on windows; kill
>> >> the process, file is removed from the abstract namespace.
>> >
>> >
>> > POSIX advisory file locks (flock) do not have this problem either. See:
>> > man(2) fcntl, section "Advisory record locking". When the file
>> > descriptor is
>> > closed, the lock is released; file descriptors are closed when the
>> > process
>> > dies.
>> >
>> > We could build a mutex on top of a unix domain socket, but then we'll
>> > have
>> > something completely separate for Windows. Shared named mutex? I'm not
>> > convinced the overall solution would be any more robust, and I'm pretty
>> > sure
>> > it's going to be more complicated. Happy to be proven wrong though.
>> >
>> >> >
>> >> > We would have to contribute Windows support, but it's not hard, I've
>> >> > done it
>> >> > before.
>> >> >
>> >> >>
>> >> >> On Tue, Dec 1, 2015 at 10:55 AM, Casey Marshall
>> >> >>  wrote:
>> >> >> > How about github.com/camlistore/lock ?
>> >> >> >
>> >> >> > On Mon, Nov 30, 2015 at 5:43 PM, Tim Penhey
>> >> >> > 
>> >> >> > wrote:
>> >> >> >>
>> >> >> >> Hi folks,
>> >> >> >>
>> >> >> >> The fslock was a mistake that I added to the codebase some time
>> >> >> >> back.
>> >> >> >> It
>> >> >> >> provided an overly simplistic solution to a more complex problem.
>> >> >> >>
>> >> >> >> Really the filesystem shouldn't be used as a locking mechanism.
>> >> >> >>
>> >> >> >> Most of the code that exists for the fslock now is working around
>> >> >> >> its
>> >> >> >> deficiencies. Instead we should be looking for a better
>> >> >> >> replacement.
>> >> >> >>
>> >> >> >> Some "features" that were added to fslock were added to work
>> >> >> >> around
>> >> >> >> the
>> >> >> >> issue that the lock did not die with the process that created it,
>> >> >> >> so
>> >> >> >> some mechanism was needed to determine whether the lock should be
>> >> >> >> broken
>> >> >> >> or not.
>> >> >> >>
>> >> >> >> What we really need is a good OS agnostic abstraction that
>> >> >> >> provides
>> >> >> >> the
>> >> >> >> ability to create a "named" lock, acquire the lock, release the
>> >> >> >> lock,
>> >> >> >> and make sure that the lock dies when the process dies, so
>> >> >> >> another
>> >> >> >> process that is waiting can acquire the lock. This way no
>> >> >> >> "BreakLock"
>> >> >> >> functionality is required, nor do we need to try and do think
>> >> >> >> like
>> >> >> >> remember which process owns the lock.
>> >> >> >>
>> >> >> >> So...
>> >> >> >>
>> >> >> >> We have three current operating systems we need to support:
>> >> >> >>
>> >> >> >> Linux - Ubuntu and CentOS
>> >> >> >> MacOS - client only - bit the CLI uses a lock for the local cache
>> >> >> >> Windows
>> >> >> >>
>> >> >> >> For Linux, and possibly MacOS, 

Re: Upgrading minimum Go version?

[Curtis Hovey-Canonical]

Good discussion. I have one nuance that I think we want to discuss in
SFO. Namely, controlling the Juju tool chain.

Juju QA uses CI to make many of the things we releases, such as win
agents. We use Launchpad/Ubuntu to make Ubuntu agents and clients.
Juju QA qill soon have access to ARM hardware. When we do, we can
choose to by-pass Lp/Ubuntu for agents. The agents that were built and
tested by CI are the agents we release in streams. This permits us to
choose the best tool chain to create each series+arch combination.
This also introduces drift between what Juju officially releases, and
and Ubuntu releases.

On Thu, Nov 26, 2015 at 8:27 PM, Andrew Wilkins
 wrote:
> On Fri, Nov 27, 2015 at 7:49 AM Michael Hudson-Doyle
>  wrote:
>>
>> On 27 November 2015 at 09:39, Tim Penhey  wrote:
>> > On 27/11/15 08:43, Michael Hudson-Doyle wrote:
>> >> On 27 November 2015 at 02:24, Martin Packman
>> >>  wrote:
>> >>> On 26/11/2015, Andrew Wilkins  wrote:
>>  Hi (mostly Curtis),
>> 
>>  Is there a plan to bump the minimum Go version? Some of our
>>  dependencies do
>>  not build with Go 1.2. The LXD provider only builds with Go 1.3 (I
>>  think?),
>>  and I've got a PR up that updates the azure-sdk-for-go dependency,
>>  but it's
>>  blocked because the newer doesn't build with Go 1.2.
>> >>
>> >> Is this something we've done to ourselves or is there a third-party
>> >> library we're depending on that doesn't work with Go 1.2?
>> >
>> > The two main ones I know about are lxd and the new azure go bindings.
>>
>> By the azure go bindings you mean something Canonical didn't write,
>> like https://github.com/Azure/azure-sdk-for-go? That sort of thing
>> sounds like a good argument for the 1.5-in-trusty SRU thing.




-- 
Curtis Hovey
Canonical Cloud Development and Operations
http://launchpad.net/~sinzui

-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev