Re: [Proposal] Core Bloodhound - basic concepts

[Gary Martin]

On Mon, 19 Mar 2018, at 2:09 PM, Greg Stein wrote:
> On Mon, Mar 19, 2018 at 8:43 AM, Gary Martin 
> wrote:
> 
> > Hi everyone,
> >
> > I am going to start building out a basic core implementation to play
> > around with some of the ideas we have discussed. I'll try not to take
> > things too far before getting this into source control.
> 
> 
> I would recommend using an svn branch, and committing "hourly". Why not?
> You won't be affecting trunk, and you'll get early feedback.

That is true. I can obviously avoid the need to discover how to create the git 
repo by just using a branch for the experiment of course.

> >...
> 
> > On that last item, I believe I have the ability to create a git repo for
> > the project at the asf. I would expect the project to also be available via
> > github.
> >
> 
> Infra would like to see some community consensus, if you plan to switch
> from svn to git. (ie. discuss it first)

You may well be right that there has not been enough discussion on this 
although it has been mentioned a number of times. I will make it a firm 
proposal again before taking that suggestion any further.
 
Cheers,
Gary

Re: [Proposal] Core Bloodhound - basic concepts

[Greg Stein]

On Mon, Mar 19, 2018 at 8:43 AM, Gary Martin 
wrote:

> Hi everyone,
>
> I am going to start building out a basic core implementation to play
> around with some of the ideas we have discussed. I'll try not to take
> things too far before getting this into source control.


I would recommend using an svn branch, and committing "hourly". Why not?
You won't be affecting trunk, and you'll get early feedback.

>...

> On that last item, I believe I have the ability to create a git repo for
> the project at the asf. I would expect the project to also be available via
> github.
>

Infra would like to see some community consensus, if you plan to switch
from svn to git. (ie. discuss it first)

>...

Cheers,
-g

Re: [Proposal] Core Bloodhound - basic concepts

[Gary Martin]

Hi everyone,

I am going to start building out a basic core implementation to play around 
with some of the ideas we have discussed. I'll try not to take things too far 
before getting this into source control. I am having a quick play with pipenv 
[1] as the new base for helping to develop and install the project. I'll also 
be hoping to get unit testing and some selenium tests in place relatively early 
along with some basic instructions for getting started with development.

At the moment I am expecting that we will be working with python >=3.5, we are 
using django >= 2.0, unit testing probably to be based on pytest [2] and that 
there is a desire to use a git repo for the work.

On that last item, I believe I have the ability to create a git repo for the 
project at the asf. I would expect the project to also be available via github.

Anything else I may have missed?

Cheers,
Gary

[1] https://docs.pipenv.org/
[2] https://docs.pytest.org/


On Thu, 15 Mar 2018, at 1:01 AM, Gary Martin wrote:
> On Mon, 12 Mar 2018, at 12:10 PM, Jason Morgan wrote:
> > Rather than Celery, [...]
> 
> I was assuming that Allan was referring to the work of constructing the 
> current state of a ticket. I may have misunderstood of course. I was 
> more expecting to go down the route of cached state when we want to 
> start addressing the limitations of speed but this is not to say that 
> making use of some kind of message queue could be in scope. I'm seeing 
> this for the moment as an implementation detail.
> 
> That said, using a message queue could be one effective way of bringing 
> a separation between a core project and a ui project. In that model we 
> could allow there to be a server for the ui that acts as an intermediary 
> and provides a websockets implementation assuming that is required. A 
> message queue could also be an effective way of implementing 
> notifications for ticket change events and similar.
> 
> I have played with zeromq a bit which is good fun... but I understand we 
> can't use it due to its current choice of license. I mostly mention it 
> as at the moment I sort of prefer the socket library approach rather 
> than requiring a message broker. I wouldn't be sure that nanomsg is a 
> viable alternative.
> 
> > [...] I've had a lot of experience and success with Django 
> > Channels (websockets).
> > 
> > See https://channels.readthedocs.io
> > And my post here: 
> > https://stackoverflow.com/questions/42815835/how-to-make-webpage-to-show-real-time-values-sent-from-raspberry-pi/42817627#42817627
> > 
> > Websockets are supported natively by all modern browsers so it makes a 
> > very portable solution with a low client overhead and with Channels, all 
> > the benefits of Django.
> 
> Good stuff. I am certainly happy with use of websockets for the main 
> frontend project. Again, as I suggest above, websockets might not make 
> it into the core project but I don't think that should be seen as a big 
> deal. If a frontend project wants to make use of them then we can make 
> that happen in some part of the architecture.
> 
> One thing that this does bring up is the support for reduced browsers 
> with reduced functionality. I can see the benefit of not having to 
> consider older browsers but it would be nice if ticket management is 
> possible in some form for older browsers. The lack of support for some 
> ways of interacting might not be seen as all that problematic though.
> 
> > j.
> > 
> > 
> > -Original Message-
> > From: Allan Swanepoel [mailto:allanice...@gmail.com] 
> > Sent: 11 March 2018 21:36
> > To: dev@bloodhound.apache.org
> > Subject: Re: [Proposal] Core Bloodhound - basic concepts
> > 
> > Possibly take a look at what has already been done using Django in this 
> > arena?
> >  One project that jumps out with some Google foo is 
> > https://djacket.github.io/
> > 
> > Also, if you want to separate (loosely couple) the front-end and the 
> > back-end, possibly build the whole front-end with some js framework like 
> > react?
> > All the functionality would be provided by the api, and all you need is 
> > some Ajax calls.
> > 
> > For the slow stuff, possibly make these more asynchronous and use some 
> > form of scheduler (celery / ampq bus)  to offload the work. Again, the 
> > result is posted back via the api.
> > 
> > Just my 2c's worth
> > 
> > On 11 Mar 2018 23:21, "Gary" <gary.mar...@physics.org> wrote:
> > 
> > > Hi everyone,
> > >
> > > Sorry it has taken so long to get this written. With the understanding 
> > > that we have expressed the desire to move away from T

Re: [Proposal] Core Bloodhound - basic concepts

[Gary Martin]

On Mon, 12 Mar 2018, at 12:10 PM, Jason Morgan wrote:
> Rather than Celery, [...]

I was assuming that Allan was referring to the work of constructing the current 
state of a ticket. I may have misunderstood of course. I was more expecting to 
go down the route of cached state when we want to start addressing the 
limitations of speed but this is not to say that making use of some kind of 
message queue could be in scope. I'm seeing this for the moment as an 
implementation detail.

That said, using a message queue could be one effective way of bringing a 
separation between a core project and a ui project. In that model we could 
allow there to be a server for the ui that acts as an intermediary and provides 
a websockets implementation assuming that is required. A message queue could 
also be an effective way of implementing notifications for ticket change events 
and similar.

I have played with zeromq a bit which is good fun... but I understand we can't 
use it due to its current choice of license. I mostly mention it as at the 
moment I sort of prefer the socket library approach rather than requiring a 
message broker. I wouldn't be sure that nanomsg is a viable alternative.

> [...] I've had a lot of experience and success with Django 
> Channels (websockets).
> 
> See https://channels.readthedocs.io
> And my post here: 
> https://stackoverflow.com/questions/42815835/how-to-make-webpage-to-show-real-time-values-sent-from-raspberry-pi/42817627#42817627
> 
> Websockets are supported natively by all modern browsers so it makes a 
> very portable solution with a low client overhead and with Channels, all 
> the benefits of Django.

Good stuff. I am certainly happy with use of websockets for the main frontend 
project. Again, as I suggest above, websockets might not make it into the core 
project but I don't think that should be seen as a big deal. If a frontend 
project wants to make use of them then we can make that happen in some part of 
the architecture.

One thing that this does bring up is the support for reduced browsers with 
reduced functionality. I can see the benefit of not having to consider older 
browsers but it would be nice if ticket management is possible in some form for 
older browsers. The lack of support for some ways of interacting might not be 
seen as all that problematic though.

> j.
> 
> 
> -Original Message-
> From: Allan Swanepoel [mailto:allanice...@gmail.com] 
> Sent: 11 March 2018 21:36
> To: dev@bloodhound.apache.org
> Subject: Re: [Proposal] Core Bloodhound - basic concepts
> 
> Possibly take a look at what has already been done using Django in this arena?
>  One project that jumps out with some Google foo is https://djacket.github.io/
> 
> Also, if you want to separate (loosely couple) the front-end and the 
> back-end, possibly build the whole front-end with some js framework like 
> react?
> All the functionality would be provided by the api, and all you need is 
> some Ajax calls.
> 
> For the slow stuff, possibly make these more asynchronous and use some 
> form of scheduler (celery / ampq bus)  to offload the work. Again, the 
> result is posted back via the api.
> 
> Just my 2c's worth
> 
> On 11 Mar 2018 23:21, "Gary" <gary.mar...@physics.org> wrote:
> 
> > Hi everyone,
> >
> > Sorry it has taken so long to get this written. With the understanding 
> > that we have expressed the desire to move away from Trac as the base 
> > for the project,  I'd like to put forward the following as an initial 
> > proposal for the way we progress. You should not expect that this is 
> > high on details. I only intend to provide some guidance around the 
> > concepts that I would like to see used. I am hoping that this will be 
> > enough to get a reasonable idea of what the core model might look like.
> >
> > In broad terms I am proposing that we
> > * take the opportunity to design the project from the ground up as 
> > opposed to attempting to build up the existing interfaces
> > * build the project using the Django web framework
> > * separate out a core sub-project with a well defined REST api for a 
> > frontend UI sub-project to build upon
> >
> > The rest of this proposal sets out to define parts of the core sub-project.
> >
> > The core should provide enough services to support the main part of a 
> > issue tracking UI. I am also expecting there to be further 
> > sub-projects to support additional optional functionality.
> >
> > The core sub-project will probably provide a basic UI to make progress 
> > ahead of work on the UI sub-project. I am not looking to define the UI 
> > sub-project any further in this proposal although it is likely that it 
> > will be h

RE: [Proposal] Core Bloodhound - basic concepts

[Jason Morgan]

Rather than Celery, I've had a lot of experience and success with Django 
Channels (websockets).

See https://channels.readthedocs.io
And my post here: 
https://stackoverflow.com/questions/42815835/how-to-make-webpage-to-show-real-time-values-sent-from-raspberry-pi/42817627#42817627

Websockets are supported natively by all modern browsers so it makes a very 
portable solution with a low client overhead and with Channels, all the 
benefits of Django.

j.


-Original Message-
From: Allan Swanepoel [mailto:allanice...@gmail.com] 
Sent: 11 March 2018 21:36
To: dev@bloodhound.apache.org
Subject: Re: [Proposal] Core Bloodhound - basic concepts

Possibly take a look at what has already been done using Django in this arena?
 One project that jumps out with some Google foo is https://djacket.github.io/

Also, if you want to separate (loosely couple) the front-end and the back-end, 
possibly build the whole front-end with some js framework like react?
All the functionality would be provided by the api, and all you need is some 
Ajax calls.

For the slow stuff, possibly make these more asynchronous and use some form of 
scheduler (celery / ampq bus)  to offload the work. Again, the result is posted 
back via the api.

Just my 2c's worth

On 11 Mar 2018 23:21, "Gary" <gary.mar...@physics.org> wrote:

> Hi everyone,
>
> Sorry it has taken so long to get this written. With the understanding 
> that we have expressed the desire to move away from Trac as the base 
> for the project,  I'd like to put forward the following as an initial 
> proposal for the way we progress. You should not expect that this is 
> high on details. I only intend to provide some guidance around the 
> concepts that I would like to see used. I am hoping that this will be 
> enough to get a reasonable idea of what the core model might look like.
>
> In broad terms I am proposing that we
> * take the opportunity to design the project from the ground up as 
> opposed to attempting to build up the existing interfaces
> * build the project using the Django web framework
> * separate out a core sub-project with a well defined REST api for a 
> frontend UI sub-project to build upon
>
> The rest of this proposal sets out to define parts of the core sub-project.
>
> The core should provide enough services to support the main part of a 
> issue tracking UI. I am also expecting there to be further 
> sub-projects to support additional optional functionality.
>
> The core sub-project will probably provide a basic UI to make progress 
> ahead of work on the UI sub-project. I am not looking to define the UI 
> sub-project any further in this proposal although it is likely that it 
> will be highly dynamic, perhaps making use of websockets. The basic UI 
> of the core project should remain well-maintained throughout the life 
> of the project but at the moment my expectation would be that it is a 
> no-thrills interface.
>
> Back to the core sub-project, I am looking for this to be built on the 
> following concepts:
>
>  #0 a change in nomenclature from product to tracker as the top level
>  #1 uuids for pretty much everything
>  #2 tickets as the accumulation of ticket change events
>  #3 'labels' and 'label types' as a unifying concept for 
> categorisation of tickets
>
> One of the early decisions made in handling of multiple projects in 
> bloodhound was to call this feature multi-product. The reasoning for 
> this was that the term 'project' is overloaded (if I recall, there may 
> have been an excuse to add projects as a layer in bloodhound that 
> could be used to group tickets across products.) I would like to see 
> the basic functionality brought in to handle the ability to host 
> multiple projects (in the top level organisation sense) but change the 
> nomenclature to tracker to be as generic as possible. To be fair, the 
> name of the feature may not matter so much as I would also like to see 
> users given the ability to define their levels of organisation to a 
> greater extent. More on this in the discussion of #3.
>
> Note that I will use the term tracker in the rest of this proposal 
> where you might prefer to see product or project.
>
> The idea behind #1, uuids, is that it may be useful to be able to 
> unambiguously identify instances of objects even across bloodhound 
> instances. Pending decisions on how tickets are referenced, these 
> uuids could be used as a reference that is unchanging as it moved 
> between trackers.
>
> How tickets are stored as a whole is also worth tackling, which is 
> what #2 is trying to broadly decide. Here I am suggesting that the 
> state of a ticket can be built up from a query that collects all the 
> change events and applies the deltas in order. This could prove to be 
> a slow proce

Re: [Proposal] Core Bloodhound - basic concepts

[Greg Stein]

On Sun, Mar 11, 2018 at 4:21 PM, Gary  wrote:
>...

> How tickets are stored as a whole is also worth tackling, which is what #2
> is trying to broadly decide. Here I am suggesting that the state of a
> ticket can be built up from a query that collects all the change events and
> applies the deltas in order. This could prove to be a slow process so at
> some point we may want to look at keeping a record of the current state or
> a checkpoint but, given that ticket views are expected to show the history,
> these are details that are required anyway. I suggest that we can look at
> optimising for speed later. Knowing that we can update a ticket from deltas
> may be useful.
>

The Apache Subversion team has a LOT of experience in this kind of storage
:-)

When we started, we stored the "latest" in full text, and then had a series
of "reverse deltas" to go backwards in time. We switched that around, and
store the original in full text, and then apply "forward" deltas to reach
"latest".

We have two mechanisms to speed this up: a sophisticated cache system.
Invariably, "latest" will be cached and fast to retrieve. The more
important part of our "series of delta" system, allowing us to rapidly
construct any point in history is to use "skip deltas". This is analogous
to the "skip list"[1] concept, where we assemble N deltas into a single
delta allowing us to skip over many deltas with a single application.
Generally speaking, if there are M deltas in a file's entire history, then
we can reassemble any point in history by applying log(M) deltas.

(note we used the skip delta mechanism for both directions; it is effective
in both directions)

Finally, I am suggesting in #3 that as much as possible we generalise
> ticket categorisation. Categorisation is a central concept to capture and,
> in trac we inherited categorisations such as statuses (open, in progress,
> etc), types (bug, enhancement, task, etc), milestones, versions, etc, and
> these had separate implementations.


When my team designed the issue tracker for Google Code's project hosting,
we did the same thing. Most issue trackers are highly-structured with a
field or this and that. It complicates issue creation, issue management,
querying, etc. I recall sitting down with 20 fields from a typical Bugzilla
install, and reducing it to about 8, where we used labels for "everything".
We did not bother with "issue types".

A second thing that we did is allow labels such as "milestone-14", and then
enable our "list display" to be configured for a "milestone" column that
would extract any labels that started with "milestone-", showing just the
"14".

It made for a very robust, easy to understand, and flexible metadata system
for each issue.

You can still see some of these design choices a decade later in Monorail,
a descendent of our tracker, now being used by the Chromium project.

>...

Cheers,
-g

[1] https://en.wikipedia.org/wiki/Skip_list

Re: [Proposal] Core Bloodhound - basic concepts

[Gary]

Hi Allan,

Thank you for the suggestions. Please take my word for it that the input is 
definitely appreciated.

I do want to delay discussion on how the front-end could work, or at least 
limit that discussion in this thread. The main thing I want to focus on here is 
getting agreement on the core platform. OK, temptation gets the better of me.. 
hopefully it is enough to say that I would see React as a reasonable choice for 
helping build the web front-end but Allan appears to have the gist of what I am 
expecting.

I'm also not hugely worried about anything being particularly slow for the 
moment. Slow operations will be possible to speed up later. I will attempt to 
keep the scheduler method in mind for the future as I might not have come up 
with that as an obvious option!

Cheers,
Gary

On Sun, 11 Mar 2018, at 9:35 PM, Allan Swanepoel wrote:
> Possibly take a look at what has already been done using Django in this
> arena?
>  One project that jumps out with some Google foo is
> https://djacket.github.io/
> 
> Also, if you want to separate (loosely couple) the front-end and the
> back-end, possibly build the whole front-end with some js framework like
> react?
> All the functionality would be provided by the api, and all you need is
> some Ajax calls.
> 
> For the slow stuff, possibly make these more asynchronous and use some form
> of scheduler (celery / ampq bus)  to offload the work. Again, the result is
> posted back via the api.
> 
> Just my 2c's worth
> 
> On 11 Mar 2018 23:21, "Gary"  wrote:
> 
> > Hi everyone,
> >
> > Sorry it has taken so long to get this written. With the understanding
> > that we have expressed the desire to move away from Trac as the base for
> > the project,  I'd like to put forward the following as an initial proposal
> > for the way we progress. You should not expect that this is high on
> > details. I only intend to provide some guidance around the concepts that I
> > would like to see used. I am hoping that this will be enough to get a
> > reasonable idea of what the core model might look like.
> >
> > In broad terms I am proposing that we
> > * take the opportunity to design the project from the ground up as opposed
> > to attempting to build up the existing interfaces
> > * build the project using the Django web framework
> > * separate out a core sub-project with a well defined REST api for a
> > frontend UI sub-project to build upon
> >
> > The rest of this proposal sets out to define parts of the core sub-project.
> >
> > The core should provide enough services to support the main part of a
> > issue tracking UI. I am also expecting there to be further sub-projects to
> > support additional optional functionality.
> >
> > The core sub-project will probably provide a basic UI to make progress
> > ahead of work on the UI sub-project. I am not looking to define the UI
> > sub-project any further in this proposal although it is likely that it will
> > be highly dynamic, perhaps making use of websockets. The basic UI of the
> > core project should remain well-maintained throughout the life of the
> > project but at the moment my expectation would be that it is a no-thrills
> > interface.
> >
> > Back to the core sub-project, I am looking for this to be built on the
> > following concepts:
> >
> >  #0 a change in nomenclature from product to tracker as the top level
> >  #1 uuids for pretty much everything
> >  #2 tickets as the accumulation of ticket change events
> >  #3 'labels' and 'label types' as a unifying concept for categorisation of
> > tickets
> >
> > One of the early decisions made in handling of multiple projects in
> > bloodhound was to call this feature multi-product. The reasoning for this
> > was that the term 'project' is overloaded (if I recall, there may have been
> > an excuse to add projects as a layer in bloodhound that could be used to
> > group tickets across products.) I would like to see the basic functionality
> > brought in to handle the ability to host multiple projects (in the top
> > level organisation sense) but change the nomenclature to tracker to be as
> > generic as possible. To be fair, the name of the feature may not matter so
> > much as I would also like to see users given the ability to define their
> > levels of organisation to a greater extent. More on this in the discussion
> > of #3.
> >
> > Note that I will use the term tracker in the rest of this proposal where
> > you might prefer to see product or project.
> >
> > The idea behind #1, uuids, is that it may be useful to be able to
> > unambiguously identify instances of objects even across bloodhound
> > instances. Pending decisions on how tickets are referenced, these uuids
> > could be used as a reference that is unchanging as it moved between
> > trackers.
> >
> > How tickets are stored as a whole is also worth tackling, which is what #2
> > is trying to broadly decide. Here I am suggesting that the state of a
> > ticket can be built up 

Re: [Proposal] Core Bloodhound - basic concepts

[Allan Swanepoel]

Possibly take a look at what has already been done using Django in this
arena?
 One project that jumps out with some Google foo is
https://djacket.github.io/

Also, if you want to separate (loosely couple) the front-end and the
back-end, possibly build the whole front-end with some js framework like
react?
All the functionality would be provided by the api, and all you need is
some Ajax calls.

For the slow stuff, possibly make these more asynchronous and use some form
of scheduler (celery / ampq bus)  to offload the work. Again, the result is
posted back via the api.

Just my 2c's worth

On 11 Mar 2018 23:21, "Gary"  wrote:

> Hi everyone,
>
> Sorry it has taken so long to get this written. With the understanding
> that we have expressed the desire to move away from Trac as the base for
> the project,  I'd like to put forward the following as an initial proposal
> for the way we progress. You should not expect that this is high on
> details. I only intend to provide some guidance around the concepts that I
> would like to see used. I am hoping that this will be enough to get a
> reasonable idea of what the core model might look like.
>
> In broad terms I am proposing that we
> * take the opportunity to design the project from the ground up as opposed
> to attempting to build up the existing interfaces
> * build the project using the Django web framework
> * separate out a core sub-project with a well defined REST api for a
> frontend UI sub-project to build upon
>
> The rest of this proposal sets out to define parts of the core sub-project.
>
> The core should provide enough services to support the main part of a
> issue tracking UI. I am also expecting there to be further sub-projects to
> support additional optional functionality.
>
> The core sub-project will probably provide a basic UI to make progress
> ahead of work on the UI sub-project. I am not looking to define the UI
> sub-project any further in this proposal although it is likely that it will
> be highly dynamic, perhaps making use of websockets. The basic UI of the
> core project should remain well-maintained throughout the life of the
> project but at the moment my expectation would be that it is a no-thrills
> interface.
>
> Back to the core sub-project, I am looking for this to be built on the
> following concepts:
>
>  #0 a change in nomenclature from product to tracker as the top level
>  #1 uuids for pretty much everything
>  #2 tickets as the accumulation of ticket change events
>  #3 'labels' and 'label types' as a unifying concept for categorisation of
> tickets
>
> One of the early decisions made in handling of multiple projects in
> bloodhound was to call this feature multi-product. The reasoning for this
> was that the term 'project' is overloaded (if I recall, there may have been
> an excuse to add projects as a layer in bloodhound that could be used to
> group tickets across products.) I would like to see the basic functionality
> brought in to handle the ability to host multiple projects (in the top
> level organisation sense) but change the nomenclature to tracker to be as
> generic as possible. To be fair, the name of the feature may not matter so
> much as I would also like to see users given the ability to define their
> levels of organisation to a greater extent. More on this in the discussion
> of #3.
>
> Note that I will use the term tracker in the rest of this proposal where
> you might prefer to see product or project.
>
> The idea behind #1, uuids, is that it may be useful to be able to
> unambiguously identify instances of objects even across bloodhound
> instances. Pending decisions on how tickets are referenced, these uuids
> could be used as a reference that is unchanging as it moved between
> trackers.
>
> How tickets are stored as a whole is also worth tackling, which is what #2
> is trying to broadly decide. Here I am suggesting that the state of a
> ticket can be built up from a query that collects all the change events and
> applies the deltas in order. This could prove to be a slow process so at
> some point we may want to look at keeping a record of the current state or
> a checkpoint but, given that ticket views are expected to show the history,
> these are details that are required anyway. I suggest that we can look at
> optimising for speed later. Knowing that we can update a ticket from deltas
> may be useful.
>
> Finally, I am suggesting in #3 that as much as possible we generalise
> ticket categorisation. Categorisation is a central concept to capture and,
> in trac we inherited categorisations such as statuses (open, in progress,
> etc), types (bug, enhancement, task, etc), milestones, versions, etc, and
> these had separate implementations. What I would like to see is that, while
> we might provide an initial set of ways to categorise tickets, we should
> allow users to add the categorisations that they require. I'm suggesting
> calling this generalisation 'labels' and