Re: GSoC: Adding a web interface similar to the Hydra web interface

[Clément Lassieur]

Gábor Boskovits  writes:

> P
>
> Clément Lassieur  ezt írta (időpont: 2018. júl. 29.,
> V 14:01):
>
>> Danny Milosavljevic  writes:
>>
>> > Hi Tatiana,
>> >
>> > On Sun, 8 Jul 2018 21:48:32 +0200
>> > Tatiana Sholokhova  wrote:
>> >
>> >> Do you have ideas on how to
>> >> implement tuple comparison and other routines in SQL and guile in a
>> >> convenient and flexible way?
>> >
>> > sqlite3 supports row values, so the comparison can be
>> > written like this:
>> >
>> >   select * from foo where (a,b,c) = (2,'foo',3);
>> >
>> > It even supports NULLs for wildcards, though it's a little more
>> complicated:
>> >
>> >   select * from foo where coalesce((a,b,c) = (2,NULL,3), 1) = 1;
>> >
>> > The sqlite C interface doesn't support parameter bindings for the entire
>> > row, though, so you'd have to specify 3 values.
>> >
>> > This works:
>> >
>> >   (sqlite-exec db "select * from foo where (a,b,c) = (" 2 "," "foo" ","
>> 3 ");")
>> >
>> > but this doesn't work, unfortunately:
>> >
>> >   (sqlite-exec db "select * from foo where (a,b,c) = " '(2 "foo" 3) ";")
>> >
>> > See also https://www.sqlite.org/rowvalue.html
>>
>> With the '<' operator, it doesn't give the results we are looking for, I
>> think.
>>
>> For example:
>>
>> select (0,1) < (1,0); -- returns 1
>> select (0,0) < (0,1); -- returns 1
>>
>
> This is working as expected. Actually this:
> (a,b)<(c,d) is a shortcut for a
> In both cases, we'd want it to return 0.
>>
>
> How do we use it? Why this is the expected result?
>
>
>> I think we should use:
>>
>> select (0 < 1) and (1 < 0); -- returns 0
>> select (0 < 0) and (0 < 1); -- returns 0
>>
>
> Could you please clarify which numbers are the placeholders for which
> quantities?
>
>>
>> instead, for the pagination borders code.

Actually, forget what I said, I was wrong ;-)

Re: GSoC: Adding a web interface similar to the Hydra web interface

[Gábor Boskovits]

P

Clément Lassieur  ezt írta (időpont: 2018. júl. 29.,
V 14:01):

> Danny Milosavljevic  writes:
>
> > Hi Tatiana,
> >
> > On Sun, 8 Jul 2018 21:48:32 +0200
> > Tatiana Sholokhova  wrote:
> >
> >> Do you have ideas on how to
> >> implement tuple comparison and other routines in SQL and guile in a
> >> convenient and flexible way?
> >
> > sqlite3 supports row values, so the comparison can be
> > written like this:
> >
> >   select * from foo where (a,b,c) = (2,'foo',3);
> >
> > It even supports NULLs for wildcards, though it's a little more
> complicated:
> >
> >   select * from foo where coalesce((a,b,c) = (2,NULL,3), 1) = 1;
> >
> > The sqlite C interface doesn't support parameter bindings for the entire
> > row, though, so you'd have to specify 3 values.
> >
> > This works:
> >
> >   (sqlite-exec db "select * from foo where (a,b,c) = (" 2 "," "foo" ","
> 3 ");")
> >
> > but this doesn't work, unfortunately:
> >
> >   (sqlite-exec db "select * from foo where (a,b,c) = " '(2 "foo" 3) ";")
> >
> > See also https://www.sqlite.org/rowvalue.html
>
> With the '<' operator, it doesn't give the results we are looking for, I
> think.
>
> For example:
>
> select (0,1) < (1,0); -- returns 1
> select (0,0) < (0,1); -- returns 1
>

This is working as expected. Actually this:
(a,b)<(c,d) is a shortcut for a

How do we use it? Why this is the expected result?


> I think we should use:
>
> select (0 < 1) and (1 < 0); -- returns 0
> select (0 < 0) and (0 < 1); -- returns 0
>

Could you please clarify which numbers are the placeholders for which
quantities?

>
> instead, for the pagination borders code.
>

Re: GSoC: Adding a web interface similar to the Hydra web interface

[Clément Lassieur]

Danny Milosavljevic  writes:

> Hi Tatiana,
>
> On Sun, 8 Jul 2018 21:48:32 +0200
> Tatiana Sholokhova  wrote:
>
>> Do you have ideas on how to
>> implement tuple comparison and other routines in SQL and guile in a
>> convenient and flexible way?
>
> sqlite3 supports row values, so the comparison can be
> written like this:
>
>   select * from foo where (a,b,c) = (2,'foo',3);
>
> It even supports NULLs for wildcards, though it's a little more complicated:
>
>   select * from foo where coalesce((a,b,c) = (2,NULL,3), 1) = 1;
>
> The sqlite C interface doesn't support parameter bindings for the entire
> row, though, so you'd have to specify 3 values.
>
> This works:
>
>   (sqlite-exec db "select * from foo where (a,b,c) = (" 2 "," "foo" "," 3 
> ");")
>
> but this doesn't work, unfortunately:
>
>   (sqlite-exec db "select * from foo where (a,b,c) = " '(2 "foo" 3) ";")
>
> See also https://www.sqlite.org/rowvalue.html

With the '<' operator, it doesn't give the results we are looking for, I
think.

For example:

select (0,1) < (1,0); -- returns 1
select (0,0) < (0,1); -- returns 1

In both cases, we'd want it to return 0.

I think we should use:

select (0 < 1) and (1 < 0); -- returns 0
select (0 < 0) and (0 < 1); -- returns 0

instead, for the pagination borders code.

Re: GSoC: Adding a web interface similar to the Hydra web interface

[Amirouche Boubekki]

Fwiw, you can use bootstrap without js.

Causal reply,

Amirouche

Le jeu. 19 juil. 2018 à 22:11, Tatiana Sholokhova  a
écrit :

> Hi Clément,
>
> Thank you for the clarifications on the job structure!
>
> I have read your changes to the web interface and everything looks good to
> me. Also, it works nicely on your server. So, let's prepare for the merge.
> Let me know if you want me to make some fixes before the merge. In the
> meantime, I continue working on the next features for the interface locally
> and then integrate them into the updated codebase.
>
> Best regards,
> Tatiana
>
> ср, 18 июл. 2018 г. в 12:37, Clément Lassieur :
>
>> Hi Tatiana,
>>
>> Tatiana Sholokhova  writes:
>>
>> > Could you please review the last 3 commits and maybe find some more
>> issues
>> > besides that?
>>
>> I've integrated your work onto my Cuirass instance[1], and I really like
>> it!  I had to fix a few things and adapt it[2] so that it works with
>> multiple inputs.
>>
>> I will do a review as soon as possible, and then we can merge it.  I'm a
>> bit late: going through the whole conversation history took more time
>> than I expected.
>>
>> Clément
>>
>> [1]: https://cuirass.lassieur.org:8081/
>> [2]: https://git.lassieur.org/cgit/cuirass.git/
>>
>

Re: GSoC: Adding a web interface similar to the Hydra web interface

[Tatiana Sholokhova]

Hi Clément,

Thank you for the clarifications on the job structure!

I have read your changes to the web interface and everything looks good to
me. Also, it works nicely on your server. So, let's prepare for the merge.
Let me know if you want me to make some fixes before the merge. In the
meantime, I continue working on the next features for the interface locally
and then integrate them into the updated codebase.

Best regards,
Tatiana

ср, 18 июл. 2018 г. в 12:37, Clément Lassieur :

> Hi Tatiana,
>
> Tatiana Sholokhova  writes:
>
> > Could you please review the last 3 commits and maybe find some more
> issues
> > besides that?
>
> I've integrated your work onto my Cuirass instance[1], and I really like
> it!  I had to fix a few things and adapt it[2] so that it works with
> multiple inputs.
>
> I will do a review as soon as possible, and then we can merge it.  I'm a
> bit late: going through the whole conversation history took more time
> than I expected.
>
> Clément
>
> [1]: https://cuirass.lassieur.org:8081/
> [2]: https://git.lassieur.org/cgit/cuirass.git/
>

Re: GSoC: Adding a web interface similar to the Hydra web interface

[Clément Lassieur]

Hi Tatiana,

Tatiana Sholokhova  writes:

> Could you please review the last 3 commits and maybe find some more issues
> besides that?

I've integrated your work onto my Cuirass instance[1], and I really like
it!  I had to fix a few things and adapt it[2] so that it works with
multiple inputs.

I will do a review as soon as possible, and then we can merge it.  I'm a
bit late: going through the whole conversation history took more time
than I expected.

Clément

[1]: https://cuirass.lassieur.org:8081/
[2]: https://git.lassieur.org/cgit/cuirass.git/

Re: GSoC: Adding a web interface similar to the Hydra web interface

[Clément Lassieur]

Hi Tatiana,

Tatiana Sholokhova  writes:

> Am I right that in terms of Cuirass database derivations correspond to
> jobs?

Yes, but to be more precise, a job is a structure containing:
  - derivation
  - job-name
  - system
  - nix-name
  - eval-id

The database table called "Derivations" should be called "Jobs", so the
name is confusing indeed.

A derivation, as Ricardo explained, is a file (.drv) representing
low-level build actions and the environment in which they are performed.

At each evaluation, there is a new set of jobs returned by the
evaluator, each job having its 'eval-id' incremented.  That means that
two different jobs for the same job-name (i.e. linux-libre-4.17.6-job)
could embed the same derivation.  In that case, it's useless to build
that job in my opinion, see that bug[1].

I hope it's clearer,
Clément

[1]: https://debbugs.gnu.org/cgi/bugreport.cgi?bug=32190

Re: GSoC: Adding a web interface similar to the Hydra web interface

[Clément Lassieur]

Dear all,

Ludovic Courtès  writes:

> Hello Tatiana & all,
>
> Ricardo Wurmus  skribis:
>
>>> I am a bit confused about the database structure. As far as I understand,
>>> there are project_name (project) and branch_name (jobset) properties, but
>>> project_name is a primary key, so a project can't have several branches?
>>
>> I share your confusion.  Maybe Ludovic or Mathieu can shed some more
>> light on this.
>
> It’s confusing indeed, I think it’s a mistake that has yet to be fixed.
> Basically what we do now is that we use a different ‘repo_name’ when we
> just want to add a branch…

The notion of "project" has been removed[1].  It was previously the
specification name, which is a primary key indeed, so it didn't make
sense because one project couldn't have several branches.

Now, Hydra's jobsets are the exact same thing as Cuirass'
specifications.  So if you want to build the "master" and "core-update"
branches of Guix, you need two specifications.

However it wasn't even possible, before, to build several branches,
because specifications names were used by the evaluator: they had to be
"guix" or "guix-modular".  Since the name was a primary key, we could
only have two specifications.  It is now[2] possible, because the
evaluator uses the input name instead of the specification name.

If you think there is a need for the notion of "Project" in Cuirass, we
could add it, but it needs to be a new SQL table.  And each
specification would be associated with one project.

Clément

[1]: 
https://git.savannah.gnu.org/cgit/guix/guix-cuirass.git/commit/?id=be713f8a30788861806a74865b07403aa6774117
[2]: 
https://git.savannah.gnu.org/cgit/guix/guix-cuirass.git/commit/?id=7b2f9e0de1ad2d320973b7aea132a8afcad8bece

Re: GSoC: Adding a web interface similar to the Hydra web interface

[Clément Lassieur]

Ricardo Wurmus  writes:

> You can download a copy of the Cuirass database as it is used on
> berlin.guixsd.org, one of the build farms of the Guix project.  I have
> copied it here:
>
> http://bootstrappable.org/cuirass.db
>
> It is 12G(!), which indicates that Cuirass adds way too many entries
> than absolutely needed.  Ludovic wrote on IRC that we don’t seem to
> check if a record already exists when two subsequent evaluations yield
> the same build.

Adding bug-g...@gnu.org to keep track of that bug.

Re: GSoC: Adding a web interface similar to the Hydra web interface

[Gábor Boskovits]

Tatiana Sholokhova  ezt írta (időpont: 2018. júl.
8., V, 21:48):

>  Hello all!
>
> Thank you for your helpful comments and ideas!
>
> I've committed an improved version of the pagination. As you advised I
> chose and implemented (2) variant. I alter sorting order in SQL query
> depending on the type of the current page border. So, now all
> operators (gotofirst, gotolast, next and previous) are working.
>
> Also, I added pagination for builds page ("eval" query). Here I face a
> problem that Denny mentioned before.
>
>> The tuple of data cells should uniquely identify one row in the result.
>> (If it
>> didn't, you'd skip the other same-value rows when going to the next page)
>
> I order builds by stoptime and there are some builds with identical
> stoptime timestamps. I am not sure what is the best way to support
> pagination filtering by multiple columns. Do you have ideas on how to
> implement tuple comparison and other routines in SQL and guile in a
> convenient and flexible way?
>
>
Please have a look at this. https://www.sqlite.org/rowvalue.html. I did not
check the sqlite version.


> Could you please review the last 3 commits and maybe find some more issues
> besides that?
>
> Best regards,
> Tatiana
>
>

Re: GSoC: Adding a web interface similar to the Hydra web interface

[Danny Milosavljevic]

Hi Tatiana,

On Sun, 8 Jul 2018 21:48:32 +0200
Tatiana Sholokhova  wrote:

> Do you have ideas on how to
> implement tuple comparison and other routines in SQL and guile in a
> convenient and flexible way?

sqlite3 supports row values, so the comparison can be
written like this:

  select * from foo where (a,b,c) = (2,'foo',3);

It even supports NULLs for wildcards, though it's a little more complicated:

  select * from foo where coalesce((a,b,c) = (2,NULL,3), 1) = 1;

The sqlite C interface doesn't support parameter bindings for the entire
row, though, so you'd have to specify 3 values.

This works:

  (sqlite-exec db "select * from foo where (a,b,c) = (" 2 "," "foo" "," 3 ");")

but this doesn't work, unfortunately:

  (sqlite-exec db "select * from foo where (a,b,c) = " '(2 "foo" 3) ";")

See also https://www.sqlite.org/rowvalue.html


pgpK8_s3Kc_Iz.pgp
Description: OpenPGP digital signature

Re: GSoC: Adding a web interface similar to the Hydra web interface

[Tatiana Sholokhova]

 Hello all!

Thank you for your helpful comments and ideas!

I've committed an improved version of the pagination. As you advised I
chose and implemented (2) variant. I alter sorting order in SQL query
depending on the type of the current page border. So, now all
operators (gotofirst, gotolast, next and previous) are working.

Also, I added pagination for builds page ("eval" query). Here I face a
problem that Denny mentioned before.

> The tuple of data cells should uniquely identify one row in the result.
> (If it
> didn't, you'd skip the other same-value rows when going to the next page)

I order builds by stoptime and there are some builds with identical
stoptime timestamps. I am not sure what is the best way to support
pagination filtering by multiple columns. Do you have ideas on how to
implement tuple comparison and other routines in SQL and guile in a
convenient and flexible way?

Could you please review the last 3 commits and maybe find some more issues
besides that?

Best regards,
Tatiana

Re: GSoC: Adding a web interface similar to the Hydra web interface

[Gábor Boskovits]

Danny Milosavljevic  ezt írta (időpont: 2018. júl.
5., Cs 10:28):

> Hi Tatiana,
>
> On Wed, 4 Jul 2018 22:54:46 +0200
> Tatiana Sholokhova  wrote:
>
> > If we want to maintain a link to
> > the previous page we have to filter the database table entries with to
> > types of filters: one with lower bound on the id, and the other with the
> > upper bound.
>
> Yeah, I know what you mean.
>
> I'd suggest one of the following alternatives for implementing "Previous
> page":
>
> (1) Remember all the page boundaries in the query string (or maybe hidden
> form elements).  So instead of finding out where the beginning of the
> previous page was all over again just remember it from before.
> (2) Reverse the ordering in the query and the boundary check and run
> the query, reverse the result of the query.  Handle finished result as
> before.
> (3) Just use the browser's back button.  In fact, you can just put a
> "Previous" link that presses the back button via Javascript for the time
> being.
>
> I suggest (3) - and implement one of the others later.
>
> I agree that going with (3) now makes sense. The most flexible solution is
(2). I usually do that, as it does not rely on having the earlier pages
seen. I usually abstract this away in a cursor, which has the first, the
last, the current page first and last ids, and a gotofirst, gotolast, next
and previous operators.WDYT?

> > The current implementation of pagination works correctly but it does not
> > support link to the previous page (first, and next only).
>
> Cool!
>

Re: GSoC: Adding a web interface similar to the Hydra web interface

[Danny Milosavljevic]

Hi Tatiana,

On Wed, 4 Jul 2018 22:54:46 +0200
Tatiana Sholokhova  wrote:

> If we want to maintain a link to
> the previous page we have to filter the database table entries with to
> types of filters: one with lower bound on the id, and the other with the
> upper bound. 

Yeah, I know what you mean.

I'd suggest one of the following alternatives for implementing "Previous page":

(1) Remember all the page boundaries in the query string (or maybe hidden
form elements).  So instead of finding out where the beginning of the
previous page was all over again just remember it from before.
(2) Reverse the ordering in the query and the boundary check and run
the query, reverse the result of the query.  Handle finished result as before.
(3) Just use the browser's back button.  In fact, you can just put a
"Previous" link that presses the back button via Javascript for the time being.

I suggest (3) - and implement one of the others later.

> The current implementation of pagination works correctly but it does not
> support link to the previous page (first, and next only).

Cool!


pgp3fOZK322cj.pgp
Description: OpenPGP digital signature

Re: GSoC: Adding a web interface similar to the Hydra web interface

[Jelle Licht]

2018-07-04 22:54 GMT+02:00 Tatiana Sholokhova :

> Hi all,
>
>
Hi Tatiana,


> I just committed the code I wrote trying to improve pagination. I screwed
> up a bit with the new pagination.
> The problem I encountered is following. If we want to maintain a link to
> the previous page we have to filter the database table entries with to
> types of filters: one with lower bound on the id, and the other with the
> upper bound. As we do so simultaneously with limiting the query output to
> the maximal page size, we need to change the ordering type depending on the
> type of the filter. At the moment I am not sure, what it the best way to
> implement database query with such properties. Could you please take a look
> on the commit and share your ideas on that?
>

> The current implementation of pagination works correctly but it does not
> support link to the previous page (first, and next only).
>

It has been some time since I last had a look at databases, so you have my
apologies
in advance if what I am saying does not really apply, or is even not
entirely correct.

You could perhaps have a look at reversing the sort order, and replace ">"
with "<" and "<"
with "<" in your WHERE clauses. The query to the previous page would be
similar to
retrieving the next page, but basically reversing the order you would page
through the results,
if that makes any sense.

If this works, you could also hack together a maybe-efficient query to
retrieve the items
for the last page; simply insert the maximum possible value in your query,
and retrieve the
previous page with regards to that maximum value. In the current case, you
could enter the
highest possible value any id can have.

If it is possible for new items to show up on previous pages as well (so
with id's that are lower
than the highest id), you would need to replace your sorting and filtering
to act on a composite
value of e.g. , instead of on only the id value.


>
> I have been trying to improve pagination for a while, and I now am
> thinking about starting the parallel work on the implementation of the
> features we listed before. What do you think about it?
>

> Best regards,
> Tatiana
>
> Good luck, and HTH!

- Jelle

Re: GSoC: Adding a web interface similar to the Hydra web interface

[Tatiana Sholokhova]

Hi all,

I just committed the code I wrote trying to improve pagination. I screwed
up a bit with the new pagination.
The problem I encountered is following. If we want to maintain a link to
the previous page we have to filter the database table entries with to
types of filters: one with lower bound on the id, and the other with the
upper bound. As we do so simultaneously with limiting the query output to
the maximal page size, we need to change the ordering type depending on the
type of the filter. At the moment I am not sure, what it the best way to
implement database query with such properties. Could you please take a look
on the commit and share your ideas on that?

The current implementation of pagination works correctly but it does not
support link to the previous page (first, and next only).

I have been trying to improve pagination for a while, and I now am thinking
about starting the parallel work on the implementation of the features we
listed before. What do you think about it?

Best regards,
Tatiana

ср, 27 июн. 2018 г. в 21:56, Ludovic Courtès :

> Hi Tatiana,
>
> Tatiana Sholokhova  skribis:
>
> > On the last week, I had fallen out of the process. I had been having
> exams
> > at the university since the beginning of June. The last exam was
> > rescheduled and that has affected my plans. I am sorry for that. Now the
> > semester is finished and I am having much more time to focus on our
> > project. I am very happy to pass the first evaluation. It is a pleasure
> for
> > me to work with such a great team!
>
> Thank you, and thanks for letting us know.
>
> > In a few days, I am going to implement the whitelist for the static files
> > and improve pagination tool performance as we discussed before. Also, I
> > intend to make the pagination tool universal and add it to the page which
> > displays all builds of a selected evaluation.
>
> Sounds cool!  I haven’t followed your work as closely as I was hoping
> for, but it seems like we could merge it quickly, notably because
> everyone wants these enhancements.  :-)
>
> > Do you think it would be okay if I add auxiliary pagination filters to
> > the request which retrieves builds from the database?
>
> Why not.  Depending on the filter, We may need to add indexes to the
> database (see the bottom of ‘schema.sql’) so that requests aren’t too
> slow.
>
> Thank you for the update!
>
> Ludo’.
>

Re: GSoC: Adding a web interface similar to the Hydra web interface

[Ludovic Courtès]

Hi Tatiana,

Tatiana Sholokhova  skribis:

> On the last week, I had fallen out of the process. I had been having exams
> at the university since the beginning of June. The last exam was
> rescheduled and that has affected my plans. I am sorry for that. Now the
> semester is finished and I am having much more time to focus on our
> project. I am very happy to pass the first evaluation. It is a pleasure for
> me to work with such a great team!

Thank you, and thanks for letting us know.

> In a few days, I am going to implement the whitelist for the static files
> and improve pagination tool performance as we discussed before. Also, I
> intend to make the pagination tool universal and add it to the page which
> displays all builds of a selected evaluation.

Sounds cool!  I haven’t followed your work as closely as I was hoping
for, but it seems like we could merge it quickly, notably because
everyone wants these enhancements.  :-)

> Do you think it would be okay if I add auxiliary pagination filters to
> the request which retrieves builds from the database?

Why not.  Depending on the filter, We may need to add indexes to the
database (see the bottom of ‘schema.sql’) so that requests aren’t too
slow.

Thank you for the update!

Ludo’.

Re: GSoC: Adding a web interface similar to the Hydra web interface

[Tatiana Sholokhova]

Hello all,

On the last week, I had fallen out of the process. I had been having exams
at the university since the beginning of June. The last exam was
rescheduled and that has affected my plans. I am sorry for that. Now the
semester is finished and I am having much more time to focus on our
project. I am very happy to pass the first evaluation. It is a pleasure for
me to work with such a great team!

In a few days, I am going to implement the whitelist for the static files
and improve pagination tool performance as we discussed before. Also, I
intend to make the pagination tool universal and add it to the page which
displays all builds of a selected evaluation. Do you think it would be okay
if I add auxiliary pagination filters to the request which retrieves builds
from the database?

Best regards,
Tatiana


2018-06-25 13:46 GMT+03:00 Gábor Boskovits :

> Can you please send us an update on your project?
>

Re: GSoC: Adding a web interface similar to the Hydra web interface

[Gábor Boskovits]

Can you please send us an update on your project?

Re: GSoC: Adding a web interface similar to the Hydra web interface

[Joshua Branson]

Gábor Boskovits  writes:

> Joshua Branson  ezt írta (időpont: 2018. jún. 13., Sze 
> 15:52):
>
>  Tatiana Sholokhova  writes:
>
>  > Hello,
>  >
>  > Thank you for your reviews!
>  >
>  > I've just fixed codestyle issues and replaced HTML5 preamble with XHTML.
>
>  Just cause I'm curious, why XHTML instead HTML5?  Is XHTML better to parse?
>  (This question comes from non guix developer by the way.  Just an 
> enthusiast).
>
>  >
>  >
>  >  Thanks again for your excellent work!
>  >
>  >  --
>  >  Ricardo
>
> Joshua, both formats have advantages, and also disadvantages. I think this 
> issue deserves its own discussion. It would be nice to collect the opinions, 
> but I think that currently we have a good
> quality and well integrated xml generator, and we do not have a html5 
> generator. Please correct me if I am wrong.

Nope you sound right.  I'm not arguing in favor or html5 over
xhtml. Both are widely supported by all browsers.  Just trying to
learn.  Thanks!

Re: GSoC: Adding a web interface similar to the Hydra web interface

[Gábor Boskovits]

Joshua Branson  ezt írta (időpont: 2018. jún. 13.,
Sze 15:52):

> Tatiana Sholokhova  writes:
>
> > Hello,
> >
> > Thank you for your reviews!
> >
> > I've just fixed codestyle issues and replaced HTML5 preamble with XHTML.
>
> Just cause I'm curious, why XHTML instead HTML5?  Is XHTML better to parse?
> (This question comes from non guix developer by the way.  Just an
> enthusiast).
>
> >
> >
> >  Thanks again for your excellent work!
> >
> >  --
> >  Ricardo
>
Joshua, both formats have advantages, and also disadvantages. I think this
issue deserves its own discussion. It would be nice to collect the
opinions, but I think that currently we have a good quality and well
integrated xml generator, and we do not have a html5 generator. Please
correct me if I am wrong.

>

Re: GSoC: Adding a web interface similar to the Hydra web interface

[Joshua Branson]

Tatiana Sholokhova  writes:

> Hello,
>
> Thank you for your reviews!
>
> I've just fixed codestyle issues and replaced HTML5 preamble with XHTML.

Just cause I'm curious, why XHTML instead HTML5?  Is XHTML better to parse?
(This question comes from non guix developer by the way.  Just an enthusiast).

>
>
>  Thanks again for your excellent work!
>
>  --
>  Ricardo

Re: GSoC: Adding a web interface similar to the Hydra web interface

[Danny Milosavljevic]

Hi Tatiana,

On Wed, 13 Jun 2018 01:43:31 +0300
Tatiana Sholokhova  wrote:

> I've just fixed codestyle issues and replaced HTML5 preamble with XHTML.

Yeah, looks much more regular now.

> I adopted the static file serving procedure from code shared by Ricardo.
> 
> https://github.com/BIMSBbioinfo/rcas-web/blob/22a114a0f281845117ed0ab105267f132fc525e4/rcas/web/render.scm#L68

I understand.

It's common to do stuff like that - but it's just an unnecessary risk.

Especially since we have cuirass build (and presumably sign) everything for us,
it would be very very bad for an attacker to be able to read out the private key
used to sign everything.

Blacklists are a whack-a-mole approach (maintaining it will never end) - and in
this case nothing in POSIX states that these characters are the only
problematic ones (and I know of quite a few filesystems where there are a lot
more problematic strings).

Much better to have a whitelist.

> I am considering the following possible implementation of a whitelist. We
> can create association list with allowed file names and their mime types (to
> replace file-mime-type list). What do you think about it?

Yes, that would be a simple way - and easily maintainable, too.  Let's do it
like that.

> Now we have the only one parameter of the request
> which is related to pagination. It is the page number. Should we add more
> parameters to the request in order to request evaluations with specific
> timestamps and IDs? Or there is some other way of doing that?

I don't think we would need the parameter for the page number anymore.

I think we should add a parameter like "boundary" or something
(or multiple ones - I defer to your judgement).

It would also be possible to use "boundary=a=b" (note: same name).
Not sure whether that would be obtuse or not.

From a technical standpoint it doesn't matter - you just have to have get a
tuple of data cells (of the last row) into the query string somehow.
Also, it has to be able to be entirely optional (in order to get the first 
page).

The tuple of data cells should uniquely identify one row in the result.  (If it
didn't, you'd skip the other same-value rows when going to the next page)

(There would be an alternative possible to be able to dispense of this
requirement, but I think it would be too obtuse to maintain)

For the evaluations, the relevant data cells would be (starttime, id,
revision) or so.

I suggest adding starttime to the "Evaluations" database table to
improve usability.

(If we wanted to eventually provide a way for the user to sort columns,
we'd have to also adapt what columns this cell list contains - although
it doesn't make sense to stay on page 243 when you change the sort order :) )

What do you think?

> I have
> checked the Hydra pagination request structure. It uses the same form of
> the request path for pagination buttons that we have now:
> "?page=".

Yeah, the practice is widespread.  Let's do better than that if we can.


pgpyfnczUsiQT.pgp
Description: OpenPGP digital signature

Re: GSoC: Adding a web interface similar to the Hydra web interface

[Gábor Boskovits]

2018-06-13 0:43 GMT+02:00 Tatiana Sholokhova :

> Hello,
>
> Thank you for your reviews!
>
> I've just fixed codestyle issues and replaced HTML5 preamble with XHTML.
>
>
> respond-static-file: We should not second-guess the VFS layer.  Checking
>> for ".." gives an illusion of security when in fact random things could be
>> mounted and also the VFS could have funny syntax for who knows what on the
>> filesystem.  Let's rather have a static list of permissible names and allow
>> those (whitelist).  That's the intention of the check anyway, right?
>>
>
> I adopted the static file serving procedure from code shared by Ricardo.
>
> https://github.com/BIMSBbioinfo/rcas-web/blob/
> 22a114a0f281845117ed0ab105267f132fc525e4/rcas/web/render.scm#L68
>
> I am considering the following possible implementation of a whitelist. We
> can create association list with allowed file names and their mime types (to
> replace file-mime-type list). What do you think about it?
>
> Also, in light of an ever-changing backing store (cuirass continusly
>> evaluates things), the way you are doing pagination is not the correct way
>> to do it because the data set will scroll underneath you and you will miss
>> items (or see duplicate items) as an user.  Also, it's slow and the DBMS
>> can't reuse anything because you are cutting it off and offseting it over
>> and over again and the transaction isolation level is too low for the DBMS
>> to be able to do anything about it[1].
>
>
> I understand how to fix the pagination on the database level. However, it
> is not completely clear to me how to incorporate the solution in the
> web-request handling. Now we have the only one parameter of the request
> which is related to pagination. It is the page number.
>

Actually one parameter is still enough, but it will be the last item on the
previous page. This style of pagination supports sequential visiting of
pages from the first
page. It is later possible to extend on this for example by interpolating
on a counter, but on the first run this also needs only one parameter...


> Should we add more parameters to the request in order to
> request evaluations with specific timestamps and IDs? Or there is some
> other way of doing that? I have checked the Hydra pagination request
> structure. It uses the same form of the request path for pagination buttons
> that we have now: "?page=".
>
> It is good practise to make small commits, one for every set of
>> logically connected changes.  This makes the review simpler and it makes
>> it easier to merge some parts while leaving others for later.
>
>
> Okay, I will follow this commit strategy.
>
> Best regards,
> Tatiana
>
>
>
> 2018-06-13 0:52 GMT+03:00 Ricardo Wurmus :
>
>>
>> Hi Tatiana,
>>
>> I have only little to add to Danny’s comments.
>>
>> > db-get-builds looks fine and we could merge this change to master.
>> > But you have other changes in the same commit, so we can't directly
>> > cherry-pick it.  (not so bad, but somewhat cumbersome now)
>>
>> It is good practise to make small commits, one for every set of
>> logically connected changes.  This makes the review simpler and it makes
>> it easier to merge some parts while leaving others for later.
>>
>> As you work on the changes that Danny’s comments imply, please take the
>> opportunity to group related changes and commit only those together.  It
>> is fine and desirable to have many independent small commits.
>>
>> Thanks again for your excellent work!
>>
>> --
>> Ricardo
>>
>>
>

Re: GSoC: Adding a web interface similar to the Hydra web interface

[Tatiana Sholokhova]

Hello,

Thank you for your reviews!

I've just fixed codestyle issues and replaced HTML5 preamble with XHTML.


respond-static-file: We should not second-guess the VFS layer.  Checking
> for ".." gives an illusion of security when in fact random things could be
> mounted and also the VFS could have funny syntax for who knows what on the
> filesystem.  Let's rather have a static list of permissible names and allow
> those (whitelist).  That's the intention of the check anyway, right?
>

I adopted the static file serving procedure from code shared by Ricardo.

https://github.com/BIMSBbioinfo/rcas-web/blob/22a114a0f281845117ed0ab105267f132fc525e4/rcas/web/render.scm#L68

I am considering the following possible implementation of a whitelist. We
can create association list with allowed file names and their mime types (to
replace file-mime-type list). What do you think about it?

Also, in light of an ever-changing backing store (cuirass continusly
> evaluates things), the way you are doing pagination is not the correct way
> to do it because the data set will scroll underneath you and you will miss
> items (or see duplicate items) as an user.  Also, it's slow and the DBMS
> can't reuse anything because you are cutting it off and offseting it over
> and over again and the transaction isolation level is too low for the DBMS
> to be able to do anything about it[1].


I understand how to fix the pagination on the database level. However, it
is not completely clear to me how to incorporate the solution in the
web-request handling. Now we have the only one parameter of the request
which is related to pagination. It is the page number. Should we add more
parameters to the request in order to request evaluations with specific
timestamps and IDs? Or there is some other way of doing that? I have
checked the Hydra pagination request structure. It uses the same form of
the request path for pagination buttons that we have now:
"?page=".

It is good practise to make small commits, one for every set of
> logically connected changes.  This makes the review simpler and it makes
> it easier to merge some parts while leaving others for later.


Okay, I will follow this commit strategy.

Best regards,
Tatiana



2018-06-13 0:52 GMT+03:00 Ricardo Wurmus :

>
> Hi Tatiana,
>
> I have only little to add to Danny’s comments.
>
> > db-get-builds looks fine and we could merge this change to master.
> > But you have other changes in the same commit, so we can't directly
> > cherry-pick it.  (not so bad, but somewhat cumbersome now)
>
> It is good practise to make small commits, one for every set of
> logically connected changes.  This makes the review simpler and it makes
> it easier to merge some parts while leaving others for later.
>
> As you work on the changes that Danny’s comments imply, please take the
> opportunity to group related changes and commit only those together.  It
> is fine and desirable to have many independent small commits.
>
> Thanks again for your excellent work!
>
> --
> Ricardo
>
>

Re: GSoC: Adding a web interface similar to the Hydra web interface

[Ricardo Wurmus]

Hi Tatiana,

I have only little to add to Danny’s comments.

> db-get-builds looks fine and we could merge this change to master.
> But you have other changes in the same commit, so we can't directly
> cherry-pick it.  (not so bad, but somewhat cumbersome now)

It is good practise to make small commits, one for every set of
logically connected changes.  This makes the review simpler and it makes
it easier to merge some parts while leaving others for later.

As you work on the changes that Danny’s comments imply, please take the
opportunity to group related changes and commit only those together.  It
is fine and desirable to have many independent small commits.

Thanks again for your excellent work!

--
Ricardo

Re: GSoC: Adding a web interface similar to the Hydra web interface

[Danny Milosavljevic]

Hi Tatiana,

nice work!

I have a few comments:

db-get-builds looks fine and we could merge this change to master.  But you 
have other changes in the same commit, so we can't directly cherry-pick it.  
(not so bad, but somewhat cumbersome now)

I'd suggest to rename "db-get-evaluations-info" to 
"db-get-evaluations-build-summary".  Please don't use nonsensical names like 
"succ" or "que" in public interfaces (f.e. #:que ).

Nitpick: You have a few variables with "_" like "eval_cnt".  This is not 
customary in Scheme.  Also, having funny abbreviations like "cnt" isn't either. 
 I suggest "evaluation-count".

For db-get-evaluations-count: Please add a short docstring documenting what 
it's for (i.e. "Return the number of evaluations of the given specification 
SPEC").

In src/cuirass/http.scm :

I don't think that HTML5 is XML-compatible.  So if not, please use XHTML 
preamble and content-type (I think that that is even broken on master - but 
it's not difficult to fix).  It's also easier on web browsers if they can 
assume well-formedness once their XML parser is done parsing.

respond-static-file: We should not second-guess the VFS layer.  Checking for 
".." gives an illusion of security when in fact random things could be mounted 
and also the VFS could have funny syntax for who knows what on the filesystem.  
Let's rather have a static list of permissible names and allow those 
(whitelist).  That's the intention of the check anyway, right?

Also, in light of an ever-changing backing store (cuirass continusly evaluates 
things), the way you are doing pagination is not the correct way to do it 
because the data set will scroll underneath you and you will miss items (or see 
duplicate items) as an user.  Also, it's slow and the DBMS can't reuse anything 
because you are cutting it off and offseting it over and over again and the 
transaction isolation level is too low for the DBMS to be able to do anything 
about it[1].

See also 
https://use-the-index-luke.com/blog/2013-07/pagination-done-the-postgresql-way 
for a much better way.  Basically, remember the last value and use it as a 
limit in the WHERE condition by this one (also, sorting always).

That means if you have the (sorted) list:

A
B
C
E
G
H

and for some reason only 3 fit the page, print first page:

A
B
C <--- print, and remember this one for the boundary of the next page

Second page is "after C", so "... WHERE ... > 'C'":

E
G
H <--- print; and remember that no next page exists.

Note that this also works if, in the mean time, "D" appeared in the backing 
store.

Then second page is "after C", so "... WHERE ... > 'C'":

D
E
G <--- print, and remember this one for the boundary of the next page

This would only cause a problem if items vanished, so we should keep this in 
mind.  And jumps to random pages by page number would be difficult now (but who 
needs that in this case?  Much better to be able to jump by actual timestamp, 
which then would be easily possible).

In the case of evaluations, it would be good to sort by timestamp and then by 
evaluation id - since no matter what this would be a stable sort criteria.
(maybe revision, too)

[1] We should fix the latter eventually, too.


pgpFrSgoylwRL.pgp
Description: OpenPGP digital signature

Re: GSoC: Adding a web interface similar to the Hydra web interface

[Ludovic Courtès]

Hello Tatiana & Ricardo!

Ricardo Wurmus  skribis:

> I wasn’t sure about this, so I asked on the #guix IRC channel.  Ludovic
> replied there that the Cuirass repository contains a “random”
> specification in “examples/random.scm”.  It uses
> “examples/random-jobs.scm” to generate … random jobs :)

Specifically, here’s how I would launch Cuirass for testing purposes:

--8<---cut here---start->8---
$ ./pre-inst-env cuirass -D cuirass.db -I 10 -S examples/random.scm
2018-06-11T13:20:58 running Fibers on 4 kernel threads
2018-06-11T13:20:58 marking stale builds as "scheduled"...
2018-06-11T13:20:58 listening on 127.0.0.1:8080
2018-06-11T13:20:58 retrieving list of pending builds...
2018-06-11T13:20:58 heap: 11.82 MiB; threads: 10; file descriptors: 55
2018-06-11T13:20:58 considering spec 'random', URL 'file:///data/src/cuirass'
2018-06-11T13:20:58 canceling 3 stale builds
2018-06-11T13:20:58 restarting 0 pending builds
2018-06-11T13:20:58 building 0 derivations in batches of 200
2018-06-11T13:20:58 done with 0 derivations
2018-06-11T13:20:58 done with restarted builds
2018-06-11T13:20:58 spec 'random': fetched commit 
"238f856e48ee333ed3e19fa32ce5e1742c650c67" (stamp was 
"43be95c40a433d21f65c9e6bfb04ccc9fa8e2db4")
2018-06-11T13:20:58 next evaluation in 10 seconds
2018-06-11T13:20:58 evaluating 'random' with commit 
"238f856e48ee333ed3e19fa32ce5e1742c650c67"
evaluating random jobs from directory 
"/gnu/store/bb7x9wgc91h9jndyd9k36dysqnamjmyl-cuirass-238f856", commit 
"238f856e48ee333ed3e19fa32ce5e1742c650c67"
2018-06-11T13:20:59 created evaluation 5 for random, commit 
238f856e48ee333ed3e19fa32ce5e1742c650c67
2018-06-11T13:20:59 building 11 jobs for 'random'
2018-06-11T13:20:59 building 11 derivations in batches of 200
2018-06-11T13:20:59 building batch of 200 derivations (0/11)
2018-06-11T13:21:00 build started: 
'/gnu/store/npkk2v9n3lrs99j6hfm2sa7z839q00lz-random0.drv'
2018-06-11T13:21:00 build started: 
'/gnu/store/xbsa9sk4aipcvkqpxai73pzad523mwnc-random1.drv'
2018-06-11T13:21:08 considering spec 'random', URL 'file:///data/src/cuirass'
2018-06-11T13:21:08 spec 'random': fetched commit 
"238f856e48ee333ed3e19fa32ce5e1742c650c67" (stamp was 
"238f856e48ee333ed3e19fa32ce5e1742c650c67")
2018-06-11T13:21:08 next evaluation in 10 seconds
--8<---cut here---end--->8---

This example instructs Cuirass to populate the ‘cuirass.db’ file from
the current directory, to check the repo in the current directory every
10 seconds, and to use the job specification from ‘examples/random.scm’.
The HTTP server of Cuirass is listening on ‘localhost’, port 8080.

Let me know if you have any questions!  (I’m civodul on #guile.)

HTH,
Ludo’.

Re: GSoC: Adding a web interface similar to the Hydra web interface

[Ricardo Wurmus]

Hi Tatiana,

> I've just committed a new version of the interface. I've implemented the
> first feature and create a more friendly interface based on bootstrap.

I’ve looked at the screenshots and have to say that this is really
looking good already.  Exciting!

> I had to add new database requests: db-get-evaluations-count,
> db-get-evaluations-info for the feature. I have added new endpoints:
> ("jobset" name), ("eval" id) and changed "status" endpoint to "/".

Okay.

> Now, when you launch Cuirass you can see all specifications on the
> main page (localhost:PORT/); when you click on the specification name
> you can see a list of all evaluations of a specification displaying
> numbers of successful, failed and pending builds for each evaluation;
> when you click on the evaluation ID you can see a list of builds with
> their statuses.

Excellent.

> The evaluation list is broken down into a set of pages with 20
> evaluations on each page. I have implemented a page navigation tool
> which may be used for other pages of this kind that we will implement
> later.

Sounds good.

> Could you please take a look at the commit and new functions?

I will take a look today and reply with some comments on your commits.

> I am still facing the local testing issue. When I tried to launch Cuirass
> with the large database you sent before it crashed with some git
> error.

Could you share the error message with us?

> Maybe you could recommend me some
> specifications to add to my local database?

I wasn’t sure about this, so I asked on the #guix IRC channel.  Ludovic
replied there that the Cuirass repository contains a “random”
specification in “examples/random.scm”.  It uses
“examples/random-jobs.scm” to generate … random jobs :)

Note that you’ll need to have Guix installed to use this, and you need
to start Cuirass from your source checkout.

> Now I am going to implement separate pages for builds with different
> statuses and implementation of the first feature will be finished. Also, I
> think It will be useful if I add some more navigation buttons to the
> header. Now it has only one link to the main page with Guix logo.

These sound like good next steps.  Thank you, Tatiana!

--
Ricardo

Re: GSoC: Adding a web interface similar to the Hydra web interface

[Danny Milosavljevic]

Hi Tatiana,

> I afraid that I am not familiar with typical Hydra use cases

Generally, the continuous integration process should enable developers
to get feedback about the effects of their changes.

This means that as soon as a commit is made, usually an evaluation of
the build source on the continuous integration server starts.

(Sometimes there are exceptions to this (for example in order to not overload
the build servers) - but generally it's true)

For a new evaluation, as a developer I'd like to know:

* Are now more packages broken than before?  Which ones?
* Are now more packages working than before?  Which ones?
* Do some packages work on more architectures than before? Fewer?
* Is the build server still building my change?  Or is it done and I
can trust that the information I see is now complete?  If not,
what is it building now or later?

"before" means "with the previous evaluation" or "with some specific past
evaluation" or "in another branch".

I think this would be the most basic functionality.

More advanced functionalities would include automatic tracking on the reason
of the failure:

* If it's dependency failure, specifically mark this package so I know I don't
have to fix this package - I have to fix a package this one depends on (which 
one?).
* What kind of failure is it? What's the latest non-noise error message in the
build log?  Display suggestions on what to do about it.

What do you think?

>4. Add additional information about previous builds (latest successful,
>first broken, etc) on this build page. For this feature, we need to extend
>database requests functionality.

Sounds good.


pgp8BRkFwBHUX.pgp
Description: OpenPGP digital signature

Re: GSoC: Adding a web interface similar to the Hydra web interface

[Ricardo Wurmus]

Hi Tatiana,

> Yes, I've taken a look at Hydra. I think my problem is that I don't totally
> understand what means some pages at Hydra and how it is related to Cuirass
> database tables. Hydra has Projects, Jobsets for each project, Evaluations
> and Jobs for each Jobset. And Cuirass has Specifications, Evaluations and
> Derivations. Could you please clarify how these entities are related to
> each other?

This is a good question and I can relate to your confusion about these
terms :)

A Cuirass “Specification” roughly combines what Hydra calls a “Project”
with a “Jobset”.  On Hydra you would have a Project “gnu” (arbitrary
name) with a Jobset for building all packages from the “master” branch,
and another Jobset for building all packages from the “core-updates”
branch.  A Cuirass specification looks something like this (taken from
the manual):

--8<---cut here---start->8---
 `((#:name . "hello")
   (#:url . "git://git.savannah.gnu.org/guix.git")
   (#:branch . "master")
   (#:no-compile? . #t)
   (#:load-path . ".")
   (#:proc . cuirass-jobs)
   (#:file . "/tmp/drv-file.scm")
   (#:arguments (subset . "hello")))
--8<---cut here---end--->8---

As you can see it includes a name and the repository URL (~= “Project”),
as well as the branch and a procedure “proc” that specifies how this is
to be built.

Evaluations in Hydra and Cuirass are the same concept.  The Cuirass
manual says:

An evaluation relates a specification with the revision of the
repository specified therein.  Derivations and builds (see below)
each belong to a specific evaluation.

When an evaluation is started, we take a look at the current state of
the repository and then determine all build processes that need to be
performed.

Derivations are a fundamental concept in Guix.  The Guix manual says
this about derivations:

--8<---cut here---start->8---
Low-level build actions and the environment in which they are performed
are represented by “derivations”.  A derivation contains the following
pieces of information:

   • The outputs of the derivation—derivations produce at least one file
 or directory in the store, but may produce more.

   • The inputs of the derivations, which may be other derivations or
 plain files in the store (patches, build scripts, etc.)

   • The system type targeted by the derivation—e.g., ‘x86_64-linux’.

   • The file name of a build script in the store, along with the
 arguments to be passed.

   • A list of environment variables to be defined.

   Derivations allow clients of the daemon to communicate build actions
to the store.
--8<---cut here---end--->8---

When the build actions that are prescribed by a derivation are
performed, the result is a build.  This could be a failed build or a
directory containing the files that were generated by compiling a
package.

In the Hydra web interface “derivations” are not mentioned; instead they
speak of “Jobs”, which is roughly the same thing.  After all, a
derivation needs to be processed to get a build, so we can consider the
collection of derivations in a particular evaluation of a specification
to be a collection of build jobs.

> I have come up with the following list of features: […]

The list looks very good.  Please see my comments below.

> 1. Add the list of all evaluations of a specification displaying numbers of
> successful and failed builds for each evaluation (like this list
> https://hydra.nixos.org/jobset/gnu/guile-2-0/evals).

Just a note: The example URL you picked may be a little misleading
because hydra.nixos.org is used slightly differently from hydra.gnu.org
— for our specifications the number of successful, failed, and ongoing
builds would be considerably larger than in this example.

This would be a very useful feature, because we could observe a trend:
as the repository is changed and an increasing number of evaluations are
completed, do more builds fail now than before, or are we improving…?

Clicking on the number of failed/successful/pending builds we should get
a list of those builds/derivations.

> 2. Add the table with status information of each job-evaluation pair like
> https://hydra.nixos.org/jobset/gnu/guile-2-0/#tabs-jobs.

A table like this would be very large, because in our case evaluations
result in thousands of derivations (= jobs).  So you need to design this
to use pagination.

With a very long table, collecting all this past information (on the
columns) can be expensive, so I’d suggest to only display the *current*
state, not the state over a range of past days.

> Am I right that
> in terms of Cuirass database derivations correspond to jobs?

Correct.

> So, we need to
> display "derivations" table with derivations as rows and evaluations as
> columns and fill the cells using information from the "builds" table?

Correct.

> Also,
> it is not clear to me how 

Re: GSoC: Adding a web interface similar to the Hydra web interface

[Tatiana Sholokhova]

Hello, Ricardo!

Yes, I've taken a look at Hydra. I think my problem is that I don't totally
understand what means some pages at Hydra and how it is related to Cuirass
database tables. Hydra has Projects, Jobsets for each project, Evaluations
and Jobs for each Jobset. And Cuirass has Specifications, Evaluations and
Derivations. Could you please clarify how these entities are related to
each other? I formulate more precise questions below.

I have come up with the following list of features:

1. Add the list of all evaluations of a specification displaying numbers of
successful and failed builds for each evaluation (like this list
https://hydra.nixos.org/jobset/gnu/guile-2-0/evals).
2. Add the table with status information of each job-evaluation pair like
https://hydra.nixos.org/jobset/gnu/guile-2-0/#tabs-jobs.  Am I right that
in terms of Cuirass database derivations correspond to jobs? So, we need to
display "derivations" table with derivations as rows and evaluations as
columns and fill the cells using information from the "builds" table? Also,
it is not clear to me how to order evaluations by their creation time like
it's done in Hydra.
3. Add page displaying information about a build similar to
https://hydra.nixos.org/build/74870692. Am I right that here we should
display the information stored in hydra-like build description (defined
in http.scm) alongside with links to the build log and outputs?
4. Add additional information about previous builds (latest successful,
first broken, etc) on this build page. For this feature, we need to extend
database requests functionality.
5. Add job information page containing the list of job's latest builds like
https://hydra.nixos.org/job/gnu/guile-2-0/build.x86_64-linux.

What do you think about these features? I afraid that I am not familiar
with typical Hydra use cases, so I may miss some important features.

Best regards,
Tatiana


2018-06-03 22:40 GMT+03:00 Ricardo Wurmus :

> Hi Tatiana,
>
> > I fixed my previous commits as you adviced. I have tested some features
> on
> > examples databases and it was correct. But I can't try your database
> > (small-cuirass.db) file, the queries just return empty 'queue' and
> 'latest
> > build' lists, and I haven't tried other requests yet.
>
> Oh, maybe there’s something wrong with that file.  I’m sorry.
>
> > Now we have a web application with two pages: main page with current
> > specifications status and pages for each specification. I think, for the
> > next stage I should display some more information from the database but
> I'm
> > not sure what else I should display.
>
> Have you taken a look at https://hydra.nixos.org yet?  This is the hydra
> instance of the Nix project.  (We use an older variant of the same
> software on https://hydra.gnu.org, but it’s not very responsive.)
>
> On https://hydra.nixos.org/jobset/nixos/staging you can see a particular
> branch of the nixos project.  It lists evaluations, displays the number
> of successes, failures, and pending builds, and it links to a detailed
> overview of the evaluation.
>
> For example, at https://hydra.nixos.org/eval/1459429 we see the list of
> builds that are associated with a particular evaluation and we can
> follow a link to a description of that build.  The build page shows us
> some information about the derivation/package and links to the build
> logs.
>
> It also shows us when the build first failed, what change in the
> repository lead to the build failure, and so on.
>
> > Now I'm not sure what features I should implement next? What else do you
> > want to see by the first deadline?
>
> We are not just looking for a status page that displays the database
> contents.  Some of the bits of information have to be derived from more
> than one database record.  When you compare the current state of the
> Cuirass web interface to that of Hydra, what do you see is still missing
> and should be implemented next?  Could you identify the top 5 features
> that you think are missing and could be added to Cuirass?
>
> --
> Ricardo
>
>

Re: GSoC: Adding a web interface similar to the Hydra web interface

[Ricardo Wurmus]

Hi Tatiana,

> I fixed my previous commits as you adviced. I have tested some features on
> examples databases and it was correct. But I can't try your database
> (small-cuirass.db) file, the queries just return empty 'queue' and 'latest
> build' lists, and I haven't tried other requests yet.

Oh, maybe there’s something wrong with that file.  I’m sorry.

> Now we have a web application with two pages: main page with current
> specifications status and pages for each specification. I think, for the
> next stage I should display some more information from the database but I'm
> not sure what else I should display.

Have you taken a look at https://hydra.nixos.org yet?  This is the hydra
instance of the Nix project.  (We use an older variant of the same
software on https://hydra.gnu.org, but it’s not very responsive.)

On https://hydra.nixos.org/jobset/nixos/staging you can see a particular
branch of the nixos project.  It lists evaluations, displays the number
of successes, failures, and pending builds, and it links to a detailed
overview of the evaluation.

For example, at https://hydra.nixos.org/eval/1459429 we see the list of
builds that are associated with a particular evaluation and we can
follow a link to a description of that build.  The build page shows us
some information about the derivation/package and links to the build
logs.

It also shows us when the build first failed, what change in the
repository lead to the build failure, and so on.

> Now I'm not sure what features I should implement next? What else do you
> want to see by the first deadline?

We are not just looking for a status page that displays the database
contents.  Some of the bits of information have to be derived from more
than one database record.  When you compare the current state of the
Cuirass web interface to that of Hydra, what do you see is still missing
and should be implemented next?  Could you identify the top 5 features
that you think are missing and could be added to Cuirass?

--
Ricardo

Re: GSoC: Adding a web interface similar to the Hydra web interface

[Tatiana Sholokhova]

Hello Ricardo,

I fixed my previous commits as you adviced. I have tested some features on
examples databases and it was correct. But I can't try your database
(small-cuirass.db) file, the queries just return empty 'queue' and 'latest
build' lists, and I haven't tried other requests yet.

Now we have a web application with two pages: main page with current
specifications status and pages for each specification. I think, for the
next stage I should display some more information from the database but I'm
not sure what else I should display.

Now I'm not sure what features I should implement next? What else do you
want to see by the first deadline?

Best regards,
Tatiana

Re: GSoC: Adding a web interface similar to the Hydra web interface

[Ricardo Wurmus]

Hi Tatiana,

did you find the database files useful?

Could you please let us know what the current state of your project is?

Thanks,
Ricardo


Ricardo Wurmus  writes:

> Hi Tatiana,
>
>> I've just made a new commit. I tried to fix codestyle mistakes but I'm not
>> sure that I managed to cover them all.
>
> Thank you for making the extra effort!
>
> One more thing I’ve noticed is that your patches add trailing whitespace
> to some lines (i.e. a line that ends on a space, or an empty line that
> only contains a space character), which we try to avoid.  This is just a
> minor problem, but we would probably edit these commits to remove these
> whitespace changes before merging the commits into the “master” branch.
>
> You are welcome to rewrite history in the “web-interface” branch,
> i.e. you can fix the previous commits in your local repository and then
> force push them to “web-interface”.  (Please be extra careful not to
> force push to the “master” branch.)
>
> Another thing I saw is things like this:
>
> '(
>   something else …)
>
> i.e. an opening parenthesis followed by a line break.  Please try to
> avoid those.
>
>> I've added static files support and an example of css style file.
>>
>> Also, I've added new html templates displaying builds of a specification.
>> It is available via "/status/" or via a link from the main page.
>
> Excellent.
>
> For your latest commit please use this commit message body instead:
>
> --8<---cut here---start->8---
> * src/cuirass/http.scm (%static-directory, file-mime-types): New variables.
> (url-handler): Add handler for “/status/”; add handler for
> static files.
> * src/static/style.css: New file.
> …
> --8<---cut here---end--->8---
>
> I recommend merging this commit with the previous commit.  You can use
> “git rebase -i $start” to interactively rebase commits and mark the last
> two commits to be squashed into one commit.  Please reword the commit
> message as well.
>
> I know that this may seem like nitpicking, but it’s a good habit to
> acquire early — fixing up commits at the very end is much harder in my
> experience.
>
>
>> Now I'm not sure what I should display in the tables and what pages to
>> implement in the interface? I think it would be helpful if I have a more
>> realistic database for understanding database structure and testing
>> purposes. But I don't know how can I get one.
>
> You can download a copy of the Cuirass database as it is used on
> berlin.guixsd.org, one of the build farms of the Guix project.  I have
> copied it here:
>
> http://bootstrappable.org/cuirass.db
>
> It is 12G(!), which indicates that Cuirass adds way too many entries
> than absolutely needed.  Ludovic wrote on IRC that we don’t seem to
> check if a record already exists when two subsequent evaluations yield
> the same build.
>
> I have also put up a smaller database at
>
> http://bootstrappable.org/cuirass-small.db
>
> which also came from berlin.guixsd.org.  I don’t know if that one would
> be useful to you, though, as it is only 48kB in size.


--
Ricardo

Re: GSoC: Adding a web interface similar to the Hydra web interface

[Gábor Boskovits]

2018-05-29 18:07 GMT+02:00 Ludovic Courtès :

> Hello Tatiana & all,
>
> Ricardo Wurmus  skribis:
>
> >> I am a bit confused about the database structure. As far as I
> understand,
> >> there are project_name (project) and branch_name (jobset) properties,
> but
> >> project_name is a primary key, so a project can't have several branches?
> >
> > I share your confusion.  Maybe Ludovic or Mathieu can shed some more
> > light on this.
>
> It’s confusing indeed, I think it’s a mistake that has yet to be fixed.
> Basically what we do now is that we use a different ‘repo_name’ when we
> just want to add a branch…
>
> We should fix it at some point.  Suggestions welcome!
>
>
I believe the confusion arises from the fact, that there is a naming
inconsistency.
If we could come up with good names to the things involved, then that might
suggest a clean solution. It seems that the problem is caused by a project
and
a repository is somehow intermixed. That might suggest a solution to keep
project-name as is, as primary key, and add a separate repository field...
or not.
Does that make sense?


> I would encourage you to write commits in a way to minimize friction
> when we are to merge them—that is, following the conventions that
> Ricardo outlined.  That way Mathieu, Ricardo, or myself can take a look
> and quickly cherry-pick to master.
>
> Anyway, kudos on what you’ve already achieved!  Getting started on an
> existing code base is never easy, so I think you’ve done a good job.
>
> Thank you,
> Ludo’.
>
>

Re: GSoC: Adding a web interface similar to the Hydra web interface

[Ludovic Courtès]

Hello Tatiana & all,

Ricardo Wurmus  skribis:

>> I am a bit confused about the database structure. As far as I understand,
>> there are project_name (project) and branch_name (jobset) properties, but
>> project_name is a primary key, so a project can't have several branches?
>
> I share your confusion.  Maybe Ludovic or Mathieu can shed some more
> light on this.

It’s confusing indeed, I think it’s a mistake that has yet to be fixed.
Basically what we do now is that we use a different ‘repo_name’ when we
just want to add a branch…

We should fix it at some point.  Suggestions welcome!

I would encourage you to write commits in a way to minimize friction
when we are to merge them—that is, following the conventions that
Ricardo outlined.  That way Mathieu, Ricardo, or myself can take a look
and quickly cherry-pick to master.

Anyway, kudos on what you’ve already achieved!  Getting started on an
existing code base is never easy, so I think you’ve done a good job.

Thank you,
Ludo’.

Re: GSoC: Adding a web interface similar to the Hydra web interface

[Ricardo Wurmus]

Hi Tatiana,

> I've just made a new commit. I tried to fix codestyle mistakes but I'm not
> sure that I managed to cover them all.

Thank you for making the extra effort!

One more thing I’ve noticed is that your patches add trailing whitespace
to some lines (i.e. a line that ends on a space, or an empty line that
only contains a space character), which we try to avoid.  This is just a
minor problem, but we would probably edit these commits to remove these
whitespace changes before merging the commits into the “master” branch.

You are welcome to rewrite history in the “web-interface” branch,
i.e. you can fix the previous commits in your local repository and then
force push them to “web-interface”.  (Please be extra careful not to
force push to the “master” branch.)

Another thing I saw is things like this:

'(
  something else …)

i.e. an opening parenthesis followed by a line break.  Please try to
avoid those.

> I've added static files support and an example of css style file.
>
> Also, I've added new html templates displaying builds of a specification.
> It is available via "/status/" or via a link from the main page.

Excellent.

For your latest commit please use this commit message body instead:

--8<---cut here---start->8---
* src/cuirass/http.scm (%static-directory, file-mime-types): New variables.
(url-handler): Add handler for “/status/”; add handler for
static files.
* src/static/style.css: New file.
…
--8<---cut here---end--->8---

I recommend merging this commit with the previous commit.  You can use
“git rebase -i $start” to interactively rebase commits and mark the last
two commits to be squashed into one commit.  Please reword the commit
message as well.

I know that this may seem like nitpicking, but it’s a good habit to
acquire early — fixing up commits at the very end is much harder in my
experience.


> Now I'm not sure what I should display in the tables and what pages to
> implement in the interface? I think it would be helpful if I have a more
> realistic database for understanding database structure and testing
> purposes. But I don't know how can I get one.

You can download a copy of the Cuirass database as it is used on
berlin.guixsd.org, one of the build farms of the Guix project.  I have
copied it here:

http://bootstrappable.org/cuirass.db

It is 12G(!), which indicates that Cuirass adds way too many entries
than absolutely needed.  Ludovic wrote on IRC that we don’t seem to
check if a record already exists when two subsequent evaluations yield
the same build.

I have also put up a smaller database at

http://bootstrappable.org/cuirass-small.db

which also came from berlin.guixsd.org.  I don’t know if that one would
be useful to you, though, as it is only 48kB in size.

--
Ricardo

Re: GSoC: Adding a web interface similar to the Hydra web interface

[Ricardo Wurmus]

Hi Tatiana,

> I have committed the first HTML template page (with table of specifications
> stored in the database) to web-interface branch. Can you please review it?

That’s great!  Congratulations on your first commit!

I’ll make a couple of extra comments on style and conventions because
this is your first commit.  You aren’t expected to remember all of these
conventions right away.

> commit a4fe6dd0d0c82c84a810d3368dd60fea3aa1b2b0 (HEAD -> web-interface, 
> origin/web-interface)
> Author: TSholokhova 
> Date:   Wed May 23 16:37:23 2018 +0300
>
> basic html templates

Please remember to make expressive commit messages.  We normally use a
format similar to the ChangeLog convention, which consists of a one-line
summary that is usually a sentence, and a listing of all changes.

In this case it would be something like:

--8<---cut here---start->8---
Add basic HTML templates.

* src/cuirass/templates.scm: New file.
* Makefile.am (dist_pkgmodule_DATA): Add it.
* src/cuirass/http.scm (url-handler): Add handler for “status”
endpoint.
--8<---cut here---end--->8---


> diff --git a/src/cuirass/http.scm b/src/cuirass/http.scm
> index e911b9b..f5e3ac1 100644
> --- a/src/cuirass/http.scm
> +++ b/src/cuirass/http.scm
> @@ -1,3 +1,4 @@
> +
>   http.scm -- HTTP API
>  ;;; Copyright © 2016 Mathieu Lirzin 
>  ;;; Copyright © 2017 Mathieu Othacehe 

Please make sure to add a copyright line for yourself to the bottom of
the copyright block.  Also try to avoid pure whitespace changes like the
addition of an empty line at the top of the file.

> @@ -135,6 +139,12 @@ Hydra format."
>   #:body
>   (object->json-string
>`((error . ,message)
> +
> +  (define (respond-html body)
> +(respond '((content-type . (text/html)))
> + #:body (lambda (port)
> +  (sxml->xml body port)
> +  )))

Please don’t leave closing parentheses on a line by itself; they feel
lonely and prefer to be on the previous line.

> @@ -223,6 +233,11 @@ Hydra format."
>,@params
>(order status+submission-time)
> (respond-json-with-error 500 "Parameter not defined!"
> +(("status")
> + (respond-html (templatize
> + "Status"
> + (specifications-table
> +  (with-critical-section db-channel (db) 
> (db-get-specifications db))

Here and in other places you used tabs, but we’re using spaces for
indentation in all source files.  Please configure your editor to use
spaces instead of tabs.

I feel that the “url-handler” procedure is already very large, so it may
be a good idea to break out the handler to a separate procedure instead
of having the details inline.  This is not a problem yet, but it may be
good to keep in mind in case you need to grow this handler in the
future.  As long as it stays this small it’s fine to keep it there.

> diff --git a/src/cuirass/templates.scm b/src/cuirass/templates.scm
> new file mode 100644
> index 000..ff63469
> --- /dev/null
> +++ b/src/cuirass/templates.scm
> @@ -0,0 +1,32 @@
> +(define-module (cuirass templates)
> +  #:export (templatize
> +specifications-table))
> +
> +

Please add the usual copyright header to the top of this module.

> +(define (templatize title body)
> +  `(html
> +,(head title)
> +(body ,body)))

Please also add a docstring to every toplevel definition to explain what
the procedure is supposed to do.  “templatize” is quite a fancy name for
what I’d call “html-page” :)

> +
> +(define (head title)
> +  `(head
> +(meta (@ (charset "utf-8")))
> +(title ,title)))

This could become part of “templatize” instead.  It’s generally good to
have small independent procedures, but in this case I don’t see us ever
using “head” without “templatize”.

> +(define (specifications-table specs)
> +  `(table
> +(@ (class "table-fill"))
> +(thead
> + (tr
> +  (th (@ (class "text-left")) Name)
> +  (th (@ (class "text-left")) Branch)))
> +(tbody
> + (@ (class "table-fill"))
> + ,@(map
> +(lambda (spec)
> +  `(tr
> +(td ,(assq-ref spec #:name))
> +(td ,(assq-ref spec #:branch
> +specs

Please also add a docstring to this procedure.  What is the result of
this procedure if “specs” is empty?  Should that case be covered to
communicate this to the viewer?

> I am a bit confused about the database structure. As far as I understand,
> there are project_name (project) and branch_name (jobset) properties, but
> project_name is a primary key, so a project can't have several branches?

I share your confusion.  Maybe Ludovic or Mathieu can shed some more
light on this.

It is true that “repo_name” (not project_name) in the Specifications
table is the 

Re: GSoC: Adding a web interface similar to the Hydra web interface

[Tatiana Sholokhova]

Hi Ricardo,

Thanks for the helpful answer. Now I almost figured out how to serve static
files.

I have committed the first HTML template page (with table of specifications
stored in the database) to web-interface branch. Can you please review it?

I am a bit confused about the database structure. As far as I understand,
there are project_name (project) and branch_name (jobset) properties, but
project_name is a primary key, so a project can't have several branches?

I'm working on static file serving but I don't know how to set the path of
the static file directory. Now I just wrote my absolute path to the
style.css file. So, I have two questions. 1. Where should I place the
static files? 2. How can I execute getcwd function (as you do it in
rcas-web/rcas/config.scm.in)? I tried to add something like

(define-public %current-directory
`(,(getcwd)))

to my config.scm.in but it does not work.

Best regards,
Tatiana Sholokhova

2018-05-22 8:33 GMT+03:00 Ricardo Wurmus :

>
> Hi Tatiana,
>
> > I have already created a small module implementing basic HTML templates
> in
> > Scheme. Also, I have made an extension to the Cuirass Web API. It
> responds
> > on the "/status" request and generates a page with a minimalistic HTML
> > table displaying the list of specifications stored in the database.
>
> Excellent.
>
> > My login on Savannah is "TSholokhova". I am looking forward to making my
> > first commit. I would be glad to receive comments on my code to be sure
> > that I am moving in the right direction.
>
> The Cuirass repository is available at
> http://git.savannah.gnu.org/cgit/guix/guix-cuirass.git
>
> I have added your account to the group, so you should have write access
> to the repository now.  Please push your commits to a new branch on that
> repository.  Please also read the section “Coding Style” in the Guix
> manual.
>
> You are welcome to rewrite your own published commits in your own
> branch, but not in other branches.  When you want us to comment on your
> work, please let us know the range of commits that you would like us to
> review.  Ideally, we would be able to merge your work into the “master”
> branch regularly.
>
> > You have mentioned that many users would prefer an interface working
> > without javascript running. Am I right that we would like to have a
> > non-interactive (js-free) interface working and also add some
> functionality
> > (e. g. search tools for tables) via javascript?
>
> Yes, this sounds like a good idea.  It is not absolutely necessary, but
> it would be nice if basic features of the interface would still be
> usable even when JavaScript is disabled.
>
> > Also, I have a couple of questions regarding the frontend part. What
> > frontend framework we would prefer? If I get it right, Hydra uses
> > Bootstrap.
>
> It is fine to use Bootstrap.  Please include the unminified Bootstrap
> CSS and JS files to the repository.  It is easy to minify them later,
> but it is virtually impossible to turn minified code into something
> readable.
>
> > For the frontend implementation, we need to include some static
> > css files in the interface and serve them somehow. Is it a good idea
> to
> > serve the static files by Cuirass web server in Scheme?
>
> Yes, this can be done by the Cuirass web server in Scheme.
>
> In the web interface for RCAS I’m dispatching to “render-static-asset”
> in the controller:
>
> https://github.com/BIMSBbioinfo/rcas-web/blob/
> master/rcas/web/controller.scm
>
> “render-static-asset” is defined here:
>
> https://github.com/BIMSBbioinfo/rcas-web/blob/
> master/rcas/web/render.scm#L65
>
> As you can see, this module defines a bunch of mime types and includes
> the appropriate type in the headers along with the contents of the file
> it serves.
>
> Hope this helps!
>
> --
> Ricardo
>
>
>

Re: GSoC: Adding a web interface similar to the Hydra web interface

[Ricardo Wurmus]

Hi Tatiana,

> I have already created a small module implementing basic HTML templates in
> Scheme. Also, I have made an extension to the Cuirass Web API. It responds
> on the "/status" request and generates a page with a minimalistic HTML
> table displaying the list of specifications stored in the database.

Excellent.

> My login on Savannah is "TSholokhova". I am looking forward to making my
> first commit. I would be glad to receive comments on my code to be sure
> that I am moving in the right direction.

The Cuirass repository is available at
http://git.savannah.gnu.org/cgit/guix/guix-cuirass.git

I have added your account to the group, so you should have write access
to the repository now.  Please push your commits to a new branch on that
repository.  Please also read the section “Coding Style” in the Guix
manual.

You are welcome to rewrite your own published commits in your own
branch, but not in other branches.  When you want us to comment on your
work, please let us know the range of commits that you would like us to
review.  Ideally, we would be able to merge your work into the “master”
branch regularly.

> You have mentioned that many users would prefer an interface working
> without javascript running. Am I right that we would like to have a
> non-interactive (js-free) interface working and also add some functionality
> (e. g. search tools for tables) via javascript?

Yes, this sounds like a good idea.  It is not absolutely necessary, but
it would be nice if basic features of the interface would still be
usable even when JavaScript is disabled.

> Also, I have a couple of questions regarding the frontend part. What
> frontend framework we would prefer? If I get it right, Hydra uses
> Bootstrap.

It is fine to use Bootstrap.  Please include the unminified Bootstrap
CSS and JS files to the repository.  It is easy to minify them later,
but it is virtually impossible to turn minified code into something
readable.

> For the frontend implementation, we need to include some static
> css files in the interface and serve them somehow. Is it a good idea to
> serve the static files by Cuirass web server in Scheme?

Yes, this can be done by the Cuirass web server in Scheme.

In the web interface for RCAS I’m dispatching to “render-static-asset”
in the controller:

https://github.com/BIMSBbioinfo/rcas-web/blob/master/rcas/web/controller.scm

“render-static-asset” is defined here:

https://github.com/BIMSBbioinfo/rcas-web/blob/master/rcas/web/render.scm#L65

As you can see, this module defines a bunch of mime types and includes
the appropriate type in the headers along with the contents of the file
it serves.

Hope this helps!

--
Ricardo

Re: GSoC: Adding a web interface similar to the Hydra web interface

[Tatiana Sholokhova]

Hi Ricardo,

I agree with your point of view on the interface implementation approaches.
Indeed, the approaches are not supposed to be mutually exclusive. As you
said the first approach is easier and I have started working closer to this
approach.

I have already created a small module implementing basic HTML templates in
Scheme. Also, I have made an extension to the Cuirass Web API. It responds
on the "/status" request and generates a page with a minimalistic HTML
table displaying the list of specifications stored in the database.

My login on Savannah is "TSholokhova". I am looking forward to making my
first commit. I would be glad to receive comments on my code to be sure
that I am moving in the right direction.

You have mentioned that many users would prefer an interface working
without javascript running. Am I right that we would like to have a
non-interactive (js-free) interface working and also add some functionality
(e. g. search tools for tables) via javascript?

Also, I have a couple of questions regarding the frontend part. What
frontend framework we would prefer? If I get it right, Hydra uses
Bootstrap. For the frontend implementation, we need to include some static
css files in the interface and serve them somehow. Is it a good idea to
serve the static files by Cuirass web server in Scheme?

Best regards,
Tatiana

2018-05-18 23:35 GMT+03:00 Ricardo Wurmus :

>
> Hi Tatiana,
>
> > I have started thinking about the type of web interface we want to have
> for
> > Cuirass in this project. As far as I see, there are two options:
> >
> >- a web application served by Cuirass web server;
> >- a standalone static site which sends queries to the Cuirass web API
> >(this is similar to Danny's application);
> >
> > I suppose that the first option has more benefits since it allows to
> choose
> > the exact type of information required to be extracted from the database
> by
> > a specific part of the web interface. What do think regarding these
> options?
>
> You are free to extend the Cuirass web API to suit your application’s
> needs.  Having a standalone site is a valid way of providing a web
> interface, but it doesn’t have to be the only way of accessing the
> information.
>
> Even if you go for the first route, the HTML you serve could talk to the
> API.  These two options don’t have to be mutually exclusive.
>
> FWIW, I expect the first approach to be easier because you can use
> Scheme for the most part; the pages it serves could be progressively
> enhanced with JavaScript, if the client supports it.  I’m sure there are
> many users who would prefer a system that would work fine even without
> running JavaScript in the browser.
>
> > How will we organize the development process? More precisely, where
> > should I place the implemented code in order to provide access to it
> > for our team? In my experience, I have used to create the separate
> > branch in the git repository. I would like to know which way of doing
> > this you would prefer.
>
> I forgot how we did this for past GSoC, but my preference is to do this
> in a separate branch of the Cuirass git repository.  Do you have an
> account on Savannah yet?  Once you do we could give you permissions to
> push your work to a separate branch on the repository.
>
> (You are free to host the code elsewhere as long as we have read
> access via git.)
>
> --
> Ricardo
>
>
>

Re: GSoC: Adding a web interface similar to the Hydra web interface

[Ricardo Wurmus]

Hi Tatiana,

> I have started thinking about the type of web interface we want to have for
> Cuirass in this project. As far as I see, there are two options:
>
>- a web application served by Cuirass web server;
>- a standalone static site which sends queries to the Cuirass web API
>(this is similar to Danny's application);
>
> I suppose that the first option has more benefits since it allows to choose
> the exact type of information required to be extracted from the database by
> a specific part of the web interface. What do think regarding these options?

You are free to extend the Cuirass web API to suit your application’s
needs.  Having a standalone site is a valid way of providing a web
interface, but it doesn’t have to be the only way of accessing the
information.

Even if you go for the first route, the HTML you serve could talk to the
API.  These two options don’t have to be mutually exclusive.

FWIW, I expect the first approach to be easier because you can use
Scheme for the most part; the pages it serves could be progressively
enhanced with JavaScript, if the client supports it.  I’m sure there are
many users who would prefer a system that would work fine even without
running JavaScript in the browser.

> How will we organize the development process? More precisely, where
> should I place the implemented code in order to provide access to it
> for our team? In my experience, I have used to create the separate
> branch in the git repository. I would like to know which way of doing
> this you would prefer.

I forgot how we did this for past GSoC, but my preference is to do this
in a separate branch of the Cuirass git repository.  Do you have an
account on Savannah yet?  Once you do we could give you permissions to
push your work to a separate branch on the repository.

(You are free to host the code elsewhere as long as we have read
access via git.)

--
Ricardo

Re: GSoC: Adding a web interface similar to the Hydra web interface

[Tatiana Sholokhova]

Hello everyone,

Thank you for the proposed ways of dealing with the problem I've faced.
Eventually, I have managed to run Danny's application locally. I have
learned all the queries to Cuirass web API which the application sends and
the general workflow of the interface.

I have started thinking about the type of web interface we want to have for
Cuirass in this project. As far as I see, there are two options:

   - a web application served by Cuirass web server;
   - a standalone static site which sends queries to the Cuirass web API
   (this is similar to Danny's application);

I suppose that the first option has more benefits since it allows to choose
the exact type of information required to be extracted from the database by
a specific part of the web interface. What do think regarding these options?

Since I am a newcomer to the guix developers team, I have a question
related to the interface development workflow. Once we choose the desired
type of the web interface, I will start implementing it. How will we
organize the development process? More precisely, where should I place the
implemented code in order to provide access to it for our team? In my
experience, I have used to create the separate branch in the git
repository. I would like to know which way of doing this you would prefer.

Best regards,
Tatiana

2018-05-14 7:20 GMT+03:00 Ricardo Wurmus :

>
> Hi Tatiana,
>
> > Unfortunately, I have no experience of IRC usage and I'm not sure how to
> do
> > in a proper way. Of course, I would like to communicate with the
> community,
> > but I'm not sure which discussions are relevant in the IRC. For instance,
> > could I ask questions on building issues similar to ones I've
> encountered?
>
> Yes, anything relating to the project that is easier discussed live is
> appropriate for the IRC channel.
>
> If you don’t have an IRC client you can use
>
> https://webchat.freenode.net/
>
> Input any nickname and #guix as the channel.
>
> --
> Ricardo
>
>
>

Re: GSoC: Adding a web interface similar to the Hydra web interface

[Chris Marusich]

Tatiana Sholokhova  writes:

> Unfortunately, I have no experience of IRC usage and I'm not sure how to do
> in a proper way. Of course, I would like to communicate with the community,
> but I'm not sure which discussions are relevant in the IRC. For instance,
> could I ask questions on building issues similar to ones I've encountered?

If you need a specific recommendation, I'd suggest you try HexChat for
IRC.  It's available in Guix via "guix package -i hexchat".  There are
other clients, but I've found this one works quite nicely for my simple
needs.

Another popular option I know of is Pidgin, which is also packaged in
Guix.  I haven't used the Guix-installed Pidgin before, though, so I
don't know how well it works at this time.

-- 
Chris


signature.asc
Description: PGP signature

Re: GSoC: Adding a web interface similar to the Hydra web interface

[Ricardo Wurmus]

Hi Tatiana,

> Unfortunately, I have no experience of IRC usage and I'm not sure how to do
> in a proper way. Of course, I would like to communicate with the community,
> but I'm not sure which discussions are relevant in the IRC. For instance,
> could I ask questions on building issues similar to ones I've encountered?

Yes, anything relating to the project that is easier discussed live is
appropriate for the IRC channel.

If you don’t have an IRC client you can use

https://webchat.freenode.net/

Input any nickname and #guix as the channel.

-- 
Ricardo

Re: GSoC: Adding a web interface similar to the Hydra web interface

[Tonton]

Hi!

As far as IRC is concerned you're welcome to take any questions or
conversations that you have there. When people are online it tends to be
faster, and some problems - like live troubleshooting - is better suited to
IRC.

Personally I ask on IRC when I have a question. If it isn't answered or if my
problem is more complex, or maybe it requires a longer explanation, I'll send
it to the mailing list.

All you really need to try is an IRC client (there are also ones you can run
in your browser). And join server chat.freenode.net (port 6697 (or 6667
without ssl)) and channel #guix

Drop by and say hello if you like.

I'm not familiar with cuirass, so I can't help there, sorry. But good luck!

On Sun, 13 May 2018 21:45:04 +0300
Tatiana Sholokhova  wrote:

> Hi all,
> 
> Thank you for your help and the provided resources which are very useful
> for me at the first stage of the project.
> 
> I have built Cuirass on my local computer but I have encountered a few
> problems while doing it. It turned out that I had an old version of
> guile-sqlite3 installed by guix. I have spent some time discovering this
> issue. Eventually, I've found that all I had to do is to update guix
> packages list invoking "guix pull" and then update the packages.
> 
> Unfortunately, I have no experience of IRC usage and I'm not sure how to do
> in a proper way. Of course, I would like to communicate with the community,
> but I'm not sure which discussions are relevant in the IRC. For instance,
> could I ask questions on building issues similar to ones I've encountered?
> 
> I have taken a look at Danny's Cuirass frontend application. Now I try to
> run it locally. I have already figured out that I need to change URLPREFIX
> and name of the repository and the branch in the code. But I still can't
> get it working. According to the browser console, all the queries to
> localhost sent by js receive following error:
> 
> "No 'Access-Control-Allow-Origin' header is present on the requested
> resource. Origin 'null' is therefore not allowed access."
> 
> But similar queries work well if I enter them manually via the browser or
> curl. Could you give me any hints on that?
> 
> Best regards,
> Tatiana
> 
> 2018-05-09 20:21 GMT+03:00 Ricardo Wurmus :
> 
> > Hi Tatiana,
> >
> > we haven’t heard from you since about a week, nor have I seen you on
> > IRC.  (Granted, I was offline for some time, so I probably just missed
> > your introduction.)
> >
> > The official so-called “community bonding period” of GSoC is soon coming
> > to an end, and it would be advisable to use the remaining time before
> > coding begins for some community bonding :)
> >
> > I’d like to repeat that frequent communication is the key to a
> > successful GSoC project.  If you have any questions about the project,
> > the code, or how to proceed, please don’t hesitate to let us know.
> >
> > Thanks!
> >
> > --
> > Ricardo
> >
> >  



-- 
I use gpg to sign my emails. All the symbols you may see at the bottom
of this mail is my cryptographic signature. It can be ignored, or used
to check that it really is me sending this email. Learn more by asking
me or see: https://u.fsf.org/zb or https://ssd.eff.org/


pgp0gGg_0ZhAj.pgp
Description: OpenPGP digital signature

Re: GSoC: Adding a web interface similar to the Hydra web interface

[Danny Milosavljevic]

Hi Tatiana,

> I have taken a look at Danny's Cuirass frontend application. Now I try to
> run it locally. I have already figured out that I need to change URLPREFIX
> and name of the repository and the branch in the code. But I still can't
> get it working. According to the browser console, all the queries to
> localhost sent by js receive following error:
> 
> "No 'Access-Control-Allow-Origin' header is present on the requested
> resource. Origin 'null' is therefore not allowed access."
> 
> But similar queries work well if I enter them manually via the browser or
> curl. Could you give me any hints on that?

You're trying to run it as a file from your local computer, right?

Then it's security settings in your web browser preventing cross-site scripting
attacks.

If you use icecat, please install this extension:

  https://addons.mozilla.org/de/firefox/addon/cors-everywhere/

There's an square icon at the top right of the icecat window then (it says 
"cors").

If you click on it it will toggle between green and red.

When it's green it means that it will let all requests pass.  WHEN IT'S GREEN, 
DON'T USE ONLINE BANKING or similar in the same session.

But the index.html frontend application will work just fine then :)

When it's red it means that cross-site scripting attack protection is online.  
If in doubt, leave it red.  It should be red most of the time.

I myself develop like that:

For testing,

* Switch CorsE to green
* Refresh Cuirass frontend index.html page
* Try some stuff
* Close Cuirass frontend index.html page
* Switch CorsE to red

Re: GSoC: Adding a web interface similar to the Hydra web interface

[Gábor Boskovits]

2018-05-13 20:45 GMT+02:00 Tatiana Sholokhova :

> Hi all,
>
> Thank you for your help and the provided resources which are very useful
> for me at the first stage of the project.
>
> I have built Cuirass on my local computer but I have encountered a few
> problems while doing it. It turned out that I had an old version of
> guile-sqlite3 installed by guix. I have spent some time discovering this
> issue. Eventually, I've found that all I had to do is to update guix
> packages list invoking "guix pull" and then update the packages.
>
> Unfortunately, I have no experience of IRC usage and I'm not sure how to
> do in a proper way. Of course, I would like to communicate with the
> community, but I'm not sure which discussions are relevant in the IRC. For
> instance, could I ask questions on building issues similar to ones I've
> encountered?
>
> I have taken a look at Danny's Cuirass frontend application. Now I try to
> run it locally. I have already figured out that I need to change URLPREFIX
> and name of the repository and the branch in the code. But I still can't
> get it working. According to the browser console, all the queries to
> localhost sent by js receive following error:
>
> "No 'Access-Control-Allow-Origin' header is present on the requested
> resource. Origin 'null' is therefore not allowed access."
>
> But similar queries work well if I enter them manually via the browser or
> curl. Could you give me any hints on that?
>
>
Hello Tatiana,

by any chance do you run this code through a file:///... url? Origin null
usually means that. If so, it would be better to test in a locally
installed webserver.


> Best regards,
> Tatiana
>
> 2018-05-09 20:21 GMT+03:00 Ricardo Wurmus :
>
>> Hi Tatiana,
>>
>> we haven’t heard from you since about a week, nor have I seen you on
>> IRC.  (Granted, I was offline for some time, so I probably just missed
>> your introduction.)
>>
>> The official so-called “community bonding period” of GSoC is soon coming
>> to an end, and it would be advisable to use the remaining time before
>> coding begins for some community bonding :)
>>
>> I’d like to repeat that frequent communication is the key to a
>> successful GSoC project.  If you have any questions about the project,
>> the code, or how to proceed, please don’t hesitate to let us know.
>>
>> Thanks!
>>
>> --
>> Ricardo
>>
>>
>

Re: GSoC: Adding a web interface similar to the Hydra web interface

[Tatiana Sholokhova]

Hi all,

Thank you for your help and the provided resources which are very useful
for me at the first stage of the project.

I have built Cuirass on my local computer but I have encountered a few
problems while doing it. It turned out that I had an old version of
guile-sqlite3 installed by guix. I have spent some time discovering this
issue. Eventually, I've found that all I had to do is to update guix
packages list invoking "guix pull" and then update the packages.

Unfortunately, I have no experience of IRC usage and I'm not sure how to do
in a proper way. Of course, I would like to communicate with the community,
but I'm not sure which discussions are relevant in the IRC. For instance,
could I ask questions on building issues similar to ones I've encountered?

I have taken a look at Danny's Cuirass frontend application. Now I try to
run it locally. I have already figured out that I need to change URLPREFIX
and name of the repository and the branch in the code. But I still can't
get it working. According to the browser console, all the queries to
localhost sent by js receive following error:

"No 'Access-Control-Allow-Origin' header is present on the requested
resource. Origin 'null' is therefore not allowed access."

But similar queries work well if I enter them manually via the browser or
curl. Could you give me any hints on that?

Best regards,
Tatiana

2018-05-09 20:21 GMT+03:00 Ricardo Wurmus :

> Hi Tatiana,
>
> we haven’t heard from you since about a week, nor have I seen you on
> IRC.  (Granted, I was offline for some time, so I probably just missed
> your introduction.)
>
> The official so-called “community bonding period” of GSoC is soon coming
> to an end, and it would be advisable to use the remaining time before
> coding begins for some community bonding :)
>
> I’d like to repeat that frequent communication is the key to a
> successful GSoC project.  If you have any questions about the project,
> the code, or how to proceed, please don’t hesitate to let us know.
>
> Thanks!
>
> --
> Ricardo
>
>

Re: GSoC: Adding a web interface similar to the Hydra web interface

[Ricardo Wurmus]

Hi Danny,

> Hi Tatjana and welcome!
>
> On Fri, 04 May 2018 14:55:01 +0200
> l...@gnu.org (Ludovic Courtès) wrote:
>
>> In terms of Web interface, Danny Milosavljevic wrote a simple HTML page
>> with JavaScript code that queries Cuirass using its HTTP interface².
>> You can see it here:
>> 
>>   https://berlin.guixsd.org/status/
>
> For a very quick way to get started, you can click "View Source" in your web
> browser and you'll see its source code.
>
> It queries the Cuirass CI server.
>
> If you have any questions, I'm reachable via the list - but not via IRC.
>
> There's a more current version of the program:
>
> https://git.savannah.gnu.org/cgit/guix/maintenance.git/tree/hydra/nginx/html/status/index.html
>
> @Ludo: Can you update it on berlin.guixsd.org ?

I’ll see if I can do that today.

Thanks!

-- 
Ricardo

Re: GSoC: Adding a web interface similar to the Hydra web interface

[Ricardo Wurmus]

Hi Tatiana,

we haven’t heard from you since about a week, nor have I seen you on
IRC.  (Granted, I was offline for some time, so I probably just missed
your introduction.)

The official so-called “community bonding period” of GSoC is soon coming
to an end, and it would be advisable to use the remaining time before
coding begins for some community bonding :)

I’d like to repeat that frequent communication is the key to a
successful GSoC project.  If you have any questions about the project,
the code, or how to proceed, please don’t hesitate to let us know.

Thanks!

--
Ricardo

Re: GSoC: Adding a web interface similar to the Hydra web interface

[Danny Milosavljevic]

Hi Tatjana and welcome!

On Fri, 04 May 2018 14:55:01 +0200
l...@gnu.org (Ludovic Courtès) wrote:

> In terms of Web interface, Danny Milosavljevic wrote a simple HTML page
> with JavaScript code that queries Cuirass using its HTTP interface².
> You can see it here:
> 
>   https://berlin.guixsd.org/status/

For a very quick way to get started, you can click "View Source" in your web
browser and you'll see its source code.

It queries the Cuirass CI server.

If you have any questions, I'm reachable via the list - but not via IRC.

There's a more current version of the program:

https://git.savannah.gnu.org/cgit/guix/maintenance.git/tree/hydra/nginx/html/status/index.html

@Ludo: Can you update it on berlin.guixsd.org ?


pgpPLWvQvXvgX.pgp
Description: OpenPGP digital signature

Re: GSoC: Adding a web interface similar to the Hydra web interface

[Ricardo Wurmus]

Hi Tatiana,

welcome to Guix!

> A good starting point for you will be to take a look at the basic ideas
> behind Cuirass: what it does, how it structures information about builds
> that it performs (which can be inferred from its database¹).

Some time ago I tried to document the database schema.  This can be
found in the manual.  The source file for the manual is here:

  https://git.savannah.gnu.org/cgit/guix/guix-cuirass.git/tree/doc/cuirass.texi

This may not be 100% accurate, but it may be helpful in getting to
understand what Cuirass stores and why.

--
Ricardo

Re: GSoC: Adding a web interface similar to the Hydra web interface

[Ludovic Courtès]

Hello Tatiana,

Tatiana Sholokhova  skribis:

> I'm Tatiana, I'm a GSoC intern. Now, I'm pursuing my master degree
> in Computer Science at Moscow State University.
>
> My project in GSoC is implementation web interface analogous to Hydra
> interface.

Thanks for your message, and welcome to Guix!

A good starting point for you will be to take a look at the basic ideas
behind Cuirass: what it does, how it structures information about builds
that it performs (which can be inferred from its database¹).

In terms of Web interface, Danny Milosavljevic wrote a simple HTML page
with JavaScript code that queries Cuirass using its HTTP interface².
You can see it here:

  https://berlin.guixsd.org/status/

It’s rough on the edges but it gives you an idea.

Anyway, these are the entry points.  Please do get in touch with us as
you start digging into it.  People on the mailing list and on IRC can
help (I’m “civodul” on IRC.)

Happy hacking!  :-)

Ludo’.

¹ https://git.savannah.gnu.org/cgit/guix/guix-cuirass.git/tree/src/schema.sql
² 
https://git.savannah.gnu.org/cgit/guix/guix-cuirass.git/tree/src/cuirass/http.scm

Re: GSoC: Adding a web interface similar to the Hydra web interface

[Maxim Cournoyer]

Hello!

Tatiana Sholokhova  writes:

> Hello!
>
> I'm Tatiana, I'm a GSoC intern. Now, I'm pursuing my master degree
> in Computer Science at Moscow State University.

Welcome :)

> My project in GSoC is implementation web interface analogous to Hydra
> interface.

Sounds interesting and useful!

> I am really happy to be accepted for GSoC and excited about the project!

See you in #guix; enjoy your SoC!

Maxim