Re: [GSoC 2017] Community Bonding

[Clemens Lang]

Hi,

On Fri, May 26, 2017 at 12:57:11AM +0530, Umesh Singla wrote:
> Since we are planning on 3 different actions which can also be used
> independently like snapshot and migrate, having 2 scripts in the
> macports1.0 directory directly or keeping these 2 commands in a single
> script doesn't seem to be good from design pov. We can have a separate
> folder for new (upcoming) actions with a hope that the existing code
> be split into "action" modules over the time.

Why wouldn't migrate.tcl and snapshot.tcl in the macports1.0 folder be
appropriate? That's more or less what we did with diagnose.tcl and
reclaim.tcl.

> > If you need any additional special SQL queries, that can be done,
> > yes. There should be abstractions available in Tcl already so that
> > you don't have to deal with SQLite's C API directly, though.
>
> The cregistry folder seems to have most of the sql queries implemented
> and to deal with the registry database, I'll only have create, update
> and select queries to do, so looks like I won't have to deal with C. I
> went through an entire "port uninstall" example to see how from Tcl
> gets to sqlite C through registry, so things seem a bit clear than
> before.

Yeah, I wouldn't expect too much extra SQL to be necessary, since
registry2.0 should already provide most (all?) the things you need.


> Is there anything even sacred to Tcl? Till now, it looks like you can
> just use anything in the language with pretty much anything else. This
> calls for a new blog post. And planning to discuss on the testing part
> in next email.

Yeah, Tcl is very versatile. If necessary, you can also redefine builtin
functions. I'm not sure if you can actually also redefine keywords (such
as if and for) – you can certainly define new actions that behave like
keywords, though. And to make it even more complicated, variable names
can contain pretty much everything, e.g. "is_valid?" would be a valid
variable name in Tcl.

> Also, can we have a small README with two to three lines kept in every
> folder in the base code at least? That'll be actually very helpful.

I agree with Rainer that we should extend HACKING and our existing
documentation with this rather than spreading it. Ideally, I'd find the
time to write down the general overview that I gave in the video.

-- 
Clemens

Re: [GSoC 2017] Community Bonding

[Jackson Isaac]

Hi,

On Thu, May 18, 2017 at 11:37 PM, Jackson Isaac  wrote:
> Hi GSoC Students and Mentors,
>
> I hope you are utilizing the community bonding period well and had
> some kind of introductory get to know interaction between
> student-mentor(s) already. If not yet, please do so, it would help you
> to get to know each other better, set up future plans and discuss any
> changes/shortcomings.
>
> neverpanic's session[1] during MacPorts Meeting should help you get
> started with macports-base and give you a good overview of macports
> internals. Please go through it if not yet done.
>
> It would be great to hear from you how is community bonding is going
> on till now. Let us know if you face any issues or need any help to
> setup meetings.
>
> [1] https://www.youtube.com/watch?v=46qshiDskrM
>

We are fast approaching end of Community Bonding Period and coding
starts immediately after that on 30th May 2017.

Kindly send in your experience till now to -dev ML and what things are
planned further. I hope you have connected at least once with your
mentor till now and are clear about your goals to begin with.

Let us know if you faced any issues till now e.g., setting up meeting,
code base related, mentor/student availability, etc..

-- 
Jackson Isaac

Re: [GSoC 2017] Community Bonding

[Rainer Müller]

On 05/25/2017 09:27 PM, Umesh Singla wrote:
> > About the project, so the `restore`, `snapshot` and `migrate` actions
> > are going to the action_array list [1]. While the flow was made clear
> > in the proposal, I think the first step should be to decide on an
> > exhaustive list of arguments/flags these 3 actions can possibly take.
> > This will help to design the procedures.
> >
> > Also, should we keep the code in a different Tcl script migrate.tcl
> > like `reclaim`?
> 
> I would prefer that, yes. You'll just need
>   package provides macports 1.0
> line at the top and then you can write code as if it was in
> macports.tcl. Alternatively, you can do what reclaim.tcl does and use
>   package provides migrate 1.0
> and add 'package require migrate 1.0' in macports.tcl.
> 
> 
> Since we are planning on 3 different actions which can also be used
> independently like snapshot and migrate, having 2 scripts in the macports1.0
> directory directly or keeping these 2 commands in a single script doesn't seem
> to be good from design pov. We can have a separate folder for new (upcoming)
> actions with a hope that the existing code be split into "action" modules over
> the time.

Not sure what you mean with "a new folder" here.

The macports 1.0 package is the public MacPorts API. Every feature should be
reachable from there. If you think separate files make more sense, split them
into multiple files (as done with reclaim.tcl or diagnose.tcl).

It sounds logical to me to keep snapshot and restore together, since they will
export to/import from the same data format. Migrate could be in its own file.

As a side note, while 'port migrate' seems important enough for a new port
action, aren't the former two only useful together? Then I would expect the
functionality in the same port action, such as for example:

  port snapshot --create
  port snapshot --restore

> Yes, the testing is not trivial at all. It might make sense to work with
> mocking a lot here (e.g. overwrite functions that would usually be
> called by your code to do nothing). That would allow you to run tests
> without actually changing your system or doing a lot of environment
> setup.
> 
> Fortunately Tcl makes this quite easy because you can change any
> function definition at runtime. 
> 
> 
> Is there anything even sacred to Tcl? Till now, it looks like you can just use
> anything in the language with pretty much anything else. This calls for a new
> blog post. And planning to discuss on the testing part in next email.
> 
> Also, can we have a small README with two to three lines kept in every folder 
> in
> the base code at least? That'll be actually very helpful. 

I am not convinced separate README files make sense instead of a single
consolidated documentation at one place. If I do not know where something is
implemented, how would I find the correct directory with the README to look at?
A single document seems easier to search and skim through to understand the
structure.

Developer documentation is somewhat shattered across multiple places. Some
information and pointers to the wiki are in HACKING at the top-level, while
other stuff is also in the guide [1]. If unsure, about anything please just ask.

[1] https://guide.macports.org/#internals

> Also, I tried StackOverflow but does tclsh support all bash commands?

There are different modes in which tclsh can be used.

In interactive mode, tclsh will fall back and call external binaries when no
proc or builtin exists with the name of the command you tried to run. This is
similar to the behavior of normal shells like bash. However, this will not
happen in non-interactive use, i.e. when interpreting a script (as in running
the port(1) command).

Rainer

Re: [GSoC 2017] Community Bonding

[Umesh Singla]

Hi



> > About the project, so the `restore`, `snapshot` and `migrate` actions
> > are going to the action_array list [1]. While the flow was made clear
> > in the proposal, I think the first step should be to decide on an
> > exhaustive list of arguments/flags these 3 actions can possibly take.
> > This will help to design the procedures.
> >
> > Also, should we keep the code in a different Tcl script migrate.tcl
> > like `reclaim`?
>
> I would prefer that, yes. You'll just need
>   package provides macports 1.0
> line at the top and then you can write code as if it was in
> macports.tcl. Alternatively, you can do what reclaim.tcl does and use
>   package provides migrate 1.0
> and add 'package require migrate 1.0' in macports.tcl.
>
>
Since we are planning on 3 different actions which can also be used
independently like snapshot and migrate, having 2 scripts in the
macports1.0 directory directly or keeping these 2 commands in a single
script doesn't seem to be good from design pov. We can have a separate
folder for new (upcoming) actions with a hope that the existing code be
split into "action" modules over the time.


>
> If you need any additional special SQL queries, that can be done, yes.
> There should be abstractions available in Tcl already so that you don't
> have to deal with SQLite's C API directly, though.
>
>
The cregistry folder seems to have most of the sql queries implemented and
to deal with the registry database, I'll only have create, update and
select queries to do, so looks like I won't have to deal with C. I went
through an entire "port uninstall" example to see how from Tcl gets to
sqlite C through registry, so things seem a bit clear than before.


>
> Yes, the testing is not trivial at all. It might make sense to work with
> mocking a lot here (e.g. overwrite functions that would usually be
> called by your code to do nothing). That would allow you to run tests
> without actually changing your system or doing a lot of environment
> setup.
>
> Fortunately Tcl makes this quite easy because you can change any
> function definition at runtime.


Is there anything even sacred to Tcl? Till now, it looks like you can just
use anything in the language with pretty much anything else. This calls for
a new blog post. And planning to discuss on the testing part in next email.

Also, can we have a small README with two to three lines kept in every
folder in the base code at least? That'll be actually very helpful.

As Clemens suggested on IRC, will start with an *honest* short daily log of
the work like what I did for the project and what I did related to MP etc
from today onwards.

Also, I tried StackOverflow but does tclsh support all bash commands?


Regards,
Umesh Singla

Re: [GSoC 2017] Community Bonding

[Umesh Singla]

I've got it. It's umeshksin...@macports.org and the GitHub account I'll be
using is github.com/umeshksingla.

Thanks

On May 23, 2017 3:39 AM, "Bradley Giesbrecht"  wrote:

> On May 22, 2017, at 12:22 PM, Umesh Singla  wrote:
>
> I have written two introductory posts on my blog [0] on Blogger.
>
> [0]: http://umeshsingla.blogspot.in/

Umesh, thanks for sharing this link.

In your second blog post you mention that by now you should have received
your MacPorts @macports.org handle. Please share your handle when you
receive it.

Also, please share the GitHub account you will be using for your MacPorts
work.


Regards,
Bradley Giesbrecht (pixilla)

Re: [GSoC 2017] Community Bonding

[Bradley Giesbrecht]

> On May 22, 2017, at 12:22 PM, Umesh Singla  wrote:
> 
> I have written two introductory posts on my blog [0] on Blogger.
> 
> [0]: http://umeshsingla.blogspot.in/

Umesh, thanks for sharing this link.

In your second blog post you mention that by now you should have received your 
MacPorts @macports.org handle. Please share your handle when you receive it.

Also, please share the GitHub account you will be using for your MacPorts work.


Regards,
Bradley Giesbrecht (pixilla)

Re: [GSoC 2017] Community Bonding

[Clemens Lang]

Hi,

On Tue, May 23, 2017 at 12:52:55AM +0530, Umesh Singla wrote:
> I have written two introductory posts on my blog [0] on Blogger. I
> thought I shared the link to it in the proposal but no, I didn't. The
> one is just when the results got announced and other about my take on
> the macports-base video.

Let me know if you have any questions about the codebase that you'd like
to see answered.


> About the project, so the `restore`, `snapshot` and `migrate` actions
> are going to the action_array list [1]. While the flow was made clear
> in the proposal, I think the first step should be to decide on an
> exhaustive list of arguments/flags these 3 actions can possibly take.
> This will help to design the procedures.
> 
> Also, should we keep the code in a different Tcl script migrate.tcl
> like `reclaim`?

I would prefer that, yes. You'll just need
  package provides macports 1.0
line at the top and then you can write code as if it was in
macports.tcl. Alternatively, you can do what reclaim.tcl does and use
  package provides migrate 1.0
and add 'package require migrate 1.0' in macports.tcl.

> and code dealing with the database is going to be in C in cregistry's
> sql.c [2], right? (like Tcl calling registry2.0 code currently?)

If you need any additional special SQL queries, that can be done, yes.
There should be abstractions available in Tcl already so that you don't
have to deal with SQLite's C API directly, though.


> All this is still trivial. What worries me most is the testing part.
> I'm not sure Brad, how are we going to take on the testing phase. We
> can virtually change specifications to see if the script works or
> compare the two docker images? Also, it needs a set of tests that
> should include all combinations of variants,
> requested-activate-deactivate scenarios. This is one big task to be
> considered.

Yes, the testing is not trivial at all. It might make sense to work with
mocking a lot here (e.g. overwrite functions that would usually be
called by your code to do nothing). That would allow you to run tests
without actually changing your system or doing a lot of environment
setup.

Fortunately Tcl makes this quite easy because you can change any
function definition at runtime. See [1] and [2] where I used this
technique to make the test think it runs as root or runs on a machine
with a different number of processors.


> Guide me to the first step, thanks. It'd be great if you want to have
> a FaceTime or Skype. I think after 0900 (UTC -8) would suit you best?

Let me know if you have any questions after that. I'm always happy to
help!


[1] 
https://github.com/macports/macports-base/blob/master/src/port1.0/tests/portactivate.test#L29-L48
[2] 
https://github.com/macports/macports-base/blob/master/src/port1.0/tests/portbuild.test#L26-L46

-- 
Clemens

Re: [GSoC 2017] Community Bonding

[Umesh Singla]

Hi

I have written two introductory posts on my blog [0] on Blogger. I thought
I shared the link to it in the proposal but no, I didn't. The one is just
when the results got announced and other about my take on the macports-base
video.

About the project, so the `restore`, `snapshot` and `migrate` actions are
going to the action_array list [1]. While the flow was made clear in the
proposal, I think the first step should be to decide on an exhaustive list
of arguments/flags these 3 actions can possibly take. This will help to
design the procedures.

Also, should we keep the code in a different Tcl script migrate.tcl like
`reclaim`? and code dealing with the database is going to be in C in
cregistry's sql.c [2], right? (like Tcl calling registry2.0 code
currently?)

All this is still trivial. What worries me most is the testing part. I'm
not sure Brad, how are we going to take on the testing phase. We can
virtually change specifications to see if the script works or compare the
two docker images? Also, it needs a set of tests that should include all
combinations of variants, requested-activate-deactivate scenarios. This is
one big task to be considered.

Guide me to the first step, thanks. It'd be great if you want to have a
FaceTime or Skype. I think after 0900 (UTC -8) would suit you best?


[0]: http://umeshsingla.blogspot.in/
[1]: https://github.com/macports/macports-base/blob/master/
src/port/port.tcl#L4264
[2]: https://github.com/macports/macports-base/blob/master/
src/cregistry/sql.c

On Mon, May 22, 2017 at 9:07 PM, Bradley Giesbrecht 
wrote:

> > On May 18, 2017, at 11:07 AM, Jackson Isaac 
> wrote:
> >
> > neverpanic's session[1] during MacPorts Meeting should help you get
> > started with macports-base and give you a good overview of macports
> > internals. Please go through it if not yet done.
> >
> > It would be great to hear from you how is community bonding is going
> > on till now. Let us know if you face any issues or need any help to
> > setup meetings.
> >
> > [1] https://www.youtube.com/watch?v=46qshiDskrM
>
>
> > On May 4, 2017, at 1:27 PM, Umesh Singla  wrote:
> >
> > Regarding the work, I plan to keep a blog to report my progress. I'll
> write a post about every two weeks, and keep email communication with my
> mentors.
>
> Hi Umesh,
>
> Do you have a blog address you are ready to share?
> Have you watched the video [1] Jackson referenced?
> Do you have any questions about MacPorts base, this project or MacPorts in
> general?
>
> Looking forward to hearing from you.
>
> Regards,
> Bradley Giesbrecht (pixilla)
>
>
>


-- 
Umesh Singla

Re: [GSoC 2017] Community Bonding

[Bradley Giesbrecht]

> On May 18, 2017, at 11:07 AM, Jackson Isaac  wrote:
> 
> neverpanic's session[1] during MacPorts Meeting should help you get
> started with macports-base and give you a good overview of macports
> internals. Please go through it if not yet done.
> 
> It would be great to hear from you how is community bonding is going
> on till now. Let us know if you face any issues or need any help to
> setup meetings.
> 
> [1] https://www.youtube.com/watch?v=46qshiDskrM


> On May 4, 2017, at 1:27 PM, Umesh Singla  wrote:
> 
> Regarding the work, I plan to keep a blog to report my progress. I'll write a 
> post about every two weeks, and keep email communication with my mentors.

Hi Umesh,

Do you have a blog address you are ready to share?
Have you watched the video [1] Jackson referenced?
Do you have any questions about MacPorts base, this project or MacPorts in 
general?

Looking forward to hearing from you.

Regards,
Bradley Giesbrecht (pixilla)

Re: [GSoC 2017] Community Bonding

[Mojca Miklavec]

Hi,

On 18 May 2017 at 20:07, Jackson Isaac wrote:
> Hi GSoC Students and Mentors,
>
> I hope you are utilizing the community bonding period well and had
> some kind of introductory get to know interaction between
> student-mentor(s) already. If not yet, please do so, it would help you
> to get to know each other better, set up future plans and discuss any
> changes/shortcomings.

I would also like to encourage everyone to keep all the discussions
both during community bonding and coding period in the open. Other
developers that are not your mentors might provide useful feedback and
quicker answers and many are simply eager to follow the progress of
the projects.

So unless you are sharing passwords or personal details that are
unrelated to the project, please try to keep the community involved in
the discussions. If traffic goes out of control, we can always set up
another list :) :) :)

Mojca