Re: [9fans] Software preservation in the post-hg era

[Dave MacFarlane]

>
>
>
> How does it compare performance wise with git9 ?
>
> https://github.com/oridb/git9
>

Based on the unscientific testing I just did of cloning the same repo
(dgit's) a few times,  they're in the same ballpark (~13s on 9front amd64.)
git9 had a couple faster runs and dgit a couple slower ones,  but nothing
outside of what could be explained by my wifi connection since the network
is the bottleneck in cloning.

--
9fans: 9fans
Permalink: 
https://9fans.topicbox.com/groups/9fans/T303744e1ec6d2108-Mc0071e2d08745673c7852cd2
Delivery options: https://9fans.topicbox.com/groups/9fans/subscription

Re: [9fans] Software preservation in the post-hg era

[ori]

> Den tors 7 maj 2020 16:17Dave MacFarlane  skrev:
> 
>>
>> On Mon, Mar 30, 2020 at 9:12 PM Sean Hinchee  wrote:
>>
>>> As a footnote, there's a decent git client written in Go that works
>>> alright on plan9 [4], but it's slow and memory intensive at the
>>> moment.
>>>
>>>
>> [...]
>>
>> [4] https://github.com/driusan/dgit
>>
>> This (and the fact that the speed of Go on Plan9/amd64 seems to be finally
>> be useable enough to do development again as of 1.14..) finally gave me the
>> kick I needed to fix some of the hacks that were causing performance
>> problems on clone. The self-clone time went from ~160s to ~13s on my
>> machine (compared to ~8s with "real" git) If there's other parts that you
>> were referring to as being slow and memory intensive let me know (or if you
>> still find it's memory intensive, I didn't benchmark that part..)
>>
>> - Dave
>>
> 
> 
> How does it compare performance wise with git9 ?
> 
> https://github.com/oridb/git9

I'll be honest, I'm using git9 because of the improved
interactions, rather than performance -- it's fast enough
for most of my usage. Still, this got curious enough to
test a bit. Here are the results:

It's close for cloning dgit -- I'm seeing about 3 seconds
for dgit with git/clone, 4.5 using dgit to clone itself.

% time git/clone https://github.com/driusan/dgit
0.81u 1.08s 2.70r 

(Looking closer, about 1.5 seconds of that comes
from the dircp to pull data out of /mnt/git/ and
into the working directory.)

When testing dgit, I redirected output to /dev/null,
since it printed enough that it affected the time.
It's *really* chatty -- for the larger test, it
produced more than 50 megabytes of status text.

cpu% time rc -c 'dgit clone https://github.com/driusan/dgit 
>[2]/dev/null'
0.47u 0.55s 4.32r

It seems like there's something accidentally
quadratic, though. Cloning a larger repository
-- in this case, perl5 -- takes 160s on git9,
and 1200 seconds on dgit. For comparison, git
on OpneBSD with different (but comparable)
hardware takes about 90 seconds.

cpu% time git/clone https://github.com/Perl/perl5.git
94.40u 14.16s 159.30r

cpu% time ./dgit clone https://github.com/Perl/perl5.git 
>[2]/tmp/dgit.log
121.93u 22.16s 1211.30r

I only skimmed the dgit code quickly, and didn't see
an obvious answer: do you cache objects that you've
decompressed, or do you iterate over full delta chains
every time?

One other bug report -- it seems that dgit hard-codes the
default branch as origin/master, but perl uses 'origin/blead',
so the checkout fails with 'Could not find origin/master'

There are still places where git9 is very slow. Sending lots
of commits at once in big repositories stands out.

Two reasons for this: we don't deltify, and we walk too much
data deciding what should go into the pack. There's also a
bug that causes certain kinds of merge to push the whole
history spuriously, which is.. only wasteful rather than
incorrect -- but wasteful isn't good.

Pushing all perl commits to an empty repository, for example:

# this is the size of the packfile git gives us
cpu% du -sh .git
297.043M.git

# pushing to git is slow
cpu% git/push -u git+ssh://192.168.1.10/tmp/p5.git
1783.08u 444.15s 2835.86r

# and our undeltified packfiles are 10x the size
# that they should be
$ du -sh p5.git; 
4.2Gp5.git

I can't compare with dgit, since dgit doesn't support ssh
pushes, and I'm not going to set up http pushes right now.


--
9fans: 9fans
Permalink: 
https://9fans.topicbox.com/groups/9fans/T303744e1ec6d2108-M9d351fd4564a026a1d6a58e8
Delivery options: https://9fans.topicbox.com/groups/9fans/subscription

Re: [9fans] Software preservation in the post-hg era

[Jim Manley]

I don't know if he's on here, but Al Kossow (aek at bitsavers dot org), the
Software Curator at the Computer History Museum in Mountain View, CA
(SillyCon Valley, just down the road from the Googleplex) might be
interested in this, if he isn't already aware of what's going on.  He's an
expert on The Good Fight Against Bit-Rot and has tools for preserving
pretty much everything in Original Flavor media formats, as well as on-line.

In addition to the software library collection, the museum also has a
separate library dedicated to preservation of the all-important
documentation which, if lost, often results in the loss of much of the
ability to comprehend the functionality of, let alone maintain, executables
and associated data.


On Mon, Mar 30, 2020 at 7:12 PM Sean Hinchee  wrote:

> In the wake of Bitbucket removing hg (Mercurial) support [1], I feel
> it's topical to bring up software preservation for the plan9
> community.
> 
> A lot of community contributed software has been put up on Bitbucket
> or other hg hosts over time (RIP Google Code), but no consolidated
> effort, to my knowledge, seems to have been made to index, let alone
> mirror, this software.
> 
> For now, as a stop-gap, I've made a GitHub organization in which I've
> consolidated most of what I had indexed from Bitbucket and a few other
> places.
> 
> Thanks to people like Ori Bernstein, we have a native git client for
> plan9 [3]; without a native client, this kind of transition wouldn't
> be nearly as simple, thank you.
> 
> I'm more than happy to add anyone interested in the curation of this
> archive to the GitHub organization. It would be nice to have spare
> hands around to add README's, mkfiles, and attributions where they
> have been missed or never existed.
> 
> In the long term, it would be nice to have a federated or otherwise
> decentralized solution to pooling community contributed software,
> especially keeping in mind ease of mirroring and picking up old
> projects as contributors come and go.
> 
> The contrib/ directory on sources and 9front are fine and good, but
> they are centralized. I don't have a proposed solution to this
> problem, but it would be nice to have ideas or insight posted ☺.
> 
> I recognize that GitHub is also centralized and doesn't solve the
> centralization problem, but at least git is really straightforward to
> mirror with multiple remotes, etc. and having an index/archive is
> valuable at least to me.
> 
> If anyone has further thoughts, anything they want added, or any lists
> or indices of works they want archived/mirrored, I would love to see
> these posted.
> 
> If anyone wants to mirror the archive, that would be wonderful. I was
> considering mirroring everything to a remote in sr.ht in the future,
> but haven't gotten around to it.
> 
> As a footnote, there's a decent git client written in Go that works
> alright on plan9 [4], but it's slow and memory intensive at the
> moment.
> 
> Cheers,
> Sean
> 
> [1] https://twitter.com/traverser/status/1244398479591563265
> [2] https://github.com/Plan9-Archive
> [3] https://github.com/oridb/git9
> [4] https://github.com/driusan/dgit

--
9fans: 9fans
Permalink: 
https://9fans.topicbox.com/groups/9fans/T303744e1ec6d2108-Ma9aa770fc935c1c4c175566a
Delivery options: https://9fans.topicbox.com/groups/9fans/subscription

Re: [9fans] Software preservation in the post-hg era

[hiro]

> state of shiny for Plan 9 from your advantage point? It deserves a

since you posted this question to the list:
what is shiny?

--
9fans: 9fans
Permalink: 
https://9fans.topicbox.com/groups/9fans/T303744e1ec6d2108-M24bd2c157af206cb71b81655
Delivery options: https://9fans.topicbox.com/groups/9fans/subscription

Re: [9fans] Software preservation in the post-hg era

[Jens Staal]

Den tors 7 maj 2020 16:17Dave MacFarlane  skrev:

>
> On Mon, Mar 30, 2020 at 9:12 PM Sean Hinchee  wrote:
>
>> As a footnote, there's a decent git client written in Go that works
>> alright on plan9 [4], but it's slow and memory intensive at the
>> moment.
>>
>>
> [...]
>
> [4] https://github.com/driusan/dgit
>
> This (and the fact that the speed of Go on Plan9/amd64 seems to be finally
> be useable enough to do development again as of 1.14..) finally gave me the
> kick I needed to fix some of the hacks that were causing performance
> problems on clone. The self-clone time went from ~160s to ~13s on my
> machine (compared to ~8s with "real" git) If there's other parts that you
> were referring to as being slow and memory intensive let me know (or if you
> still find it's memory intensive, I didn't benchmark that part..)
>
> - Dave
>


How does it compare performance wise with git9 ?

https://github.com/oridb/git9


*9fans * / 9fans / see discussions
>  + participants
>  + delivery options
>  Permalink
> 
>

--
9fans: 9fans
Permalink: 
https://9fans.topicbox.com/groups/9fans/T303744e1ec6d2108-M9c55a9ba9a94eef7fa085c7f
Delivery options: https://9fans.topicbox.com/groups/9fans/subscription

Re: [9fans] Software preservation in the post-hg era

[Lucio De Re]

Kudos to you, Dave, in spades!

Could you please contact me off-list with a very brief summary of the
state of shiny for Plan 9 from your advantage point? It deserves a
full conference, but that isn't going to happen, is it?

Lucio.

--
9fans: 9fans
Permalink: 
https://9fans.topicbox.com/groups/9fans/T303744e1ec6d2108-M36eb934a6c8fb41819b44dc5
Delivery options: https://9fans.topicbox.com/groups/9fans/subscription

Re: [9fans] Software preservation in the post-hg era

[Dave MacFarlane]

On Mon, Mar 30, 2020 at 9:12 PM Sean Hinchee  wrote:

> As a footnote, there's a decent git client written in Go that works
> alright on plan9 [4], but it's slow and memory intensive at the
> moment.
>
>
[...]

[4] https://github.com/driusan/dgit

This (and the fact that the speed of Go on Plan9/amd64 seems to be finally
be useable enough to do development again as of 1.14..) finally gave me the
kick I needed to fix some of the hacks that were causing performance
problems on clone. The self-clone time went from ~160s to ~13s on my
machine (compared to ~8s with "real" git) If there's other parts that you
were referring to as being slow and memory intensive let me know (or if you
still find it's memory intensive, I didn't benchmark that part..)

- Dave

--
9fans: 9fans
Permalink: 
https://9fans.topicbox.com/groups/9fans/T303744e1ec6d2108-Mbc073232aabcf86a9843
Delivery options: https://9fans.topicbox.com/groups/9fans/subscription

Re: [9fans] Software preservation in the post-hg era

[Charles Forsyth]

I moved all my hg repositories to git some time ago

On Tue, Mar 31, 2020 at 7:39 PM Dave MacFarlane  wrote:

> On Tue, Mar 31, 2020 at 10:46 AM  wrote:
> > > If anyone has further thoughts, anything they want added, or any lists
> > > or indices of works they want archived/mirrored, I would love to see
> > > these posted.
> >
> > I think the lede got buried here, and people went of discussing git
> > clients.
> >
> > Given that the aptly-named bitbucket is deleting mercurial repositories,
> > is there anything of interest that should be retrieved before they empty
> > the bitbucket onto the curb?
> >
> 
> Is Inferno already mirrored?

--
9fans: 9fans
Permalink: 
https://9fans.topicbox.com/groups/9fans/T303744e1ec6d2108-M9682d3b12c0c9f00a4afed22
Delivery options: https://9fans.topicbox.com/groups/9fans/subscription

Re: [9fans] Software preservation in the post-hg era

[Fazlul Shahriar]

It's worth highlighting the fact that Bitbucket will be deleting *all* hg
repositories on May 30, 2020. Thankfully, the Software Heritage (
https://www.softwareheritage.org/) seems to be doing a great job of
archiving all open source work. For example:
https://archive.softwareheritage.org/browse/origin/https://bitbucket.org/inferno-os/inferno-os/directory/
I don't know how much they have archived or how up-to-date it is. It's
worth converting repos to git anyway, especially if we want to continue
maintaining them.

On Mon, Mar 30, 2020 at 9:11 PM Sean Hinchee  wrote:

> In the wake of Bitbucket removing hg (Mercurial) support [1], I feel
> it's topical to bring up software preservation for the plan9
> community.
> 
> A lot of community contributed software has been put up on Bitbucket
> or other hg hosts over time (RIP Google Code), but no consolidated
> effort, to my knowledge, seems to have been made to index, let alone
> mirror, this software.
> 
> For now, as a stop-gap, I've made a GitHub organization in which I've
> consolidated most of what I had indexed from Bitbucket and a few other
> places.
> 
> Thanks to people like Ori Bernstein, we have a native git client for
> plan9 [3]; without a native client, this kind of transition wouldn't
> be nearly as simple, thank you.
> 
> I'm more than happy to add anyone interested in the curation of this
> archive to the GitHub organization. It would be nice to have spare
> hands around to add README's, mkfiles, and attributions where they
> have been missed or never existed.
> 
> In the long term, it would be nice to have a federated or otherwise
> decentralized solution to pooling community contributed software,
> especially keeping in mind ease of mirroring and picking up old
> projects as contributors come and go.
> 
> The contrib/ directory on sources and 9front are fine and good, but
> they are centralized. I don't have a proposed solution to this
> problem, but it would be nice to have ideas or insight posted ☺.
> 
> I recognize that GitHub is also centralized and doesn't solve the
> centralization problem, but at least git is really straightforward to
> mirror with multiple remotes, etc. and having an index/archive is
> valuable at least to me.
> 
> If anyone has further thoughts, anything they want added, or any lists
> or indices of works they want archived/mirrored, I would love to see
> these posted.
> 
> If anyone wants to mirror the archive, that would be wonderful. I was
> considering mirroring everything to a remote in sr.ht in the future,
> but haven't gotten around to it.
> 
> As a footnote, there's a decent git client written in Go that works
> alright on plan9 [4], but it's slow and memory intensive at the
> moment.
> 
> Cheers,
> Sean
> 
> [1] https://twitter.com/traverser/status/1244398479591563265
> [2] https://github.com/Plan9-Archive
> [3] https://github.com/oridb/git9
> [4] https://github.com/driusan/dgit

--
9fans: 9fans
Permalink: 
https://9fans.topicbox.com/groups/9fans/T303744e1ec6d2108-M3889534780ac09733b7bf1cd
Delivery options: https://9fans.topicbox.com/groups/9fans/subscription

Re: [9fans] Software preservation in the post-hg era

[Sigrid Solveig Haflínudóttir]

On Tue, Mar 31, 2020 at 8:38 PM Dave MacFarlane  wrote:
>
> On Tue, Mar 31, 2020 at 10:46 AM  wrote:
> > > If anyone has further thoughts, anything they want added, or any lists
> > > or indices of works they want archived/mirrored, I would love to see
> > > these posted.
> >
> > I think the lede got buried here, and people went of discussing git
> > clients.
> >
> > Given that the aptly-named bitbucket is deleting mercurial repositories,
> > is there anything of interest that should be retrieved before they empty
> > the bitbucket onto the curb?
> >
>
> Is Inferno already mirrored?

If https://github.com/Plan9-Archive/inferno-os is the one then yes.
There are multiple other forks of Inferno there.

--
9fans: 9fans
Permalink: 
https://9fans.topicbox.com/groups/9fans/T303744e1ec6d2108-Mb217461f21ece2f1e8125545
Delivery options: https://9fans.topicbox.com/groups/9fans/subscription

Re: [9fans] Software preservation in the post-hg era

[Xiao-Yong Jin]

Thanks for doing this.
Please add me (jxy) to the GitHub org.
I plan to maintain drawterm-metal so long as I'm using it.


--
9fans: 9fans
Permalink: 
https://9fans.topicbox.com/groups/9fans/T303744e1ec6d2108-Mb35e89ba494b39cee28abc84
Delivery options: https://9fans.topicbox.com/groups/9fans/subscription

Re: [9fans] Software preservation in the post-hg era

[Dave MacFarlane]

On Tue, Mar 31, 2020 at 10:46 AM  wrote:
> > If anyone has further thoughts, anything they want added, or any lists
> > or indices of works they want archived/mirrored, I would love to see
> > these posted.
>
> I think the lede got buried here, and people went of discussing git
> clients.
>
> Given that the aptly-named bitbucket is deleting mercurial repositories,
> is there anything of interest that should be retrieved before they empty
> the bitbucket onto the curb?
>

Is Inferno already mirrored?

--
9fans: 9fans
Permalink: 
https://9fans.topicbox.com/groups/9fans/T303744e1ec6d2108-M8ffeabfca5531d4d9caffcb1
Delivery options: https://9fans.topicbox.com/groups/9fans/subscription

Re: [9fans] Software preservation in the post-hg era

[Lucio De Re]

On 3/31/20, o...@eigenstate.org  wrote:
>
> I think the lede got buried here, and people went of discussing git
> clients.
>
Entirely your fault: that's the thanks you get for doing such a good job.

Lucio.

--
9fans: 9fans
Permalink: 
https://9fans.topicbox.com/groups/9fans/T303744e1ec6d2108-M76529bf4c6408c5c3b474d4f
Delivery options: https://9fans.topicbox.com/groups/9fans/subscription

Re: [9fans] Software preservation in the post-hg era

[ori]

> If anyone has further thoughts, anything they want added, or any lists
> or indices of works they want archived/mirrored, I would love to see
> these posted.

I think the lede got buried here, and people went of discussing git
clients.

Given that the aptly-named bitbucket is deleting mercurial repositories,
is there anything of interest that should be retrieved before they empty
the bitbucket onto the curb?


--
9fans: 9fans
Permalink: 
https://9fans.topicbox.com/groups/9fans/T303744e1ec6d2108-M43dd6e182ede2169dd8b55b7
Delivery options: https://9fans.topicbox.com/groups/9fans/subscription

Re: [9fans] Software preservation in the post-hg era

[Fazlul Shahriar]

>
> As a footnote, there's a decent git client written in Go that works
> alright on plan9 [4], but it's slow and memory intensive at the
> moment.


 go-git also works well. I recently wrote a wrapper around it for plan9:
https://github.com/fhs/gig

--
9fans: 9fans
Permalink: 
https://9fans.topicbox.com/groups/9fans/T303744e1ec6d2108-M5a0ab076eb11ff1ddb369ae6
Delivery options: https://9fans.topicbox.com/groups/9fans/subscription

[9fans] Software preservation in the post-hg era

[Sean Hinchee]

In the wake of Bitbucket removing hg (Mercurial) support [1], I feel
it's topical to bring up software preservation for the plan9
community.

A lot of community contributed software has been put up on Bitbucket
or other hg hosts over time (RIP Google Code), but no consolidated
effort, to my knowledge, seems to have been made to index, let alone
mirror, this software.

For now, as a stop-gap, I've made a GitHub organization in which I've
consolidated most of what I had indexed from Bitbucket and a few other
places.

Thanks to people like Ori Bernstein, we have a native git client for
plan9 [3]; without a native client, this kind of transition wouldn't
be nearly as simple, thank you.

I'm more than happy to add anyone interested in the curation of this
archive to the GitHub organization. It would be nice to have spare
hands around to add README's, mkfiles, and attributions where they
have been missed or never existed.

In the long term, it would be nice to have a federated or otherwise
decentralized solution to pooling community contributed software,
especially keeping in mind ease of mirroring and picking up old
projects as contributors come and go.

The contrib/ directory on sources and 9front are fine and good, but
they are centralized. I don't have a proposed solution to this
problem, but it would be nice to have ideas or insight posted ☺.

I recognize that GitHub is also centralized and doesn't solve the
centralization problem, but at least git is really straightforward to
mirror with multiple remotes, etc. and having an index/archive is
valuable at least to me.

If anyone has further thoughts, anything they want added, or any lists
or indices of works they want archived/mirrored, I would love to see
these posted.

If anyone wants to mirror the archive, that would be wonderful. I was
considering mirroring everything to a remote in sr.ht in the future,
but haven't gotten around to it.

As a footnote, there's a decent git client written in Go that works
alright on plan9 [4], but it's slow and memory intensive at the
moment.

Cheers,
Sean

[1] https://twitter.com/traverser/status/1244398479591563265
[2] https://github.com/Plan9-Archive
[3] https://github.com/oridb/git9
[4] https://github.com/driusan/dgit

--
9fans: 9fans
Permalink: 
https://9fans.topicbox.com/groups/9fans/T303744e1ec6d2108-Ma1c49d00e7042a1a8f6713d2
Delivery options: https://9fans.topicbox.com/groups/9fans/subscription