[gentoo-portage-dev] Portage API questions
Hello! As part of my Gentoo Google Summer of Code activities I am making use of the Portage Python API. As some of you can tell much quicker than me - if the API can answer question X, and - where to find functionality for task Y I come here to ask for precise pointers related to gathering these sets of data: - publically known distfile mirror URLs - publically known global use flags - entries for package.keywords - entries for package.mask - entries for package.unmask - entries for package.use - installed packages with version numbers - entries for /var/lib/portage/world - available local use for ebuild file F - applied use flags for installed package P So my two questions are: - Which of these sets of data can be obtained through the Portage API, which of them not? - Which classes and functions should I look at? That's it for now. I'm thankful for any helpful bytes. Sebastian
Re: [gentoo-portage-dev] Portage API questions
Thanks guys, very helpful! More questions later. Sebastian
Re: [gentoo-portage-dev] Portage API questions
in case anybody wants to have a look what i made of that you can find my current sources over here: http://git.goodpoint.de/?p=smolt-gentoo.git;a=tree;f=client/distros/gentoo;hb=refs/heads/gentoo thanks again to andy for his code and zac for help on IRC. any feedback and code review is very welcome, better off-list though. sebastian
[gentoo-portage-dev] portage.settings.profiles
Hello! In my current code for GSoC/Gentoo/Smolt I access the list of folders that /etc/make.profile and parents are resolved to: portage.settings.profiles I do this to be able access - profile package.mask, - user package.mask, and - user package.unmask independently, which portage.settings._getMaskAtom portage.settings._getProfileMaskAtom seem not be able to at the moment, besides looking quite private :-) Rationale: Among other things I want to find out if the user unmasked a certain package in the past even if the related mask entry for it was removed in the meantime. Also, I want to be able to distinct between a a stable on stable package and an unmasked masked package. What I would like to ask for is we could decide that portage.settings.profiles is declared part of the public API and not allow to change anymore or if we could introduce some getter for now so I could surround the current access to portage.settings.profiles by a portage API version check and use the getter from a certain version on. Integrating my code for portage related code into smolt does not seem reasonable to me as it would require depending on a version of portage that will not be stable before 2010 or so. Please share your thoughts with me. Sebastian
[gentoo-portage-dev] [PATCH] Move regex creation out of PhaseCheck constructor (Revision: 15458)
From a07a07166354e328711d1574eb06f7d357f21907 Mon Sep 17 00:00:00 2001
From: Sebastian Pipping sebast...@pipping.org
Date: Fri, 26 Feb 2010 21:19:06 +0100
Subject: [PATCH] Move regex creation out of PhaseCheck constructor
---
pym/repoman/checks.py | 16 +---
1 files changed, 9 insertions(+), 7 deletions(-)
diff --git a/pym/repoman/checks.py b/pym/repoman/checks.py
index 27e02d2..02b8d03 100644
--- a/pym/repoman/checks.py
+++ b/pym/repoman/checks.py
@@ -30,21 +30,23 @@ class LineCheck(object):
def end(self):
pass
+
+_ebuild_phases = ('pkg_pretend', 'pkg_setup', 'src_unpack', 'src_prepare',
'src_configure', 'src_compile',
+ 'src_test', 'src_install', 'pkg_preinst',
'pkg_postinst', 'pkg_prerm', 'pkg_postrm', 'pkg_config')
+_phase_re = '(%s)' % '|'.join(_ebuild_phases)
+del _ebuild_phases
+
+
class PhaseCheck(LineCheck):
basic class for function detection
ignore_line = re.compile(r'(^\s*#)')
func_end_re = re.compile(r'^\}$')
+ phases_re = re.compile(_phase_re)
in_phase = ''
def __init__(self):
- self.phases = ('pkg_pretend', 'pkg_setup', 'src_unpack',
'src_prepare', 'src_configure', 'src_compile',
- 'src_test', 'src_install', 'pkg_preinst',
'pkg_postinst', 'pkg_prerm', 'pkg_postrm', 'pkg_config')
- phase_re = '('
- for phase in self.phases:
- phase_re += phase + '|'
- phase_re = phase_re[:-1] + ')'
- self.phases_re = re.compile(phase_re)
+ pass
def check(self, num, line):
m = self.phases_re.match(line)
--
1.6.6
[gentoo-portage-dev] Composite exceptions?
Hello! I was wondering how to best handle a case with functions that I would like to collect several exceptions from. Is there an existing standard way to solve this? I was thinking of using the composite pattern for this allowing to throw a tree of exceptions with the option to flatten it for display later. How far off does that sound to you? Sebastian
[gentoo-portage-dev] VCS used for development of portage
Hello! Is moving portage development over to Git planned anytime soon? Anything keeping you from the move? Anything I can do to speed it up? Sebastian
Re: [gentoo-portage-dev] Composite exceptions?
On 02/27/10 04:20, Zac Medico wrote: Do you have an example case where you want to use this? Multiple defects in metadata.xml are such a case. At some point all the exceptions will have to collected, e.g. two invalid herds are mentioned. In that case a single exception with a list of invalid herds may suffice but it gets worse when combining errors of slightly different types. Is this a common practice? Maybe other approaches are better? No idea. Sebastian
Re: [gentoo-portage-dev] Composite exceptions?
Good idea, thank you! Sebastian
Re: [gentoo-portage-dev] VCS used for development of portage
Hello!
I've been playing with Git conversions of the portage repository.
The current demo conversion from anon SVN is up here:
http://git.goodpoint.de/?p=portage.git;a=summary
NOTE: Do not use it for development, yet - it's a demo!
At the end you can find the script I made for the conversion.
Notes on the script:
- An author map portage-author-map.txt is required, get from [1]
(Note to zmedico/robbat2: fixed version, do not use the older one)
- If updated, the latest version of this script is up at [2]
- It works with anon SVN and rsync. For the final run
- use developer SVN instead
- be sure to remove --rewrite-root !
- The final repository will be about 22MB in size
- Main conversion takes about two hours on my machine
Next we need to decide on
- a final location (git.overlays.gentoo.org/proj/portage?)
- when we convert (and freeze SVN)
- who runs the final conversion (zmedico, roobat2, me?)
Sebastian
[1] http://www.hartwork.org/public/portage-author-map.txt
[2] http://www.hartwork.org/public/portage-svn-to-git.sh
=
#!/usr/bin/env bash
starttime=$(date +'%Y-%m-%d--%H-%M-%S')
output_dir=portage-git-repo--${starttime}
# Open logged subshell
(
# Print executed bash commands
set -x
# Rip/sync anon SVN using rsync
rsync -r rsync://anonvcs.gentoo.org/vcs-public-svnroot/portage/ \
portage-anon-svn-repo-dump/ || exit 1
# Init git-svn repo, note double use of --trunk
[ -d ${output_dir} ] exit 1
git svn init file://${PWD}/portage-anon-svn-repo-dump/ \
--rewrite-root=svn://anonsvn.gentoo.org/portage/ \
--trunk=main/trunk --tags=main/tags --branches=main/branches \
${output_dir}
cd ${output_dir} || exit 1
# Convert commits
git config svn.authorsfile ../portage-author-map.txt
time git svn fetch || exit 1
# Make real Git tags from remotes/tags/* (two special cases)
for branch_tag in $(git branch -r | grep 'tags/' \
| fgrep -v 'tags/portage-2.1_pre5'); do
tag=$(sed 's|^tags/||' ${branch_tag})
git tag v${tag} remotes/${branch_tag} || exit 1
done
git tag 'tr...@1888' 'remotes/tr...@1888' || exit 1
# Make local branches from remotes/* (excluding tags and trunk)
for branch in $(git branch -r | grep -v 'tags\|trunk'); do
git checkout -b ${branch} remotes/${branch} || exit 1
done
git checkout master || exit 1
# Reduce size of repository
dotgitsize() { du --human --total .git | tail -n 1; }
dotgitsize
git gc --aggressive
dotgitsize
# Wipe all traces of SVN
git config --remove-section 'svn-remote.svn'
git config --remove-section 'svn'
rm -R .git/svn
rm -R .git/logs/refs/remotes
for file in .git/info/refs .git/packed-refs ; do
sed -e '/remotes\//d' -i ${file}
done
# Hide executed bash commands
set +x
cat INFO
DONE.
NEXT STEPS:
# cd ${output_dir}
Verify that branches and tags
1. make sense
2. are unambiguous
3. are free of SVN
# git branch -a
# git show-branch --list
# git tag -l
Push full repository
# git remote add \${remote_name} \${push_url}
# git push --mirror \${remote_name}
INFO
) | tee conversion--${starttime}.log
#
Re: [gentoo-portage-dev] VCS used for development of portage
On 03/05/10 04:58, Zac Medico wrote: http://git.goodpoint.de/?p=portage.git;a=summary NOTE: Do not use it for development, yet - it's a demo! It looks very nice to me. I noticed that it preserved continuity when branches got moved around, so the history seems like it will be fully intact. Great job! Still, maybe we should not jump on this version yet: - with svn2git we could split it to several repositories easily (see [1] for status you on related experiments) - neither svn2git nor git-svn seem to support proper conversion of changes to svn:ignore Summer of code could help about the latter: http://en.gentoo-wiki.com/wiki/Google_Summer_of_Code_2010_ideas#Add_support_for_svn:ignore_to_svn2git But pushing the conversion further into the future could also be a trade-off reducing efficiency and the number of contributions. I don't feel like proposing anything on that matter at the moment. With that said: what do you and Robin think? Sebastian [1] http://bugs.gentoo.org/show_bug.cgi?id=196025#c41
Re: [gentoo-portage-dev] portage sources have moved from SVN to GIT
On 03/24/10 09:07, Zac Medico wrote: If you have push access then you can commit something to the master branch like this: git clone git+ssh://g...@git.overlays.gentoo.org/proj/portage.git cd portage # edit files git commit -a git push origin master Let me add a few more words and pointers: let me get you started. In this mail - First thing to do - On origin - Not like Subversion: Commits in Git - Committing versus pushing - Non-linear history - Commits and the staging area - You in the future - Resources (lots of recommendable ones) - Questions? First thing to do = After cloning you need to set up your commit identity: git config --global user.name 'Dr. First Middle Last' git config --global user.email 'n...@gentoo.org' Missing that up front is more work later. On origin === The origin Zac mentioned is the name of a remote - a URI Git can pull from and (sometimes even) push to. Soon you will work with more than one remote: From personal experience I recommend to rename that remote to something more meaningful, something reflecting the involved host at best, e.g. git remote rename origin overlays-gentoo-org ^old^new Not like Subversion: Commits in Git Committing versus pushing - In Git you commit locally, even without network connectivity. You do a few local commits and push them to the server in an extra step: git commit git commit .. git push overlays-gentoo-org master In the beginning this separation may feel like a burden. You'll soon appreciate to have it. Commits and the staging area When you do git commit the content of the staging area (called index sometimes) is written into a new commit object. So modifying the staging area you change what goes into the next commit. Commands like git add file3.txt git add -u git add -p git reset do changes in the index for you. The index is one of the core features and differences to other systems including Subversion. Understanding the index is essential to working with Git. Please study online material on that topic. Non-linear history -- Due to its distributed nature - history is a directed acyclic graph (DAG) in Git, not a list - revision IDs are SHA1s, not plain numbers I can recommend emerging dev-vcs/gitg for a visual history browser. Present is on top, moving down is moving into the past You in the future = Now that we're on Git you'll soon be able (and expected) to re-order merge split clean-up past commits, i.e. re-write history. The related commands are git commit --amend and git rebase -i See here if you want to know more: http://book.git-scm.com/4_interactive_rebasing.html Resources (lots of recommendable ones) === Video Talks on Git -- - (2007-05-03) Linus Torvalds Source code control the way it was meant to be! http://www.youtube.com/watch?v=4XpnKHJAok8 - (2007-10-12) Randal Schwartz Git http://www.youtube.com/watch?v=8dhZ9BXQgc4 - (2008-06-01) Scott Chacon Getting Git http://www.markrichman.com/2008/06/14/railsconf-git-talk/ http://www.vimeo.com/1099027?pg=embedsec=1099027 - (2008-07-09) Tommi Virtanen Version Control for Developers http://blip.tv/file/1114793/ - (2008-07-09) Bart Trojanowski Git the basics http://excess.org/article/2008/07/ogre-git-tutorial/ - (2008-10-27) Johannes Schindelin Contributing with Git http://www.youtube.com/watch?v=j45cs5_nY2k Online Reading -- Introductions ` - Git Magic http://www-cs-students.stanford.edu/~blynn/gitmagic/ - The Git Community Book http://book.git-scm.com/ - Git from the bottom up http://ftp.newartisans.com/pub/git.from.bottom.up.pdf - The Git Parable http://tom.preston-werner.com/2009/05/19/the-git-parable.html - Pro Git http://progit.org/book/ Task-oriented material `` - Git Ready http://gitready.com/ - Git Casts (actually short films) http://gitcasts.com/ - Git FAQ https://git.wiki.kernel.org/index.php/GitFaq Questions? == - Check #git on Freenode -- very helpful - Mail me - Call me: +49 177 / 460 46 17 Thanks for reading! Sebastian Pipping
[gentoo-portage-dev] Handling merge issues (on the case of prefix)
Hello!
As the prefix branch is not the last case where people may run into
problems with merging due to the conversion from SVN to Git I feel like
writing about it here.
In this mail
- The problem
- Merges in SVN
- Merges in Git
- Consequences
- Dealing with it
- The concept
- Realization
The problem
===
Merges in SVN
-
In SVN people select what commits are merged from where to where
manually: SVN, merge commits 3 to 10 to branch prefix please. What
has been merge is tracked in the mind of the person merging and in log
messages.
Merges in Git
-
In Git there's another place where merges are saved: in the DAG of commits:
- A commit with several children/successors is a branch point
- A commit with several parents is a merge point
When merging one branch into another Git runs the DAG up (into the past)
from both commits until it finds a shared merge or branch point: that's
the point up to where both branches were synced last time. History
after that point is then merged over, nothing before.
Consequences
So Git relies on the existence of merge commits to detect what has been
merged already. Creating all these merge commits is a tough job for a
conversion tool (like svn2git) as it would have to distinguish between
cases where just a few commist have been cherry-picked over and cases
where all previous commits have made it over.
So the portage Git history does not have merge commits at these points
but plain single-parent commits.
Dealing with it
===
The concept
---
So to not get diffs way bigger than needed when merging what we can do
is we can manually (and permanently) teach Git what we know more about
portage's history.
Let's look the case of the prefix branch.
The current head on prefix has this log message:
[head on prefix]
Merged from trunk -r15842:15843
Looking a few commits back (using gitg or git log) on branch master I
find this commit:
[commit f52e83b0982c9c18d96757ab55109d43a9873b3f on master]
install_qa_check: make sure init.d and conf.d files do not have
syntax errors in them #310805
svn path=/main/trunk/; revision=15843
So that's the commit where grobian merged trunk into prefix last time:
perfect.
Realization
---
How do we teach Git that all that stuff has been merged already?
By creating a merge commit with two parents:
1) f52e83b0982c9c18d96757ab55109d43a9873b3f
2) head on prefix
This is how to do it on the shell
# Checkout prefix locally
git checkout -b prefix overlays-gentoo-org/prefix
# Create merge commit manually (_NOT_ something to do usually)
MERGE_COMMIT=$(echo \
'Teach Git that tr...@15843 has been merged into prefix' \
| git commit-tree 'prefix^{tree}' -p prefix -p
f52e83b0982c9c18d96757ab55109d43a9873b3f)
# Inspect result
echo ${MERGE_COMMIT}
# Move prefix head forward to include that commit
git merge ${MERGE_COMMIT}
# Inspect what we have done visually
gitg
That's where we leave the land of dirty hacks and Git's so-called
plumbings. We can can continue merging the few remaining commits from
here as usual.
# Get a feeling for what would be merged
# Should list ~10 commits (instead of ~8000 without the hack before)
git cherry -v prefix overlays-gentoo-org/master
# Merge master into prefix
git merge overlays-gentoo-org/master # Fails with conflicts, fine
git status
git mergetool
git commit
git push overlays-gentoo-org prefix
I hope this mail was helpful to you.
Sebastian
Re: [gentoo-portage-dev] Handling merge issues (on the case of prefix)
On 03/26/10 06:23, Sebastian Pipping wrote: Consequences So Git relies on the existence of merge commits to detect what has been merged already. Creating all these merge commits is a tough job for a conversion tool (like svn2git) as it would have to distinguish between cases where just a few commist have been cherry-picked over and cases where all previous commits have made it over. So the portage Git history does not have merge commits at these points but plain single-parent commits. I was wondering if this could be something that git-svn can do better than svn2git: after all what does git-svn do in these 100 minutes taking longer than svn2git? So I ran a git-svn conversion again but it seems it does no better: # git cherry -v prefix master | wc -l 7988 So that looks like about the same. Sebastian
Re: [gentoo-portage-dev] Google Summer of Code: Bash AST and grammar
On 07/21/10 15:49, Nathan Eloe wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Hello portage developers My name is Nathan Eloe, and I'm working on a Google Summer of Code project in which I'm writing a library that will create an Abstract Syntax Tree of the bash grammar. In essence I am recreating the bash grammar and structuring it as an AST. I am writing to you because you have shown interest in this project. I have attached a bash script which demonstrates the supported functionality of the grammar thus far and the resulting AST. Nice, you have covered quite a few constructs alraedy. I would greatly appreciate input on the tree I'm unsure if such a big file and tree is good to handle. I propose to split this test suite to many small cases. as well as insight into missing functionality you might see. A few things I haven't seen, yet: # Adjacent changes in quotation echo abc''def # Makes abcdef # Escape sequences like .. echo abc\def # Invocation from ibnside strings echo foo $(echo foo), bar `echo bar` # Heredocs (and their variations ..) cat EOF /dev/stdout 1 2 EOF # Construct of $((..)) # Refs to variable withzout $ upfront z=3; echo $((z+3)) Best, Sebastian
Re: [gentoo-portage-dev] Google Summer of Code: Bash AST and grammar
On 07/21/10 16:56, Sebastian Pipping wrote: A few things I haven't seen, yet: # Adjacent changes in quotation echo abc''def # Makes abcdef # Escape sequences like .. echo abc\def # Invocation from ibnside strings echo foo $(echo foo), bar `echo bar` # Heredocs (and their variations ..) cat EOF /dev/stdout 1 2 EOF # Construct of $((..)) # Refs to variable withzout $ upfront z=3; echo $((z+3)) An important one I forgot: # Line continuation echo one \ two Potentially hase quite a few sub cases .. Best, Sebastian
Re: [gentoo-portage-dev] portage docbook documentation - why not asciidoc ?
On 11/27/10 13:27, Lionel Orry wrote: [Lionel] In fact you can customize the toc level with a 'toclevel' attribute on the command-line, see the Makefile. I did several tries but you can get the same level as the original doc with '-a toclevels=4' if I remember well. In short, it's not a problem. I see, thanks. But one thing I don't know, is that whether that doc asks for external contributions or is kept as an internal reference and should only be taken care of by the devs. That would mean it stays unnecessarily hard for new devs to join if I understand your point correctly. Best, Sebastian
