I have been waiting for GSoC to begin in order to present a more concrete
So, I have discussed my idea with João S. Bueno and he agreed to being my
mentor for GSoC. We live nearby and have the same Alma Mater (UNICAMP), so
it was easy to get in touch.
Basically, I have proposed to first merge the soc-2009-siox-drb branch with
the main branch and then implement my algorithm as another backend for the
foreground selection tool. For the first part, it would be nice if I could
get some advice from Gerald if possible.
In my proposal I have stated everything that I told you guys on this thread,
so there are not many surprises. Also, João and I did some brainstorming and
came up with a few ideas which are not described on the attached file, but
that I'll try to implement if there's time.
Thiago Vallin Spina
Laboratory of Visual Informatics
Institute of Computing -- Unicamp
Merging of the detail refinement brush branch and implementation of an
IFT-based segmentation tool
= Abstract =
This proposal aims at first merging the code from the detail
refinement brush, of the foreground selection tool, with the main
development branch. Then, an algorithm of hard and soft object
segmentation based on the Image Foresting Transform will be
implemented as an alternative to the current SIOX-based method of
= Background =
I was born and raised in Brazil, and I got a Computer Science
Bachelor's degree at the University of Campinas (UNICAMP - from
Brazil, we have a great history of participations on GSoC). Currently,
I have just been admitted to the Computer Science MSc program at
UNICAMP, but I've been researching image processing algorithms (mostly
related to foreground segmentation on images and videos) for the past
2 years. Before that I worked with robotics, so I have experience on:
* Building C/C++ modules applied to indoor navigation of robots for
about 2 years as a Research Scholarship Holder -- This project was
also applied to the mix of robotics and entertainment, and involved
a lot of C/Shell Script hacking and kernel adaptation (also some
* Developing image and video segmentation algorithms [1,2,3] using
the Image Foresting Transform (created by my advisor
prof. Alexandre X. Falcão ) mostly in C/C++ (some Python) --
Also as a Research Scholarship Holder. In that project I have also
gained a lot of experience on code optimization using mostly gcc
vector extensions for SIMD.
* Dealing with libraries such as wxWidgets (Gtk) to implement the GUI
of a software used in my image processing research.
= Motivation =
Foreground segmentation is an important part of image processing and
has been a crucial topic of research for the past decades, but it is
still an open problem. The foreground selection tool implemented on
GIMP is based on the Simple Interactive Object Extraction (SIOX)
algorithm. On the last year's Google Summer of Code, that tool was
improved to do some detail refinement on the hard segmentation. That
improvement was implemented on a separate git branch named
soc-2009-siox-drb, and now there is the need to merge it with the main
development branch for the next release of GIMP. Thus, one of the
goals of this proposal is to merge those branches.
Although SIOX does a great job when segmenting the desired foreground,
specially on noisy images, it presents some limitations related to
color similarity between foreground and background. In such cases it
may take a lot of user interaction to obtain a satisfactory
The main goal of my current research is to make user interaction more
effective and simple for both hard and soft image/video
segmentation. In that spirit, the Image Foresting Transform is a great
graph-based alternative to algorithms using the graph cut technique
(such as GrabCut ), and it was proven to provide better results
. IFT is a generalization of the Dijkstra's algorithm, which works
quite fast and has served for many purposes in image processing, image
analysis, and pattern recognition. The IFT-based framework for hard
image segmentation that I have been developing is able to deal with
the shortcomings of SIOX, as I have demonstrated on this video:
http://vimeo.com/9803589. Thus, the second goal of this proposal is to
implement my framework as an alternative to SIOX.
Finally, I have talked to João S. Bueno about this proposal, because
it is not on the GIMP's ideas list, and he has agreed to being my
mentor. Also, he studied at UNICAMP and lives nearby, so communication
will be easier.
= Implementation and GUI =
First, the implementation of the detail refinement brush, developed
for the 2009 Google Summer of Code, is going to be merged with GIMP's
main development branch.
Then, given that the current code for the foreground selection tool is
prepared for multiple algorithms, the choice between SIOX and IFT will
be done using a combo box or a similar widget. When selected, the
IFT-based foreground extraction will work as described on :
1) Once the tool is loaded the image is going to be automatically
pre-segmented into a few regions and the result are going to be shown
to the user.
2) The user will then draw foreground markers (strokes) selecting the
regions which compose the object, and inserting background markers
when a region contains both foreground and background pixels (those
regions will be split as near to the object's border as possible).
3) Afterwards, the markers are used to improve the automatic
pre-segmentation (by learning the color distribution and texture of
the object), to select the regions that compose the object, and to
divide the ones that need to be divided.
4) Finally, the user may add/remove object and background markers to
finalize the hard segmentation (merging/splitting regions). These
corrections are local and do not affect other regions.
The method has three parameteres which may be selected:
* Number of regions which should be pre-segmented (a slider to select
between "less" and "more" regions)
* The percentage of the object's information that should be used (the
color and texture information learned from the object markers is
combined with information extracted from the image).
* The radius of the strokes.
As a post-processing step, the user will be able to draw a new set of
strokes which will allow detail refinement on features such as hair,
using an IFT-based operator for soft segmentation. There might be a
"color" threshold paramater (a percentage actually) for this step, but
a default value of 50% yields good results.
Furthermore, the IFT-based framework is quite flexible regarding the
user interaction style, so the current form used for SIOX of rough
foreground selection using a lasso followed by marker drawing can also
be used. In fact, this possibility might be evaluated during GSoC.
The framework is going to be implemented using GEGL because part of it
uses floating point operations, thus, it will be easier to optimize
the code using gcc vector extensions.
= Timeline =
April 26 - May 23: Learn GIMP's API and look at the code from
soc-2009-siox-drb branch, analyzing what will be necessary to
merge the branches.
May 24 - June 25: Merge branches.
June 25 - July 10: Start coding the IFT-based framework inside GIMP.
July 05 - July 12: Write and submit the mid-term evaluation.
July 12 - August 08: Finish coding the IFT-based framework, focusing
on the graphical user interface, testing, and documenting.
August 09 - 15: Do some code scrubbing and write final evaluation.
August 16: Submit final evaluation.
August 30: Submit code samples to Google.
= Availability =
In Brazil we only have one month of Winter break (northern hemisphere
Summer) on July, but since I am only taking one class this semester I
will be able to work full time on this project. Also, my research will
be put partially on hold so that I can have even more time for coding.
= References =
 - http://portal.acm.org/citation.cfm?id=1700473
 - http://www.springerlink.com/content/7415482725175nm7/
 - R. Minetto, J. P. Papa, T. V. Spina, A. X. Falcão, N. J. Leite,
and J. Stolfi. Fast and robust object tracking using image foresting
transform. In 16th International Workshop on Systems, Signals and
Image Processing, Chalkida, Greece, Jun 2009. IEEE.
 - http://www.ic.unicamp.br/~afalcao/downloads.html
 - http://portal.acm.org/citation.cfm?id=1574529
Gimp-developer mailing list