[
https://issues.apache.org/jira/browse/HADOOP-11680?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14349394#comment-14349394
]
Sean Busbey commented on HADOOP-11680:
--------------------------------------
The rest of Colin's work
{quote}
And here is the de-duplicator program I wrote (in Go):
{code}
package main
import (
"path/filepath"
"flag"
"fmt"
"os"
)
var basePathToFullPath map[string][]string = map[string][]string{ }
func visit(path string, f os.FileInfo, err error) error {
if err != nil {
panic(err)
}
if f.Mode().IsDir() {
return nil
}
base := filepath.Base(path)
bases := basePathToFullPath[base]
if bases == nil {
bases = make([]string, 0, 1)
}
bases = append(bases, path)
basePathToFullPath[base] = bases
fmt.Printf("%s -> %s\n", base, path)
return nil
}
func main() {
flag.Parse()
if len(os.Args) < 2 {
fmt.Printf("Usage: %s [path]\n", os.Args[0])
os.Exit(1)
}
root := os.Args[1]
err := filepath.Walk(root, visit)
if err != nil {
fmt.Printf("Error while traversing %s: %s\n", root, err.Error())
os.Exit(1)
}
for basePath, fullPaths := range basePathToFullPath {
if len(fullPaths) <= 1 {
continue
}
absPath, err := filepath.Abs(fullPaths[0])
if err != nil {
fmt.Printf("failed to find abspath of %s: %s\n",
fullPaths[0], err.Error())
os.Exit(1)
}
fmt.Printf("Handling %s\n", basePath)
for i := 1; i < len(fullPaths); i++ {
fmt.Printf("rm %s\n", fullPaths[i])
err = os.Remove(fullPaths[i])
if err != nil {
panic(err)
}
fmt.Printf("ln %s %s\n", absPath, fullPaths[i])
err = os.Symlink(absPath, fullPaths[i])
if err != nil {
panic(err)
}
}
}
}
{code}
{quote}
and things that he thinks are lacking
{quote}
My simple script would also have to be modified (we probably would want
relative symlinks rather than absolute ones, for one thing...) Also we probably
want to have Maven somehow do this (either by writing a Maven plugin, or
tweaking the assembly one...) Not something I know how to do, but maybe there
are some maven experts in the audience?
{quote}
> Deduplicate jars in convenience binary distribution
> ---------------------------------------------------
>
> Key: HADOOP-11680
> URL: https://issues.apache.org/jira/browse/HADOOP-11680
> Project: Hadoop Common
> Issue Type: Improvement
> Components: build
> Reporter: Sean Busbey
> Assignee: Sean Busbey
>
> Pulled from discussion on HADOOP-11656 Colin wrote:
> {quote}
> bq. Andrew wrote: One additional note related to this, we can spend a lot of
> time right now distributing 100s of MBs of jar dependencies when launching a
> YARN job. Maybe this is ameliorated by the new shared distributed cache, but
> I've heard this come up quite a bit as a complaint. If we could meaningfully
> slim down our client, it could lead to a nice win.
> I'm frustrated that nobody responded to my earlier suggestion that we
> de-duplicate jars. This would drastically reduce the size of our install, and
> without rearchitecting anything.
> In fact I was so frustrated that I decided to write a program to do it myself
> and measure the delta. Here it is:
> Before:
> {code}
> du -h /h
> 249M /h
> {code}
> After:
> {code}
> du -h /h
> 140M /h
> {code}
> Seems like deduplicating jars would be a much better project than splitting
> into a client jar, if we really cared about this.
> <snip>
> {quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
