[
https://issues.apache.org/jira/browse/PIG-2599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16506911#comment-16506911
]
Niels Basjes commented on PIG-2599:
-----------------------------------
HI all,
I decided to try and see how far I would get with this and also try to
understand why this (for me) very desirable change remains stuck for so long..
My current attempt is based upon previous attempts by others and myself and can
be found here [https://github.com/nielsbasjes/MavenizePig]
What I have right now is that I am able to compile and build Pig as a normal
multi module Maven project with Spark 2 (not spark 1 yet).
At this point some tests work and many do not yet work. Or they do work when
run in isolation but not as part of a full build cycle (due to static
MiniGenericCluster instance).
There are however a few situations that are hard in Maven and easy in Ant:
* Building against multiple versions of (for example) Spark. So my current
attempt is only capable of building with Spark 2.
* Running the unit tests that are MR/Tez/Spark specific. Here the specific
subset of tests is started with a specific environment variable. Now the
problem is that this variable gets stuck in the MiniGenericCluster as the
static variable INSTANCE and as such running Tez/Spark/Mr tests sequentially
becomes really hard.
Now in order to make this actually work I propose the following steps
# Step by step change the tests to no longer rely on ant and those nasty
environment variables. These should be commits that can just be committed.
# Then as a 'big bang' rearrange the entire source tree with a script similar
to what I have right now.
# Have a few small 'cleanup' commits if needed.
I think the spark shims problem can be solved by making several version
dedicated modules that eventually get shaded into the final jar file.
This way we can have several steps that all keeps the code compiling and only a
single "massive tree change" that rearranges everything.
Now the key question is: Do you guys agree with this? Do you think this would
work?
If so then I'm going to actually try to see how far I can get and put up some
patches.
> Mavenize Pig
> ------------
>
> Key: PIG-2599
> URL: https://issues.apache.org/jira/browse/PIG-2599
> Project: Pig
> Issue Type: New Feature
> Components: build
> Reporter: Daniel Dai
> Assignee: Vimuth Fernando
> Priority: Major
> Labels: gsoc2014
> Fix For: 0.18.0
>
> Attachments: PIG-2599-wip.zip, maven-pig.1.zip, maven-wip.xml
>
>
> Switch Pig build system from ant to maven.
> This is a candidate project for Google summer of code 2014. More information
> about the program can be found at
> https://cwiki.apache.org/confluence/display/PIG/GSoc2014
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
