[jira] [Commented] (PIG-5343) Upgrade developer build environment
[ https://issues.apache.org/jira/browse/PIG-5343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16607128#comment-16607128 ] Adam Szita commented on PIG-5343: - [~nielsbasjes], Thanks for the investigation on this one. +1 on [^PIG-5343-3.patch], it is now committed to trunk. > Upgrade developer build environment > --- > > Key: PIG-5343 > URL: https://issues.apache.org/jira/browse/PIG-5343 > Project: Pig > Issue Type: Improvement > Components: build >Reporter: Niels Basjes >Assignee: Niels Basjes >Priority: Major > Fix For: 0.18.0 > > Attachments: PIG-5343-1.patch, PIG-5343-2.patch, PIG-5343-3.patch > > > The docker image that can be used to setup the build environment still uses > Java 1.7 and is based on a very old version of Ubuntu. > Both of these should be updated. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (PIG-5343) Upgrade developer build environment
[
https://issues.apache.org/jira/browse/PIG-5343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16538726#comment-16538726
]
Niels Basjes commented on PIG-5343:
---
[~szita]
In the BUILDING.md is states that these commands must be done for a patch
{{ ANT_OPTS='-Djavac.args="-Xlint -Xmaxwarns 1000"'}}
{{ ant ${ANT_OPTS} clean piggybank jar compile-test test-commit}}
{{ cd contrib/piggybank/java && ant ${ANT_OPTS} test}}{{ }}
So I reverted the changes of PIG-5282 and did this. All passes except two that
failed because dependencies were introduced that actually need Java 8.
I then also did the same with the current Pig (without the -Xmaxwarns 1000) and
this patch (i.e. do everything in Java 8) and these also all passed.
So the tests everyone runs before submitting a patch all work fine.
However when I simply do ant test then apparently a more extensive set of tests
is run.
There I found that also in the Java 7 setup there are failing tests.
The first one that fails in my experiments is the
org.apache.pig.builtin.TestOrcStoragePushdown which is also the first one I ran
into while working on the mavenization effort.
So my current asesment (without doing a complete analysis) is that
* There are failing tests under Java 7.
* The same tests pass/fail under both Java 7 and Java 8.
So this patch (the docker thing) seems correct to me.
I created PIG-5348 to handle the problem of these tests separately.
> Upgrade developer build environment
> ---
>
> Key: PIG-5343
> URL: https://issues.apache.org/jira/browse/PIG-5343
> Project: Pig
> Issue Type: Improvement
> Components: build
>Reporter: Niels Basjes
>Assignee: Niels Basjes
>Priority: Major
> Attachments: PIG-5343-1.patch, PIG-5343-2.patch, PIG-5343-3.patch
>
>
> The docker image that can be used to setup the build environment still uses
> Java 1.7 and is based on a very old version of Ubuntu.
> Both of these should be updated.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (PIG-5343) Upgrade developer build environment
[ https://issues.apache.org/jira/browse/PIG-5343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16530007#comment-16530007 ] Adam Szita commented on PIG-5343: - [~nielsbasjes], are the failing tests passing under java 7? > Upgrade developer build environment > --- > > Key: PIG-5343 > URL: https://issues.apache.org/jira/browse/PIG-5343 > Project: Pig > Issue Type: Improvement > Components: build >Reporter: Niels Basjes >Assignee: Niels Basjes >Priority: Major > Attachments: PIG-5343-1.patch, PIG-5343-2.patch, PIG-5343-3.patch > > > The docker image that can be used to setup the build environment still uses > Java 1.7 and is based on a very old version of Ubuntu. > Both of these should be updated. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (PIG-5343) Upgrade developer build environment
[
https://issues.apache.org/jira/browse/PIG-5343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16522044#comment-16522044
]
Adam Szita commented on PIG-5343:
-
In {{TestDriverPig.pm}} we might want to leave the MaxPermSize setting. E2E
tests can be used to compare results of the same Pig script using a new and an
old release of Pig, and some test clusters might have java7 installed on them.
> Upgrade developer build environment
> ---
>
> Key: PIG-5343
> URL: https://issues.apache.org/jira/browse/PIG-5343
> Project: Pig
> Issue Type: Improvement
> Components: build
>Reporter: Niels Basjes
>Assignee: Niels Basjes
>Priority: Major
> Attachments: PIG-5343-1.patch, PIG-5343-2.patch
>
>
> The docker image that can be used to setup the build environment still uses
> Java 1.7 and is based on a very old version of Ubuntu.
> Both of these should be updated.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (PIG-5343) Upgrade developer build environment
[ https://issues.apache.org/jira/browse/PIG-5343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16520364#comment-16520364 ] Niels Basjes commented on PIG-5343: --- I'm still working on this because I'm having problems with what I have created so far. I was trying to get PIG-2599 (Mavenize Pig) running and I got stuck on a large number of unit tests that simply fail with hard to trace errors. Get this [https://github.com/nielsbasjes/MavenizePig] and the run.sh script will locally create a rearranged version of the Pig source tree which compiles pig and piggybank. So I decided to see how the ant based build works and compare where it goes wrong (i.e. this issue popped up to fix the docker setup to do just that) At this point my best guess is that there is a problem in relation to the PigServer + a 'non-local' MiniCluster and the use of HDFS which cause jobs to fail under Java 1.8. Try this one for example: in TestOrcStoragePushdown the BeforeClass method createInputData the very simple pigserver job to create the test file on HDFS fails and produces an empty file (hence all subsequent tests in that file fail too). It seems to be Java 1.7 / 1.8 related but at this point I'm stuck and I could use some help/pointers where to look next. Most importantly: Do you have these effects too? > Upgrade developer build environment > --- > > Key: PIG-5343 > URL: https://issues.apache.org/jira/browse/PIG-5343 > Project: Pig > Issue Type: Improvement > Components: build >Reporter: Niels Basjes >Assignee: Niels Basjes >Priority: Major > Attachments: PIG-5343-1.patch, PIG-5343-2.patch > > > The docker image that can be used to setup the build environment still uses > Java 1.7 and is based on a very old version of Ubuntu. > Both of these should be updated. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (PIG-5343) Upgrade developer build environment
[ https://issues.apache.org/jira/browse/PIG-5343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16519662#comment-16519662 ] Koji Noguchi commented on PIG-5343: --- [~szita], [~rohini], can you help reviewing this? > Upgrade developer build environment > --- > > Key: PIG-5343 > URL: https://issues.apache.org/jira/browse/PIG-5343 > Project: Pig > Issue Type: Improvement > Components: build >Reporter: Niels Basjes >Assignee: Niels Basjes >Priority: Major > Attachments: PIG-5343-1.patch, PIG-5343-2.patch > > > The docker image that can be used to setup the build environment still uses > Java 1.7 and is based on a very old version of Ubuntu. > Both of these should be updated. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (PIG-5343) Upgrade developer build environment
[ https://issues.apache.org/jira/browse/PIG-5343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16519060#comment-16519060 ] Niels Basjes commented on PIG-5343: --- I changed: * Latest LTS version of Ubuntu for the build image: _Ubuntu_ 18.04 LTS (_Bionic_ Beaver) * Remove MaxPermSize which is ignored in Java 1.8 and only triggers warnings. * Small documentation fix. > Upgrade developer build environment > --- > > Key: PIG-5343 > URL: https://issues.apache.org/jira/browse/PIG-5343 > Project: Pig > Issue Type: Improvement > Components: build >Reporter: Niels Basjes >Assignee: Niels Basjes >Priority: Major > Attachments: PIG-5343-1.patch, PIG-5343-2.patch > > > The docker image that can be used to setup the build environment still uses > Java 1.7 and is based on a very old version of Ubuntu. > Both of these should be updated. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
