Apache policy is that jars cannot be in source releases. It’s impossible for PMC members voting on it and for users downloading it to determine the contents of the jar and thus whether they are willing to stand behind it or use it.
Could we add an ant target that fetches these jars and puts them in the right place so that users who download the source jar could easily fetch them and run the tests? Alan. On Jun 17, 2014, at 3:13 PM, Daniel Dai <[email protected]> wrote: > Oh, probably it is intentional. Alan, do you know if we shall include > test jars into the source release? > > On Tue, Jun 17, 2014 at 2:56 PM, Cheolsoo Park <[email protected]> wrote: >> I just confirmed that pig-0.12.1.tar.gz has the same issue. I've never seen >> this before since I always ran e2e with the repository. >> >> But looking at the build.xml, it seems intentional? >> >> <exclude name="test/**/*.jar"/> >> >> I guess we can just fix NPE. >> >> >> On Tue, Jun 17, 2014 at 2:48 PM, Cheolsoo Park <[email protected]> wrote: >> >>> Regarding e2e Native tests, they only fail in source tarball. >>> >>> 1) The source tarball doesn't include the following jars- >>> >>> test/e2e/pig/lib/hadoop-0.23.0-streaming.jar >>> test/e2e/pig/lib/hadoop-examples.jar >>> test/e2e/pig/lib/hadoop-streaming.jar >>> >>> 2) Due to #1, the following line throws NPE- >>> >>> >>> https://github.com/apache/pig/blob/branch-0.13/src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/JobControlCompiler.java#L1707 >>> >>> It seems trivial to fix #2, but not sure whether #1 is a regression or not. >>> >>> >>> On Tue, Jun 17, 2014 at 11:52 AM, Daniel Dai <[email protected]> >>> wrote: >>> >>>> I will take a look of TestHBaseStorage today. >>>> >>>> Also run e2e tests on Hadoop 2.0.3. All clear for me. >>>> >>>> On Tue, Jun 17, 2014 at 11:31 AM, Rohini Palaniswamy >>>> <[email protected]> wrote: >>>>> I will look at the e2e test failures I reported. Can someone pick up >>>>> TestHBaseStorage? >>>>> >>>>> >>>>> On Mon, Jun 16, 2014 at 2:59 PM, Aniket Mokashi <[email protected]> >>>> wrote: >>>>> >>>>>> Any updates on this? Please let me know if you need me to look into >>>> any of >>>>>> these. >>>>>> >>>>>> >>>>>> On Fri, Jun 13, 2014 at 9:08 AM, Cheolsoo Park <[email protected]> >>>>>> wrote: >>>>>> >>>>>>> In my e2e on Hadoop 1 run, I found Native_1,2,3 fail. I'll file a >>>> jira to >>>>>>> fix them. >>>>>>> >>>>>>> Caused by: java.lang.NullPointerException >>>>>>> at >>>>>>> >>>>>>> >>>>>> >>>> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.shipToHDFS(JobControlCompiler.java:1707) >>>>>>> >>>>>>> >>>>>>> In my unit test run with jdk7, I found 4 failures. However, I only >>>> see >>>>>>> TestHBaseStorage fails in my jenkins build, so the rest could be due >>>> to >>>>>> my >>>>>>> environment. Nevertheless, I'll look into them- >>>>>>> >>>>>>> cheolsoop@cheolsoop-gateway[502]source-tarball: grep FAILED >>>> unittest.log >>>>>>> [junit] Test org.apache.pig.test.TestHBaseStorage FAILED >>>>>>> [junit] Test org.apache.pig.test.TestMRCompiler FAILED >>>>>>> [junit] Test org.apache.pig.test.TestNativeMapReduce FAILED >>>>>>> [junit] Test org.apache.pig.test.TestPigRunner FAILED >>>>>>> >>>>>>> >>>>>>> On Thu, Jun 12, 2014 at 12:55 PM, Rohini Palaniswamy < >>>>>>> [email protected]> wrote: >>>>>>> >>>>>>>> I ran e2e on Hadoop 0.23 >>>>>>>> BigData_4 failed with OOM - This certainly needs investigation >>>>>>>> JobManagement_3, CmdErrors_3 failed with >>>>>> java.lang.NoClassDefFoundError: >>>>>>>> org/apache/zookeeper/Shell >>>>>>>> >>>>>>>> Unit tests except for HBaseStorage was good. piggybank unit tests >>>> are >>>>>>> good. >>>>>>>> >>>>>>>> Regards, >>>>>>>> Rohini >>>>>>>> >>>>>>>> >>>>>>>> On Wed, Jun 11, 2014 at 1:05 PM, Daniel Dai <[email protected] >>>>> >>>>>>> wrote: >>>>>>>> >>>>>>>>> I will run e2e on Hadoop 2. >>>>>>>>> >>>>>>>>> On Wed, Jun 11, 2014 at 12:36 PM, Cheolsoo Park < >>>>>> [email protected]> >>>>>>>>> wrote: >>>>>>>>>> I am able to run e2e on Hadoop 1 cluster. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Wed, Jun 11, 2014 at 12:28 PM, Rohini Palaniswamy < >>>>>>>>>> [email protected]> wrote: >>>>>>>>>> >>>>>>>>>>> I have kicked off a unit test run as well. Did someone run e2e >>>>>>> tests? >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> On Wed, Jun 11, 2014 at 12:26 PM, Cheolsoo Park < >>>>>>> [email protected] >>>>>>>>> >>>>>>>>>>> wrote: >>>>>>>>>>> >>>>>>>>>>>>>> Do we publish another RC? >>>>>>>>>>>> >>>>>>>>>>>> Unfortunately, I think we need another RC. Can you wait >>>> before >>>>>>>> rolling >>>>>>>>>>> out >>>>>>>>>>>> a new RC until we collect as many problems as possible? I'll >>>>>>> report >>>>>>>> my >>>>>>>>>>> unit >>>>>>>>>>>> test results by tomorrow. >>>>>>>>>>>> >>>>>>>>>>>> On Wed, Jun 11, 2014 at 12:09 PM, Rohini Palaniswamy < >>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> I am also seeing that TestHBaseStorage has 6 failures. >>>>>>>>>>>>> >>>>>>>>>>>>> Regards, >>>>>>>>>>>>> Rohini >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> On Wed, Jun 11, 2014 at 11:37 AM, Aniket Mokashi < >>>>>>>>> [email protected]> >>>>>>>>>>>>> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>> Thanks Cheolsoo, thanks for catching this. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Do you know what's the best way of fixing this? Do we >>>>>> publish >>>>>>>>> another >>>>>>>>>>>> RC? >>>>>>>>>>>>>> I will make sure to clean working directory before I >>>> package >>>>>>>> next >>>>>>>>>>> time. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Thanks, >>>>>>>>>>>>>> Aniket >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> On Wed, Jun 11, 2014 at 8:38 AM, Cheolsoo Park < >>>>>>>>> [email protected] >>>>>>>>>>>> >>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>>> Hi Aniket, >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Thank you so much for the hard work! >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> I discovered that several temporary files are >>>> included in >>>>>>> the >>>>>>>>>>> source >>>>>>>>>>>>>>> tarball. We should remove them from the tarball. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Here is the diff between the source tarball and the >>>>>>>> repository- >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Only in repo/contrib/piggybank/java/lib: .gitignore >>>>>>>>>>>>>>> Only in source-tarball/contrib/piggybank/java: >>>>>>>>>>>>>>> testPathPartitioner-testGetKeys-1402332960101 >>>>>>>>>>>>>>> Only in repo: .eclipse.templates >>>>>>>>>>>>>>> Only in repo: .git >>>>>>>>>>>>>>> Only in repo: .gitignore >>>>>>>>>>>>>>> Only in source-tarball: script3.txt >>>>>>>>>>>>>>> Only in >>>>>>>> source-tarball/src/docs/src/documentation/content/xdocs: >>>>>>>>>>>>>>> func.xml.orig >>>>>>>>>>>>>>> Only in repo/test/e2e/pig/lib: >>>> hadoop-0.23.0-streaming.jar >>>>>>>>>>>>>>> Only in repo/test/e2e/pig/lib: hadoop-examples.jar >>>>>>>>>>>>>>> Only in repo/test/e2e/pig/lib: hadoop-streaming.jar >>>>>>>>>>>>>>> Only in >>>>>>> source-tarball/test/org/apache/pig/builtin/avro/data: >>>>>>>>> avro >>>>>>>>>>>>>>> Only in >>>>>>>>> source-tarball/test/org/apache/pig/builtin/avro/data/json: >>>>>>>>>>>>>>> evenFileNameTestDirectoryCounts.json >>>>>>>>>>>>>>> Only in >>>>>>>>> source-tarball/test/org/apache/pig/builtin/avro/data/json: >>>>>>>>>>>>>>> simpleRecordsTrevni.json >>>>>>>>>>>>>>> Only in >>>>>>>>> source-tarball/test/org/apache/pig/builtin/avro/data/json: >>>>>>>>>>>>>>> testdirectory >>>>>>>>>>>>>>> Only in >>>>>>>>> source-tarball/test/org/apache/pig/builtin/avro/data/json: >>>>>>>>>>>>>>> testDirectoryCounts.json >>>>>>>>>>>>>>> Only in >>>>>>> source-tarball/test/org/apache/pig/builtin/avro/data: >>>>>>>>>>> trevni >>>>>>>>>>>>>>> Only in repo/test/org/apache/pig/test/data: >>>>>>> pigtestloader.jar >>>>>>>>>>>>>>> Only in repo/test/org/apache/pig/test/data: >>>>>>> TestWordCount.jar >>>>>>>>>>>>>>> Only in repo/test/perf/pigmix/lib: sdsuLibJKD12.jar >>>>>>>>>>>>>>> Only in source-tarball: >>>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>> >>>>>>>> >>>>>> >>>> TestDataBagAccess-testBagStoreLoad-org.apache.pig.builtin.BinStorage.txt >>>>>>>>>>>>>>> Only in source-tarball: >>>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>> >>>>>>>> >>>>>> >>>> TestDataBagAccess-testBagStoreLoad-org.apache.pig.builtin.PigStorage.txt >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Thanks, >>>>>>>>>>>>>>> Cheolsoo >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> On Mon, Jun 9, 2014 at 11:59 AM, Aniket Mokashi < >>>>>>>>>>> [email protected] >>>>>>>>>>>>> >>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Hi, >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> I have created a candidate build for Pig 0.13.0. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Keys used to sign the release are available at >>>>>>>>>>>>>>>> >>>> http://svn.apache.org/viewvc/pig/trunk/KEYS?view=markup >>>>>> . >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Please download, test, and try it out: >>>>>>>>>>>>>>>> http://people.apache.org/~aniket486/pig-0.13.0-rc0 >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Release notes and the rat report are available at >>>> the >>>>>> same >>>>>>>>>>>> location. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Should we release this? Vote closes on Tuesday, June >>>>>> 17th >>>>>>>>> 2014. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Thanks, >>>>>>>>>>>>>>>> Aniket >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> -- >>>>>>>>>>>>>> "...:::Aniket:::... Quetzalco@tl" >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> CONFIDENTIALITY NOTICE >>>>>>>>> NOTICE: This message is intended for the use of the individual or >>>>>>> entity >>>>>>>> to >>>>>>>>> which it is addressed and may contain information that is >>>>>> confidential, >>>>>>>>> privileged and exempt from disclosure under applicable law. If >>>> the >>>>>>> reader >>>>>>>>> of this message is not the intended recipient, you are hereby >>>>>> notified >>>>>>>> that >>>>>>>>> any printing, copying, dissemination, distribution, disclosure or >>>>>>>>> forwarding of this communication is strictly prohibited. If you >>>> have >>>>>>>>> received this communication in error, please contact the sender >>>>>>>> immediately >>>>>>>>> and delete it from your system. Thank You. >>>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> "...:::Aniket:::... Quetzalco@tl" >>>>>> >>>> >>>> -- >>>> CONFIDENTIALITY NOTICE >>>> NOTICE: This message is intended for the use of the individual or entity >>>> to >>>> which it is addressed and may contain information that is confidential, >>>> privileged and exempt from disclosure under applicable law. If the reader >>>> of this message is not the intended recipient, you are hereby notified >>>> that >>>> any printing, copying, dissemination, distribution, disclosure or >>>> forwarding of this communication is strictly prohibited. If you have >>>> received this communication in error, please contact the sender >>>> immediately >>>> and delete it from your system. Thank You. >>>> >>> >>> -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
