[jira] [Commented] (PIG-5282) Upgade to Java 8
[ https://issues.apache.org/jira/browse/PIG-5282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16130884#comment-16130884 ] Satish Subhashrao Saley commented on PIG-5282: -- Updated the patch. In test/e2e/pig/udfs/java/build.xml and test/perf/pigmix/build.xml, we don't specify any java version. So the version of java which we use to run ant command will gets picked up by default. > Upgade to Java 8 > > > Key: PIG-5282 > URL: https://issues.apache.org/jira/browse/PIG-5282 > Project: Pig > Issue Type: Improvement >Reporter: Nandor Kollar >Assignee: Satish Subhashrao Saley > Fix For: 0.18.0 > > Attachments: PIG-5282-1.patch, PIG-5282-2.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (PIG-5282) Upgade to Java 8
[ https://issues.apache.org/jira/browse/PIG-5282?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Satish Subhashrao Saley updated PIG-5282: - Attachment: PIG-5282-2.patch > Upgade to Java 8 > > > Key: PIG-5282 > URL: https://issues.apache.org/jira/browse/PIG-5282 > Project: Pig > Issue Type: Improvement >Reporter: Nandor Kollar >Assignee: Satish Subhashrao Saley > Fix For: 0.18.0 > > Attachments: PIG-5282-1.patch, PIG-5282-2.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (PIG-5294) Spark unit tests are always run in spark1 mode
[
https://issues.apache.org/jira/browse/PIG-5294?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16130828#comment-16130828
]
Adam Szita commented on PIG-5294:
-
Some unit test related targets should not depend on {{jar}}, but rather on
{{jar-simple}} instead because:
* There is no point of compiling against two versions of Spark when at the end
we only use one to run the tests with
* There is no way to override what Spark version we want, it will always be
spark1
Please find the fix in [^PIG-5294.0.patch]
> Spark unit tests are always run in spark1 mode
> --
>
> Key: PIG-5294
> URL: https://issues.apache.org/jira/browse/PIG-5294
> Project: Pig
> Issue Type: Bug
> Components: build, spark
>Reporter: Adam Szita
>Assignee: Adam Szita
> Attachments: PIG-5294.0.patch
>
>
> The ant target {{test}} depends on {{jar}} which builds Pig with
> sparkversion=2 then sparkversion=1, hence we will always run the tests with
> Spark 1 (1.6.1 currently to be exact)
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (PIG-5294) Spark unit tests are always run in spark1 mode
[
https://issues.apache.org/jira/browse/PIG-5294?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Adam Szita updated PIG-5294:
Status: Patch Available (was: Open)
> Spark unit tests are always run in spark1 mode
> --
>
> Key: PIG-5294
> URL: https://issues.apache.org/jira/browse/PIG-5294
> Project: Pig
> Issue Type: Bug
> Components: build, spark
>Reporter: Adam Szita
>Assignee: Adam Szita
> Attachments: PIG-5294.0.patch
>
>
> The ant target {{test}} depends on {{jar}} which builds Pig with
> sparkversion=2 then sparkversion=1, hence we will always run the tests with
> Spark 1 (1.6.1 currently to be exact)
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (PIG-5294) Spark unit tests are always run in spark1 mode
[
https://issues.apache.org/jira/browse/PIG-5294?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Adam Szita updated PIG-5294:
Attachment: PIG-5294.0.patch
> Spark unit tests are always run in spark1 mode
> --
>
> Key: PIG-5294
> URL: https://issues.apache.org/jira/browse/PIG-5294
> Project: Pig
> Issue Type: Bug
> Components: build, spark
>Reporter: Adam Szita
>Assignee: Adam Szita
> Attachments: PIG-5294.0.patch
>
>
> The ant target {{test}} depends on {{jar}} which builds Pig with
> sparkversion=2 then sparkversion=1, hence we will always run the tests with
> Spark 1 (1.6.1 currently to be exact)
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (PIG-5282) Upgade to Java 8
[ https://issues.apache.org/jira/browse/PIG-5282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16130820#comment-16130820 ] Rohini Palaniswamy commented on PIG-5282: - Please check other build.xml's as well like piggybank, e2e, etc > Upgade to Java 8 > > > Key: PIG-5282 > URL: https://issues.apache.org/jira/browse/PIG-5282 > Project: Pig > Issue Type: Improvement >Reporter: Nandor Kollar >Assignee: Satish Subhashrao Saley > Fix For: 0.18.0 > > Attachments: PIG-5282-1.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (PIG-5282) Upgade to Java 8
[ https://issues.apache.org/jira/browse/PIG-5282?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Satish Subhashrao Saley updated PIG-5282: - Attachment: PIG-5282-1.patch > Upgade to Java 8 > > > Key: PIG-5282 > URL: https://issues.apache.org/jira/browse/PIG-5282 > Project: Pig > Issue Type: Improvement >Reporter: Nandor Kollar >Assignee: Satish Subhashrao Saley > Fix For: 0.18.0 > > Attachments: PIG-5282-1.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (PIG-5282) Upgade to Java 8
[ https://issues.apache.org/jira/browse/PIG-5282?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Satish Subhashrao Saley updated PIG-5282: - Status: Patch Available (was: Open) > Upgade to Java 8 > > > Key: PIG-5282 > URL: https://issues.apache.org/jira/browse/PIG-5282 > Project: Pig > Issue Type: Improvement >Reporter: Nandor Kollar >Assignee: Satish Subhashrao Saley > Fix For: 0.18.0 > > Attachments: PIG-5282-1.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Assigned] (PIG-5282) Upgade to Java 8
[ https://issues.apache.org/jira/browse/PIG-5282?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Satish Subhashrao Saley reassigned PIG-5282: Assignee: Satish Subhashrao Saley > Upgade to Java 8 > > > Key: PIG-5282 > URL: https://issues.apache.org/jira/browse/PIG-5282 > Project: Pig > Issue Type: Improvement >Reporter: Nandor Kollar >Assignee: Satish Subhashrao Saley > Fix For: 0.18.0 > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (PIG-5273) _SUCCESS file should be created at the end of the job
[ https://issues.apache.org/jira/browse/PIG-5273?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16130777#comment-16130777 ] Rohini Palaniswamy commented on PIG-5273: - Can you put the javac.version change from 1.7 to 1.8 as a patch in PIG-5282? > _SUCCESS file should be created at the end of the job > - > > Key: PIG-5273 > URL: https://issues.apache.org/jira/browse/PIG-5273 > Project: Pig > Issue Type: Bug >Reporter: Satish Subhashrao Saley >Assignee: Satish Subhashrao Saley > Attachments: PIG-5273-1.patch > > > One of the users ran into issues because _SUCCESS file was created by > FileOutputCommitter.commitJob() and storeCleanup() called after that in > PigOutputCommitter failed to store schema due to network outage. abortJob was > then called and the StoreFunc.cleanupOnFailure method in it deleted the > output directory. Downstream jobs that started because of _SUCCESS file ran > with empty data > Possible solutions: > 1) Move storeCleanup before commit. Found that order was reversed in > https://issues.apache.org/jira/browse/PIG-2642, probably due to > FileOutputCommitter version 1 and might not be a problem with > FileOutputCommitter version 2. This would still not help when there are > multiple outputs as main problem is cleanupOnFailure in abortJob deleting > directories. > 2) We can change cleanupOnFailure not delete output directories. It still > does not help. The Oozie action retry might kick in and delete the directory > while the downstream has already started running because of the _SUCCESS > file. > 3) It cannot be done in the OutputCommitter at all as multiple output > committers are called in parallel in Tez. We can have Pig suppress _SUCCESS > creation and try creating them all at the end in TezLauncher if job has > succeeded before calling cleanupOnSuccess. Can probably add it as a > configurable setting and turn on by default in our clusters. This is probably > the possible solution > Thank you [~rohini] for finding out the issue and providing solution. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (PIG-5294) Spark unit tests are always run in spark1 mode
Adam Szita created PIG-5294:
---
Summary: Spark unit tests are always run in spark1 mode
Key: PIG-5294
URL: https://issues.apache.org/jira/browse/PIG-5294
Project: Pig
Issue Type: Bug
Components: build, spark
Reporter: Adam Szita
Assignee: Adam Szita
The ant target {{test}} depends on {{jar}} which builds Pig with sparkversion=2
then sparkversion=1, hence we will always run the tests with Spark 1 (1.6.1
currently to be exact)
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] Subscription: PIG patch available
Issue Subscription Filter: PIG patch available (35 issues) Subscriber: pigdaily Key Summary PIG-5290User Cache upload contention can cause job failures https://issues.apache.org/jira/browse/PIG-5290 PIG-5273_SUCCESS file should be created at the end of the job https://issues.apache.org/jira/browse/PIG-5273 PIG-5268Review of org.apache.pig.backend.hadoop.datastorage.HDataStorage https://issues.apache.org/jira/browse/PIG-5268 PIG-5267Review of org.apache.pig.impl.io.BufferedPositionedInputStream https://issues.apache.org/jira/browse/PIG-5267 PIG-5256Bytecode generation for POFilter and POForeach https://issues.apache.org/jira/browse/PIG-5256 PIG-5191Pig HBase 2.0.0 support https://issues.apache.org/jira/browse/PIG-5191 PIG-5160SchemaTupleFrontend.java is not thread safe, cause PigServer thrown NPE in multithread env https://issues.apache.org/jira/browse/PIG-5160 PIG-5115Builtin AvroStorage generates incorrect avro schema when the same pig field name appears in the alias https://issues.apache.org/jira/browse/PIG-5115 PIG-5106Optimize when mapreduce.input.fileinputformat.input.dir.recursive set to true https://issues.apache.org/jira/browse/PIG-5106 PIG-5081Can not run pig on spark source code distribution https://issues.apache.org/jira/browse/PIG-5081 PIG-5080Support store alias as spark table https://issues.apache.org/jira/browse/PIG-5080 PIG-5057IndexOutOfBoundsException when pig reducer processOnePackageOutput https://issues.apache.org/jira/browse/PIG-5057 PIG-5029Optimize sort case when data is skewed https://issues.apache.org/jira/browse/PIG-5029 PIG-4926Modify the content of start.xml for spark mode https://issues.apache.org/jira/browse/PIG-4926 PIG-4913Reduce jython function initiation during compilation https://issues.apache.org/jira/browse/PIG-4913 PIG-4849pig on tez will cause tez-ui to crash,because the content from timeline server is too long. https://issues.apache.org/jira/browse/PIG-4849 PIG-4750REPLACE_MULTI should compile Pattern once and reuse it https://issues.apache.org/jira/browse/PIG-4750 PIG-4684Exception should be changed to warning when job diagnostics cannot be fetched https://issues.apache.org/jira/browse/PIG-4684 PIG-4656Improve String serialization and comparator performance in BinInterSedes https://issues.apache.org/jira/browse/PIG-4656 PIG-4598Allow user defined plan optimizer rules https://issues.apache.org/jira/browse/PIG-4598 PIG-4551Partition filter is not pushed down in case of SPLIT https://issues.apache.org/jira/browse/PIG-4551 PIG-4539New PigUnit https://issues.apache.org/jira/browse/PIG-4539 PIG-4515org.apache.pig.builtin.Distinct throws ClassCastException https://issues.apache.org/jira/browse/PIG-4515 PIG-4323PackageConverter hanging in Spark https://issues.apache.org/jira/browse/PIG-4323 PIG-4313StackOverflowError in LIMIT operation on Spark https://issues.apache.org/jira/browse/PIG-4313 PIG-4251Pig on Storm https://issues.apache.org/jira/browse/PIG-4251 PIG-4002Disable combiner when map-side aggregation is used https://issues.apache.org/jira/browse/PIG-4002 PIG-3952PigStorage accepts '-tagSplit' to return full split information https://issues.apache.org/jira/browse/PIG-3952 PIG-3911Define unique fields with @OutputSchema https://issues.apache.org/jira/browse/PIG-3911 PIG-3877Getting Geo Latitude/Longitude from Address Lines https://issues.apache.org/jira/browse/PIG-3877 PIG-3873Geo distance calculation using Haversine https://issues.apache.org/jira/browse/PIG-3873 PIG-3864ToDate(userstring, format, timezone) computes DateTime with strange handling of Daylight Saving Time with location based timezones https://issues.apache.org/jira/browse/PIG-3864 PIG-3668COR built-in function when atleast one of the coefficient values is NaN https://issues.apache.org/jira/browse/PIG-3668 PIG-3587add functionality for rolling over dates https://issues.apache.org/jira/browse/PIG-3587 PIG-1804Alow Jython function to implement Algebraic and/or Accumulator interfaces https://issues.apache.org/jira/browse/PIG-1804 You may edit this subscription at: https://issues.apache.org/jira/secure/FilterSubscription!default.jspa?subId=16328=12322384
