[jira] [Comment Edited] (PIG-5305) Enable yarn-client mode execution of tests in Spark (1) mode

[liyunzhang_intel (JIRA)]

[ 
https://issues.apache.org/jira/browse/PIG-5305?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16167284#comment-16167284
 ] 

liyunzhang_intel edited comment on PIG-5305 at 9/15/17 3:49 AM:


[~szita]: several suggestions
1. can we only add {{pigtest-jar}} to {{test-spark}} target in build.xml? I 
guess for {{test-tez}}, there is no need for {{pigtest-jar}}
meanwhile there is no need to add {{jar-simple}} in the dependency of 
{{test-tez}} as the dependency of {{compile-test}} includes {{jar-simple}}. If 
my understanding is not right, tell me.
2.please add comment {{added feature to re-initialize SparkContext when 
switching between cluster and local mode PigServers}} on related code. 

Besides, is there any unit test failures if convert  {{SPARK_MASTER}} from 
"local" to "yarn-client"?


was (Author: kellyzly):
[~szita]: several suggestions
1. can we only add {{pigtest-jar}} to {{test-spark}} target in build.xml? I 
guess for {{test-tez}}, there is no need for {{pigtest-jar}}
meanwhile there is no need to add {{jar-simple}} in the dependency of 
{{test-tez}} as the dependency of {{compile-test}} includes {{jar-simple}}. If 
my understanding is not right, tell me.
2.please add comment {{added feature to re-initialize SparkContext when 
switching between cluster and local mode PigServers}} on related code. 

Besides, is there any unit test failures if convert  {{SPARK_MASTER}} from 
"local" to "yarn-client"?

> Enable yarn-client mode execution of tests in Spark (1) mode
> 
>
> Key: PIG-5305
> URL: https://issues.apache.org/jira/browse/PIG-5305
> Project: Pig
>  Issue Type: Sub-task
>  Components: spark
>Reporter: Adam Szita
>Assignee: Adam Szita
> Attachments: PIG-5305.0.patch
>
>
> See parent jira (PIG-5305) for problem description



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (PIG-5302) Remove HttpClient dependency

[Nandor Kollar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/PIG-5302?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nandor Kollar updated PIG-5302:
---
Attachment: PIG-5302_2.patch

> Remove HttpClient dependency
> 
>
> Key: PIG-5302
> URL: https://issues.apache.org/jira/browse/PIG-5302
> Project: Pig
>  Issue Type: Bug
>Reporter: Nandor Kollar
>Assignee: Nandor Kollar
> Attachments: PIG-5302_1.patch, PIG-5302_2.patch
>
>
> Pig depends on Apache Commons HttpClient 3.1 which is an old version with 
> security problems 
> ([CVE-2015-5262|https://cve.mitre.org/cgi-bin/cvename.cgi?name=%20CVE-2015-5262])
> Also, Pig depends on Apache HttpComponents (it also needs update to newer 
> version due to similar reason), which is the successor of HttpClient, thus we 
> should remove HttpClient dependency, and update HttpComponents to 4.4+



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (PIG-5302) Remove HttpClient dependency

[Nandor Kollar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/PIG-5302?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nandor Kollar updated PIG-5302:
---
Attachment: PIG-5302_2.patch

> Remove HttpClient dependency
> 
>
> Key: PIG-5302
> URL: https://issues.apache.org/jira/browse/PIG-5302
> Project: Pig
>  Issue Type: Bug
>Reporter: Nandor Kollar
>Assignee: Nandor Kollar
> Attachments: PIG-5302_1.patch, PIG-5302_2.patch
>
>
> Pig depends on Apache Commons HttpClient 3.1 which is an old version with 
> security problems 
> ([CVE-2015-5262|https://cve.mitre.org/cgi-bin/cvename.cgi?name=%20CVE-2015-5262])
> Also, Pig depends on Apache HttpComponents (it also needs update to newer 
> version due to similar reason), which is the successor of HttpClient, thus we 
> should remove HttpClient dependency, and update HttpComponents to 4.4+



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (PIG-5305) Enable yarn-client mode execution of tests in Spark (1) mode

[Adam Szita (JIRA)]

[ 
https://issues.apache.org/jira/browse/PIG-5305?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16166543#comment-16166543
 ] 

Adam Szita commented on PIG-5305:
-

Attached [^PIG-5305.0.patch] to enable running tests in yarn-client mode for 
Spark execution.

Main changes:
* build.xml: added target to build a jar with all test classes. This is 
required so that we can pass this test jar onto SparkContext which then 
distributes it among Spark executors + set SPARK_MASTER env var to "yarn-client"
* SparkLauncher: added feature to re-initialize SparkContext when switching 
between cluster and local mode PigServers + only setting 
ChildFirstURLClassLoader during cluster mode

[~kellyzly] can you please take a look?


> Enable yarn-client mode execution of tests in Spark (1) mode
> 
>
> Key: PIG-5305
> URL: https://issues.apache.org/jira/browse/PIG-5305
> Project: Pig
>  Issue Type: Sub-task
>  Components: spark
>Reporter: Adam Szita
>Assignee: Adam Szita
> Attachments: PIG-5305.0.patch
>
>
> See parent jira (PIG-5305) for problem description



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (PIG-5305) Enable yarn-client mode execution of tests in Spark (1) mode

[Adam Szita (JIRA)]

 [ 
https://issues.apache.org/jira/browse/PIG-5305?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Adam Szita updated PIG-5305:

Description: See parent jira (PIG-5305) for problem description

> Enable yarn-client mode execution of tests in Spark (1) mode
> 
>
> Key: PIG-5305
> URL: https://issues.apache.org/jira/browse/PIG-5305
> Project: Pig
>  Issue Type: Sub-task
>  Components: spark
>Reporter: Adam Szita
>Assignee: Adam Szita
> Attachments: PIG-5305.0.patch
>
>
> See parent jira (PIG-5305) for problem description



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (PIG-5305) Enable yarn-client mode execution of tests in Spark (1) mode

[Adam Szita (JIRA)]

 [ 
https://issues.apache.org/jira/browse/PIG-5305?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Adam Szita updated PIG-5305:

Attachment: PIG-5305.0.patch

> Enable yarn-client mode execution of tests in Spark (1) mode
> 
>
> Key: PIG-5305
> URL: https://issues.apache.org/jira/browse/PIG-5305
> Project: Pig
>  Issue Type: Sub-task
>  Components: spark
>Reporter: Adam Szita
>Assignee: Adam Szita
> Attachments: PIG-5305.0.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (PIG-5297) Yarn-client mode doesn't work with Spark 2

[Adam Szita (JIRA)]

 [ 
https://issues.apache.org/jira/browse/PIG-5297?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Adam Szita updated PIG-5297:

Attachment: PIG-5297.0.patch

> Yarn-client mode doesn't work with Spark 2
> --
>
> Key: PIG-5297
> URL: https://issues.apache.org/jira/browse/PIG-5297
> Project: Pig
>  Issue Type: Sub-task
>  Components: spark
>Reporter: Adam Szita
>Assignee: Adam Szita
>
> When running tests in yarn-client mode that were built with Spark 2 I'm 
> getting the following exception:
> {code}
> Caused by: java.lang.IllegalStateException: Library directory 
> './pig/assembly/target/scala-2.11/jars' does not exist; make sure Spark 
> is built.
>   at 
> org.apache.spark.launcher.CommandBuilderUtils.checkState(CommandBuilderUtils.java:248)
>   at 
> org.apache.spark.launcher.CommandBuilderUtils.findJarsDir(CommandBuilderUtils.java:368)
>   at 
> org.apache.spark.launcher.YarnCommandBuilderUtils$.findJarsDir(YarnCommandBuilderUtils.scala:38)
>   at 
> org.apache.spark.deploy.yarn.Client.prepareLocalResources(Client.scala:558)
>   at 
> org.apache.spark.deploy.yarn.Client.createContainerLaunchContext(Client.scala:882)
> {code}
> After overcoming this with symlinks and setting SPARK_HOME I hit another 
> issue:
> {code}
> Caused by: java.lang.NoSuchMethodError: 
> io.netty.channel.DefaultFileRegion.(Ljava/io/File;JJ)V
>   at 
> org.apache.spark.network.buffer.FileSegmentManagedBuffer.convertToNetty(FileSegmentManagedBuffer.java:133)
>   at 
> org.apache.spark.network.protocol.MessageEncoder.encode(MessageEncoder.java:58)
>   at 
> org.apache.spark.network.protocol.MessageEncoder.encode(MessageEncoder.java:33)
>   at 
> io.netty.handler.codec.MessageToMessageEncoder.write(MessageToMessageEncoder.java:89)
> {code}
> I believe this will be an incompatibility between netty-all versions required 
> by hadoop and spark..



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (PIG-5297) Yarn-client mode doesn't work with Spark 2

[Adam Szita (JIRA)]

 [ 
https://issues.apache.org/jira/browse/PIG-5297?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Adam Szita updated PIG-5297:

Attachment: (was: PIG-5297.0.patch)

> Yarn-client mode doesn't work with Spark 2
> --
>
> Key: PIG-5297
> URL: https://issues.apache.org/jira/browse/PIG-5297
> Project: Pig
>  Issue Type: Sub-task
>  Components: spark
>Reporter: Adam Szita
>Assignee: Adam Szita
>
> When running tests in yarn-client mode that were built with Spark 2 I'm 
> getting the following exception:
> {code}
> Caused by: java.lang.IllegalStateException: Library directory 
> './pig/assembly/target/scala-2.11/jars' does not exist; make sure Spark 
> is built.
>   at 
> org.apache.spark.launcher.CommandBuilderUtils.checkState(CommandBuilderUtils.java:248)
>   at 
> org.apache.spark.launcher.CommandBuilderUtils.findJarsDir(CommandBuilderUtils.java:368)
>   at 
> org.apache.spark.launcher.YarnCommandBuilderUtils$.findJarsDir(YarnCommandBuilderUtils.scala:38)
>   at 
> org.apache.spark.deploy.yarn.Client.prepareLocalResources(Client.scala:558)
>   at 
> org.apache.spark.deploy.yarn.Client.createContainerLaunchContext(Client.scala:882)
> {code}
> After overcoming this with symlinks and setting SPARK_HOME I hit another 
> issue:
> {code}
> Caused by: java.lang.NoSuchMethodError: 
> io.netty.channel.DefaultFileRegion.(Ljava/io/File;JJ)V
>   at 
> org.apache.spark.network.buffer.FileSegmentManagedBuffer.convertToNetty(FileSegmentManagedBuffer.java:133)
>   at 
> org.apache.spark.network.protocol.MessageEncoder.encode(MessageEncoder.java:58)
>   at 
> org.apache.spark.network.protocol.MessageEncoder.encode(MessageEncoder.java:33)
>   at 
> io.netty.handler.codec.MessageToMessageEncoder.write(MessageToMessageEncoder.java:89)
> {code}
> I believe this will be an incompatibility between netty-all versions required 
> by hadoop and spark..



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (PIG-5305) Enable yarn-client mode execution of tests in Spark (1) mode

[Adam Szita (JIRA)]

Adam Szita created PIG-5305:
---

 Summary: Enable yarn-client mode execution of tests in Spark (1) 
mode
 Key: PIG-5305
 URL: https://issues.apache.org/jira/browse/PIG-5305
 Project: Pig
  Issue Type: Sub-task
  Components: spark
Reporter: Adam Szita
Assignee: Adam Szita






--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (PIG-5304) Support Pig build with Hadoop 2.8.1

[Anton Chevychalov (JIRA)]

[ 
https://issues.apache.org/jira/browse/PIG-5304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16166287#comment-16166287
 ] 

Anton Chevychalov commented on PIG-5304:


[~nkollar], Ok I got a problem. If there is no plan to switch to 2.8.x I think 
it's better to make some condition. 

> Support Pig build with Hadoop 2.8.1
> ---
>
> Key: PIG-5304
> URL: https://issues.apache.org/jira/browse/PIG-5304
> Project: Pig
>  Issue Type: Improvement
>  Components: build
>Reporter: Anton Chevychalov
> Attachments: 0001-PIG-5304-Fix-build-issue-with-Hadoop-2.8.patch
>
>
> There is no way to build Pig with Hadoop 2.8 due to 
> [javac] 
> /code/build/pig/rpm/BUILD/pig-0.15.0-src/src/org/apache/pig/backend/hadoop/datastorage/HDataStorage.java:33:
>  error: cannot find symbol
> [javac] import org.apache.hadoop.hdfs.DistributedFileSystem;
> [javac]  ^
> [javac]   symbol:   class DistributedFileSystem
> [javac]   location: package org.apache.hadoop.hdfs
> That trouble was caused by HDFS-8740
> So we need to add hadoop-hdfs-client as dependency. 
> I'll upload patch shortly. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (PIG-5304) Support Pig build with Hadoop 2.8.1

[Nandor Kollar (JIRA)]

[ 
https://issues.apache.org/jira/browse/PIG-5304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16166178#comment-16166178
 ] 

Nandor Kollar commented on PIG-5304:


[~achevychalov] with your patch ant clean jar doesn't compile now, it can't 
find org.apache.hadoop#hadoop-hdfs-client;2.7.3. hadoop-hdfs-client is a new 
artifact which didn't exist before 2.8.x. Upgrading this dependency would mean 
Pig requires 2.8+ Hadoop now. [~rohini] should we upgrade to Hadoop version 
2.8.x? That would require doc change as well by the way.

> Support Pig build with Hadoop 2.8.1
> ---
>
> Key: PIG-5304
> URL: https://issues.apache.org/jira/browse/PIG-5304
> Project: Pig
>  Issue Type: Improvement
>  Components: build
>Reporter: Anton Chevychalov
> Attachments: 0001-PIG-5304-Fix-build-issue-with-Hadoop-2.8.patch
>
>
> There is no way to build Pig with Hadoop 2.8 due to 
> [javac] 
> /code/build/pig/rpm/BUILD/pig-0.15.0-src/src/org/apache/pig/backend/hadoop/datastorage/HDataStorage.java:33:
>  error: cannot find symbol
> [javac] import org.apache.hadoop.hdfs.DistributedFileSystem;
> [javac]  ^
> [javac]   symbol:   class DistributedFileSystem
> [javac]   location: package org.apache.hadoop.hdfs
> That trouble was caused by HDFS-8740
> So we need to add hadoop-hdfs-client as dependency. 
> I'll upload patch shortly. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (PIG-5304) Support Pig build with Hadoop 2.8.1

[Anton Chevychalov (JIRA)]

 [ 
https://issues.apache.org/jira/browse/PIG-5304?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Anton Chevychalov updated PIG-5304:
---
Attachment: 0001-PIG-5304-Fix-build-issue-with-Hadoop-2.8.patch

> Support Pig build with Hadoop 2.8.1
> ---
>
> Key: PIG-5304
> URL: https://issues.apache.org/jira/browse/PIG-5304
> Project: Pig
>  Issue Type: Improvement
>  Components: build
>Reporter: Anton Chevychalov
> Attachments: 0001-PIG-5304-Fix-build-issue-with-Hadoop-2.8.patch
>
>
> There is no way to build Pig with Hadoop 2.8 due to 
> [javac] 
> /code/build/pig/rpm/BUILD/pig-0.15.0-src/src/org/apache/pig/backend/hadoop/datastorage/HDataStorage.java:33:
>  error: cannot find symbol
> [javac] import org.apache.hadoop.hdfs.DistributedFileSystem;
> [javac]  ^
> [javac]   symbol:   class DistributedFileSystem
> [javac]   location: package org.apache.hadoop.hdfs
> That trouble was caused by HDFS-8740
> So we need to add hadoop-hdfs-client as dependency. 
> I'll upload patch shortly. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: Review Request 27820: Setup for Macros in DataFu. Basic setup, no automated testing. Need feedback.

[Eyal Allweil via Review Board]

> On Nov. 14, 2014, 2:40 a.m., Matthew Hayes wrote:
> > datafu-pig/src/main/macros/nlp/tf_idf.pig
> > Lines 72 (patched)
> > 
> >
> > Shouldn't this be SUM?

As far as I can tell, it's OK that this is COUNT, if we're counting documents 
(and as I understand it TF-IDF we're dividing by documents for the IDF part, 
not actual occurences.


- Eyal


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/27820/#review61348
---


On Nov. 10, 2014, 8:33 p.m., Russell Jurney wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/27820/
> ---
> 
> (Updated Nov. 10, 2014, 8:33 p.m.)
> 
> 
> Review request for DataFu, pig, Joseph Adler, Jakob Homan, Matthew Hayes, and 
> Sam Shah.
> 
> 
> Repository: datafu
> 
> 
> Description
> ---
> 
> DATAFU-61 - Add TF-IDF Macro to DataFu
> 
> 
> Diffs
> -
> 
>   datafu-pig/src/main/macros/nlp/tf_idf.pig PRE-CREATION 
>   datafu-pig/src/test/macros/nlp/test_tf_idf.pig PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/27820/diff/1/
> 
> 
> Testing
> ---
> 
> Works for me, but testing not automated. See 
> https://issues.apache.org/jira/browse/DATAFU-61
> 
> 
> Thanks,
> 
> Russell Jurney
> 
>

[jira] [Updated] (PIG-5302) Remove HttpClient dependency

[Nandor Kollar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/PIG-5302?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nandor Kollar updated PIG-5302:
---
Status: Patch Available  (was: In Progress)

> Remove HttpClient dependency
> 
>
> Key: PIG-5302
> URL: https://issues.apache.org/jira/browse/PIG-5302
> Project: Pig
>  Issue Type: Bug
>Reporter: Nandor Kollar
>Assignee: Nandor Kollar
> Attachments: PIG-5302_1.patch
>
>
> Pig depends on Apache Commons HttpClient 3.1 which is an old version with 
> security problems 
> ([CVE-2015-5262|https://cve.mitre.org/cgi-bin/cvename.cgi?name=%20CVE-2015-5262])
> Also, Pig depends on Apache HttpComponents (it also needs update to newer 
> version due to similar reason), which is the successor of HttpClient, thus we 
> should remove HttpClient dependency, and update HttpComponents to 4.4+



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (PIG-5302) Remove HttpClient dependency

[Nandor Kollar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/PIG-5302?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nandor Kollar updated PIG-5302:
---
Attachment: PIG-5302_1.patch

> Remove HttpClient dependency
> 
>
> Key: PIG-5302
> URL: https://issues.apache.org/jira/browse/PIG-5302
> Project: Pig
>  Issue Type: Bug
>Reporter: Nandor Kollar
>Assignee: Nandor Kollar
> Attachments: PIG-5302_1.patch
>
>
> Pig depends on Apache Commons HttpClient 3.1 which is an old version with 
> security problems 
> ([CVE-2015-5262|https://cve.mitre.org/cgi-bin/cvename.cgi?name=%20CVE-2015-5262])
> Also, Pig depends on Apache HttpComponents (it also needs update to newer 
> version due to similar reason), which is the successor of HttpClient, thus we 
> should remove HttpClient dependency, and update HttpComponents to 4.4+



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (PIG-5304) Support Pig build with Hadoop 2.8.1

[Anton Chevychalov (JIRA)]

Anton Chevychalov created PIG-5304:
--

 Summary: Support Pig build with Hadoop 2.8.1
 Key: PIG-5304
 URL: https://issues.apache.org/jira/browse/PIG-5304
 Project: Pig
  Issue Type: Improvement
  Components: build
Reporter: Anton Chevychalov


There is no way to build Pig with Hadoop 2.8 due to 

[javac] 
/code/build/pig/rpm/BUILD/pig-0.15.0-src/src/org/apache/pig/backend/hadoop/datastorage/HDataStorage.java:33:
 error: cannot find symbol
[javac] import org.apache.hadoop.hdfs.DistributedFileSystem;
[javac]  ^
[javac]   symbol:   class DistributedFileSystem
[javac]   location: package org.apache.hadoop.hdfs

That trouble was caused by HDFS-8740

So we need to add hadoop-hdfs-client as dependency. 

I'll upload patch shortly. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Work started] (PIG-5302) Remove HttpClient dependency

[Nandor Kollar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/PIG-5302?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Work on PIG-5302 started by Nandor Kollar.
--
> Remove HttpClient dependency
> 
>
> Key: PIG-5302
> URL: https://issues.apache.org/jira/browse/PIG-5302
> Project: Pig
>  Issue Type: Bug
>Reporter: Nandor Kollar
>Assignee: Nandor Kollar
>
> Pig depends on Apache Commons HttpClient 3.1 which is an old version with 
> security problems 
> ([CVE-2015-5262|https://cve.mitre.org/cgi-bin/cvename.cgi?name=%20CVE-2015-5262])
> Also, Pig depends on Apache HttpComponents (it also needs update to newer 
> version due to similar reason), which is the successor of HttpClient, thus we 
> should remove HttpClient dependency, and update HttpComponents to 4.4+



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Assigned] (PIG-5303) Remove HttpClient dependency

[Nandor Kollar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/PIG-5303?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nandor Kollar reassigned PIG-5303:
--

Assignee: (was: Nandor Kollar)

> Remove HttpClient dependency
> 
>
> Key: PIG-5303
> URL: https://issues.apache.org/jira/browse/PIG-5303
> Project: Pig
>  Issue Type: Bug
>Reporter: Nandor Kollar
>
> Pig depends on Apache Commons HttpClient 3.1 which is an old version with 
> security problems 
> ([CVE-2015-5262|https://cve.mitre.org/cgi-bin/cvename.cgi?name=%20CVE-2015-5262])
> Also, Pig depends on Apache HttpComponents (it also needs update to newer 
> version due to similar reason), which is the successor of HttpClient, thus we 
> should remove HttpClient dependency, and update HttpComponents to 4.4+



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Resolved] (PIG-5303) Remove HttpClient dependency

[Nandor Kollar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/PIG-5303?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nandor Kollar resolved PIG-5303.

Resolution: Duplicate

> Remove HttpClient dependency
> 
>
> Key: PIG-5303
> URL: https://issues.apache.org/jira/browse/PIG-5303
> Project: Pig
>  Issue Type: Bug
>Reporter: Nandor Kollar
>
> Pig depends on Apache Commons HttpClient 3.1 which is an old version with 
> security problems 
> ([CVE-2015-5262|https://cve.mitre.org/cgi-bin/cvename.cgi?name=%20CVE-2015-5262])
> Also, Pig depends on Apache HttpComponents (it also needs update to newer 
> version due to similar reason), which is the successor of HttpClient, thus we 
> should remove HttpClient dependency, and update HttpComponents to 4.4+



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (PIG-5302) Remove HttpClient dependency

[Nandor Kollar (JIRA)]

Nandor Kollar created PIG-5302:
--

 Summary: Remove HttpClient dependency
 Key: PIG-5302
 URL: https://issues.apache.org/jira/browse/PIG-5302
 Project: Pig
  Issue Type: Bug
Reporter: Nandor Kollar
Assignee: Nandor Kollar


Pig depends on Apache Commons HttpClient 3.1 which is an old version with 
security problems 
([CVE-2015-5262|https://cve.mitre.org/cgi-bin/cvename.cgi?name=%20CVE-2015-5262])

Also, Pig depends on Apache HttpComponents (it also needs update to newer 
version due to similar reason), which is the successor of HttpClient, thus we 
should remove HttpClient dependency, and update HttpComponents to 4.4+



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (PIG-5303) Remove HttpClient dependency

[Nandor Kollar (JIRA)]

Nandor Kollar created PIG-5303:
--

 Summary: Remove HttpClient dependency
 Key: PIG-5303
 URL: https://issues.apache.org/jira/browse/PIG-5303
 Project: Pig
  Issue Type: Bug
Reporter: Nandor Kollar
Assignee: Nandor Kollar


Pig depends on Apache Commons HttpClient 3.1 which is an old version with 
security problems 
([CVE-2015-5262|https://cve.mitre.org/cgi-bin/cvename.cgi?name=%20CVE-2015-5262])

Also, Pig depends on Apache HttpComponents (it also needs update to newer 
version due to similar reason), which is the successor of HttpClient, thus we 
should remove HttpClient dependency, and update HttpComponents to 4.4+



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] Subscription: PIG patch available

[jira]

Issue Subscription
Filter: PIG patch available (35 issues)

Subscriber: pigdaily

Key Summary
PIG-5298Verify if org.mortbay.jetty is removable
https://issues-test.apache.org/jira/browse/PIG-5298
PIG-5290User Cache upload contention can cause job failures
https://issues-test.apache.org/jira/browse/PIG-5290
PIG-5273_SUCCESS file should be created at the end of the job
https://issues-test.apache.org/jira/browse/PIG-5273
PIG-5267Review of org.apache.pig.impl.io.BufferedPositionedInputStream
https://issues-test.apache.org/jira/browse/PIG-5267
PIG-5256Bytecode generation for POFilter and POForeach
https://issues-test.apache.org/jira/browse/PIG-5256
PIG-5191Pig HBase 2.0.0 support
https://issues-test.apache.org/jira/browse/PIG-5191
PIG-5160SchemaTupleFrontend.java is not thread safe, cause PigServer thrown 
NPE in multithread env
https://issues-test.apache.org/jira/browse/PIG-5160
PIG-5115Builtin AvroStorage generates incorrect avro schema when the same 
pig field name appears in the alias
https://issues-test.apache.org/jira/browse/PIG-5115
PIG-5106Optimize when mapreduce.input.fileinputformat.input.dir.recursive 
set to true
https://issues-test.apache.org/jira/browse/PIG-5106
PIG-5081Can not run pig on spark source code distribution
https://issues-test.apache.org/jira/browse/PIG-5081
PIG-5080Support store alias as spark table
https://issues-test.apache.org/jira/browse/PIG-5080
PIG-5057IndexOutOfBoundsException when pig reducer processOnePackageOutput
https://issues-test.apache.org/jira/browse/PIG-5057
PIG-5029Optimize sort case when data is skewed
https://issues-test.apache.org/jira/browse/PIG-5029
PIG-4926Modify the content of start.xml for spark mode
https://issues-test.apache.org/jira/browse/PIG-4926
PIG-4913Reduce jython function initiation during compilation
https://issues-test.apache.org/jira/browse/PIG-4913
PIG-4849pig on tez will cause tez-ui to crash,because the content from 
timeline server is too long. 
https://issues-test.apache.org/jira/browse/PIG-4849
PIG-4750REPLACE_MULTI should compile Pattern once and reuse it
https://issues-test.apache.org/jira/browse/PIG-4750
PIG-4684Exception should be changed to warning when job diagnostics cannot 
be fetched
https://issues-test.apache.org/jira/browse/PIG-4684
PIG-4656Improve String serialization and comparator performance in 
BinInterSedes
https://issues-test.apache.org/jira/browse/PIG-4656
PIG-4598Allow user defined plan optimizer rules
https://issues-test.apache.org/jira/browse/PIG-4598
PIG-4551Partition filter is not pushed down in case of SPLIT
https://issues-test.apache.org/jira/browse/PIG-4551
PIG-4539New PigUnit
https://issues-test.apache.org/jira/browse/PIG-4539
PIG-4515org.apache.pig.builtin.Distinct throws ClassCastException
https://issues-test.apache.org/jira/browse/PIG-4515
PIG-4323PackageConverter hanging in Spark
https://issues-test.apache.org/jira/browse/PIG-4323
PIG-4313StackOverflowError in LIMIT operation on Spark
https://issues-test.apache.org/jira/browse/PIG-4313
PIG-4251Pig on Storm
https://issues-test.apache.org/jira/browse/PIG-4251
PIG-4002Disable combiner when map-side aggregation is used
https://issues-test.apache.org/jira/browse/PIG-4002
PIG-3952PigStorage accepts '-tagSplit' to return full split information
https://issues-test.apache.org/jira/browse/PIG-3952
PIG-3911Define unique fields with @OutputSchema
https://issues-test.apache.org/jira/browse/PIG-3911
PIG-3877Getting Geo Latitude/Longitude from Address Lines
https://issues-test.apache.org/jira/browse/PIG-3877
PIG-3873Geo distance calculation using Haversine
https://issues-test.apache.org/jira/browse/PIG-3873
PIG-3864ToDate(userstring, format, timezone) computes DateTime with strange 
handling of Daylight Saving Time with location based timezones
https://issues-test.apache.org/jira/browse/PIG-3864
PIG-3668COR built-in function when atleast one of the coefficient values is 
NaN
https://issues-test.apache.org/jira/browse/PIG-3668
PIG-3587add functionality for rolling over dates
https://issues-test.apache.org/jira/browse/PIG-3587
PIG-1804Alow Jython function to implement Algebraic and/or Accumulator 
interfaces
https://issues-test.apache.org/jira/browse/PIG-1804

You may edit this subscription at:
https://issues-test.apache.org/jira/secure/EditSubscription!default.jspa?subId=16328=12322384

[jira] Subscription: PIG patch available

[jira]

Issue Subscription
Filter: PIG patch available (34 issues)

Subscriber: pigdaily

Key Summary
PIG-5298Verify if org.mortbay.jetty is removable
https://issues.apache.org/jira/browse/PIG-5298
PIG-5273_SUCCESS file should be created at the end of the job
https://issues.apache.org/jira/browse/PIG-5273
PIG-5267Review of org.apache.pig.impl.io.BufferedPositionedInputStream
https://issues.apache.org/jira/browse/PIG-5267
PIG-5256Bytecode generation for POFilter and POForeach
https://issues.apache.org/jira/browse/PIG-5256
PIG-5191Pig HBase 2.0.0 support
https://issues.apache.org/jira/browse/PIG-5191
PIG-5160SchemaTupleFrontend.java is not thread safe, cause PigServer thrown 
NPE in multithread env
https://issues.apache.org/jira/browse/PIG-5160
PIG-5115Builtin AvroStorage generates incorrect avro schema when the same 
pig field name appears in the alias
https://issues.apache.org/jira/browse/PIG-5115
PIG-5106Optimize when mapreduce.input.fileinputformat.input.dir.recursive 
set to true
https://issues.apache.org/jira/browse/PIG-5106
PIG-5081Can not run pig on spark source code distribution
https://issues.apache.org/jira/browse/PIG-5081
PIG-5080Support store alias as spark table
https://issues.apache.org/jira/browse/PIG-5080
PIG-5057IndexOutOfBoundsException when pig reducer processOnePackageOutput
https://issues.apache.org/jira/browse/PIG-5057
PIG-5029Optimize sort case when data is skewed
https://issues.apache.org/jira/browse/PIG-5029
PIG-4926Modify the content of start.xml for spark mode
https://issues.apache.org/jira/browse/PIG-4926
PIG-4913Reduce jython function initiation during compilation
https://issues.apache.org/jira/browse/PIG-4913
PIG-4849pig on tez will cause tez-ui to crash,because the content from 
timeline server is too long. 
https://issues.apache.org/jira/browse/PIG-4849
PIG-4750REPLACE_MULTI should compile Pattern once and reuse it
https://issues.apache.org/jira/browse/PIG-4750
PIG-4684Exception should be changed to warning when job diagnostics cannot 
be fetched
https://issues.apache.org/jira/browse/PIG-4684
PIG-4656Improve String serialization and comparator performance in 
BinInterSedes
https://issues.apache.org/jira/browse/PIG-4656
PIG-4598Allow user defined plan optimizer rules
https://issues.apache.org/jira/browse/PIG-4598
PIG-4551Partition filter is not pushed down in case of SPLIT
https://issues.apache.org/jira/browse/PIG-4551
PIG-4539New PigUnit
https://issues.apache.org/jira/browse/PIG-4539
PIG-4515org.apache.pig.builtin.Distinct throws ClassCastException
https://issues.apache.org/jira/browse/PIG-4515
PIG-4323PackageConverter hanging in Spark
https://issues.apache.org/jira/browse/PIG-4323
PIG-4313StackOverflowError in LIMIT operation on Spark
https://issues.apache.org/jira/browse/PIG-4313
PIG-4251Pig on Storm
https://issues.apache.org/jira/browse/PIG-4251
PIG-4002Disable combiner when map-side aggregation is used
https://issues.apache.org/jira/browse/PIG-4002
PIG-3952PigStorage accepts '-tagSplit' to return full split information
https://issues.apache.org/jira/browse/PIG-3952
PIG-3911Define unique fields with @OutputSchema
https://issues.apache.org/jira/browse/PIG-3911
PIG-3877Getting Geo Latitude/Longitude from Address Lines
https://issues.apache.org/jira/browse/PIG-3877
PIG-3873Geo distance calculation using Haversine
https://issues.apache.org/jira/browse/PIG-3873
PIG-3864ToDate(userstring, format, timezone) computes DateTime with strange 
handling of Daylight Saving Time with location based timezones
https://issues.apache.org/jira/browse/PIG-3864
PIG-3668COR built-in function when atleast one of the coefficient values is 
NaN
https://issues.apache.org/jira/browse/PIG-3668
PIG-3587add functionality for rolling over dates
https://issues.apache.org/jira/browse/PIG-3587
PIG-1804Alow Jython function to implement Algebraic and/or Accumulator 
interfaces
https://issues.apache.org/jira/browse/PIG-1804

You may edit this subscription at:
https://issues.apache.org/jira/secure/FilterSubscription!default.jspa?subId=16328=12322384