[jira] [Updated] (HADOOP-14992) Upgrade Avro patch version

2017-10-28 Thread Bharat Viswanadham (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HADOOP-14992?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bharat Viswanadham updated HADOOP-14992:

Status: Patch Available  (was: Open)

> Upgrade Avro patch version
> --
>
> Key: HADOOP-14992
> URL: https://issues.apache.org/jira/browse/HADOOP-14992
> Project: Hadoop Common
>  Issue Type: Improvement
>  Components: build
>Reporter: Chris Douglas
>Assignee: Bharat Viswanadham
> Attachments: HADOOP-14992.00.patch
>
>
> Hadoop uses Avro 1.7.4, we should upgrade to 1.7.7



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Updated] (HADOOP-14992) Upgrade Avro patch version

2017-10-28 Thread Bharat Viswanadham (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HADOOP-14992?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bharat Viswanadham updated HADOOP-14992:

Attachment: HADOOP-14992.00.patch

> Upgrade Avro patch version
> --
>
> Key: HADOOP-14992
> URL: https://issues.apache.org/jira/browse/HADOOP-14992
> Project: Hadoop Common
>  Issue Type: Improvement
>  Components: build
>Reporter: Chris Douglas
>Assignee: Bharat Viswanadham
> Attachments: HADOOP-14992.00.patch
>
>
> Hadoop uses Avro 1.7.4, we should upgrade to 1.7.7



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Assigned] (HADOOP-14992) Upgrade Avro patch version

2017-10-28 Thread Bharat Viswanadham (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HADOOP-14992?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bharat Viswanadham reassigned HADOOP-14992:
---

Assignee: Bharat Viswanadham

> Upgrade Avro patch version
> --
>
> Key: HADOOP-14992
> URL: https://issues.apache.org/jira/browse/HADOOP-14992
> Project: Hadoop Common
>  Issue Type: Improvement
>  Components: build
>Reporter: Chris Douglas
>Assignee: Bharat Viswanadham
>
> Hadoop uses Avro 1.7.4, we should upgrade to 1.7.7



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Comment Edited] (HADOOP-14157) FsUrlStreamHandlerFactory "Illegal character in path" parsing file URL on Windows

2017-10-28 Thread Bruno Orsi Berton (JIRA) 

[ 
https://issues.apache.org/jira/browse/HADOOP-14157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16223743#comment-16223743
 ] 

Bruno Orsi Berton edited comment on HADOOP-14157 at 10/28/17 9:33 PM:
--

Guys, I made a pull request to fix this in version 3.0.0. 

The patch provided by Simon Scott alters one test and the test is failing.
I created a new test and now is passing.

Could someone review this please?
Thanks.


was (Author: bberton):
Guys, I made a pull request to fix this in version 3.0.0. 

The patch provided by Simon Scott alters one test and the test is failing.
I created a new test and now is passing.

Could someone reviews this please?
Thanks.

> FsUrlStreamHandlerFactory "Illegal character in path" parsing file URL on 
> Windows
> -
>
> Key: HADOOP-14157
> URL: https://issues.apache.org/jira/browse/HADOOP-14157
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: fs
>Affects Versions: 2.7.3, 2.6.5, 3.0.0-alpha2
> Environment: Windows
>Reporter: Simon Scott
>Priority: Minor
> Attachments: HADOOP-14157.001.patch
>
>
> After registering the FsUrlStreamHandlerFactory with the JVM, subsequent 
> calls to convert a "file" URL to a URI can fail with "Illegal character in 
> path" where the illegal character is a backslash.
> For example:
> {code}
> URL.setURLStreamHandlerFactory(new FsUrlStreamHandlerFactory());
> File file = new File("C:/Users");
> final URL url = new URL("file:///" + file.getAbsolutePath());
> {code}
> gives stack trace:
> {noformat}
> java.net.URISyntaxException: Illegal character in path at index 8: 
> file:/C:\Users
> at java.net.URI$Parser.fail(URI.java:2848)
> at java.net.URI$Parser.checkChars(URI.java:3021)
> at java.net.URI$Parser.parseHierarchical(URI.java:3105)
> at java.net.URI$Parser.parse(URI.java:3053)
> at java.net.URI.(URI.java:588)
> at java.net.URL.toURI(URL.java:946)
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Comment Edited] (HADOOP-14157) FsUrlStreamHandlerFactory "Illegal character in path" parsing file URL on Windows

2017-10-28 Thread Bruno Orsi Berton (JIRA) 

[ 
https://issues.apache.org/jira/browse/HADOOP-14157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16223743#comment-16223743
 ] 

Bruno Orsi Berton edited comment on HADOOP-14157 at 10/28/17 9:32 PM:
--

Guys, I made a pull request to fix this in version 3.0.0. 

The patch provided by Simon Scott alters one test and the test is failing.
I created a new test and now is passing.

Could someone reviews this please?
Thanks.


was (Author: bberton):
Guys, I made a pull request to fix this in version 3.0.0. The problem with the 
patch of Simon Scott is that the changed test is failing.

> FsUrlStreamHandlerFactory "Illegal character in path" parsing file URL on 
> Windows
> -
>
> Key: HADOOP-14157
> URL: https://issues.apache.org/jira/browse/HADOOP-14157
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: fs
>Affects Versions: 2.7.3, 2.6.5, 3.0.0-alpha2
> Environment: Windows
>Reporter: Simon Scott
>Priority: Minor
> Attachments: HADOOP-14157.001.patch
>
>
> After registering the FsUrlStreamHandlerFactory with the JVM, subsequent 
> calls to convert a "file" URL to a URI can fail with "Illegal character in 
> path" where the illegal character is a backslash.
> For example:
> {code}
> URL.setURLStreamHandlerFactory(new FsUrlStreamHandlerFactory());
> File file = new File("C:/Users");
> final URL url = new URL("file:///" + file.getAbsolutePath());
> {code}
> gives stack trace:
> {noformat}
> java.net.URISyntaxException: Illegal character in path at index 8: 
> file:/C:\Users
> at java.net.URI$Parser.fail(URI.java:2848)
> at java.net.URI$Parser.checkChars(URI.java:3021)
> at java.net.URI$Parser.parseHierarchical(URI.java:3105)
> at java.net.URI$Parser.parse(URI.java:3053)
> at java.net.URI.(URI.java:588)
> at java.net.URL.toURI(URL.java:946)
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-14157) FsUrlStreamHandlerFactory "Illegal character in path" parsing file URL on Windows

2017-10-28 Thread Bruno Orsi Berton (JIRA) 

[ 
https://issues.apache.org/jira/browse/HADOOP-14157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16223743#comment-16223743
 ] 

Bruno Orsi Berton commented on HADOOP-14157:


Guys, I made a pull request to fix this in version 3.0.0. The problem with the 
patch of Simon Scott is that the changed test is failing.

> FsUrlStreamHandlerFactory "Illegal character in path" parsing file URL on 
> Windows
> -
>
> Key: HADOOP-14157
> URL: https://issues.apache.org/jira/browse/HADOOP-14157
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: fs
>Affects Versions: 2.7.3, 2.6.5, 3.0.0-alpha2
> Environment: Windows
>Reporter: Simon Scott
>Priority: Minor
> Attachments: HADOOP-14157.001.patch
>
>
> After registering the FsUrlStreamHandlerFactory with the JVM, subsequent 
> calls to convert a "file" URL to a URI can fail with "Illegal character in 
> path" where the illegal character is a backslash.
> For example:
> {code}
> URL.setURLStreamHandlerFactory(new FsUrlStreamHandlerFactory());
> File file = new File("C:/Users");
> final URL url = new URL("file:///" + file.getAbsolutePath());
> {code}
> gives stack trace:
> {noformat}
> java.net.URISyntaxException: Illegal character in path at index 8: 
> file:/C:\Users
> at java.net.URI$Parser.fail(URI.java:2848)
> at java.net.URI$Parser.checkChars(URI.java:3021)
> at java.net.URI$Parser.parseHierarchical(URI.java:3105)
> at java.net.URI$Parser.parse(URI.java:3053)
> at java.net.URI.(URI.java:588)
> at java.net.URL.toURI(URL.java:946)
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-14157) FsUrlStreamHandlerFactory "Illegal character in path" parsing file URL on Windows

2017-10-28 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/HADOOP-14157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16223736#comment-16223736
 ] 

ASF GitHub Bot commented on HADOOP-14157:
-

GitHub user bberton-ciandt opened a pull request:

https://github.com/apache/hadoop/pull/284

[HADOOP-14157] fix parseUrl to replace '\' for '/'



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/bberton-ciandt/hadoop HADOOP-14157

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/hadoop/pull/284.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #284


commit 2d563d66946888e4c906370fd7a07fa33c83bd61
Author: bberton-ciandt 
Date:   2017-10-28T21:08:07Z

[HADOOP-14157] fix parseUrl to replace '\' for '/'




> FsUrlStreamHandlerFactory "Illegal character in path" parsing file URL on 
> Windows
> -
>
> Key: HADOOP-14157
> URL: https://issues.apache.org/jira/browse/HADOOP-14157
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: fs
>Affects Versions: 2.7.3, 2.6.5, 3.0.0-alpha2
> Environment: Windows
>Reporter: Simon Scott
>Priority: Minor
> Attachments: HADOOP-14157.001.patch
>
>
> After registering the FsUrlStreamHandlerFactory with the JVM, subsequent 
> calls to convert a "file" URL to a URI can fail with "Illegal character in 
> path" where the illegal character is a backslash.
> For example:
> {code}
> URL.setURLStreamHandlerFactory(new FsUrlStreamHandlerFactory());
> File file = new File("C:/Users");
> final URL url = new URL("file:///" + file.getAbsolutePath());
> {code}
> gives stack trace:
> {noformat}
> java.net.URISyntaxException: Illegal character in path at index 8: 
> file:/C:\Users
> at java.net.URI$Parser.fail(URI.java:2848)
> at java.net.URI$Parser.checkChars(URI.java:3021)
> at java.net.URI$Parser.parseHierarchical(URI.java:3105)
> at java.net.URI$Parser.parse(URI.java:3053)
> at java.net.URI.(URI.java:588)
> at java.net.URL.toURI(URL.java:946)
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Created] (HADOOP-14992) Upgrade Avro patch version

2017-10-28 Thread Chris Douglas (JIRA) 
Chris Douglas created HADOOP-14992:
--

 Summary: Upgrade Avro patch version
 Key: HADOOP-14992
 URL: https://issues.apache.org/jira/browse/HADOOP-14992
 Project: Hadoop Common
  Issue Type: Improvement
  Components: build
Reporter: Chris Douglas


Hadoop uses Avro 1.7.4, we should upgrade to 1.7.7



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-13684) Snappy may complain Hadoop is built without snappy if libhadoop is not found.

2017-10-28 Thread Steve Loughran (JIRA) 

[ 
https://issues.apache.org/jira/browse/HADOOP-13684?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16223406#comment-16223406
 ] 

Steve Loughran commented on HADOOP-13684:
-

Before doing that, is it possible to have the setup: no native libhadoop, but 
snappy native code around and accessible. If so, it should be bypassing the 
native lib check; it just needs some other check

> Snappy may complain Hadoop is built without snappy if libhadoop is not found.
> -
>
> Key: HADOOP-13684
> URL: https://issues.apache.org/jira/browse/HADOOP-13684
> Project: Hadoop Common
>  Issue Type: Improvement
>Reporter: Wei-Chiu Chuang
>Assignee: Wei-Chiu Chuang
>Priority: Minor
>  Labels: supportability
> Fix For: 2.8.0, 3.0.0-alpha2
>
> Attachments: HADOOP-13684.001.patch, HADOOP-13684.002.patch
>
>
> If for some reason libhadoop can not be found/loaded, Snappy complains Hadoop 
> is not built with Snappy but it actually is.
> {code:title=SnappyCodec.java}
> public static void checkNativeCodeLoaded() {
>   if (!NativeCodeLoader.isNativeCodeLoaded() ||
>   !NativeCodeLoader.buildSupportsSnappy()) {
> throw new RuntimeException("native snappy library not available: " +
> "this version of libhadoop was built without " +
> "snappy support.");
>   }
> {code}
> This case may happen with MAPREDUCE-6577.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-14973) [s3a] Log StorageStatistics

2017-10-28 Thread Steve Loughran (JIRA) 

[ 
https://issues.apache.org/jira/browse/HADOOP-14973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16223404#comment-16223404
 ] 

Steve Loughran commented on HADOOP-14973:
-

yes, that's the one.: linked direct. It's using  specific file as a sink, which 
also means isolated from all other noise. Try that today, if you want something 
in the rolling logs, let's add a new sync. I think I'd like that for testing 
anyway, as it would get all stats into test results automatically

> [s3a] Log StorageStatistics
> ---
>
> Key: HADOOP-14973
> URL: https://issues.apache.org/jira/browse/HADOOP-14973
> Project: Hadoop Common
>  Issue Type: Sub-task
>  Components: fs/s3
>Affects Versions: 3.0.0-beta1, 2.8.1
>Reporter: Sean Mackrory
>Assignee: Sean Mackrory
>
> S3A is currently storing much more detailed metrics via StorageStatistics 
> than are logged in a MapReduce job. Eventually, it would be nice to get 
> Spark, MapReduce and other workloads to retrieve and store these metrics, but 
> it may be some time before they all do that. I'd like to consider having S3A 
> publish the metrics itself in some form. This is tricky, as S3A has no daemon 
> but lives inside various other processes.
> Perhaps writing to a log file at some configurable interval and on close() 
> would be the best we could do. Other ideas would be welcome.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-13032) Refactor FileSystem$Statistics to use StorageStatistics

2017-10-28 Thread Steve Loughran (JIRA) 

[ 
https://issues.apache.org/jira/browse/HADOOP-13032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16223399#comment-16223399
 ] 

Steve Loughran commented on HADOOP-13032:
-

Like Andrew says, this is going to be incompatible. Doesn't stop us making the 
news storage stats visible directly though, will it

> Refactor FileSystem$Statistics to use StorageStatistics
> ---
>
> Key: HADOOP-13032
> URL: https://issues.apache.org/jira/browse/HADOOP-13032
> Project: Hadoop Common
>  Issue Type: Sub-task
>  Components: fs
>Reporter: Mingliang Liu
>Assignee: Mingliang Liu
> Attachments: HADOOP-13032.000.patch, HADOOP-13032.001.patch, 
> HADOOP-13032.002.patch, HADOOP-13032.003.patch, HADOOP-13032.004.patch, 
> HADOOP-13032.005.patch
>
>
> [HADOOP-13065] added a new interface for retrieving FS and FC Statistics. 
> This jira is to track the effort of moving the {{Statistics}} class out of 
> {{FileSystem}}, and make it use that new interface.
> We should keep the thread local implementation. Benefits are:
> # they could be used in both {{FileContext}} and {{FileSystem}}
> # unified stats data structure
> # shorter source code
> Please note this will be an backwards-incompatible change.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (HADOOP-13435) Add thread local mechanism for aggregating file system storage stats

2017-10-28 Thread Steve Loughran (JIRA) 

[ 
https://issues.apache.org/jira/browse/HADOOP-13435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16223396#comment-16223396
 ] 

Steve Loughran commented on HADOOP-13435:
-

h3. main code

I'd like to be able to have a single thread access its stats in isolation. This 
will let a single threaded task execution build up the stats which I can then 
include in the {{_SUCCESS}} manifest. for later aggregation

{code}
setupTask(taskContext): stats0 = fs.getThreadStats()
...do work
commitTask(taskContext): stats1 = fs.getThreadStats(); diff = 
stats1.diff(stats0), log(diff), _SUCCESS.stats=diff
{code}

other: New FS keynames should go into CommonStatisticNames if we plan to use 
them more broadly, referencing from the enum.


h3. tests should 

* hand in some -ve read ranges
* try to read nonexistent key
* look @ using Callable & futures for testIncrementAndGet for errors to come up 
(); if a child has an error, just throw the exception directly, rather than an 
assertNull
* could refactor out common {{assertEquals(0, stats.get(MyStatType.FIRST))}} 
tome {{assertStatEquals(expected, sts, key)}} with error text to include key 
name


> Add thread local mechanism for aggregating file system storage stats
> 
>
> Key: HADOOP-13435
> URL: https://issues.apache.org/jira/browse/HADOOP-13435
> Project: Hadoop Common
>  Issue Type: Sub-task
>  Components: fs
>Reporter: Mingliang Liu
>Assignee: Mingliang Liu
> Attachments: HADOOP-13435.000.patch, HADOOP-13435.001.patch, 
> HADOOP-13435.002.patch, HADOOP-13435.003.patch, HADOOP-13435.004.patch
>
>
> As discussed in [HADOOP-13032], this is to add thread local mechanism for 
> aggregating file system storage stats. This class will also be used in 
> [HADOOP-13031], which is to separate the distance-oriented rack-aware read 
> bytes logic from {{FileSystemStorageStatistics}} to new 
> DFSRackAwareStorageStatistics as it's DFS-specific. After this patch, the 
> {{FileSystemStorageStatistics}} can live without the to-be-removed 
> {{FileSystem$Statistics}} implementation.
> A unit test should also be added.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Updated] (HADOOP-13435) Add thread local mechanism for aggregating file system storage stats

2017-10-28 Thread Steve Loughran (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HADOOP-13435?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Steve Loughran updated HADOOP-13435:

Affects Version/s: 2.9.0
 Target Version/s: 3.1.0  (was: 3.0.0)

> Add thread local mechanism for aggregating file system storage stats
> 
>
> Key: HADOOP-13435
> URL: https://issues.apache.org/jira/browse/HADOOP-13435
> Project: Hadoop Common
>  Issue Type: Sub-task
>  Components: fs
>Affects Versions: 2.9.0
>Reporter: Mingliang Liu
>Assignee: Mingliang Liu
> Attachments: HADOOP-13435.000.patch, HADOOP-13435.001.patch, 
> HADOOP-13435.002.patch, HADOOP-13435.003.patch, HADOOP-13435.004.patch
>
>
> As discussed in [HADOOP-13032], this is to add thread local mechanism for 
> aggregating file system storage stats. This class will also be used in 
> [HADOOP-13031], which is to separate the distance-oriented rack-aware read 
> bytes logic from {{FileSystemStorageStatistics}} to new 
> DFSRackAwareStorageStatistics as it's DFS-specific. After this patch, the 
> {{FileSystemStorageStatistics}} can live without the to-be-removed 
> {{FileSystem$Statistics}} implementation.
> A unit test should also be added.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org