[jira] [Created] (HADOOP-15499) Performance several drop when running RawErasureCoderBenchmark with NativeRSRawErasureCoder

2018-05-29 Thread SammiChen (JIRA) 
SammiChen created HADOOP-15499:
--

 Summary: Performance several drop when running 
RawErasureCoderBenchmark with NativeRSRawErasureCoder
 Key: HADOOP-15499
 URL: https://issues.apache.org/jira/browse/HADOOP-15499
 Project: Hadoop Common
  Issue Type: Improvement
Affects Versions: 3.0.2, 3.0.1, 3.0.0
Reporter: SammiChen
Assignee: SammiChen


Run RawErasureCoderBenchmark  which is a micro-benchmark to test EC codec 
encoding/decoding performance. 

50 concurrency Native ISA-L coder has the less throughput than 1 concurrency 
Native ISA-L case. It's abnormal. 

 

bin/hadoop jar ./share/hadoop/common/hadoop-common-3.2.0-SNAPSHOT-tests.jar 
org.apache.hadoop.io.erasurecode.rawcoder.RawErasureCoderBenchmark encode 3 1 
1024 1024
Using 126MB buffer.
ISA-L coder encode 1008MB data, with chunk size 1024KB
Total time: 0.19 s.
Total throughput: 5390.37 MB/s
Threads statistics:
1 threads in total.
Min: 0.18 s, Max: 0.18 s, Avg: 0.18 s, 90th Percentile: 0.18 s.

 

bin/hadoop jar ./share/hadoop/common/hadoop-common-3.2.0-SNAPSHOT-tests.jar 
org.apache.hadoop.io.erasurecode.rawcoder.RawErasureCoderBenchmark encode 3 50 
1024 10240
Using 120MB buffer.
ISA-L coder encode 54000MB data, with chunk size 10240KB
Total time: 11.58 s.
Total throughput: 4662 MB/s
Threads statistics:
50 threads in total.
Min: 0.55 s, Max: 11.5 s, Avg: 6.32 s, 90th Percentile: 10.45 s.

 

RawErasureCoderBenchmark shares a single coder between all concurrent threads. 
While 

NativeRSRawEncoder and NativeRSRawDecoder has synchronized key work on doDecode 
and doEncode function. So 50 concurrent threads are forced to use the shared 
coder encode/decode function one by one. 

 

To resolve the issue, there are two approaches. 
 # Refactor RawErasureCoderBenchmark  to use dedicated coder for each 
concurrent thread.
 # Refactor NativeRSRawEncoder  and NativeRSRawDecoder  to get better 
concurrency.  Since the synchronized key work is to try to protect the private 
variable nativeCoder from being checked in doEncode/doDecode and  being 
modified in release.  We can use reentrantReadWriteLock to increase the 
concurrency since doEncode/doDecode can be called multiple times without change 
the nativeCoder state.

 I prefer approach 2 and will upload a patch later. 

 

 

 

 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] SammiChen shared a search result with you

2018-01-10 Thread SammiChen (JIRA) 
SammiChen shared a search result with you



https://issues.apache.org/jira/secure/IssueNavigator.jspa?reset=true=%22Target+Version%2Fs%22+%3D+2.9.1+AND+%28project+%3D+hadoop+OR+project+%3D+%22Hadoop+HDFS%22+OR+project+%3D+%22Hadoop+YARN%22+OR+project+%3D+%22Hadoop+Map%2FReduce%22+OR+project+%3D+%22Hadoop+Common%22%29+AND+status+%21%3D+resolved+ORDER+BY+priority+DESC

Be public

 Also shared with
  yarn-...@hadoop.apache.org
  mapreduce-...@hadoop.apache.org
  hdfs-...@hadoop.apache.org



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Resolved] (HADOOP-15041) XInclude support in .xml configuration file is broken after "5eb7dbe9b31a45f57f2e1623aa1c9ce84a56c4d1" commit

2017-11-15 Thread SammiChen (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HADOOP-15041?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

SammiChen resolved HADOOP-15041.

Resolution: Fixed

> XInclude support in .xml configuration file is broken after 
> "5eb7dbe9b31a45f57f2e1623aa1c9ce84a56c4d1" commit
> -
>
> Key: HADOOP-15041
> URL: https://issues.apache.org/jira/browse/HADOOP-15041
> Project: Hadoop Common
>  Issue Type: Bug
>Reporter: SammiChen
>Assignee: Arun Suresh
>Priority: Critical
>
> XInclude support in .xml configuration file is broken after following 
> check-in.  
> Since there is no JIRA number in the following commit message, create a new 
> JIRA to track the issue. 
> commit 5eb7dbe9b31a45f57f2e1623aa1c9ce84a56c4d1
> Author: Arun Suresh 
> Date:   Thu Nov 9 15:15:51 2017 -0800
> Fixing Job History Server Configuration parsing. (Jason Lowe via asuresh)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-15041) XInclude support in .xml configuration file is broken after "5eb7dbe9b31a45f57f2e1623aa1c9ce84a56c4d1" commit

2017-11-15 Thread SammiChen (JIRA) 
SammiChen created HADOOP-15041:
--

 Summary: XInclude support in .xml configuration file is broken 
after "5eb7dbe9b31a45f57f2e1623aa1c9ce84a56c4d1" commit
 Key: HADOOP-15041
 URL: https://issues.apache.org/jira/browse/HADOOP-15041
 Project: Hadoop Common
  Issue Type: Bug
Reporter: SammiChen
Assignee: Arun Suresh
Priority: Critical


XInclude support in .xml configuration file is broken after following check-in. 
 

Since there is no JIRA number in the following commit message, create a new 
JIRA to track the issue. 

commit 5eb7dbe9b31a45f57f2e1623aa1c9ce84a56c4d1
Author: Arun Suresh 
Date:   Thu Nov 9 15:15:51 2017 -0800

Fixing Job History Server Configuration parsing. (Jason Lowe via asuresh)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-15024) AliyunOSS: Provide oss client side Hadoop version information to oss server

2017-11-08 Thread SammiChen (JIRA) 
SammiChen created HADOOP-15024:
--

 Summary: AliyunOSS: Provide oss client side Hadoop version 
information to oss server
 Key: HADOOP-15024
 URL: https://issues.apache.org/jira/browse/HADOOP-15024
 Project: Hadoop Common
  Issue Type: Sub-task
Reporter: SammiChen
Assignee: SammiChen


Provide oss client side Hadoop version to oss server, to help build access 
statistic metrics. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-14865) Mvnsite fail to execute macro defined in the document HDFSErasureCoding.md

2017-09-12 Thread SammiChen (JIRA) 
SammiChen created HADOOP-14865:
--

 Summary: Mvnsite fail to execute macro defined in the document 
HDFSErasureCoding.md
 Key: HADOOP-14865
 URL: https://issues.apache.org/jira/browse/HADOOP-14865
 Project: Hadoop Common
  Issue Type: Bug
  Components: build
Reporter: SammiChen


[ERROR] Failed to execute goal 
org.apache.maven.plugins:maven-site-plugin:3.6:site (default-site) on project 
hadoop-hdfs: Error parsing 
'/testptch/hadoop/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HDFSErasureCoding.md':
 line [-1] Error parsing the model: Unable to execute macro in the document: 
toc -> [Help 1]



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-14593) Improve TestCodecRawCoderMapping to support native RS coder

2017-06-26 Thread SammiChen (JIRA) 
SammiChen created HADOOP-14593:
--

 Summary: Improve TestCodecRawCoderMapping to support native RS 
coder
 Key: HADOOP-14593
 URL: https://issues.apache.org/jira/browse/HADOOP-14593
 Project: Hadoop Common
  Issue Type: Improvement
  Components: common
Reporter: SammiChen
Assignee: SammiChen


Currently this unit test will fail due to not support the native RS coder.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-13504) Refactor jni_common to conform to C89 restrictions imposed by Visual Studio 2010

2016-08-16 Thread SammiChen (JIRA) 
SammiChen created HADOOP-13504:
--

 Summary: Refactor jni_common to conform to C89 restrictions 
imposed by Visual Studio 2010
 Key: HADOOP-13504
 URL: https://issues.apache.org/jira/browse/HADOOP-13504
 Project: Hadoop Common
  Issue Type: Sub-task
Reporter: SammiChen
Assignee: SammiChen


Piece of code in jni_common declares variables after the first statement in 
function. This behavior is not allowed in compilers, such as Visual Studio 
2010, which only supports C89 C standards. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org