[jira] [Commented] (MAPREDUCE-5968) Work directory is not deleted in DistCache if Exception happen in downloadCacheObject.
[ https://issues.apache.org/jira/browse/MAPREDUCE-5968?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14085864#comment-14085864 ] Karthik Kambatla commented on MAPREDUCE-5968: - Latest patch looks good to me. +1. Committing this. Work directory is not deleted in DistCache if Exception happen in downloadCacheObject. --- Key: MAPREDUCE-5968 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5968 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv1 Affects Versions: 1.2.1 Reporter: zhihai xu Assignee: zhihai xu Attachments: MAPREDUCE-5968.branch1.patch, MAPREDUCE-5968.branch1_new.patch, MAPREDUCE-5968.branch1_new1.patch Work directory is not deleted in DistCache if Exception happen in downloadCacheObject. In downloadCacheObject, the cache file will be copied to temporarily work directory first, then the work directory will be renamed to the final directory. If IOException happens during the copy, the work directory will not be deleted. This will cause garbage data left in local disk cache. For example If the MR application use Distributed Cache to send a very large Archive/file(50G), if the disk is full during the copy, then the IOException will be triggered, the work directory will be not deleted or renamed and the work directory will occupy a big chunk of disk space. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (MAPREDUCE-5968) Work directory is not deleted when downloadCacheObject throws IOException
[ https://issues.apache.org/jira/browse/MAPREDUCE-5968?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karthik Kambatla updated MAPREDUCE-5968: Summary: Work directory is not deleted when downloadCacheObject throws IOException (was: Work directory is not deleted in DistCache if Exception happen in downloadCacheObject.) Work directory is not deleted when downloadCacheObject throws IOException - Key: MAPREDUCE-5968 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5968 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv1 Affects Versions: 1.2.1 Reporter: zhihai xu Assignee: zhihai xu Attachments: MAPREDUCE-5968.branch1.patch, MAPREDUCE-5968.branch1_new.patch, MAPREDUCE-5968.branch1_new1.patch Work directory is not deleted in DistCache if Exception happen in downloadCacheObject. In downloadCacheObject, the cache file will be copied to temporarily work directory first, then the work directory will be renamed to the final directory. If IOException happens during the copy, the work directory will not be deleted. This will cause garbage data left in local disk cache. For example If the MR application use Distributed Cache to send a very large Archive/file(50G), if the disk is full during the copy, then the IOException will be triggered, the work directory will be not deleted or renamed and the work directory will occupy a big chunk of disk space. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (MAPREDUCE-5968) Work directory is not deleted when downloadCacheObject throws IOException
[ https://issues.apache.org/jira/browse/MAPREDUCE-5968?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karthik Kambatla updated MAPREDUCE-5968: Resolution: Fixed Fix Version/s: 1.3.0 Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Thanks for the contribution, [~zxu]. Just committed this to branch-1. Work directory is not deleted when downloadCacheObject throws IOException - Key: MAPREDUCE-5968 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5968 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv1 Affects Versions: 1.2.1 Reporter: zhihai xu Assignee: zhihai xu Fix For: 1.3.0 Attachments: MAPREDUCE-5968.branch1.patch, MAPREDUCE-5968.branch1_new.patch, MAPREDUCE-5968.branch1_new1.patch Work directory is not deleted in DistCache if Exception happen in downloadCacheObject. In downloadCacheObject, the cache file will be copied to temporarily work directory first, then the work directory will be renamed to the final directory. If IOException happens during the copy, the work directory will not be deleted. This will cause garbage data left in local disk cache. For example If the MR application use Distributed Cache to send a very large Archive/file(50G), if the disk is full during the copy, then the IOException will be triggered, the work directory will be not deleted or renamed and the work directory will occupy a big chunk of disk space. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (MAPREDUCE-2015) GridMix always throws an FileAlreadyExistsException even ouput directory is not available in HDFS.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14086369#comment-14086369 ] Brian Husted commented on MAPREDUCE-2015: - [~ravidotg] [~inna.balasanyan] I am running into the exact same issue with Gridmix on 1.2.1 where it generates no input data with an empty _SUCCESS. I get from Gridmix: GRIDMIX_GENERATE_INPUT_DATA (job_local641395695_0001) success Any pointers how to resolve this issue? GridMix always throws an FileAlreadyExistsException even ouput directory is not available in HDFS. -- Key: MAPREDUCE-2015 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2015 Project: Hadoop Map/Reduce Issue Type: Bug Components: contrib/gridmix Affects Versions: 0.20.1 Reporter: Vinay Kumar Thota Assignee: Ranjit Mathew Gridmix always throws an FileAlreadyExistsException even ouput directory is not available in HDFS. Actually I was launching the Gridmix in a command line for generating the data, before launching I just make sure the output directory is not available in the HDFS by deleting the folder if already exists.However, I could see output directory already exists exception every time. Please see the attached logs for more information. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (MAPREDUCE-6007) Create a new option for distcp -p which causes raw.* namespace extended attributes to not be preserved
[
https://issues.apache.org/jira/browse/MAPREDUCE-6007?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Charles Lamb updated MAPREDUCE-6007:
Attachment: MAPREDUCE-6007.002.patch
{code}
-px : preserve raw and non-raw xattrs
-pr : no xattrs are preserved
-p : preserve raw xattrs
-pxr: preserve non-raw xattrs
: no xattrs are preserved
{code}
Yes, this is almost all correct. Assuming that -pr (above) is the same as -pd
in the patch, then the only typo above is the last line which should be no raw
xattrs are preserved, but I think you had the raw part implied so I throw
that in just to be crystal clear.
{quote}
raw src, raw dst: the options apply as specified above
raw src, not-raw dst, dst supports xattrs but no /reserved/.raw: we will
fail to set raw xattrs at runtime.
raw src, dst doesn't support xattrs: if -pX is specified, throws an
exception. Else, silently discards raw xattrs.
{quote}
bq. If the src is /reserved/.raw, the user is expecting preservation of raw
xattrs when -p or -pX is specified. In this scenario, we should test that the
dest is /.reserved/raw and that it's present on the dstFS. There might be other
weird cases, haven't thought through all of them
Good idea. I've cleaned this up so that anytime there's a /.reserved/raw src
path present, and either pd or a non- /.reserved/raw target path is specified,
an exception is thrown (actually a DistCpConstants.INVALID_ARGUMENT exit code
is returned and a message logged). This will presumably help admins from
shooting themselves in the foot.
bq. We have both noPreserveRaw and preserveRaw booleans, can we standardize on
one everywhere? I'd like a negative one, call it disableRaw or excludeRaw since
it better captures the meaning of the flag. exclude feels a bit better IMO, but
it looks like -pe is taken.
I changed all of the booleans to be disableRawXattrs (vs disableRaw). This
way it matches the preserveXattrs siblings in the code. However, in
SimpleCopyList#doBuildListing, the boolean is still named copyRawXAttrs to
mirror its copyAcls and copyXAttrs siblings. It felt wrong to change that to a
disableRawXattrs. Ditto #traverseNonEmptyDirectory. Not to mention that if we
negated the sense of the boolean from copyRawXattrs to disableRawXattrs, it
would hair up the calls to toCopyListFileStatus when they are 'd with
child.isDirectory(). It feels like it's cleaner to leave all three booleans in
the positive in these two cases.
I left the name of the enum the same (DISABLERAWXATTRS) since (1) it is an
available letter (d), and (2) I think you were only suggesting changing the
name of the booleans in the code. I can change it to DISABLERAW if you prefer.
bq. What's the expected behavior when the dest doesn't support xattrs or
reserved raw, or supports xattrs but not reserved raw?
If the dest doesn't support xattrs or raw, then (assuming -px was specified) an
exception is thrown (per current behavior). If the dest supports xattrs but not
reserved raw, then it depends on whether reserved raw was specified in the
source or not. If it was, then an INVALID_ARGUMENT (-1) is returned as the exit
code. If reserved/raw was not specified on any source path, then silence.
bq.CopyListing, this is where we'd also test to see if the destFS has a
/.reserved/raw directory
Yes, I've added a test for src/dst reserved/raw mismatch here.
I've updated the documentation per your suggestions.
{quote}
if (!Path.getPathWithoutSchemeAndAuthority(target).toString().
What if the target is a relative path here?
{quote}
Check me on this. I convinced myself that a relative path could never be
relative to /.reserved/raw since you can't set your working directory to that.
bq. Any reason this isn't part of the existing XAttr test? They seem pretty
similar, and you also added a PXD test to the existing test.
I started out with a combined test, but during development, they diverged
enough in terms of structure, @BeforeClass, files that they operated on,
initializing files, directories, xattrs, etc. that it felt cleaner to leave
them split into two tests.
Create a new option for distcp -p which causes raw.* namespace extended
attributes to not be preserved
--
Key: MAPREDUCE-6007
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6007
Project: Hadoop Map/Reduce
Issue Type: New Feature
Components: distcp
Affects Versions: fs-encryption
Reporter: Charles Lamb
Assignee: Charles Lamb
Attachments: MAPREDUCE-6007.001.patch, MAPREDUCE-6007.002.patch
As part of the Data at Rest Encryption work (HDFS-6134), we need to create a
new option for distcp which causes raw.* namespace extended attributes to not
be preserved. See the doc in
[jira] [Updated] (MAPREDUCE-4815) FileOutputCommitter.commitJob can be very slow for jobs with many output files
[ https://issues.apache.org/jira/browse/MAPREDUCE-4815?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siqi Li updated MAPREDUCE-4815: --- Status: Open (was: Patch Available) FileOutputCommitter.commitJob can be very slow for jobs with many output files -- Key: MAPREDUCE-4815 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4815 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 2.4.1, 2.0.1-alpha, 0.23.3 Reporter: Jason Lowe Assignee: Siqi Li Attachments: MAPREDUCE-4815.v1.patch If a job generates many files to commit then the commitJob method call at the end of the job can take minutes. This is a performance regression from 1.x, as 1.x had the tasks commit directly to the final output directory as they were completing and commitJob had very little to do. The commit work was processed in parallel and overlapped the processing of outstanding tasks. In 0.23/2.x, the commit is single-threaded and waits until all tasks have completed before commencing. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (MAPREDUCE-4815) FileOutputCommitter.commitJob can be very slow for jobs with many output files
[ https://issues.apache.org/jira/browse/MAPREDUCE-4815?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siqi Li updated MAPREDUCE-4815: --- Attachment: MAPREDUCE-4815.v2.patch FileOutputCommitter.commitJob can be very slow for jobs with many output files -- Key: MAPREDUCE-4815 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4815 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.3, 2.0.1-alpha, 2.4.1 Reporter: Jason Lowe Assignee: Siqi Li Attachments: MAPREDUCE-4815.v1.patch, MAPREDUCE-4815.v2.patch If a job generates many files to commit then the commitJob method call at the end of the job can take minutes. This is a performance regression from 1.x, as 1.x had the tasks commit directly to the final output directory as they were completing and commitJob had very little to do. The commit work was processed in parallel and overlapped the processing of outstanding tasks. In 0.23/2.x, the commit is single-threaded and waits until all tasks have completed before commencing. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (MAPREDUCE-4815) FileOutputCommitter.commitJob can be very slow for jobs with many output files
[ https://issues.apache.org/jira/browse/MAPREDUCE-4815?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siqi Li updated MAPREDUCE-4815: --- Status: Patch Available (was: Open) FileOutputCommitter.commitJob can be very slow for jobs with many output files -- Key: MAPREDUCE-4815 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4815 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 2.4.1, 2.0.1-alpha, 0.23.3 Reporter: Jason Lowe Assignee: Siqi Li Attachments: MAPREDUCE-4815.v1.patch, MAPREDUCE-4815.v2.patch If a job generates many files to commit then the commitJob method call at the end of the job can take minutes. This is a performance regression from 1.x, as 1.x had the tasks commit directly to the final output directory as they were completing and commitJob had very little to do. The commit work was processed in parallel and overlapped the processing of outstanding tasks. In 0.23/2.x, the commit is single-threaded and waits until all tasks have completed before commencing. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (MAPREDUCE-4815) FileOutputCommitter.commitJob can be very slow for jobs with many output files
[
https://issues.apache.org/jira/browse/MAPREDUCE-4815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14086607#comment-14086607
]
Hadoop QA commented on MAPREDUCE-4815:
--
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12659925/MAPREDUCE-4815.v2.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4788//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4788//console
This message is automatically generated.
FileOutputCommitter.commitJob can be very slow for jobs with many output files
--
Key: MAPREDUCE-4815
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4815
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Affects Versions: 0.23.3, 2.0.1-alpha, 2.4.1
Reporter: Jason Lowe
Assignee: Siqi Li
Attachments: MAPREDUCE-4815.v1.patch, MAPREDUCE-4815.v2.patch
If a job generates many files to commit then the commitJob method call at the
end of the job can take minutes. This is a performance regression from 1.x,
as 1.x had the tasks commit directly to the final output directory as they
were completing and commitJob had very little to do. The commit work was
processed in parallel and overlapped the processing of outstanding tasks. In
0.23/2.x, the commit is single-threaded and waits until all tasks have
completed before commencing.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (MAPREDUCE-6014) New task status field in task attempts table can lead to an empty web page
[ https://issues.apache.org/jira/browse/MAPREDUCE-6014?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14086849#comment-14086849 ] Jason Lowe commented on MAPREDUCE-6014: --- +1 lgtm as well. Committing this. New task status field in task attempts table can lead to an empty web page --- Key: MAPREDUCE-6014 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6014 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 2.5.0 Reporter: Mit Desai Assignee: Mit Desai Attachments: MAPREDUCE-6014.patch MAPREDUCE-5550 added a new task attempts field but didn't Javascript-escape the contents. Tasks with status messages that have newlines or other characters can then break the parsing of the web page and leave the user with a blank page. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (MAPREDUCE-6014) New task status field in task attempts table can lead to an empty web page
[ https://issues.apache.org/jira/browse/MAPREDUCE-6014?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Lowe updated MAPREDUCE-6014: -- Resolution: Fixed Fix Version/s: 2.6.0 3.0.0 Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Thanks to Mit for the contribution and Jon for additional review! I committed this to trunk and branch-2. New task status field in task attempts table can lead to an empty web page --- Key: MAPREDUCE-6014 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6014 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 2.5.0 Reporter: Mit Desai Assignee: Mit Desai Fix For: 3.0.0, 2.6.0 Attachments: MAPREDUCE-6014.patch MAPREDUCE-5550 added a new task attempts field but didn't Javascript-escape the contents. Tasks with status messages that have newlines or other characters can then break the parsing of the web page and leave the user with a blank page. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (MAPREDUCE-6014) New task status field in task attempts table can lead to an empty web page
[ https://issues.apache.org/jira/browse/MAPREDUCE-6014?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14086922#comment-14086922 ] Hudson commented on MAPREDUCE-6014: --- SUCCESS: Integrated in Hadoop-trunk-Commit #6018 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/6018/]) MAPREDUCE-6014. New task status field in task attempts table can lead to an empty web page. Contributed by Mit Desai (jlowe: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1616018) * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/webapp/TaskPage.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/webapp/TasksBlock.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/webapp/TestBlocks.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/webapp/HsTaskPage.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/java/org/apache/hadoop/mapreduce/v2/hs/webapp/TestBlocks.java New task status field in task attempts table can lead to an empty web page --- Key: MAPREDUCE-6014 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6014 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 2.5.0 Reporter: Mit Desai Assignee: Mit Desai Fix For: 3.0.0, 2.6.0 Attachments: MAPREDUCE-6014.patch MAPREDUCE-5550 added a new task attempts field but didn't Javascript-escape the contents. Tasks with status messages that have newlines or other characters can then break the parsing of the web page and leave the user with a blank page. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (MAPREDUCE-6007) Create a new option for distcp -p which causes raw.* namespace extended attributes to not be preserved
[
https://issues.apache.org/jira/browse/MAPREDUCE-6007?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14087131#comment-14087131
]
Hadoop QA commented on MAPREDUCE-6007:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12659910/MAPREDUCE-6007.002.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 4 new
or modified test files.
{color:red}-1 javac{color:red}. The patch appears to cause the build to
fail.
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4789//console
This message is automatically generated.
Create a new option for distcp -p which causes raw.* namespace extended
attributes to not be preserved
--
Key: MAPREDUCE-6007
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6007
Project: Hadoop Map/Reduce
Issue Type: New Feature
Components: distcp
Affects Versions: fs-encryption
Reporter: Charles Lamb
Assignee: Charles Lamb
Attachments: MAPREDUCE-6007.001.patch, MAPREDUCE-6007.002.patch
As part of the Data at Rest Encryption work (HDFS-6134), we need to create a
new option for distcp which causes raw.* namespace extended attributes to not
be preserved. See the doc in HDFS-6509 for details. The default for this
option will be to preserve raw.* xattrs.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (MAPREDUCE-5978) native-task CompressTest failure on Ubuntu
[ https://issues.apache.org/jira/browse/MAPREDUCE-5978?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14087254#comment-14087254 ] Todd Lipcon commented on MAPREDUCE-5978: [~clockfly] -- since you're a branch committer, feel free to commit Manu's patch. Feel free to ping me if you run into any issues native-task CompressTest failure on Ubuntu -- Key: MAPREDUCE-5978 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5978 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: task Reporter: Todd Lipcon Assignee: Manu Zhang Attachments: mapreduce-5978.txt The MR-2841 branch fails the following unit tests on my box: CompressTest.testBzip2Compress:84 file compare result: if they are the same ,then return true expected:true but was:false CompressTest.testDefaultCompress:116 file compare result: if they are the same ,then return true expected:true but was:false We need to fix these before merging. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (MAPREDUCE-5976) native-task should not fail to build if snappy is missing
[
https://issues.apache.org/jira/browse/MAPREDUCE-5976?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14087260#comment-14087260
]
Todd Lipcon commented on MAPREDUCE-5976:
Sorry for the delay here. The patch mostly looks good. Just a couple of nits:
{code}
+ if (job.getBoolean(MRJobConfig.MAP_OUTPUT_COMPRESS, false) == true) {
{code}
style: just use {{if (job.getBoolean(...))}} instead of comparing {{ == true }}.
{code}
+String codec = job.get(MRJobConfig.MAP_OUTPUT_COMPRESS_CODEC);
+if
(NativeRuntime.supportCompressionCodec(codec.getBytes(Charsets.UTF_8)) ==
false) {
{code}
Same -- just use {{!NativeRuntime.supportCompressionCodec(...)}} instead of
comparing {{ == false}}
Also, English nit: should be named {{supportsCompressionCodec}} (note the 's')
{code}
+ String message = Native output collector don't support compression
codec + codec;
{code}
Nit: should be doesn't support instead of don't support
{code}
+ *
+ * @param codecName
+ * @return
{code}
Remove these empty JavaDoc lines, since the main description of the function is
sufficient to understand it.
native-task should not fail to build if snappy is missing
-
Key: MAPREDUCE-5976
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5976
Project: Hadoop Map/Reduce
Issue Type: Sub-task
Components: task
Reporter: Todd Lipcon
Assignee: Sean Zhong
Attachments: mapreduce-5976-v2.txt, mapreduce-5976.txt
Other native parts of Hadoop will automatically disable snappy support if
snappy is not present and -Drequire.snappy is not passed. native-task should
do the same. (right now, it fails to build if snappy is missing)
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (MAPREDUCE-5987) native-task: Unit test TestGlibCBug fails on ubuntu
[ https://issues.apache.org/jira/browse/MAPREDUCE-5987?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14087267#comment-14087267 ] Todd Lipcon commented on MAPREDUCE-5987: Hey [~decster] -- is this fixed now by MAPREDUCE-6005? Seems like you fixed the memcpy-memmove thing there. If so, let's resolve this JIRA. native-task: Unit test TestGlibCBug fails on ubuntu --- Key: MAPREDUCE-5987 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5987 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: task Reporter: Sean Zhong Assignee: Sean Zhong Priority: Minor On ubuntu12, glibc: 2.15-0ubuntu10.3, UT TestGlibCBug fails [ RUN ] IFile.TestGlibCBug 14/07/21 15:55:30 INFO TestGlibCBug ./testData/testGlibCBugSpill.out /home/decster/projects/hadoop-trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/test/TestIFile.cc:186: Failure Value of: realKey Actual: 1127504685 Expected: expect[index] Which is: 4102672832 [ FAILED ] IFile.TestGlibCBug (0 ms) [--] 2 tests from IFile (240 ms total) -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (MAPREDUCE-5987) native-task: Unit test TestGlibCBug fails on ubuntu
[ https://issues.apache.org/jira/browse/MAPREDUCE-5987?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14087270#comment-14087270 ] Binglin Chang commented on MAPREDUCE-5987: -- At least on my ubuntu env, the bug doesn't show up. ] Sean can you give more comments? native-task: Unit test TestGlibCBug fails on ubuntu --- Key: MAPREDUCE-5987 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5987 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: task Reporter: Sean Zhong Assignee: Sean Zhong Priority: Minor On ubuntu12, glibc: 2.15-0ubuntu10.3, UT TestGlibCBug fails [ RUN ] IFile.TestGlibCBug 14/07/21 15:55:30 INFO TestGlibCBug ./testData/testGlibCBugSpill.out /home/decster/projects/hadoop-trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/test/TestIFile.cc:186: Failure Value of: realKey Actual: 1127504685 Expected: expect[index] Which is: 4102672832 [ FAILED ] IFile.TestGlibCBug (0 ms) [--] 2 tests from IFile (240 ms total) -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (MAPREDUCE-5976) native-task should not fail to build if snappy is missing
[ https://issues.apache.org/jira/browse/MAPREDUCE-5976?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Manu Zhang updated MAPREDUCE-5976: -- Attachment: mapreduce-5976-v3.txt native-task should not fail to build if snappy is missing - Key: MAPREDUCE-5976 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5976 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: task Reporter: Todd Lipcon Assignee: Sean Zhong Attachments: mapreduce-5976-v2.txt, mapreduce-5976-v3.txt, mapreduce-5976.txt Other native parts of Hadoop will automatically disable snappy support if snappy is not present and -Drequire.snappy is not passed. native-task should do the same. (right now, it fails to build if snappy is missing) -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (MAPREDUCE-5976) native-task should not fail to build if snappy is missing
[ https://issues.apache.org/jira/browse/MAPREDUCE-5976?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14087289#comment-14087289 ] Manu Zhang commented on MAPREDUCE-5976: --- thanks, Todd. I've updated the patch accordingly and also fixed don't support in other places. native-task should not fail to build if snappy is missing - Key: MAPREDUCE-5976 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5976 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: task Reporter: Todd Lipcon Assignee: Sean Zhong Attachments: mapreduce-5976-v2.txt, mapreduce-5976-v3.txt, mapreduce-5976.txt Other native parts of Hadoop will automatically disable snappy support if snappy is not present and -Drequire.snappy is not passed. native-task should do the same. (right now, it fails to build if snappy is missing) -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (MAPREDUCE-5984) native-task: upgrade lz4 to lastest version
[ https://issues.apache.org/jira/browse/MAPREDUCE-5984?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14087294#comment-14087294 ] Todd Lipcon commented on MAPREDUCE-5984: [~clockfly]/[~decster] -- feel free to commit native-task: upgrade lz4 to lastest version --- Key: MAPREDUCE-5984 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5984 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: task Reporter: Binglin Chang Assignee: Binglin Chang Priority: Minor Attachments: MAPREDUCE-5984.v1.patch, MAPREDUCE-5984.v2.patch -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (MAPREDUCE-5984) native-task: reuse lz4 sources in hadoop-common
[ https://issues.apache.org/jira/browse/MAPREDUCE-5984?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Binglin Chang updated MAPREDUCE-5984: - Summary: native-task: reuse lz4 sources in hadoop-common (was: native-task: upgrade lz4 to lastest version) native-task: reuse lz4 sources in hadoop-common --- Key: MAPREDUCE-5984 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5984 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: task Reporter: Binglin Chang Assignee: Binglin Chang Priority: Minor Attachments: MAPREDUCE-5984.v1.patch, MAPREDUCE-5984.v2.patch -- This message was sent by Atlassian JIRA (v6.2#6252)
