[jira] [Updated] (HADOOP-8303) Clean up hadoop-streaming

[Harsh J (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-8303?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Harsh J updated HADOOP-8303:


Attachment: 0008-MapReduce-Streaming-Cleanup.patch

Patch that cleans all warnings of hadoop-streaming sub-module save for usage of 
MiniMRCluster and DistributedCache and one instance of StreamJob's non usual 
constructor cause a job reference is still required there.

 Clean up hadoop-streaming
 -

 Key: HADOOP-8303
 URL: https://issues.apache.org/jira/browse/HADOOP-8303
 Project: Hadoop Common
  Issue Type: Sub-task
  Components: build
Affects Versions: 3.0.0
Reporter: Harsh J
Assignee: Harsh J
Priority: Minor
 Attachments: 0008-MapReduce-Streaming-Cleanup.patch


 Clean up a bunch of existing javac warnings in hadoop-streaming module.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HADOOP-8303) Clean up hadoop-streaming

[Harsh J (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-8303?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Harsh J updated HADOOP-8303:


Status: Patch Available  (was: Open)

 Clean up hadoop-streaming
 -

 Key: HADOOP-8303
 URL: https://issues.apache.org/jira/browse/HADOOP-8303
 Project: Hadoop Common
  Issue Type: Sub-task
  Components: build
Affects Versions: 3.0.0
Reporter: Harsh J
Assignee: Harsh J
Priority: Minor
 Attachments: 0008-MapReduce-Streaming-Cleanup.patch


 Clean up a bunch of existing javac warnings in hadoop-streaming module.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HADOOP-8303) Clean up hadoop-streaming

[Harsh J (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-8303?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Harsh J updated HADOOP-8303:


Attachment: 0008-MapReduce-Streaming-Cleanup.patch

A few things:

# Looks like even Hadoop-Common QA does not pick up hadoop-tools changes.
# The eclipse's auto unused import fixer removed RecRecord imports which caused 
TestIO not to compile. I fixed those manually in this new patch.
# TestStreaming had a change in original patch that caused it to lose info 
about the running jobs and such. I tweaked the change to use a non ToolRunner 
approach, and TestStreamingCounters/Combiner now work nicely again - minus 
deprecation warnings.

Since Hadoop QA bot can't run the tests, here is the test result manually run, 
with patch applied:

{{cd hadoop-tools/hadoop-streaming; mvn clean test}}

{code}
---
 T E S T S
---
Running org.apache.hadoop.streaming.TestAutoInputFormat
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.663 sec
Running org.apache.hadoop.streaming.TestClassWithNoPackage
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.261 sec
Running org.apache.hadoop.streaming.TestDumpTypedBytes
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 4.21 sec
Running org.apache.hadoop.streaming.TestFileArgs
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 19.49 sec
Running org.apache.hadoop.streaming.TestGzipInput
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.188 sec
Running org.apache.hadoop.streaming.TestLoadTypedBytes
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 3.486 sec
Running org.apache.hadoop.streaming.TestMRFramework
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.265 sec
Running org.apache.hadoop.streaming.TestMultipleArchiveFiles
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 22.727 sec
Running org.apache.hadoop.streaming.TestMultipleCachefiles
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 25.324 sec
Running org.apache.hadoop.streaming.TestRawBytesStreaming
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.07 sec
Running org.apache.hadoop.streaming.TestStreamAggregate
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.062 sec
Running org.apache.hadoop.streaming.TestStreamDataProtocol
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.058 sec
Running org.apache.hadoop.streaming.TestStreaming
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.165 sec
Running org.apache.hadoop.streaming.TestStreamingBackground
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.09 sec
Running org.apache.hadoop.streaming.TestStreamingBadRecords
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 9.769 sec
Running org.apache.hadoop.streaming.TestStreamingCombiner
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 3.21 sec
Running org.apache.hadoop.streaming.TestStreamingCounters
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.191 sec
Running org.apache.hadoop.streaming.TestStreamingExitStatus
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 6.031 sec
Running org.apache.hadoop.streaming.TestStreamingFailure
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.788 sec
Running org.apache.hadoop.streaming.TestStreamingKeyValue
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 3.324 sec
Running org.apache.hadoop.streaming.TestStreamingOutputKeyValueTypes
Tests run: 11, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 13.626 sec
Running org.apache.hadoop.streaming.TestStreamingSeparator
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.068 sec
Running org.apache.hadoop.streaming.TestStreamingStderr
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 14.579 sec
Running org.apache.hadoop.streaming.TestStreamingTaskLog
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 15.659 sec
Running org.apache.hadoop.streaming.TestStreamJob
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.761 sec
Running org.apache.hadoop.streaming.TestStreamReduceNone
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.146 sec
Running org.apache.hadoop.streaming.TestStreamXmlMultipleRecords
Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 6.238 sec
Running org.apache.hadoop.streaming.TestStreamXmlRecordReader
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.152 sec
Running org.apache.hadoop.streaming.TestSymLink
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 26.418 sec
Running org.apache.hadoop.streaming.TestTypedBytesStreaming
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.146 sec
Running