Re: making a hadoop-common test run if a property is set
thanks, I'l; have a look. I've always wanted to add the notion of skipped
to test runs -all the way through to the XML and generated reports, but
you'd have to do a new junit runner for this and tweak the reporting code.
Which, if it involved going near maven source, is not something I am
prepared to do
On 14 December 2012 18:57, Colin McCabe cmcc...@alumni.cmu.edu wrote:
One approach we've taken in the past is making the junit test skip
itself when some precondition is not true. Then, we often create a
property which people can use to cause the skipped tests to become a
hard error.
For example, all the tests that rely on libhadoop start with these lines:
@Test
public void myTest() {
Assume.assumeTrue(NativeCodeLoader.isNativeCodeLoaded());
...
}
This causes them to be silently skipped when libhadoop.so is not
available or loaded (perhaps because it hasn't been built.)
However, if you want to cause this to be a hard error, you simply run
mvn test -Drequire.test.libhadoop
See TestHdfsNativeCodeLoader.java to see how this is implemented.
The main idea is that your Jenkins build slaves use all the -Drequire
lines, but people running tests locally are not inconvenienced by the
need to build libhadoop.so in every case. This is especially good
because libhadoop.so isn't known to build on certain platforms like
AIX, etc. It seems to be a good tradeoff so far. I imagine that s3
could do something similar.
cheers,
Colin
On Fri, Dec 14, 2012 at 9:56 AM, Steve Loughran ste...@hortonworks.com
wrote:
The swiftfs tests need only to run if there's a target filesystem;
copying
the s3/s3n tests, something like
property
nametest.fs.swift.name/name
valueswift://your-object-store-herel//value
/property
How does one actually go about making junit tests optional in mvn-land?
Should the probe/skip logic be in the code -which can make people think
the
test passed when it didn't actually run? Or can I turn it on/off in
maven?
-steve
Re: making a hadoop-common test run if a property is set
There are some tests like the S3 tests that end with Test (e.g.
Jets3tNativeS3FileSystemContractTest) - unlike normal tests which
start with Test. Only those that start with Test are run
automatically (see the surefire configuration in
hadoop-project/pom.xml). You have to run the others manually with mvn
test -Dtest=
The mechanism that Colin describes is probably better though, since
the environment-specific tests can be run as a part of a full test run
by Jenkins if configured appropriately.
Tom
On Mon, Dec 17, 2012 at 10:06 AM, Steve Loughran ste...@hortonworks.com wrote:
thanks, I'l; have a look. I've always wanted to add the notion of skipped
to test runs -all the way through to the XML and generated reports, but
you'd have to do a new junit runner for this and tweak the reporting code.
Which, if it involved going near maven source, is not something I am
prepared to do
On 14 December 2012 18:57, Colin McCabe cmcc...@alumni.cmu.edu wrote:
One approach we've taken in the past is making the junit test skip
itself when some precondition is not true. Then, we often create a
property which people can use to cause the skipped tests to become a
hard error.
For example, all the tests that rely on libhadoop start with these lines:
@Test
public void myTest() {
Assume.assumeTrue(NativeCodeLoader.isNativeCodeLoaded());
...
}
This causes them to be silently skipped when libhadoop.so is not
available or loaded (perhaps because it hasn't been built.)
However, if you want to cause this to be a hard error, you simply run
mvn test -Drequire.test.libhadoop
See TestHdfsNativeCodeLoader.java to see how this is implemented.
The main idea is that your Jenkins build slaves use all the -Drequire
lines, but people running tests locally are not inconvenienced by the
need to build libhadoop.so in every case. This is especially good
because libhadoop.so isn't known to build on certain platforms like
AIX, etc. It seems to be a good tradeoff so far. I imagine that s3
could do something similar.
cheers,
Colin
On Fri, Dec 14, 2012 at 9:56 AM, Steve Loughran ste...@hortonworks.com
wrote:
The swiftfs tests need only to run if there's a target filesystem;
copying
the s3/s3n tests, something like
property
nametest.fs.swift.name/name
valueswift://your-object-store-herel//value
/property
How does one actually go about making junit tests optional in mvn-land?
Should the probe/skip logic be in the code -which can make people think
the
test passed when it didn't actually run? Or can I turn it on/off in
maven?
-steve
[jira] [Resolved] (HADOOP-8981) TestMetricsSystemImpl fails on Windows
[ https://issues.apache.org/jira/browse/HADOOP-8981?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suresh Srinivas resolved HADOOP-8981. - Resolution: Fixed Fix Version/s: trunk-win Hadoop Flags: Reviewed I committed the patch. Thank you Xuan. TestMetricsSystemImpl fails on Windows -- Key: HADOOP-8981 URL: https://issues.apache.org/jira/browse/HADOOP-8981 Project: Hadoop Common Issue Type: Bug Components: metrics Affects Versions: trunk-win Reporter: Chris Nauroth Assignee: Xuan Gong Fix For: trunk-win Attachments: HADOOP-8981-branch-trunk-win.1.patch, HADOOP-8981-branch-trunk-win.2.patch, HADOOP-8981-branch-trunk-win.3.patch, HADOOP-8981-branch-trunk-win.4.patch, HADOOP-8981-branch-trunk-win.5.patch The test is failing on an expected mock interaction. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HADOOP-9150) Unnecessary DNS resolution attempts for logical URIs
Todd Lipcon created HADOOP-9150: --- Summary: Unnecessary DNS resolution attempts for logical URIs Key: HADOOP-9150 URL: https://issues.apache.org/jira/browse/HADOOP-9150 Project: Hadoop Common Issue Type: Bug Components: ha Affects Versions: 2.0.2-alpha, 3.0.0 Reporter: Todd Lipcon Priority: Critical In the FileSystem code, we accidentally try to DNS-resolve the logical name before it is converted to an actual domain name. In some DNS setups, this can cause a big slowdown - eg in one misconfigured cluster we saw a 2-3x drop in terasort throughput, since every task wasted a lot of time waiting for slow not found responses from DNS. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: making a hadoop-common test run if a property is set
On 17 December 2012 16:06, Tom White t...@cloudera.com wrote:
There are some tests like the S3 tests that end with Test (e.g.
Jets3tNativeS3FileSystemContractTest) - unlike normal tests which
start with Test. Only those that start with Test are run
automatically (see the surefire configuration in
hadoop-project/pom.xml). You have to run the others manually with mvn
test -Dtest=
The mechanism that Colin describes is probably better though, since
the environment-specific tests can be run as a part of a full test run
by Jenkins if configured appropriately.
I'd like that -though one problem with the current system is that you need
to get the s3 (and soon: openstack) credentials into
src/test/resources/core-site.xml, which isn't the right approach. If we
could get them into properties files things would be easier.
another tactic could be to have specific test projects: test-s3,
test-openstack, test-... which contain nothing but test cases. You'd set
jenkins up those test projects too -the reason for having the separate
names is to make it blatantly clear which tests you've not run
That's overkill for adding a few more openstack tests -but I would like to
make it easier to turn those and the rackspace ones without sticking my
secrets into an XML file under SCM
Tom
On Mon, Dec 17, 2012 at 10:06 AM, Steve Loughran ste...@hortonworks.com
wrote:
thanks, I'l; have a look. I've always wanted to add the notion of skipped
to test runs -all the way through to the XML and generated reports, but
you'd have to do a new junit runner for this and tweak the reporting
code.
Which, if it involved going near maven source, is not something I am
prepared to do
On 14 December 2012 18:57, Colin McCabe cmcc...@alumni.cmu.edu wrote:
One approach we've taken in the past is making the junit test skip
itself when some precondition is not true. Then, we often create a
property which people can use to cause the skipped tests to become a
hard error.
For example, all the tests that rely on libhadoop start with these
lines:
@Test
public void myTest() {
Assume.assumeTrue(NativeCodeLoader.isNativeCodeLoaded());
...
}
This causes them to be silently skipped when libhadoop.so is not
available or loaded (perhaps because it hasn't been built.)
However, if you want to cause this to be a hard error, you simply run
mvn test -Drequire.test.libhadoop
See TestHdfsNativeCodeLoader.java to see how this is implemented.
The main idea is that your Jenkins build slaves use all the -Drequire
lines, but people running tests locally are not inconvenienced by the
need to build libhadoop.so in every case. This is especially good
because libhadoop.so isn't known to build on certain platforms like
AIX, etc. It seems to be a good tradeoff so far. I imagine that s3
could do something similar.
cheers,
Colin
On Fri, Dec 14, 2012 at 9:56 AM, Steve Loughran ste...@hortonworks.com
wrote:
The swiftfs tests need only to run if there's a target filesystem;
copying
the s3/s3n tests, something like
property
nametest.fs.swift.name/name
valueswift://your-object-store-herel//value
/property
How does one actually go about making junit tests optional in
mvn-land?
Should the probe/skip logic be in the code -which can make people
think
the
test passed when it didn't actually run? Or can I turn it on/off in
maven?
-steve
[jira] [Resolved] (HADOOP-9018) Reject invalid Windows URIs
[ https://issues.apache.org/jira/browse/HADOOP-9018?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal resolved HADOOP-9018. --- Resolution: Fixed Fixed as part of HADOOP-9056. Reject invalid Windows URIs --- Key: HADOOP-9018 URL: https://issues.apache.org/jira/browse/HADOOP-9018 Project: Hadoop Common Issue Type: Improvement Components: fs Affects Versions: trunk-win Environment: Windows Reporter: Arpit Agarwal Assignee: Arpit Agarwal Attachments: HADOOP-9018.1.patch, HADOOP-9018.2.patch, HADOOP-9018.patch Original Estimate: 24h Remaining Estimate: 24h This JIRA is to make handling of improperly constructed file URIs for Windows local paths more rigorous. e.g. reject file:///c:\\Windows Valid file URI syntax explained at http://blogs.msdn.com/b/ie/archive/2006/12/06/file-uris-in-windows.aspx. Also see https://issues.apache.org/jira/browse/HADOOP-8953 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HADOOP-9151) Include RPC error info in RpcResponseHeader instead of sending it separately
Sanjay Radia created HADOOP-9151: Summary: Include RPC error info in RpcResponseHeader instead of sending it separately Key: HADOOP-9151 URL: https://issues.apache.org/jira/browse/HADOOP-9151 Project: Hadoop Common Issue Type: Sub-task Reporter: Sanjay Radia Assignee: Sanjay Radia -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
