[
https://issues.apache.org/jira/browse/HBASE-7938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13602729#comment-13602729
]
Nick Dimiduk commented on HBASE-7938:
-------------------------------------
bq. I guess you had an open issue for this, right?
Yeah, that's what I ment when I said "The test works as intended when run
stand-alone, but FAILS when run via maven. It looks like HFileOutputFormat is
creating the paritions file on localFs instead of HDFS." The test does spin up
a cluster + mapred via IntegrationTestingUtility. Scroll back in the logs and
you'll see {{INFO mapreduce.HFileOutputFormat: Writing partition information
to}} and then a long local file path. That path ends up not existing when the
TotalOrderPartitioner gets around to looking for it; I don't know why. You'll
also see an error about "unable to create symlink" a little later on, because
that file path no longer exists.
This business is fixed in HBASE-4285. I do away with the symlink stuff and drop
the partitions file in /tmp instead of the working directory. I also request
that the file be deleted at FS close.
bq. Second question is that, whether we need to run the other 2 tests in
TestImportTsv in IT mode?
The point of the integration test (in my mind) is that it can be run on a real
cluster. If that's desirable for the tests in TestImportTsv, they might as well
be merged. The alternative is to try setting up TestImportTsv with a main that
can be run w.o maven.
Personally, I don't think we gain very much by having the integration test
stuff tied into maven. We don't have a persistent cluster that Jenkins can run
them against, so when they run through maven they're using the minicluster,
same as regular tests. After my experience with this thing, I think we're
better off writing them all as Tools with a main that can be run against an
externally specified conf.
> Add integration test for ImportTsv/LoadIncrementalHFiles workflow
> -----------------------------------------------------------------
>
> Key: HBASE-7938
> URL: https://issues.apache.org/jira/browse/HBASE-7938
> Project: HBase
> Issue Type: Sub-task
> Components: mapreduce
> Reporter: Nick Dimiduk
> Assignee: Nick Dimiduk
> Fix For: 0.95.0, 0.98.0
>
> Attachments:
> 0001-HBASE-7938-Add-integration-test-for-ImportTsv-LoadIn.patch,
> 0001-HBASE-7938-Add-integration-test-for-ImportTsv-LoadIn.patch,
> 0001-HBASE-7938-Add-integration-test-for-ImportTsv-LoadIn.patch
>
>
> We have existing unit tests for smoke-testing the packaged MR jobs, however
> they do not create a runtime environment that is true to running on a real MR
> cluster. This is particularly true in regard to classpaths (HBASE-7934) but
> also other static state (HBASE-4802). An integration test that can be pointed
> to run on a pseudo-distributed Hadoop deployed on localhost would find these
> kinds of problems.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
