[
https://issues.apache.org/jira/browse/SOLR-4916?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13683270#comment-13683270
]
Mark Miller commented on SOLR-4916:
-----------------------------------
Thanks for taking a look AB!
bq. Re. Hadoop dependencies: the patch adds a hard dependency on Hadoop and its
dependencies directly to Solr core. I wonder if it's possible to refactor it so
that it could be optional and the functionality itself moved to contrib/ - this
way only users who want to use HdfsDirectory would need Hadoop deps.
Yeah, I don't really beleive in Solr contribs - they are not so useful IMO -
it's a pain to actually pull them out and it has to be done after the fact.
Given the size of the dependencies is such a small percentage of the current
size, that we don't want to support the UpdateLog as actually pluggable, and
that it would be nice that hdfs was supported out of the box just as local
filesystem, I don't see being a contrib being much of a win. It saves a few
megabytes when we are already well over 100 - and that's if you are willing to
pull it apart after you download it. From what I've seen, even with the *huge*
extract contrib, most people don't bother repackaging. It's hard to imagine
they would for a few megabytes.
bq. Cache and BlockCache imple
We have done some casual benchmarking - loading tweets at a high rate of speed
while sending queries at a high rate of speed with 1 second NRT - essentially
the worst case NRT scenerio. By and large, performance has been similiar to
local filesystem performance. We will likely share some numbers when we have
some less casual results. You do of course have to warm up the block cache
before it really kicks in.
In terms of impl, as I mentioned, the orig HdfsDirectory comes from the Blur
guys - we tried not to change it too much currently - not until we figure out
if we might evolve it with them in the future - eg as a Lucene module or
something.
> Add support to write and read Solr index files and transaction log files to
> and from HDFS.
> ------------------------------------------------------------------------------------------
>
> Key: SOLR-4916
> URL: https://issues.apache.org/jira/browse/SOLR-4916
> Project: Solr
> Issue Type: New Feature
> Reporter: Mark Miller
> Assignee: Mark Miller
> Attachments: SOLR-4916.patch
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
