[
https://issues.apache.org/jira/browse/BLUR-290?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13831127#comment-13831127
]
Aaron McCurry commented on BLUR-290:
------------------------------------
- I was just trying a proof-of-concept stuff. There are many nocommits and
loose ends, so haven't really gone to the perf aspect. But if you can eval,
will be great. I have fixed some bugs and attached few test-cases, in case you
want to try it.
I will try to give it a go this week.
- +1 for introducing table-types. A less adventurous table-type can even write
unsorted segments to disk and take a hit during searches, while doing the
balancing act during merge.
I think that it will be a property that we read from the table properties on
open. That way if someone else comes up with their own they don't have to
modify anymore code than necessary.
- It would be great if we can detect that NRT re-opens are going to suffer
under load and hence adjust accordingly. Shouldn't it be a policy that is
applicable to all table-types instead of it being a table-type itself?
Hmm, not sure. I have started creating a simple blur index that will not show
any uncommitted data. Thus the visibility of the data is going to be very
slow. I have also starting working on an alternate version that allows for the
inbound data to be written to a hdfs backed queue for indexing. That way the
indexer can take larger and larger chunks of data if it gets behind the inbound
data. So perhaps we could combine all of these? Maybe not. Not sure at this
point. I did notice that I will need to modify the BlurIndex api to allow for
partial Row updates (Record appending or something).
- PS: Sorry for the huge number of files. Is there place to check-in, may be an
experimental-branch?
At this point there's no where I can give you access. We should probably have
infrastructure mirror apache git to github. That way it would be easier for
you to branch and post pull requests etc.
Aaron
> NRT Updates using RAMDirectory & Swap
> -------------------------------------
>
> Key: BLUR-290
> URL: https://issues.apache.org/jira/browse/BLUR-290
> Project: Apache Blur
> Issue Type: New Feature
> Affects Versions: experimental-dev
> Reporter: Ravikumar
> Attachments: BlurFieldsConsumer.java, BlurFieldsConsumer.java,
> BlurFieldsConsumer.java, BlurFlushingIndexWriter.java, BlurIndexTracker.java,
> BlurPostingsConsumer.java, BlurPostingsConsumer.java,
> BlurPostingsFormat.java, BlurPostingsFormat.java, BlurRealTimeIndex.java,
> BlurRealTimeIndex.java, BlurRealTimeIndexTest.java,
> BlurRealTimeIndexWriter.java, BlurRealTimeManager.java,
> BlurRealTimeManagerReopenThread.java, BlurRowCodec.java, BlurRowCodec.java,
> BlurSegmentInfoFormat.java, BlurSegmentInfoWriter.java,
> BlurTermsConsumer.java, BlurTermsConsumer.java,
> CompressingRowIndexReader.java, CompressingRowIndexWriter.java,
> CompressingRowReader.java, CompressingRowReader.java,
> CompressingRowReader.java, CompressingRowWriter.java,
> CompressingRowWriter.java, CompressingRowWriter.java,
> GrowableByteArrayDataOutput.java, PrimeDocCache.java,
> RealTimeTransactionRecorder.java, RealTimeTransactionRecorder.java,
> RowCache.java, RowDocsCollector.java, RowDocsCollector.java,
> RowReaderCache.java, RowReaderCache.java, SlabAllocator.java,
> SlabRAMDirectory.java, SlabRAMFile.java, SlabRAMInputStream.java,
> SlabRAMOutputStream.java, SortingMultiReader.java, SortingMultiReader.java,
> TestCompressingRowWriter.java, TestCompressingRowWriter.java
>
>
> We have been discussing about handling humungous rows in Blur (BLUR-220).
> Explore the idea of using RAMDirectory at the front, backed by
> persistent-index.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
