Hello Joerg ,
I was still thinking how well will this handle cases where i have like 10
Million to insert in the translog and i ask ES to index them all in a
single flush.
Is a heap dump likely to happen.
Thanks
Vineeth
On Mon, Feb 24, 2014 at 1:08 AM, vineeth mohan <[email protected]>wrote:
> Hello Joerg ,
>
> So if i disable it , ES wont write the feeds to lucene until i make a
> manual flush...
> I believe translog is written to a file and its not resident in the memory.
> This also means that translogs are maintained between restarts and we will
> never loose data.
>
> If all the above are right , then this might be a good candidate for my
> purpose.
>
> Thanks
> Vineeth
>
>
> On Mon, Feb 24, 2014 at 12:54 AM, [email protected] <
> [email protected]> wrote:
>
>> Oops, the correct parameter is index.translog.disable_flush : true
>>
>> index.gateway.local.flush: -1 is controlling the gateway.
>>
>> Jörg
>>
>>
>> On Sun, Feb 23, 2014 at 8:21 PM, [email protected] <
>> [email protected]> wrote:
>>
>>> Yes, it is possible to disable the translog sync (the component where
>>> the operations are passed from ES to Lucene) with
>>> index.gateway.local.flush: -1 and use the flush action for "manual commit"
>>> instead.
>>>
>>> I have never done that practically, though.
>>>
>>> Jörg
>>>
>>>
>>>
>>> On Sun, Feb 23, 2014 at 5:42 PM, vineeth mohan <
>>> [email protected]> wrote:
>>>
>>>> Hello Michael - Thanks for the configuration.
>>>>
>>>> Hello Jörg - I was thinking more in lines of translog -
>>>> http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/index-modules-translog.html
>>>>
>>>> I believe the index operation is first written to translog ( Which i am
>>>> not sure if is a part of lucene ) and then written to lucene later.
>>>> Here if we can ask ES , to accumulate a huge amount of feeds to index
>>>> and index it later , will that do the trick ?
>>>>
>>>> Thanks
>>>> Vineeth
>>>>
>>>>
>>>> On Sun, Feb 23, 2014 at 7:03 PM, Michael Sick <
>>>> [email protected]> wrote:
>>>>
>>>>> Also, if there are no other clients wanting a faster refresh, you can
>>>>> set index.refresh_interval to a higher value than the 1s default either in
>>>>> general for your index or just during the times when you're doing your
>>>>> bulk
>>>>> updates.
>>>>> http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/index-modules.html
>>>>>
>>>>>
>>>>> On Sun, Feb 23, 2014 at 8:28 AM, [email protected] <
>>>>> [email protected]> wrote:
>>>>>
>>>>>> Best method to achieve this would be to implement this in front of ES
>>>>>> so the bulk indexing client runs only at the time it should run.
>>>>>>
>>>>>> For the gathering plugin which I am working on, I plan to separate
>>>>>> the two phases of gathering documents and indexing documents. So, by
>>>>>> giving
>>>>>> a scheduling option, it will be possible to index (or even reindex)
>>>>>> gathered documents at a later time, for example, documents are
>>>>>> continuously
>>>>>> collected from various sources, like JDBC, web, or file system, and then
>>>>>> indexed at some later time (for example at night). Such collected
>>>>>> documents
>>>>>> will be stored in an archive format at each gatherer node, like the
>>>>>> archive
>>>>>> formats supported in the knapsack plugin.
>>>>>>
>>>>>> Jörg
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Sun, Feb 23, 2014 at 6:52 AM, vineeth mohan <
>>>>>> [email protected]> wrote:
>>>>>>
>>>>>>> Hi ,
>>>>>>>
>>>>>>> I am doing a lot of bulk insert into Elasticsearch and at the same
>>>>>>> time doing lots of read in another index.
>>>>>>>
>>>>>>> Because of the bulk insert my searches on other index are slow.
>>>>>>>
>>>>>>> It is not very urgent that these bulk indexes actually gets indexed
>>>>>>> and are immediately searchable.
>>>>>>>
>>>>>>> Is there anyway , I can ask Elasticsearch to receive the bulk
>>>>>>> inserts but do the actual indexing ( Which should be the CPU consuming
>>>>>>> part
>>>>>>> ) later.
>>>>>>>
>>>>>>> I figured out that Elasticsearch would wait for 1 second before
>>>>>>> making the documents searchable.
>>>>>>> Here , what is it waiting for ? Is it to index the document or
>>>>>>> reopening the indexWriter ?
>>>>>>> Will it help me if i can configure this 1 second to 1 hour ?
>>>>>>> If so , which parameter should i tweak.
>>>>>>>
>>>>>>> Kindly let me know if there are any other similar features out there
>>>>>>> which can be of any help.
>>>>>>>
>>>>>>> Thanks
>>>>>>> Vineeth
>>>>>>>
>>>>>>> --
>>>>>>> You received this message because you are subscribed to the Google
>>>>>>> Groups "elasticsearch" group.
>>>>>>> To unsubscribe from this group and stop receiving emails from it,
>>>>>>> send an email to [email protected].
>>>>>>> To view this discussion on the web visit
>>>>>>> https://groups.google.com/d/msgid/elasticsearch/CAGdPd5kwxwB%2Bi%3DHZDS1y%2B6Ad-VTax8hLSpgSVaSNH7CbzagB3Q%40mail.gmail.com
>>>>>>> .
>>>>>>> For more options, visit https://groups.google.com/groups/opt_out.
>>>>>>>
>>>>>>
>>>>>> --
>>>>>> You received this message because you are subscribed to the Google
>>>>>> Groups "elasticsearch" group.
>>>>>> To unsubscribe from this group and stop receiving emails from it,
>>>>>> send an email to [email protected].
>>>>>> To view this discussion on the web visit
>>>>>> https://groups.google.com/d/msgid/elasticsearch/CAKdsXoFFewVHjeoEyZVktYEEqtbBXoD4VH3K-Tx9KAh%3DTfj%3D1Q%40mail.gmail.com
>>>>>> .
>>>>>>
>>>>>> For more options, visit https://groups.google.com/groups/opt_out.
>>>>>>
>>>>>
>>>>> --
>>>>> You received this message because you are subscribed to the Google
>>>>> Groups "elasticsearch" group.
>>>>> To unsubscribe from this group and stop receiving emails from it, send
>>>>> an email to [email protected].
>>>>> To view this discussion on the web visit
>>>>> https://groups.google.com/d/msgid/elasticsearch/CAP8axnCq-PE%3Du0ZSC6d7rDxME%3DpkzpBo%3D9-tq_rT%2BCZjQgzFxg%40mail.gmail.com
>>>>> .
>>>>>
>>>>> For more options, visit https://groups.google.com/groups/opt_out.
>>>>>
>>>>
>>>> --
>>>> You received this message because you are subscribed to the Google
>>>> Groups "elasticsearch" group.
>>>> To unsubscribe from this group and stop receiving emails from it, send
>>>> an email to [email protected].
>>>> To view this discussion on the web visit
>>>> https://groups.google.com/d/msgid/elasticsearch/CAGdPd5%3DMvTYj0amH46nkm%3DkAEZ6HS2yaAYX5fadS7vaY6cmRvw%40mail.gmail.com
>>>> .
>>>>
>>>> For more options, visit https://groups.google.com/groups/opt_out.
>>>>
>>>
>>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "elasticsearch" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to [email protected].
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/elasticsearch/CAKdsXoHtevuu0%2BdrH1x%2B1JfPbG0brZYm503Fe96qNRjoeg8uSg%40mail.gmail.com
>> .
>>
>> For more options, visit https://groups.google.com/groups/opt_out.
>>
>
>
--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAGdPd5mTEsy9vEk8eifZnVxnn%3D%3DCJ%2BZr_RRtR696uBQq_C0_7w%40mail.gmail.com.
For more options, visit https://groups.google.com/groups/opt_out.
