Re: Welcome Julie Tibshirani as Lucene/Solr committer

[Tomás Fernández Löbbe]

Welcome Julie!

On Wed, Nov 18, 2020 at 6:59 PM Ilan Ginzburg  wrote:

> Welcome Julie and congrats!
>
> On Thu, Nov 19, 2020 at 3:51 AM Julie Tibshirani 
> wrote:
>
>> Thank you for the warm welcome! It’s a big honor for me -- I’ve been a
>> Lucene fan since the start of my software career. I’m excited to contribute
>> to such a great project.
>>
>> I’m a developer at Elastic focused on core search features. My
>> professional background is in information retrieval and data systems. I
>> also have an interest in statistical computing and machine learning
>> software. I’m originally from Canada but have lived in the SF Bay Area for
>> many years now. Some of my favorite things…
>> * Color: purple
>> * Album: Siamese Dream
>> * Java keyword: final
>>
>> Julie
>>
>> On Wed, Nov 18, 2020 at 6:33 PM Ishan Chattopadhyaya <
>> ichattopadhy...@gmail.com> wrote:
>>
>>> Welcome Julie!
>>>
>>> On Thu, 19 Nov, 2020, 12:10 am Erick Erickson, 
>>> wrote:
>>>
 Welcome Julie!

 > On Nov 18, 2020, at 1:21 PM, Alexandre Rafalovitch <
 arafa...@gmail.com> wrote:
 >
 > Juliet from the house of Elasticsearch meets a interesting,
 relevancy-aware  committer from the house of Solr.
 >
 > Such a romantic beginning. Not sure I want to know the end of that
 heroine's journey.
 >
 > :-)
 >
 > On Wed., Nov. 18, 2020, 12:59 p.m. Dawid Weiss, <
 dawid.we...@gmail.com> wrote:
 >
 > Congratulations and welcome, Julie.
 >
 > I think juliet is not a bad nick at all, you just need to who -all |
 grep "romeo"... :)
 >
 > Dawid
 >
 > On Wed, Nov 18, 2020 at 4:08 PM Michael Sokolov 
 wrote:
 > I'm pleased to announce that Julie Tibshirani has accepted the PMC's
 > invitation to become a committer.
 >
 > Julie, the tradition is that new committers introduce themselves with
 > a brief bio.
 >
 > I think we may still be sorting out the details of your Apache account
 > (julie@ may have been taken?), but as soon as that has been sorted
 out
 >  and karma has been granted, you can use your new powers to add
 > yourself to the committers section of the Who We Are page on the
 > website: 
 >
 > Congratulations and welcome!
 >
 > Mike Sokolov
 >
 > -
 > To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
 > For additional commands, e-mail: dev-h...@lucene.apache.org
 >


 -
 To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
 For additional commands, e-mail: dev-h...@lucene.apache.org

Re: Welcome Julie Tibshirani as Lucene/Solr committer

[Ilan Ginzburg]

Welcome Julie and congrats!

On Thu, Nov 19, 2020 at 3:51 AM Julie Tibshirani 
wrote:

> Thank you for the warm welcome! It’s a big honor for me -- I’ve been a
> Lucene fan since the start of my software career. I’m excited to contribute
> to such a great project.
>
> I’m a developer at Elastic focused on core search features. My
> professional background is in information retrieval and data systems. I
> also have an interest in statistical computing and machine learning
> software. I’m originally from Canada but have lived in the SF Bay Area for
> many years now. Some of my favorite things…
> * Color: purple
> * Album: Siamese Dream
> * Java keyword: final
>
> Julie
>
> On Wed, Nov 18, 2020 at 6:33 PM Ishan Chattopadhyaya <
> ichattopadhy...@gmail.com> wrote:
>
>> Welcome Julie!
>>
>> On Thu, 19 Nov, 2020, 12:10 am Erick Erickson, 
>> wrote:
>>
>>> Welcome Julie!
>>>
>>> > On Nov 18, 2020, at 1:21 PM, Alexandre Rafalovitch 
>>> wrote:
>>> >
>>> > Juliet from the house of Elasticsearch meets a interesting,
>>> relevancy-aware  committer from the house of Solr.
>>> >
>>> > Such a romantic beginning. Not sure I want to know the end of that
>>> heroine's journey.
>>> >
>>> > :-)
>>> >
>>> > On Wed., Nov. 18, 2020, 12:59 p.m. Dawid Weiss, 
>>> wrote:
>>> >
>>> > Congratulations and welcome, Julie.
>>> >
>>> > I think juliet is not a bad nick at all, you just need to who -all |
>>> grep "romeo"... :)
>>> >
>>> > Dawid
>>> >
>>> > On Wed, Nov 18, 2020 at 4:08 PM Michael Sokolov 
>>> wrote:
>>> > I'm pleased to announce that Julie Tibshirani has accepted the PMC's
>>> > invitation to become a committer.
>>> >
>>> > Julie, the tradition is that new committers introduce themselves with
>>> > a brief bio.
>>> >
>>> > I think we may still be sorting out the details of your Apache account
>>> > (julie@ may have been taken?), but as soon as that has been sorted out
>>> >  and karma has been granted, you can use your new powers to add
>>> > yourself to the committers section of the Who We Are page on the
>>> > website: 
>>> >
>>> > Congratulations and welcome!
>>> >
>>> > Mike Sokolov
>>> >
>>> > -
>>> > To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
>>> > For additional commands, e-mail: dev-h...@lucene.apache.org
>>> >
>>>
>>>
>>> -
>>> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
>>> For additional commands, e-mail: dev-h...@lucene.apache.org
>>>
>>>

Re: Welcome Julie Tibshirani as Lucene/Solr committer

[Julie Tibshirani]

 Thank you for the warm welcome! It’s a big honor for me -- I’ve been a
Lucene fan since the start of my software career. I’m excited to contribute
to such a great project.

I’m a developer at Elastic focused on core search features. My professional
background is in information retrieval and data systems. I also have an
interest in statistical computing and machine learning software. I’m
originally from Canada but have lived in the SF Bay Area for many years
now. Some of my favorite things…
* Color: purple
* Album: Siamese Dream
* Java keyword: final

Julie

On Wed, Nov 18, 2020 at 6:33 PM Ishan Chattopadhyaya <
ichattopadhy...@gmail.com> wrote:

> Welcome Julie!
>
> On Thu, 19 Nov, 2020, 12:10 am Erick Erickson, 
> wrote:
>
>> Welcome Julie!
>>
>> > On Nov 18, 2020, at 1:21 PM, Alexandre Rafalovitch 
>> wrote:
>> >
>> > Juliet from the house of Elasticsearch meets a interesting,
>> relevancy-aware  committer from the house of Solr.
>> >
>> > Such a romantic beginning. Not sure I want to know the end of that
>> heroine's journey.
>> >
>> > :-)
>> >
>> > On Wed., Nov. 18, 2020, 12:59 p.m. Dawid Weiss, 
>> wrote:
>> >
>> > Congratulations and welcome, Julie.
>> >
>> > I think juliet is not a bad nick at all, you just need to who -all |
>> grep "romeo"... :)
>> >
>> > Dawid
>> >
>> > On Wed, Nov 18, 2020 at 4:08 PM Michael Sokolov 
>> wrote:
>> > I'm pleased to announce that Julie Tibshirani has accepted the PMC's
>> > invitation to become a committer.
>> >
>> > Julie, the tradition is that new committers introduce themselves with
>> > a brief bio.
>> >
>> > I think we may still be sorting out the details of your Apache account
>> > (julie@ may have been taken?), but as soon as that has been sorted out
>> >  and karma has been granted, you can use your new powers to add
>> > yourself to the committers section of the Who We Are page on the
>> > website: 
>> >
>> > Congratulations and welcome!
>> >
>> > Mike Sokolov
>> >
>> > -
>> > To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
>> > For additional commands, e-mail: dev-h...@lucene.apache.org
>> >
>>
>>
>> -
>> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
>> For additional commands, e-mail: dev-h...@lucene.apache.org
>>
>>

Re: Welcome Julie Tibshirani as Lucene/Solr committer

[Ishan Chattopadhyaya]

Welcome Julie!

On Thu, 19 Nov, 2020, 12:10 am Erick Erickson, 
wrote:

> Welcome Julie!
>
> > On Nov 18, 2020, at 1:21 PM, Alexandre Rafalovitch 
> wrote:
> >
> > Juliet from the house of Elasticsearch meets a interesting,
> relevancy-aware  committer from the house of Solr.
> >
> > Such a romantic beginning. Not sure I want to know the end of that
> heroine's journey.
> >
> > :-)
> >
> > On Wed., Nov. 18, 2020, 12:59 p.m. Dawid Weiss, 
> wrote:
> >
> > Congratulations and welcome, Julie.
> >
> > I think juliet is not a bad nick at all, you just need to who -all |
> grep "romeo"... :)
> >
> > Dawid
> >
> > On Wed, Nov 18, 2020 at 4:08 PM Michael Sokolov 
> wrote:
> > I'm pleased to announce that Julie Tibshirani has accepted the PMC's
> > invitation to become a committer.
> >
> > Julie, the tradition is that new committers introduce themselves with
> > a brief bio.
> >
> > I think we may still be sorting out the details of your Apache account
> > (julie@ may have been taken?), but as soon as that has been sorted out
> >  and karma has been granted, you can use your new powers to add
> > yourself to the committers section of the Who We Are page on the
> > website: 
> >
> > Congratulations and welcome!
> >
> > Mike Sokolov
> >
> > -
> > To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
> > For additional commands, e-mail: dev-h...@lucene.apache.org
> >
>
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
> For additional commands, e-mail: dev-h...@lucene.apache.org
>
>

Re: Possible resource leak in IndexWriter.deleteAll()/FieldNumbers.clear()

[Michael Froh]

Thanks David!

Created https://issues.apache.org/jira/browse/LUCENE-9617 and posted a PR:
https://github.com/apache/lucene-solr/pull/2088

On Wed, Nov 18, 2020 at 10:26 AM David Smiley  wrote:

> Thanks for sharing the background of your indexing serialization
> shenanigans :-) -- interesting.
>
> I think IndexWriter.deleteAll() should ultimately reset
> lowestUnassignedFieldNumber.  globalFieldNumberMap.clear() is only called
> by deleteAll, so this simple proposal makes sense to me.  File a JIRA issue.
>
> ~ David Smiley
> Apache Lucene/Solr Search Developer
> http://www.linkedin.com/in/davidwsmiley
>
>
> On Wed, Nov 18, 2020 at 1:17 PM Michael Froh  wrote:
>
>> I have some code that is kind of abusing IndexWriter.deleteAll(). In
>> short, I'm basically experimenting with using tiny (one block of joined
>> parent/child documents) indexes as a serialized format to index on one
>> fleet and then merge these tiny indexes on another fleet. I'm doing this by
>> indexing a block, committing, storing the contents of the index directory
>> in a zip file, invoking deleteAll(), and repeating. Believe it or not, the
>> performance is not terrible. (Currently getting about 20% of the throughput
>> I see with regular indexing.)
>>
>> Regardless of my serialization shenanigans above, I've found that
>> performance degrades over time for the process, as it spends more time
>> allocating and freeing memory. Analyzing some heap dumps, it's because
>> FieldInfos.byNumber is getting bigger and bigger. IndexWriter.deleteAll()
>> doesn't truly reset state. Specifically, it calls
>> globalFieldNumberMap.clear(), which clears all of the FieldNumbers
>> collections, but it doesn't reset lowestUnassignedFieldNumber. So, that
>> number keeps counting up, and new instances of FieldInfos allocate larger
>> and larger arrays (and only use the top indices).
>>
>> Has anyone else encountered this? Can I open an issue for resetting
>> lowestUnassignedFieldNumber in FieldNumbers.clear()? Is there any risk in
>> doing so?
>>
>> (For my specific use-case, I would be okay with not clearing
>> globalFieldNumberMap at all, since the set of fields is bounded, but
>> assigning new field numbers is probably among the least of my costs.)
>>
>

Re: Possible resource leak in IndexWriter.deleteAll()/FieldNumbers.clear()

[Michael Sokolov]

Yeah that sounds as if it would be too expensive. I wasn't quite sure
what would be involved..

On Wed, Nov 18, 2020 at 3:56 PM Michael Froh  wrote:
>
> I didn't try creating a new IndexWriter for each batch, but I was assuming 
> that would be heavier, as it would allocate a new DocumentsWriter, and 
> through that new DocumentsWriterPerThreads. Skimming through the code for 
> DWPT, it looks like there are various pools involved in creating each DWPT's 
> instance of DefaultIndexingChain, which might be expensive to create 
> frequently, rather than reusing on flush().
>
> Also I was partly motivated by laziness. The production code I'm borrowing 
> for this prototype doesn't make it easy to recreate the IndexWriterConfig, 
> and IWC is not reusable across IndexWriter instances.
>
> On Wed, Nov 18, 2020 at 12:25 PM Michael Sokolov  wrote:
>>
>> I'm curious if you tried creating a new IndexWriter for each batch?
>>
>> On Wed, Nov 18, 2020 at 1:18 PM Michael Froh  wrote:
>> >
>> > I have some code that is kind of abusing IndexWriter.deleteAll(). In 
>> > short, I'm basically experimenting with using tiny (one block of joined 
>> > parent/child documents) indexes as a serialized format to index on one 
>> > fleet and then merge these tiny indexes on another fleet. I'm doing this 
>> > by indexing a block, committing, storing the contents of the index 
>> > directory in a zip file, invoking deleteAll(), and repeating. Believe it 
>> > or not, the performance is not terrible. (Currently getting about 20% of 
>> > the throughput I see with regular indexing.)
>> >
>> > Regardless of my serialization shenanigans above, I've found that 
>> > performance degrades over time for the process, as it spends more time 
>> > allocating and freeing memory. Analyzing some heap dumps, it's because 
>> > FieldInfos.byNumber is getting bigger and bigger. IndexWriter.deleteAll() 
>> > doesn't truly reset state. Specifically, it calls 
>> > globalFieldNumberMap.clear(), which clears all of the FieldNumbers 
>> > collections, but it doesn't reset lowestUnassignedFieldNumber. So, that 
>> > number keeps counting up, and new instances of FieldInfos allocate larger 
>> > and larger arrays (and only use the top indices).
>> >
>> > Has anyone else encountered this? Can I open an issue for resetting 
>> > lowestUnassignedFieldNumber in FieldNumbers.clear()? Is there any risk in 
>> > doing so?
>> >
>> > (For my specific use-case, I would be okay with not clearing 
>> > globalFieldNumberMap at all, since the set of fields is bounded, but 
>> > assigning new field numbers is probably among the least of my costs.)
>>
>> -
>> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
>> For additional commands, e-mail: dev-h...@lucene.apache.org
>>

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: Possible resource leak in IndexWriter.deleteAll()/FieldNumbers.clear()

[Michael Froh]

I didn't try creating a new IndexWriter for each batch, but I was assuming
that would be heavier, as it would allocate a new DocumentsWriter, and
through that new DocumentsWriterPerThreads. Skimming through the code for
DWPT, it looks like there are various pools involved in creating each
DWPT's instance of DefaultIndexingChain, which might be expensive to create
frequently, rather than reusing on flush().

Also I was partly motivated by laziness. The production code I'm borrowing
for this prototype doesn't make it easy to recreate the IndexWriterConfig,
and IWC is not reusable across IndexWriter instances.

On Wed, Nov 18, 2020 at 12:25 PM Michael Sokolov  wrote:

> I'm curious if you tried creating a new IndexWriter for each batch?
>
> On Wed, Nov 18, 2020 at 1:18 PM Michael Froh  wrote:
> >
> > I have some code that is kind of abusing IndexWriter.deleteAll(). In
> short, I'm basically experimenting with using tiny (one block of joined
> parent/child documents) indexes as a serialized format to index on one
> fleet and then merge these tiny indexes on another fleet. I'm doing this by
> indexing a block, committing, storing the contents of the index directory
> in a zip file, invoking deleteAll(), and repeating. Believe it or not, the
> performance is not terrible. (Currently getting about 20% of the throughput
> I see with regular indexing.)
> >
> > Regardless of my serialization shenanigans above, I've found that
> performance degrades over time for the process, as it spends more time
> allocating and freeing memory. Analyzing some heap dumps, it's because
> FieldInfos.byNumber is getting bigger and bigger. IndexWriter.deleteAll()
> doesn't truly reset state. Specifically, it calls
> globalFieldNumberMap.clear(), which clears all of the FieldNumbers
> collections, but it doesn't reset lowestUnassignedFieldNumber. So, that
> number keeps counting up, and new instances of FieldInfos allocate larger
> and larger arrays (and only use the top indices).
> >
> > Has anyone else encountered this? Can I open an issue for resetting
> lowestUnassignedFieldNumber in FieldNumbers.clear()? Is there any risk in
> doing so?
> >
> > (For my specific use-case, I would be okay with not clearing
> globalFieldNumberMap at all, since the set of fields is bounded, but
> assigning new field numbers is probably among the least of my costs.)
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
> For additional commands, e-mail: dev-h...@lucene.apache.org
>
>

Re: Possible resource leak in IndexWriter.deleteAll()/FieldNumbers.clear()

[Michael Sokolov]

I'm curious if you tried creating a new IndexWriter for each batch?

On Wed, Nov 18, 2020 at 1:18 PM Michael Froh  wrote:
>
> I have some code that is kind of abusing IndexWriter.deleteAll(). In short, 
> I'm basically experimenting with using tiny (one block of joined parent/child 
> documents) indexes as a serialized format to index on one fleet and then 
> merge these tiny indexes on another fleet. I'm doing this by indexing a 
> block, committing, storing the contents of the index directory in a zip file, 
> invoking deleteAll(), and repeating. Believe it or not, the performance is 
> not terrible. (Currently getting about 20% of the throughput I see with 
> regular indexing.)
>
> Regardless of my serialization shenanigans above, I've found that performance 
> degrades over time for the process, as it spends more time allocating and 
> freeing memory. Analyzing some heap dumps, it's because FieldInfos.byNumber 
> is getting bigger and bigger. IndexWriter.deleteAll() doesn't truly reset 
> state. Specifically, it calls globalFieldNumberMap.clear(), which clears all 
> of the FieldNumbers collections, but it doesn't reset 
> lowestUnassignedFieldNumber. So, that number keeps counting up, and new 
> instances of FieldInfos allocate larger and larger arrays (and only use the 
> top indices).
>
> Has anyone else encountered this? Can I open an issue for resetting 
> lowestUnassignedFieldNumber in FieldNumbers.clear()? Is there any risk in 
> doing so?
>
> (For my specific use-case, I would be okay with not clearing 
> globalFieldNumberMap at all, since the set of fields is bounded, but 
> assigning new field numbers is probably among the least of my costs.)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: Welcome Julie Tibshirani as Lucene/Solr committer

[Erick Erickson]

Welcome Julie!

> On Nov 18, 2020, at 1:21 PM, Alexandre Rafalovitch  wrote:
> 
> Juliet from the house of Elasticsearch meets a interesting, relevancy-aware  
> committer from the house of Solr.
> 
> Such a romantic beginning. Not sure I want to know the end of that heroine's 
> journey.
> 
> :-) 
> 
> On Wed., Nov. 18, 2020, 12:59 p.m. Dawid Weiss,  wrote:
> 
> Congratulations and welcome, Julie. 
> 
> I think juliet is not a bad nick at all, you just need to who -all | grep 
> "romeo"... :)
> 
> Dawid
> 
> On Wed, Nov 18, 2020 at 4:08 PM Michael Sokolov  wrote:
> I'm pleased to announce that Julie Tibshirani has accepted the PMC's
> invitation to become a committer.
> 
> Julie, the tradition is that new committers introduce themselves with
> a brief bio.
> 
> I think we may still be sorting out the details of your Apache account
> (julie@ may have been taken?), but as soon as that has been sorted out
>  and karma has been granted, you can use your new powers to add
> yourself to the committers section of the Who We Are page on the
> website: 
> 
> Congratulations and welcome!
> 
> Mike Sokolov
> 
> -
> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
> For additional commands, e-mail: dev-h...@lucene.apache.org
> 


-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: Possible resource leak in IndexWriter.deleteAll()/FieldNumbers.clear()

[David Smiley]

Thanks for sharing the background of your indexing serialization
shenanigans :-) -- interesting.

I think IndexWriter.deleteAll() should ultimately reset
lowestUnassignedFieldNumber.  globalFieldNumberMap.clear() is only called
by deleteAll, so this simple proposal makes sense to me.  File a JIRA issue.

~ David Smiley
Apache Lucene/Solr Search Developer
http://www.linkedin.com/in/davidwsmiley


On Wed, Nov 18, 2020 at 1:17 PM Michael Froh  wrote:

> I have some code that is kind of abusing IndexWriter.deleteAll(). In
> short, I'm basically experimenting with using tiny (one block of joined
> parent/child documents) indexes as a serialized format to index on one
> fleet and then merge these tiny indexes on another fleet. I'm doing this by
> indexing a block, committing, storing the contents of the index directory
> in a zip file, invoking deleteAll(), and repeating. Believe it or not, the
> performance is not terrible. (Currently getting about 20% of the throughput
> I see with regular indexing.)
>
> Regardless of my serialization shenanigans above, I've found that
> performance degrades over time for the process, as it spends more time
> allocating and freeing memory. Analyzing some heap dumps, it's because
> FieldInfos.byNumber is getting bigger and bigger. IndexWriter.deleteAll()
> doesn't truly reset state. Specifically, it calls
> globalFieldNumberMap.clear(), which clears all of the FieldNumbers
> collections, but it doesn't reset lowestUnassignedFieldNumber. So, that
> number keeps counting up, and new instances of FieldInfos allocate larger
> and larger arrays (and only use the top indices).
>
> Has anyone else encountered this? Can I open an issue for resetting
> lowestUnassignedFieldNumber in FieldNumbers.clear()? Is there any risk in
> doing so?
>
> (For my specific use-case, I would be okay with not clearing
> globalFieldNumberMap at all, since the set of fields is bounded, but
> assigning new field numbers is probably among the least of my costs.)
>

Re: Welcome Julie Tibshirani as Lucene/Solr committer

[Alexandre Rafalovitch]

Juliet from the house of Elasticsearch meets a interesting,
relevancy-aware  committer from the house of Solr.

Such a romantic beginning. Not sure I want to know the end of that
heroine's journey.

:-)

On Wed., Nov. 18, 2020, 12:59 p.m. Dawid Weiss, 
wrote:

>
> Congratulations and welcome, Julie.
>
> I think juliet is not a bad nick at all, you just need to who -all | grep
> "romeo"... :)
>
> Dawid
>
> On Wed, Nov 18, 2020 at 4:08 PM Michael Sokolov 
> wrote:
>
>> I'm pleased to announce that Julie Tibshirani has accepted the PMC's
>> invitation to become a committer.
>>
>> Julie, the tradition is that new committers introduce themselves with
>> a brief bio.
>>
>> I think we may still be sorting out the details of your Apache account
>> (julie@ may have been taken?), but as soon as that has been sorted out
>>  and karma has been granted, you can use your new powers to add
>> yourself to the committers section of the Who We Are page on the
>> website: 
>>
>> Congratulations and welcome!
>>
>> Mike Sokolov
>>
>> -
>> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
>> For additional commands, e-mail: dev-h...@lucene.apache.org
>>
>>

Possible resource leak in IndexWriter.deleteAll()/FieldNumbers.clear()

[Michael Froh]

I have some code that is kind of abusing IndexWriter.deleteAll(). In short,
I'm basically experimenting with using tiny (one block of joined
parent/child documents) indexes as a serialized format to index on one
fleet and then merge these tiny indexes on another fleet. I'm doing this by
indexing a block, committing, storing the contents of the index directory
in a zip file, invoking deleteAll(), and repeating. Believe it or not, the
performance is not terrible. (Currently getting about 20% of the throughput
I see with regular indexing.)

Regardless of my serialization shenanigans above, I've found that
performance degrades over time for the process, as it spends more time
allocating and freeing memory. Analyzing some heap dumps, it's because
FieldInfos.byNumber is getting bigger and bigger. IndexWriter.deleteAll()
doesn't truly reset state. Specifically, it calls
globalFieldNumberMap.clear(), which clears all of the FieldNumbers
collections, but it doesn't reset lowestUnassignedFieldNumber. So, that
number keeps counting up, and new instances of FieldInfos allocate larger
and larger arrays (and only use the top indices).

Has anyone else encountered this? Can I open an issue for resetting
lowestUnassignedFieldNumber in FieldNumbers.clear()? Is there any risk in
doing so?

(For my specific use-case, I would be okay with not clearing
globalFieldNumberMap at all, since the set of fields is bounded, but
assigning new field numbers is probably among the least of my costs.)

Re: Welcome Julie Tibshirani as Lucene/Solr committer

[Yonik Seeley]

Congrats Julie!
-Yonik


On Wed, Nov 18, 2020 at 10:07 AM Michael Sokolov  wrote:

> I'm pleased to announce that Julie Tibshirani has accepted the PMC's
> invitation to become a committer.
>
> Julie, the tradition is that new committers introduce themselves with
> a brief bio.
>
> I think we may still be sorting out the details of your Apache account
> (julie@ may have been taken?), but as soon as that has been sorted out
>  and karma has been granted, you can use your new powers to add
> yourself to the committers section of the Who We Are page on the
> website: 
>
> Congratulations and welcome!
>
> Mike Sokolov
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
> For additional commands, e-mail: dev-h...@lucene.apache.org
>
>

Re: Welcome Julie Tibshirani as Lucene/Solr committer

[Dawid Weiss]

Congratulations and welcome, Julie.

I think juliet is not a bad nick at all, you just need to who -all | grep
"romeo"... :)

Dawid

On Wed, Nov 18, 2020 at 4:08 PM Michael Sokolov  wrote:

> I'm pleased to announce that Julie Tibshirani has accepted the PMC's
> invitation to become a committer.
>
> Julie, the tradition is that new committers introduce themselves with
> a brief bio.
>
> I think we may still be sorting out the details of your Apache account
> (julie@ may have been taken?), but as soon as that has been sorted out
>  and karma has been granted, you can use your new powers to add
> yourself to the committers section of the Who We Are page on the
> website: 
>
> Congratulations and welcome!
>
> Mike Sokolov
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
> For additional commands, e-mail: dev-h...@lucene.apache.org
>
>

Re: Welcome Julie Tibshirani as Lucene/Solr committer

[David Smiley]

Congratulations Julie!

~ David Smiley
Apache Lucene/Solr Search Developer
http://www.linkedin.com/in/davidwsmiley


On Wed, Nov 18, 2020 at 10:08 AM Michael Sokolov  wrote:

> I'm pleased to announce that Julie Tibshirani has accepted the PMC's
> invitation to become a committer.
>
> Julie, the tradition is that new committers introduce themselves with
> a brief bio.
>
> I think we may still be sorting out the details of your Apache account
> (julie@ may have been taken?), but as soon as that has been sorted out
>  and karma has been granted, you can use your new powers to add
> yourself to the committers section of the Who We Are page on the
> website: 
>
> Congratulations and welcome!
>
> Mike Sokolov
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
> For additional commands, e-mail: dev-h...@lucene.apache.org
>
>

Re: Welcome Julie Tibshirani as Lucene/Solr committer

[Anshum Gupta]

Congratulations and welcome, Julie! :)

On Wed, Nov 18, 2020 at 7:07 AM Michael Sokolov  wrote:

> I'm pleased to announce that Julie Tibshirani has accepted the PMC's
> invitation to become a committer.
>
> Julie, the tradition is that new committers introduce themselves with
> a brief bio.
>
> I think we may still be sorting out the details of your Apache account
> (julie@ may have been taken?), but as soon as that has been sorted out
>  and karma has been granted, you can use your new powers to add
> yourself to the committers section of the Who We Are page on the
> website: 
>
> Congratulations and welcome!
>
> Mike Sokolov
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
> For additional commands, e-mail: dev-h...@lucene.apache.org
>
>

-- 
Anshum Gupta

Re: Welcome Julie Tibshirani as Lucene/Solr committer

[Nhat Nguyen]

Congrats and welcome Julie!

On Wed, Nov 18, 2020 at 11:32 AM Mike Drob  wrote:

> Congratulations and welcome, Julie!
>
> On Wed, Nov 18, 2020 at 8:29 AM Christian Moen  wrote:
>
>> Congrats, Julie.
>>
>> On Thu, Nov 19, 2020 at 0:07 Michael Sokolov  wrote:
>>
>>> I'm pleased to announce that Julie Tibshirani has accepted the PMC's
>>> invitation to become a committer.
>>>
>>> Julie, the tradition is that new committers introduce themselves with
>>> a brief bio.
>>>
>>> I think we may still be sorting out the details of your Apache account
>>> (julie@ may have been taken?), but as soon as that has been sorted out
>>>  and karma has been granted, you can use your new powers to add
>>> yourself to the committers section of the Who We Are page on the
>>> website: 
>>>
>>> Congratulations and welcome!
>>>
>>> Mike Sokolov
>>>
>>> -
>>> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
>>> For additional commands, e-mail: dev-h...@lucene.apache.org
>>>
>>>

Re: Welcome Julie Tibshirani as Lucene/Solr committer

[Mike Drob]

Congratulations and welcome, Julie!

On Wed, Nov 18, 2020 at 8:29 AM Christian Moen  wrote:

> Congrats, Julie.
>
> On Thu, Nov 19, 2020 at 0:07 Michael Sokolov  wrote:
>
>> I'm pleased to announce that Julie Tibshirani has accepted the PMC's
>> invitation to become a committer.
>>
>> Julie, the tradition is that new committers introduce themselves with
>> a brief bio.
>>
>> I think we may still be sorting out the details of your Apache account
>> (julie@ may have been taken?), but as soon as that has been sorted out
>>  and karma has been granted, you can use your new powers to add
>> yourself to the committers section of the Who We Are page on the
>> website: 
>>
>> Congratulations and welcome!
>>
>> Mike Sokolov
>>
>> -
>> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
>> For additional commands, e-mail: dev-h...@lucene.apache.org
>>
>>

Re: Welcome Julie Tibshirani as Lucene/Solr committer

[Christian Moen]

Congrats, Julie.

On Thu, Nov 19, 2020 at 0:07 Michael Sokolov  wrote:

> I'm pleased to announce that Julie Tibshirani has accepted the PMC's
> invitation to become a committer.
>
> Julie, the tradition is that new committers introduce themselves with
> a brief bio.
>
> I think we may still be sorting out the details of your Apache account
> (julie@ may have been taken?), but as soon as that has been sorted out
>  and karma has been granted, you can use your new powers to add
> yourself to the committers section of the Who We Are page on the
> website: 
>
> Congratulations and welcome!
>
> Mike Sokolov
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
> For additional commands, e-mail: dev-h...@lucene.apache.org
>
>

Re: Welcome Julie Tibshirani as Lucene/Solr committer

[Steve Rowe]

Congrats and welcome, Julie!

--
Steve

> On Nov 18, 2020, at 10:06 AM, Michael Sokolov  wrote:
> 
> I'm pleased to announce that Julie Tibshirani has accepted the PMC's
> invitation to become a committer.
> 
> Julie, the tradition is that new committers introduce themselves with
> a brief bio.
> 
> I think we may still be sorting out the details of your Apache account
> (julie@ may have been taken?), but as soon as that has been sorted out
> and karma has been granted, you can use your new powers to add
> yourself to the committers section of the Who We Are page on the
> website: 
> 
> Congratulations and welcome!
> 
> Mike Sokolov
> 
> -
> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
> For additional commands, e-mail: dev-h...@lucene.apache.org
> 


-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: Welcome Julie Tibshirani as Lucene/Solr committer

[Alexandre Rafalovitch]

Congratulations Julie and welcome,

I guess you will have to do a follow-up post to your "Finding a home"
entry now :-) 
https://www.elastic.co/blog/culture-finding-a-home-and-career-in-the-open-source-community

Regards,
   Alex.

On Wed, 18 Nov 2020 at 10:07, Michael Sokolov  wrote:
>
> I'm pleased to announce that Julie Tibshirani has accepted the PMC's
> invitation to become a committer.
>
> Julie, the tradition is that new committers introduce themselves with
> a brief bio.
>
> I think we may still be sorting out the details of your Apache account
> (julie@ may have been taken?), but as soon as that has been sorted out
>  and karma has been granted, you can use your new powers to add
> yourself to the committers section of the Who We Are page on the
> website: 
>
> Congratulations and welcome!
>
> Mike Sokolov
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
> For additional commands, e-mail: dev-h...@lucene.apache.org
>

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: Welcome Julie Tibshirani as Lucene/Solr committer

[Michael McCandless]

Welcome Julie!

Mike McCandless

http://blog.mikemccandless.com


On Wed, Nov 18, 2020 at 11:04 AM Gus Heck  wrote:

> Congratulations and welcome :)
>
> On Wed, Nov 18, 2020 at 10:56 AM Houston Putman 
> wrote:
>
>> Congrats and welcome Julie!!
>>
>> - Houston
>>
>> On Wed, Nov 18, 2020 at 10:30 AM Eric Pugh <
>> ep...@opensourceconnections.com> wrote:
>>
>>> I’ve seen all your contributions, really great stuff. Welcome!
>>>
>>>
>>> On Nov 18, 2020, at 10:22 AM, Uwe Schindler  wrote:
>>>
>>> Welcome Julie!
>>>
>>> -
>>> Uwe Schindler
>>> Achterdiek 19, D-28357 Bremen
>>> https://www.thetaphi.de
>>> eMail: u...@thetaphi.de
>>>
>>> -Original Message-
>>> From: Michael Sokolov 
>>> Sent: Wednesday, November 18, 2020 4:07 PM
>>> To: dev@lucene.apache.org
>>> Subject: Welcome Julie Tibshirani as Lucene/Solr committer
>>>
>>> I'm pleased to announce that Julie Tibshirani has accepted the PMC's
>>> invitation to become a committer.
>>>
>>> Julie, the tradition is that new committers introduce themselves with
>>> a brief bio.
>>>
>>> I think we may still be sorting out the details of your Apache account
>>> (julie@ may have been taken?), but as soon as that has been sorted out
>>> and karma has been granted, you can use your new powers to add
>>> yourself to the committers section of the Who We Are page on the
>>> website: 
>>>
>>> Congratulations and welcome!
>>>
>>> Mike Sokolov
>>>
>>> -
>>> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
>>> For additional commands, e-mail: dev-h...@lucene.apache.org
>>>
>>>
>>>
>>> -
>>> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
>>> For additional commands, e-mail: dev-h...@lucene.apache.org
>>>
>>>
>>> ___
>>> *Eric Pugh **| *Founder & CEO | OpenSource Connections, LLC | 434.466.1467
>>> | http://www.opensourceconnections.com | My Free/Busy
>>> 
>>> Co-Author: Apache Solr Enterprise Search Server, 3rd Ed
>>> 
>>> This e-mail and all contents, including attachments, is considered to be
>>> Company Confidential unless explicitly stated otherwise, regardless
>>> of whether attachments are marked as such.
>>>
>>>
>
> --
> http://www.needhamsoftware.com (work)
> http://www.the111shift.com (play)
>

Re: Welcome Julie Tibshirani as Lucene/Solr committer

[Gus Heck]

Congratulations and welcome :)

On Wed, Nov 18, 2020 at 10:56 AM Houston Putman 
wrote:

> Congrats and welcome Julie!!
>
> - Houston
>
> On Wed, Nov 18, 2020 at 10:30 AM Eric Pugh <
> ep...@opensourceconnections.com> wrote:
>
>> I’ve seen all your contributions, really great stuff. Welcome!
>>
>>
>> On Nov 18, 2020, at 10:22 AM, Uwe Schindler  wrote:
>>
>> Welcome Julie!
>>
>> -
>> Uwe Schindler
>> Achterdiek 19, D-28357 Bremen
>> https://www.thetaphi.de
>> eMail: u...@thetaphi.de
>>
>> -Original Message-
>> From: Michael Sokolov 
>> Sent: Wednesday, November 18, 2020 4:07 PM
>> To: dev@lucene.apache.org
>> Subject: Welcome Julie Tibshirani as Lucene/Solr committer
>>
>> I'm pleased to announce that Julie Tibshirani has accepted the PMC's
>> invitation to become a committer.
>>
>> Julie, the tradition is that new committers introduce themselves with
>> a brief bio.
>>
>> I think we may still be sorting out the details of your Apache account
>> (julie@ may have been taken?), but as soon as that has been sorted out
>> and karma has been granted, you can use your new powers to add
>> yourself to the committers section of the Who We Are page on the
>> website: 
>>
>> Congratulations and welcome!
>>
>> Mike Sokolov
>>
>> -
>> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
>> For additional commands, e-mail: dev-h...@lucene.apache.org
>>
>>
>>
>> -
>> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
>> For additional commands, e-mail: dev-h...@lucene.apache.org
>>
>>
>> ___
>> *Eric Pugh **| *Founder & CEO | OpenSource Connections, LLC | 434.466.1467
>> | http://www.opensourceconnections.com | My Free/Busy
>> 
>> Co-Author: Apache Solr Enterprise Search Server, 3rd Ed
>> 
>> This e-mail and all contents, including attachments, is considered to be
>> Company Confidential unless explicitly stated otherwise, regardless
>> of whether attachments are marked as such.
>>
>>

-- 
http://www.needhamsoftware.com (work)
http://www.the111shift.com (play)

Re: Welcome Julie Tibshirani as Lucene/Solr committer

[Houston Putman]

Congrats and welcome Julie!!

- Houston

On Wed, Nov 18, 2020 at 10:30 AM Eric Pugh 
wrote:

> I’ve seen all your contributions, really great stuff. Welcome!
>
>
> On Nov 18, 2020, at 10:22 AM, Uwe Schindler  wrote:
>
> Welcome Julie!
>
> -
> Uwe Schindler
> Achterdiek 19, D-28357 Bremen
> https://www.thetaphi.de
> eMail: u...@thetaphi.de
>
> -Original Message-
> From: Michael Sokolov 
> Sent: Wednesday, November 18, 2020 4:07 PM
> To: dev@lucene.apache.org
> Subject: Welcome Julie Tibshirani as Lucene/Solr committer
>
> I'm pleased to announce that Julie Tibshirani has accepted the PMC's
> invitation to become a committer.
>
> Julie, the tradition is that new committers introduce themselves with
> a brief bio.
>
> I think we may still be sorting out the details of your Apache account
> (julie@ may have been taken?), but as soon as that has been sorted out
> and karma has been granted, you can use your new powers to add
> yourself to the committers section of the Who We Are page on the
> website: 
>
> Congratulations and welcome!
>
> Mike Sokolov
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
> For additional commands, e-mail: dev-h...@lucene.apache.org
>
>
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
> For additional commands, e-mail: dev-h...@lucene.apache.org
>
>
> ___
> *Eric Pugh **| *Founder & CEO | OpenSource Connections, LLC | 434.466.1467
> | http://www.opensourceconnections.com | My Free/Busy
> 
> Co-Author: Apache Solr Enterprise Search Server, 3rd Ed
> 
> This e-mail and all contents, including attachments, is considered to be
> Company Confidential unless explicitly stated otherwise, regardless
> of whether attachments are marked as such.
>
>

Re: Welcome Julie Tibshirani as Lucene/Solr committer

[Eric Pugh]

I’ve seen all your contributions, really great stuff. Welcome!


> On Nov 18, 2020, at 10:22 AM, Uwe Schindler  wrote:
> 
> Welcome Julie!
> 
> -
> Uwe Schindler
> Achterdiek 19, D-28357 Bremen
> https://www.thetaphi.de
> eMail: u...@thetaphi.de
> 
>> -Original Message-
>> From: Michael Sokolov 
>> Sent: Wednesday, November 18, 2020 4:07 PM
>> To: dev@lucene.apache.org
>> Subject: Welcome Julie Tibshirani as Lucene/Solr committer
>> 
>> I'm pleased to announce that Julie Tibshirani has accepted the PMC's
>> invitation to become a committer.
>> 
>> Julie, the tradition is that new committers introduce themselves with
>> a brief bio.
>> 
>> I think we may still be sorting out the details of your Apache account
>> (julie@ may have been taken?), but as soon as that has been sorted out
>> and karma has been granted, you can use your new powers to add
>> yourself to the committers section of the Who We Are page on the
>> website: 
>> 
>> Congratulations and welcome!
>> 
>> Mike Sokolov
>> 
>> -
>> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
>> For additional commands, e-mail: dev-h...@lucene.apache.org
> 
> 
> -
> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
> For additional commands, e-mail: dev-h...@lucene.apache.org
> 

___
Eric Pugh | Founder & CEO | OpenSource Connections, LLC | 434.466.1467 | 
http://www.opensourceconnections.com  | 
My Free/Busy   
Co-Author: Apache Solr Enterprise Search Server, 3rd Ed 


This e-mail and all contents, including attachments, is considered to be 
Company Confidential unless explicitly stated otherwise, regardless of whether 
attachments are marked as such.

RE: Welcome Julie Tibshirani as Lucene/Solr committer

[Uwe Schindler]

Welcome Julie!

-
Uwe Schindler
Achterdiek 19, D-28357 Bremen
https://www.thetaphi.de
eMail: u...@thetaphi.de

> -Original Message-
> From: Michael Sokolov 
> Sent: Wednesday, November 18, 2020 4:07 PM
> To: dev@lucene.apache.org
> Subject: Welcome Julie Tibshirani as Lucene/Solr committer
> 
> I'm pleased to announce that Julie Tibshirani has accepted the PMC's
> invitation to become a committer.
> 
> Julie, the tradition is that new committers introduce themselves with
> a brief bio.
> 
> I think we may still be sorting out the details of your Apache account
> (julie@ may have been taken?), but as soon as that has been sorted out
>  and karma has been granted, you can use your new powers to add
> yourself to the committers section of the Who We Are page on the
> website: 
> 
> Congratulations and welcome!
> 
> Mike Sokolov
> 
> -
> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
> For additional commands, e-mail: dev-h...@lucene.apache.org


-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: Welcome Julie Tibshirani as Lucene/Solr committer

[Shalin Shekhar Mangar]

Congratulations and welcome Julie!

On Wed, Nov 18, 2020 at 8:38 PM Michael Sokolov  wrote:

> I'm pleased to announce that Julie Tibshirani has accepted the PMC's
> invitation to become a committer.
>
> Julie, the tradition is that new committers introduce themselves with
> a brief bio.
>
> I think we may still be sorting out the details of your Apache account
> (julie@ may have been taken?), but as soon as that has been sorted out
>  and karma has been granted, you can use your new powers to add
> yourself to the committers section of the Who We Are page on the
> website: 
>
> Congratulations and welcome!
>
> Mike Sokolov
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
> For additional commands, e-mail: dev-h...@lucene.apache.org
>
>

-- 
Regards,
Shalin Shekhar Mangar.

Re: Welcome Julie Tibshirani as Lucene/Solr committer

[Joel Bernstein]

Welcome Julie!

On Wed, Nov 18, 2020 at 10:14 AM Adrien Grand  wrote:

> Welcome Julie!
>
> On Wed, Nov 18, 2020 at 4:09 PM Alan Woodward 
> wrote:
>
>> Congratulations and welcome Julie!
>>
>> > On 18 Nov 2020, at 15:06, Michael Sokolov  wrote:
>> >
>> > I'm pleased to announce that Julie Tibshirani has accepted the PMC's
>> > invitation to become a committer.
>> >
>> > Julie, the tradition is that new committers introduce themselves with
>> > a brief bio.
>> >
>> > I think we may still be sorting out the details of your Apache account
>> > (julie@ may have been taken?), but as soon as that has been sorted out
>> > and karma has been granted, you can use your new powers to add
>> > yourself to the committers section of the Who We Are page on the
>> > website: 
>> >
>> > Congratulations and welcome!
>> >
>> > Mike Sokolov
>> >
>> > -
>> > To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
>> > For additional commands, e-mail: dev-h...@lucene.apache.org
>> >
>>
>>
>> -
>> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
>> For additional commands, e-mail: dev-h...@lucene.apache.org
>>
>>
>
> --
> Adrien
>

Re: Welcome Julie Tibshirani as Lucene/Solr committer

[Adrien Grand]

Welcome Julie!

On Wed, Nov 18, 2020 at 4:09 PM Alan Woodward  wrote:

> Congratulations and welcome Julie!
>
> > On 18 Nov 2020, at 15:06, Michael Sokolov  wrote:
> >
> > I'm pleased to announce that Julie Tibshirani has accepted the PMC's
> > invitation to become a committer.
> >
> > Julie, the tradition is that new committers introduce themselves with
> > a brief bio.
> >
> > I think we may still be sorting out the details of your Apache account
> > (julie@ may have been taken?), but as soon as that has been sorted out
> > and karma has been granted, you can use your new powers to add
> > yourself to the committers section of the Who We Are page on the
> > website: 
> >
> > Congratulations and welcome!
> >
> > Mike Sokolov
> >
> > -
> > To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
> > For additional commands, e-mail: dev-h...@lucene.apache.org
> >
>
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
> For additional commands, e-mail: dev-h...@lucene.apache.org
>
>

-- 
Adrien

Re: Welcome Julie Tibshirani as Lucene/Solr committer

[Alan Woodward]

Congratulations and welcome Julie!

> On 18 Nov 2020, at 15:06, Michael Sokolov  wrote:
> 
> I'm pleased to announce that Julie Tibshirani has accepted the PMC's
> invitation to become a committer.
> 
> Julie, the tradition is that new committers introduce themselves with
> a brief bio.
> 
> I think we may still be sorting out the details of your Apache account
> (julie@ may have been taken?), but as soon as that has been sorted out
> and karma has been granted, you can use your new powers to add
> yourself to the committers section of the Who We Are page on the
> website: 
> 
> Congratulations and welcome!
> 
> Mike Sokolov
> 
> -
> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
> For additional commands, e-mail: dev-h...@lucene.apache.org
> 


-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Welcome Julie Tibshirani as Lucene/Solr committer

[Michael Sokolov]

I'm pleased to announce that Julie Tibshirani has accepted the PMC's
invitation to become a committer.

Julie, the tradition is that new committers introduce themselves with
a brief bio.

I think we may still be sorting out the details of your Apache account
(julie@ may have been taken?), but as soon as that has been sorted out
 and karma has been granted, you can use your new powers to add
yourself to the committers section of the Who We Are page on the
website: 

Congratulations and welcome!

Mike Sokolov

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: [JENKINS] Lucene-Solr-8.x-Linux (64bit/jdk-14.0.1) - Build # 5047 - Unstable!

[Dawid Weiss]

Can't reproduce (on master) with some beasting:

./gradlew test -p lucene/core --tests
TestLatLonPointQueries.testAllLatEqual  -Dtests.iters=100
-Dtests.multiplier=3 -Dtests.slow=true


On Wed, Nov 18, 2020 at 8:15 AM Policeman Jenkins Server <
jenk...@thetaphi.de> wrote:

> Build: https://jenkins.thetaphi.de/job/Lucene-Solr-8.x-Linux/5047/
> Java: 64bit/jdk-14.0.1 -XX:+UseCompressedOops -XX:+UseSerialGC
>
> 8 tests failed.
> FAILED:  org.apache.lucene.search.TestLatLonPointQueries.testAllLatEqual
>
> Error Message:
>
>
> Stack Trace:
> java.lang.AssertionError
> at
> __randomizedtesting.SeedInfo.seed([576081CCB1847199:EAF5D742B8495394]:0)
> at org.apache.lucene.geo.Rectangle.(Rectangle.java:61)
> at
> org.apache.lucene.geo.GeoEncodingUtils.createComponentPredicate(GeoEncodingUtils.java:185)
> at
> org.apache.lucene.document.LatLonPointInGeometryQuery.createWeight(LatLonPointInGeometryQuery.java:148)
> at
> org.apache.lucene.search.IndexSearcher.createWeight(IndexSearcher.java:726)
> at
> org.apache.lucene.search.AssertingIndexSearcher.createWeight(AssertingIndexSearcher.java:57)
> at
> org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:445)
> at
> org.apache.lucene.geo.BaseGeoPointTestCase.searchIndex(BaseGeoPointTestCase.java:1126)
> at
> org.apache.lucene.geo.BaseGeoPointTestCase.verifyRandomGeometries(BaseGeoPointTestCase.java:1068)
> at
> org.apache.lucene.geo.BaseGeoPointTestCase.verify(BaseGeoPointTestCase.java:776)
> at
> org.apache.lucene.geo.BaseGeoPointTestCase.testAllLatEqual(BaseGeoPointTestCase.java:500)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.base/java.lang.reflect.Method.invoke(Method.java:564)
> at
> com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1750)
> at
> com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:938)
> at
> com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:974)
> at
> com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:988)
> at
> org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
> at
> org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
> at
> org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
> at
> org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
> at
> org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
> at org.junit.rules.RunRules.evaluate(RunRules.java:20)
> at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> at
> com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
> at
> com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
> at
> com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
> at
> com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:947)
> at
> com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:832)
> at
> com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:883)
> at
> com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:894)
> at
> org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
> at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> at
> org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
> at
> com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
> at
> com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
> at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> at
> com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> at
> org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
> at
> org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
> 

Re: [JENKINS] Lucene-Solr-master-Linux (64bit/jdk-15) - Build # 28659 - Still Failing!

[Dawid Weiss]

> We should fix these build notification emails where the subject says
"failing" and then the subject says "All tests passed" :)

This is actually the correct message though -- the tests passed but
additional checks didn't.

I agree the amount of information and how it's selected should be improved.
We now have the tools to do it (for example, we could
create a separate message to be mailed from the CI only). I don't think
I'll have the time to do it soon but I filed an issue here -

https://issues.apache.org/jira/browse/LUCENE-9598

D.

On Tue, Nov 17, 2020 at 4:43 PM Michael McCandless <
luc...@mikemccandless.com> wrote:

> We should fix these build notification emails where the subject says
> "failing" and then the subject says "All tests passed" :)
>
> In this case it was javadoc linting issue.
>
> Unfortunately, I think to fix this we have to edit the most horrific
> regexp I've ever seen, that Jenkins uses to extract what actually failed
> from the sometimes massive console log ...
>
> Mike McCandless
>
> http://blog.mikemccandless.com
>
>
> On Mon, Nov 16, 2020 at 10:13 PM Policeman Jenkins Server <
> jenk...@thetaphi.de> wrote:
>
>> Build: https://jenkins.thetaphi.de/job/Lucene-Solr-master-Linux/28659/
>> Java: 64bit/jdk-15 -XX:+UseCompressedOops -XX:+UseSerialGC
>>
>> All tests passed
>>
>> -
>> To unsubscribe, e-mail: builds-unsubscr...@lucene.apache.org
>> For additional commands, e-mail: builds-h...@lucene.apache.org
>
>