Re: Gradle build

[Gus Heck]

+1 to option 2

On Thu, Nov 7, 2019 at 6:23 PM David Smiley 
wrote:

>
> Doing 2 doesn’t stop us going to 3 soon if we want.  Easier to fix/improve
> on one branch while it’s new.
>
> On Thu, Nov 7, 2019 at 5:41 PM Adrien Grand  wrote:
>
>> I'd be fine with option 2 but I have a slight preference for option 3.
>> If we see the Gradle build as the future default build, then we need
>> to start using it and I wonder that having to use a different workflow
>> on branch_8x would be an incentive to keep using the Ant build
>> instead.
>>
>> On Thu, Nov 7, 2019 at 11:15 AM Cassandra Targett 
>> wrote:
>> >
>> > I’m fine with Option 2.
>> >
>> > Putting my project manager hat on, I’d really like to see a central
>> list or Jira issues of the things we want to make sure to do before we can
>> turn off Ant. The list/sub-tasks could be compiled after it’s merged to
>> master, but it would be nice if we could approach this in a coordinated way
>> so we’re all able to figure out quickly what remains - I think we’ll have a
>> higher chance of faster success that way. Hopefully also people would sign
>> up for the things that they will volunteer to drive to completion instead
>> of hanging back because they aren’t sure what needs to be done.
>> >
>> > Dawid and I worked on the Ref Guide transition to Gradle in a separate
>> branch and it’s either finished or very close to it IMO. It includes the PR
>> I put up last night to remove the PDF build tasks. I just need to merge it
>> into the main Gradle branch (jira/SOLR-13452_gradle_8), which I will try to
>> do by tomorrow.
>> >
>> > Cassandra
>> > On Nov 6, 2019, 3:07 PM -0600, David Smiley ,
>> wrote:
>> >
>> > Option 2.
>> >
>> > ~ David Smiley
>> > Apache Lucene/Solr Search Developer
>> > http://www.linkedin.com/in/davidwsmiley
>> >
>> >
>> > On Wed, Nov 6, 2019 at 12:36 PM Erick Erickson 
>> wrote:
>> >>
>> >> All:
>> >>
>> >> re: SOLR-13452. I’m now coming down on both sides of the issue. My
>> motive here is if this was pushed, even in its current incomplete state,
>> people would have an easier time contributing to the effort. Of course this
>> means I would be asking people to use the gradle build at least on master
>> if at all possible.
>> >>
>> >> There are several assumptions I’m making here:
>> >>
>> >> - we can keep running the Ant build as long as necessary. Ant would be
>> our primary build process. The purpose of pushing the current Gradle effort
>> is to make it easier for others to contribute and “just try it”.
>> >>
>> >> - There are no conflicts between the Gradle and Ant builds, so we can
>> continue to use Ant as necessary until we make the switch.
>> >>
>> >> - people will commit up front to putting some effort into this. I flat
>> guarantee I won’t carry the load alone. If nobody else steps up, I’ll table
>> it. I will volunteer to push fixes from non-committers.
>> >> — Yes, people can do this already with the gradle_8 branch, it’d just
>> be easier if it was already in the pull.
>> >>
>> >> - moving to Gradle as our primary workflow won’t happen until we work
>> out some of the kinks, things like.
>> >> — running on Jenkins.
>> >> — Getting the equivalent of “ant server dist” to run.
>> >> — Getting the ref guide built.
>> >> — I’m sure other things will crop up.
>> >>
>> >>
>> >> So there are several options, please let me know which one you prefer:
>> >>
>> >> 1. do nothing. People can check out the gradle_8 build and work on it.
>> There has been some of this so far, many thanks.
>> >>
>> >> 2. merge it into master only. TBD is when we take Ant out of master.
>> >>
>> >> 3. merge into both master and 8x. Assuming we can continue to use
>> both, I’m not sure what advantage there is to merging into 8x. This seems
>> like something that should come along with a major version release.
>> >>
>> >> 4. wait until it’s feature-complete. Based on the evidence so far,
>> this may be a long time coming.
>> >>
>> >> Also, the timing is fungible. I don’t see a downside as long as we can
>> continue to build with Ant. I certainly _do_ see a downside if we have to
>> do everything Ant does immediately after pushing to whatever branches.
>> >>
>> >> Erick
>> >>
>> >>
>> >>
>> >>
>> >> -
>> >> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
>> >> For additional commands, e-mail: dev-h...@lucene.apache.org
>> >>
>>
>>
>> --
>> Adrien
>>
>> -
>> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
>> For additional commands, e-mail: dev-h...@lucene.apache.org
>>
>> --
> Sent from Gmail Mobile
>


-- 
http://www.needhamsoftware.com (work)
http://www.the111shift.com (play)

Re: Tracking down inconsistent failure in jenkins

[Raphael Bircher]

Hi all

On 2019/11/08 23:56:00, Raphael Bircher  wrote: 

> I was building solr and running the JUnit Tests now.

The tests was running, but I don't find the testlogs ;-)

I got two errors with a self builded solr from the head. I've also seen a 
Ubuntu machine on Jenkins with two errors. I let now run the test for a second 
time.

   [junit4] Execution time total: 1 hour 19 minutes 20 seconds
   [junit4] Tests summary: 888 suites (6 ignored), 4543 tests, 2 errors, 212 
ignored (183 assumptions)

Regards, Raphael

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: Tracking down inconsistent failure in jenkins

[Raphael Bircher]

Hi Erick

On 2019/11/08 22:49:31, Erick Erickson  wrote: 
> Raphael:
> 
> Thanks for becoming involved!
> 
> It’s super-frustrating that some of the tests on Jenkins do (or do not) 
> reproduce, even if you “beast” them. Hoss’ reports come from many different 
> environments, from Windows to various Java releases to… So “does it fail 
> locally” is a tricky question. Plus, many of the intermittent failures are 
> timing-related, so the speed of your local machine, the other tasks running 
> on your machine etc. can be a factor.

Ok, I expected something like this. Why are some test timing related? Are there 
any informations about this.

> 
> What I do is use Mark Miller’s “beast” script. See: 
> https://gist.github.com/markrmiller/dbdb792216dc98b018ad
> 
> Two important parameters to the script above are
> - how many separate tests you want to run in parallel. This helps when the 
> failures are timing-related
> - how many iterations of the tests you want to run. Each test puts its output 
> in a separate subdirectory, so when a test fails you have the full logs in 
> the corresponding subdirectory.
> 
> Then I run the failing test over and over and over. If I can get it to fail 
> (and if you’re getting 0.5% failures, it’s _really_ hit or miss) then I can 
> diagnose the logs in the appropriate directory, possibly add logging and run 
> it all again.
> 
> Unfortunately, for intermittently-failing tests, you never _quite_ know if 
> you’ve fixed the problem because your 10,000 iterations may have just lucked 
> out.

So you never get a consistent result, even if you run the same test on one 
build several times? Can others confirm this behavior?

I was building solr and running the JUnit Tests now.

Regards, Raphael


-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: Tracking down inconsistent failure in jenkins

[Erick Erickson]

Raphael:

Thanks for becoming involved!

It’s super-frustrating that some of the tests on Jenkins do (or do not) 
reproduce, even if you “beast” them. Hoss’ reports come from many different 
environments, from Windows to various Java releases to… So “does it fail 
locally” is a tricky question. Plus, many of the intermittent failures are 
timing-related, so the speed of your local machine, the other tasks running on 
your machine etc. can be a factor.

What I do is use Mark Miller’s “beast” script. See: 
https://gist.github.com/markrmiller/dbdb792216dc98b018ad

Two important parameters to the script above are
- how many separate tests you want to run in parallel. This helps when the 
failures are timing-related
- how many iterations of the tests you want to run. Each test puts its output 
in a separate subdirectory, so when a test fails you have the full logs in the 
corresponding subdirectory.

Then I run the failing test over and over and over. If I can get it to fail 
(and if you’re getting 0.5% failures, it’s _really_ hit or miss) then I can 
diagnose the logs in the appropriate directory, possibly add logging and run it 
all again.

Unfortunately, for intermittently-failing tests, you never _quite_ know if 
you’ve fixed the problem because your 10,000 iterations may have just lucked 
out.

Welcome to the joys of distributed computing ;)

Best,
Erick

> On Nov 8, 2019, at 5:31 PM, Raphael Bircher  wrote:
> 
> Hi all
> 
> In my welcome thread, Jason pointed me to Hoss's Jenkins failure page. I 
> think this is a good point to start for me. I like to help you tracking down 
> this inconsistent failures. At first, please, if I got something wrong, 
> correct me!
> 
> As far as I understand, there are tests in Jenkins who sometimes fail, and 
> sometimes not. It looks like nobody really know why. right?
> 
> So I need some information from you. First of all. Do you see the same 
> behavior, if you do the test locally. First I want to exclude that the bug is 
> within the Jenkins system.
> 
> Are there bugs who are possibly related to the tests in question?
> 
> When this behavior was first detected?
> 
> This is it for the start. Thanks for your help and your information
> 
> Regards, Raphael
> 
> -
> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
> For additional commands, e-mail: dev-h...@lucene.apache.org
> 


-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Tracking down inconsistent failure in jenkins

[Raphael Bircher]

Hi all

In my welcome thread, Jason pointed me to Hoss's Jenkins failure page. I think 
this is a good point to start for me. I like to help you tracking down this 
inconsistent failures. At first, please, if I got something wrong, correct me!

As far as I understand, there are tests in Jenkins who sometimes fail, and 
sometimes not. It looks like nobody really know why. right?

So I need some information from you. First of all. Do you see the same 
behavior, if you do the test locally. First I want to exclude that the bug is 
within the Jenkins system.

Are there bugs who are possibly related to the tests in question?

When this behavior was first detected?

This is it for the start. Thanks for your help and your information

Regards, Raphael

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: Hi I'm Raphael

[Raphael Bircher]

On 2019/11/08 05:42:01, Shawn Heisey  wrote: 
> On 11/7/2019 6:14 PM, Raphael Bircher wrote:
> > Wich source tree I should build for testing purpose? I heard rumors, that 
> > installing Lucene is a bit rocket science.
> 
> I'm here for Solr, and my interest in Lucene is only incidental, because 
> Solr derives most of its functionality from Lucene.
> 
> Almost all development work here begins in the master branch, and is 
> backported to other branches as required.  We release from the stable 
> branch, which is currently branch_8x.  At some point, likely next year, 
> we will create branch_9x and master will be renumbered to major version 10.
> 
> Lucene is not really geared for "installation" ... it is a development 
> library.  It is used by developers to write their own software, and it 
> will be up to that developer to work out how their software is 
> installed.  Lucene is a development dependency, much like this one:
> 
> https://dev.mysql.com/downloads/connector/j/

Thanks a load for your information. I expected something like this. So for 
testing it would be good, to have access to some external project wich are 
based on solr/lucene. So if someone want to invite me, you can write me an 
E-mail. The best way ist to write au Raphael (at) Vefko (dot) ch

Regards, Raphael

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: [lucene-solr] branch branch_8x_tmp created (now f4eea9b)

[Ishan Chattopadhyaya]

Thanks for catching this, Hoss. This is absolutely unacceptable. It
shouldn't have been there. I just deleted it.

Noble, please be careful about such mishaps. You could've named it
"jira/_branch8x" or something. Please don't create any branch
that doesn't start with "jira/" (unless you're the RM).

On Fri, Nov 8, 2019 at 10:41 PM Chris Hostetter
 wrote:
>
>
> : Subject: [lucene-solr] branch branch_8x_tmp created (now f4eea9b)
>
> WTF is this branch_8x_tmp !?!?!?!???
>
> Since when is this ok?
>
> -Hoss
> http://www.lucidworks.com/
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
> For additional commands, e-mail: dev-h...@lucene.apache.org
>

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: [lucene-solr] branch branch_8x_tmp created (now f4eea9b)

[Chris Hostetter]

: Subject: [lucene-solr] branch branch_8x_tmp created (now f4eea9b)

WTF is this branch_8x_tmp !?!?!?!???

Since when is this ok?

-Hoss
http://www.lucidworks.com/

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: Hi I'm Raphael

[Jason Gerlowski]

Hi Raphael,

Welcome and thanks for taking an interest in helping out!  Shawn's
pointers above are all good for getting started.  If there's any other
information you need, let us know.

You know best what you want to get involved with.  But if you're
interested in QA and are interested in the Solr side of things, you
might be interested in a test-failure aggregator that Hoss, a member
of the community here, has set up:
http://fucit.org/solr-jenkins-reports/failure-report.html.  Some tests
fail regularly but others have low failure rates and are challenging
to reproduce.  We always need help with tests and appreciate help
tracking down issues.

Looking forward to working with you as you get ramped up.

Best,

Jason

On Fri, Nov 8, 2019 at 12:42 AM Shawn Heisey  wrote:
>
> On 11/7/2019 6:14 PM, Raphael Bircher wrote:
> > Wich source tree I should build for testing purpose? I heard rumors, that 
> > installing Lucene is a bit rocket science.
>
> I'm here for Solr, and my interest in Lucene is only incidental, because
> Solr derives most of its functionality from Lucene.
>
> Almost all development work here begins in the master branch, and is
> backported to other branches as required.  We release from the stable
> branch, which is currently branch_8x.  At some point, likely next year,
> we will create branch_9x and master will be renumbered to major version 10.
>
> Lucene is not really geared for "installation" ... it is a development
> library.  It is used by developers to write their own software, and it
> will be up to that developer to work out how their software is
> installed.  Lucene is a development dependency, much like this one:
>
> https://dev.mysql.com/downloads/connector/j/
>
> If you want to work on a complete application, you can opt to work on
> Solr instead of Lucene.  Or you can go to project outside the Apache
> umbrella that happens to use Lucene as a dependency.
>
> Thanks,
> Shawn
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
> For additional commands, e-mail: dev-h...@lucene.apache.org
>

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: Solr 8.3 Solrj streaming expressions do not return all field values

[Joel Bernstein]

Before moving to a jira let's take a look at the underlying Solr queries in
the log. The Streaming Expressions just creates solr queries, in this case
queries to the /export handler. So when something is not working as
expected we want to strip away the streaming expression and debug the
actual queries that are being run.

You can find the Solr queries that appear in the log after running the
expressions and then try running them outside of the expression as plain
Solr queries.

You can also post the Solr queries to this thread and we discuss what the
logs say.

In these cases the logs always are the way to debug whats going on.


Joel Bernstein
http://joelsolr.blogspot.com/


On Wed, Nov 6, 2019 at 4:14 PM Jörn Franke  wrote:

> I created a JIRA for this:
> https://issues.apache.org/jira/browse/SOLR-13894
>
> On Wed, Nov 6, 2019 at 10:45 AM Jörn Franke  wrote:
>
>> I have checked now Solr 8.3 server in admin UI. Same issue.
>>
>> Reproduction:
>> select(search(testcollection,q=“test”,df=“Default”,defType=“edismax”,fl=“id”,
>> qt=“/export”, sort=“id asc”),id,if(eq(1,1),Y,N) as found)
>>
>> In 8.3 it returns only the id field.
>> In 8.2 it returns id,found field.
>>
>> Since found is generated by select (and not coming from the collection)
>> there must be an issue with select.
>>
>> Any idea why this is happening.
>>
>> Debug logs do not show any error and the expression is correctly received
>> by Solr.
>>
>> Thank you.
>>
>> Best regards
>>
>> > Am 05.11.2019 um 14:59 schrieb Jörn Franke :
>> >
>> > Thanks I will check and come back to you. As far as I remember (but
>> have to check) the queries generated by Solr were correct
>> >
>> > Just to be clear the same thing works with Solr 8.2 server and Solr 8.2
>> client.
>> >
>> > It show the odd behaviour with Solr 8.2 server and Solr 8.3 client.
>> >
>> >> Am 05.11.2019 um 14:49 schrieb Joel Bernstein :
>> >>
>> >> I'll probably need some more details. One thing that's useful is to
>> look at
>> >> the logs and see the underlying Solr queries that are generated. Then
>> try
>> >> those underlying queries against the Solr index and see what comes
>> back. If
>> >> you're not seeing the fields with the plain Solr queries then we know
>> it's
>> >> something going on below streaming expressions. If you are seeing the
>> >> fields then it's the expressions themselves that are not handling the
>> data
>> >> as expected.
>> >>
>> >>
>> >> Joel Bernstein
>> >> http://joelsolr.blogspot.com/
>> >>
>> >>
>>  On Mon, Nov 4, 2019 at 9:09 AM Jörn Franke 
>> wrote:
>> >>>
>> >>> Most likely this issue can bei also reproduced in the admin UI for the
>> >>> streaming handler of a collection.
>> >>>
>> > Am 04.11.2019 um 13:32 schrieb Jörn Franke :
>> 
>>  Hi,
>> 
>>  I use streaming expressions, e.g.
>>  Sort(Select(search(...),id,if(eq(1,1),Y,N) as found), by=“field A
>> asc”)
>>  (Using export handler, sort is not really mandatory , I will remove
>> it
>> >>> later anyway)
>> 
>>  This works perfectly fine if I use Solr 8.2.0 (server + client). It
>> >>> returns Tuples in the form { “id”,”12345”, “found”:”Y”}
>> 
>>  However, if I use Solr 8.2.0 as server and Solr 8.3.0 as client then
>> the
>> >>> above statement only returns the id field, but not the found field.
>> 
>>  Questions:
>>  1) is this expected behavior, ie Solr client 8.3.0 is in this case
>> not
>> >>> compatible with Solr 8.2.0 and server upgrade to Solr 8.3.0 will fix
>> this?
>>  2) has the syntax for the above expression changed? If so how?
>>  3) is this not expected behavior and I should create a Jira for it?
>> 
>>  Thank you.
>>  Best regards
>> >>>
>>
>

Re: ReleaseWizard tool

[Jan Høydahl]

Hi,

Looking forward to feedback on the tool Ishan!

One concrete question:
I see that v8.2.0 is still in the release repo 
http://www.apache.org/dist/lucene/solr/ 

The Wizard step "9.10. Stop mirroring old releases" should have told you to 
remove v8.2.0 form the repo.
Did you not yet complete all steps in the Wizard or was there a bug so that 
step did not display for you?

--
Jan Høydahl, search solution architect
Cominvent AS - www.cominvent.com

> 6. nov. 2019 kl. 21:46 skrev Ishan Chattopadhyaya :
> 
> Jan, I owe you the list of issues and filing of jiras regarding issues i 
> faced with the tool during 8.3.0. I'll try to do so this weekend.
> 
> On Thu, 7 Nov, 2019, 2:03 AM Cassandra Targett,  > wrote:
> I was far too swamped back in July to be able to take a look at this new 
> tool, but I’ve had a few minutes now and while I’ve never done a release, it 
> really looks very helpful.
> 
> Jan, I note in your initial description in this thread and in LUCENE-8852 you 
> mention the release wizard tool allows you to "generate a full Asciidoc guide 
> for the release” - what is it generating? I’m not a python expert, but I did 
> notice it requires & uses the asciidoctor gem. That’s not how we make the Ref 
> Guide so it must be for another purpose, but I’m not able to fully figure it 
> out.
> 
> Sorry for the late thread resurrection, as I said I only now have a bit of 
> time to catch up here.
> 
> Thanks,
> Cassandra
> On Jul 5, 2019, 3:11 PM -0500, Jan Høydahl  >, wrote:
>> Go for it. For me it was a very interesting experience, and I will likely do 
>> it again at some point!
>> 
>> Jan Høydahl
>> 
>> 5. jul. 2019 kl. 21:00 skrev David Smiley > >:
>> 
>>> Nice Jan!  Maybe I'll be an RM one day, now that there's a nice tool to 
>>> help :-)
>>> 
>>> ~ David Smiley
>>> Apache Lucene/Solr Search Developer
>>> http://www.linkedin.com/in/davidwsmiley 
>>> 
>>> 
>>> On Thu, Jul 4, 2019 at 2:53 PM Jan Høydahl >> > wrote:
>>> I wrote an article at LinkedIN pulse about the release process and the tool:
>>> https://www.linkedin.com/pulse/releasing-lucene-just-61-steps-jan-høydahl/ 
>>> 
>>> 
>>> --
>>> Jan Høydahl, search solution architect
>>> Cominvent AS - www.cominvent.com 
>>> 
 11. jun. 2019 kl. 10:46 skrev Jan Høydahl >>> >:
 
 I have now pushed the ReleaseWizard tool in 
 https://issues.apache.org/jira/browse/LUCENE-8852 
 
 Appreciate all kind of feedback!
 
 --
 Jan Høydahl, search solution architect
 Cominvent AS - www.cominvent.com 
 
> 1. jun. 2019 kl. 20:26 skrev Jan Høydahl  >:
> 
> As I said, I’ll start a thread about this, please reply to that instead 
> of continuing discussion in this thread which is about releaseWizard :)
> 
> Jan Høydahl
> 
> 1. jun. 2019 kl. 15:53 skrev Michael Sokolov  >:
> 
>> I'm not sure what the proper way to use fix version is. Suppose you back 
>> port a fix to multiple branches? Should fixVersion list all of them? 
>> Just pick one?
>> 
>> On Wed, May 29, 2019, 6:00 PM Jan Høydahl > > wrote:
>> My releaseWizard tool is getting more complete as the 7.7.2 release 
>> progresses. Will share the code just after I complete all steps.
>> 
>> I tested relasedocmaker and it digs up all the JIRA issues marked as 
>> RESOLVED for the version and creates two files.
>> CHANGELOG.md simply lists all issues under headings IMPROVEMENTS, BUG 
>> FIXES etc
>> One problem I found with how the CHANGELOG works is that it adds all 
>> issues having the version in fixVersion, even if the feature
>> was already released in an earlier version. That is because of the way 
>> we use JIRA fixVersion, adding both e.g. "master (9.0)" and "8.2"
>> at the same time, even if we know that 8.2 is the version the feature 
>> will be released. If we stop always adding "master" to fixVersion
>> but strive to keep it a list of version the feature/bugfix is FIRST 
>> introduced, then this tool will do the correct job.
>> 
>> RELEASENOTES.md lists "...new developer and user-facing 
>> incompatibilities, important issues, features, and major improvements.".
>> And if we enable the JIRA field "Release Notes" (we don't have it now), 
>> the content of that field will be used in the release notes instead of 
>> the JIRA description.
>> You can select any issue to surface in RELEASENOTES by adding a certain