Connie, Okay, so if I understand correctly:
1. Your proposal to stick with ICU4NET was a temporary measure, not a permanent one. 2. We have an alternative option that allows us to compile 32-bit with icu-dotnet, so we can eliminate the blocking issues for Lucene.Net.Analysis.Common and (hopefully) Lucene.Net.Analysis.ICU. > Is it better to solve this issue creating a separate PR against the master > branch that switches to using icu-dotnet? Depends on what the plan is. My hope is that #191 can be merged into master so rework doesn’t have to be done every time master is changed. While there are issues in master awaiting this merge (sweeping API changes and failing tests), there is still at least 3 weeks of porting work to do in the meantime, which could be stretched to at least 5 weeks by porting some of the non-essential packages (including Lucene.Net.Analysis.ICU). Is it realistic to expect #191 to be merged by then? What else needs to be done to make it ready, since the current status seems to indicate that it is? What I am saying is that solving the ICU issues (about blocking, bitness, deployment, cross-platform, documentation, etc.) permanently are more important than solving them quickly. Let's take the time we have to solve them for the long term, rather than theorizing about ways to solve them faster (which will ultimately result in wasteful rework). That is why I am asking about porting the Lucene.Net.Analysis.ICU package ASAP so we can put all of the ICU issues to bed once and for good. Or would it be better to hold off on Lucene.Net.Analysis.ICU to free up your time to work on other things? I have some doubts about whether icu-dotnet has everything we need for Lucene.Net.Analysis.ICU, but less so about Lucene.Net.Analysis.Collation and Lucene.Net.Analysis.Highlighter. Going by type name, icu-dotnet is missing some dependent types that are used by Lucene.Net.Analysis.ICU, which are: import com.ibm.icu.text.Normalizer2; import com.ibm.icu.text.Transliterator; import com.ibm.icu.text.Replaceable; import com.ibm.icu.text.UTF16; import com.ibm.icu.text.FilteredNormalizer2; import com.ibm.icu.util.ULocale; import com.ibm.icu.text.RawCollationKey; It may be that we just need to do an in-depth analysis of the features in order to find the corresponding functionality, but it also may be that we need more functionality ported in icu-dotnet in order to make the library complete enough to fully support Lucene.Net. Thanks, Shad Storhaug (NightOwl888) P.S. After you submitted #191, the first thing I did was attempt to use icu-dotnet in a standalone project to test some of its features. However, I couldn't seem to get it working. It would be helpful (not just for me, but for anyone using the library) if there were some documentation explaining how to setup a project to use it and/or just make the functionality work out of the box by installing the NuGet package. -----Original Message----- From: Connie Yau [mailto:[email protected]] Sent: Saturday, November 12, 2016 9:13 AM To: Shad Storhaug Cc: Itamar Syn-Hershko; [email protected] Subject: RE: ICU4NET 64bit? When you execute `dotnet build` in the current .NET Core toolchain uses csc.dll which in turn calls vbcscompiler (Roslyn) to compile the C# code into IL. This also applies to .xprojs when you run `msbuild Lucene.Net.Portable.sln`, it'll call `dotnet build` under the covers which then calls csc.dll. dotnet.exe and msbuild.exe understand how to process project.jsons and csproj/xproj files so that they can pass the appropriate arguments to csc.exe for compilation. The issue I was mentioning is that when I run `dotnet build --runtime win7-x86`, the output binaries will be x64 binaries because my .NET Core SDK is x64. This can be worked around by installing .NET Core SDK x86 and running `dotnet build` with that instead. In future releases of the .NET Core tooling, we switch to msbuild, which is why that issue (https://github.com/dotnet/cli/issues/3387) wasn't fixed. Icu-dotnet has RuleBasedCollator, so that shouldn't be a blocker anymore once I move to using icu-dotnet. Is it better to solve this issue creating a separate PR against the master branch that switches to using icu-dotnet? Connie -----Original Message----- From: Shad Storhaug [mailto:[email protected]] Sent: Friday, November 11, 2016 6:22 AM To: Connie Yau <[email protected]> Cc: Itamar Syn-Hershko <[email protected]>; [email protected] Subject: RE: ICU4NET 64bit? Hi Connie, Maybe this is a naïve question, but do we need to use MSBuild anymore? I mean, doesn't Roslyn have the capability to compile both the .NET 4.6 and .NET Core stacks? And if we used Roslyn exclusively for builds, would it help with solving the bitness issue? Also, ICU4Net is missing the RuleBasedCollator, which is a dependency for the last missing piece of Lucene.Net.Analysis.Common (Lucene.Net.Analysis.Collation). Using ICU4NET on 32 bit builds, we won't be able to add this missing piece to the Lucene.Net.Analysis.Common package. Note that most of the work has already been completed for Lucene.Net.Analysis.Collation on this branch: https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FChristopherHaws%2Flucenenet%2Ftree%2Fanalysis-work-2%2Fsrc%2FLucene.Net.Analysis.Common%2FCollation&data=02%7C01%7Cconniey%40microsoft.com%7C8e9fee949f9d43bd570a08d40a3e12cf%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636144709110996071&sdata=AOZcWZOLMs5bQ19WAj7GZMF7UrtaYgisa7Y%2Bi8TNqvk%3D&reserved=0 I am not sure if Lucene.Net.Analysis.Collation is critical for the Lucene.Net.Analysis.Common package release - in the .NET world there is no RuleBasedCollator, so it will most likely not be missed if this extension for it is not there. But, it would be nice to include it if we can. Perhaps providing Lucene.Net.Analysis.Collation to the 64 bit release and excluding it for 32 bit using conditional compilation is an option, although it seems like an uncommon thing to do. Another package (important for Lucene.Net release) that is dependent on ICU's (BreakIterator) is Lucene.Net.Highlighter, which is (patiently) awaiting the resolution of the ICU dependency before it is ported. Status ===== List of (known) sections remaining to port/complete for release: 1. Lucene.Net.Analysis.Collation (Namespace in Lucene.Net.Analysis.Common) 2. Lucene.Net.Highlighter 3. Lucene.Net.Store (Namespace in Lucene.Net.Misc - Native (C++) directories for Unix and Windows - important??) 4. Lucene.Net.Spatial (currently WIP) 5. Lucene.Net.QueryParsers.Flexible (Namespace in Lucene.Net.QueryParser) Other useful packages not yet ported that are not critical for release: 1. Lucene.Net.Analysis.ICU 2. Lucene.Net.Analysis.Kuromoji 3. Lucene.Net.Analysis.Morfologik 4. Lucene.Net.Analysis.Phonetic 5. Lucene.Net.Analysis.SmartCn 6. Lucene.Net.Analysis.UIMA 7. Lucene.Net.Demo Lucene.Net.Analysis.ICU also depends on icu.net. Although Itamar said that Lucene.Net.Analysis.ICU is not critical to port for release, would it make sense to port it sooner rather than later to make sure we have all of the support we need in icu.net? Thanks, Shad Storhaug (NightOwl888) -----Original Message----- From: Connie Yau [mailto:[email protected]] Sent: Friday, November 11, 2016 6:01 AM To: Itamar Syn-Hershko; [email protected] Subject: RE: ICU4NET 64bit? Hi, I apologise for the delayed response. In the .NET Core migration<https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Flucenenet%2Fpull%2F191%2F&data=02%7C01%7Cconniey%40microsoft.com%7C8e9fee949f9d43bd570a08d40a3e12cf%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636144709111006079&sdata=c65gRVNTMchVpOCh8Cef1iz69S0TzJsOdYEh4WEl84g%3D&reserved=0> I am working on, with the current .NET Core toolchain, it compiles for x64 even if a 32-bit runtime is specified. :( (It’s a known issue<https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fdotnet%2Fcli%2Fissues%2F3387&data=02%7C01%7Cconniey%40microsoft.com%7C8e9fee949f9d43bd570a08d40a3e12cf%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636144709111006079&sdata=nouDNzHkONT4xy0wA5bYJSnLYzpBPP4828IT7CsbzWc%3D&reserved=0> they are not going to fix because of the migration to msbuild.) As a result, the only way I could get the Lucene.Net.Analysis.Common library running was to change it to use 64-bit version of icu.net<https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.myget.org%2Ffeed%2Ficu-dotnet%2Fpackage%2Fnuget%2Ficu.net-x64&data=02%7C01%7Cconniey%40microsoft.com%7C8e9fee949f9d43bd570a08d40a3e12cf%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636144709111006079&sdata=kOaNxMGNyVgctWSJ7KRDLai2ugYC8Wkh1y5yzyyJTZg%3D&reserved=0> that I compiled from the sillsdev/icu-dotnet repository<https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fsillsdev%2Ficu-dotnet&data=02%7C01%7Cconniey%40microsoft.com%7C8e9fee949f9d43bd570a08d40a3e12cf%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636144709111006079&sdata=v2%2BRqQNZW%2FSJT5DIJ01ffJs0EnuH0xXUuDzCND%2BUXUI%3D&reserved=0>. As an aside, I can create a separate PR which uses x86 version of icu-dotnet that I’ve replaced ICU4NET with as a solution for this issue. Connie From: [email protected] [mailto:[email protected]] On Behalf Of Itamar Syn-Hershko Sent: Thursday, November 10, 2016 12:44 PM To: [email protected] Cc: Connie Yau <[email protected]> Subject: Re: ICU4NET 64bit? Ok, let's see what ICU support Shad and Connie come up with. Worst case we can release all analyzers requiring the ICU packages as a separate package and remove the ICU dependency from the more commonly used parts. -- Itamar Syn-Hershko https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fcode972.com&data=02%7C01%7Cconniey%40microsoft.com%7C8e9fee949f9d43bd570a08d40a3e12cf%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636144709111006079&sdata=LpwVcwgjpFB9SBWyiMMBLSMyE4gUW8BSKpilw23yw5Y%3D&reserved=0<https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fcode972.com%2F&data=01%7C01%7Cconniey%40microsoft.com%7C876a41dab5fd4e99406008d409aa4498%7C72f988bf86f141af91ab2d7cd011db47%7C1&sdata=hQQo6v997y%2Bdt1fJvs7k3Q6zV1lZjBQHyVXIDqGhuCs%3D&reserved=0> | @synhershko<https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftwitter.com%2Fsynhershko&data=01%7C01%7Cconniey%40microsoft.com%7C876a41dab5fd4e99406008d409aa4498%7C72f988bf86f141af91ab2d7cd011db47%7C1&sdata=4H05YsiyAsrOceyRCLuMgtqlOavX8%2FeL%2BUvz8%2FdU2fE%3D&reserved=0> Freelance Developer & Consultant Lucene.NET committer and PMC member On Wed, Nov 9, 2016 at 11:53 PM, Roethinger, Alexander <[email protected]<mailto:[email protected]>> wrote: Good morming! I can confirm that this is an issue. Spent two hours the other night trying to deploy my solution on a Windows Server 2012 VM to no avail. I kept on getting the known Assembly-Loading exceptions despite setting PATH variable or copying the native DLLs to the Windows directory. Forking Lucene.Net.Analysis.Common in order to manually remove the Thai-Analyzer and Wordbreaker is not an option as it defies the advantage of your release packages as well as other compilation issues. I finally ended up simply removing the assembly references to ICU4NET and ICU4NET.Extensions from my projects that use Lucene.Net.Analysis.Common and voila, the code still runs and I could deploy without needing the native DLLs. While I know that my application would crash if I were to access the Thai analyzer, since I'm not using it, it seemed safe to proceed that way. In any case it would be great to get rid of ICU4NET dependency, either by moving the Thai-Analyzer and WordBreaker to a separate assembly (which only solves the problem partially) or some true CLR alternative. Cheers Alexander -----Ursprüngliche Nachricht----- Von: [email protected]<mailto:[email protected]> [mailto:[email protected]<mailto:[email protected]>] Im Auftrag von Itamar Syn-Hershko Gesendet: Donnerstag, 10. November 2016 05:09 An: [email protected]<mailto:[email protected]> Cc: Connie Yau <[email protected]<mailto:[email protected]>> Betreff: Re: ICU4NET 64bit? I've had several reports on ICU4NET dependencies mostly on cloud environments - probably due to the native assemblies that are shipped with it. Let's make sure we are using a solution that runs on the CLR and work well. -- Itamar Syn-Hershko https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fcode972.com&data=02%7C01%7Cconniey%40microsoft.com%7C8e9fee949f9d43bd570a08d40a3e12cf%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636144709111006079&sdata=LpwVcwgjpFB9SBWyiMMBLSMyE4gUW8BSKpilw23yw5Y%3D&reserved=0<https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fcode972.com&data=01%7C01%7Cconniey%40microsoft.com%7C876a41dab5fd4e99406008d409aa4498%7C72f988bf86f141af91ab2d7cd011db47%7C1&sdata=YkdTdD8FbJ6kjuriq8oEJEIHZEdWCmLaOPJTWQTWILs%3D&reserved=0> | @synhershko <https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftwitter.com%2Fsynhershko&data=02%7C01%7Cconniey%40microsoft.com%7C8e9fee949f9d43bd570a08d40a3e12cf%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636144709111006079&sdata=AciVwx7K%2B6B87%2FDWjZnUjNPeqaJwx%2F89QbZ8%2Fe0NCmg%3D&reserved=0<https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftwitter.com%2Fsynhershko&data=01%7C01%7Cconniey%40microsoft.com%7C876a41dab5fd4e99406008d409aa4498%7C72f988bf86f141af91ab2d7cd011db47%7C1&sdata=4H05YsiyAsrOceyRCLuMgtqlOavX8%2FeL%2BUvz8%2FdU2fE%3D&reserved=0>> Freelance Developer & Consultant Lucene.NET committer and PMC member On Thu, Nov 3, 2016 at 11:55 AM, Laimonas Simutis <[email protected]<mailto:[email protected]>> wrote: > Alexander, > > I am pretty sure that is the same code that you linked. I just created > a nuget package with all the ICU4C dlls included so that we could > reference them in Lucene port via nuget without having to include c > libs somewhere else. > > I will need to go back and see where I ended up with the 64 bit port. > Although as Shad mentions, is this still relevant since for the long > term the port are going away from ICU4NET? > > > > On Thu, Nov 3, 2016 at 10:04 AM, Shad Storhaug > <[email protected]<mailto:[email protected]>> > wrote: > > > Alexander, > > > > Actually, Connie Yau from Microsoft is working on porting to .NET > > core, and one of the things that is happening is that she is > > switching the dependency from ICU4NET to > > icu.net<https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Ficu.net&data=01%7C01%7Cconniey%40microsoft.com%7C876a41dab5fd4e99406008d409aa4498%7C72f988bf86f141af91ab2d7cd011db47%7C1&sdata=iDTVkPtaza84wfQvIP1WUi2U%2FStVLCUvD6LCs3hYSiE%3D&reserved=0>. > > > > https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgit > > hub.com%2Fsillsdev%2Ficu-dotnet&data=02%7C01%7Cconniey%40microsoft.c > > om%7C8e9fee949f9d43bd570a08d40a3e12cf%7C72f988bf86f141af91ab2d7cd011 > > db47%7C1%7C0%7C636144709111006079&sdata=v2%2BRqQNZW%2FSJT5DIJ01ffJs0 > > EnuH0xXUuDzCND%2BUXUI%3D&reserved=0<https://na01.safelinks.protectio > > n.outlook.com/?url=https%3A%2F%2Fna01.safelinks.protec&data=02%7C01% > > 7Cconniey%40microsoft.com%7C8e9fee949f9d43bd570a08d40a3e12cf%7C72f98 > > 8bf86f141af91ab2d7cd011db47%7C1%7C0%7C636144709111006079&sdata=gLBrp > > ri7Q7bT1BG9Q1zgXfqwZ%2ByZKu%2FSM%2B7U8lLeip8%3D&reserved=0 > > tion.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fsillsdev%2Ficu-dotn > > et&data=01%7C01%7Cconniey%40microsoft.com%7C876a41dab5fd4e99406008d4 > > 09aa4498%7C72f988bf86f141af91ab2d7cd011db47%7C1&sdata=r8itnCstaCa0FO > > L6mrSjWa2xYEPFMN0Sat9EQGFdpng%3D&reserved=0> > > https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Flucenenet%2Fpull%2F191&data=02%7C01%7Cconniey%40microsoft.com%7C8e9fee949f9d43bd570a08d40a3e12cf%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636144709111006079&sdata=OEzuuiBgZLPUGfLsaMaJ6bRFa5UKxfOsI%2FeZ5BMyNKM%3D&reserved=0<https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fna01.safelinks&data=02%7C01%7Cconniey%40microsoft.com%7C8e9fee949f9d43bd570a08d40a3e12cf%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636144709111006079&sdata=2sSzbRZdT0POme2gxcmuT1KSqHvl8V2Rid5A9BmED2s%3D&reserved=0. > > protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fluce > > nenet%2Fpull%2F191&data=01%7C01%7Cconniey%40microsoft.com%7C876a41da > > b5fd4e99406008d409aa4498%7C72f988bf86f141af91ab2d7cd011db47%7C1&sdat > > a=4L4penTOgo4T0EjS35aviV0m6cK63hxdAyk5JYhP%2FSg%3D&reserved=0> > > > > So at this point, making changes to ICU4NET is moot. However, I am > > not sure what that means in terms of 64 bit compatibility. Both of > > these packages are just wrappers around the ICU4C library. > > > > Connie, perhaps you can share what you know? If Alexander is willing > > to update that package to work with 64 bit (and fix the issues with > > auto-loading), that would be great. > > > > > > Thanks, > > Shad Storhaug (NightOwl888) > > > > From: Roethinger, Alexander > > [mailto:[email protected]<mailto:[email protected]>] > > Sent: Thursday, November 3, 2016 9:37 PM > > To: [email protected]<mailto:[email protected]> > > Subject: ICU4NET 64bit? > > > > Dear Devs, > > > > can anyone point me to the source-code (if available) for the > > "ICU4NET packaged together with ICU4C 55.1 32-bit release" > > package available on Nuget and provided by Laimonas Simutis? > > > > Or is this the same code as in > > "Simple wrapper for ICU4C > > (https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fsite.icu-project.org%2F&data=02%7C01%7Cconniey%40microsoft.com%7C8e9fee949f9d43bd570a08d40a3e12cf%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636144709111006079&sdata=dczijNKi8O9XQBYNxlDSSYqpo1pc3gjizdhn36JMk2Q%3D&reserved=0)<https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fsite.icu-project.org%2F)&data=01%7C01%7Cconniey%40microsoft.com%7C876a41dab5fd4e99406008d409aa4498%7C72f988bf86f141af91ab2d7cd011db47%7C1&sdata=OlmEp4NxmBWbGIlQ3MW7PsEGyU4oWB79LT31mJATHNM%3D&reserved=0>." > > https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fniaher%2Ficu4net&data=02%7C01%7Cconniey%40microsoft.com%7C8e9fee949f9d43bd570a08d40a3e12cf%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636144709111006079&sdata=7DsF8ufyaitRhzsJimjIt2BeR6RAqsElnOwGEjvMVKQ%3D&reserved=0<https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fna01.safelinks.protection&data=02%7C01%7Cconniey%40microsoft.com%7C8e9fee949f9d43bd570a08d40a3e12cf%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636144709111006079&sdata=9K3xPGl6EoWMw4NXE7M1cIP1KV5zzvNJ7F%2FwN0GIU%2Fc%3D&reserved=0. > > outlook.com/?url=https%3A%2F%2Fgithub.com%2Fniaher%2Ficu4net&data=01 > > %7C01%7Cconniey%40microsoft.com%7C876a41dab5fd4e99406008d409aa4498%7 > > C72f988bf86f141af91ab2d7cd011db47%7C1&sdata=vurBYCkwS1BiswMx2vg2%2B4 > > gLu92ReatRBS7A3Fgni2w%3D&reserved=0> > > ? > > > > > > I would like to build a 64bit (resp. AnyCPU) version using the 64bit > > DLLs available at ICU-Homepage. > > I believe Laimonas had mentioned in a post that he apparently had > > already successfully tested a 64bit version. > > > > Are there plans to provide 64bit support for ICU4NET? > > (I know there's a discussion going regarding the dependency on > > ICU4NET) > > > > Thanks > > Alexander > > >
