This is an automated email from the ASF dual-hosted git repository.
nightowl888 pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/lucenenet.git
The following commit(s) were added to refs/heads/master by this push:
new 41ad67643 Added net6.0 target to Lucene.Net.Analysis.OpenNLP and
changed to using MavenReference (#892)
41ad67643 is described below
commit 41ad6764342a618cd62ab6580aca772e9ba43b76
Author: Shad Storhaug <[email protected]>
AuthorDate: Thu Feb 1 16:10:32 2024 +0700
Added net6.0 target to Lucene.Net.Analysis.OpenNLP and changed to using
MavenReference (#892)
* upgrade targets to target .net core 6, in addition to .net framework
* update net 4.6 version
* Lucene.Net.Tests.OpenNLP: Patched IDE behavior to use net48 when net461
is selected and net7.0 when net5.0 is selected. In CI, we set
IsTestProject=false and IsPublishable=false to skip these tests.
* publish-test-results-for-test-projects.yml: Added support for net7.0 and
net6.0 for Lucene.Net.Tests.Analysis.OpenNLP tests.
* .github/workflows: Regenerated to add net7.0 as a test framework for
Lucene.Net.Tests.Analysis.OpenNLP
* .build/dependencies.props: Upgrade System.Memory to 4.5.5 to match IKVM
8.2.0
* .build/dependencies.props: Bumped System.Runtime.CompilerServices.Unsafe
to 6.0.0 to match IKVM 8.5.0
* Lucene.Net.csproj: Added direct dependency on
System.Runtime.CompilerServices.Unsafe for netstandard2.0 and net462 to ensure
the version will work with any combination of Lucene.Net components. This is a
transitive dependency in a few 3rd party DLLs, but there may be version
conflicts if this isn't done on .NET Framework.
* Lucene.Net.Facet.csproj: Added explicit dependency on System.Memory for
netstandard2.0 and net462, since it is being used in Lucene.Net.Facet.
* Lucene.Net.TestFramework.csproj: Added dependency on System.Text.Json to
pin the version so it matches the reference of IKVM 8.5.0 (6.0.6).
* upgrade targets to target .net core 6, in addition to .net framework
* Lucene.Net.Analysis.OpenNLP.csproj, .build/dependencies.props: Changed to
use <MavenReference> to build opennlp-tools instead of using the pre-built
OpenNLP.NET NuGet package.
* .build/dependencies.props: bumped IKVM to 8.7.3 and IKVM.Maven.Sdk to
1.6.7
* Lucene.Net.Analysis.OpenNLP.csproj: Removed duplicate TargetFrameworks
declaration
* Lucene.Net.Analysis.OpenNLP: Changed target from net462 > net472, the
minimum supported by IKVM.
* Directory.Build.targets: Updated FEATURE_OPENNLP to be available on .NET
Core
* Lucene.Net.Tests.AllProjects: Updated references so we can successfully
compile with IKVM in the mix, both on .NET Framework and .NET Core
* .build/dependencies.props: Added OpenNLP MavenReference version so it can
be managed with the other packages
* Lucene.Net.Analysis.OpenNLP: Added Maven dependency on org.osgi.core to
eliminate build warnings (at least 1 type is referenced in opennlp-tools)
* Lucene.Net.Analysis.OpenNLP: For now, making net472 conditional based on
Windows due to lack of non-Windows build support in IKVM 8.7.3 (see:
https://github.com/ikvmnet/ikvm-maven/issues/49).
* .build/dependencies.props: Reverted back to OpenNLP 1.9.1 because of
build issues with opennlp-uima on 1.9.4. This aligns with Lucene 8.2.0.
* publish-nuget-packages.yml: Remove forward slash
* .build/dependencies.props: Bumped IKVM to 8.7.5
* Lucene.Net.Analysis.OpenNLP/overview.md: Added missing docs from Lucene
and link to MavenReference demo. Fixes #890.
* FEATURE: Lucene.Net.Analysis.Miscellaneous: Added TypeAsSynonymFilter
from Lucene 8.2.0 because it is called out in the docs as part of the process
of configuring Lucene.Net.Analysis.OpenNLP. Changed CannedTokenStream to set
ITypeAttribute.Type because it is required by the tests for TypeAsSynonymFilter.
* Lucene.Net.Analysis.Miscellaneous.TestTypeAsSynonymFilterFactory: Added
comment with lucene version compatibility level (to indicate we ported it from
Lucene 8.2.0)
* Lucene.Net.Analysis.OpenNLP.overview.md: Corrected information about
which filters are included in the package (there is no NER filter in the box)
---------
Co-authored-by: Laimonas Simutis <[email protected]>
---
.build/TestReferences.Common.targets | 4 -
.build/azure-templates/publish-nuget-packages.yml | 4 +-
.../publish-test-results-for-test-projects.yml | 23 ++++-
.build/dependencies.props | 12 ++-
.../Lucene-Net-Tests-Analysis-OpenNLP.yml | 2 +-
Directory.Build.targets | 4 +-
.../Analysis/Miscellaneous/TypeAsSynonymFilter.cs | 97 ++++++++++++++++++++
.../Miscellaneous/TypeAsSynonymFilterFactory.cs | 62 +++++++++++++
.../Lucene.Net.Analysis.OpenNLP.csproj | 15 +++-
src/Lucene.Net.Analysis.OpenNLP/overview.md | 100 ++++++++++++++++++++-
src/Lucene.Net.Facet/Lucene.Net.Facet.csproj | 10 ++-
.../Analysis/CannedTokenStream.cs | 10 ++-
.../Lucene.Net.TestFramework.csproj | 5 ++
.../Lucene.Net.Tests.AllProjects.csproj | 10 +--
.../TestTypeAsSynonymFilterFactory.cs | 56 ++++++++++++
.../Lucene.Net.Tests.Analysis.OpenNLP.csproj | 18 ++--
src/Lucene.Net/Lucene.Net.csproj | 4 +
.../Lucene.Net.Tests.Cli.csproj | 11 ++-
18 files changed, 405 insertions(+), 42 deletions(-)
diff --git a/.build/TestReferences.Common.targets
b/.build/TestReferences.Common.targets
index 3811a42fd..9eab94e1c 100644
--- a/.build/TestReferences.Common.targets
+++ b/.build/TestReferences.Common.targets
@@ -25,8 +25,4 @@
<PackageReference Include="NUnit" Version="$(NUnitPackageVersion)" />
<PackageReference Include="NUnit3TestAdapter"
Version="$(NUnit3TestAdapterPackageVersion)" />
</ItemGroup>
-
- <ItemGroup Condition=" '$(TargetFramework)' == 'net461' ">
- <PackageReference Include="System.Runtime.CompilerServices.Unsafe"
Version="$(SystemRuntimeCompilerServicesUnsafePackageVersion)" />
- </ItemGroup>
</Project>
\ No newline at end of file
diff --git a/.build/azure-templates/publish-nuget-packages.yml
b/.build/azure-templates/publish-nuget-packages.yml
index 184fa2de6..92d63d213 100644
--- a/.build/azure-templates/publish-nuget-packages.yml
+++ b/.build/azure-templates/publish-nuget-packages.yml
@@ -1,4 +1,4 @@
-# Licensed to the Apache Software Foundation (ASF) under one
+# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
@@ -57,7 +57,7 @@ steps:
inputs:
command: push
packagesToPush: '${{ parameters.nugetArtifactName }}/*.nupkg;!${{
parameters.nugetArtifactName }}/*.symbols.nupkg'
- publishVstsFeed: '/${{ parameters.artifactFeedID }}'
+ publishVstsFeed: '${{ parameters.artifactFeedID }}'
allowPackageConflicts: true
- task: PublishSymbols@2
diff --git a/.build/azure-templates/publish-test-results-for-test-projects.yml
b/.build/azure-templates/publish-test-results-for-test-projects.yml
index 7ee98b752..11ebac78f 100644
--- a/.build/azure-templates/publish-test-results-for-test-projects.yml
+++ b/.build/azure-templates/publish-test-results-for-test-projects.yml
@@ -102,7 +102,28 @@ steps:
testResultsArtifactName: '${{ parameters.testResultsArtifactName }}'
testResultsFileName: '${{ parameters.testResultsFileName }}'
-# Special case: Only supports net48
+# Special case: Only supports net7.0, net6.0 and net48
+
+- template: publish-test-results.yml
+ parameters:
+ testProjectName: 'Lucene.Net.Tests.Analysis.OpenNLP'
+ framework: 'net7.0' # Since condtions are not supported for templates, we
check for the file existence within publish-test-results.yml
+ vsTestPlatform: '${{ parameters.vsTestPlatform }}'
+ osName: '${{ parameters.osName }}'
+ testResultsFormat: '${{ parameters.testResultsFormat }}'
+ testResultsArtifactName: '${{ parameters.testResultsArtifactName }}'
+ testResultsFileName: '${{ parameters.testResultsFileName }}'
+
+- template: publish-test-results.yml
+ parameters:
+ testProjectName: 'Lucene.Net.Tests.Analysis.OpenNLP'
+ framework: 'net6.0' # Since condtions are not supported for templates, we
check for the file existence within publish-test-results.yml
+ vsTestPlatform: '${{ parameters.vsTestPlatform }}'
+ osName: '${{ parameters.osName }}'
+ testResultsFormat: '${{ parameters.testResultsFormat }}'
+ testResultsArtifactName: '${{ parameters.testResultsArtifactName }}'
+ testResultsFileName: '${{ parameters.testResultsFileName }}'
+
- template: publish-test-results.yml
parameters:
testProjectName: 'Lucene.Net.Tests.Analysis.OpenNLP'
diff --git a/.build/dependencies.props b/.build/dependencies.props
index a37cc27a3..22d4e0e23 100644
--- a/.build/dependencies.props
+++ b/.build/dependencies.props
@@ -38,6 +38,8 @@
<ICU4NLanguageDataPackageVersion>$(ICU4NPackageVersion)</ICU4NLanguageDataPackageVersion>
<ICU4NRegionDataPackageVersion>$(ICU4NPackageVersion)</ICU4NRegionDataPackageVersion>
<ICU4NTransliteratorPackageVersion>$(ICU4NPackageVersion)</ICU4NTransliteratorPackageVersion>
+ <IKVMPackageVersion>8.7.5</IKVMPackageVersion>
+ <IKVMMavenSdkPackageVersion>1.6.7</IKVMMavenSdkPackageVersion>
<J2NPackageVersion>2.0.0</J2NPackageVersion>
<LiquidTestReportsMarkdownPackageVersion>1.0.9</LiquidTestReportsMarkdownPackageVersion>
<MicrosoftAspNetCoreHttpAbstractionsPackageVersion>2.0.0</MicrosoftAspNetCoreHttpAbstractionsPackageVersion>
@@ -68,19 +70,23 @@
<NewtonsoftJsonPackageVersion>13.0.1</NewtonsoftJsonPackageVersion>
<NUnit3TestAdapterPackageVersion>3.17.0</NUnit3TestAdapterPackageVersion>
<NUnitPackageVersion>3.13.1</NUnitPackageVersion>
- <OpenNLPNETPackageVersion>1.9.1.1</OpenNLPNETPackageVersion>
<RandomizedTestingGeneratorsPackageVersion>2.7.8</RandomizedTestingGeneratorsPackageVersion>
<SharpZipLibPackageVersion>1.4.2</SharpZipLibPackageVersion>
<Spatial4nPackageVersion>0.4.1.1</Spatial4nPackageVersion>
- <SystemMemoryPackageVersion>4.5.4</SystemMemoryPackageVersion>
+ <SystemMemoryPackageVersion>4.5.5</SystemMemoryPackageVersion>
<SystemReflectionEmitPackageVersion>4.3.0</SystemReflectionEmitPackageVersion>
<SystemReflectionEmitILGenerationPackageVersion>4.3.0</SystemReflectionEmitILGenerationPackageVersion>
<SystemReflectionTypeExtensionsPackageVersion>4.3.0</SystemReflectionTypeExtensionsPackageVersion>
-
<SystemRuntimeCompilerServicesUnsafePackageVersion>5.0.0</SystemRuntimeCompilerServicesUnsafePackageVersion>
+
<SystemRuntimeCompilerServicesUnsafePackageVersion>6.0.0</SystemRuntimeCompilerServicesUnsafePackageVersion>
<SystemRuntimeInteropServicesRuntimeInformationPackageVersion>4.3.0</SystemRuntimeInteropServicesRuntimeInformationPackageVersion>
<SystemSecurityCryptographyXmlPackageVersion>6.0.1</SystemSecurityCryptographyXmlPackageVersion>
<SystemTextEncodingCodePagesPackageVersion>4.3.0</SystemTextEncodingCodePagesPackageVersion>
<SystemTextEncodingCodePagesPackageVersion Condition="
'$(TargetFramework)' == 'net461'
">5.0.0</SystemTextEncodingCodePagesPackageVersion>
+ <SystemTextJsonPackageVersion>6.0.6</SystemTextJsonPackageVersion>
<TimeZoneConverterPackageVersion>6.1.0</TimeZoneConverterPackageVersion>
</PropertyGroup>
+ <PropertyGroup Label="Maven Package Reference Versions">
+
<OpenNLPToolsMavenReferenceVersion>1.9.1</OpenNLPToolsMavenReferenceVersion>
+ <OSGICoreMavenReferenceVersion>4.2.0</OSGICoreMavenReferenceVersion>
+ </PropertyGroup>
</Project>
\ No newline at end of file
diff --git a/.github/workflows/Lucene-Net-Tests-Analysis-OpenNLP.yml
b/.github/workflows/Lucene-Net-Tests-Analysis-OpenNLP.yml
index d23d10fe7..ca1236afc 100644
--- a/.github/workflows/Lucene-Net-Tests-Analysis-OpenNLP.yml
+++ b/.github/workflows/Lucene-Net-Tests-Analysis-OpenNLP.yml
@@ -56,7 +56,7 @@ jobs:
fail-fast: false
matrix:
os: [windows-latest, ubuntu-latest]
- framework: [net48]
+ framework: [net7.0, net48]
platform: [x64]
configuration: [Release]
exclude:
diff --git a/Directory.Build.targets b/Directory.Build.targets
index c39e315ef..042d18844 100644
--- a/Directory.Build.targets
+++ b/Directory.Build.targets
@@ -129,9 +129,9 @@
<DebugType>full</DebugType>
</PropertyGroup>
- <!-- Features in .NET Framework 4.5+ but not in .NET Standard 2.0 -->
+ <!-- Features in .NET Framework 4.5+ and .NET 6.0+ but not in .NET Standard
2.0 or .NET Standard 2.1 -->
<!-- net461 is used to test .NET Standard 2.0, so we treat it like it is not
part of this group -->
- <PropertyGroup Condition="$(TargetFramework.StartsWith('net4')) And
'$(TargetFramework)' != 'net461'">
+ <PropertyGroup Condition=" ($(TargetFramework.StartsWith('net4')) And
'$(TargetFramework)' != 'net461') Or $(TargetFramework.StartsWith('net6.')) Or
$(TargetFramework.StartsWith('net7.'))">
<DefineConstants>$(DefineConstants);FEATURE_OPENNLP</DefineConstants>
diff --git
a/src/Lucene.Net.Analysis.Common/Analysis/Miscellaneous/TypeAsSynonymFilter.cs
b/src/Lucene.Net.Analysis.Common/Analysis/Miscellaneous/TypeAsSynonymFilter.cs
new file mode 100644
index 000000000..b72ec02e0
--- /dev/null
+++
b/src/Lucene.Net.Analysis.Common/Analysis/Miscellaneous/TypeAsSynonymFilter.cs
@@ -0,0 +1,97 @@
+// Lucene version compatibility level 8.2.0
+// LUCENENET NOTE: Ported because Lucene.Net.Analysis.OpenNLP requires this to
be useful.
+using Lucene.Net.Analysis.TokenAttributes;
+using Lucene.Net.Util;
+#nullable enable
+
+namespace Lucene.Net.Analysis.Miscellaneous
+{
+ /*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+ /// <summary>
+ /// Adds the <see cref="ITypeAttribute.Type"/> as a synonym,
+ /// i.e. another token at the same position, optionally with a specified
prefix prepended.
+ /// </summary>
+ public sealed class TypeAsSynonymFilter : TokenFilter
+ {
+ private readonly ICharTermAttribute termAtt;
+ private readonly ITypeAttribute typeAtt;
+ private readonly IPositionIncrementAttribute posIncrAtt;
+ private readonly string? prefix;
+
+ private State? savedToken = null;
+
+ /// <summary>
+ /// Initializes a new instance of <see cref="TypeAsSynonymFilter"/>
with
+ /// the specified token stream.
+ /// </summary>
+ /// <param name="input">Input token stream.</param>
+ public TypeAsSynonymFilter(TokenStream input)
+ : this(input, null)
+ {
+ }
+
+ /// <summary>
+ /// Initializes a new instance of <see cref="TypeAsSynonymFilter"/>
with
+ /// the specified token stream and prefix.
+ /// </summary>
+ /// <param name="input">Input token stream.</param>
+ /// <param name="prefix">Prepend this string to every token type
emitted as token text.
+ /// If <c>null</c>, nothing will be prepended.</param>
+ public TypeAsSynonymFilter(TokenStream input, string? prefix)
+ : base(input)
+ {
+ this.prefix = prefix;
+ termAtt = AddAttribute<ICharTermAttribute>();
+ typeAtt = AddAttribute<ITypeAttribute>();
+ posIncrAtt = AddAttribute<IPositionIncrementAttribute>();
+ }
+
+
+ public override bool IncrementToken()
+ {
+ if (savedToken != null)
+ {
+ // Emit last token's type at the same position
+ RestoreState(savedToken);
+ savedToken = null;
+ termAtt.SetEmpty();
+ if (prefix != null)
+ {
+ termAtt.Append(prefix);
+ }
+ termAtt.Append(typeAtt.Type);
+ posIncrAtt.PositionIncrement = 0;
+ return true;
+ }
+ else if (m_input.IncrementToken())
+ {
+ // Ho pending token type to emit
+ savedToken = CaptureState();
+ return true;
+ }
+ return false;
+ }
+
+ public override void Reset()
+ {
+ base.Reset();
+ savedToken = null;
+ }
+ }
+}
diff --git
a/src/Lucene.Net.Analysis.Common/Analysis/Miscellaneous/TypeAsSynonymFilterFactory.cs
b/src/Lucene.Net.Analysis.Common/Analysis/Miscellaneous/TypeAsSynonymFilterFactory.cs
new file mode 100644
index 000000000..462be60bd
--- /dev/null
+++
b/src/Lucene.Net.Analysis.Common/Analysis/Miscellaneous/TypeAsSynonymFilterFactory.cs
@@ -0,0 +1,62 @@
+// Lucene version compatibility level 8.2.0
+// LUCENENET NOTE: Ported because Lucene.Net.Analysis.OpenNLP requires this to
be useful.
+using Lucene.Net.Analysis.Util;
+using System;
+using System.Collections.Generic;
+#nullable enable
+
+namespace Lucene.Net.Analysis.Miscellaneous
+{
+ /*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+ /// <summary>
+ /// Factory for <see cref="TypeAsSynonymFilter"/>.
+ /// <code>
+ /// <fieldType name="text_type_as_synonym" class="solr.TextField"
positionIncrementGap="100">
+ /// <analyzer>
+ /// <tokenizer class="solr.UAX29URLEmailTokenizerFactory"/>
+ /// <filter class="solr.TypeAsSynonymFilterFactory" prefix="_type_"
/>
+ /// </analyzer>
+ /// </fieldType>
+ /// </code>
+ ///
+ /// <para/>
+ /// If the optional <c>prefix</c> parameter is used, the specified value
will be prepended
+ /// to the type, e.g.with prefix = "_type_", for a token "example.com"
with type "<URL>",
+ /// the emitted synonym will have text "_type_<URL>".
+ /// </summary>
+ public class TypeAsSynonymFilterFactory : TokenFilterFactory
+ {
+ private readonly string prefix;
+
+ public TypeAsSynonymFilterFactory(IDictionary<string, string> args)
+ : base(args)
+ {
+ prefix = Get(args, "prefix"); // default value is null
+ if (args.Count > 0)
+ {
+ throw new
ArgumentException(string.Format(J2N.Text.StringFormatter.CurrentCulture,
"Unknown parameters: {0}", args));
+ }
+ }
+
+ public override TokenStream Create(TokenStream input)
+ {
+ return new TypeAsSynonymFilter(input, prefix);
+ }
+ }
+}
diff --git a/src/Lucene.Net.Analysis.OpenNLP/Lucene.Net.Analysis.OpenNLP.csproj
b/src/Lucene.Net.Analysis.OpenNLP/Lucene.Net.Analysis.OpenNLP.csproj
index 625d6ee9c..33920cdd4 100644
--- a/src/Lucene.Net.Analysis.OpenNLP/Lucene.Net.Analysis.OpenNLP.csproj
+++ b/src/Lucene.Net.Analysis.OpenNLP/Lucene.Net.Analysis.OpenNLP.csproj
@@ -30,7 +30,10 @@
<Import Project="$(SolutionDir).build/nuget.props" />
<PropertyGroup>
- <TargetFramework>net462</TargetFramework>
+ <!-- Currently, IKVM doesn't officially support building NetFX on anything
but Windows, so we skip it for contributors who may be on various platforms.
+ We can remove the condition once that has been addressed. See:
https://github.com/ikvmnet/ikvm-maven/issues/49 -->
+ <TargetFrameworks>net6.0</TargetFrameworks>
+ <TargetFrameworks
Condition="$([MSBuild]::IsOsPlatform('Windows'))">$(TargetFrameworks);net472</TargetFrameworks>
<AssemblyTitle>Lucene.Net.Analysis.OpenNLP</AssemblyTitle>
<PackageTags>$(PackageTags);analysis;natural;language;processing;opennlp</PackageTags>
@@ -39,8 +42,6 @@
<RootNamespace>Lucene.Net.Analysis.OpenNlp</RootNamespace>
</PropertyGroup>
-
-
<ItemGroup>
<ProjectReference Include="..\dotnet\Lucene.Net.ICU\Lucene.Net.ICU.csproj"
/>
<ProjectReference Include="..\Lucene.Net\Lucene.Net.csproj" />
@@ -49,7 +50,13 @@
<ItemGroup>
<PackageReference Include="ICU4N" Version="$(ICU4NPackageVersion)" />
- <PackageReference Include="OpenNLP.NET"
Version="$(OpenNLPNETPackageVersion)" />
+ <PackageReference Include="IKVM" Version="$(IKVMPackageVersion)" />
+ <PackageReference Include="IKVM.Maven.Sdk"
Version="$(IKVMMavenSdkPackageVersion)" />
+ </ItemGroup>
+
+ <ItemGroup>
+ <MavenReference Include="org.apache.opennlp:opennlp-tools"
Version="$(OpenNLPToolsMavenReferenceVersion)" />
+ <MavenReference Include="org.osgi:org.osgi.core"
Version="$(OSGICoreMavenReferenceVersion)" />
</ItemGroup>
</Project>
diff --git a/src/Lucene.Net.Analysis.OpenNLP/overview.md
b/src/Lucene.Net.Analysis.OpenNLP/overview.md
index ba4f9fc9b..882ddff08 100644
--- a/src/Lucene.Net.Analysis.OpenNLP/overview.md
+++ b/src/Lucene.Net.Analysis.OpenNLP/overview.md
@@ -21,4 +21,102 @@ summary: *content
limitations under the License.
-->
-OpenNLP Library Integration
\ No newline at end of file
+OpenNLP Library Integration
+
+This module exposes functionality from Apache OpenNLP to Apache Lucene.NET.
The Apache OpenNLP library is a machine learning based toolkit for the
processing of natural language text.
+
+For an introduction to Lucene's analysis API, see the
[Lucene.Net.Analysis](../core/Lucene.Net.Analysis.html) namespace documentation.
+
+The OpenNLP Tokenizer behavior is similar to the
<xref:Lucene.Net.Analysis.Core.WhiteSpaceTokenizer> but is smart about
inter-word punctuation. The term stream looks very much like the way you parse
words and punctuation while reading. The major difference between this
tokenizer and most other tokenizers shipped with Lucene is that punctuation is
tokenized. This is required for the following taggers to operate properly.
+
+The OpenNLP taggers annotate terms using the
<xref:Lucene.Net.Analysis.TokenAttributes.ITypeAttribute>.
+
+- <xref:Lucene.Net.Analysis.OpenNlp.OpenNLPTokenizer> segments text into
sentences or words. This Tokenizer uses the OpenNLP Sentence Detector and/or
Tokenizer classes. When used together, the Tokenizer receives sentences and can
do a better job.
+- <xref:Lucene.Net.Analysis.OpenNlp.OpenNLPPOSFilter> tags words for
Part-of-Speech and <xref:Lucene.Net.Analysis.OpenNlp.OpenNLPChunkerFilter> tags
words for Chunking. These tags are assigned as token types. Note that only one
of these operations will tag
+Since the <xref:Lucene.Net.Analysis.TokenAttributes.ITypeAttribute> is not
stored in the index, it is recommended that one of these filters is used
following OpenNLPFilter to enable search against the assigned tags:
+
+- <xref:Lucene.Net.Analysis.Payloads.TypeAsPayloadTokenFilter> copies the
<xref:Lucene.Net.Analysis.TokenAttributes.ITypeAttribute> value to the
<xref:Lucene.Net.Analysis.TokenAttributes.IPayloadAttribute>
+- <xref:Lucene.Net.Analysis.Miscellaneous.TypeAsSynonymFilter> creates a
cloned token at the same position as each tagged token, and copies the
<xref:Lucene.Net.Analysis.TokenAttributes.ITypeAttribute> value to the
<xref:Lucene.Net.Analysis.TokenAttributes.ICharTermAttribute>, optionally with
a customized prefix (so that tags effectively occupy a different namespace from
token text).
+
+Named Entity Recognition is also supported by OpenNLP, but there is no
OpenNLPNERFilter included. For an implementation, see the
[lucenenet-opennlp-mavenreference-demo](https://github.com/NightOwl888/lucenenet-opennlp-mavenreference-demo).
+
+## MavenReference Primer
+
+When a `<PackageReference>` is included for this NuGet package in your
SDK-style MSBuild project, it will automatically include transient dependencies
to [`opennlp-tools` on
maven.org](https://search.maven.org/artifact/org.apache.opennlp/opennlp-tools/1.9.4/bundle).
The transient dependency will automatically include a `<MavenReference>` in
your MSBuild project.
+
+The `<MavenReference>` item group operates similar to a dependency in Maven.
All transitive dependencies are collected and resolved, and then the final
output is produced. However, unlike `PackageReference`s, `MavenReference`s are
collected by the final output project, and reassessed. That is, each dependent
Project within your .NET SDK-style solution contributes its `MavenReference`s
to project(s) which include it, and each project makes its own dependency
graph. Projects do not contrib [...]
+
+> **NOTE** `<MavenReference>` is only supported on SDK-style MSBuild projects.
+
+## MavenReference Example
+
+This means this package can be combined with other related packages on Maven
in your project and they can be accessed using the same path as in Java like a
namespace in .NET. For example, you can add a `<MavenReference>` to your
project to include a reference to
[`opennlp-uima`](https://search.maven.org/artifact/org.apache.opennlp/opennlp-uima/1.9.1/jar).
The UIMA (Unstructured Information Management Architecture) integration module
is designed to work with the Apache UIMA framework. UIM [...]
+
+Here's a basic outline of how you might extend an existing Lucene.NET analyzer
to incorporate OpenNLP-UIMA annotators:
+
+```xml
+<Project Sdk="Microsoft.NET.Sdk">
+ <PropertyGroup>
+ <TargetFramework>net7.0</TargetFramework>
+ </PropertyGroup>
+
+ <ItemGroup>
+ <PackageReference Include="Lucene.Net.Analysis.OpenNLP"
Version="4.8.0-beta00017" />
+ </ItemGroup>
+
+ <ItemGroup>
+ <MavenReference Include="org.apache.opennlp:opennlp-uima" Version="1.9.1"
/>
+ </ItemGroup>
+</Project>
+```
+
+```c#
+using Lucene.Net.Analysis;
+using Lucene.Net.Analysis.Core;
+using Lucene.Net.Analysis.Util;
+using Lucene.Net.Util;
+using org.apache.uima.analysis_engine;
+using System.IO;
+
+public class CustomOpenNLPAnalyzer : OpenNLPTokenizerFactory
+{
+ // ... constructor and other methods ...
+
+ public override Tokenizer Create(AttributeFactory factory, TextReader
reader)
+ {
+ Tokenizer tokenizer = base.Create(factory, reader);
+
+ // Wrap the tokenizer with UIMA annotators
+ AnalysisEngineDescription uimaSentenceAnnotator =
CreateUIMASentenceAnnotator();
+ AnalysisEngineDescription uimaTokenAnnotator =
CreateUIMATokenAnnotator();
+
+ // Combine OpenNLP-UIMA annotators with the existing tokenizer
+ AnalysisEngine tokenizerAndUIMAAnnotators =
CreateAggregate(uimaSentenceAnnotator, uimaTokenAnnotator);
+
+ return new UIMATokenizer(tokenizer, tokenizerAndUIMAAnnotators);
+ }
+
+ // ... other methods ...
+
+ private AnalysisEngineDescription CreateUIMASentenceAnnotator() {
+ // Create and configure UIMA sentence annotator
+ // ...
+
+ return /* UIMA sentence annotator description */;
+ }
+
+ private AnalysisEngineDescription CreateUIMATokenAnnotator() {
+ // Create and configure UIMA token annotator
+ // ...
+
+ return /* UIMA token annotator description */;
+ }
+}
+```
+
+In the above example, `CustomOpenNLPAnalyzer` extends
`OpenNLPTokenizerFactory` (assuming that's the analyzer you're using), and it
wraps the OpenNLP tokenizer with UIMA annotators. You'll need to replace the
placeholder methods (`CreateUIMASentenceAnnotator` and
`CreateUIMATokenAnnotator`) with the actual code to create and configure your
UIMA annotators. Please note that configuring NLP can be complex. See the
[OpenNLP 1.9.4 Manual](https://opennlp.apache.org/docs/1.9.4/manual/opennlp.
[...]
+
+> [!NOTE]
+> IKVM (and `<MavenReference>`) does not support Java SE higher than version
8. So it will not be possible to add a `<MavenReference>` to OpenNLP 2.x until
support is added for it in IKVM.
+
+For a more complete example, see the
[lucenenet-opennlp-mavenreference-demo](https://github.com/NightOwl888/lucenenet-opennlp-mavenreference-demo).
\ No newline at end of file
diff --git a/src/Lucene.Net.Facet/Lucene.Net.Facet.csproj
b/src/Lucene.Net.Facet/Lucene.Net.Facet.csproj
index 5f489ec87..2406b72f5 100644
--- a/src/Lucene.Net.Facet/Lucene.Net.Facet.csproj
+++ b/src/Lucene.Net.Facet/Lucene.Net.Facet.csproj
@@ -38,13 +38,19 @@
<NoWarn>$(NoWarn);1591;1573</NoWarn>
</PropertyGroup>
-
-
<ItemGroup>
<ProjectReference Include="..\Lucene.Net.Join\Lucene.Net.Join.csproj" />
<ProjectReference
Include="..\Lucene.Net.Queries\Lucene.Net.Queries.csproj" />
</ItemGroup>
+ <ItemGroup Condition=" '$(TargetFramework)' == 'netstandard2.0' ">
+ <PackageReference Include="System.Memory"
Version="$(SystemMemoryPackageVersion)" />
+ </ItemGroup>
+
+ <ItemGroup Condition=" '$(TargetFramework)' == 'net462' ">
+ <PackageReference Include="System.Memory"
Version="$(SystemMemoryPackageVersion)" />
+ </ItemGroup>
+
<ItemGroup>
<InternalsVisibleTo Include="Lucene.Net.Tests.Facet" />
</ItemGroup>
diff --git a/src/Lucene.Net.TestFramework/Analysis/CannedTokenStream.cs
b/src/Lucene.Net.TestFramework/Analysis/CannedTokenStream.cs
index e7d787d04..824eaec95 100644
--- a/src/Lucene.Net.TestFramework/Analysis/CannedTokenStream.cs
+++ b/src/Lucene.Net.TestFramework/Analysis/CannedTokenStream.cs
@@ -1,4 +1,4 @@
-using Lucene.Net.Analysis.TokenAttributes;
+using Lucene.Net.Analysis.TokenAttributes;
namespace Lucene.Net.Analysis
{
@@ -31,6 +31,7 @@ namespace Lucene.Net.Analysis
private readonly IPositionLengthAttribute posLengthAtt;
private readonly IOffsetAttribute offsetAtt;
private readonly IPayloadAttribute payloadAtt;
+ private readonly ITypeAttribute typeAtt; // LUCENENET specific - See
IncrementToken()
private readonly int finalOffset;
private readonly int finalPosInc;
@@ -49,6 +50,7 @@ namespace Lucene.Net.Analysis
posLengthAtt = AddAttribute<IPositionLengthAttribute>();
offsetAtt = AddAttribute<IOffsetAttribute>();
payloadAtt = AddAttribute<IPayloadAttribute>();
+ typeAtt = AddAttribute<ITypeAttribute>(); // LUCENENET specific -
See IncrementToken()
this.tokens = tokens;
this.finalOffset = finalOffset;
@@ -76,6 +78,12 @@ namespace Lucene.Net.Analysis
posLengthAtt.PositionLength = token.PositionLength;
offsetAtt.SetOffset(token.StartOffset, token.EndOffset);
payloadAtt.Payload = token.Payload;
+
+ // LUCENENET: This change is from
https://github.com/apache/lucene/commit/72eaeab7151d421a28ecec1634b8c48599e524f5.
+ // We need it for the TestTypeAsSynonymFilterFactory tests to
pass (from lucene 8.2.0).
+ // But we don't yet have all of the PackedTokenAttributeImpl
plumbing it takes to do it the way they did,
+ // so setting it explicitly as a workaround.
+ typeAtt.Type = token.Type;
return true;
}
else
diff --git a/src/Lucene.Net.TestFramework/Lucene.Net.TestFramework.csproj
b/src/Lucene.Net.TestFramework/Lucene.Net.TestFramework.csproj
index 712ec3b0e..4eb51d734 100644
--- a/src/Lucene.Net.TestFramework/Lucene.Net.TestFramework.csproj
+++ b/src/Lucene.Net.TestFramework/Lucene.Net.TestFramework.csproj
@@ -66,6 +66,11 @@
<PackageReference Include="RandomizedTesting.Generators"
Version="$(RandomizedTestingGeneratorsPackageVersion)" />
</ItemGroup>
+ <ItemGroup Condition=" '$(TargetFramework)' == 'net462' ">
+ <!-- This is a transitive dependency of
Microsoft.Extensions.Configuration.Json and IKVM. It is just here to pin the
version to avoid conflicts. -->
+ <PackageReference Include="System.Text.Json"
Version="$(SystemTextJsonPackageVersion)" />
+ </ItemGroup>
+
<ItemGroup Condition=" '$(TargetFramework)' == 'net462' ">
<Reference Include="System.IO.Compression" />
<Reference Include="System.Numerics" />
diff --git
a/src/Lucene.Net.Tests.AllProjects/Lucene.Net.Tests.AllProjects.csproj
b/src/Lucene.Net.Tests.AllProjects/Lucene.Net.Tests.AllProjects.csproj
index 6219d7596..92056ba0a 100644
--- a/src/Lucene.Net.Tests.AllProjects/Lucene.Net.Tests.AllProjects.csproj
+++ b/src/Lucene.Net.Tests.AllProjects/Lucene.Net.Tests.AllProjects.csproj
@@ -118,18 +118,10 @@
</ItemGroup>
- <ItemGroup Condition=" '$(TargetFramework)' == 'net48' ">
+ <ItemGroup Condition=" '$(TargetFramework)' == 'net48' Or
$(TargetFramework.StartsWith('net6.')) Or
$(TargetFramework.StartsWith('net7.')) ">
<ProjectReference
Include="..\Lucene.Net.Analysis.OpenNLP\Lucene.Net.Analysis.OpenNLP.csproj" />
</ItemGroup>
- <ItemGroup Condition=" '$(TargetFramework)' == 'net48' ">
- <Reference Include="System" />
- </ItemGroup>
-
- <ItemGroup Condition=" '$(TargetFramework)' == 'net461' ">
- <Reference Include="System" />
- </ItemGroup>
-
<Import Project="$(SolutionDir).build/TestReferences.Common.targets" />
</Project>
diff --git
a/src/Lucene.Net.Tests.Analysis.Common/Analysis/Miscellaneous/TestTypeAsSynonymFilterFactory.cs
b/src/Lucene.Net.Tests.Analysis.Common/Analysis/Miscellaneous/TestTypeAsSynonymFilterFactory.cs
new file mode 100644
index 000000000..8f363f4f9
--- /dev/null
+++
b/src/Lucene.Net.Tests.Analysis.Common/Analysis/Miscellaneous/TestTypeAsSynonymFilterFactory.cs
@@ -0,0 +1,56 @@
+// Lucene version compatibility level 8.2.0
+// LUCENENET NOTE: Ported because Lucene.Net.Analysis.OpenNLP requires this to
be useful.
+using Lucene.Net.Analysis.Util;
+using NUnit.Framework;
+
+namespace Lucene.Net.Analysis.Miscellaneous
+{
+ /*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+ public class TestTypeAsSynonymFilterFactory :
BaseTokenStreamFactoryTestCase
+ {
+ private static readonly Token[] TOKENS = { token("Visit",
"<ALPHANUM>"), token("example.com", "<URL>") };
+
+ [Test]
+ public void TestBasic()
+ {
+ TokenStream stream = new CannedTokenStream(TOKENS);
+ stream = TokenFilterFactory("TypeAsSynonym").Create(stream);
+ AssertTokenStreamContents(stream, new string[] { "Visit",
"<ALPHANUM>", "example.com", "<URL>" },
+ null, null, new string[] { "<ALPHANUM>", "<ALPHANUM>",
"<URL>", "<URL>" }, new int[] { 1, 0, 1, 0 });
+ }
+
+ [Test]
+ public void TestPrefix()
+ {
+ TokenStream stream = new CannedTokenStream(TOKENS);
+ stream = TokenFilterFactory("TypeAsSynonym", "prefix",
"_type_").Create(stream);
+ AssertTokenStreamContents(stream, new string[] { "Visit",
"_type_<ALPHANUM>", "example.com", "_type_<URL>" },
+ null, null, new string[] { "<ALPHANUM>", "<ALPHANUM>",
"<URL>", "<URL>" }, new int[] { 1, 0, 1, 0 });
+ }
+
+ private static Token token(string term, string type)
+ {
+ Token token = new Token();
+ token.SetEmpty();
+ token.Append(term);
+ token.Type = type;
+ return token;
+ }
+ }
+}
diff --git
a/src/Lucene.Net.Tests.Analysis.OpenNLP/Lucene.Net.Tests.Analysis.OpenNLP.csproj
b/src/Lucene.Net.Tests.Analysis.OpenNLP/Lucene.Net.Tests.Analysis.OpenNLP.csproj
index 1a3bbbc6b..d88058462 100644
---
a/src/Lucene.Net.Tests.Analysis.OpenNLP/Lucene.Net.Tests.Analysis.OpenNLP.csproj
+++
b/src/Lucene.Net.Tests.Analysis.OpenNLP/Lucene.Net.Tests.Analysis.OpenNLP.csproj
@@ -24,19 +24,21 @@
<Import Project="$(SolutionDir)TestTargetFramework.props" />
<PropertyGroup>
- <TargetFrameworks></TargetFrameworks>
- <TargetFramework>net48</TargetFramework>
-
+ <!-- Allow specific target framework to flow in from
TestTargetFrameworks.props -->
+ <TargetFrameworks Condition=" '$(TargetFramework)' == ''
">net7.0;net6.0</TargetFrameworks>
+ <TargetFrameworks Condition=" '$(TargetFramework)' == '' AND
$([MSBuild]::IsOsPlatform('Windows'))
">$(TargetFrameworks);net48</TargetFrameworks>
+
<AssemblyTitle>Lucene.Net.Tests.Analysis.OpenNLP</AssemblyTitle>
<RootNamespace>Lucene.Net.Analysis.OpenNlp</RootNamespace>
+ </PropertyGroup>
+ <PropertyGroup Condition=" '$(TargetFramework)' == 'net461' Or
$(TargetFramework.StartsWith('net5.')) ">
+ <!-- For CI, we don't publish the projects and exclude them from the test
run by setting IsPublishable=false and IsTestProject=false -->
<IsPublishable>false</IsPublishable>
- <IsPublishable Condition=" '$(TargetFramework)' == 'net48'
">true</IsPublishable>
-
- <!-- Workaround since there are no targets on non-Windows OS. We need at
least 1 TargetFramework
- registered or MSBuild's validation will fail, so we explicitly disable
it as a test project instead. -->
<IsTestProject>false</IsTestProject>
- <IsTestProject
Condition="$([MSBuild]::IsOsPlatform('Windows'))">true</IsTestProject>
+ <!-- For the IDE, the above doesn't work. Redirect to a supported test
framework instead. -->
+ <TargetFramework Condition=" $(TargetFramework.StartsWith('net5.'))
">net7.0</TargetFramework>
+ <TargetFramework Condition=" '$(TargetFramework)' == 'net461'
">net48</TargetFramework>
</PropertyGroup>
<ItemGroup>
diff --git a/src/Lucene.Net/Lucene.Net.csproj b/src/Lucene.Net/Lucene.Net.csproj
index a1889b96b..2491eccf4 100644
--- a/src/Lucene.Net/Lucene.Net.csproj
+++ b/src/Lucene.Net/Lucene.Net.csproj
@@ -60,10 +60,14 @@
<ItemGroup Condition=" '$(TargetFramework)' == 'netstandard2.0' ">
<PackageReference Include="System.Memory"
Version="$(SystemMemoryPackageVersion)" />
+ <!-- NOTE: This is a transitive dependency only, but we are forcing an
upgrade to ensure there are no conflicts with dependencies. -->
+ <PackageReference Include="System.Runtime.CompilerServices.Unsafe"
Version="$(SystemRuntimeCompilerServicesUnsafePackageVersion)" />
</ItemGroup>
<ItemGroup Condition=" '$(TargetFramework)' == 'net462' ">
<PackageReference Include="System.Memory"
Version="$(SystemMemoryPackageVersion)" />
+ <!-- NOTE: This is a transitive dependency only, but we are forcing an
upgrade to ensure there are no conflicts with dependencies. -->
+ <PackageReference Include="System.Runtime.CompilerServices.Unsafe"
Version="$(SystemRuntimeCompilerServicesUnsafePackageVersion)" />
</ItemGroup>
<ItemGroup>
diff --git a/src/dotnet/tools/Lucene.Net.Tests.Cli/Lucene.Net.Tests.Cli.csproj
b/src/dotnet/tools/Lucene.Net.Tests.Cli/Lucene.Net.Tests.Cli.csproj
index d0cf2a720..9e9a0105a 100644
--- a/src/dotnet/tools/Lucene.Net.Tests.Cli/Lucene.Net.Tests.Cli.csproj
+++ b/src/dotnet/tools/Lucene.Net.Tests.Cli/Lucene.Net.Tests.Cli.csproj
@@ -26,12 +26,15 @@
<PropertyGroup>
<!-- Allow specific target framework to flow in from
TestTargetFrameworks.props -->
<TargetFrameworks Condition=" '$(TargetFramework)' == ''
">net7.0;net6.0</TargetFrameworks>
- <!-- If .NET Frameowrk is specified, just target the latest version -->
- <TargetFramework Condition=" $(TargetFramework.StartsWith('net4'))
">net7.0</TargetFramework>
<AssemblyTitle>Lucene.Net.Tests.Cli</AssemblyTitle>
+ </PropertyGroup>
- <IsPublishable Condition="$(TargetFramework.StartsWith('net4')) Or
$(TargetFramework.StartsWith('net5.'))">false</IsPublishable>
- <IsTestProject Condition="$(TargetFramework.StartsWith('net4')) Or
$(TargetFramework.StartsWith('net5.'))">false</IsTestProject>
+ <PropertyGroup Condition=" $(TargetFramework.StartsWith('net4')) Or
$(TargetFramework.StartsWith('net5.')) ">
+ <!-- For CI, we don't publish the projects and exclude them from the test
run by setting IsPublishable=false and IsTestProject=false -->
+ <IsPublishable>false</IsPublishable>
+ <IsTestProject>false</IsTestProject>
+ <!-- For the IDE, the above doesn't work. Redirect to a supported test
framework instead. -->
+ <TargetFramework>net7.0</TargetFramework>
</PropertyGroup>
<ItemGroup>
