This is an automated email from the ASF dual-hosted git repository.
aradzinski pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-nlpcraft-website.git
The following commit(s) were added to refs/heads/master by this push:
new d00d96b WIP.
d00d96b is described below
commit d00d96b3cca5c562ce9ab1c10fc31ed4c9ee2480
Author: Aaron Radzinzski <[email protected]>
AuthorDate: Sat Jul 4 11:44:10 2020 -0700
WIP.
---
examples/sql_model.html | 415 ++-----------------------------------------
images/sql_example_model.png | Bin 0 -> 203011 bytes
2 files changed, 18 insertions(+), 397 deletions(-)
diff --git a/examples/sql_model.html b/examples/sql_model.html
index 8d2412f..b72db2f 100644
--- a/examples/sql_model.html
+++ b/examples/sql_model.html
@@ -146,415 +146,36 @@ id: sql_model
we should be ready now to develop our data model.
</p>
</section>
- <section id="model">
- <h3 class="section-title">Data Model</h3>
+ <section id="sql">
+ <h3 class="section-title">Sample Database</h3>
<p>
- We are going to start with declaring the static part of our
semantic model using JSON which we will later load using
- <code>NCModelFileAdapter</code> in our Java-based model
implementation. Create new <code>weather_model.json</code>
- file and add the following model declaration into it:
+ We are going to be building natural language interface against
slightly modified "Northwind" sample database
+ from <a target="_"
href="https://docs.microsoft.com/en-us/dotnet/framework/data/adonet/sql/linq/downloading-sample-databases#get-the-northwind-sample-database-for-sql-server";>Microsoft
SQL Server</a>:
</p>
- <pre class="brush: js, highlight: [10, 42, 50, 59]">
-{
- "id": "nlpcraft.weather.ex",
- "name": "Weather Example Model",
- "version": "1.0",
- "description": "Weather example model.",
- "examples": [
- "What's the local weather forecast?",
- "What's the weather in Moscow?"
- ],
- "macros": [
- {
- "name": "<OF>",
- "macro": "{of|for|per}"
- },
- {
- "name": "<CHANCE>",
- "macro":
"{chance|possibility|probability|odds|likelihood|potential|risk|opportunity}"
- },
- {
- "name": "<PHENOMENON>",
- "macro": "{high sea|severe weather|hail|heat wave|cold
wave|derecho|supercell|avalanche|cyclone|wildfire|landslide|firestorm|dust
storm|thunder snow|winter
storm|cloudburst|shower|condensation|precipitation|drizzle|rainstorm|rain
storm|rainfall|rain|storm|sun|sunshine|cloud|hot|cold|dry|wet|wind||hurricane|typhoon|sand-storm|sand
storm|tornado|humid|fog|snow|smog|black
ice|haze|thundershower|thundersnow|sleet|drought|wildfire|blizzard|avalanche|mist|thunderstorm}"
- },
- {
- "name": "<CUR>",
- "macro": "{current|present|moment|now}"
- },
- {
- "name": "<WEATHER>",
- "macro": "{weather
{condition|temp|temperature|data|*}|condition|temp|temperature}"
- },
- {
- "name": "<FORECAST>",
- "macro": "{forecast|prognosis|prediction}"
- },
- {
- "name": "<HISTORY>",
- "macro": "{history|past}"
- }
- ],
- "elements": [
- {
- "id": "wt:hist",
- "description": "Past weather conditions.",
- "synonyms": [
- "{<WEATHER>|*} <HISTORY>",
- "<HISTORY> {<OF>|*}
{<WEATHER>|<PHENOMENON>}"
- ]
- },
- {
- "id": "wt:curr",
- "description": "Current weather conditions.",
- "synonyms": [
- "{<CUR>|*} {<WEATHER>|<PHENOMENON>}",
- "<CHANCE> <OF> <PHENOMENON>",
- "<PHENOMENON> {<CHANCE>|*}"
- ]
- },
- {
- "id": "wt:fcast",
- "description": "Future weather forecast.",
- "synonyms": [
- "{<PHENOMENON>|<WEATHER>|*} <FORECAST>",
- "<FORECAST> {<OF>|*}
{<WEATHER>|<PHENOMENON>}"
- ]
- }
- ]
-}
- </pre>
- <p>There are number of important points here:</p>
- <ul>
- <li>
- <code>Line 10</code> defines several macros that are used
later on throughout the model's elements
- to shorten the synonym declarations. Note how macros coupled
with option groups
- shorten overall synonym declarations 1000:1 vs. manually
listing all possible word permutations.
- </li>
- <li>
- <code>Lines 42, 50, 59</code> define three model elements: the
past, present and future (forecast) weather
- condition.
- </li>
- </ul>
+ <figure>
+ <img class="img-fluid" src="/images/sql_example_model.png" alt="">
+ <figcaption><b>Fig 1.</b> SQL Schema</figcaption>
+ </figure>
<p>
- Now that our model is ready let's create a Java class that would
load this model and define intents
- that use the model elements we have just defined.
+ You can find SQL script creating this database and populating it
with the sample data at <code>db/northwind.sql</code>
+ file in the root of the example.
</p>
</section>
- <section id="code">
- <h3 class="section-title">Model Class</h3>
+ <section id="h2">
+ <h3 class="section-title">H2 Database</h3>
<p>
- Our Java implementation uses one auxiliary class <a
target="github"
href="https://github.com/apache/incubator-nlpcraft/blob/master/src/main/scala/org/apache/nlpcraft/examples/weather/WeatherResultWrapper.java";>WeatherResultWrapper.java</a>
- that is used for JSON result formatting. Note that the main <a
target="github"
href="https://github.com/apache/incubator-nlpcraft/blob/master/src/main/scala/org/apache/nlpcraft/examples/weather/WeatherModel.java";>WeatherModel</a>
class
- have number of utility methods that we'll skip here (you can see
the entire class by following the link above) and
- we'll concentrate on the intents only:
+ In our example we'll be using H2 database instance. For
convenience, example provides a command line application
<code>db/SqlServer.java</code>
+ that automatically starts local H2 database instance with default
configuration (localhost on port 9092) and initializes it
+ using <code>db/northwind.sql</code> script.
</p>
- <pre class="brush: java, highlight: [1, 2, 3, 4, 5, 10, 11, 12, 13,
14, 19, 20, 21, 22, 23]">
- @NCIntent("intent=fcast term={id == 'wt:fcast'} term(city)={id ==
'nlpcraft:city'}? term(date)={id == 'nlpcraft:date'}?")
- public NCResult onForecastMatch(
- NCIntentMatch ctx,
- @NCIntentTerm("city") Optional<NCToken> cityTokOpt,
- @NCIntentTerm("date") Optional<NCToken> dateTokOpt
- ) {
- return onPeriodMatch(ctx, cityTokOpt, dateTokOpt, 5);
- }
-
- @NCIntent("intent=hist term={id == 'wt:hist'} term(city)={id ==
'nlpcraft:city'}? term(date)={id == 'nlpcraft:date'}?")
- public NCResult onHistoryMatch(
- NCIntentMatch ctx,
- @NCIntentTerm("city") Optional<NCToken> cityTokOpt,
- @NCIntentTerm("date") Optional<NCToken> dateTokOpt
- ) {
- return onPeriodMatch(ctx, cityTokOpt, dateTokOpt, -5);
- }
-
- @NCIntent("intent=curr term={id == 'wt:curr'} term(city)={id ==
'nlpcraft:city'}? term(date)={id == 'nlpcraft:date'}?")
- public NCResult onCurrentMatch(
- NCIntentMatch ctx,
- @NCIntentTerm("city") Optional<NCToken> cityTokOpt,
- @NCIntentTerm("date") Optional<NCToken> dateTokOpt
- ) {
- checkMatch(ctx);
-
- try {
- Coordinate cr = prepGeo(ctx, cityTokOpt);
-
- if (dateTokOpt.isPresent()) {
- DateRange range = extractDate(dateTokOpt.get());
-
- return makeResult(srv.getTimeMachine(cr.latitude,
cr.longitude, range.from, range.to), ctx.getIntentId());
- }
-
- return makeResult(srv.getCurrent(cr.latitude, cr.longitude),
ctx.getIntentId());
- }
- catch (DarkSkyException e) {
- throw new NCRejection(e.getLocalizedMessage());
- }
- catch (NCRejection e) {
- throw e;
- }
- catch (Exception e) {
- throw new NCRejection("Weather provider error.", e);
- }
- }
- </pre>
- <p>
- There three methods define three intents. Each intent is defined
"in place", i.e. as an annotation on the
- method that acts as a callback for that intent:
- </p>
- <ul>
- <li>
- Line 1 defines intent and the callback for the weather
forecast (weather in the future):
- <ul>
- <li>Intent is ordered (default), supports conversation
(default) and has ID <code>fcast</code></li>
- <li>Intent has one mandatory term and two optional terms:
- <ul>
- <li>
- Mandatory term is defined as a token with ID
<code>wt:fcast</code> as defined
- in the model.
- </li>
- <li>
- Two optional terms (their IDs are
<code>city</code> and <code>date</code>) define
- optional city and date for the weather report.
If not provided, their will be taken
- from either conversation context or assume the
default value (current IP geo location
- and current time, if available).
- </li>
- </ul>
- </li>
- </ul>
- </li>
- <li>
- Lines 3-5 define formal parameters for the <code>fcast</code>
intent callback method. Note that we use term IDs
- (<code>city</code> and <code>date</code>) in
<code>@NCIntentTerm</code> annotations to automatically
- assign tokens from the detected terms to their corresponding
method formal parameters.
- </li>
- <li>
- Lines 10-14 and lines 19-23 define two other intents and their
callbacks methods in the same manner.
- </li>
- <li>
- Implementation of all intent callback method is pretty
straightforward and deals mostly with figuring
- out the default location and time if either one isn't provided.
- </li>
- </ul>
- </section>
- <section id="tools">
- <h3 class="section-title">External Tools</h3>
- <p>
- This example uses several external tools to implement its
functionality:
- </p>
- <ul>
- <li>
- <a target=_ href="https://darksky.net";>Apple DarkSky</a> - to
provide actual weather data service
- See <code>org.apache.nlpcraft.examples.misc.darksky</code>
package for details.
- </li>
- <li>
- <a target="_" href="https://tools.keycdn.com/geo";>KeyCDN's IP
Location Finder</a> - to provide IP location
- service. See
<code>org.apache.nlpcraft.examples.misc.geo.keycdn</code> package for details.
- </li>
- <li>
- City to timezone mapper. See
<code>org.apache.nlpcraft.examples.misc.geo.cities</code> package for details.
- </li>
- </ul>
</section>
- <section id="start_probe">
- <h3 class="section-title">Start Data Probe <sub>optional</sub></h3>
- <div class="bq warn">
- <p><b>Embedded Probe</b></p>
- <p>
- If you are using the <a href="#testing">unit test</a> that
comes with this example you <b>do not</b>
- need to start the data probe standalone as this unit test uses
embedded probe mode. In this mode, the unit
- test will automatically start and stop the data probe from
within the test itself.
- </p>
- <p>
- <b>If using <a href="#testing">unit test</a> below - skip this
step, you only need to start the server.</b>
- </p>
- </div>
- <p>
- NLPCraft data models get deployed into data probe. Let's start a
standalone data probe with our newly
- created data model. To start data probe we need to configure Run
Configuration in IDEA with
- the following parameters:
- </p>
- <ul>
- <li>
- <b>Main class:</b> <code>org.apache.nlpcraft.NCStart</code>
- </li>
- <li>
- <b>VM arguments:</b>
<code>-Dconfig.override_with_env_vars=true</code>
- </li>
- <li>
- <b>Environment variable:</b>
<code>CONFIG_FORCE_nlpcraft_probe_models.0=org.apache.nlpcraft.examples.weather.WeatherModel</code>
- </li>
- <li>
- <b>Program arguments: </b> <code>-probe</code>
- </li>
- </ul>
- <div class="bq info">
- <p>
- <b>NOTE:</b> instead of supplying a <a
href="/server-and-probe.html">full configuration file</a> we just
- use the default configuration and override one configuration
property using
- configuration override via environment variables.
- </p>
- </div>
- <p>
- Start this run configuration and make sure you have positive
console output indicating that our model
- has been successfully loaded and probe started.
- </p>
- </section>
- <section id="start_server">
- <h3 class="section-title">Start REST Server</h3>
- <p>
- REST server listens for requests from client applications and
routes them to the requested data models
- via connected data probes. REST server starts the same way as the
data probe. Configure new
- Run Configuration in IDEA with the following parameters:
- </p>
- <ul>
- <li>
- <b>Main class:</b> <code>org.apache.nlpcraft.NCStart</code>
- </li>
- <li>
- <b>Program arguments: </b> <code>-server</code>
- </li>
- </ul>
- <p>
- Once started ensure that your REST server console output shows
that data probe is connected and the
- REST server is listening on the default
<code>localhost:8081</code> endpoint.
- </p>
- <p>
- At this point we've developed our data model, deployed it into the
data probe, and started the REST server.
- To test it, we'll use the built-in <a
href="/tools/test_framework.html">test framework</a>
- that allows you to write convenient unit tests against your data
model.
- </p>
- </section>
- <section id="testing">
- <h3 class="section-title">Testing</h3>
- <p>
- NLPCraft comes with easy to use <a
href="/tools/test_framework.html">test framework</a> for
- data models that can be used with
- any unit testing framework like JUnit or ScalaTest. It is
essentially a simplified
- version of Java REST client that is custom designed for data model
testing.
- </p>
- <p>
- We would like to test with following user requests:
- </p>
- <ul>
- <li><code>"What's the local weather forecast?"</code></li>
- <li><code>"What's the weather in Moscow?"</code></li>
- <li><code>"Chance of snow?"</code> (using conversation
context)</li>
- <li><code>"Moscow?"</code> (using conversation context).</li>
- </ul>
- <p>
- Let's create new Java class <code>WeatherTest.java</code> with the
following code:
- </p>
- <pre class="brush: java, highlight: [47, 43, 55, 59]">
-package org.apache.nlpcraft.examples.weather;
-
-import com.google.gson.Gson;
-import com.google.gson.reflect.TypeToken;
-import org.apache.nlpcraft.common.NCException;
-import org.apache.nlpcraft.model.tools.test.NCTestClient;
-import org.apache.nlpcraft.model.tools.test.NCTestClientBuilder;
-import org.apache.nlpcraft.model.tools.test.NCTestResult;
-import org.apache.nlpcraft.probe.embedded.NCEmbeddedProbe;
-import org.junit.jupiter.api.AfterEach;
-import org.junit.jupiter.api.BeforeEach;
-import org.junit.jupiter.api.Test;
-
-import java.io.IOException;
-import java.lang.reflect.Type;
-import java.util.HashMap;
-import java.util.Map;
-
-import static org.junit.jupiter.api.Assertions.*;
-
-class WeatherTest {
- private static final Gson GSON = new Gson();
- private static final Type TYPE_MAP_RESP = new
TypeToken<HashMap<String, Object>>() {}.getType();
- private NCTestClient cli;
-
- private void checkIntent(String txt, String intentId, boolean
shouldBeSame) throws NCException, IOException {
- NCTestResult res = cli.ask(txt);
-
- assertTrue(res.isOk(), () -> res.getResultError().get());
-
- assert res.getResult().isPresent();
-
- Map<String, Object> map = GSON.fromJson(res.getResult().get(),
TYPE_MAP_RESP);
-
- if (shouldBeSame)
- assertEquals(intentId, map.get("intentId"));
- else
- assertNotEquals(intentId, map.get("intentId"));
- }
-
- @BeforeEach
- void setUp() throws NCException, IOException {
- NCEmbeddedProbe.start(WeatherModel.class);
-
- cli = new NCTestClientBuilder().newBuilder().build();
-
- cli.open("nlpcraft.weather.ex"); // See weather_model.json
- }
-
- @AfterEach
- void tearDown() throws NCException, IOException {
- if (cli != null)
- cli.close();
-
- NCEmbeddedProbe.stop();
- }
-
- @Test
- void test() throws NCException, IOException {
- // Empty parameter.
- assertTrue(cli.ask("").isFailed());
-
- // Only latin charset is supported.
- assertTrue(cli.ask("El tiempo en España").isFailed());
-
- // Should be passed.
- checkIntent("What's the local weather forecast?", "fcast", true);
- checkIntent("What's the weather in Moscow?", "curr", true);
- // Can be answered with conversation.
- checkIntent("Chance of snow?", "curr", true);
- checkIntent("Moscow", "curr", true);
-
- cli.clearConversation();
-
- // Cannot be answered without conversation.
- assertTrue(cli.ask("Moscow").isFailed());
- }
-}
- </pre>
- <p>
- This test is pretty straight forward:
- </p>
- <ul>
- <li>
- On line 47 we open the test client with the model ID (see
<code>weather_model.yaml</code>
- file for where we declared it).
- </li>
- <li>
- Test on line 59 is where we issue our test sentences and we
should see
- the confirmation messages in our test console output.
- </li>
- </ul>
- <div class="bq info">
- <p><b>Embedded Prove</b></p>
- <p>
- This test uses <a href="/tools/embedded_probe.html">embedded
probe</a> which automatically
- start and stops the data probe from within the tests itself.
See lines 43 and 55 for details.
- </p>
- <p>
- <b>NOTE:</b> when using test you don't need to start data
probe standalone in a previous step.
- </p>
- </div>
- <p>
- Right click on this class in the project view and run it. You
should be getting standard output in
- JUnit panel as well as the output in the data probe console.
- </p>
+ <section id="model">
+ <h3 class="section-title">Data Model</h3>
</section>
<section>
<h2 class="section-title">Done! 👌</h2>
<p>
- You've created pretty full featured weather bot data model,
deployed it into the data probe, started the
+ You've created a data model for fairly complete natural language
interface to SQL database, deployed it into the data probe, started the
REST server and tested this model using JUnit 5 and the built-in
test framework.
</p>
</section>
diff --git a/images/sql_example_model.png b/images/sql_example_model.png
new file mode 100644
index 0000000..f2dc118
Binary files /dev/null and b/images/sql_example_model.png differ
