spmallette opened a new pull request, #3742:
URL: https://github.com/apache/tinkerpop/pull/3742
This adds natural language intents to the Gherkin feature tests and an
internal tool, gremlin-fluency, that uses them to measure how well AI agents
write Gremlin.
### Intents
A scenario can now carry an And the intent step: a plain-English description
of what its traversal does.
```
Given the modern graph
And the intent
"""
How many vertices are there of each label? Return one map from label to
count.
"""
And the traversal of
"""
g.V().groupCount().by(T.label)
"""
```
The step has no effect on the test. Every language variant accepts and
ignores it, and the generated GLV test code is unchanged. About 250 scenarios
carry one. Each was selected for step and pattern coverage and reviewed by hand
against one rule: an intent should admit only one reasonable reading, and that
reading should give the expected result on any graph with the same schema, not
just the one the test uses. The guidance is in the "Gremlin Language Test
Cases" section of for-committers.asciidoc. Providers with their own step
definitions need to accept and ignore the new step, which is noted in the 4.x
provider upgrade docs.
### New feature test content
- Air-routes scenarios adapted from Practical Gremlin, plus scenarios over
four new domain schemas: ledger, telemetry, organization and contacts. They're
grouped under features/patterns/ by the kind of question they ask:
reachability, paths, aggregation, ranking and so on.
- A domain schema defines a test graph by its schema alone. Each scenario's
initializer adds just the data it needs on the empty graph. This covers data
types the preloaded graphs lack, such as BigDecimal, UUID, Duration, char,
datetime, multi-properties and meta-properties, on graphs that models have not
memorized.
- The schemas are GQL graph-type files in
gremlin-test/src/main/resources/.../test/schemas, included into the developer
docs. @Data* tags mark scenarios by the data types they depend on.
### gremlin-fluency (gremlin-tools/gremlin-fluency)
This is an internal, non-deployed tool. For each setup in a matrix (model,
harness and context, such as none or gremlin-mcp), it prompts an agent with an
intent only, never the graph name, expected result or scenario name. It then
scores the agent's traversal by running a copy of the scenario through the same
Cucumber step definitions on TinkerGraph. The feature files are never modified.
- Checks against coincidental passes. Passes are re-run against the
reference on seeded, perturbed graphs. On domain graphs the perturbations
follow the schema.
- Outputs. Each run writes JSONL results, a snapshot and a single-file HTML
report.
- More detail. The README covers how it works, how to run it, and what
TinkerPop intends to use it for.
VOTE +1
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
