Hi,
Your proposal is very close to how we are using the `CallStep` in Sqlg.
The `CallStep` itself is always just a placeholder and is replaced in the strategies and the `CallStep.serviceName` is the marker.
Here is the current code in the strategy which replaces the `CallStep`.
```
private void handleCallStep(ListIterator<Step<?, ?>> stepIterator, CallStep<?, ?> callStep, MutableInt pathCount) {
String serviceName = callStep.getServiceName();
switch (serviceName) {
case SqlgPGRoutingFactory.NAME:
handlePgRouting(serviceName, callStep, pathCount);
break;
case SqlgPGVectorFactory.NAME:
handlePGVector(serviceName, callStep, pathCount);
break;
case SqlgFunctionFactory.NAME:
handleFunction(serviceName, callStep, pathCount);
break;
default:
throw new IllegalStateException("Unknown serviceName: " + serviceName);
}
}
```
I think your approach is very much the same as this so I am onboard with your proposal.
I certianly felt that the grammar should also be generated from the `@GremlinDsl`, but alas having no experience with ANTLR I parked that for when I have more time to investigate. For now back to groovy was the quick fix.
Thanks
On Wed, 2025-08-27 at 15:35 +0200, Andrii Lomakin wrote:
> > Good day.
> >
> >
> > This question has already been raised several times, but typically,
> > it was
> > discussed in the context of a specific extension of steps.
> >
> >
> > Here, I want to propose a general approach that we are going to use
> > in
> > YouTrackDB to extend the Gremlin language.
> >
> >
> > Our intention in this discussion is to receive initial feedback from
> > the TP
> > team. If they agree, we will eventually contribute part of this
> > approach or
> > the complete approach in the next versions of the TP framework.
> >
> >
> > We used the following core principle when developing this approach:
> > existing Gremlin tools should be able to take advantage of all the
> > functionality provided by Gremlin extensions. The only difference is
> > that it
> > will not be as handy for users as using the new features provided.
> >
> >
> > As a result, we paid attention to the two tools already at hand for
> > TP
> > users:
> >
> > 1. DSL annotation processor.
> > 2. The call step.
> >
> >
> > So first of all, if we can not convert our DSL using the previously
> > mentioned annotation processor into the standard steps, we convert it
> > into
> > the `call` of the service. This service is actually a stub, not a
> > real
> > service.
> >
> > The first parameter of this `call` step is the new step name, and the
> > rest
> > are arguments specified in free form, supported by TP, and
> > understandable
> > to humans, as well as the optimization strategy that will convert
> > them to
> > the real step in the phase of traversal preprocessing. This approach
> > allows
> > for the addition of new steps and types of arguments, such as
> > predicates.
> >
> >
> > Let us look at a fake example: If I call `g.addSchemaClass("Test")`,
> > DSL
> > converts it into the standard step -> `g.call("extDSL",
> > Map.of("stepName",
> > "addSchemaClass", "className", "Test"))`, which, during the Traversal
> > preprocessing phase, is converted to the AddSchemaClass step by the
> > related
> > optimization strategy.
> >
> >
> > This approach:
> >
> > 1. Allows the use of new steps in already existing tools.
> > 2. The straightforward design to follow to extend Gremlin drivers
> > allows for
> > quick incorporation of those drivers by the tool's developers and
> > application developers.
> > 3. It allows DB vendors to add new steps themselves easily.
> >
> >
> > However, this does not solve the issue when DSL is passed as text.
> >
> > To solve this last-mile issue, we propose the following approach:
> >
> > 1. ANTLR grammar is distributed as a separate artifact, and users
> > are
> > allowed to extend it with new step definitions by contract. I am
> > not sure
> > about predicates, but that is likely also possible.
> > 2. A new Customizer that provides instances of GremlinLexer,
> > GermlinParser
> > and a new interface, GremlinASTTransformer, if specified is
> > introduced.
> > 3. The responsibility of a new GremlinASTTransformer class would
> > be to
> > transform the AST provided by the extended parser into the
> > standard AST
> > generated by the standard Gremlin ANTLR grammar.
> > 4. After that, the same optimization routine that converts fake
> > service
> > into the new Step instance is applied.
> >
> >
> > This routine can also be made more developer-friendly.
> >
> >
> > For example, there could be handy automations that will allow you not
> > to
> > write AST transformation code manually, but merely map it to the
> > specialized TraversalStrategy interface implementation that will
> > provide
> > all information needed for both transformations:
> >
> > 1. Forward -> from the `call` step to the new Step instance
> > conversion.
> > 2. Backward -> from the Gremllin step in the extended AST to the
> > `call `
> > step in Traversal.
> >
> > Moreover, a new step, "stepExt," can be introduced to make the
> > Gremlin
> > extension more semantically clear. In such a case, a new type of
> > TraversalStrategy can be specialized to work only with a given type
> > of step.
> >
> >
> > To recap, after this massive wall of text, the benefits of this
> > approach
> > are:
> >
> > 1. Despite being somewhat convoluted, extending the Gremlin
> > language
> > will not require spending a large amount of resources for
> > developers with
> > knowledge of ANTLR, especially if AbstractGremlinASTTransformer
> > and
> > StepExtensionTraversalStrategy are implemented, which will do all
> > the heavy
> > lifting of step transformation.
> > 2. Existing gremlin tools can be used to call new steps, though
> > less
> > conveniently than in the presence of new steps in Traveral.
> >
> > Looking forward to reading your feedback.
