bamaer opened a new pull request, #8811:
URL: https://github.com/apache/hop/pull/8811
…, Apache AGE and Gremlin
Adds pluggable graph database connections and makes the graph transforms and
actions work with any graph database, not only Neo4j.
## Graph database connections
- New metadata type `graph-database-connection` in core
(`GraphDatabaseMeta`, `IGraphDatabase`, `IGraphConnection`) with a plugin type
for database types.
- Types:
- Neo4j, Memgraph, Amazon Neptune: Bolt, in the Neo4j plugin.
- FalkorDB: new `hop-tech-falkordb`, Redis protocol.
- Apache AGE: new `hop-tech-age`, PostgreSQL JDBC.
- Gremlin Server and JanusGraph: new `hop-tech-gremlin`, TinkerPop driver.
- Each type has a Cypher dialect (or Gremlin), which drives the index and
constraint syntax and what the dialogs offer.
- Existing Neo4j connections keep working unchanged. *Convert...* and
*Convert all...* in the Neo4j connection editor turn them into graph database
connections.
## Transforms and actions
All of them work through the generic connection. Plugin IDs and serialized
keys are unchanged, so existing pipelines and workflows load as before. Display
names, icons and docs are now vendor neutral:
| Before | Now |
|---|---|
| Neo4j Cypher | Graph query |
| Neo4j graph output | Graph output |
| Neo4j output | Cypher output |
| Neo4j Cypher builder | Cypher builder |
| Neo4j split graph | Split graph |
| Neo4j generate CSVs | Generate graph CSVs |
| Get Neo4j logging info | Get graph logging info |
| Neo4j Cypher script | Graph script |
| Neo4j index / constraint | Graph index / constraint |
| Check Neo4j connections | Check graph database connections |
| Neo4j perspective | Graph logging perspective |
| Neo4j location | Graph database location |
Neo4j import (neo4j-admin) stays Neo4j only.
Highlights:
- Graph output writes to every database, Gremlin included (upserts as
traversals). Model validation checks the model everywhere and the indexes
wherever the database can list them.
- Nodes, relationships and paths come back as core value types from every
connector. "Return graph" and Graph return values work everywhere.
- Execution logging (`HOP_GRAPH_LOGGING_CONNECTION`, with
`NEO4J_LOGGING_CONNECTION` as fallback), the perspective and the execution
information location work on every Cypher database, AGE included.
- Generate graph CSVs gets a plain header format for loading into other
databases.
- On Memgraph, schema and `SHOW` statements run in auto-commit transactions,
which Memgraph requires.
## New
- Get graph schema: the labels, relationship types and properties of a graph.
- Graph vector search: nearest nodes or relationships through a vector
index. Graph index creates vector indexes.
- Extract graph (AI plugin): entities and relationships from text with an AI
provider, for GraphRAG.
- Create property graph action: SQL/PGQ property graphs on relational
databases (tested on Oracle 23ai).
## Fixes in existing behavior
- Cypher builder wrote parameters as `{param}`, which Neo4j 4+ rejects. It
also failed from the second batch.
- Get graph logging info's previous-execution lookups never matched, so they
always returned no date. Pipelines that relied on that now get real dates.
- Workflow logging now writes the workflow status.
- Graph index, Graph constraint, Graph script and Check connections report
their own success after a failure hop.
- Model validation used `db.indexes()`, which Neo4j 5 removed.
## Tests
- Unit tests for the parsers, converters and dialects in core and each
plugin.
- Container ITs in the FalkorDB, AGE and Gremlin plugins.
- IT projects `neo4j`, `neo4j-bolt`, `memgraph`, `falkordb`, `age`,
`gremlin` (156 tests), with compose files. Oracle test 0011 covers SQL/PGQ.
## Docs
New pages for graph database connections, graph databases (with upgrade
notes), FalkorDB, Apache AGE, Gremlin and the new transforms. The graph pages
moved to vendor-neutral paths (`pipeline/transforms/graph-query`,
`metadata-types/graphs/…`) with redirects from the old URLs.
------------------------
Thank you for your contribution! Follow this checklist to help us
incorporate your contribution quickly and easily:
- [x] Run `mvn clean install apache-rat:check` to make sure basic checks
pass. A more thorough check will be performed on your pull request
automatically.
- [x] If you have a group of commits related to the same change, please
squash your commits into one and force push your branch using `git rebase -i`.
- [x] Mention the appropriate issue in your description (for example:
`addresses #123`), if applicable.
To make clear that you license your contribution under the [Apache License
Version 2.0, January 2004](http://www.apache.org/licenses/LICENSE-2.0)
you have to acknowledge this by using the following check-box.
- [x] I hereby declare this contribution to be licensed under the [Apache
License Version 2.0, January 2004](http://www.apache.org/licenses/LICENSE-2.0)
- [ ] In any other case, please file an [Apache Individual Contributor
License Agreement](https://www.apache.org/licenses/icla.pdf).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
