Alchuang22-dev opened a new pull request, #15651:
URL: https://github.com/apache/iotdb/pull/15651
## Description
This PR implements support for CREATE MODEL statements in Apache IoTDB,
enabling users to register and manage AI/ML models within the IoTDB ecosystem.
The implementation includes model registration, lifecycle management, and
integration with the AI Node component.
**The PR is used for showing the changes in our code, SO PLEASE DO NOT PASS
THE PR !**
If you find that there are **comments or output in Chinese** in the code,
this is just for easy testing on different machines. The release version will
not include these notes.
### Model Registration and Management
The implementation introduces a comprehensive model management system that
allows users to register models using URI-based references. The system supports
various model types including user-defined models, with built-in support for
TimerXL and Sundial forecasting models. The design emphasizes flexibility by
allowing external model repositories and local file system references.
Key design decisions:
- **URI-based model registration**: Models are registered using URIs to
support both local and remote model storage, providing flexibility for
different deployment scenarios
- **Asynchronous model loading**: Models are loaded asynchronously to
prevent blocking the main database operations during potentially time-consuming
model initialization
- **State-based lifecycle management**: Models progress through LOADING →
ACTIVE → INACTIVE states with proper error handling for each transition
### SQL Grammar and Parsing
The SQL grammar has been extended to support CREATE MODEL, DROP MODEL, and
SHOW MODELS statements. The parser integrates seamlessly with IoTDB's existing
SQL infrastructure while maintaining backward compatibility.
Design considerations:
- **Consistent syntax**: The CREATE MODEL syntax follows IoTDB's existing
patterns (similar to CREATE TIMESERIES)
- **Flexible parameter passing**: Support for both URI-based and inline
configuration parameters
- **Error-first parsing**: Comprehensive syntax validation with clear error
messages
Alternative considered: We evaluated using a JSON-based configuration
approach but chose the URI pattern for consistency with IoTDB's existing
LOAD/REMOVE commands.
### AI Node Integration
The implementation leverages IoTDB's AI Node component for actual model
execution and inference. The DataNode acts as a coordinator, delegating model
operations to appropriate AI Nodes while maintaining metadata consistency.
Architecture decisions:
- **Separation of concerns**: Model metadata management in DataNode,
execution in AI Node
- **Distributed model registry**: Models can be deployed across multiple AI
Nodes for scalability
- **Fault tolerance**: Graceful handling of AI Node failures with automatic
retry mechanisms
### Error Handling and Resource Management
The system implements comprehensive error handling for various failure
scenarios including network issues, insufficient resources, and invalid model
configurations.
Error handling strategies:
- **Graceful degradation**: System remains operational even when individual
models fail to load
- **Resource cleanup**: Automatic cleanup of partially loaded models on
failure
- **User-friendly error messages**: Clear error reporting for common issues
like missing files or invalid URIs
<hr>
This PR has:
- been self-reviewed.
- concurrent read
- concurrent write
- [x] concurrent read and write
- [x] added documentation for new or modified features or behaviors.
- [x] added Javadocs for most classes and all non-trivial methods.
- [x] added or updated version, __license__, or notice information
- [x] added comments explaining the "why" and the intent of the code
wherever would not be obvious
for an unfamiliar reader.
- [x] added unit tests or modified existing tests to cover new code paths,
ensuring the threshold
for code coverage.
- [x] added integration tests.
- [x] been tested in a test IoTDB cluster.
<hr>
##### Key changed/added classes (or packages if there are too many classes)
in this PR
org.apache.iotdb.ainode.model.ModelLoader
org.apache.iotdb.it.ainode.CreateModelIT
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
