Changeset: a271627eff36 for MonetDB URL: https://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=a271627eff36 Added Files: documentation/embedded/authors.rst documentation/embedded/conf.py documentation/embedded/dev_notes.rst documentation/embedded/examples.rst documentation/embedded/index.rst documentation/embedded/installation.rst documentation/embedded/introduction.rst documentation/embedded/monetdbe_api.rst Branch: default Log Message:
A first shot at the documentation of MonetDBe diffs (truncated from 369 to 300 lines): diff --git a/documentation/embedded/authors.rst b/documentation/embedded/authors.rst new file mode 100644 --- /dev/null +++ b/documentation/embedded/authors.rst @@ -0,0 +1,5 @@ +Authors +======= + +- Niels Nes +- Martin Kersten diff --git a/documentation/embedded/conf.py b/documentation/embedded/conf.py new file mode 100644 --- /dev/null +++ b/documentation/embedded/conf.py @@ -0,0 +1,58 @@ +# Configuration file for the Sphinx documentation builder. +# +# This file only contains a selection of the most common options. For a full +# list see the documentation: +# https://www.sphinx-doc.org/en/master/usage/configuration.html + +# -- Path setup -------------------------------------------------------------- + +# If extensions (or modules to document with autodoc) are in another directory, +# add these directories to sys.path here. If the directory is relative to the +# documentation root, use os.path.abspath to make it absolute, like shown here. +# +# import os +# import sys +# sys.path.insert(0, os.path.abspath('.')) + + +# -- Project information ----------------------------------------------------- + +project = 'monetdbe' +copyright = '2020, MonetDB Solutions' +author = 'Niels Nes' + +# The full version, including alpha/beta/rc tags +release = '0.1' + + +# -- General configuration --------------------------------------------------- + +# Add any Sphinx extension module names here, as strings. They can be +# extensions coming with Sphinx (named 'sphinx.ext.*') or your custom +# ones. +extensions = [ +] + +# Add any paths that contain templates here, relative to this directory. +templates_path = ['_templates'] + +# List of patterns, relative to source directory, that match files and +# directories to ignore when looking for source files. +# This pattern also affects html_static_path and html_extra_path. +exclude_patterns = [] + +# This is needed to keep readthedocs happy +master_doc = 'index' + + +# -- Options for HTML output ------------------------------------------------- + +# The theme to use for HTML and HTML Help pages. See the documentation for +# a list of builtin themes. +# +html_theme = 'alabaster' + +# Add any paths that contain custom static files (such as style sheets) here, +# relative to this directory. They are copied after the builtin static files, +# so a file named "default.css" will overwrite the builtin "default.css". +html_static_path = ['_static'] diff --git a/documentation/embedded/dev_notes.rst b/documentation/embedded/dev_notes.rst new file mode 100644 --- /dev/null +++ b/documentation/embedded/dev_notes.rst @@ -0,0 +1,10 @@ +Developer notes +=============== + +``libmonetdbe.so`` is developed using the MonetDB code repository. + +Installation for development +---------------------------- + +In order to create a version of ``libmonetdbe.so`` do the following: + diff --git a/documentation/embedded/examples.rst b/documentation/embedded/examples.rst new file mode 100644 --- /dev/null +++ b/documentation/embedded/examples.rst @@ -0,0 +1,6 @@ +Examples +======== + +The following example illustrates the basics of a MonetDBe application. +See the repositary ... for more starting points of your yourney. + diff --git a/documentation/embedded/index.rst b/documentation/embedded/index.rst new file mode 100644 --- /dev/null +++ b/documentation/embedded/index.rst @@ -0,0 +1,25 @@ +.. monetdbe documentation master file, created by + sphinx-quickstart on Wed May 13 12:42:53 2020. + You can adapt this file completely to your liking, but it should at least + contain the root `toctree` directive. + +Welcome to MonetDBe's documentation! +========================================= + +.. toctree:: + :maxdepth: 2 + :caption: Contents: + + introduction + installation + monetdbe_api + examples + dev_notes + authors + +Indices and tables +================== + +* :ref:`genindex` +* :ref:`modindex` +* :ref:`search` diff --git a/documentation/embedded/installation.rst b/documentation/embedded/installation.rst new file mode 100644 --- /dev/null +++ b/documentation/embedded/installation.rst @@ -0,0 +1,4 @@ +Installation +============ + +The ``libmonetdbe.so`` can be downloaded from the location... diff --git a/documentation/embedded/introduction.rst b/documentation/embedded/introduction.rst new file mode 100644 --- /dev/null +++ b/documentation/embedded/introduction.rst @@ -0,0 +1,81 @@ +Introduction +============ + +``MonetDBe`` is the embedded version of MonetDB. It provides all the +functionality offered by the MonetDB SQL environment, but makes +this available as a single ``libmonetdbe.so`` library. This library +can be directly linked with a C application to benefit from the +functionality. +//Sample mature offerings in the space are SQLanywhere(SAP), SQLite, Cache, ExtremeDB, solidDB, Pervasive PSQL, Raima, Interbas + +``An embedded database system is a database management system (DBMS) which +is tightly integrated with an application software that requires access to stored data, +such that the database system is "hidden" from the application’s end-user +and requires little or no ongoing maintenance. `` [wikipedia] + +The embedded database market has for a long time been dominated by traditional row-stores, +often with limited SQL functionality, scalability or performance. +The space lacks a mature open-source column store deemed essential to push +the performance envelope for data analytics applications. + +What is our proposed solution? +------------------------------ + +Wouldn’t it be great if a full fledged column-store SQL database server be harnessed +to the point it can work as an embedded DBMS. A fully parallel execution engine +to exploit the common multi-core platforms. This way addressing the needs for +data analytics and machine learning, without relying on expensive interprocess communication or ETL. +A strong embeddedness with the application to easily use the UDFs at close range without overhead in data exchanges. + +How we solved it +--------------- + +The solution brought to you is illustrated below. On the left we see MonetDB in +the role of a client/server architecture as it has worked all along. +The interaction between the applications and the server is regulated using standard API’s, such as JDBC/ODBC, and the MonetDB API. +In this architecture, the underlying assumptions are: +. The server is fully isolated from the application code for security, stability, reliability. +. The server serves multiple users concurrently. +. The server manages a large shared storage space. +. The server can use replication for resilience and load sharing. +. The server can communicate with its peers to execute queries over partitioned tables. + +A large part of the server code, however, can be decoupled to reach an +embeddable database system, illustrated at the right with a focus +on the programming language C. +A single library to be linked into the application code directly. With the obvious benefits: + +. No client-server communication overhead. +. No intermediate serialization. +. Full use of the multi-core architectures. +. Seamless integration with the application. +. Single user control over the resources. +. Working with :inmemory: databases with controlled RAM footprint. +. Boosting your data analytics + +TODO include picture + +How does it work? +---------------- +MonetDBe encapsulates the full SQL functionality offered by MonetDB. +To achieve this, we had to create a single shared library (libmonetdbe) to +expose the main sql interfaces using a small api (monetdbe.h). +It is an integral component of the server version nowadays, which avoids maintaining multiple code paths. + +What to expect? +--------------- +The MonetDBe library supports three different storage schemes. +A light-weight scheme for small databases confined to main memory, i.e. connect(‘ :inmemory:’). +The maximum footprint can be set explicitly or is derived from the host (VM) resource limits. +Using a directory on your local machine is the way to go if you need persistence, +but with exclusive access as it concerns ACID properties. If the application works +with :inmemory: or a local database then switching to a MonetDB server merely +requires the connect call to be provided with the access point information. + +The MonetDBe code is available from the development branch of MonetDB as an **alpha-release**.. +For now, the embedded version can serve a single database at a time, but coarse grain locking +is considered to enable sharing the same persistent storage. +There is no such thing as distributed queries over partitioned tables and +stored in different MonetDBe instances or instant database replication between them. +These features we will be added to the interface soon. + diff --git a/documentation/embedded/monetdbe_api.rst b/documentation/embedded/monetdbe_api.rst new file mode 100644 --- /dev/null +++ b/documentation/embedded/monetdbe_api.rst @@ -0,0 +1,140 @@ +MonetDBe API +============ + +The application built around the MonetDBe library. All instructions return a string which contains possible +errors encountered during the interpretation. If all went well they return a NULL. + +Data Types +--------------- +The API wraps the internal data types to those more convenient for programming directly in C. + + +| MonetDBe type | MonetDB internal type | +| --------------- | --------------------- | +|int8_t | bool| +|int8_t | int8_t| +|int16_t | int16_t| +|int32_t | int32_t| +|int64_t | int64_t| +|__int128 | int128_t| +|size_t | size_t [TODO really needed ?]| +|float | float| +|double | double| +|char * | str| +|monetdb_data_blob | blob| +|monetdb_data_date | date| +|monetdb_data_time | time| +|monetdb_data_timestamp | timestamp| + +Data Types +---------- +The MonetDBe interface uses a few structures you need to be aware of. +Data Types +---------- +The MonetDBe interface uses a few structures you need to be aware of. +See the monetdbe.h file + +Connection and server options +--------------------------------- +.. c:char* monetdbe_startup(char* dbdir, bool sequential); + + Initialize the database component. The dbname identifies the path to the directory holding the persistent data. + It will be locked for exclusive access. An in-memory only database is created by passing a NULL argument. + + The sequential argument indicates [WHAT] + +[TODO, consider this seriously! ] +.. c:char* monetdbe_connect_mapi(char* dbdir, char *username, char *password, char *host, int port); + + A simplified wrapper around the official MAPI protocol. It allows for development of the application + using a local directory and simply replacing the startup call with a mapi_connect. + +.. c:char* monetdb_connect(monetdb_connection *conn); + + Create a separate connection channel with the database. Don't use it in parallel threads to submit + queries, because they would become mangled and even could hang your application. + [TODO protect against it] + + +.. c:char* monetdb_disconnect(monetdb_connection *conn); + + From here on the connection can not be used anymore to pass queries or any pending result set is cleaned up. + +.. c:char* monetdb_shutdown(void); + + Force a shutdown of the database server. This will not shutdown your application, you can simply restart + the database server. However, be aware that the content of an ':inmemory:' database is discarded at shutdown. + +Transaction management +--------------------- +.. c: char* monetdb_get_autocommit(monetdb_connection conn, int* result); + _______________________________________________ checkin-list mailing list [email protected] https://www.monetdb.org/mailman/listinfo/checkin-list
