http://git-wip-us.apache.org/repos/asf/incubator-trafodion/blob/286a6e5c/docs/src/site/markdown/faq.md ---------------------------------------------------------------------- diff --git a/docs/src/site/markdown/faq.md b/docs/src/site/markdown/faq.md new file mode 100644 index 0000000..a532123 --- /dev/null +++ b/docs/src/site/markdown/faq.md @@ -0,0 +1,220 @@ +<!-- + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the + License. +--> +This page provides frequently-asked questions about the Trafodion project and their answers. + +# Project +## What is Project Trafodion? +Project Trafodion is an open source initiative originally cultivated by HP Labs and HP IT to develop an enterprise-class SQL-on-HBase solution targeted for big data transactional or operational workloads as opposed to analytic workloads. + +## When is Trafodion available? +Trafodion is currently available at downloads.trafodion.org. There, you can find an installer and executable code. For installation instructions, see Installation. + +## What are the key features of Trafodion? + +The key features are: + +* Full-functioned ANSI SQL language support +* JDBC/ODBC connectivity for Linux/Windows clients +* Distributed ACID transaction protection across multiple statements, tables and/or rows +* Transaction recovery to achieve database consistency +* Optimization for low-latency read and write transactions +* Support for large data sets using a parallel-aware query optimizer +* Performance improvements for OLTP workloads with compile-time and run-time optimizations +* Distributed parallel-processing architecture designed for scalability + +## What are the key benefits of Trafodion? +Trafodion delivers a full-featured and optimized transactional SQL-on-HBase DBMS solution with full transactional data protection. These capabilities help overcome basic Hadoop limitations in supporting transactional workloads. + +With Trafodion, users gain the following benefits: + +* Ability to leverage in-house SQL expertise versus complex MapReduce programming +* Seamless support for existing transactional applications +* Ability to develop next generation highly scalable, real-time transaction processing applications +* Reduction in data latency for down-steam analytic workloads +* Adoption of data reuse by different application domains + +And operational SQL users also gain the following benefits inherent in Hadoop ecosystem. + +* Reduced infrastructure costs +* Massive scalability and granular elasticity +* Improved data availability and disaster recovery protection + +## Why is it called Trafodion? +Trafodion is Welsh for transactions. + +## What are the primary use cases for Trafodion? +Primary use cases on Trafodion consist of existing HBase workloads enhanced through SQL with transactional data protection. Another class of use cases consist of new or rehosted existing operational applications to address scalability issues, complex application programming, or prohibitive licensing costs. + +## What is the heritage of the Trafodion software? +Trafodion is based on HPâs 20+ year history in developing database products for OLTP. Building on this heritage, Trafodion provides distributed transaction management protection over multiple SQL statements, multiple tables and multiple rows. + +## What do you mean by "Transactional or Operational SQL"? +Transactional or Operational SQL describe workloads were previously described simply as OLTP (online transaction processing) workloads. This expands that definition from the broad range of enterprise-level transactional applications (ERP, CRM, etc.) to include the new transactions generated from social and mobile data interactions and observations and the new mixing of structured and semi-structured data. + +<!-- 20151117 Gunnar Tapper + + Commented out this section since the reference is no longer valid. + + Also, markdown does not like the equal signs in the actual URL, which is why tinyurl + is used. Original URL is: + + http://www.gartner.com/technology/reprints.do?id=1-1M9YEHW&ct=131028&st=sb + +Further details on Operational DBMS can be found in Gartnerâs [Magic Quadrant for Operational Database Management Systems][gartner]. + +[gartner]: http://tinyurl.com/p5cwass + +--> + +---- + +# Using +## Why would Trafodion be used over a commercial OLTP database? +Trafodion is an open source product and like other products in the Hadoop ecosystem, it obtains two key advantages over the tradional proprietary OLTP database systems: Cost structure and Data reuse. + +## Is Trafodion mainly an extension to HBase? +Not quite. Trafodion runs on top of HBase using the HBase APIs to perform the typical SQL functions of insert, update, delete, and query data with transactional protection. However, there are plans to include database capabilities using the coprocessor mechanism provided by HBase. + +## Besides data access via SQL database connection, what other advantages does Trafodion offer over existing distributed database technologies? +With Trafodion, users can make use of standard HBase APIs as well as use Trafodion for SQL access to the HBase tables. This includes Trafodion's ability to use powerful SQL capabilities to do joins against both HBase and Trafodion tables and run queries against them while providing transaction management capability. + +This allows application developers to choose the best access methods for each particular data usage, based on skill set, data size and usage, and access pattern requirements. + +## Can Trafodion modify tables? +Yes, Trafodion provides full-featured ANSI DML operations including INSERT, UPDATE, DELETE, MERGE, and UPSERT statement support. + +## What is the benchmarked performance for a transaction workload? +Benchmark results will be posted as the community publishes them. + +## Where would you position Trafodion according to the CAP theorem? Is it CP (consistent and partition tolerant as HBase) or CA (consistent and highly available)? +Trafodion is hosted on top of HBase and HDFS. HBase is generally viewed as being CA (Consistent, Available) in the context of the CAP theorem. + +Unlike regular HBase, Trafodion extends the definition of consistency to provide ACID protection across transactions comprised of multiple SQL statements, tables, and rows. + +## How does Trafodion scale in terms of database size? +As Trafodion is hosted on HBase/HDFS, in theory, Trafodionâs database should scale as HBase/HDFS scales. To this point (due to available hardware limitations), Trafodion has been tested with configurations up to 10 nodes and 50 terabytes. + +## What about high availability? Does Trafodion have a single point of failure? +HBase and HDFS are considered highly available with many built in features for HA including name node redundancy, HDFS k-safety data replication, HBase replication, HBase snapshots, Zookeeperâs highly reliable distributed coordination of Hadoop hosted services, and so on. + +Additionally, the planned release of HBase 1.0 is advertised as having many additional HA features which Trafodion incorporates as it supports HBase 1.0. Furthermore, many Hadoop distributions have added their own HA features at the HBase or HDFS layers. + +Trafodion leverages HBase and HDFS capabilities for providing extended HA to the connectivity and SQL layers. For example, Trafodion registers its connection services using Zookeeper to ensure persistent connectivity services. Trafodion is designed as an MPP service with replicated service points for HA and no single points of failure. + +## Is Trafodion best suited for reads or for writes? +Trafodion provides a bulk load capability that interfaces directly with HDFS for high-performance data loading. HBase leverages large-scale cached memory to host data in memory until the HFile buffer becomes filled and is then written to HDFS. + +Additionally, Trafodion provides many low-latency read optimizations on top of HBase to significantly improve the both the random and parallel read performance of the database engine. These include: + +* Statistics-based plan generation +* Degree of parallelism optimization +* In-memory data-flow, scheduler-driven executor +* Query plan caching +* Key-based access with SQL âpushdownâ +* Multi-dimensional access method (MDAM) +* Composite key support +* Secondary index support +* Table structure optimizations +* Salting of row-keys for workload balancing + +## Does Trafodion have full text search capabilities? +Trafodion provides SQL search capabilities using search functions such as POSITION (equivalent to INDEX) and SUBSTRING. Once large object (LOB) support is made available, a user-written or third-party UDF could be plugged in to add search capabilities into documents and images. This is a great area for community contribution. + +---- + +# Transactions +## Is Trafodionâs transaction management an eventually consistent model? +No, Trafodion supports full ACID properties. Distributed Transaction Management provides transaction consistency across multiple row updates, updates across multiple tables, and transactions spanning multiple update statements. At the return from a COMMIT WORK statement, all rows and tables are in consistent form with regards to that transaction. + +## Does Trafodion provide transaction management for native HBase tables? +Yes, Trafodion supports transaction management for native HBase tables. + +## Does Trafodion support read consistency? +Yes, all the reads in Trafodion are completely consistent. + +## Does Trafodion offer ability to rollback transactions? +Yes, Trafodion offers full ACID protection across multiple SQL statements, tables, or rows. Transactions can be aborted using the ROLLBACK WORK statement. + +## Is there a transaction log and can the log be shipped to a DR site? +Not really. There are two logs that the transaction manager uses to coordinate transaction history, one at an HBase regionserver level and the other at the transaction manager level. But there is no support for shipping and replaying these logs on a remote DR site in a transactionally consistent manner. + +## Does Trafodion support distributed transaction management across multiple data centers? +There is no support for transaction management between Trafodion instances in different data centers. + +## How does two-phase commit scale linearly on large clusters? +Trafodion employs a Distributed Transaction Manager (DTM) model where a DTM runs on every datanode and each DTM works with its peers in other datanodes to coordinate the two-phase commit protocol. For efficiency purposes, the DTM is invoked only when necessary: + +* If the transaction involves only a single region, then two-phase commit coordination is not necessary. +* The DTM is not involved with read-only transactions. +* The DTM is not involved in loading data into empty tables or during index creation. +* If the transaction impacts only a single table row, then standard HBase ACID protection is used instead of the DTM right now. In the future,these transactions might be fast-tracked using a different mechanism. + +---- + +# Hadoop Integration +## Does Trafodion work as a Hadoop Yarn application? +Trafodion has not been tested with Yarn yet, but there is nothing in Trafodion that precludes it from being treated as a regular application running in a Yarn container. + +## Can Trafodion access Hive stored data? +Trafodion does provide the capability for SQL statements submitted to Trafodion to access, join, and aggregate a combination of Trafodion database objects, native HBase objects, and Hive tables (via the HCatalog). + +## Can we access Trafodion data with MapReduce? +While theoretically possible since Trafodion data is stored in HBase/HDFS, the data would not be easily interpreted due to the encoding mechanisms Trafodion employs for transaction performance and efficiency. + +Trafodion provides automatic access parallelism without the need for writing MapReduce. If the optimized plan calls for parallel execution, the Trafodion Master process divides the work among Executive Server Processes (ESPs) to perform the work in parallel on behalf of the Master process. The results are passed back to the Master for consolidation. + +In some situations where there a highly complex plan specified (for example, large n-way joins or aggregations), multiple layers of ESPs may be requested. See the diagram below. + +<center><img src="images/multi-layer-esps.png"/></center> + +## Can Trafodion data be accessed through Hive? Can we publish a Trafodion table to HCatalog and query it using Hive? +While theoretically possible since Trafodion data is stored in HBase/HDFS, the data would not be easily interpreted due to the encoding mechanisms Trafodion employs for transaction performance and efficiency. + +Trafodion maintains its own catalog for metadata that provides the relational abstraction layer for accessing the physical storage layer via SQL commands. Trafodion objects are complex as they can have tables, indexes, views, constraints, which are represented in Trafodionâs own catalog. HCatalog can represent only simple objects like tables or column families currently but does not contain support for representing other more complex relational objects. + +Alternatively, Trafodion data could be loaded into Hive/HDFS and then accessed using Hive specific operations or native MapReduce jobs. + +---- + +# Security +## How is Database security management supported in Trafodion? +Currently, Trafodion provides security management based on HBase capabilities. There are projects as part of roadmap to provide full GRANT/REVOKE capability at the table and schema level for users and roles, along with support for directory based authentication. + +## Does Trafodion have role-based security authorization? +Yes, Trafodion supports ANSI GRANT/REVOKE on both an individual user and role basis. See Enabling Security Features in Trafodion. For details on the GRANT/REVOKE syntax, see the Trafodion SQL Reference Manual (pdf, 3.98 MB). + +## What is Trafodionâs security authorization granularity? +Trafodion supports authorization (that is, privileges) at the system and database-object level. System privileges are valid across SQL, such as the ability to create schemas and objects. Object privileges include tables, views, and other databse objects. Views can be used to restrict access to the subset of table rows referenced within the view. + +---- + +# Infrastructure +## What hardware do I need to run Trafodion? Is it hardware vendor neutral? +Trafodion is designed to be hosted on vendor neutral, commodity hardware. As Trafodion runs on top of HBase and HDFS, a vendor's recommended reference architectur such as HP Reference Architecture for Hadoop or similar server configuration is recommended as a starting point for a new installation. + +## Does Trafodion need to be installed on all the Hadoop cluster nodes? +Trafodion needs to be installed on all nodes that host an HBase RegionServer. Trafodion comes with an install script to make the installation process easy. For more information, see the Trafodion installation instructions. + +## What does Trafodion software require? +Trafodion software requires an x86-64 bit hardware platform running RedHat Enterprise Linux or CentOS 6.x kernel (64-bit). It supports Cloudera and Hortonworks Hadoop distributions; for details, see Supported Hadoop Distributions. The software is intended to be Linux and Hadoop distribution neutral, but other distributions have not been tested at this time. + +## Are there any recommended configurations in terms of nodes, memory, etc.? +Recommended sizings for Trafodion are still being developed, and vary based on the amount of data, anticipated query workload (types, rates/respose times, concurrency) and the peak number of supported concurrent connections). + +Whatâs the process like when we add a new node to the cluster? Does the system grow elastically? +The underlying HBase/HDFS infrastructure supports elastic scalability allowing for incremental node expansion, that is, datanode expansion. Trafodion could then access the HBase tables (or âregionsâ of the HBase tables) that now span the new node(s). Trafodion services (that is, connection, compiler, master, ESP) can be made to leverage these additional nodes by restarting Trafodion with a new coniguration. + +## Does Trafodion need dedicated hardware? +Trafodion can run on dedicated hardware, cloud or VM configurations. For best and consistent performance, dedicated hardware is recommended where feasible. \ No newline at end of file
http://git-wip-us.apache.org/repos/asf/incubator-trafodion/blob/286a6e5c/docs/src/site/markdown/install-preparation.md ---------------------------------------------------------------------- diff --git a/docs/src/site/markdown/install-preparation.md b/docs/src/site/markdown/install-preparation.md new file mode 100644 index 0000000..3848746 --- /dev/null +++ b/docs/src/site/markdown/install-preparation.md @@ -0,0 +1,138 @@ +<!-- + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the + License. +--> +This page describes the steps you need to perform before installing Trafodion. + +# Cluster Requirements +The following configuration settings have been tested and are known to work with the Trafodion installation: + +Setting | Value +------------------------------|------------------------------------------------- +**Hardware Platform** | x86-64 +**Operating System** | RedHat Enterprise Linux, 6.x kernel (64 bit) or CentOS, 6.x kernel (64 bit) +**Install User IDs** | A user ID with passwordless and sudo access. Please refer to [Install User ID](#Install User ID) below. +**Cluster Size** | A cluster consisting of 1 to N nodes. There is currently no upper limit. Two nodes is the recommended minimum, but a single node cluster should work. +**Disk Space** | Minimum of 20 GB. Please refer to [Checking Disk Space](#Checking_Disk_Space) below. +**Memory** | Minimal 1 GB * number of connectivity servers (MXOSRVR processes) configured per cluster. + +## Checking Disk Space +### Cloudera Manager +This step is required for Cloudera distributions only. + +Before installing Cloudera Manager, a minimum of 20 GB is needed to support the Cloudera management database. Default location: **```/var/lib/cloudera-scm-server-db```**. + +Check the available space for **```/var```** as follows: + +* Start a putty or VNC terminal window agains the node where Cloudera Manager will be installed. +* As a user with **```root```** privileges, verify that 20 GB of space is available to **```/var```**. + + cd /var + df -hP + +If the **```/var```** space is insufficient, then you can provide a soft link to another drive for your Cloudera Manager database. + +* Locate a drive with sufficient disk space. +* Create the soft link. + +``` +cd <new drive> # (e.g. cd /DATA) +mkdir cloudera-scm-server-db +chmod 777 cloudera-scm-server-db +cd /var/lib +ln -s <new drive>/cloudera-scm-server-db . +``` + +**Considerations** + +* If **```/var```** is a sub-directory in your cluster's root filesystem, then the Cloudera Manager database should have sufficient space available. +* If Cloudera Manager has already been installed and it is showing red for log directories, it means that Cloudera Manager was installed using **```/var/lib```** and may not have a large enough **```/var```** file system. In this case, there is an unsupported script that can be used to move the directories. Please refer to the **```clouderaMoveDB.sh```** script in the **```installer/tools```** directory, which is created when the installer tar file is unpacked. Run **```clouderaMoveDB.sh```** without syntax to display help information. + +# Security Requirements +## Install User ID +The Trafodion installation requires a user ID (referred to as **```<sudo-user>```**) with these attributes on all nodes where Trafodion is installed: + +* **```/etc/sudoers```* on all nodes: + * **```sudo```** access â allows the install scripts to run privileged commands. + * **requiretty** disabled â allows Trafodion install scripts to use embedded **```sudo```** commands +* passwordless ssh to all nodes â allows Trafodion install scripts to communicate with remote nodes without prompting for password. + +Please create a the **```<sudo-user>```** per the practices of your environment. For example, you may need to request permission from your cluster-management team to obtain this type of access. [Passwordless SSH(passwordless-ssh.html) describes how to set up passwordless ssh. + +## Configure LDAP Identity Store +If you plan to enable Trafodion authentication, then you need to have an LDAP identity store available to perform authentication. + +The Trafodion installer prompts you to set up an authentication configuration file that points to an LDAP server (or servers), which enables security (that is, authentication and authorization) in the Trafodion system. + +If you wish to manually set up the authentication configuration file and enable security, then please refer to [Enabling Security Features in Trafodion](enable-secure-trafodion.html). + +# PC Installation +If you are using a Windows PC, then additional PC software may be needed throughout the installation process. It is recommended that you pre-install the PC software that you need before continuing with the Trafodion installation. + +Install the following software: + +* putty and puttygen (download from the [PuTTY website](http://www.chiark.greenend.org.uk/~sgtatham/putty/download.html)) +* VNC client (download from http://www.realvnc.com) +* Firefox or Chrome browser +* SFTP client to transfer files from your PC to the server: WinSCP or FileZilla + +# Install Hadoop Distribution +Trafodion requires that the following Hadoop services are installed: + +* HDFS +* MapReduce +* ZooKeeper +* HBase +* Hive + +Please ensure that these services are installed for one of the following Hadoop distributions. + +<table><tr><td><strong>IMPORTANT</strong><br />Trafodion does not support a Hadoop server running on a node that is not part of the Trafodion cluster. You must specify the Hadoop nodes to be part of the Trafodion list of nodes during the installation of Trafodion (using the <strong><code>--nodes</code></strong> parameter). The Trafodion installer needs to run from the same node where you installed the Hadoop distribution).</td></tr></table> + +<table> + <tr> + <th>Distribution</th> + <th>Version</th> + <th>HBase Version</th> + <th>Installation</th> + <th>Notes</th> + </tr> + <tr> + <td><strong>Cloudera</strong></td> + <td>CDH 5.2 or 5.3</td> + <td>0.98.6</td> + <td>Refer to <a href="http://www.cloudera.com/content/cloudera/en/documentation/core/latest/topics/installation.html"; target="_blank">CDH 5.3.x Installation and Upgrade Guide</a> for installation instructions.</td> + <td> + <ul> + <li>Select <strong>Cloudera Standard</strong> edition.</li> + <li>Install using <strong>packages</strong>. Do not install using tarballs and parcels. Trafodion does not currently support tarball/parcel installation.</li> + <li>CDH 5.2 is supported but has not been officially tested with Trafodion Release 1.3.</li> + </ul> + </td> + </tr> + <tr> + <td><strong>Hortonworks Data Platform (HDP)</strong></td> + <td>HDP 2.2</td> + <td>0.98.4</td> + <td> + <ul> + <li><strong>Install</strong>: <a href="http://docs.hortonworks.com/HDPDocuments/Ambari-1.7.0.0/Ambari_Install_v170/Ambari_Install_v170.pdf";>Ambari 1.7.0 Install Guide</a></li> + <li><strong>Upgrade:</strong> <a href="http://docs.hortonworks.com/HDPDocuments/Ambari-1.7.0.0/Ambari_Upgrade_v170/Ambari_Upgrade_v170.pdf";>Ambari 1.7.0 Upgrade Guide</a></li> + </ul> + </td> + <td></td> + </tr> +</table> + +# Validation +Once the Hadoop distribution has been installed and started, please ensure that the required services are running. At this point, you're ready to [Install Trafodion](install.html). \ No newline at end of file http://git-wip-us.apache.org/repos/asf/incubator-trafodion/blob/286a6e5c/docs/src/site/markdown/install.md ---------------------------------------------------------------------- diff --git a/docs/src/site/markdown/install.md b/docs/src/site/markdown/install.md new file mode 100644 index 0000000..07fe6af --- /dev/null +++ b/docs/src/site/markdown/install.md @@ -0,0 +1,203 @@ +<!-- + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the + License. +--> +This page describes how to install the Trafodion product environment. [Create Development Environment](create-dev-environment.html) describes how to create the Trafodion Development Test Environment. + +The Trafodion product environment is installed using the Trafodion Installer, which operates on Trafodion binaries only. Refer to the [Download](download.html) page for instructions about how you download/create the Trafodion binaries. + +# Preparation +The Trafodion Installer assumes that you've performed the following steps before a Trafodion install: + +1. **Cluster Requirements**: ensure that you cluster meets the Trafodion requirements. +2. **Security Requirements**: Create the user ID that is used to run the Trafodion Installer. Review and configure LDAP, if desired. +3. **Prepare PC**: install software used when installing Trafodion on your PC. +4. **Install Hadoop**: Install Hadoop software supported by Trafodion. +5. **Validate Hadoop**: Validate that required Hadoop services are running. + +Please refer to [Install Preparation](install-preparation.html) for information about how to perform the steps above. + +# Security Considerations +You use the following user IDs and passwords during the Trafodion installation. + +Logon | User ID | Password | Description +-----------------------------------|----------------------------|-----------------------------|-------------------------------------------------------- +Cloudera Manager Web GUI logon | **```admin (default)```** | **```admin (default)```** | After installing Cloudera, you will be instructed to log on to the Cloudera Manager Web GUI. Use the default user ID and password (**```admin```**, **```admin```**). If you already had Cloudera installed, please use your previously defined user ID and password. +Ambari Web GUI logon | **```admin (default)```** | **```admin (default)```** | After downloading and setting up the Ambari repository, you will be instructed to log on to the Ambari Web GUI. Use the default user ID and password (**```admin```**, **```admin```**). If you already had Ambari installed, please use your previously defined user ID and password. +User ID with sudo Access | **```<sudo-username>```** | **```<password>```** | In the installation steps, you may be instructed to use **```sudo```** or **```sudo userid```** access. You will be using your user ID, which has been enabled with **```sudo```** access and [passwordless ssh](passwordless-ssg.html) to all nodes of the cluster. +Trafodion Logon | **```trafodion```** | **```traf123 (default)```** | The Trafodion installer **automatically creates** this user IDwhen Trafodion is installed. Do not create this user ID manually. +VNC Server Logon | **```trafodion```** | **```traf123 (default)```** | After installing Trafodion, you will be instructed to log on to the VNC server at port 1. Use the **```trafodion```** user ID and **```password```**. + +# Install Procedure +## Upgrade +Follow these steps if you have an older version of Trafodion that you want to upgrade: + +1. *Shutdown Trafodion*: Please refer to [Manage Trafodion](management.html). +2. *Backup Data*: Backup metadata and user objects. Please refer to [Backup/Restore](backuprestore.html). +3. *Install Trafodion*: The **```trafodion_install```** script automatically detects the previous version of Trafodion and performs an upgrade operation, including initializing Trafodion, upgrading the metadata, and restarting all processes. Please refer to [Installation](#Installation) below for more information about **```trafodion_install```**. + + If something goes wrong during the installation, restore your data from the backed up objects and repeat the previous step. + +<table><tr><td><strong>NOTE</strong> + <p>If you are doing an upgrade outside of the installation environment, then you will need to restart your services and processes, such as sqlci, the DCS server, and the qms servers. If you do not restart those services and processes, your system will not work properly.</p> +</td></tr></table> + +## Installation + +<table><tr><td><strong>NOTE</strong><br /> If performing an upgrade: Please refer to [Upgrading Trafodion](#Upgrading_Trafodion) before proceeding.</td></tr></table> + +At a high level, the following instructions consist of a single step: + +Command | User ID | Comments +----------------------------|---------------------------|--------------------------------------------------------------------------------------------------------- +**```trafodion_install```** | **```<sudo-username>```** | Installs necessary RPMs, creates the **```trafodion```** user ID, sets up passwordless ssh for the **```trafodion```** user ID, copies the Trafodion distribution files across the cluster, generates startup files, and starts Trafodion, including Database Connectivity Services (DCS) (the ODBC/JDBC server), among other things. + + +<table><tr><td> + <strong>IMPORTANT</strong> + <p>The Trafodion commands should all be done from the first node of the cluster. Do not run them from a machine that is not part of the Trafodion cluster. Trafodion must be installed on all nodes that host an HBase RegionServer (that is, where a supported Hadoop distribution is installed).</p> + <p>For example, if Cloudera or Hortonworks is installed on nodes <strong><code>n001</code></strong>, <strong><code>n002</code></strong>, and <strong><code>n003</code></strong>, then you must install Trafodion on those same nodes (<strong><code>--nodes "n001 n002 n003"</code></strong>).</p> +</td></tr></table> + +<table> + <tr> + <th width="5%">Step</th> + <th width="35%">Description</th> + <th width="60%">Commands</th> + </tr> + <tr> + <td><strong>Download</strong></td> + <td>If you haven't done so already, then please download the Trafodion software.</td> + <td>Please refer to <a href="download.html">Download</a>.</td> + </tr> + <tr> + <td><strong>Upgrade</strong></td> + <td> + <p>If you are upgrading to a newer version of Trafodion, copy the installation files for the previous version to a directory outside of <strong><code>$HOME/trafodion_downloads</code></strong>.</p> + <p>For example:</p> + <pre>mkdir $HOME/trafodion_downloads_v1.0</pre> + </td> + <td><pre>cp âr $HOME/trafodion_downloads/* $HOME/trafodion_downloads_v1.0 +rm ârf $HOME/trafodion_downloads/*</pre> + </td> + </tr> + <tr> + <td><strong>Create <code>trafodion_downloads</code></strong></td> + <td>Create the <strong><code>trafodion_downloads</code></strong> directory if you have not done so already and place the downloaded <strong><code>trafodion-1.1.<n>.tar.gz</code></strong> and <strong><code>installer-1.1.<n>_v<num>.tar.gz</code></strong> tar files into it.</td> + <td> + <pre> +mkdir $HOME/trafodion_downloads +mv <your-download-path>/trafodion-1.1.<n>.tar.gz $HOME/trafodion_downloads +mv <your-download-path>/installer-1.1.<n>_v<num>.tar.gz $HOME/trafodion_downloads +cd $HOME/trafodion_downloads + </pre> + </td> + </tr> + <tr> + <td><strong>Unpack tar File</strong></td> + <td>Untar the downloaded installer file.</td> + <td><pre>tar -xzf installer-1.1.<n>_v<num>.tar.gz</pre></td> + </tr> + <tr> + <td><strong>Run <code>trafodion_install</code></strong></td> + <td> + <p>The scripts is run without parameters. Instead, you'll be prompted for information.</p> + <p><strong>NOTE</strong></p> + <p>The trafodion_install script automatically invokes the <a href="scanner.html">Trafodion Scanner</a> to check the cluster for attributes that commonly cause installation issues, runtime performance problems, and so on. Also, various system information is collected for future reference, such as to aid with troubleshooting. The installation will abort if any configuration errors are discovered. You will need to correct such errors before re-running the <strong><code>trafodion_install</code></strong>script.</p> + </td> + <td> + <pre> +cd installer +./trafodion_install + </pre> + <p>Type "accept" to accept the license agreement when prompted.</p> + <p>Provide the requested information according to the prompts.</p> + <p><strong>NOTE</strong></p> + <p>If you choose not to start Trafodion after the installation (that is, if you enter <strong><code>N</code></strong> for <strong><code>Start Trafodion after install (Y/N)</code></strong>), you will need to manually start and initialize Trafodion after <strong><code>trafodion_install</code></strong> completes.</p> + </td> + </tr> + <tr> + <td><strong><code>trafodion_install</code> Completion</strong></td> + <td>Wait for <strong><code>trafodion_install</code></strong> to complete. </td> + <td><p>You will see the following message:</p><pre>***INFO: Installation completed successfully.</pre></td> + </tr> + <tr> + <td><strong>Connect</strong></td> + <td>Once <strong><code>trafodion_install</code></strong> completes without error, your Trafodion system should be up and running.</td> + <td> + <p>Start a new VNC session using port :1.</p> + <p>Log in as the <strong><code>trafodion</code></strong> user with the password for the VNC, which is <strong><code>traf123</code></strong> by default unless you changed it.</p> + </td> + </tr> +</table> + +## Start and Initialize +If you chose not to start Trafodion after the installation, start and initialize Trafodion as follows: + + cds + sqstart + + [trafodion@n001 ~]$ sqlci + Trafodion Conversational Interface 1.1.0 + (c) Copyright 2015 Apache Software Foundation + >>initialize trafodion ; + . + . + . + +## Validate Installation +Perform a quick sanity check using the Trafodion Conversational Interface (**```sqlci```**). Create a table with a couple of records. + +**Example** + + [trafodion@n001 ~]$ sqlci + Trafodion Conversational Interface 1.1.0 + (c) Copyright 2015 Apache Software Foundation + >> create table test1 (f1 int, f2 int); + + --- SQL operation complete. + >> insert into test1 values(1,1); + + --- 1 row(s) inserted. + >> insert into test1 values(2,2); + + --- 1 row(s) inserted. + >> select * from test1; + + F1 F2 + ----------- ----------- + 1 1 + 2 2 + + --- 2 row(s) selected. + >> get tables; + + Tables in Schema TRAFODION.SEABASE + ================================== + + TEST1 + + --- SQL operation complete. + >> exit; + +The SQL commands above should run successfully. + +## Install Client Software +Download and install the Trafodion JDBC and/or ODBC drivers on your client workstation to be able to connect to Trafodion from a client application. Please refer to the [Trafodion Client Installation Guide](docs/Trafodion_Client_Installation_Guide.html), which describes how to install the JDBC and ODBC drivers, how to connect to Trafodion, and how to run sample programs to test the connection. + +# Troubleshooting +Please refer to [Troubleshoot Trafodion Installation](install-troubleshoot.html). + +# Uninstall +Please refer to [Unistall Trafodion](uninstall.html). + http://git-wip-us.apache.org/repos/asf/incubator-trafodion/blob/286a6e5c/docs/src/site/markdown/ldapcheck.md ---------------------------------------------------------------------- diff --git a/docs/src/site/markdown/ldapcheck.md b/docs/src/site/markdown/ldapcheck.md new file mode 100644 index 0000000..f5f4d1a --- /dev/null +++ b/docs/src/site/markdown/ldapcheck.md @@ -0,0 +1,41 @@ +<!-- + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the + License. +--> +This page describes ldapcheck, which is a tool that is used to validate the Trafodion authentication configuration and attempt to connect to a configured LDAP server. + +# Usage + +``` +ldapcheck [<option>]... +<option> ::= --help|-h display usage information + --username=<LDAP-username> + --password[=<password>] + --primary Use first configuration + --local Use first configuration + --enterprise Use first configuration + --secondary Use second configuration + --remote Use second configuration + --cluster Use second configuration + --verbose Display non-zero retry counts + and LDAP errors +``` + +# Considerations + +* Aliases for primary include enterprise and local. Aliases for secondary include cluster and remote. If no configuration is specified, primary is assumed. +* The equals sign is required when supplying a value to username or password. +* To be prompted for a password value with no echo, specify the password argument but omit the equals sign and value. +* Passwords that contain special characters may need to be escaped if the password is specified on the command line or within a script file. +* If the password keyword is not specified, only the username will be checked. The tool can therefore be used to test the LDAP configuration and connection to the configured LDAP server(s) without knowing a valid username or password. + http://git-wip-us.apache.org/repos/asf/incubator-trafodion/blob/286a6e5c/docs/src/site/markdown/ldapconfigcheck.md ---------------------------------------------------------------------- diff --git a/docs/src/site/markdown/ldapconfigcheck.md b/docs/src/site/markdown/ldapconfigcheck.md new file mode 100644 index 0000000..806150f --- /dev/null +++ b/docs/src/site/markdown/ldapconfigcheck.md @@ -0,0 +1,48 @@ +<!-- + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the + License. +--> +This page describes the **```ldapconfigcheck```** tool, which validates the syntactic correctness of a Trafodion authentication configuration file. Trafodion does not need to be running to run the tool. + +# Considerations +If the configuration filename is not specified, the tool will look for a file using environment variables. Those environment variables and the search order are: + +1. TRAFAUTH_CONFIGFILE + + A fully qualified name is expected. +2. TRAFAUTH_CONFIGDIR + + Filename **```.traf_authentication_config/```** is appended to the specified directory +3. MY_SQROOT + + **```/sql/scripts/.traf_authentication_config```** is appended to the value of **```MY_SQROOT```**. + +# Errors +One of the following is output when the tool is run. Only the first error encountered is reported. + +Code | Text +-----------------|-------------------------------------------------------------------------------- +**```0```** | File *filename* is valid. +**```1```** | File *filename* not found. +**```2```** | File: *filename*<br /><br />Invalid attribute name on line *line-number*. +**```3```** | File: *filename*<br /><br />Missing required value on line *line-number*. +**```4```** | File: *filename*<br /><br />Value out of range on line *line-number*. +**```5```** | File: *filename*<br /><br />Open of traf_authentication_config file failed. +**```6```** | File: *filename*<br /><br />Read of traf_authentication_config file failed. +**```7```** | No file provided. Either specify a file parameter or verify environment variables. +**```8```** | TLS was requested in at least one section, but TLS_CACERTFilename was not provided. +**```9```** | Missing host name in at least one section.<br /><br />Each LDAP connection configuration section must provide at least one host name. +**```10```** | Missing unique identifier in at least one section.<br /><br />Each LDAP connection configuration section must provide at least one unique identifier. +**```11```** | At least one LDAP connection configuration section must be specified. +**```12```** | Internal error parsing . traf_authentication_config. + http://git-wip-us.apache.org/repos/asf/incubator-trafodion/blob/286a6e5c/docs/src/site/markdown/manage-dev-environment.md ---------------------------------------------------------------------- diff --git a/docs/src/site/markdown/manage-dev-environment.md b/docs/src/site/markdown/manage-dev-environment.md new file mode 100644 index 0000000..2559c8d --- /dev/null +++ b/docs/src/site/markdown/manage-dev-environment.md @@ -0,0 +1,51 @@ +<!-- + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the + License. +--> +This page describes how you manage the Trafodion development test environment, which is intended for people that are contributing to the Trafodion source tree. Please refer to [Download](download.html) if you want to try the Trafodion product environment. + +# Prerequisites +You must have created the [Trafodion Development Environment](create-dev-environment.html) before using the instructions on this page. + +# Hadoop Enviroment +## Pre-Existing Hadoop +If you are doing Trafodion development on a pre-existing Hadoop distribution, then do the following: + +* **Distribution**: Use the distribution management too. For example: Apache Ambari or Cloudera Manager. +* **Regular Hadoop**: Use the start/stop script for each Hadoop environment. + +## Local Hadoop +Use the following commands to manage the Hadoop environment. + +Command | Usage +-----------------------------------|----------------------------------------- +**```swstartall```** | Start the complete Hadoop environment. +**```swstopall```** | Stops the complete Hadoop environment. +**```swstatus```** | Checks the status of the Hadoop environment. +**```swuninstall_local_hadoop```** | Removes the Hadoop installation. + +# Trafodion +Please refer to [Trafodion Management](management.html). + +# New Source Download +You need to do the following each time you download new source code. + + cd <Trafodion source directory> + source ./env.sh + cd $MY_SQROOT/etc + # delete ms.env, if it exists + rm ms.env + cd $MY_SQROOT/sql/scripts + sqgen + + http://git-wip-us.apache.org/repos/asf/incubator-trafodion/blob/286a6e5c/docs/src/site/markdown/management.md ---------------------------------------------------------------------- diff --git a/docs/src/site/markdown/management.md b/docs/src/site/markdown/management.md new file mode 100644 index 0000000..e651cc4 --- /dev/null +++ b/docs/src/site/markdown/management.md @@ -0,0 +1,62 @@ +<!-- + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the + License. +--> +This page describes how you manage Trafodion. + +# Management Scripts +The following table provides an overview of the different Trafodion management scripts. + +Component | Start | Stop | Status +--------------------------------------|----------------------|---------------------|--------------------- +All of Trafodion | **```sqstart```** | **```sqstop```** | **```sqcheck```** +DCS (Database Connectivity Services) | **```dcsstart```** | **```dcsstop```** | **```dcscheck```** +REST Server | **```reststart```** | **```reststop```** | â +LOB Server | **```lobstart```** | **```lobstop```** | â +RMS Server | **```rmsstart```** | **```rmsstop```** | **```rmscheck```** + +**Example: Start Trafodion** + + cd $MY_SQROOT/sql/scripts + sqstart + sqcheck + +# Validate Trafodion Installation +You can use **```sqlci```** or **```trafci```** (connects via DCS) to validate your installation. + + get schemas; + create table table1 (a int); + invoke table1; + insert into table1 values (1), (2), (3), (4); + select * from table1; + exit; + +Assuming no errors, your installation has been successful. You can start working on your modifications. + +# Troubleshooting Tips +If you are not able to start up the environment or if there are problems running **```sqlci```** or **```trafci```**, then verify that the all the processes are up and running. + +* **```swstatus```** should show at 6 java servers and 2 mysql processes. +* **```sqcheck```** should indicate all processes are running. + +If processes are not running as expected, then: + +* **```sqstop```** to shut down Traodion. If some Trafodion processes do not terminate cleanly, then run **```ckillall```**. +* **```swstopall```** to shut down the Hadoop ecosystem. +* **```swstartall```** to restart the Hadoop ecosystem. +* **```sqstart```** to restart Trafodion. + +If problems persist please review logs: + +* **```$MY_SQROOT/sql/local_hadoop/\*/log```**: Hadoop, HBase, and Hive logs. +* **```$MY_SQROOT/logs```**: Trafodion logs. \ No newline at end of file http://git-wip-us.apache.org/repos/asf/incubator-trafodion/blob/286a6e5c/docs/src/site/markdown/new-features.md ---------------------------------------------------------------------- diff --git a/docs/src/site/markdown/new-features.md b/docs/src/site/markdown/new-features.md new file mode 100644 index 0000000..ada9081 --- /dev/null +++ b/docs/src/site/markdown/new-features.md @@ -0,0 +1,15 @@ +<!-- + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the + License. +--> +This page will provide information about features that have recently been added to Trafodion. \ No newline at end of file http://git-wip-us.apache.org/repos/asf/incubator-trafodion/blob/286a6e5c/docs/src/site/markdown/passwordless-ssh.md ---------------------------------------------------------------------- diff --git a/docs/src/site/markdown/passwordless-ssh.md b/docs/src/site/markdown/passwordless-ssh.md new file mode 100644 index 0000000..9935d8b --- /dev/null +++ b/docs/src/site/markdown/passwordless-ssh.md @@ -0,0 +1,131 @@ +<!-- + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the + License. +--> +This page describes how to setup passwordless ssh for a Trafodion user. + +# Overview +Please use these commands to enable passwordless ssh access for a user ID; for example: **```trafodion```** + +You need to define the authorization keys on each node on the instance. To simplify this procedure, this procedure starts from scratch; that is, it removes the **```ssh```** configuration so that new files can be created. The procedure on this page defines what you need to do to set up passwordless ssh manually: + +* Run all commands from a single node in the instance. This node is labeled **master node** in the procedure. +* The commands will reach out to all other nodes in the instance. You will have to provide the correct password manually until the procedure has been completed. +* Ensure that you run the validation step. If passwordless ssh isn't working, then you likely missed a step in the procedure; you'll have to start over. + +<table> + <tr> + <td><strong>Note</strong> <br /><br />You run these commands for each node in the instance, including the master node; that is, you run these commands as well if you are on a single-node instance. + </td> + </tr> +</table> + +In this procedure, you perform the following steps: + +* Generate the **```~/.ssh/id_dsa.pub```** file. +* Populate the local **```~/.ssh/authorized_keys```** file. +* Copy files to all nodes in the instance. +* Add **```localhost```** key to each node's **```~/.ssh/known_hosts```** file. +* Turn on **```NoHostAuthenticationForLocalhost```** on each node. +* Secure the files for passwordless ssh access. +* Validate passwordless ssh access. + +The result of the above procedure should be that the content of the **```~/.ssh/authorized_keys```** file is the same on all nodes in your instance and that the security is set up the same way. This allows you to use ssh to login in to any node from any node in the instance without having to specify a password. This is key for many scripts and support commands, which assume that passwordless ssh is working properly. For example, most support commands uses the **```pdsh $MY_NODES```** or **```pdsh $MY_HADOOP_NODES```** command for status checks. + +# Command Summary +A summary of the commands used in this procedures to help experienced users: + + # Skip this step if logged in as a user that has run EsgynDB sqenv.sh, which defines NODE_LIST. + # Modify the NODE_LIST content to list the nodes on which you are installing Trafodion. + export NODE_LIST="node01.host.com node02.host.com node03.host.com node04.host.com node05.host.com" + + # Get the user ID we're doing this work for. Makes the rest of the commands cleaner. + user=$(whoami) + + # Generate ~/.ssh/id_dsa.pub on each node in the instance + for node in $NODE_LIST; do ssh $user@$node "rmdir ~/.ssh; mkdir ~/.ssh; cd ~/.ssh; echo -e 'y\n' | ssh-keygen -t rsa -N '' -f $~/.ssh/id_rsa"; done + + # Populate ~/.ssh/authorized_keys file on the master node. + for node in $NODE_LIST; do ssh $user@$node "cat ~/.ssh/id_dsa.pub" >> ~/.ssh/authorized_keys; done + + # Copy files to all nodes in the instance but the master node. + my_node=$(echo $HOSTNAME | cut -d '.' -f 1) + for node in $NODE_LIST; do if [ $node <> $my_node ]; then scp ~/.ssh/authorized_keys ~/.ssh/known_hosts $user@$node:~/.ssh/.; fi; done + + # Add localhost public key to each node's ~/.ssh/known_hosts file. + for node in $NODE_LIST; do ssh $user@$node "echo localhost $(cat /etc/ssh/ssh_host_rsa_key.pub) >> ~/.ssh/known_hosts"; done + + # Turn on NoHostAuthenticationForLocalhost on each node. + for node in $NODE_LIST; do ssh $user@$node "echo "NoHostAuthenticationForLocalhost=yes" >> ~/.ssh/config"; done + + # Change security of ~/.ssh directory and key files. + for node in $NODE_LIST; do ssh $user@$node "chmod 755 ~/.ssh; chmod 600 ~/.ssh/authorized_keys; cd ~/.ssh; chmod 700 .."; done + + # Validate that passwordless ssh works. No password prompts should be seen. Try from all or some nodes in the instance. + for node in $NODE_LIST; do ssh $user@$node "ls -al /etc/passwd"; done + + +# Detailed Commands + +## Define ```NODE_LIST``` +The steps below assumes that the **```NODE_LIST```** environmental variable contains the node names onto which Trafodion should be installed. You populate this variable using one of the following methods: + +1. Invoke the Trafodion **```sqenv.sh```** script, which is present after Trafodion has been installed. +2. Create **```NODE_LIST```** manually; for example: + + export NODE_LIST="node01.host.com node02.host.com node03.host.com node04.host.com node05.host.com" + +## Generate the ```~/.ssh/id_dsa.pub file``` +In this step, you create the authorization key for each node in the instance. This key will be located in the **```~/.ssh/id_dsa.pub```** file. Use the following commands for each node of the instance: + + $ user=$(whoami) + $ for node in $NODE_LIST; do ssh $user@$node "rmdir ~/.ssh; mkdir ~/.ssh; cd ~/.ssh; echo -e 'y\n' | ssh-keygen -t rsa -N '' -f $~/.ssh/id_rsa"; done + +## Populate the local ```~/.ssh/authorized_keys file``` +In this step, you populate the **```~/.ssh/authorized_keys```** file on master node by copying the content from each node's **```~/.ssh/id_dsa.pub```** file. + +Use the following command string to copy the content of **```~/.ssh/id_dsa.pub```** file for each node into the **```~/.ssh/authorized_keys```** on the master node: + + $ for node in $NODE_LIST; do ssh $user@$node "cat ~/.ssh/id_dsa.pub" >> ~/.ssh/authorized_keys; done + +## Copy Files to All Nodes in the Instance +In this step, you copy the **```~/.ssh/authorized_keys```** and **```~/.ssh/known_host```** files to all other nodes in the instance. Use the following commands to copy the files to all nodes in the instance '''except''' the master node: + + $ my_node=$(echo $HOSTNAME | cut -d '.' -f 1) + $ for node in $NODE_LIST; do if [ $node <> $my_node ]; then scp ~/.ssh/authorized_keys ~/.ssh/known_hosts $user@$node:~/.ssh/.; fi; done + +## Add ```localhost``` key to each node's ```~/.ssh/known_hosts file``` +In this step, you add the public key for **```localhost```** to the **```~/.ssh/known_hosts```** file + + $ for node in $NODE_LIST; do ssh $user@$node "echo localhost $(cat /etc/ssh/ssh_host_rsa_key.pub) >> ~/.ssh/known_hosts"; done + +## Turn on ```NoHostAuthenticationForLocalhost``` +In this step, you turn the **```NoHostAuthenticationForLocalhost```** on in each node in the instance. + + $ for node in $NODE_LIST; do ssh $user@$node "echo "NoHostAuthenticationForLocalhost=yes" >> ~/.ssh/config"; done + +## Secure the Files for Passwordless ```ssh``` Access +In this step, you change the security of the **```~/.ssh/authorized_keys```** file and the **```~/.ssh```** directory so that passwordless ssh will be allowed. Use the following commands for each node of the instance: + + $ for node in $NODE_LIST; do ssh $user@$node "chmod 755 ~/.ssh; chmod 600 ~/.ssh/authorized_keys; cd ~/.ssh; chmod 700 .."; done + +## Validate Passwordless ssh Access +Test by starting a remote shell on the different nodes in the instance â no password should be necessary. Be sure to check **```ssh```** access to the node itself as well. Use the following command string to check that you can reach each node in the instance without having to provide a password: + + $ for node in $NODE_LIST; do ssh $user@$node "ls -al /etc/passwd"; done + +Try the validation command string from all or some nodes in the instance. + +# Additional Information +Passwordless ssh isn't considered secure in some installations. We don't have a solution for this situation yet. + http://git-wip-us.apache.org/repos/asf/incubator-trafodion/blob/286a6e5c/docs/src/site/markdown/performance.md ---------------------------------------------------------------------- diff --git a/docs/src/site/markdown/performance.md b/docs/src/site/markdown/performance.md new file mode 100644 index 0000000..3151472 --- /dev/null +++ b/docs/src/site/markdown/performance.md @@ -0,0 +1,15 @@ +<!-- + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the + License. +--> +This page will provide information about Trafodion performance characteristics. \ No newline at end of file http://git-wip-us.apache.org/repos/asf/incubator-trafodion/blob/286a6e5c/docs/src/site/markdown/port-assignment.md ---------------------------------------------------------------------- diff --git a/docs/src/site/markdown/port-assignment.md b/docs/src/site/markdown/port-assignment.md new file mode 100644 index 0000000..de27501 --- /dev/null +++ b/docs/src/site/markdown/port-assignment.md @@ -0,0 +1,134 @@ +<!-- + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the + License. +--> +The following table lists the default ports used by the different Trafodion components plus the configuration file and configuration attribute associated with each port setting. + +<table> + <tr> + <th>Default Port</th> + <th>Configuration File</th> + <th>Configuration Entry</th> + <th>Required</th> + <th>Range</th> + <th>Protocol</th> + <th>Comment</th> + </tr> + <tr> + <td><strong>4200</strong></td> + <td><code>rest-site.xml</code></td> + <td><code>trafodion.rest.port</code></td> + <td>Yes</td> + <td>1</td> + <td>REST</td> + <td>Trafodion REST Server.</td> + </tr> + <tr> + <td><strong>4201</strong></td> + <td><code>rest-site.xml</code></td> + <td><code>trafodion.rest.https.port</code></td> + <td>Yes</td> + <td>1</td> + <td>HTTPS</td> + <td>Trafodion REST Server (HTTPS).</td> + </tr> + <tr> + <td><strong>23400</strong></td> + <td><code>dcs-site.xml</code></td> + <td><code>dcs.master.port</code></td> + <td>Yes</td> + <td>n</td> + <td>binary</td> + <td>Start of Trafodion DCS port range. (37800 for Trafodion 1.1)</td> + </tr> + <tr> + <td><strong>24400</strong></td> + <td><code>dcs-site.xml</code></td> + <td><code>dcs.master.info.port</code></td> + <td>Yes</td> + <td>1</td> + <td>HTTP</td> + <td>DCS master web GUI. (40010 for Trafodion 1.1)</td> + </tr> + <tr> + <td><strong>24410</strong></td> + <td><code>dcs-site.xml</code></td> + <td><code>dcs.server.info.port</code></td> + <td>Yes</td> + <td>n</td> + <td>HTTP</td> + <td>Start of range for DCS server web GUIs. (40020 for Trafodion 1.1)</td> + </tr> + <tr> + <td><strong>50030</strong></td> + <td><code>mapred-site.xml</code></td> + <td><code>mapred.job.tracker.http.address</code></td> + <td>No</td> + <td>1</td> + <td>HTTP</td> + <td>MapReduce Job Tracker web GUI.</td> + </tr> + <tr> + <td><strong>50070</strong></td> + <td><code>hdfs-site.xml</code></td> + <td><code>dfs.http.address</code></td> + <td>No</td> + <td>1</td> + <td>HTTP</td>. + <td>HDFS Name Node web GUI.</td> + </tr> + <tr> + <td><strong>50075</strong></td> + <td><code>hdfs-site.xml</code></td> + <td><code>dfs.datanode.http.address</code></td> + <td>No</td> + <td>1</td> + <td>HTTP</td>. + <td>HDFS Data Node web GUI.</td> + </tr> + <tr> + <td><strong>50090</strong></td> + <td><code>hdfs-site.xml</code></td> + <td><code>dfs.secondary.http.address</code></td> + <td>No</td> + <td>1</td> + <td>HTTP</td>. + <td>HDFS Secondary Name Node web GUI.</td> + </tr> + <tr> + <td><strong>60010</strong></td> + <td><code>hbase-site.xml</code></td> + <td><code>hbase.master.info.port</code></td> + <td>No</td> + <td>1</td> + <td>HTTP</td>. + <td>HBase Master web GUI.</td> + </tr> + <tr> + <td><strong>60030</strong></td> + <td><code>hbase-site.xml</code></td> + <td><code>hbase.regionserver.info.port</code></td> + <td>No</td> + <td>1</td> + <td>HTTP</td>. + <td>HBase Region Server web GUI.</td> + </tr> +</table> + +<div class="alert alert-dismissible alert-info"> + <button type="button" class="close" data-dismiss="alert">&close;</button> + <p style="color:black">There are two port ranges used by Trafodion. <ul><li style="color:black">23400 is a range, to allow multiple mxosrvr processes on each node. Allow a range of a few ports, enough to cover all the servers per node that are listed in the "servers" file in the DCS configuration directory.</li> <li style="color:black">24410 is a range as well, enough to cover the DCS servers per node, usually 1 or 2.</li></ul></p> +<p style="color:black">If you use Trafodion 1.1, then the older port numbers are 37800 and 40010.</p> +<p style="color:black">On top of that you would need the ports required by your Hadoop distribution. Most of this information comes from a script to sand-box a development install, including the major ports used. The script's name is <strong><code>trafodion/core/sqf/sql/scripts/install_local_hadoop</code></strong>.</p> + <p style="color:black">Although not all the ports will be used on every node of the cluster, you would need to open most of them for all the nodes in the cluster that have Trafodion, HBase or HDFS servers on them.</p> +</div> \ No newline at end of file http://git-wip-us.apache.org/repos/asf/incubator-trafodion/blob/286a6e5c/docs/src/site/markdown/presentations.md ---------------------------------------------------------------------- diff --git a/docs/src/site/markdown/presentations.md b/docs/src/site/markdown/presentations.md new file mode 100644 index 0000000..911c6a3 --- /dev/null +++ b/docs/src/site/markdown/presentations.md @@ -0,0 +1,18 @@ +<!-- + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the + License. +--> + +Presentation | Description +-----------------------------------------------------------|---------------------------------------------------- +**[DTM Architecture](presentations/dtm-architecture.pdf)** | In-depth descriptions of the Trafodion Distributed Transaction Management service. \ No newline at end of file http://git-wip-us.apache.org/repos/asf/incubator-trafodion/blob/286a6e5c/docs/src/site/markdown/quickstart.md ---------------------------------------------------------------------- diff --git a/docs/src/site/markdown/quickstart.md b/docs/src/site/markdown/quickstart.md new file mode 100644 index 0000000..2a8e1f4 --- /dev/null +++ b/docs/src/site/markdown/quickstart.md @@ -0,0 +1,15 @@ +<!-- + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the + License. +--> +This page will describe how to get going with Trafodion. \ No newline at end of file http://git-wip-us.apache.org/repos/asf/incubator-trafodion/blob/286a6e5c/docs/src/site/markdown/release-notes-0-8-0.md ---------------------------------------------------------------------- diff --git a/docs/src/site/markdown/release-notes-0-8-0.md b/docs/src/site/markdown/release-notes-0-8-0.md new file mode 100644 index 0000000..a6c020b --- /dev/null +++ b/docs/src/site/markdown/release-notes-0-8-0.md @@ -0,0 +1,165 @@ +<!-- + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +--> + +This is the initial source-only release of the Apache Trafodion (incubating) project. + +Build instructions are available [here](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61316378). + +# Supported Platforms +The following platforms are supported in this release. + +<span> + <table> + <tr> + <td>**Operating Systems**</td> + <td>CentOS 6.5 -- 6.7</td> + </tr> + <tr> + <td>**Hadoop Distributions**</td> + <td>Hortonworks distribution HDP 2.0</td> + </tr> + <tr> + <td>**Java Version**</td> + <td>JDK 1.7.0_67 or newer</td> + </tr> + <tr> + <td>**HBase Version**</td> + <td>HBase 0.94.x</td> + </tr> + </table> +</span> + +# Enhancements + +This release contains the following new features. + +<span> + <table> + <tr> + <th>Category</th> + <th>Feature</th> + <th>Defect ID</th> + </tr> + <tr> + <td>**SQL**</td> + <td> + <ul> + <li>Foreign key (referential integrity), check, and unique constraints in addition to primary key and not null constraints.</li> + <li>Stored Procedures in Java (SPJs) (not yet fully tested).</li> + <li>User-Defined Functions (UDFs) written in C (not yet fully tested).</li> + <li>ACID properties apart from failure recovery.</li> + <li>Transactional atomicity and consistency provided by the standard two-phased commit protocol with presumed abort semantics, supporting commit and rollback.</li> + <li>Serializability based on a Multi-Version Concurrency Control (MVCC) algorithm in HBase-trx.</li> + <li>Distributed/global transaction support across the cluster, including transaction propagation between SQL components.</li> + <li>Support for multithreaded SQL clients.</li> + </ul> + </td> + <td></td> + </tr> + </table> +</span> + +# Fixes + +## 0.8.0 +Fixes to the Trafodion ODBC driver for Windows and the installer, which handles the installation of the Trafodion software. This release also includes performance fixes and cleanup to the Trafodion software code. + +## 0.8.1 +Release 0.8.1 provides these fixes to the Trafodion installer: + +* An updated license agreement in the installer. +* Fix to the installation problem noted in [Launchpad Question #250671](https://answers.launchpad.net/trafodion/+question/250671). + +## 0.8.2 +Release 0.8.2 has a corrected license file in the installer. + +## 0.8.3 +Release 0.8.3 contains fixes to more than 35 critical and high defects, which were filed through [Launchpad](https://bugs.launchpad.net/trafodion/), and includes some performance enhancements based on DC and YCSB benchmarks. Native Expressions and Transaction Manager Recovery were disabled in order to avoid two critical issues. Those issues will be addressed in an upcoming release. + +# Known Issues + +## sqlci prints no banner or prompt when used in a su session + +**Defect:** [1329001](https://bugs.launchpad.net/trafodion/+bug/1329001) + +**Symptom:** <code>sqlci</code> hangs and is missing output after â<code>su</code>â command. + +**Cause:** The <code>sqlci</code> program may appear hung (producing no output) when you use the â<code>su</code>â command to switch to another user. This is caused by a permission problem in Linux, making the standard output of <code>sqlci</code> disappear. + +**Solution:** There are two possible workarounds: + +1. Create a new session for the changed user; for example: with the <code>ssh</code> or <code>xterm</code> commands, or +2. Invoke the program like this: <code>sqlci | cat</code> + +## Getting TM error 97 when tables split or get moved + +**Defect:** [1274651](https://bugs.launchpad.net/trafodion/+bug/1274651) + +**Symptom:** HBase Region Splits, Load Balancing, and Error 97. + +**Cause:** As part of an HBase environmentâs ongoing operations (and based on the policies configured for the HBase environment), an HBase region can either get split (into two daughter regions) or moved to a different region server. (Please see the blog: [http://hortonworks.com/blog/apache-hbase-region-splitting-and-merging/](http://hortonworks.com/blog/apache-hbase-region-splitting-and-merging/).) If that happens when a Trafodion transaction is active (and operates on rows within the region that is being split or load-balanced), then a subsequent transaction commit operation by the application might encounter an error 97. Please note that under such conditions the Trafodion Transaction Manager will abort the transaction and will preserve the integrity of the database. + +**Solution:** To minimize disruptions when this happens, we suggest that you use one or more of the following approaches: + +<ol> + <li>Enhance your JDBC application logic to retry when an error 97 is returned for a commit operation.</li> + <li>Update the HBase configuration to reduce the times when such disruptions happen. It involves updates to some properties that can be set in <code>hbase-site.xml</code> (or via the manageability interface of your Hadoop distribution).</li> + <ul> + <li>Set the maximum file size of an HBase Region to 100 GB. For example, set the value of the property <code>hbase.hregion.max.filesize</code> to <code>107374182400</code>.</li> + <li><p>Set the HBase region split policy to '<code>ConstantSizeRegionSplitPolicy</code>'. For example, set the value of the property <code>hbase.regionserver.region.split.policy</code> to <code>org.apache.hadoop.hbase.regionserver.ConstantSizeRegionSplitPolicy</code>.</p> + <p><strong>NOTE</strong>: The split policy should already be set to '<code>onstantSizeRegionSplitPolicy</code>' by the Trafodion installer.</p> + <p><strong>Summary</strong></p> + <p> + <span> + <table> + <tr> + <th>Property</th> + <th>Value</th> + </tr> + <tr> + <td><code>hbase.hregion.max.filesize</code></td> + <td><code>107374182400</code></td> + </tr> + <tr> + <td><code>hbase.regionserver.region.split.policy</code></td> + <td><code>org.apache.hadoop.hbase.regionserver.ConstantSizeRegionSplitPolicy</code></td> + </tr> + </table> + </span> + More information: [http://hortonworks.com/blog/apache-hbase-region-splitting-and-merging/](http://hortonworks.com/blog/apache-hbase-region-splitting-and-merging/) + </p> + </li> + </ul> + <li><p>Disable HBase Region Load Balancing. Use the HBase shell command balance_switch false to disable the movement of a region from one server to another.</p> + <p><strong>Example</strong></p> + <pre> +hbase shell +hbase(main):002:0> balance_switch false +true -- Output will be the last setting of the balance_switch value +0 row(s) in 0.0080 seconds + </pre> + </li> + <li> + Pre-split the table into multiple regions by using the <code>SALT USING n PARTITIONS</code> clause when creating the table. The number of partitions that you specify could be a function of the number of region servers present in the HBase cluster. Here is a simple example in which the table INVENTORY is pre-split into four regions when created: + <pre> +CREATE TABLE INVENTORY + ( + ITEM_ID INT UNSIGNED NO DEFAULT NOT NULL + , ITEM_TYPE INT UNSIGNED NO DEFAULT NOT NULL + , ITEM_COUNT INT UNSIGNED NO DEFAULT NOT NULL + , PRIMARY KEY (ITEM_ID ASC) + ) SALT USING 4 PARTITIONS + ; </pre> + </li> +</ol> http://git-wip-us.apache.org/repos/asf/incubator-trafodion/blob/286a6e5c/docs/src/site/markdown/release-notes-0-9-0.md ---------------------------------------------------------------------- diff --git a/docs/src/site/markdown/release-notes-0-9-0.md b/docs/src/site/markdown/release-notes-0-9-0.md new file mode 100644 index 0000000..a8f0757 --- /dev/null +++ b/docs/src/site/markdown/release-notes-0-9-0.md @@ -0,0 +1,210 @@ +<!-- + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +--> + +This is the source-only release of the Apache Trafodion (incubating) project. This release provides support for HBase 0.98 and many new product features. + +Build instructions are available [here](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61316378). + +# Supported Platforms +The following platforms are supported in this release. + +<span> + <table> + <tr> + <td>**Operating Systems**</td> + <td>CentOS 6.5 -- 6.7</td> + </tr> + <tr> + <td>**Hadoop Distributions**</td> + <td>Cloudera CDH distribution 5.1 and Hortonworks distribution HDP 2.1</td> + </tr> + <tr> + <td>**Java Version**</td> + <td>JDK 1.7.0_67 or newer</td> + </tr> + <tr> + <td>**HBase Version**</td> + <td>HBase 0.98.x</td> + </tr> + </table> +</span> + +# Enhancements + +This release contains the following new features. + +<span> + <table> + <tr> + <th>Category</th> + <th>Feature</th> + <th>Defect ID</th> + </tr> + <tr> + <td>**Performance**</td> + <td> + <ul> + <li>Move to HBase 0.98.</li> + <li>Significant improvements in benchmarks over Trafodion 0.8.3:</li> + <ul> + <li>YCSB</li> + <li>Debit/Credit</li> + <li>Order Entry</li> + <li>Atomics</li> + </ul> + </ul> + </td> + <td></td> + </tr> + <tr> + <td>**Infrastructure and High Availability**</td> + <td> + <ul> + <li>Support on newer Hadoop distributions: + <ul> + <li>Cloudera CDH 5.1</li> + <li>Hortonworks HDP 2.1</li> + </ul> + </li> + <li>Better integration with HBase: + <ul> + <li>Use of coprocessors</li> + <li>Use of HLOG (HBase Write Ahead Log)</li> + </ul> + </li> + <li>Recovery from catastrophic HBase failures.</li> + </ul> + </td> + <td></td> + </tr> + <tr> + <td>**Database and Query Performance**</td> + <td> + <ul> + <li>Improvement in scan performance: + <ul> + <li>Tuning Degree of Parallelism</li> + <li>Tuning scan buffer size</li> + </ul> + </li> + <li>Prefetch</li> + <li>Enhancements in JNI interface</li> + <li>Improved partitioning for salted tables</li> + <li>Push down predicates to HBase layer</li> + <li>Enhancement in data movement/message traffic by sending compressed (internal format) data from one operator to another</li> + <li>Improved memory usage</li> + <li>Improvements to query compile time through embedded compilers</li> + <li>Improvements in performance of update statistics feature</li> + </ul> + </td> + <td></td> + </tr> + <tr> + <td>**Database Movement and Connectivity**</td> + <td> + <ul> + <li>Bulk Loader</li> + <li>Support for compression in ODBC drivers</li> + </ul> + </td> + <td></td> + </tr> + <tr> + <td>**Security**</td> + <td> + <ul> + <li>Support for Grant/Revoke</li> + </ul> + </td> + <td></td> + </tr> + <tr> + <td>**Stability**</td> + <td> + <ul> + <li>Fixes for 140+ defects.</li> + </ul> + </td> + <td></td> + </tr> + </table> +</span> + +# Fixes + +This release contains fixes to 140 defects. Those defects were filed through [Launchpad](https://launchpad.net/trafodion/+milestone/r1.1). + +# Known Issues + +## Getting TM error 97 when tables split or get moved + +**Defect:** [1274651](https://bugs.launchpad.net/trafodion/+bug/1274651) + +**Symptom:** HBase Region Splits, Load Balancing, and Error 97. + +**Cause:** As part of an HBase environmentâs ongoing operations (and based on the policies configured for the HBase environment), an HBase region can either get split (into two daughter regions) or moved to a different region server. (Please see the blog: [http://hortonworks.com/blog/apache-hbase-region-splitting-and-merging/](http://hortonworks.com/blog/apache-hbase-region-splitting-and-merging/).) If that happens when a Trafodion transaction is active (and operates on rows within the region that is being split or load-balanced), then a subsequent transaction commit operation by the application might encounter an error 97. Please note that under such conditions the Trafodion Transaction Manager will abort the transaction and will preserve the integrity of the database. + +**Solution:** To minimize disruptions when this happens, we suggest that you use one or more of the following approaches: + +<ol> + <li>Enhance your JDBC application logic to retry when an error 97 is returned for a commit operation.</li> + <li>Update the HBase configuration to reduce the times when such disruptions happen. It involves updates to some properties that can be set in <code>hbase-site.xml</code> (or via the manageability interface of your Hadoop distribution).</li> + <ul> + <li>Set the maximum file size of an HBase Region to 100 GB. For example, set the value of the property <code>hbase.hregion.max.filesize</code> to <code>107374182400</code>.</li> + <li><p>Set the HBase region split policy to '<code>ConstantSizeRegionSplitPolicy</code>'. For example, set the value of the property <code>hbase.regionserver.region.split.policy</code> to <code>org.apache.hadoop.hbase.regionserver.ConstantSizeRegionSplitPolicy</code>.</p> + <p><strong>NOTE</strong>: The split policy should already be set to '<code>onstantSizeRegionSplitPolicy</code>' by the Trafodion installer.</p> + <p><strong>Summary</strong></p> + <p> + <span> + <table> + <tr> + <th>Property</th> + <th>Value</th> + </tr> + <tr> + <td><code>hbase.hregion.max.filesize</code></td> + <td><code>107374182400</code></td> + </tr> + <tr> + <td><code>hbase.regionserver.region.split.policy</code></td> + <td><code>org.apache.hadoop.hbase.regionserver.ConstantSizeRegionSplitPolicy</code></td> + </tr> + </table> + </span> + More information: [http://hortonworks.com/blog/apache-hbase-region-splitting-and-merging/](http://hortonworks.com/blog/apache-hbase-region-splitting-and-merging/) + </p> + </li> + </ul> + <li><p>Disable HBase Region Load Balancing. Use the HBase shell command balance_switch false to disable the movement of a region from one server to another.</p> + <p><strong>Example</strong></p> + <pre> +hbase shell +hbase(main):002:0> balance_switch false +true -- Output will be the last setting of the balance_switch value +0 row(s) in 0.0080 seconds + </pre> + </li> + <li> + Pre-split the table into multiple regions by using the <code>SALT USING n PARTITIONS</code> clause when creating the table. The number of partitions that you specify could be a function of the number of region servers present in the HBase cluster. Here is a simple example in which the table INVENTORY is pre-split into four regions when created: + <pre> +CREATE TABLE INVENTORY + ( + ITEM_ID INT UNSIGNED NO DEFAULT NOT NULL + , ITEM_TYPE INT UNSIGNED NO DEFAULT NOT NULL + , ITEM_COUNT INT UNSIGNED NO DEFAULT NOT NULL + , PRIMARY KEY (ITEM_ID ASC) + ) SALT USING 4 PARTITIONS + ; </pre> + </li> +</ol>
