[TRAFODION-1507] Added new files. jenkins, skip test
Project: http://git-wip-us.apache.org/repos/asf/incubator-trafodion/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-trafodion/commit/286a6e5c Tree: http://git-wip-us.apache.org/repos/asf/incubator-trafodion/tree/286a6e5c Diff: http://git-wip-us.apache.org/repos/asf/incubator-trafodion/diff/286a6e5c Branch: refs/heads/master Commit: 286a6e5c1797f725c9942d6bba45540bd3c844af Parents: 1c617ae Author: Gunnar Tapper <[email protected]> Authored: Mon Dec 7 14:55:20 2015 -0700 Committer: Gunnar Tapper <[email protected]> Committed: Mon Dec 7 14:55:20 2015 -0700 ---------------------------------------------------------------------- docs/src/site/markdown/advocate.md | 15 + docs/src/site/markdown/build-tools-manual.md | 197 ++++++++++++ docs/src/site/markdown/build.md | 80 +++++ docs/src/site/markdown/code.md | 32 ++ .../markdown/cplusplus-coding-guidelines.md | 310 +++++++++++++++++++ .../src/site/markdown/create-dev-environment.md | 153 +++++++++ docs/src/site/markdown/develop.md | 245 +++++++++++++++ docs/src/site/markdown/document.md | 121 ++++++++ docs/src/site/markdown/documentation.md | 25 ++ docs/src/site/markdown/download.md | 36 +++ .../site/markdown/enable-secure-trafodion.md | 224 ++++++++++++++ docs/src/site/markdown/faq.md | 220 +++++++++++++ docs/src/site/markdown/install-preparation.md | 138 +++++++++ docs/src/site/markdown/install.md | 203 ++++++++++++ docs/src/site/markdown/ldapcheck.md | 41 +++ docs/src/site/markdown/ldapconfigcheck.md | 48 +++ .../src/site/markdown/manage-dev-environment.md | 51 +++ docs/src/site/markdown/management.md | 62 ++++ docs/src/site/markdown/new-features.md | 15 + docs/src/site/markdown/passwordless-ssh.md | 131 ++++++++ docs/src/site/markdown/performance.md | 15 + docs/src/site/markdown/port-assignment.md | 134 ++++++++ docs/src/site/markdown/presentations.md | 18 ++ docs/src/site/markdown/quickstart.md | 15 + docs/src/site/markdown/release-notes-0-8-0.md | 165 ++++++++++ docs/src/site/markdown/release-notes-0-9-0.md | 210 +++++++++++++ docs/src/site/markdown/release-notes-1-0-0.md | 255 +++++++++++++++ docs/src/site/markdown/release-notes-1-0-1.md | 249 +++++++++++++++ docs/src/site/markdown/release-notes-1-1-0.md | 181 +++++++++++ docs/src/site/markdown/release.md | 225 ++++++++++++++ docs/src/site/markdown/roadmap.md | 15 + .../site/markdown/setup-build-environment.md | 160 ++++++++++ docs/src/site/markdown/testing.md | 179 +++++++++++ docs/src/site/markdown/tests.md | 71 +++++ .../site/markdown/traf_authentication_config.md | 96 ++++++ docs/src/site/markdown/wiki.md | 23 ++ .../site/resources/images/multi-layer-esps.png | Bin 0 -> 116303 bytes 37 files changed, 4358 insertions(+) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/incubator-trafodion/blob/286a6e5c/docs/src/site/markdown/advocate.md ---------------------------------------------------------------------- diff --git a/docs/src/site/markdown/advocate.md b/docs/src/site/markdown/advocate.md new file mode 100644 index 0000000..425c70e --- /dev/null +++ b/docs/src/site/markdown/advocate.md @@ -0,0 +1,15 @@ +<!-- + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the + License. +--> +This page provides how to contribute to Trafodion via advocacy. Please refer to the [Contribute](contribute.html) page for information about other ways to contribute to the Trafodion project. \ No newline at end of file http://git-wip-us.apache.org/repos/asf/incubator-trafodion/blob/286a6e5c/docs/src/site/markdown/build-tools-manual.md ---------------------------------------------------------------------- diff --git a/docs/src/site/markdown/build-tools-manual.md b/docs/src/site/markdown/build-tools-manual.md new file mode 100644 index 0000000..6c02481 --- /dev/null +++ b/docs/src/site/markdown/build-tools-manual.md @@ -0,0 +1,197 @@ +<!-- + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the + License. +--> +This page describes how to perform manual installs of the required [Trafodion Build Tools](setup-build-environment.html#Install_Build_Tools). + +In the sections below, the **```<tool installation directory>```** is the directory where you want the tool to be installed. + +# MPICH +**Tested Version**: 3.0.4 + +**Download**: http://www.mpich.org/static/downloads/3.0.4/mpich-3.0.4.tar.gz (http://www.mpich.org/downloads) + +**Considerations**: For more detailed instructions, see the README file that comes with the source. + +**Install**: + + tar -xzf mpich-3.0.4.tar.gz + cd mpich-3.0.4 + ./configure --prefix=<tool installation directory>/dest-mpich-3.0.4 --with-device=ch3:sock --disable-f77 --disable-fc + make + make check + make install + +**```<tool installation directory>```** is the directory where you want MPICH to be installed. If you do not specify the **```--prefix```** option, the default location is **```/usr/local``**. + +# Bison +**Tested Version**: 3.0 + +**Download**: http://ftp.gnu.org/gnu/bison/bison-3.0.tar.gz (http://ftp.gnu.org/gnu/bison/) + +**Considerations**: Refer to the bison INSTALL file for detailed instructions. + +**Determine Bison Version**: + + which bison + bison --version + +If the version is older than 3.0, then do the following: + + tar -xzf bison-3.0.tar.gz + cd bison-3.0 + ./configure --prefix=<tool installation directory>/bison_3_linux + make + make check + make install + +**Note**: The **```make check```** step may return errors like the following that can be ignored: + + make[3]: Entering directory `<mydir>/bison-3.0' + YACC examples/calc++/calc++-parser.stamp + CXX examples/calc++/examples_calc___calc__-calc++-driver.o + LEX examples/calc++/calc++-scanner.cc + CXX examples/calc++/examples_calc___calc__-calc++-scanner.o + g++: ./examples/calc++/calc++-scanner.cc: No such file or directory + g++: no input file + +Adjust your **```PATH```** to ensure that the correct version is chosen. Rerun the **```bison --version```** to verify. + +# Udis86 +**Tested Version**: 1.7.2 + +**Download**: http://sourceforge.net/projects/udis86/files/udis86/1.7/udis86-1.7.2.tar.gz (http://udis86.sourceforge.net) + +**Consideration**: Udis86 is a prerequisite to building the LLVM product. + +**Install**: + + tar xzf udis86-1.7.2.tar.gz + cd udis86-1.7.2 + ./configure --prefix=<tool installation directory>/udis86-1.7.2 --enable-shared + make + make install + +# LLVM +**Tested Version**: 3.2 + +**Download**: http://llvm.org/releases/3.2/llvm-3.2.src.tar.gz (http://llvm.org/releases/download.html) + +**Consideration**: Udis86 must be installed on the system before LLVM is built and installed. Building LLVM takes some time to complete, be patient. + +**Install**: + + # Set BASE_DIR to the top-level directory where the LLVM source will be + # unpacked and the objects compiled. + BASE_DIR=<your-base-dir> + cd $BASE_DIR + tar xzf llvm-3.2.src.tar.gz + + export MY_UDIS_INSTALL_DIR=<udis-installation-directory>/udis86-1.7.2 + export MY_LLVM_INSTALL_DIR=<llvm-installation-directory>/dest-llvm-3.2/ + export MY_LLVM_SRC_DIR=$BASE_DIR/llvm-3.2.src + export MY_LLVM_OBJ_DIR=$BASE_DIR/llvm-3.2.obj/ + export LD_LIBRARY_PATH=$MY_UDIS_INSTALL_DIR/lib:$LD_LIBRARY_PATH + export C_INCLUDE_PATH=$MY_UDIS_INSTALL_DIR/include + export CPATH=$MY_UDIS_INSTALL_DIR/include + + mkdir -p $MY_LLVM_OBJ_DIR/release + cd $MY_LLVM_OBJ_DIR/release + + $MY_LLVM_SRC_DIR/configure --prefix=$MY_LLVM_INSTALL_DIR/release \ + --enable-optimized --enable-jit \ + --enable-shared --enable-targets=x86,x86_64,cpp \ + --with-udis86=$MY_UDIS_INSTALL_DIR/lib \ + CFLAGS=-fgnu89-inline + + make libs-only + make install-libs + + mkdir -p $MY_LLVM_OBJ_DIR/debug + cd $MY_LLVM_OBJ_DIR/debug + + $MY_LLVM_SRC_DIR/configure --prefix=$MY_LLVM_INSTALL_DIR/debug \ + --enable-optimized --enable-jit \ + --enable-debug-runtime --enable-debug-symbols \ + --enable-shared --enable-targets=x86,x86_64,cpp \ + --with-udis86=$MY_UDIS_INSTALL_DIR/lib \ + CFLAGS=-fgnu89-inline + + make libs-only + make install-libs + +# ICU +**Tested Version**: 4.4.0 + +**Download**: http://download.icu-project.org/files/icu4c/4.4/icu4c-4_4-src.tgz (http://site.icu-project.org/download) + +**Install**: + + tar -xzf icu4c-4_4-src.tgz + cd icu/source + ./runConfigureICU Linux --with-library-suffix=Nv44 --prefix=<tool installation directory>/icu4.4/linux64 + make && make check + make install + +**Note**: The following **```make check```** errors can be ignored. + + [All tests passed successfully...] + Elapsed Time: 00:00:12.126 + make[2]: Leaving directory `/home/centos/icu/source/test/cintltst' + --------------- + ALL TESTS SUMMARY: + ok: testdata iotest cintltst + ===== ERRS: intltest + make[1]: *** [check-recursive] Error 1 + make[1]: Leaving directory `/home/centos/icu/source/test' + make: *** [check-recursive] Error 2 + +# Zookeeper +**Tested Version**: 3.4.5 + +**Download**: https://archive.apache.org/dist/zookeeper/zookeeper-3.4.5/zookeeper-3.4.5.tar.gz + +**Install**: + + tar -xzf zookeeper-3.4.5.tar.gz + cd zookeeper-3.4.5/src/c + ./configure --prefix=<tool installation directory>/zookeeper-3.4.5 + make + make install + +# Thrift +**Tested Version**: 0.9.0 + +**Download**: http://archive.apache.org/dist/thrift/0.9.0/ + +**Consideration**: Behind a firewall, you may need the ant flags to specify a proxy. + +**Install**: + + tar -xzf thrift-0.9.0.tar.gz + cd thrift-0.9.0 + ./configure --prefix=<tool installation dir>/thrift-0.9.0 --without-qt + make + make install + +# Maven +**Tested Version**: 3.3.3 + +**Download**: http://archive.apache.org/dist/maven/maven-3/3.3.3/binaries/apache-maven-3.3.3-bin.tar.gz. + +**Considerations**: Add Maven to your **```PATH```** once it has been installed. + +**Install**: + + tar -xzf apache-maven-3.3.3-bin.tar.gz -C <tool installation directory> + \ No newline at end of file http://git-wip-us.apache.org/repos/asf/incubator-trafodion/blob/286a6e5c/docs/src/site/markdown/build.md ---------------------------------------------------------------------- diff --git a/docs/src/site/markdown/build.md b/docs/src/site/markdown/build.md new file mode 100644 index 0000000..960b928 --- /dev/null +++ b/docs/src/site/markdown/build.md @@ -0,0 +1,80 @@ +<!-- + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the + License. +--> +This page describes how to build the Trafodion source code. + +# Prerequisites +You need to [Setup Build Environment](setup-build-environment.html) before trying to build the Trafodion source code. + +# Download Source +You should already have downloaded the source code when setting up the build environment. + +## Git +Please refer to [Making Changes](develop.html#making_changes) on the [Develop](develop.html) page. + +## tar file +The source code for Apache Trafodion can be downloaded from [Apache Trafodion Incubator Release](https://dist.apache.org/repos/dist/release/incubator) as a tar file. + +* Download the source tar file to your **```<trafodion download directory>```**. +* Check the tar file validity by checking signatures, please refer to [Verify Signatures](release.html#Verify_Signatures). The Trafodion releases have been signed using The GNU Privacy Guard. + +**Unpack the tar file** + + cd <trafodion download directory> + tar -xzf <tar file> + +# Setup Environmental Variables +Start a new **```ssh```** session. Use the following commands to set up the Trafodion environmental variables. + + cd <Trafodion source directory> + export TOOLSDIR=<tools installation directory> + source ./env.sh + +* **```<Trafodion source directory>```**: Source tree base for Trafodion. +* **```<tools installation directory>```**: where Trafodion required tools are located. The following example assumes that you installed all the required tools in a single location. If you installed or used pre-installed tools in different directories, then you need to export the location of each tool as described in [Build Tools Manual Installation](build-tools-manual.html) prior to sourcing in **```env.sh```**. + +# Build Commands +Build a debug version of Trafodion using one of the following options. + +Command | What It Builds +------------------------------------|---------------------------------------------------------------------------------- +**```make all```** | Trafodion, DCS, and REST. +**```make package```** | Trafodion, DCS, REST, and Client Drivers. +**```make package-all```** | Trafodion, DCS, REST, Client Drivers, and tests for all components. + +If the build fails, you might want to rerun the **```make```** step. Trafodion downloads many dependencies and sometimes one of the download operations fail. Rerunning the build generally works. + +# Verify Build +Use **```sqvers -u```** to verify the build. + + $ sqvers -u + MY_SQROOT=/home/centos/apache-trafodion-1.3.0-incubating/core/sqf + who@host=centos@mysystem + JAVA_HOME=/usr/lib/jvm/java-1.7.0-openjdk-1.7.0.91.x86_64 + SQ_MBTYPE=64d (64-debug) + linux=2.6.32-504.1.3.el6.x86_64 + redhat=6.7 + NO patches + Most common Apache_Trafodion Release 1.3.0 (Build debug [centos], branch -, date 06Nov15) + UTT count is 1 + [6] Release 1.3.0 (Build debug [centos], branch -, date 06Nov15) + export/lib/hbase-trx-cdh5_3-1.3.0.jar + export/lib/hbase-trx-hbase_98_4-1.3.0.jar + export/lib/hbase-trx-hdp2_2-1.3.0.jar + export/lib/sqmanvers.jar + export/lib/trafodion-dtm-1.3.0.jar + export/lib/trafodion-sql-1.3.0.jar + +The output from the **```sqvers -u```** commands should show several jar files. The number of files differs based on the version of Trafodion you downloaded. + http://git-wip-us.apache.org/repos/asf/incubator-trafodion/blob/286a6e5c/docs/src/site/markdown/code.md ---------------------------------------------------------------------- diff --git a/docs/src/site/markdown/code.md b/docs/src/site/markdown/code.md new file mode 100644 index 0000000..71b76ab --- /dev/null +++ b/docs/src/site/markdown/code.md @@ -0,0 +1,32 @@ +<!-- + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the + License. +--> +This page describes how you set up the Trafodion build and development-test environment. + +# Supported Platforms +Red Hat or Centos 6.x (6.4 or later) versions are supported as development and production platforms. + +# Setup Build Environment +Please refer to [Setup Build Environment](setup-build-environment.html). + +# Build Trafodion +Please refer to [Build Trafodion](build.html). + +# Setup Test Environment +Please refer to the [Test Environment](test-environment.html) page. + +# Make Changes +Please refer to [Making Changes](develop.html#Making_Changes). Ensure that you run the [Regression Tests](testing.html) before checking in changes. + + http://git-wip-us.apache.org/repos/asf/incubator-trafodion/blob/286a6e5c/docs/src/site/markdown/cplusplus-coding-guidelines.md ---------------------------------------------------------------------- diff --git a/docs/src/site/markdown/cplusplus-coding-guidelines.md b/docs/src/site/markdown/cplusplus-coding-guidelines.md new file mode 100644 index 0000000..3470641 --- /dev/null +++ b/docs/src/site/markdown/cplusplus-coding-guidelines.md @@ -0,0 +1,310 @@ +<!-- + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the + License. +--> + +These are the C++ coding guidelines that are part of the acceptance criteria for code submitted to the Trafodion. Trafodion reviewers use these guidelines when reviewing changes. + +The guidelines describe practices that are either required or preferred. In addition, areas where there is no preference between two or more practices are described. + +Trafodion is composed of several distinct sub-projects, some of which have coding guidelines that differ from the Trafodion standard; for example, a requirement in most areas of the code may be only preferred in others. + +There may also be existing code that violates one or more of the published requirements. The intent is to correct these over time, and corrections are encouraged when changing code to make a fix or implement new functionality. However, changes that are solely coding guideline changes are not recommended since this places undue burden on the reviewers. + +# Header Files +Keep **<code>#include</code>** directives to a minimum in header files. + +You may forward declare classes and structs when the only use within the header file is a pointer or reference. While includes should be kept to a minimum, if something is used in a header file and cannot be forward declared, it must be explicitly included. + +All files, headers and implementation, should include everything they need to be self-sufficient. They should never assume something will be pre-included. In other words, the contents of a header file should compile cleanly by itself. To help ensure this, all implementation files should include their respective header file first. + +Header files should not contain declarations for public items that are only used by the implementation file. Public items that are require in the implementation file but not the header file should be declared in the implementation file. The preference is NOT to create header files that consist only of includes of other header files. + +Declare as little as possible in the header file and keep as much of the actual implementation private as is reasonable. For instance, donât include in a header file declarations of types, enums, and functions that are only referenced by the implementation file. + +## Including Standard Header Files +The preference is for C++ style includes over C style includes for standard header files. + + //Preferred + #include <cstdio> + //Accepted + #include <stdio.h> + +## Include Guards +All header files must use include guards. The name of the include guard **<code>#define</code>** should be the filename in all uppercase, with underscore used in place of periods. + +For instance, if the header file is named **<code>ServerInterface_ODBC.h</code>**, the header file should begin and end as follows: + + #ifndef SERVERINTERFACE_ODBC_H + #define SERVERINTERFACE_ODBC_H + ... + #endif /* SERVERINTERFACE_ODBC_H */ + +Comments following the **<code>#endif</code>** indicating the include guard are preferred. + +# Variable Declaration and Naming Standards +Trafodion uses a combination of Pascal and Camel case. + +* Pascal case means that the first letter in each word in an identifier is capitalized. +* Camel case is similar except the first letter is in lower case. + +For both, underscores are **not** used to separate words. The general rule is that identifiers with local scope start with a lower case letter and identifiers with global scope start with an upper case letter. + + //Pascal case + class AuthenticationMessage; + //Camel case (aka lower Camel case or camelCase) + int logonCount; + Class Names + +Class names should be Pascal case and should describe the object contents (not what it does), with as little abbreviation as possible. When names include acronyms, the acronyms should be in all upper case. + +**Acceptable Examples** + + //Preferred + + class SQLSessionContext; // an object that contains the context for a SQL session + class PrivilegeList; // a list of privileges + +**Poor Examples** + + class OutputInfo; // Doesnât describe class contents, no context + class ReadTableDef; // Describes what class does, not contents + class Cmdline_Args; // Prefer Pascal case, no underscore, for class names + +## Class Member Variable Names +Private member data variables should be suffixed with an underscore and should use Camel case. When names include acronyms, the acronyms should be in all upper or all lower case, dependent on which case the first letter should be. + +**Example** + + class Employee + { + public: + Employee (); + private: + std::string firstName_; + std::string lastName_; + uint16_t departmentNumber_; + std::string departmentName_; + uint32_t irsSSN_; + } + +## Function Names +Class member functions and static file functions should use Camel case. External non-class functions should use Pascal case. Except for constructors, destructors, and operators, the function name should include a verb that describes the action the function is performing. + +**Good Examples** + + //Class member functions + int32_t getSalary() const; + int32_t setAuthID(); + int32_t closeAllCursors(); + +**Bad Examples** + + // Is it setting break enabled, returning it or ??? + int32_t SQLCLI_BreakEnabled(); + +## Enums +Enum types should use Pascal case and describe the class of enums. If the enum is declared outside of a class, the type name should include an indication of the scope of the enums. + +Enums themselves should be declared as all upper case. The names may begin with a common prefix or be independent, depending on the usage. + +When enums represent an arbitrary set of return values (that is, error codes, state codes, etc.), then avoid the values -1, 0, and 1 if using weakly typed enums, to reduce the chance of matches with Booleans or uninitialized variables. + +The preference is to declare enums as strongly typed. + + enum class EnumName {...}; + +## Boolean Variables +Boolean variables names should include a verb, state, and optionally a noun (object whose state is in question) indicating the nature of the Boolean. Any combination is acceptable, however verbState is the most common. + +**Good Examples** + + bool isValid; // verbState + bool isValidTable; // verbStateNoun + bool tableIsDroppable; // nounVerbState + bool hasData; // verbState + +**Bad Examples** + + bool valid; + bool tableState; + bool empty; + +Functions that return a Boolean should also have names of the form verbState or verbStateNoun if the functions return state information. (This naming standard does not apply to functions returning Boolean as indication of success or failure.) + +**Good Examples** + + bool isValidHbaseName(); + bool isHostNameExcluded(); + bool canUseCbServer(); + +**Bad Examples** + + // Donât use get for Boolean accessors + bool getUDRAccessModeViolation(); + + // Donât use integer return for Boolean functions + short existsInHBase(); + + // Function name implies it is sending settings to the compiler, but it is + // actually only returning an indication that settings should be sent. + // A better name would be shouldSendSettingsToCompiler(). + bool sendSettingsToCompiler(); + +Parts of Trafodion code use one of two Boolean typedefs, **<code>NABoolean</code>** and **<code>ComBoolean</code>**, declared as follows: + + typedef int Int32; + typedef Int32 NABoolean; + typedef NABoolean ComBoolean; + const NABoolean TRUE = (1 == 1); + const NABoolean FALSE = (0 == 1); + +Exercise care when mixing usage of bool and **<code>NABoolean</code>**/**<code>ComBoolean</code>** types, as the latter are not guaranteed to only contain values of TRUE and FALSE. The use of non-standard Boolean types is gradually being phased out. + +## Constants +All constant names should be all upper case, regardless of how the constant is declared. That is, enums, defines, and variables with the **<code>const</code>** modifier should be named in all upper case. + +Defines, enums, and const are all permitted and used throughout Trafodion, although most code in Trafodion uses enum for numerical constants and defines for character constants. + +For new code, the use of const char or string is preferred for character constants instead of defines. + +## Namespace Names +The preference is for namespaces to be all lower case, with preference to single words (note the exception to the rule that a name with global scope should start with an upper case). + +If a namespace must be dual-worded, use underscores. If mixed case names are used, Pascal case is preferred. + +**Examples** + + //Preferred + + namespace compiler + //Accepted + + namespace Compiler + +# Indentation and Formatting +## Indentation +TAB characters are not permitted in source files except for text files (for example, makefiles) that require them. + +Trafodion code uses several indenting depths, including 2, 3, 4, and 8 spaces. Most common is 2 and 3. + +Use the style found in existing code, and when writing new code, use either 2, 3, or 4, and remain consistent. + +A variety of control block indentation styles are used throughout Trafodion, most commonly Allman, Whitesmith, Stroustrup, and GNU. Follow the predominant style when making small to medium changes to existing code. For new code, the Allman style is preferred. + + //Allman + if (x > 5) + { + error = doThis(x); + if (error != 0) + { + return false; + } + } + else + { + doThat(x); + } + + //Whitesmith + if (x > 5) + { + error = doThis(x); + if (error != 0) + { + return false; + } + } + else + { + doThat(x); + } + + //Stroustrup + if (x > 5) { + error = doThis(x); + if (error != 0) { + return false; + } + } + else { + doThat(x); + } + +Note that the Stroustrup and the similar K&R formats were popularized by usage in books where conservation of line count was a goal. + + //GNU + if (x > 5) + { + error = doThis(x); + if (error != 0) + { + return false; + } + } + else + { + doThat(x); + } + +# Comments +## Comment Style +C++ style comments are preferred, but C comments are acceptable as well. + +Some code uses Doxygen style comments, but this is not required. + +## When/Where Comments Should Be Used +Every file should have a comment at the beginning describing the purpose of the file. + +In header files where classes are declared, there should be a comment describing the class, including purpose and usage. Also describe anything out of the ordinary, such as the use of multiple inheritance. + +Within implementation files, in addition to the comment at the beginning of the file, add comments for any global or static variables defined in the file, and how the variable is handled in a multi-threaded environment (if applicable). + +Also, for each function defined, describe the purpose and intent of the function. For each parameter, list whether it is input or output (or both), how it is used, and any range restrictions imposed. For functions not returning void, describe the possible return values. + +Within the body of the function, there is no need to write comments that document the obvious. But if there is any complexity to the logic, at a minimum document the intent, and consider documenting the details (assumptions, limits, unexpected side effects from function calls, etc.) + +If a feature is only partially implemented, add a comment indicating at a high level what work remains. Prefix the comment with //TODO. + + //TODO Code is currently not thread safe. Need to protect allocation of ... + +# Error Handling +## Asserts +Trafodion uses asserts in the "debug" build. Use asserts freely, but ensure they do not contain any side effects as the code is not present in the "release" build. + +## Error Return/Retrieval +Avoid the use of integer return codes for success and error codes. Instead, use bool for simple succeeded/failed and enum types for returns with multiple conditions. + +## Exception Handling +Trafodion code mixes usage of exceptions and error returns. When using try/catch blocks, keep the scope as small as possible. Ensure all exceptions thrown are handled, potentially in main() if nowhere else. + +# General Guidelines +## Casting +Avoid using C-style casts. Use C++-style casts instead, as they are often safer and easier to search for. + + //Preferred + + int x = static_cast<int>(shortVariable); + + MyType *myVar = reinterpret_cast<MyType *>(voidPtr); + +Donât blindly cast to remove a compiler error or warning. Ensure the cast is safe. + +Use const casting sparingly. Often the use of mutable or changing a function to be const correct is a better solution. + +## Types +Use standard types defined in **<code>\<cstdint\></code>** or **<code>\<stdint.h\></code>**. Note that Trafodion defines and uses many non-standard types (for example, **<code>Int32</code>**, **<code>Lng32</code>**), but this usage is being phased out. + +Use types with explicit sizes (for example, **<code>int32_t</code>**, **<code>int64_t</code>**) when size is a factor in the code, such as an external API, or column in a table. + +Where size is not a factor (counters, indexes) a non-sized type such as **<code>int</code>** or **<code>long</code>** may be used. However, in general, **<code>size_t</code>** and **<code>ssize_t</code>** are preferred for variables where fixed size is not required. \ No newline at end of file http://git-wip-us.apache.org/repos/asf/incubator-trafodion/blob/286a6e5c/docs/src/site/markdown/create-dev-environment.md ---------------------------------------------------------------------- diff --git a/docs/src/site/markdown/create-dev-environment.md b/docs/src/site/markdown/create-dev-environment.md new file mode 100644 index 0000000..acd840d --- /dev/null +++ b/docs/src/site/markdown/create-dev-environment.md @@ -0,0 +1,153 @@ +<!-- + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the + License. +--> +This page describes how to create the test environment used for Trafodion development, which is intended for people that are contributing to the Trafodion source tree. Please refer to [Download](download.html) if you want to try the Trafodion product environment. + +# Prerequisites +The following prerequisites need to be met in order to create a functional Trafodion test environment. + +## Passwordless ```ssh``` +Check to see if you have passwordless SSH setup correctly. + + ssh localhost + Last login: Fri Nov 6 22:44:00 2015 from 192.168.1.9 + +If the **```ssh localhost```** command prompts for a password, then passwordless **```ssh```** is not set up correctly. + +The following is an example of setting up passwordless **```ssh```** using **```id_rsa```** keys. You can choose the method that best represents your environment. + +If you already have an existing set of **```ssh```** keys. Simply copy both the **```id_rsa.pub```** and **```id_rsa```** to your **```~/.ssh```** directory. + +Then, do the following to modify your **```ssh```** environment. + + cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys + chmod 600 ~/.ssh/id_rsa + echo "NoHostAuthenticationForLocalhost=yes" >>~/.ssh/config + chmod go-w ~/.ssh/config + chmod 755 ~/.ssh; chmod 640 ~/.ssh/authorized_keys; cd ~/.ssh; chmod 700 .. + +If you need to create your keys first, then do the following. + + ssh-keygen -t rsa -N "" -f ~/.ssh/id_rsa + cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys + chmod 600 ~/.ssh/id_rsa.pub + echo "NoHostAuthenticationForLocalhost=yes" >>~/.ssh/config + chmod go-w ~/.ssh/config + chmod 755 ~/.ssh; chmod 640 ~/.ssh/authorized_keys; cd ~/.ssh; chmod 700 .. + +## System Limits +Please check that the system limits in your environment are appropriate for Apache Trafodion. If they are not, then you will need to increase the limits or Trafodion cannot start. + +The recommended settings are as follows. + + $ ulimit âa + core file size (blocks, -c) 1000000 + data seg size (kbytes, -d) unlimited + scheduling priority (-e) 0 + file size (blocks, -f) unlimited + pending signals (-i) 515196 + max locked memory (kbytes, -l) 49595556 + max memory size (kbytes, -m) unlimited + open files (-n) 32000 + pipe size (512 bytes, -p) 8 + POSIX message queues (bytes, -q) 819200 + real-time priority (-r) 0 + stack size (kbytes, -s) 10240 + cpu time (seconds, -t) unlimited + max user processes (-u) 267263 + virtual memory (kbytes, -v) unlimited + file locks (-x) unlimited + +Please refer to this [article](http://www.itworld.com/article/2693414/setting-limits-with-ulimit.html) for information on how you change system limits. + +# Setup +You can create a Trafodion test environment using a: + +* **Pre-Installed Hadoop**: Trafodion installation on a system that already has a compatible version of Hadoop installed +* **Local Hadoop**: You install a Hadoop environment using the **```install_local_hadoop```** script + +Your installation approach depends on whether you already have installed Hadoop. + +## Pre-Installed Hadoop +Use the following instructions if you're installing Trafodion on a pre-installed Hadoop environment. +### Build Binary tar Files +Build the Trafodion binary tar files. + + cd <Trafodion source directory> + make package + +### Install Trafodion +Please refer to the installation instructions described in the [Installation](install.html) page. + +## Local Hadoop +Use the following instructions if you need to install a local Hadoop environment. +### Run ```install_local_hadoop``` +The **```install_local_hadoop```** script downloads compatible versions of Hadoop, HBase, Hive, and MySQL. Then, it starts Trafodion. + +<div class="alert alert-dismissible alert-info"> + <button type="button" class="close" data-dismiss="alert">&close;</button> + <p style="color:black"><strong>Time Saver</strong></p> + <p style="color:black"><strong>```install_local_hadoop```</strong> downloads Hadoop, HBase, Hive, and MySql jar files from the Internet. To avoid this overhead, you can download the required files into a separate directory and set the environment variable <strong>```MY_LOCAL_SW_DIST```</strong> to point to this directory.</p> +</div> + +Command | Usage +-----------------------------------------------|-------------------------------------------------------- +**```install_local_hadoop```** | Uses default ports for all services. +**```install_local_hadoop -p fromDisplay```** | Start Hadoop with a port number range determined from the DISPLAY environment variable. +**```install_local_hadoop -p rand```** | Start with any random port number range between 9000 and 49000. +**```install_local_hadoop -p <port>```** | Start with the specified port number. + +For a list of ports that get configured and their default values, please refer to [Port Assignments](port-assignments.html). + +### Sample Procedure +Start a new **```ssh```** session and ensure that the Trafodion environmental variables are loaded. + + cd <Trafodion source directory> + source ./env.sh + +Install the Hadoop software. + + cd $MY_SQROOT/sql/scripts + install_local_hadoop + ./install_traf_components + +Verify installation. + + $ swstatus + 6 java servers and 2 mysqld processes are running + 713 NameNode + 19513 HMaster + 1003 SecondaryNameNode + 838 DataNode + 1173 ResourceManager + 1298 NodeManager + +Six java servers as shown above and two mysqld processes should be running. + +# Install and Build Trafodion +Please refer to [Modify Code](code.html) for information on how to install and build Trafodion from its source code. + +# New Source Download +You need to do the following each time you download new source code. + + cd <Trafodion source directory> + source ./env.sh + cd $MY_SQROOT/etc + # delete ms.env, if it exists + rm ms.env + cd $MY_SQROOT/sql/scripts + sqgen + +# Manage +Please refer to [Manage Development Environment](manage-dev-environment) for instructions. http://git-wip-us.apache.org/repos/asf/incubator-trafodion/blob/286a6e5c/docs/src/site/markdown/develop.md ---------------------------------------------------------------------- diff --git a/docs/src/site/markdown/develop.md b/docs/src/site/markdown/develop.md new file mode 100644 index 0000000..79656d8 --- /dev/null +++ b/docs/src/site/markdown/develop.md @@ -0,0 +1,245 @@ +<!-- + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the + License. +--> +This page describes how to help develop the Trafodion source tree. Please refer to the [Contribute](contribute.html) page for information about other ways to contribute to the Trafodion project. + +# Prerequisites +You need to register as a Trafodion contributor before you can help us develop Trafodion. Please perform the following registration actions: + +<table> + <tr> + <th style="width:30%;">Area</th> + <th style="width:55%;">Notes</th> + <th style="width:15%;">URL</th> + </tr> + <tr> + <td><strong>Individual Contributor License Agreement (ICLA)</strong></td> + <td>You should sign the ICLA before contributing content to the Trafodion source tree. (Required to become a committer.)</td> + <td><a href="https://www.apache.org/licenses/icla.txt"; target="_blank">ICLA Agreement</a><br /> + <a href="http://www.apache.org/licenses/"; target="_blank">Approval Process</a> + </td> + </tr> + <tr> + <td><strong>Source Control</strong></td> + <td>You must have a git account in order to contribute to the Trafodion source. If you haven't already done so, please join git.</td> + <td><a href="https://github.com/join"; target="_blank">Git Signup</a></td> + </tr> + <tr> + <td><strong>Defect Tracking</strong></td> + <td>In order to have certain permissions, including assigning issues to yourself, you need to be a Contributor in the project. Be sure to sign up for a JIRA account if you don't have one.</td> + <td><a href="https://issues.apache.org/jira/secure/Signup!default.jsp"; target="_blank">Jira Signup</a></td> + </tr> +</table> + +Please send an e-mail to the [Trafodion development list](mail-lists.hmtl) with the approved ICLA attached. Include your git and Jira IDs. + +Wait for the response and then you're ready to help us develop Trafodion. + +# Development Environment +You use the following tools and guidelines to develop Trafodion: + +<table> + <body> + <tr> + <th style="width:15%;">Area</th> + <th style="width:15%;">Tool</th> + <th style="width:55%;">Notes</th> + <th style="width:15%;">Location</th> + </tr> + <tr> + <td><strong>Trafodion Architecture</strong></td> + <td>Document</td> + <td>Please review the Trafodion architecture to ensure that you understand how the different components related to each other.</td> + <td><a href="architecture-overview.html" target="_blank">Trafodion Architecture</a></td> + </tr> + <tr> + <td><strong>Defect Tracking</strong></td> + <td>Jira</td> + <td>View all the Trafodion defects and enhancements requests in the Jira system hosted by Apache.</td> + <td><a href="https://issues.apache.org/jira/browse/TRAFODION"; target="_blank">Trafodion Jiras</a></td> + </tr> + <tr> + <td><strong>Defect Management</strong></td> + <td>Document</td> + <td>Please read about our approach to defect management. Mostly, any changes you'll make will be in response to a defect reported in Jira.</td> + <td><a href="defect-management.html" target="_blank">Defect Management</a></td> + </tr> + <tr> + <td><strong>Git Tools</strong></td> + <td>git</td> + <td><p>Most of the Trafodion development is done on Linux. Development of the web site and/or documentation can successfully be done on Windows.</p><p>Please download the appropriate tool version; Linux or Windows.</p><p>Then, please refer to <a href="https://help.github.com/articles/set-up-git/"; target="_blank">GitHub Documentation</a> for information on how to set up your git environment. Ensure that you register your <a href="https://github.com/settings/ssh"; target="_blank">ssh keys</a>.</p></td> + <td><a href="http://git-scm.com/downloads"; target="_blank">Download git</a></td> + </tr> + <tr> + <td><strong>Code Repository</strong></td> + <td>git</td> + <td>The full Trafodion source tree can be retrieved from either of these repositories.</td> + <td><a href="https://git-wip-us.apache.org/repos/asf/incubator-trafodion.git"; target="_blank">Apache Repository</a><br /><a href="https://github.com/apache/incubator-trafodion"; target="_blank">GitHub Mirror</a> + </td> + <tr> + </tr> + <td><strong>Code Organization</strong></td> + <td>Document</td> + <td>Please familiarize yourself with the Trafodion code organization.</td> + <td><a href="code-organization.html" target="_blank">Code Organization</a></td> + </tr> + <tr> + <td><strong>C++ Coding Guidelines</strong></td> + <td>Document</td> + <td>Please read the coding guidelines for the Trafodion C++ code before making changes.</td> + <td><a href="cplusplus-coding-guidelines.html" target="_blank">C++ Coding Guidelines</a></td> + </tr> + <tr> + <td><strong>Debugging Tips</strong></td> + <td>Document</td> + <td>Documented tips describing how to debug your code in unit testing.</td> + <td><a href="debugging-tips.html" target="_blank">Debugging Tips</a></td> + </tr> + <tr> + <td><strong>Testing</strong></td> + <td>Document</td> + <td>Trafodion has a rich set of test suites for each of its components. You'll need to run the tests before submitting a code change for review.</td> + <td><a href="testing.html" target="_blank">How to Test</a></td> + </tr> + <tr> + <td><strong>Code Reviews</strong></td> + <td>git</td> + <td> + <p>We use GitHub pull-requests for code review. All of the activity on github is captured in ASF JIRA and/or ASF project mail archives by ASF INFRA team automation. In this way, we do not depend on github for accurate history of where contributions come from.</p> + <p>Each pull-request title should start with a JIRA ID in brackets, so that activity can be logged to the correct JIRA issue.</p> + <p>Regardless of the title, the pull-request activity is also logged to the <a href="http://mail-archives.apache.org/mod_mbox/incubator-trafodion-codereview/"; target="_blank">code-review mail list</a>.</p> + </td> + <td><a href="https://github.com/apache/incubator-trafodion/pulls"; target="_blank">Current Pull Requests</a></td> + </tr> + </body> +</table> + +# Initial Setup +This set of tasks is performed **after** downloading the git tools. Refer to [Development Tools](#development_tools) above. + +You should not have to perform these tasks more than once. + +## Setting Up the Git Enviroment +If you have not done so already, now is the time to set up your **<code>git</code>** environment. Refer to the [GitHub Documentation](https://help.github.com/articles/set-up-git/) for information on how to set up your Git environment. Please ensure that you register your [ssh keys](https://github.com/settings/ssh). + +## Fork the Trafodion Repository +You create a private fork of Trafodion on <https://github.com/apache/incubator-trafodion>. Use the **fork** button top-right on the page to create your fork, which will be named **\<your-git-id\>_fork.** + +The following examples use **trafdeveloper** to represent **\<your-git-id\>**. + +## Clone the Trafodion Repository +Use the **git shell** to perform this task. + + # Move to the directory where you want to install the Trafodion source code. + cd mysource + # Clone the Trafodion source code + git clone git://git.apache.org/incubator-trafodion.git + # Register your fork as a remote branch + git remote add trafdeveloper_fork [email protected]:trafdeveloper/incubator-trafodion + +At this point, you've finished all preparation steps. Now, you can start making changes. + +# Making Changes +## Create a Task Branch +You create a task branch to make changes to the Trafodion source. Typically, we name the branches after the Jira we are working on. In this example, the Jira is: **TRAFODION-1507**. + + # Ensure that you have the latest changes + git fetch --all + # Checkout source + git checkout -b TRAFODION-1507 origin/master + +## Change Recipes +The procedure to make changes depends on what type of problem or feature you're working on. + +Change Type | Refer To +---------------------------------|------------------------------------------------------------ +**Code** | [Modify Code](code.html) +**Documentation** | [Modify Documentation](document.html) +**QA Tests** | [Modify Tests](tests.html) +**Web Site** | [Modify Web Site](website.html) + +## Commit Changes + +<div class="alert alert-dismissible alert-info"> + <button type="button" class="close" data-dismiss="alert">&close;</button> + <p style="color:black"><strong>Reminder</strong></p> + <p style="color:black">If making code changes: please ensure that you run the <a href="testing.html">Regression Tests</a> before committing changes.</p> +</div> + +Perform the following steps to commit your changes. + + # Commit changes + git commit -a + # Dry-run check + git push -n trafdeveloper_fork HEAD + # Push changes to your private fork + git push trafdeveloper_fork TRAFODION-1507 + +## Create Pull Request +You changed code needs to be reviewed by a Trafodion committer. Therefore, you need to create a pull request for your private repository; for example: <https://github.com/trafdeveloper/incubator-trafodion>. + + # Generate pull request + git pull-request + +Ensure that you include the Jira ID at the beginning of the title in your pull request. For example: + + [TRAFODION-1507] Explanation of the changes you made. + +[Automated Tests](#automated_tests) are normally triggered to run on every pull request. If you have modified the documentation or the web site, then you can skip the automated testing by adding the following phrase to the comments of the pull request: + + jenkins, skip test + +## Review Comments +The pull request gets reviewed by the committers and once you get a consensus, then the committer merges your changes into the main incubator-trafodion branch. + +## Address Review Comments +Follow the GitHub conversation on you pull request (you should be automatically subscribed). Respond to questions and issues. + +If you need to make additional changes, then do the following: + +1. Checkout the code: **<code>git checkout TRAFODION-1507</code>** +2. Make the requested changes. +3. Run regression tests. +4. Commit the changes: **<code>git commit -a</code>** +5. Push the changes back to your private git fork: **<code>git push trafdeveloper_fork TRAFODION-1507</code>** + +## Merge Changes +If all is well, a committer will merge your change into the Apache repository, which is mirrored on github. + +You may be asked to close out the JIRA or other follow up. + +Your change is done. Thanks for your contribution to Trafodion. + +# Automated Tests +Automated tests take several hours to complete from when your pull-request was approved by a committer or updated with a new commit. + +Normally, the Traf-Jenkins user will post a message in the pull-request with a link to the results. You can also check the Jenkins server to see the status even before the tests are finished. Look in the **Build History** table for the build/test job that matches your pull-request. For example, the master branch tests are located at: <https://jenkins.esgyn.com/job/Check-PR-master/> + +## Reviewing Logs + +There are two approaches to reviewing logs. + +### Approach 1 + +* The first two columns in build-job table are links to the specific sub-job. Clicl on the link to drill down. +* The console log of each job has a link to the log file directories (close to the top). Look for **Detailed logs**. + +### Approach 2 + +* Go to: <http://traf-logs.esgyn.com/PullReq/> +* Click on the number of the pull request. Example: <http://traf-logs.esgyn.com/PullReq/18/>. The next directory level is the build number. With multiple commits or re-tests, it is possible for a pull request to have multiple builds. +* Under the build number is a directory for each specific job. Example: <http://traf-logs.esgyn.com/PullReq/18/35/regress-seabase-ahw2.2/> + +## More Information +The check tests do not include all of the automated daily tests. If you (or another contributor) want, you can run additional tests on the pull request. Refer [automated test setup](automated-tests.html) for more information. \ No newline at end of file http://git-wip-us.apache.org/repos/asf/incubator-trafodion/blob/286a6e5c/docs/src/site/markdown/document.md ---------------------------------------------------------------------- diff --git a/docs/src/site/markdown/document.md b/docs/src/site/markdown/document.md new file mode 100644 index 0000000..021f322 --- /dev/null +++ b/docs/src/site/markdown/document.md @@ -0,0 +1,121 @@ +<!-- + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the + License. +--> +This page describes how to modify the Trafodion documentation. Please refer to the [Contribute](contribute.html) page for information about other ways to contribute to the Trafodion project. + +# Organization +Documents do **not** include version information as part of the file name. + +## Source + +Document | Source Format | Source Tree | Output Format +--------------------------|-----------------------|------------------------------------------------|---------------------------- +Client Installation Guide | word | ```docs/src/resources/docs``` | PDF +Command Interface Guide | word | ```docs/src/resources/docs``` | PDF +DCS Reference Guide | asciidoc | ```dcs/src/main/asciidoc``` | Web Book, PDF +DCS APIs | javadoc | ```dcs/src/main/java``` | Web Book +odb User Guide | word | ```docs/src/resources/docs``` | PDF +REST Reference Guide | asciidoc | ```core/rest/src/main/asciidoc``` | Web Book, PDF +REST APIs | javadoc | ```core/rest/src/main/java``` | Web Book +SQL Reference Manual | word | ```docs/src/resources/docs``` | PDF + +## Web + +The documentation is copied to the **```docs/target/docs```** directory on a per-release basis. This directory is referred to as the **release document directory** below. + +**Example** + +Release Document Directory | Content | Web Site Directory +----------------------------|---------------------------------------------|---------------------------------------------------- +**```docs/target/1.3.0```** | Documentation related to the 1.3.0 release. | **```incubator.trafodion.apache.org/docs/1.3.0```** +**```docs/target/1.5.0```** | Documentation related to the 1.5.0 release. | **```incubator.trafodion.apache.org/docs/1.5.0```** + +The **release document directory** contains a sub-directory per document. The naming convention for these directories is the document name in all lower case without the manual/guide qualifier and with words separated by underscore. Web-based API definitions generated by **```javadoc```** are located under each sub-directory, if applicable. + +**Example** + +Document | Sub Directory Name +--------------------------|------------------------------------------ +Client Installation Guide | **```client_installation```** +Command Interface Guide | **```command_interface```** +DCS Reference Guide | **```dcs_reference```** +odb User Guide | **```odb_user```** +REST Reference Guide | **```rest_reference```** +SQL Reference Manual | **```sql_reference```** + +The sub-directory is organized as follows. More files may be present in the sub-directory depending on document. + +File/Directory | Content +-----------------------------|------------------------------------------------------------------------------------------------------ +**```*.pdf```** | The PDF version of the document. For example, **```Trafodion_SQL_Reference_Guide.pdf```**. +**```index.html```** | Entry point for the web book. Generated by asciidoc. +**```_chapters```** | Web book chapters; that is, the actual content. +**```apidocs```** | API documentation provided as web. Generated by javadoc. +**```apidocs/index.html```** | Entry point for API documentation. +**```css```** | CSS definitions used by the web-version of the document. +**```images```** | Images used by the web-version of the document. + +# Making Changes + +## Word Documents +Do the following: + +1. Edit the document in **```docs/src/resources/docs```**. +2. Save a PDF version of the document in **```docs/src/resources/docs```**. +3. Build the web site using **```mvn clean site```** against the **```docs```** directory. + +The Word and PDF files will be copied to **```docs/target/docs```** directory. You will need to move the files to the final **Release Document Directory**; refer to [Publishing](#Publishing) below. + +## Asciidoc Documents +Please refer to DCS [Contributing to Documentation](https://github.com/apache/incubator-trafodion/blob/master/dcs/src/main/asciidoc/_chapters/appendix_contributing_to_documentation.adoc) guidance for information about working on asciidoc-based documentation. + +Once you've made the desired changes, then do the following: + +1. Build the document using **```mvn clean site```** against the directory containing the document; for example: **```dcs```** or **```core/rest'''**. +2. Verify the content in the generated **```target```** directory. The **```target/index.html```** file provides the entry point for the web book; the **```target/apidocs/index.html```** file contains the entry point for API documentation. + +You will need to move the files to the final **Release Document Directory**; refer to [Publishing](#Publishing) below. + +# Publishing + +<div class="alert alert-dismissible alert-info"> + <button type="button" class="close" data-dismiss="alert">&close;</button> + <p style="color:black">Publication is done when a committer is ready to update the external web site. You do <strong>not</strong> perform these steps as part of checking in changes.</p></div> + +**Prerequisite:** You need to build the web site before using these instructions. Please refer to [Modifying Web Site](website.html). + +You will need to copy/move the documentation from its generated location to the **Release Document Directory**. + +1. Create the **Release Document Directory** under **```docs/target/docs```** if it doesn't already exist. +2. Copy the generated document files as follows: + +Document | Populate Release Document Directory +--------------------------|------------------------------------------ +Client Installation Guide | Delete **```.docx```*** file, copy the **```.pdf```** file in **```docs/target/docs```** to the **Release Document Directory**. +Command Interface Guide | Delete **```.docx```*** file, copy the **```.pdf```** file in **```docs/target/docs```** to the **Release Document Directory**. +DCS Reference Guide | Copy **```dcs/target/site/```** to the **Release Document Directory**. +odb User Guide | Delete **```.docx```*** file, copy the **```.pdf```** file in **```docs/target/docs```** to the **Release Document Directory**. +REST Reference Guide | Copy **```core/rest/target/site/```** to the **Release Document Directory**. +SQL Reference Manual | Delete **```.docx```*** file, copy the **```.pdf```** file in **```docs/target/docs```** to the **Release Document Directory**. + +**Example â Publishing the DCS Reference Guide** + + cd $SQ_HOME/docs/target/docs + mkdir 1.3.0 + cd 1.3.0 + mkdir dcs_reference + cp -r $SQ_HOME/dcs/target/site/ . + +Once the update of the **Release Document Directory** is complete, then you can complete the [web-site publication](website.html#Publishing). + http://git-wip-us.apache.org/repos/asf/incubator-trafodion/blob/286a6e5c/docs/src/site/markdown/documentation.md ---------------------------------------------------------------------- diff --git a/docs/src/site/markdown/documentation.md b/docs/src/site/markdown/documentation.md new file mode 100644 index 0000000..eff2b45 --- /dev/null +++ b/docs/src/site/markdown/documentation.md @@ -0,0 +1,25 @@ +<!-- + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the + License. +--> +This page provides links to the per-release Trafodion documentation. + +# 1.3.0 + +Document | Formats +-----------------------------------------------|----------------------------------- +Trafodion Client Installation Guide | [PDF](docs/1.3.0/client_installation/Trafodion_Client_Installation_Guide.pdf) +Trafodion Command Interface Guide | [PDF](docs/1.3.0/command_interface/Trafodion_Command_Interface_Guide.pdf) +Trafodion Database Connectivity Services Guide | [Web Book](docs/1.3.0/dcs_reference/index.html), [API](docs/1.3.0/dcs_reference/apidocs/index.html) +Trafodion odb User Guide | [PDF](docs/1.3.0/odb_user/Trafodion_odb_User_Guide.pdf) +Trafodion SQL Reference Manual | [PDF](docs/1.3.0/sql_reference/Trafodion_SQL_Reference_Manual.pdf) http://git-wip-us.apache.org/repos/asf/incubator-trafodion/blob/286a6e5c/docs/src/site/markdown/download.md ---------------------------------------------------------------------- diff --git a/docs/src/site/markdown/download.md b/docs/src/site/markdown/download.md new file mode 100644 index 0000000..ba33c48 --- /dev/null +++ b/docs/src/site/markdown/download.md @@ -0,0 +1,36 @@ +<!-- + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the + License. +--> +This page describes how you download and install the Trafodion product environment. Please refer to [Create Development Environment](create-dev-environment.html] if you want to set up a Trafodion development environment (which includes the installation of a stand-alone Hadoop environment) based on the Trafodion source code. + +# Prerequisite +Trafodion is installed onto an existing Hadoop environment. + +# Download +The Trafodion product environment is installed using the Trafodion Installer, which operates on Trafodion binaries only. + +## Binaries +Not yet available. + +## Source +Build your own binaries from the Trafodion source code as follows: + +1. [Setup Build Environment](setup-build-environment.html). +2. [Build Trafodion](build.html) \â use **```make package```**. + +# Install +Please refer to the [Install](install.html) instructions. + + + http://git-wip-us.apache.org/repos/asf/incubator-trafodion/blob/286a6e5c/docs/src/site/markdown/enable-secure-trafodion.md ---------------------------------------------------------------------- diff --git a/docs/src/site/markdown/enable-secure-trafodion.md b/docs/src/site/markdown/enable-secure-trafodion.md new file mode 100644 index 0000000..7e23510 --- /dev/null +++ b/docs/src/site/markdown/enable-secure-trafodion.md @@ -0,0 +1,224 @@ +<!-- + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the + License. +--> +This page describes how to Trafodion security. + +# Introduction +If you do not enable security in Trafodion, a client interface to Trafodion may request a username and password, but Trafodion ignores the user name and password entered in the client interface, and the session runs as the database **```root```** user, **```DB__ROOT```**, without restrictions. If you want to restrict users, restrict access to certain users only, or restrict access to an object or operation, then you must enable security, which enforces authentication and authorization. You can enable security during installation by answering the installer's prompts or after installation by running the **```traf_authentication_setup```** script, which enables both authentication and authorization. For more information, see [Authentication Setup Script](#Authentication_Setup_Script) below. + +Trafodion does not manage user names and passwords internally but does support authentication via directory servers that support the OpenLDAP protocol, also known as LDAP servers. You can configure the LDAP servers during installation by answering the installer's prompts, or you can configure the LDAP servers manually after installation. For more information, please refer to [Configuring LDAP Servers](Configuring_LDAP_Servers) below. + +Once authentication and authorization are enabled, Trafodion allows users to be registered in the database and allows privileges on objects to be granted to users and roles (which are granted to users). Trafodion also supports component-level (or system-level) privileges, such as MANAGE_USERS, which can be granted to users and roles. Refer to [Manage Users](#Manage_Users) below. + +# Configuring LDAP Servers +To specify the LDAP server(s) to be used for authentication, you need to configure the text file **```.traf_authentication_config```**, located (by default) in **```$MY_SQROOT/sql/scripts```**. This file is a flat file, organized as a series of attribute/value pairs. Details on all the attributes and values accepted in the authentication configuration file and how to configure alternate locations can be found in [.traf_authentication_config](traf_authentication_config.html). + +A sample template file is located in **```$MY_SQROOT/sql/scripts/traf_authentication_config```**. + +Attributes and values in the authentication configuration file are separated with a colon immediately following the attribute name. In general white space is ignored, but spaces may be relevant in some values. Attribute names are always case insensitive. Multiple instances of an attribute are specified by repeating the attribute name and providing the new value. For attributes with only one instance, if the attribute is repeated, the last value provided is used. + + Attribute1: valueA + Attribute2: valueB + Attribute1: valueC + +If **```Attribute1```** has only one instance, **```valueC```** is used, otherwise, **```valueA```** and **```valueC```** are both added to the list of values for **```Attribute1```**. + +Attributes are grouped into sections; this is for future enhancements. Attributes are declared in the **```LOCAL```** section, unless otherwise specified. + +<table><tr><td><strong>Note</strong><br />Section names, attribute names, and the general layout of the authentication configuration file are subject to change in future versions of Trafodion and backward compatibility is not guaranteed.</td></tr></table> + +Specification of your directory server(s) requires at a minimum: + +<!-- This table is too complex to do in markdown --> +<table> + <tr> + <th width="15%">Setting</th> + <th width="55%">Description</th> + <th width="30%">Example</th> + </tr> + <tr> + <td><strong><code>LDAP Host Name(s)</code></strong></td> + <td>One or more names of hosts that support the OpenLDAP protocol must be specified. Trafodion will attempt to connect to all provided host names during the authentication process. The set of user names and passwords should be identical on all hosts to avoid unpredictable results. The attribute name is <strong><code>LDAPHostName</code></strong></td> + <td><pre>LDAPHostName: ldap.company.com</pre></td> + </tr> + <tr> + <td><strong><code>LDAP Port Number</code></strong></td> + <td>Port number of the LDAP server. Typically this is 389 for servers using no encryption or TLS, and 636 for servers using SSL. The attribute name is <strong><code>LDAPPort</code></strong>.</td> + <td><pre>LDAPPort: 389</pre></td> + </tr> + <tr> + <td><strong><code>LDAP Unique Identifier</code></strong></td> + <td>Attribute(s) used by the directory server that uniquely identifies the user name. You may provide one or more unique identifier specifiers.</td> + <td><pre>UniqueIdentifier: uid=,ou=users,dc=com</pre></td> + </tr> + <tr> + <td><strong><code>Encryption Level</code></strong></td> + <td>A numeric value indicating the encryption scheme used by your LDAP server. Values are: + <ul> + <li>0: Encryption not used</li> + <li>1: SSL</li> + <li>2: TLS</li> + </ul> + </td> + <td> + <pre>LDAPSSL: 2</pre> + <p>If your LDAP server uses TLS you must specify a file containing the certificate used to encrypt the password. By default the Trafodion software looks for this file in <strong><code>$MY_SQROOT/cacerts</code></strong>, but you may specify a fully qualified filename, or set the environment variable <strong><code>CACERTS_DIR</code></strong> to another directory. To specify the file containing the certificate, you set the value of the attribute <strong><code>TLS_CACERTFilename</code></strong>, located in the Defaults section.</p> + <p><strong>Example</strong></p> + <pre> +TLS_CACERTFilename: mycert.pem +TLS_CACertFilename: /usr/etc/cert.pem</pre> + </td> + </tr> + <tr> + <td><strong><code>Search username and password</code></strong></td> + <td>Some LDAP servers require a known user name and password to search the directory of user names. If your environment has that requirement, provide these âsearchâ values.</td> + <td> + <pre> +LDAPSearchDN: [email protected] +LDAPSearchPwd: Lookup123</pre> + </td> + </tr> +</table> + +There are additional optional attributes that can be used to customize Trafodion authentication. As mentioned earlier, they are described in [.traf_authentication_config](traf_authentication_config.html). + +You can test the authentication configuration file for syntactic errors using the **```ldapconfigcheck```** tool. If you have loaded the Trafodion environment (**```sqenv.sh```**), then the tool will automatically check the file at **```$MY_SQROOT/sql/scripts/.traf_authentication_config```**. If not, you can specify the file to be checked. + +**Example** + + ldapconfigcheck âfile myconfigfile + File myconfigfile is valid. + +If an error is found, the line number with the error is displayed along with the error. Please refer to [ldapconfigcheck](ldapconfigcheck.html) for more information. + +<table><tr><td><strong>Note</strong><br />The authentication configuration file needs to be propagated to all nodes, but there is a script that will do that for you described later. For now, you can test your changes on the local node.</td></tr></table> + +You can test the LDAP connection using the utility **```ldapcheck```**. To use this utility the Trafodion environment must be loaded (**```sqenv.sh```**), but the Trafodion instance does not need to be running. To test the connection only, you can specify any user name, and a name lookup will be performed using the attributes in **```.traf_authentication_config```**. + + ldapcheck [email protected] + User [email protected] not found + +If **```ldapcheck```** reports either that the user was found or the user was not found, the connection was successful. However, if an error is reported, either the configuration file is not setup correctly, or there is a problem either with your LDAP server or the connection to the server. You can get additional error detail by including the **```--verbose```** option. Please refer to [ldapcheck](ldapcheck.html) for more information. + +If you supply a password, **```ldapcheck```** will attempt to authenticate the specified **```username```** and **```password```**. The example below shows the password for illustrative purposes, but to avoid typing the password on the command line, leave the password blank (**```--password=```**) and the utility will prompt for the password with no echo. + + ldapcheck [email protected] â-password=StrongPassword + Authentication successful + +# Generate a Trafodion Certificate +Trafodion clients such as **```trafci```** will encrypt the password before sending it to Trafodion. A self-signed certificate is used to encrypt the password. The certificate and key should be generated when the **```sqgen```** script is invoked. By default, the files **```server.key```** and **```server.crt```** will be located in **```$HOME/sqcert```**. If those files are not present, since Trafodion clients will not send unencrypted passwords, you will need to manually generate those files. To do so, run the script **```sqcertgen```** located in **```$MY_SQROOT/sql/scripts```**. The script runs **```openssl```** to generate the certificate and key. + +To run openssl manually, follow the example: + + openssl req -x509 -nodes -days 365 -subj '/C=US/ST=California/L=PaloAlto/CN=host.domain.com/O=Some Company/OU=Service Connection' + - newkey rsa:2048 -keyout server.key -out server.crt + +Option | Description +------------------------------------------|----------------------- +**```-x509```** | Generate a self-signed certificate. +**```-days <validity of certificate>```** | Make the certificate valid for the days specified. +**```-newkey rsa:<bytes>```** | Generate a new private key of type RSA of length 1024 or 2048 bytes. +**```-subj <certificateinfo>```** | Specify the information that will be incorporated in the certificate. Each instance in a cluster should have a unique common name(**```CN```**). +**```-keyout <filename>```** | Write the newly generated RSA private key to the file specified. +**```-nodes```** | It is an optional parameter that specifies NOT to encrypt the private key. If you encrypt the private key, then you must enter the password every time the private key is used by an application. +**```-out <filename>```** | Write the self-signed certificate to the specified file. + +Both the public (**```server.crt```**) and private (**```server.key```**) files should be placed in the directory **```$HOME/sqcert```**. If you do not want to use the **```HOME```** directory or if you want to use different names for the private and/or public key files, then Trafodion supports environment variables to specific the alternate locations or names. + +* Trafodion first checks the environment variables **```SQCERT_PRIVKEY```** and **```SQCERT_PUBKEY```**. If they are set, Trafodion uses the fully qualified filename value of the environment variable. + + You can specify either one filename environment variable or both. + +* If at least one filename environment variable is not set, Trafodion checks the value of the environment variable **```SQCERT_DIR```**. If set, the default filename **```server.key```** or **```server.crt```** is appended to the value of the environment variable **```SQCERT_DIR```**. +* If the filename environment variable is not set and the directory environment variable is not set, Trafodion uses the default location (**```$HOME/sqcert```**) and the default filename. + +# Authentication Setup Script +The final step to enable security is to change the value of the environment variable **```TRAFODION_ENABLE_AUTHENTICATION```**from **```NO```** to **```YES```** and turn on authorization. This is achieved by invoking the **```traf_authentication_setup```** script, which is located in **```$MY_SQROOT/sql/scripts```**. + +**Usage** + +Usage: traf_authentication_setup [options] + +``` +Options: + --file <loc> Optional location of OpenLDAP configuration file + --help Prints this message + --off Disables authentication and authorization + --on Enables authentication and authorization + --setup Enables authentication + --status Returns status of authentication enablement +``` + +<!-- In HTML to control the column widths; markdown table doesn't handle Option correctly. --> +<table> + <tr> + <th width="10%">Option</th> + <th width="90%">Description</th> + </tr> + <tr> + <td><strong><code>--file</code></strong></td> + <td>If specified, then <strong><code>filename</code></strong> is copied to <strong><code>$MY_SQROOT/</code></strong>. Users working in their own private environment can refer to a site-specific configuration file from a central location.</td> + </tr> + <tr> + <td><strong><code>--on</code></strong></td> + <td> + <p><strong><code>traf_authentication_setup</code></strong> invokes <a href="ldapconfigcheck.html">ldapconfigcheck</a>) to verify the configuration file is syntactically correct. It also invokes <a href="ldapcheck.html">ldapcheck</a> to verify that a connection can be made to an LDAP server.</p> + <p>If both checks pass, the script sets the environment variable <strong><code>TRAFODION_ENABLE_AUTHENTICATION</code></strong> to <strong><code>YES</code></strong> in the file <strong><code>$MY_SQROOT/sqenvcom.sh</code></strong>, and propagates <strong><code>sqenvcom.sh</code></strong> and <strong><code>.traf_authentication_config</code></strong> to all nodes in the cluster.</p> + <p>The last step is to enable authorization by creating privilege-related metadata tables and set up default permissions with a call to the database. The list of privilege-related metadata tables, users, roles, and component privileges are logged in <strong><code>$MY_SQROOT/logs/authEnable.log</code></strong>.</p> + <p>Specifying <strong><code>--on</code></strong> requires that a valid <strong><code>.traf_authentication_config</code></strong> file exists and the Trafodion metadata initialized.</p> + </td> + </tr> + <tr> + <td><strong><code>--off</code></strong></td> + <td> + <p>If specified, then <strong><code>traf_authentication_setup</code></strong> sets the environment variable <strong><code>TRAFODION_ENABLE_AUTHENTICATION</code></strong> to <strong><code>NO</code></strong> in <strong><code>$MY_SQROOT/sqenvcom.sh</code></strong> and propagates the file to all the nodes in the cluster.</p> + <p>The last step is to disable authorization by removing any privilege-related metadata and permissions with a call to the database. The results of this operation is logged in <strong><code>$MY_SQROOT/logs/authEnable.log</code></strong>.</p> + </td> + </tr> + <tr> + <td><strong><code>--setup</code></strong></td> + <td>Use this option if the Trafodion metadata has not been initialized. This option enables authentication but does not call the database to create privilege-related metadata tables. Later, when Trafodion metadata is initialized, privilege-related metadata tables and default permissions are automatically created. + </td> + </tr> + <tr> + <td><strong><code>--status</code></strong></td> + <td>Reports the value of the environment variable <strong><code>TRAFODION_ENABLE_AUTHENTICATION</code></strong> in <strong><code>$MY_SQROOT/sqenvcom.sh</code></strong> on the current node and reports the status of security features in the database. + </td> + </tr> +</table> + +**Example** + +``` +INFO: Start of security (authentication and authorization) script Wed Mar 25 15:12:50 PDT 2xxx. + +INFO: *** Trafodion security (authentication and authorization) status *** + Authentication is ENABLED + Authorization (grant/revoke) is ENABLED + +INFO: End of security (authorization and authentication) script Wed Mar 25 15:12:54 PDT 2xxx. +``` + +<table><tr><td><strong>IMPORTANT</strong><br />Any time the environment file (<strong><code>sqenvcom.sh</code></strong>) is changed (and propagated to all nodes), Database Connectivity Services (DCS) must be restarted to pick up the new value. If the configuration file is changed, it will be re-read in 30 minutes (by default), but you can have changes take effect immediately by restarting DCS.</td></tr></table> + +To restart DCS, run the scripts **```stop-dcs.sh```** and **```start-dcs.sh```**, located in **```$MY_SQROOT/dcs-<x>.<y>.<z>/bin```**. + +# Manage Users +Users are registered in the Trafodion database and are used to enforce authorization. If security is disabled, any user can register any user at any time. However, once security is enabled, user administration is considered a secure operation, and registration of users is restricted to **```DB__ROOT```** or any user granted the **```MANAGE_USERS```** component privilege. To initially register a user, connect to Trafodion with the external user mapped to **```DB__ROOT```**(also known as the Trafodion ID). + +When security is enabled, the **```DB__ROOT```** user is registered as the **```TRAFODION```** external user name. It is recommended that the **```DB__ROOT```** user be mapped to the external user name that will be used to connect for root operations. To do this, start a **```sqlci```** session and perform the **```ALTER USER``*** command, for example: + + ALTER USER DB__ROOT SET EXTERNAL NAME trafodion_rootuser_in_ldap; + +To learn more about how to register users, grant object and component privileges, and manage users and roles, please see the [Trafodion SQL Reference Manual](docs/Trafodion_SQL_Reference_Manual.pdf). \ No newline at end of file
