http://git-wip-us.apache.org/repos/asf/incubator-trafodion/blob/681cad66/docs/cqd_reference/src/asciidoc/_chapters/schema_controls.adoc ---------------------------------------------------------------------- diff --git a/docs/cqd_reference/src/asciidoc/_chapters/schema_controls.adoc b/docs/cqd_reference/src/asciidoc/_chapters/schema_controls.adoc index 0112b9a..0f84bb0 100644 --- a/docs/cqd_reference/src/asciidoc/_chapters/schema_controls.adoc +++ b/docs/cqd_reference/src/asciidoc/_chapters/schema_controls.adoc @@ -28,12 +28,34 @@ This section describes CQDs that are used for schema controls. +[[catalog]] +== Catalog + +[cols="25%h,75%"] +|=== +| *Description* | Specifies the default catalog name for all DDL and DML statements. +| *Values* | Any valid ANSI name, including delimited names. + + + +The default is *'TRAFODION'*. +| *Usage* | Trafodion tables must be in a catalog called TRAFODION. If you mostly access Hive or native HBase tables, +then the catalog could be changed to HIVE or HBASE respectively. + + + +The default setting is overridden by any catalog name specified in a SQL statement. +| *Production Usage* | Yes. +| *Impact* | Not applicable. +| *Level* | Any. +| *Conflicts/Synergies* | Alternately you can use the SET CATALOG statement. +| *Real Problem Addressed* | Not applicable. +| *Introduced In Release* | Trafodion 1.3.0. +| *Deprecated In Release* | Not applicable. +|=== + +<<< [[schema]] == Schema [cols="25%h,75%"] |=== -| *Category* | Influencing Query Plans | *Description* | Sets the default schema for the session. | *Values* | SQL identifier. + +
http://git-wip-us.apache.org/repos/asf/incubator-trafodion/blob/681cad66/docs/cqd_reference/src/asciidoc/_chapters/table_definition.adoc ---------------------------------------------------------------------- diff --git a/docs/cqd_reference/src/asciidoc/_chapters/table_definition.adoc b/docs/cqd_reference/src/asciidoc/_chapters/table_definition.adoc new file mode 100644 index 0000000..0a93dce --- /dev/null +++ b/docs/cqd_reference/src/asciidoc/_chapters/table_definition.adoc @@ -0,0 +1,142 @@ +//// +/** +* @@@ START COPYRIGHT @@@ +* +* Licensed to the Apache Software Foundation (ASF) under one +* or more contributor license agreements. See the NOTICE file +* distributed with this work for additional information +* regarding copyright ownership. The ASF licenses this file +* to you under the Apache License, Version 2.0 (the +* "License"); you may not use this file except in compliance +* with the License. You may obtain a copy of the License at +* +* http://www.apache.org/licenses/LICENSE-2.0 +* +* Unless required by applicable law or agreed to in writing, +* software distributed under the License is distributed on an +* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +* KIND, either express or implied. See the License for the +* specific language governing permissions and limitations +* under the License. +* +* @@@ END COPYRIGHT @@@ + */ +//// + +[[table_definition]] += Table Definition + +This section describes CQDs that are used for table definition + +[[auto-query-retry-warnings]] +== ALLOW_NULLABLE_UNIQUE_KEY_CONSTRAINT + +[cols="25%h,75%"] +|=== +| *Description* | Allow Trafodion tables to be created with NULLABLE columns in the PRIMARY or STORE BY key. +| *Values* | *'ON'* or *'OFF'* + + + +The default value is *'OFF'*. +| *Usage* | Allows NULLABLE columns to be included in the PRIMARY or STORE BY key for Trafodion tables. +This CQD must be set prior to creating the table. It is not necessary to specify this CQD during DML operations on the table. + + + +When performing UPDATE STATISTICS with SAMPLE option on such tables, this CQD must be set so that an appropriate sample table can be created. +| *Production Usage* | Please consult with the Trafodion community. +| *Impact* | Two bytes are added to the key for each nullable column. +| *Level* | Session. +| *Conflicts/Synergies* | Not applicable. +| *Real Problem Addressed* | Not applicable. +| *Introduced In Release* | Trafodion 1.3.0. +| *Deprecated In Release* | Not applicable. +|=== + +<<< +[[hbase-block-size]] +== HBASE_BLOCK_SIZE + +[cols="25%h,75%"] +|=== +| *Description* | Allow Trafodion tables to be created with specified HBase block size. +| *Values* | Positive integer. + + + +The default value is *'65536'*. +| *Usage* | The value of this attribute is passed on to HBase when a Trafodion table is created in HBase. +See the https://hbase.apache.org/book.html[Apache HBase(TM) Reference Guide] for usage information . +| *Production Usage* | Yes. +| *Impact* | The impact depends on the type of table access. Choose a block size that is appropriate +for how the table is primarily accessed. +| *Level* | System. +| *Conflicts/Synergies* | Not applicable. +| *Real Problem Addressed* | Not applicable. +| *Introduced In Release* | Trafodion 1.3.0. +| *Deprecated In Release* | Not applicable. +|=== + +<<< +[[hive-default-charset]] +== HIVE_DEFAULT_CHARSET + +[cols="25%h,75%"] +|=== +| *Description* | Defines what character set the columns of Hive tables should have. +| *Values* | +*'ISO88591'* or *'UTF8'* + + + +Default: *'UTF8'* +| *Usage* | Set this to ISO88591 when reading from Hive tables with ISO8859-1 data. +| *Production Usage* | Yes. +| *Impact* | Not applicable. +| *Level* | System. +| *Conflicts/Synergies* | Not applicable. +| *Real Problem Addressed* | Not applicable. +| *Introduced In Release* | Trafodion 1.3.0. +| *Deprecated In Release* | Not applicable. +|=== + +<<< +[[hive-file-charset]] +== HIVE_FILE_CHARSET + +[cols="25%h,75%"] +|=== +| *Description* | For certain character sets that are not supported in Trafodion, you can specify the character set here +causing Trafodion to automatically convert the data to the character set specified in the HIVE_DEFAULT_CHARSET CQD. + + + +This feature is currently supported only for *'GBK'*, and only if HIVE_DEFAULT_CHARSET is set to *'UTF8'*. +| *Values* | +empty or *'GBK'* + + + +Default: empty +| *Usage* | Leave this blank, unless you want to access GBK data in Hive tables. +| *Production Usage* | Yes. +| *Impact* | Not applicable. +| *Level* | System. +| *Conflicts/Synergies* | Only applicable if HIVE_DEFAULT_CHARSET is set to *'UTF8'*. +| *Real Problem Addressed* | Not applicable. +| *Introduced In Release* | Trafodion 1.3.0. +| *Deprecated In Release* | Not applicable. +|=== + +<<< +[[hive-max-string-length]] +== HIVE_MAX_STRING_LENGTH + +[cols="25%h,75%"] +|=== +| *Description* | Hive columns of type STRING have a maximum length in Trafodion, which you can specify with this CQD. + + + +*NOTE* For UTF-8 data, this length is specified in bytes, not UTF-8 characters. +| *Values* | +1-n. + + + +Default: 32000. +| *Usage* | Set this to the lowest possible value to improve system performance. +| *Production Usage* | Yes. +| *Impact* | Not applicable. +| *Level* | System. +| *Conflicts/Synergies* | Not applicable. +| *Real Problem Addressed* | Not applicable. +| *Introduced In Release* | Trafodion 1.3.0. +| *Deprecated In Release* | Not applicable. +|=== + http://git-wip-us.apache.org/repos/asf/incubator-trafodion/blob/681cad66/docs/cqd_reference/src/asciidoc/_chapters/transactions.adoc ---------------------------------------------------------------------- diff --git a/docs/cqd_reference/src/asciidoc/_chapters/transactions.adoc b/docs/cqd_reference/src/asciidoc/_chapters/transactions.adoc index 5f590fb..6a18aab 100644 --- a/docs/cqd_reference/src/asciidoc/_chapters/transactions.adoc +++ b/docs/cqd_reference/src/asciidoc/_chapters/transactions.adoc @@ -33,7 +33,6 @@ This section describes CQDs that are used for transaction control and locking. [cols="25%h,75%"] |=== -| *Category* | Transaction Control and Locking | *Description* | Specifies the default transaction isolation level that queries use. | *Values* | *'READ UNCOMMITTED'*, *'READ COMMITTED'*, *'REPEATABLE READ'*, or *'SERIALIZABLE'*. + + @@ -76,7 +75,6 @@ then its value is used as the isolation level for updates and DDL statements. UP [cols="25%h,75%"] |=== -| *Category* | Transaction Control and Locking | *Description* | Specifies the default transaction isolation level for these update operations: INSERT, UPDATE, or DELETE. | *Values* | *'READ UNCOMMITTED'*, *'READ COMMITTED'*, *'REPEATABLE READ'*, or *'SERIALIZABLE'*. + + @@ -86,7 +84,7 @@ The default value is *'READ COMMITTED'* (ANSI). | *Impact* | When set, this CQD prevents users from doing any of these update operations: INSERT, UPDATE, or DELETE. | *Level* | Service. | *Conflicts/Synergies* | Works with the ISOLATION_LEVEL setting. Both settings are READ COMMITTED by default. -ISOLATION_LEVEL can be set to READ UNCOMMITTED. This CQD still remains READ COMMITTED. You can change it to READ UNCOMMITED to prevent queries +ISOLATION_LEVEL can be set to READ UNCOMMITTED. This CQD still remains READ COMMITTED. You can change it to READ UNCOMMITTED to prevent queries running at the service level to not perform any updates. | *Real Problem Addressed* | Not applicable. | *Introduced In Release* | Trafodion 1.3.0. http://git-wip-us.apache.org/repos/asf/incubator-trafodion/blob/681cad66/docs/cqd_reference/src/asciidoc/_chapters/update_stats.adoc ---------------------------------------------------------------------- diff --git a/docs/cqd_reference/src/asciidoc/_chapters/update_stats.adoc b/docs/cqd_reference/src/asciidoc/_chapters/update_stats.adoc index 51c0204..9695e3f 100644 --- a/docs/cqd_reference/src/asciidoc/_chapters/update_stats.adoc +++ b/docs/cqd_reference/src/asciidoc/_chapters/update_stats.adoc @@ -31,7 +31,6 @@ [cols="25%h,75%"] |=== -| *Category* | Update Statistics and Reorg | *Description* | When performing update statistics with the NECESSARY keyword or with automation, this is the number of minutes that are allowed to have elapsed since a histogram was marked as read for it to be regenerated. + + @@ -48,7 +47,7 @@ However, if a table is not used frequently, then this CQD influences how frequen + A smaller setting reduces the number of histograms being updated if there are many tables that have not been used within that interval. A larger setting updates histogram for many more tables that are not being accessed that often. -| *Production Usage* | Not applicable. +| *Production Usage* | Consult the Trafodion community. | *Impact* | Influences the number of histograms that need to be regenerated and therefore the time it takes for update statistics automation to regenerate histograms for all the tables that so qualify. | *Level* | System. @@ -67,7 +66,6 @@ This CQD qualifies the histogram to be regenerated; it is the maximum time since [cols="25%h,75%"] |=== -| *Category* | Update Statistics and Reorg | *Description* | Sets the minimum rows that need to be in a table before sampling is used to update statistics for that table. If a table has a fewer rows than the value of this CQD, then the SAMPLE option is silently ignored when performing update statistics. | *Values* | 1 through max unsigned integer. + @@ -76,7 +74,7 @@ The default value is 10000. | *Usage* | Influences for what tables sampling is used for histogram statistics generation. If the setting is smaller, then more tables qualify for sampling. If the setting is larger, then fewer tables qualify for sampling. Sampling can result in faster update statistics run times. But for smaller tables, it can also result in poor histogram accuracy which could result in poor query plans. -| *Production Usage* | Not applicable. +| *Production Usage* | Consult the Trafodion community. | *Impact* | Setting this CQD to a smaller value means that sampling is used for tables with fewer rows, when the SAMPLE option is specified as part of update statistics. This can result in less accurate histograms and poor query plans, because the sample size may be too small to generate good estimates for histograms. + @@ -90,3 +88,28 @@ However, these tables may also have more accurate histograms. | *Deprecated In Release* | Not applicable. |=== +<<< +[[ustat-min-rowcount-for-low-sample]] +== USTAT_MIN_ROWCOUNT_FOR_LOW_SAMPLE + +[cols="25%h,75%"] +|=== +| *Description* | Defines the behavior of the UPDATE STATISTICS utility. It places a lower limit on the number of sample rows that will be used when sampling. + + + +If the number of rows in the table is less than this value and sampling is used, then the sample size used will be that specified by +the <<hist-default-sample-min,HIST_DEFAULT_SAMPLE_MIN>> CQD. (which defaults to 10,000). +| *Values* | This CQD may take on any integer value greater than zero. + + + +The default for this CQD is *1,000,000*. +| *Usage* | Prevents accidental use of too-small samples when generating statistics on tables. If sample sizes are too small, +then histogram statistics will be less accurate, leading to potentially less efficient query plans. +| *Production Usage* | Consult the Trafodion community. +| *Impact* | Setting this CQD to lower values may result in smaller sample sizes for small tables. +This may slightly improve UPDATE STATISTICS run time, but at the cost of potentially less efficient queries. +| *Level* | System. +| *Conflicts/Synergies* | <<hist-default-sample-min,HIST_DEFAULT_SAMPLE_MIN>> effects the behavior of this CQD. +| *Real Problem Addressed* | Not applicable. +| *Introduced In Release* | Trafodion 1.3.0. +| *Deprecated In Release* | Not applicable. +|=== + http://git-wip-us.apache.org/repos/asf/incubator-trafodion/blob/681cad66/docs/cqd_reference/src/asciidoc/index.adoc ---------------------------------------------------------------------- diff --git a/docs/cqd_reference/src/asciidoc/index.adoc b/docs/cqd_reference/src/asciidoc/index.adoc index f67e482..9de403c 100644 --- a/docs/cqd_reference/src/asciidoc/index.adoc +++ b/docs/cqd_reference/src/asciidoc/index.adoc @@ -61,9 +61,12 @@ specific language governing permissions and limitations under the License. include::asciidoc/_chapters/about.adoc[] include::asciidoc/_chapters/introduction.adoc[] include::asciidoc/_chapters/query_plans.adoc[] +include::asciidoc/_chapters/query_execution.adoc[] include::asciidoc/_chapters/histograms.adoc[] include::asciidoc/_chapters/transactions.adoc[] include::asciidoc/_chapters/runtime_controls.adoc[] include::asciidoc/_chapters/schema_controls.adoc[] +include::asciidoc/_chapters/table_definition.adoc[] include::asciidoc/_chapters/update_stats.adoc[] include::asciidoc/_chapters/operational_controls.adoc[] +include::asciidoc/_chapters/debugging.adoc[] http://git-wip-us.apache.org/repos/asf/incubator-trafodion/blob/681cad66/docs/load_transform/pom.xml ---------------------------------------------------------------------- diff --git a/docs/load_transform/pom.xml b/docs/load_transform/pom.xml index ef00fd5..d54b0e8 100644 --- a/docs/load_transform/pom.xml +++ b/docs/load_transform/pom.xml @@ -234,15 +234,15 @@ - target/docs/<version>/<document> contains the PDF version and the web book. The web book is named index.html --> <!-- Copy the PDF file to its target directories --> - <copy file="${basedir}/target/index.pdf" tofile="${basedir}/../target/docs/load_transform_guide/Trafodion_Load_Transform_Guide.pdf" /> - <copy file="${basedir}/target/index.pdf" tofile="${basedir}/../target/docs/${project.version}/load_transform_guide/Trafodion_Load_Transform_Guide.pdf" /> + <copy file="${basedir}/target/index.pdf" tofile="${basedir}/../target/docs/load_transform/Trafodion_Load_Transform_Guide.pdf" /> + <copy file="${basedir}/target/index.pdf" tofile="${basedir}/../target/docs/${project.version}/load_transform/Trafodion_Load_Transform_Guide.pdf" /> <!-- Copy the Web Book files to their target directories --> - <copy todir="${basedir}/../target/docs/load_transform_guide"> + <copy todir="${basedir}/../target/docs/load_transform"> <fileset dir="${basedir}/target/site"> <include name="**/*.*"/> <!--All sub-directories, too--> </fileset> </copy> - <copy todir="${basedir}/../target/docs/${project.version}/load_transform_guide"> + <copy todir="${basedir}/../target/docs/${project.version}/load_transform"> <fileset dir="${basedir}/target/site"> <include name="**/*.*"/> <!--All sub-directories, too--> </fileset> http://git-wip-us.apache.org/repos/asf/incubator-trafodion/blob/681cad66/docs/load_transform/src/asciidoc/_chapters/about.adoc ---------------------------------------------------------------------- diff --git a/docs/load_transform/src/asciidoc/_chapters/about.adoc b/docs/load_transform/src/asciidoc/_chapters/about.adoc index 4ef4bd0..f15fd04 100644 --- a/docs/load_transform/src/asciidoc/_chapters/about.adoc +++ b/docs/load_transform/src/asciidoc/_chapters/about.adoc @@ -32,17 +32,19 @@ The information herein is complementary to the following Trafodion documentation * http://trafodion.apache.org/docs/odb/index.html[Trafodion odb User Guide] == Intended Audience -This guide targets anyone wanting to load data into a Trafodion database. You need to have the -following skill sets to make full of the information in this guide: +This guide targets anyone wanting to load data into a Trafodion database. + +You need to have skills in the following areas to make full use of the information in this guide: * SQL DDL and DML. -* Install and configure Linux software. +* Installation and configuration of Linux software. * Trafodion administration. * Depending on your data source, Java and/or Hadoop ecosystem usage. == New and Changed Information This is a new guide. +<<< == Notation Conventions This list summarizes the notation conventions for syntax presentation in this manual. http://git-wip-us.apache.org/repos/asf/incubator-trafodion/blob/681cad66/docs/load_transform/src/asciidoc/_chapters/bulk_load.adoc ---------------------------------------------------------------------- diff --git a/docs/load_transform/src/asciidoc/_chapters/bulk_load.adoc b/docs/load_transform/src/asciidoc/_chapters/bulk_load.adoc index 03f1385..46593b5 100644 --- a/docs/load_transform/src/asciidoc/_chapters/bulk_load.adoc +++ b/docs/load_transform/src/asciidoc/_chapters/bulk_load.adoc @@ -46,30 +46,31 @@ You copy data between two Trafodion tables by using the appropriate SELECT state === Example ``` +LOAD INTO target_table SELECT * FROM source_table WHERE custkey >= 1000 ; ``` [[bulk-load-data-from-hdfs-files]] == Load Data From HDFS Files -You can copy your data into an HDFS folder and then create an external Hive tables with the correct fields that -you point to the HDFS folder containing the data. You can also specify a WHERE clause on the source data as a filter. +You copy your data (local or remote) into an HDFS folder. Then, you create an external Hive table (with correct fields) that points +to the HDFS folder containing the data. You may also specify a WHERE clause on the source data as a filter, if needed. See the https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DDL#LanguageManualDDL-ExternalTables[External Tables] page on the https://cwiki.apache.org/confluence/display/Hive[Hive Wiki] for more information. -When Hive tables are accessed through Trafodion columns with integral, string, and char types are accessible. +Trafodion can access columns in Hive tables having integer, string and char types. See the https://cwiki.apache.org/confluence/display/Hive/LanguageManual+Types[LanguageManual Types] page on the https://cwiki.apache.org/confluence/display/Hive[Hive Wiki] for the data types available in Hive. Overall, you do the following: -1. Export the data on the remote cluster. -2. Transport files to Trafodion cluster via FTP, scp, or some other method. +1. Export the data on the local or remote cluster. +2. If applicable, transport files to Trafodion cluster via FTP, scp, or some other method. 3. Use LOAD referencing HIVE external tables. === Example - -NEED A DESCRIPTION OF THE USE CASE, THE LOAD FILE, AND ITS LAYOUT. +You have a customer-demographics in a text file, which you need to load into Trafodion. +The columns are separated by `|`. Do the following: @@ -141,12 +142,12 @@ Task: POPULATE INDEX Status: Ended ET: 00:00:05.262 [[bulk-load-data-from-hive-tables]] == Load Data From Hive Tables -You can import data from Hive using the TrafCI or sqlci command interface. Do the following: +You can import data from Hive using the trafci or sqlci command interface. Do the following: 1. Set these required Control Query Defaults (CQDs) to improve load performance: + ``` -CQD HIVE_MAX_STRING_LENGTH '1000'; // if the widest column is 1KB +CQD HIVE_MAX_STRING_LENGTH '1000'; -- if the widest column is 1KB ``` + This setting is required if there are time-related column types in the target Trafodion table. @@ -162,10 +163,10 @@ LOAD WITH NO POPULATE INDEXES INTO trafodion.sch.demo SELECT * FROM hive.hive.de ``` See the http://trafodion.incubator.apache.org/docs/sql_reference/index.html#load_statement[Trafodion SQL Reference Manual]. -for the completed syntax of the LOAD statement. +for the complete syntax of the LOAD statement. -If multiple invocations of the LOAD statement are used to incrementally load sets of data to a single target table, then several HFiles -for each partition of the target table are created. This causes inefficient access during SELECT queries and may also cause a compaction +If you use multiple LOAD statements to incrementally load sets of data into a single target table, then several HFiles are created +for each partition of the target table. This causes inefficient access during SELECT queries and may also cause a compaction to be triggered based on the policies configured in the HBase settings. To avoid this issue, it is good practice to perform a major compaction on a table that has been the target of more than two LOAD statements @@ -184,8 +185,6 @@ NOTE: The `major_compact` command returns immediately since it's not waited. Typ >> CQD HIVE_MAX_STRING_LENGTH '1000' ; >> CQD ALLOW_INCOMPATIBLE_ASSIGNMENT 'on' ; >> LOAD WITH NO POPULATE INDEXES INTO trafodion.sch.demo SELECT * FROM >> hive.hive.demo ; - -<NEED MORE HERE, INCLUDING CREATING THE TRAFODION TABLE AND WHAT TO DO ABOUT INDEXES> ``` [[bulk-load-data-from-external-databases]] @@ -243,7 +242,8 @@ export JAVA_OPTIONS=-Dmapred.child.java.opts=\-Djava.security.egd=file:/dev/uran ==== List All Oracle Tables ``` -sqoop list-tables --driver oracle.jdbc.OracleDriver --connect jdbc:oracle:thin:@<Oracle host name>:<port>/<database> +sqoop list-tables --driver oracle.jdbc.OracleDriver +--connect jdbc:oracle:thin:@<Oracle host name>:<port>/<database> --username <user-name> --password <password> ``` @@ -279,9 +279,12 @@ which does not exist in the data itself. === Example -ADD EXAMPLE HERE. - ``` +sqoop import --connect jdbc:oracle:thin:@localhost:1521/orcl +--username trafdemo --password traf123 --table CUSTOMER +--split-by CUSTNUM --hive-import --create-hive-table +--hive-table customers --hive-overwrite --null-string '' +--null-non-string '' --hive-drop-import-delims--verbose ``` http://git-wip-us.apache.org/repos/asf/incubator-trafodion/blob/681cad66/docs/load_transform/src/asciidoc/_chapters/introduction.adoc ---------------------------------------------------------------------- diff --git a/docs/load_transform/src/asciidoc/_chapters/introduction.adoc b/docs/load_transform/src/asciidoc/_chapters/introduction.adoc index 2bbc3e1..05e0cd5 100644 --- a/docs/load_transform/src/asciidoc/_chapters/introduction.adoc +++ b/docs/load_transform/src/asciidoc/_chapters/introduction.adoc @@ -36,19 +36,21 @@ There are two methods used to load data into a Trafodion table. Both methods can | *Bulk Load* | Large data volumes + Stage data and load in the batches | Trafodion Bulk Loader | *Trickle Load* | Small data volumes + -Insert data at it arrives | odb tool + -ETL tool or custom ODBC/JDBC application +Insert data as it arrives | ETL tool + +Custom ODBC/JDBC application + +User-Defined Functions + +odb Tool |=== These two methods use four types of SQL insert statements * *Bulk Load* -** http://trafodion.apache.org/docs/sql_reference/index.html#load_statement[LOAD] +** http://trafodion.incubator.apache.org/docs/sql_reference/index.html#load_statement[LOAD] * *Trickle Load* ** http://trafodion.incubator.apache.org/docs/sql_reference/index.html#insert_statement[INSERT] -** http://trafodion.apache.org/docs/sql_reference/index.html#upsert_statement[UPSERT] -** http://trafodion.apache.org/docs/sql_reference/index.html#upsert_statement[UPSERT USING LOAD] +** http://trafodion.incubator.apache.org/docs/sql_reference/index.html#upsert_statement[UPSERT] +** http://trafodion.incubator.apache.org/docs/sql_reference/index.html#upsert_statement[UPSERT USING LOAD] The http://trafodion.incubator.apache.org/docs/sql_reference/index.html[Trafodion SQL Reference Manual] provides syntax descriptions for these statements. @@ -62,6 +64,7 @@ The data source defines what type of load approach and method you use: ** _Native HBase Tables_: Direct load. ** _Disparate Data Source_: Write Java/C++ UDF to read data from source and pass rows to LOAD. +<<< * *Trickle Load* (odb utility) ** _Text Files_: Direct access ** _pipes_: Via `stdin` @@ -72,6 +75,7 @@ For more information, refer to: * <<bulk-load,Bulk Load>> * <<trickle-load, Trickle Load>> +<<< [[introduction-insert-types]] === Insert Types http://git-wip-us.apache.org/repos/asf/incubator-trafodion/blob/681cad66/docs/load_transform/src/asciidoc/_chapters/tables_indexes.adoc ---------------------------------------------------------------------- diff --git a/docs/load_transform/src/asciidoc/_chapters/tables_indexes.adoc b/docs/load_transform/src/asciidoc/_chapters/tables_indexes.adoc index 71ea7f7..a9f8f78 100644 --- a/docs/load_transform/src/asciidoc/_chapters/tables_indexes.adoc +++ b/docs/load_transform/src/asciidoc/_chapters/tables_indexes.adoc @@ -60,6 +60,7 @@ The number of partitions must also be specified during table creation. You choos cluster and the expected size of the table. A salted table can split if more data is added to it than initially estimated. If this happens, then more than one partition having rows with the same salt value, which may result in suboptimal execution plans for the table. +<<< You can also choose not to salt Trafodion tables. This is similar to range partitioning in a traditional database. The number of partitions grows with the size of the table, and range boundaries are determined by HBase based on the specified split policy. @@ -71,18 +72,11 @@ when compared to a Hive table with equivalent data since HBase stores the key se HBase provides several types of encoding to avoid storing the same key value to disk for every column in the row. HBase also supports various types of compression of the entire data block, regardless whether it is encoded or not. - See http://hbase.apache.org/book.html#compression[Appendix E: Compression and Data Block Encoding In HBase] in the -http://hbase.apache.org/book.html[Apache HBase Reference Guide] for a comparison of various compression and encoding algorithms. - -For a discussion on the space savings achieved by various types of encoding and compression schemes, please see the -https://blogs.apache.org/hbase/category/General["The Effect of ColumnFamily, RowKey and KeyValue Design on HFile Size"] HBase blog entry, -dated Friday April 11, 2014. - -The performance of an algorithm is also a relevant factor when deciding on the type of compression. For more information, see slides 6, 7, -and 8 of Govind Kamat's and Sumeet Singh's presentation, -http://www.slideshare.net/Hadoop_Summit/kamat-singh-june27425pmroom210cv2["Compression Options in Hadoop - A Tale of Tradeoffs"]. - +http://hbase.apache.org/book.html[Apache HBase Reference Guide] for a comparison of various compression and encoding algorithms. Use the +information in the http://hbase.apache.org/book.html#data.block.encoding.types[Which Compressor or Data Block Encoder To Use] section to +determine the best compression technique for your tables. +<<< [[create-trafodion-tables-and-indexes]] == Create Tables and Indexes @@ -204,5 +198,3 @@ using ON EXISTING COLUMNS. Note that using ON NECESSARY COLUMNS will only identi requested by the optimizer but do not exist. The current implementation of automation does not know which existing histograms might be stale. - -LocalWords: c7 http://git-wip-us.apache.org/repos/asf/incubator-trafodion/blob/681cad66/docs/load_transform/src/asciidoc/_chapters/trickle_load.adoc ---------------------------------------------------------------------- diff --git a/docs/load_transform/src/asciidoc/_chapters/trickle_load.adoc b/docs/load_transform/src/asciidoc/_chapters/trickle_load.adoc index 3d3bd1b..4991619 100644 --- a/docs/load_transform/src/asciidoc/_chapters/trickle_load.adoc +++ b/docs/load_transform/src/asciidoc/_chapters/trickle_load.adoc @@ -472,14 +472,14 @@ In addition, you need to convert the date format and replace all occurrences of The following map file accomplishes these goals: -*** +``` $ cat test/load_map/ml1.map + -# Map file to load TRAFODION.MFTEST.FRIENDS from friends.dat + -ID:seq:1 # Inserts into ID column a sequence starting from 1 + -NAME:4:REPLACE:Lucia:Lucy # Loads field #4 into NAME and replace all occurrences of Lucia with Lucy + -AGE:2 # Loads field #2 (they start from zero) into AGE + +# Map file to load TRAFODION.MFTEST.FRIENDS from friends.dat +ID:seq:1 # Inserts into ID column a sequence starting from 1 +NAME:4:REPLACE:Lucia:Lucy # Loads field #4 into NAME and replace all occurrences of Lucia with Lucy +AGE:2 # Loads field #2 (they start from zero) into AGE BDATE:6:DCONV:d.b.y # Loads field #6 into BDATE converting date format from dd mmm yyyy -*** +``` Load as follows: http://git-wip-us.apache.org/repos/asf/incubator-trafodion/blob/681cad66/docs/load_transform/src/asciidoc/_chapters/troubleshoot.adoc ---------------------------------------------------------------------- diff --git a/docs/load_transform/src/asciidoc/_chapters/troubleshoot.adoc b/docs/load_transform/src/asciidoc/_chapters/troubleshoot.adoc index 112b0ee..77624ec 100644 --- a/docs/load_transform/src/asciidoc/_chapters/troubleshoot.adoc +++ b/docs/load_transform/src/asciidoc/_chapters/troubleshoot.adoc @@ -112,8 +112,8 @@ you can specify the default attribute, USE_HIVE_SOURCE. This causes the Hive tab NOTE: The name of the Hive table must use the Trafodion table name as its prefix. For example, if the Trafodion table is TRAFODION.SCH.DEMO, then the Hive -table name can be DEMO_SRC. In this case, set the attribute as follows: -+ +table name can be DEMO_SRC. In this case, set the attribute as follows: + + + ``` CONTROL QUERY DEFAULT USE_HIVE_SOURCE '_SRC' ; CREATE INDEX demo_ix ON sch.demo(name) ; http://git-wip-us.apache.org/repos/asf/incubator-trafodion/blob/681cad66/docs/sql_reference/src/asciidoc/_chapters/about.adoc ---------------------------------------------------------------------- diff --git a/docs/sql_reference/src/asciidoc/_chapters/about.adoc b/docs/sql_reference/src/asciidoc/_chapters/about.adoc index de3dcb8..0cc63d6 100644 --- a/docs/sql_reference/src/asciidoc/_chapters/about.adoc +++ b/docs/sql_reference/src/asciidoc/_chapters/about.adoc @@ -87,9 +87,7 @@ Trafodion SQL statements. Management System (RMS) and describes the RMS counters that are returned. | <<OLAP_Functions,OLAP Functions>> | Describes specific on line analytical processing functions. | <<Reserved_Words,Appendix A: Reserved Words>> | Lists the words that are reserved in Trafodion SQL. -| <<Control_Query_Default,Appendix B: Control Query Default (CQD) Attributes>> | -Describes the Control Query Default (CQD) attributes that are supported in a Trafodion SQL environment. -| <<Limits,Appendix C: Limits>> | Describes limits in Trafodion SQL. +| <<Limits,Appendix B: Limits>> | Describes limits in Trafodion SQL. |=== http://git-wip-us.apache.org/repos/asf/incubator-trafodion/blob/681cad66/docs/sql_reference/src/asciidoc/_chapters/cqds.adoc ---------------------------------------------------------------------- diff --git a/docs/sql_reference/src/asciidoc/_chapters/cqds.adoc b/docs/sql_reference/src/asciidoc/_chapters/cqds.adoc deleted file mode 100644 index c96d125..0000000 --- a/docs/sql_reference/src/asciidoc/_chapters/cqds.adoc +++ /dev/null @@ -1,329 +0,0 @@ -//// -/** -* @@@ START COPYRIGHT @@@ -* -* Licensed to the Apache Software Foundation (ASF) under one -* or more contributor license agreements. See the NOTICE file -* distributed with this work for additional information -* regarding copyright ownership. The ASF licenses this file -* to you under the Apache License, Version 2.0 (the -* "License"); you may not use this file except in compliance -* with the License. You may obtain a copy of the License at -* -* http://www.apache.org/licenses/LICENSE-2.0 -* -* Unless required by applicable law or agreed to in writing, -* software distributed under the License is distributed on an -* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY -* KIND, either express or implied. See the License for the -* specific language governing permissions and limitations -* under the License. -* -* @@@ END COPYRIGHT @@@ - */ -//// - -<<< -<<< -[[cqd_attributes]] -= Control Query Default (CQD) Attributes - -This section describes CQDs that are used to override system-level -default settings. - -[[hive_environment_cqds]] -== Hive Environment CQDs - -[[hive_max_string_length]] -=== HIVE_MAX_STRING_LENGTH - -[cols="25%h,75%"] -|=== -| Category | Hive -| Description | Maximum supported string length for the _string_ data type in Hive. All -string columns in Hive tables get converted to VARCHAR(_n_ BYTES) -CHARACTER SET UTF8, with _n_ being the value of this CQD. -| Values | The default value is 32000. -|=== - -<<< -[[managing_histograms]] -== Managing Histograms - -[[cache_histograms_refresh_interval]] -=== CACHE_HISTOGRAMS_REFRESH_INTERVAL - -[cols="25%h,75%"] -|=== -| Category | Histograms -| Description | Defines the time interval after which timestamps for cached histograms -are checked to be refreshed. -| Values | Unsigned integer. Unit is seconds. + - + -The default value is '3600' (1 hour). -| Usage | Histogram statistics are cached so that the compiler can avoid access to -the metadata tables, thereby reducing compile times. The time stamp of -the tables are checked against those of the cached histograms at an -interval specified by this CQD, in order to see if the cached histograms -need to be refreshed. + - + -You can increase the interval to reduce the impact on compile times as -long as you do not need to obtain fresh statistics more frequently in -order to improve query performance. It may - + -be that the default interval is too long and you would rather refresh -the statistics more frequently than the default one hour, in order to -improve query performance at the cost of increased compile times. - + -This setting depends on how frequently you are updating statistics on -tables. There is no point in refreshing statistics frequently when -statistics are not being updated during that time. On the other hand if -you are updating statistics, or generating them for the first time on -freshly loaded tables frequently enough, and you want these to be picked -up immediately by the compiler because you have seen this to have a -dramatic impact on plan quality, then you can make the refresh more -frequent. -| Production Usage | Not applicable. -| Impact | Longer histogram refresh intervals can improve compile times. However, -the longer the refresh interval the more obsolete the histograms. That -could result in poor performance for queries that could leverage -recently updated statistics. -| Level | System or Service Not applicable -| Addressing the Real Problem | Not applicable. -|=== - -<<< -[[hist_no_stats_refresh_interval]] -=== HIST_NO_STATS_REFRESH_INTERVAL - -[cols="25%h,75%"] -|=== -| Category | Histograms -| Description | Defines the time interval after which the fake histograms in the cache should be refreshed unconditionally. -| Values | Integer. Unit is seconds. + - + -The default value is '3600' (1 hour). -| Usage | Histogram statistics are "fake" when update statistics is not being run, -but instead the customer is updating the histogram tables directly with -statistics to guide the optimizer. This may be done if the data in the -table is very volatile (such as for temporary tables), update statistics -is not possible because of constant flush and fill of the table -occurring, and statistics are manually set to provide some guidance to -the optimizer to generate a good plan. - + -If these fake statistics are updated constantly to reflect the data -churn, this default can be set to 0. This would ensure that the -histograms with fake statistics are not cached, and are always -refreshed. If these fake statistics are set and not touched again, then -this interval could be set very high. -| Production Usage | Not applicable. -| Impact | Setting a high interval improves compilation time. However, if statistics are being updated, the compiler may be working with obsolete -histogram statistics, potentially resulting in poorer plans. -| Level | Service. -| Conflicts/Synergies | Not applicable. -| Addressing the Real Problem | Not applicable. -|=== - -[[hist_prefetch]] -=== HIST_PREFETCH - -[cols="25%h,75%"] -|=== -| Category | Histograms -| Description | Influences the compiler to pre-fetch the histograms and save them in cache. -| Values | 'ON' Pre-fetches the histograms. + -'OFF' Does not pre-fetch the histograms. + - + - The default value is 'ON'. -| Usage | You may want to turn this off if you don't want to pre-fetch a large number of histograms, many of which may not be used. -| Production Usage | Not applicable. -| Impact | Though it makes compilation time faster, it may result in the histogram cache to be filled with histograms that may never be used. -| Level | System or Service. -| Conflicts/Synergies | Use this CQD with CACHE_HISTOGRAMS. If CACHE_HISTOGRAMS is OFF, then this CQD has no effect. -| Addressing the Real Problem | Not applicable. -|=== - -[[hist_rowcount_requiring_stats]] -=== HIST_ROWCOUNT_REQUIRING_STATS - -[cols="25%h,75%"] -|=== -| Category | Histograms -| Description | Specifies the minimum row count for which the optimizer needs histograms, in order to compute better cardinality estimates. The -optimizer does not issue any missing statistics warnings for tables whose size is smaller than the value of this CQD. -| Values | Integer. + - + -The default value is '50000'. -| Usage | Use this CQD to reduce the number of statistics warnings. -| Production Usage | Not applicable. -| Impact | Missing statistics warnings are not displayed for smaller tables, which in most cases don't impact plan quality much. -However, there may be some exceptions where missing statistics on small tables could result in less than optimal plans. -| Level | System -| Conflicts/Synergies | Use this CQD with HIST_MISSING_STATS_WARNING_LEVEL. If the warning level CQD is 0, then this CQD does not have any effect. Also, for tables -having fewer rows than set in this CQD, no warnings are displayed irrespective of the warning level. -| Addressing the Real Problem | Not applicable. -|=== - -<<< -[[optimizer]] -== Optimizer - -[[join_order_by_user]] -=== JOIN_ORDER_BY_USER - -[cols="25%h,75%"] -|=== -| Category | Influencing Query Plans -| Description | Enables or disables the join order in which the optimizer joins the tables to be the sequence of the tables in the FROM clause of the query. -| Values | 'ON' Join order is forced. + - + -'OFF' Join order is decided by the optimizer. + - + -The default value is 'OFF'. -| Usage | When set to ON, the optimizer considers only execution plans that have the join order matching the sequence of the tables in the FROM clause. -| Production Usage | This setting is to be used only for forcing a desired join order that was not generated by default by the optimizer. It can be used as a -workaround for query plans with inefficient join order. -| Impact | Because you are in effect forcing the optimizer to use a plan that joins the table in the order specified in the FROM clause, -the plan generated may not be the optimal one. -| Level | Query -| Conflicts/Synergies | Not applicable. -| Addressing the Real Problem | Not applicable. -|=== - -[[mdam_scan_method]] -=== MDAM_SCAN_METHOD - -[cols="25%h,75%"] -|=== -| Category | Influencing Query Plans -| Description | Enables or disables the Multi-Dimensional Access Method. -| Values | 'ON' MDAM is considered. + -'OFF' MDAM is disabled. + - + -The default value is 'ON'. -| Usage | In certain situations, the optimizer might choose MDAM inappropriately, causing poor performance. -In such situations you may want to turn MDAM OFF for the query it is effecting. -| Production Usage | Not applicable. -| Impact | Table scans with predicates on non-leading clustering key column(s) could benefit from MDAM access method if -the leading column(s) has a small number of distinct values. Turning MDAM off results in a longer scan time for such queries. -| Level | Set this CQD at the query level when MDAM is not working efficiently for a specific query. -However, there may be cases (usually a defect) where a larger set of queries is being negatively impacted by MDAM. -In those cases you may want to set it at the service or system level. -| Conflicts/Synergies | Not applicable. -| Addressing the Real Problem | Not applicable. -|=== - -[[subquery_unnesting]] -=== SUBQUERY_UNNESTING - -[cols="25%h,75%"] -|=== -| Category | Influencing Query Plans -| Description | Controls the optimizer's ability to transform nested sub-queries into regular join trees. -| Values | 'ON' Subquery un-nesting is considered. + -'OFF' Subquery un-nesting is disabled. + - + -The default value is 'ON'. -| Usage | Use this control to disable subquery un-nesting in the rare situation when un-nesting results in an inefficient query execution plan. -| Production usage | Not applicable. -| Impact | In general, subquery un-nesting results in more efficient execution plans for queries with nested sub-queries. -Use only as a workaround for observed problems due to un-nesting. -| Level | Query -| Conflicts/Synergies | Not applicable. -| Addressing the Real Problem | Not applicable. -|=== - -<<< -[[managing_schemas]] -== Managing Schemas - -[[schema]] -=== SCHEMA - -[cols="25%h,75%"] -|=== -| Category | Schema controls -| Description | Sets the default schema for the session. -| Values | SQL identifier. + - + -The default is SEABASE. -| Usage | A SET SCHEMA statement, or a CONTROL QUERY DEFAULT SCHEMA statement, can be used to override the default schema name. -| Production Usage | It is a convenience so you do not have to type in two-part names. -| Impact | Not applicable. -| Level | Any. -| Conflicts/Synergies | Alternately you can use the SET SCHEMA statement. -| Addressing the Real Problem | Not applicable. -|=== - -<<< -[[transaction_control_and_locking]] -== Transaction Control and Locking - -[[block_to_prevent_halloween]] -=== BLOCK_TO_PREVENT_HALLOWEEN - -[cols="25%h,75%"] -|=== -| Category | Runtime controls -| Description | A self-referencing insert is one which inserts into a target table and -also scans from the same target table as part of the query that produces -rows to be inserted. Inconsistent results are produced by the insert -statement if the statement scans rows which have been inserted by the -same statement. This is sometimes called the "Halloween problem." -Trafodion prevents the Halloween problem using one of two methods: 1) -the blocking method uses a SORT operation to ensure all rows have been -scanned before any are inserted, or 2) the disk process (ESAM) locks -method tracks the rows which have already been inserted and the SCAN -operator skips these rows. + - + -The compiler chooses the blocking method in cases in which static -analysis of the plan indicates that the disk process locks method cannot -be used. However, the compiler does not evaluate one condition that -would prevent the use of the disk process locks method: the AUTOCOMMIT -setting in which the statement is executed. Instead the compiler assumes -that the statement is executed with the default setting for AUTOCOMMIT, -'ON'. If AUTOCOMMIT is set to 'OFF' and self-referencing insert -statement which uses the disk process locks method is executed, then a -runtime error (SQLCODE 8107) is raised. + - + -This CQD is used to force the compiler to use the blocking method to -prevent error 8107. -| Values | 'OFF' The compiler is free to choose which method to use to prevent the Halloween problem. + -'ON' The compiler is forced to use the blocking method. + - + -The default value is 'ON'. -| Usage | Change this default to 'ON' if error 8107 is raised for a self-referencing insert statement which is -executed in a session with AUTOCOMMIT set to 'OFF'. -| Production Usage | Not applicable. -| Impact | Using the 'ON' value in conditions that require it allows successful completion of the insert statement. -Using the 'ON' value when not required can decrease performance of some self-referencing insert statements. -| Level | If self-referencing insert statements which execute with AUTOCOMMIT 'OFF' can be restricted to a service level, -then this default should be set to 'ON' only for that service level. Otherwise the setting should be made for the system. -| Conflicts/Synergies | Not applicable. -| Addressing the Real Problem | Not applicable. -|=== - -<<< -[[upd_ordered]] -=== UPD_ORDERED - -[cols="25%h,75%"] -|=== -| Category | Influencing Query Plans -| Description | Controls whether rows should be inserted, updated, or deleted in clustering key order. -| Values | 'ON' The optimizer generates and considers plans where the rows are inserted, updated, or deleted in clustering key order. + -'OFF' The optimizer does not generate plans where the rows must be inserted, updated, or deleted in clustering key order. - + -The default value is 'ON'. -| Usage | Inserting, updating or deleting rows in the clustering key order is most efficient and highly recommended. Turning this CQD OFF may result in -saving the data sorting cost but at the expense of having less efficient random I/O Insert/Update/Delete operations. + - + -If you know that the data is already sorted in clustering key order, or is mostly in clustering key order, -so that it would not result in random I/O, you could set this CQD to OFF. -| Production Usage | Not applicable. -| Impact | If turned OFF, the system may perform large number of inefficient Random I/Os when performing Insert/Update/Delete operations. -| Level | Query -| Conflicts/Synergies | Not applicable. -| Addressing the Real Problem | Not applicable. -|=== - http://git-wip-us.apache.org/repos/asf/incubator-trafodion/blob/681cad66/docs/sql_reference/src/asciidoc/_chapters/introduction.adoc ---------------------------------------------------------------------- diff --git a/docs/sql_reference/src/asciidoc/_chapters/introduction.adoc b/docs/sql_reference/src/asciidoc/_chapters/introduction.adoc index ea73f2a..2badda6 100644 --- a/docs/sql_reference/src/asciidoc/_chapters/introduction.adoc +++ b/docs/sql_reference/src/asciidoc/_chapters/introduction.adoc @@ -71,7 +71,8 @@ You can use Trafodion SQL statements to read, update, and create HBase tables. * <<trafodion_sql_tables_versus_native_hbase_tables,Trafodion SQL Tables Versus Native HBase Tables>> * <<supported_sql_statements_with_hbase_tables,Supported SQL Statements With HBase Tables>> -For a list of Control Query Default (CQD) settings for the HBase environment, see <<HBase_Environment_CQDs,HBase Environment CQDs>>. +For a list of Control Query Default (CQD) settings for the HBase environment, see the +http://trafodion.incubator.apache.org/docs/cqd_reference/index.hmtl[Trafodion Control Query Default (CQD) Reference Guide]. [[ways_to_access_hbase_tables]] === Ways to Access HBase Tables @@ -223,7 +224,8 @@ You can use Trafodion SQL statements to access Hive tables. * <<type_mapping_from_hive_to_trafodion_sql,Type Mapping From Hive to Trafodion SQL>> * <<supported_sql_statements_with_hive_tables,Supported SQL Statements With Hive Tables>> -For a list of Control Query Default (CQD) settings for the Hive environment, see <<Hive_Environment_CQDs,Hive Environment CQDs>>. +For a list of Control Query Default (CQD) settings for the Hive environment, see the +http://trafodion.incubator.apache.org/docs/cqd_reference/index.hmtl[Trafodion Control Query Default (CQD) Reference Guide]. [[ansi_names_for_hive_tables]] === ANSI Names for Hive Tables @@ -239,7 +241,7 @@ This example should work if a Hive table named `T` has already been defined: ``` set schema hive.hive; -cqd hive_max_string_length '20'; -- creates a more readable display +CQD HIVE_MAX_STRING_LENGTH '20'; -- creates a more readable display select * from t; -- implicit table name set schema trafodion.seabase; @@ -267,7 +269,8 @@ Trafodion performs the following data-type mappings: | `timestamp` | `timestamp(6)`^2^ |=== -1. The value `_n_` is determined by `CQD HIVE_MAX_STRING_LENGTH`. See <<hive_environment_cqds,Hive Environment CQDs>>. +1. The value `_n_` is determined by `CQD HIVE_MAX_STRING_LENGTH`. See the +http://trafodion.incubator.apache.org/docs/cqd_reference/index.hmtl[Trafodion Control Query Default (CQD) Reference Guide]. 2. Hive supports timestamps with nanosecond resolution (precision of 9). Trafodion SQL supports only microsecond resolution (precision 6). [[supported_sql_statements_with_hive_tables]] http://git-wip-us.apache.org/repos/asf/incubator-trafodion/blob/681cad66/docs/sql_reference/src/asciidoc/_chapters/sql_statements.adoc ---------------------------------------------------------------------- diff --git a/docs/sql_reference/src/asciidoc/_chapters/sql_statements.adoc b/docs/sql_reference/src/asciidoc/_chapters/sql_statements.adoc index 80ce893..2a33dcc 100644 --- a/docs/sql_reference/src/asciidoc/_chapters/sql_statements.adoc +++ b/docs/sql_reference/src/asciidoc/_chapters/sql_statements.adoc @@ -1086,7 +1086,7 @@ control-default-option is: * `_attribute_` + is a character string that represents an attribute name. For descriptions of these attributes, -see <<Control_Query_Default_Attributes,Control Query Default (CQD) Attributes>>. +see the http://trafodion.incubator.apache.org/docs/cqd_reference/index.html[Trafodion Control Query Default (CQD) Reference Guide]. * `_attr-value_` + @@ -7444,8 +7444,8 @@ see the <<control_query default_statement,Control Query Default Statement>>. * `_attribute-name_[, MATCH {FULL | PARTIAL }]` + displays only the defaults that match, either fully or partially, the _attribute_ used in CONTROL QUERY DEFAULT -statements. The match is not case-sensitive. For descriptions of these attributes, see -<<control_query_default_attributes,CONTROL QUERY (CQD) DEFAULT Attributes>>. +statements. The match is not case-sensitive. For descriptions of these attributes, see the +http://trafodion.incubator.apache.org/docs/cqd_reference/index.html[Trafodion Control Query Default (CQD) Reference Guide]. + MATCH FULL specifies that _attribute-name_ must be the same as the attribute name used in a control query default statement. match partial specifies that _attribute-name_ must be included in the attribute name used in a control http://git-wip-us.apache.org/repos/asf/incubator-trafodion/blob/681cad66/docs/sql_reference/src/asciidoc/index.adoc ---------------------------------------------------------------------- diff --git a/docs/sql_reference/src/asciidoc/index.adoc b/docs/sql_reference/src/asciidoc/index.adoc index f874749..0d8f81e 100644 --- a/docs/sql_reference/src/asciidoc/index.adoc +++ b/docs/sql_reference/src/asciidoc/index.adoc @@ -72,7 +72,6 @@ include::asciidoc/_chapters/sql_functions_and_expressions.adoc[] include::asciidoc/_chapters/olap_functions.adoc[] include::asciidoc/_chapters/runtime_stats.adoc[] include::asciidoc/_chapters/reserved_words.adoc[] -include::asciidoc/_chapters/cqds.adoc[] include::asciidoc/_chapters/limits.adoc[] http://git-wip-us.apache.org/repos/asf/incubator-trafodion/blob/681cad66/docs/src/site/markdown/documentation.md ---------------------------------------------------------------------- diff --git a/docs/src/site/markdown/documentation.md b/docs/src/site/markdown/documentation.md index 720f8cf..a3848eb 100644 --- a/docs/src/site/markdown/documentation.md +++ b/docs/src/site/markdown/documentation.md @@ -16,40 +16,46 @@ This page provides links to the per-release Trafodion documentation. # Latest -Document | Formats ------------------------------------------------|----------------------------------- -Trafodion Client Installation Guide | [Web Book](docs/client_install/index.html),[PDF](docs/client_install/Trafodion_Client_Installation_Guide.pdf) -Trafodion Command Interface Guide | [Web Book](docs/command_interface/index.html),[PDF](docs/command_interface/Trafodion_Command_Interface_Guide.pdf) -Trafodion Database Connectivity Services Guide | [Web Book](docs/dcs_reference/index.html),[API](docs/dcs_reference/apidocs/index.html) -Trafodion Messages Guide | [Web Book](docs/messages_guide/index.html),[PDF](docs/messages_guide/Trafodion_Messages_Guide.pdf) -Trafodion odb User Guide | [Web Book](docs/odb/index.html),[PDF](docs/odb/Trafodion_odb_User_Guide.pdf) -Trafodion Provisioning Guide | [Web Book](docs/provisioning_guide/index.html),[PDF](docs/provisioning_guide/Trafodion_Provisioning_Guide.pdf) -Trafodion REST Server Reference Guide | [Web Book](docs/rest_reference/index.html),[API](docs/rest_reference/apidocs/index.html) -Trafodion SQL Reference Manual | [Web Book](docs/sql_reference/index.html),[PDF](docs/sql_reference/Trafodion_SQL_Reference_Manual.pdf) +Document | Formats +------------------------------------------------------|----------------------------------- +Trafodion Client Installation Guide | [Web Book](docs/client_install/index.html),[PDF](docs/client_install/Trafodion_Client_Installation_Guide.pdf) +Trafodion Command Interface Guide | [Web Book](docs/command_interface/index.html),[PDF](docs/command_interface/Trafodion_Command_Interface_Guide.pdf) +Trafodion Control Query Default (CQD) Reference Guide | [Web Book](docs/cqd_reference/index.html),[PDF](docs/cqd_interface/Trafodion_CQD_Reference_Guide.pdf) +Trafodion Database Connectivity Services Guide | [Web Book](docs/dcs_reference/index.html),[API](docs/dcs_reference/apidocs/index.html) +Trafodion Load and Transform Guide | [Web Book](docs/load_transform/index.html),[PDF](docs/load_transform/Trafodion_Load_Transform_Guide.pdf) +Trafodion Messages Guide | [Web Book](docs/messages_guide/index.html),[PDF](docs/messages_guide/Trafodion_Messages_Guide.pdf) +Trafodion odb User Guide | [Web Book](docs/odb/index.html),[PDF](docs/odb/Trafodion_odb_User_Guide.pdf) +Trafodion Provisioning Guide | [Web Book](docs/provisioning_guide/index.html),[PDF](docs/provisioning_guide/Trafodion_Provisioning_Guide.pdf) +Trafodion REST Server Reference Guide | [Web Book](docs/rest_reference/index.html),[API](docs/rest_reference/apidocs/index.html) +Trafodion SQL Reference Manual | [Web Book](docs/sql_reference/index.html),[PDF](docs/sql_reference/Trafodion_SQL_Reference_Manual.pdf) # 2.0.0 (In Development) -Document | Formats ------------------------------------------------|----------------------------------- -Trafodion Client Installation Guide | [Web Book](docs/2.0.0/client_install/index.html),[PDF](docs/2.0.0/client_install/Trafodion_Client_Installation_Guide.pdf) -Trafodion Command Interface Guide | [Web Book](docs/2.0.0/command_interface/index.html),[PDF](docs/2.0.0/command_interface/Trafodion_Command_Interface_Guide.pdf) -Trafodion Database Connectivity Services Guide | [Web Book](docs/2.0.0/dcs_reference/index.html),[API](docs/2.0.0/dcs_reference/apidocs/index.html) -Trafodion Messages Guide | [Web Book](docs/2.0.0/messages_guide/index.html),[PDF](docs/2.0.0/messages_guide/Trafodion_Messages_Guide.pdf) -Trafodion odb User Guide | [Web Book](docs/2.0.0/odb/index.html),[PDF](docs/2.0.0/odb/Trafodion_odb_User_Guide.pdf) -Trafodion Provisioning Guide | [Web Book](docs/2.0.0/provisioning_guide/index.html),[PDF](docs/2.0.0/provisioning_guide/Trafodion_Provisioning_Guide.pdf) -Trafodion REST Server Reference Guide | [Web Book](docs/2.0.0/rest_reference/index.html),[API](docs/2.0.0/rest_reference/apidocs/index.html) -Trafodion SQL Reference Manual | [Web Book](docs/2.0.0/sql_reference/index.html),[PDF](docs/2.0.0/sql_reference/Trafodion_SQL_Reference_Manual.pdf) +Document | Formats +------------------------------------------------------|----------------------------------- +Trafodion Client Installation Guide | [Web Book](docs/2.0.0/client_install/index.html),[PDF](docs/2.0.0/client_install/Trafodion_Client_Installation_Guide.pdf) +Trafodion Command Interface Guide | [Web Book](docs/2.0.0/command_interface/index.html),[PDF](docs/2.0.0/command_interface/Trafodion_Command_Interface_Guide.pdf) +Trafodion Control Query Default (CQD) Reference Guide | [Web Book](docs/2.0.0/cqd_reference/index.html),[PDF](docs/2.0.0/cqd_interface/Trafodion_CQD_Reference_Guide.pdf) +Trafodion Database Connectivity Services Guide | [Web Book](docs/2.0.0/dcs_reference/index.html),[API](docs/2.0.0/dcs_reference/apidocs/index.html) +Trafodion Load and Transform Guide | [Web Book](docs/2.0.0/load_transform/index.html),[PDF](docs/2.0.0/load_transform/Trafodion_Load_Transform_Guide.pdf) +Trafodion Messages Guide | [Web Book](docs/2.0.0/messages_guide/index.html),[PDF](docs/2.0.0/messages_guide/Trafodion_Messages_Guide.pdf) +Trafodion odb User Guide | [Web Book](docs/2.0.0/odb/index.html),[PDF](docs/2.0.0/odb/Trafodion_odb_User_Guide.pdf) +Trafodion Provisioning Guide | [Web Book](docs/2.0.0/provisioning_guide/index.html),[PDF](docs/2.0.0/provisioning_guide/Trafodion_Provisioning_Guide.pdf) +Trafodion REST Server Reference Guide | [Web Book](docs/2.0.0/rest_reference/index.html),[API](docs/2.0.0/rest_reference/apidocs/index.html) +Trafodion SQL Reference Manual | [Web Book](docs/2.0.0/sql_reference/index.html),[PDF](docs/2.0.0/sql_reference/Trafodion_SQL_Reference_Manual.pdf) # 1.3.0 -Document | Formats ------------------------------------------------|----------------------------------- -Trafodion Client Installation Guide | [Web Book](docs/1.3.0/client_install/index.html),[PDF](docs/1.3.0/client_install/Trafodion_Client_Installation_Guide.pdf) -Trafodion Command Interface Guide | [Web Book](docs/1.3.0/command_interface/index.html),[PDF](docs/1.3.0/command_interface/Trafodion_Command_Interface_Guide.pdf) -Trafodion Database Connectivity Services Guide | [Web Book](docs/1.3.0/dcs_reference/index.html),[API](docs/1.3.0/dcs_reference/apidocs/index.html) -Trafodion Messages Guide | [Web Book](docs/1.3.0/messages_guide/index.html),[PDF](docs/1.3.0/messages_guide/Trafodion_Messages_Guide.pdf) -Trafodion odb User Guide | [Web Book](docs/1.3.0/odb/index.html),[PDF](docs/1.3.0/odb/Trafodion_odb_User_Guide.pdf) -Trafodion Provisioning Guide | [Web Book](docs/1.3.0/provisioning_guide/index.html),[PDF](docs/1.3.0/provisioning_guide/Trafodion_Provisioning_Guide.pdf) -Trafodion REST Server Reference Guide | [Web Book](docs/1.3.0/rest_reference/index.html),[API](docs/1.3.0/rest_reference/apidocs/index.html) -Trafodion SQL Reference Manual | [Web Book](docs/1.3.0/sql_reference/index.html),[PDF](docs/1.3.0/sql_reference/Trafodion_SQL_Reference_Manual.pdf) +Document | Formats +------------------------------------------------------|----------------------------------- +Trafodion Client Installation Guide | [Web Book](docs/1.3.0/client_install/index.html),[PDF](docs/1.3.0/client_install/Trafodion_Client_Installation_Guide.pdf) +Trafodion Command Interface Guide | [Web Book](docs/1.3.0/command_interface/index.html),[PDF](docs/1.3.0/command_interface/Trafodion_Command_Interface_Guide.pdf) +Trafodion Control Query Default (CQD) Reference Guide | [Web Book](docs/1.3.0/cqd_reference/index.html),[PDF](docs/1.3.0/cqd_interface/Trafodion_CQD_Reference_Guide.pdf) +Trafodion Database Connectivity Services Guide | [Web Book](docs/1.3.0/dcs_reference/index.html),[API](docs/1.3.0/dcs_reference/apidocs/index.html) +Trafodion Load and Transform Guide | [Web Book](docs/1.3.0/load_transform/index.html),[PDF](docs/1.3.0/load_transform/Trafodion_Load_Transform_Guide.pdf) +Trafodion Messages Guide | [Web Book](docs/1.3.0/messages_guide/index.html),[PDF](docs/1.3.0/messages_guide/Trafodion_Messages_Guide.pdf) +Trafodion odb User Guide | [Web Book](docs/1.3.0/odb/index.html),[PDF](docs/1.3.0/odb/Trafodion_odb_User_Guide.pdf) +Trafodion Provisioning Guide | [Web Book](docs/1.3.0/provisioning_guide/index.html),[PDF](docs/1.3.0/provisioning_guide/Trafodion_Provisioning_Guide.pdf) +Trafodion REST Server Reference Guide | [Web Book](docs/1.3.0/rest_reference/index.html),[API](docs/1.3.0/rest_reference/apidocs/index.html) +Trafodion SQL Reference Manual | [Web Book](docs/1.3.0/sql_reference/index.html),[PDF](docs/1.3.0/sql_reference/Trafodion_SQL_Reference_Manual.pdf) http://git-wip-us.apache.org/repos/asf/incubator-trafodion/blob/681cad66/docs/src/site/markdown/logo.md ---------------------------------------------------------------------- diff --git a/docs/src/site/markdown/logo.md b/docs/src/site/markdown/logo.md index f621a28..7dacd5f 100644 --- a/docs/src/site/markdown/logo.md +++ b/docs/src/site/markdown/logo.md @@ -44,7 +44,7 @@ when referencing our project. Download as needed. #### [trafodion-logo.jpg][3] -[][3] +[][3] [3]: images/logos/trafodion-logo.jpg http://git-wip-us.apache.org/repos/asf/incubator-trafodion/blob/681cad66/docs/src/site/resources/images/logos/pb-Trafodion.png ---------------------------------------------------------------------- diff --git a/docs/src/site/resources/images/logos/pb-Trafodion.png b/docs/src/site/resources/images/logos/pb-Trafodion.png deleted file mode 100644 index 67643a4..0000000 Binary files a/docs/src/site/resources/images/logos/pb-Trafodion.png and /dev/null differ http://git-wip-us.apache.org/repos/asf/incubator-trafodion/blob/681cad66/docs/src/site/resources/images/logos/pb-trafodion.png ---------------------------------------------------------------------- diff --git a/docs/src/site/resources/images/logos/pb-trafodion.png b/docs/src/site/resources/images/logos/pb-trafodion.png new file mode 100644 index 0000000..67643a4 Binary files /dev/null and b/docs/src/site/resources/images/logos/pb-trafodion.png differ http://git-wip-us.apache.org/repos/asf/incubator-trafodion/blob/681cad66/docs/src/site/site.xml ---------------------------------------------------------------------- diff --git a/docs/src/site/site.xml b/docs/src/site/site.xml index cf130d6..27b0f16 100644 --- a/docs/src/site/site.xml +++ b/docs/src/site/site.xml @@ -206,6 +206,7 @@ <item href="faq.html" name="FAQ"/> <item href="release-notes.html" name="Release Notes"/> <item href="https://cwiki.apache.org/confluence/display/TRAFODION/Apache+Trafodion+Home"; name="Wiki"/> + <item name="Videos" href="https://www.youtube.com/playlist?list=PLyyCp3HBB8Tv9696OnG0_mmgF5tfHeyOi"/> <item href="management.html" name="Managing Trafodion"/> <!-- item href="new-features.html" name="New Features"/--> <!-- item href="roadmap.html" name="Roadmap"/--> http://git-wip-us.apache.org/repos/asf/incubator-trafodion/blob/681cad66/install/installer/checkHBase ---------------------------------------------------------------------- diff --git a/install/installer/checkHBase b/install/installer/checkHBase new file mode 100755 index 0000000..354ed1a --- /dev/null +++ b/install/installer/checkHBase @@ -0,0 +1,37 @@ +#!/bin/bash +# @@@ START COPYRIGHT @@@ +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +# +# @@@ END COPYRIGHT @@@ + +echo "**INFO: Checking HBase status" + + +echo 'status' | hbase shell -n | grep "0 dead" + +status=$? + +if [[ "$status" -ne 0 ]]; then + echo "***ERROR: HBase not up" + echo "***ERROR: HBase MUST be installed and up for installation" + exit -1 +else + echo "***INFO: HBase up and running" + echo +fi http://git-wip-us.apache.org/repos/asf/incubator-trafodion/blob/681cad66/install/installer/dcs_installer ---------------------------------------------------------------------- diff --git a/install/installer/dcs_installer b/install/installer/dcs_installer index 5d71aab..b07a748 100755 --- a/install/installer/dcs_installer +++ b/install/installer/dcs_installer @@ -110,7 +110,7 @@ while [[ $# -gt 0 ]]; do done checkOpts - +source $TRAF_CONFIG echo "***INFO: Start of DCS install" # Make sure required env vars have been set first @@ -155,12 +155,6 @@ rm dcs-env.temp 2>/dev/null echo "***INFO: modifying $DCS_DIR/conf/dcs-site.xml" -# get zookeeper quorum - -lineNumber=$(grep -n "zookeeper\.quorum" $TRAF_WORKDIR/hbase-site.xml | sed 's/\:.*//') -lineNumber=$((lineNumber+1)) - -ZOOKEEPER_NODES=$(sed "$lineNumber!d" $TRAF_WORKDIR/hbase-site.xml | sed 's/\/value.*//' | sed 's/.*>//' | sed 's/.$//') # add zookeeper quorum property to end of configuration rm dcs-site.temp 2>/dev/null http://git-wip-us.apache.org/repos/asf/incubator-trafodion/blob/681cad66/install/installer/parseHBaseSite.py ---------------------------------------------------------------------- diff --git a/install/installer/parseHBaseSite.py b/install/installer/parseHBaseSite.py new file mode 100755 index 0000000..4fb5905 --- /dev/null +++ b/install/installer/parseHBaseSite.py @@ -0,0 +1,56 @@ +#!/usr/bin/python + +# @@@ START COPYRIGHT @@@ +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +# +# @@@ END COPYRIGHT @@@ + + +import os + +import xml.etree.ElementTree as ET + +hbaseMaster="hbase.master.info.port" +hbaseRegion="hbase.regionserver.info.port" +zooKeeperNodes="hbase.zookeeper.quorum" +pathToHome= os.environ['HOME'] + +hbaseMasterInfoPort="60010" +regionServerInfoPort="60030" +zookeeperNodeNames="" + +tree = ET.parse( pathToHome + '/hbase-site.xml') + +root = tree.getroot() + + +for x in root.findall('property'): + name = str(x.find('name').text) + if name == hbaseMaster: + hbaseMasterInfoPort = x.find('value').text + if name == hbaseRegion: + regionServerInfoPort = x.find('value').text + if name == zooKeeperNodes: + zookeeperNodeNames = x.find('value').text + +f = open( '/etc/trafodion/trafodion_config', 'a') +f.write ( 'export HBASE_MASTER_INFO_PORT="' + hbaseMasterInfoPort + '"\n' ) +f.write ( 'export REGIONSERVER_INFO_PORT="' + regionServerInfoPort + '"\n' ) +f.write ( 'export ZOOKEEPER_NODES="' + zookeeperNodeNames + '"\n' ) +f.close() http://git-wip-us.apache.org/repos/asf/incubator-trafodion/blob/681cad66/install/installer/traf_add_sudoAccess ---------------------------------------------------------------------- diff --git a/install/installer/traf_add_sudoAccess b/install/installer/traf_add_sudoAccess index 9440977..23fd67f 100755 --- a/install/installer/traf_add_sudoAccess +++ b/install/installer/traf_add_sudoAccess @@ -38,7 +38,9 @@ echo "%trafodion ALL = NOPASSWD: IP, ARP" >> $sudoFile function backupRestore { -echo "## Allow $BACKUP_USER id to run commands needed for backup and restore" >> $BACKUP_USER +source $TRAF_CONFIG + +echo "## Allow $BACKUP_USER id to run commands needed for backup and restore" > $BACKUP_USER echo "%$BACKUP_USER ALL =(hbase) NOPASSWD: /usr/bin/hbase" >> $BACKUP_USER } @@ -73,7 +75,7 @@ fi echo "***INFO: Creating $TRAF_USER sudo access file" echo -if [[ "$ENABLE_HA" == "Y" ]]; then +if [[ "$ENABLE_HA" == "true" ]]; then haSudoAdd copyTrafFile fi http://git-wip-us.apache.org/repos/asf/incubator-trafodion/blob/681cad66/install/installer/traf_cloudera_mods98 ---------------------------------------------------------------------- diff --git a/install/installer/traf_cloudera_mods98 b/install/installer/traf_cloudera_mods98 index b8da481..edb09e1 100755 --- a/install/installer/traf_cloudera_mods98 +++ b/install/installer/traf_cloudera_mods98 @@ -39,23 +39,18 @@ export PDCP_HADOOP_NODES="$PDCP $MY_HADOOP_NODES" cd $UNTAR_DIR # determine java version for hbase 0.98 only java 1.7 is allowed -echo "***INFO: Detected JAVA version $JAVA_VERSION" -if [ "$JAVA_VERSION" == "1.6" ]; then - echo "***ERROR: Java 1.6 not supported for Trafodion HBase 0.98" - exit -1 -elif [ "$JAVA_VERSION" == "1.7" ]; then - if [[ $CDH_5_3_HDP_2_2_SUPPORT == "N" ]]; then - if [[ $CDH_5_4_SUPPORT == "Y" ]]; then +if [[ $CDH_5_3_HDP_2_2_SUPPORT == "N" ]]; then + if [[ $CDH_5_4_SUPPORT == "Y" ]] || [[ $CDH_5_5_SUPPORT == "Y" ]]; then + if [[ $CDH_VERSION == "5.4" ]]; then hbase_trx_jar="hbase-trx-cdh5_4-${TRAF_VERSION}.jar" else - hbase_trx_jar="hbase-trx-${TRAF_VERSION}.jar" - fi + hbase_trx_jar="hbase-trx-cdh5_5-${TRAF_VERSION}.jar" + fi else - hbase_trx_jar="hbase-trx-cdh5_3-${TRAF_VERSION}.jar" - fi + hbase_trx_jar="hbase-trx-${TRAF_VERSION}.jar" + fi else - echo "***ERROR: unable to determine Hadoop's java version" - exit -1 + hbase_trx_jar="hbase-trx-cdh5_3-${TRAF_VERSION}.jar" fi traf_util_jar="trafodion-utility-${TRAF_VERSION}.jar" @@ -109,6 +104,25 @@ else ssh -q -n $node sudo chmod 644 $HADOOP_PATH/$traf_util_jar done fi + +#==================================== +#Make sure hbase-trx*jar got copied + +for node in $HADOOP_NODES +do + copiedOver=$(ssh -q -n $node sudo ls $HADOOP_PATH/hbase-trx* | wc -l) + if [[ $copiedOver -ne "1" ]]; then + echo "***ERROR: $hbase_trx_jar was not copied on $node" + echo "***ERROR: Please investigate why this happened" + echo "***ERROR: Trafodion can not start without this. EXITING..." + exit -1 + fi +done + +echo "***INFO: $hbase_trx_jar copied correctly! Huzzah." + + + #===================================== # create new directories for bulkload and lobs if not already there rm $LOCAL_WORKDIR/traf_temp_output 2>/dev/null http://git-wip-us.apache.org/repos/asf/incubator-trafodion/blob/681cad66/install/installer/traf_config_check ---------------------------------------------------------------------- diff --git a/install/installer/traf_config_check b/install/installer/traf_config_check index 95e3f4a..a1ed2e9 100755 --- a/install/installer/traf_config_check +++ b/install/installer/traf_config_check @@ -426,10 +426,15 @@ else fi done fi - export PATH=$PATH:$JAVA_HOME/bin + hadoopPath="$HADOOP_PATH" + hadoopBinPath="$HADOOP_BIN_PATH" + + export PATH="$PATH:$JAVA_HOME/bin" sudo chmod 777 $TRAF_CONFIG sed -i '/PATH\=/d' $TRAF_CONFIG echo "export PATH=\"$PATH\"" >>$TRAF_CONFIG + echo "export HADOOP_BIN_PATH=\"$hadoopBinPath\"" >>$TRAF_CONFIG + echo "export HADOOP_PATH=\"$hadoopPath\"" >>$TRAF_CONFIG sudo chmod 777 $TRAF_CONFIG source $TRAF_CONFIG fi @@ -489,11 +494,15 @@ fi function checkBackupUser { +TRAF_CONFIG="/etc/trafodion/trafodion_config" + +source $TRAF_CONFIG + if [ -z $BACKUP_USER ]; then echo "***ERROR: BACKUP_USER variable not set in config file." echo "***WARNING: BACKUP_USER will be set to trafodion." sudo chmod 777 $TRAF_CONFIG - sudo "export BACKUP_USER=\"trafodion\"" >> $TRAF_CONFIG + sudo echo "export BACKUP_USER=\"trafodion\"" >> $TRAF_CONFIG sudo chmod 777 $TRAF_CONFIG fi @@ -647,11 +656,11 @@ function checkClouderaVersion { if [[ $CDH_5_3_HDP_2_2_SUPPORT == "N" ]]; then #Check that Cloudera 5.2 or 5.3 are not installed. - if [[ "$CDH_5_4_SUPPORT" == "Y" ]]; then - nameOfVersion=$(ssh -q -n $node grep "Version" $HOME/hbaseVersion.txt | sed 's/,.*//' | sed 's/.*\-//' | grep cdh5.4.*) + if [[ "$CDH_5_4_SUPPORT" == "Y" ]] || [[ "$CDH_5_5_SUPPORT" == "Y" ]]; then + nameOfVersion=$(ssh -q -n $node grep "Version" $HOME/hbaseVersion.txt | sed 's/,.*//' | sed 's/.*\-//' | grep cdh5.[4-5].*) #Check that Cloudera 5.[n>4].* is not installed. if [[ -z $nameOfVersion ]]; then - versionInstalled=$(ssh -q -n $node grep "Version" $HOME/hbaseVersion.txt | sed 's/,.*//' | sed 's/.*\-//' | grep cdh[5-6].[0-9].* | wc -l) + versionInstalled=$(ssh -q -n $node grep "Version" $HOME/hbaseVersion.txt | sed 's/,.*//' | sed 's/.*\-//' | grep cdh[6-9].[0-9].* | wc -l) if [[ $versionInstalled -gt "0" ]]; then errorFound=1 echo "HADOOP VERSION" >> $ERROR_LOG @@ -686,18 +695,7 @@ else fi fi -export HADOOP_PATH="/usr/lib/hbase/lib" -#Check for Cloudera parcels of packages. -parcelsInstalled=$(ssh -q -n $node sudo ls /opt/cloudera/parcels/ | wc -l) -if [[ $parcelsInstalled -gt "0" ]]; then - export HADOOP_PATH="/opt/cloudera/parcels/CDH/lib/hbase/lib" -fi - echo "***INFO: nameOfVersion=$nameOfVersion" -echo "***INFO: HADOOP_PATH=$HADOOP_PATH" -sudo chmod 777 $TRAF_CONFIG -sed -i '/HADOOP_PATH\=/d' $TRAF_CONFIG -echo "export HADOOP_PATH=\"$HADOOP_PATH\"" >> $TRAF_CONFIG sudo chmod 777 $TRAF_CONFIG source $TRAF_CONFIG @@ -727,9 +725,8 @@ if [[ $CDH_5_3_HDP_2_2_SUPPORT == "N" ]]; then ssh -q -n $node cat hbaseVersion.txt >> $ERROR_LOG fi fi - export HADOOP_PATH="/usr/lib/hbase/lib" else - nameOfVersion=$(ssh -q -n $node grep "Version" $HOME/hbaseVersion.txt | sed 's/,.*//' | sed 's/-.*//' | grep 2.3.*) + nameOfVersion=$(ssh -q -n $node grep "Version" $HOME/hbaseVersion.txt | sed 's/,.*//' | sed 's/-.*//' | grep 2.[3-4].*) #Check that HDP 2.[n>3].* is not installed. if [[ -z $nameOfVersion ]]; then versionInstalled=$(ssh -q -n $node grep "Version" $HOME/hbaseVersion.txt | sed 's/,.*//' | sed 's/-.*//' | sed 's/Version [0-9]*.[0-9]*.[0-9].//g' | grep [1-3].[0-9].* | wc -l) @@ -741,7 +738,6 @@ if [[ $CDH_5_3_HDP_2_2_SUPPORT == "N" ]]; then ssh -q -n $node cat hbaseVersion.txt >> $ERROR_LOG fi fi - export HADOOP_PATH="/usr/hdp/current/hbase-regionserver/lib" fi else nameOfVersion=$(ssh -q -n $node grep "Version" $HOME/hbaseVersion.txt | sed 's/,.*//' | sed 's/-.*//' | grep 2.2.*) @@ -758,14 +754,9 @@ if [[ -z $nameOfVersion ]]; then ssh -q -n $node cat hbaseVersion.txt >> $ERROR_LOG fi fi -export HADOOP_PATH="/usr/hdp/current/hbase-regionserver/lib" echo "***INFO: nameOfVersion=$nameOfVersion" echo "***INFO: HADOOP_PATH=$HADOOP_PATH" -sudo chmod 777 $TRAF_CONFIG -sed -i '/HADOOP_PATH\=/d' $TRAF_CONFIG -echo "export HADOOP_PATH=\"$HADOOP_PATH\"" >> $TRAF_CONFIG -sudo chmod 777 $TRAF_CONFIG source $TRAF_CONFIG } @@ -877,6 +868,15 @@ if [[ ! -z $install_features_path ]]; then sudo chmod 777 $TRAF_CONFIG echo "export CDH_5_3_HDP_2_2_SUPPORT=\"$CDH_5_3_HDP_2_2_SUPPORT\"" >> $TRAF_CONFIG sudo chmod 777 $TRAF_CONFIG + if [[ -z $CDH_5_5_SUPPORT ]]; then + sudo chmod 777 $TRAF_CONFIG + echo "export CDH_5_5_SUPPORT=\"N\"" >> $TRAF_CONFIG + sudo chmod 777 $TRAF_CONFIG + else + sudo chmod 777 $TRAF_CONFIG + echo "export CDH_5_5_SUPPORT=\"$CDH_5_5_SUPPORT\"" >> $TRAF_CONFIG + sudo chmod 777 $TRAF_CONFIG + fi if [[ -z $CDH_5_4_SUPPORT ]]; then sudo chmod 777 $TRAF_CONFIG echo "export CDH_5_4_SUPPORT=\"N\"" >> $TRAF_CONFIG @@ -898,10 +898,12 @@ if [[ ! -z $install_features_path ]]; then else CDH_5_3_HDP_2_2_SUPPORT="N" CDH_5_4_SUPPORT="N" + CDH_5_5_SUPPORT="N" HDP_2_3_SUPPORT="N" sudo chmod 777 $TRAF_CONFIG echo "export CDH_5_3_HDP_2_2_SUPPORT=\"$CDH_5_3_HDP_2_2_SUPPORT\"" >> $TRAF_CONFIG echo "export CDH_5_4_SUPPORT=\"$CDH_5_4_SUPPORT\"" >> $TRAF_CONFIG + echo "export CDH_5_5_SUPPORT=\"$CDH_5_5_SUPPORT\"" >> $TRAF_CONFIG echo "export HDP_2_3_SUPPORT=\"$HDP_2_3_SUPPORT\"" >> $TRAF_CONFIG sudo chmod 777 $TRAF_CONFIG fi @@ -912,11 +914,7 @@ fi #=========================================== #Main -echo -echo "************************************" -echo " Trafodion Configuration File Check" -echo "************************************" -echo +echo " ***INFO: Trafodion Configuration File Check" setUpErrorLog http://git-wip-us.apache.org/repos/asf/incubator-trafodion/blob/681cad66/install/installer/traf_config_setup ---------------------------------------------------------------------- diff --git a/install/installer/traf_config_setup b/install/installer/traf_config_setup index 4837ca1..63704d5 100755 --- a/install/installer/traf_config_setup +++ b/install/installer/traf_config_setup @@ -395,6 +395,7 @@ echo "export UTT_INSTALL=\"$UTT_INSTALL\"" >> $LOCAL_TRAF_CONFIG #============================================== #Backup restore user +export BACKUP_USER="trafodion" echo -n "Enter Backup/Restore username (can be Trafodion), default is [$BACKUP_USER]: " read answer @@ -791,8 +792,6 @@ sudo mkdir -p $TRAF_CONFIG_DIR sudo chmod 777 $TRAF_CONFIG_DIR sudo cp $LOCAL_TRAF_CONFIG $TRAF_CONFIG sudo chmod 777 $TRAF_CONFIG -echo "***INFO: Configuration file: $TRAF_CONFIG" - #============================================== #Call user defined prompting script http://git-wip-us.apache.org/repos/asf/incubator-trafodion/blob/681cad66/install/installer/traf_getHadoopNodes ---------------------------------------------------------------------- diff --git a/install/installer/traf_getHadoopNodes b/install/installer/traf_getHadoopNodes index 127d0c2..e397d4a 100755 --- a/install/installer/traf_getHadoopNodes +++ b/install/installer/traf_getHadoopNodes @@ -25,22 +25,23 @@ source /etc/trafodion/trafodion_config TRAF_CONFIG=/etc/trafodion/trafodion_config -echo "***INFO: Getting list of all $HADOOP_TYPE nodes" - curlRC=0 -if [[ $HADOOP_TYPE == "cloudera" ]]; then +if [ $HADOOP_TYPE == "cloudera" ]; then curl -k -su $ADMIN:$PASSWORD $URL/api/v10/hosts > tempFile curlRC=$? numberHadoopNodes=$(grep -r "hostname" tempFile | wc -l) grep -r "hostname" tempFile > tempFile2 - if [[ -d /opt/cloudera/parcels/CDH ]]; then - HADOOP_PATH="/opt/cloudera/parcels/CDH/lib/hbase/lib" - HADOOP_BIN_PATH="/opt/cloudera/parcels/CDH/bin" + if [ -d /opt/cloudera/parcels/CDH ]; then + export HADOOP_PATH="/opt/cloudera/parcels/CDH/lib/hbase/lib" + export HADOOP_BIN_PATH="/opt/cloudera/parcels/CDH/bin" else - HADOOP_PATH="/usr/lib/hbase/lib" - HADOOP_BIN_PATH="/usr/bin" + export HADOOP_PATH="/usr/lib/hbase/lib" + export HADOOP_BIN_PATH="/usr/bin" fi + echo "***INFO: HADOOP_PATH=$HADOOP_PATH" + echo "***INFO: HADOOP_BIN_PATH=$HADOOP_BIN_PATH" + fi if [[ $HADOOP_TYPE == "hortonworks" ]]; then @@ -49,11 +50,13 @@ if [[ $HADOOP_TYPE == "hortonworks" ]]; then numberHadoopNodes=$(grep -r "host_name" tempFile | wc -l) grep -r "host_name" tempFile > tempFile2 - if [[ -d /usr/lib/hbase/lib ]]; then + if [ -d /usr/lib/hbase/lib ]; then HADOOP_PATH="/usr/lib/hbase/lib" else HADOOP_PATH="/usr/hdp/current/hbase-regionserver/lib" fi + echo "***INFO: HADOOP_PATH=$HADOOP_PATH" + fi if [ $curlRC != 0 ]; then http://git-wip-us.apache.org/repos/asf/incubator-trafodion/blob/681cad66/install/installer/traf_hortonworks_mods98 ---------------------------------------------------------------------- diff --git a/install/installer/traf_hortonworks_mods98 b/install/installer/traf_hortonworks_mods98 index 7d2be7e..cffb3be 100755 --- a/install/installer/traf_hortonworks_mods98 +++ b/install/installer/traf_hortonworks_mods98 @@ -41,29 +41,20 @@ export PDCP_HADOOP_NODES="$PDCP $MY_HADOOP_NODES" cd $UNTAR_DIR PORT=`echo $URL | sed 's/.*://'` -AMBARI_HOST=$(echo $URL | sed 's@:.*@@') +AMBARI_HOST=$(echo $URL | sed 's@.*://@@' | sed 's@:.*@@') echo "export AMBARI_HOST=\"$AMBARI_HOST\"" >> $TRAF_CONFIG sudo chmod 777 $TRAF_CONFIG source $TRAF_CONFIG #determine java version and choose corresponding jar files -echo "***INFO: Detected JAVA version $JAVA_VERSION" -if [ "$JAVA_VERSION" == "1.6" ]; then - echo "***ERROR: JAVA 1.6 not supported for Trafodion HBase 0.98" - exit -1 -elif [ "$JAVA_VERSION" == "1.7" ] || [ "$JAVA_VERSION" == "1.8" ]; then - if [[ $CDH_5_3_HDP_2_2_SUPPORT == "N" ]]; then - if [[ $HDP_2_3_SUPPORT == "Y" ]]; then - hbase_trx_jar="hbase-trx-hdp2_3-${TRAF_VERSION}.jar" - else - hbase_trx_jar="hbase-trx-hdp2_1-${TRAF_VERSION}.jar" - fi - else - hbase_trx_jar="hbase-trx-hdp2_2-${TRAF_VERSION}.jar" - fi +if [[ $CDH_5_3_HDP_2_2_SUPPORT == "N" ]]; then + if [[ $HDP_2_3_SUPPORT == "Y" ]]; then + hbase_trx_jar="hbase-trx-hdp2_3-${TRAF_VERSION}.jar" + else + hbase_trx_jar="hbase-trx-hdp2_1-${TRAF_VERSION}.jar" + fi else - echo "***ERROR: unable to determine Hadoop's java version" - exit -1 + hbase_trx_jar="hbase-trx-hdp2_2-${TRAF_VERSION}.jar" fi traf_util_jar="trafodion-utility-${TRAF_VERSION}.jar" @@ -120,6 +111,23 @@ else done fi +#======================================= +#Check that HBase-trx copied to all nodes + +for node in $HADOOP_NODES +do + copiedOver=$(ssh -q -n $node sudo ls $HADOOP_PATH/hbase-trx* | wc -l) + if [[ $copiedOver -ne "1" ]]; then + echo "***ERROR: $hbase_trx_jar was not copied on $node" + echo "***ERROR: Please investigate why this happened" + echo "***ERROR: Trafodion can not start without this. EXITING..." + exit -1 + fi +done + +echo "***INFO: $hbase_trx_jar copied correctly! Huzzah." + + #Copy hbase-site.xml file ssh -q -n $AMBARI_HOST sudo cp /etc/hbase/conf/hbase-site.xml $HOME http://git-wip-us.apache.org/repos/asf/incubator-trafodion/blob/681cad66/install/installer/traf_package_setup ---------------------------------------------------------------------- diff --git a/install/installer/traf_package_setup b/install/installer/traf_package_setup index dbf9a4a..d904090 100755 --- a/install/installer/traf_package_setup +++ b/install/installer/traf_package_setup @@ -151,7 +151,7 @@ if [[ $SUSE_LINUX == "false" ]]; then # it doesn't hurt if it was already enabled and we try to enable it again $TRAF_PDSH sudo yum-config-manager --enable RedHat-6-x86_64-Fedora-EPEL 2>> $YUM_LOG >> $YUM_LOG - package_list="sqlite expect perl-DBD-SQLite* protobuf xerces-c perl-Params-Validate perl-Time-HiRes gzip" + package_list="apr apr-util sqlite expect perl-DBD-SQLite* protobuf xerces-c perl-Params-Validate perl-Time-HiRes gzip" if [[ $internetAccess == "true" ]]; then for package in $package_list do http://git-wip-us.apache.org/repos/asf/incubator-trafodion/blob/681cad66/install/installer/traf_sqgen ---------------------------------------------------------------------- diff --git a/install/installer/traf_sqgen b/install/installer/traf_sqgen index d3e2da2..271e719 100755 --- a/install/installer/traf_sqgen +++ b/install/installer/traf_sqgen @@ -73,7 +73,7 @@ if [[ "$LDAP_SECURITY" == "Y" ]]; then #Check traf_authentication_config for errors echo "***INFO: Checking LDAP Configuration file for errors." - $SQ_ROOT/export/bin64/ldapconfigcheck -file $SQ_ROOT/sql/scripts/traf_authentication_config + $SQ_ROOT/export/bin${SQ_MBTYPE}/ldapconfigcheck -file $SQ_ROOT/sql/scripts/traf_authentication_config if [ "$?" != "0" ]; then echo "***ERROR: traf_authentication_config not configured correctly." echo "***ERROR: Install will continue WITHOUT simple security turned on."
