http://git-wip-us.apache.org/repos/asf/incubator-hawq-docs/blob/a42c05a9/markdown/datamgmt/dml.html.md.erb ---------------------------------------------------------------------- diff --git a/markdown/datamgmt/dml.html.md.erb b/markdown/datamgmt/dml.html.md.erb index 681883a..3c5d7fa 100644 --- a/markdown/datamgmt/dml.html.md.erb +++ b/markdown/datamgmt/dml.html.md.erb @@ -2,6 +2,25 @@ title: Managing Data with HAWQ --- +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. +--> + This chapter provides information about manipulating data and concurrent access in HAWQ. - **[Basic Data Operations](../datamgmt/BasicDataOperations.html)**
http://git-wip-us.apache.org/repos/asf/incubator-hawq-docs/blob/a42c05a9/markdown/datamgmt/load/client-loadtools.html.md.erb ---------------------------------------------------------------------- diff --git a/markdown/datamgmt/load/client-loadtools.html.md.erb b/markdown/datamgmt/load/client-loadtools.html.md.erb index fe291d0..6587a9f 100644 --- a/markdown/datamgmt/load/client-loadtools.html.md.erb +++ b/markdown/datamgmt/load/client-loadtools.html.md.erb @@ -1,6 +1,25 @@ --- title: Client-Based HAWQ Load Tools --- + +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. +--> HAWQ supports data loading from Red Hat Enterprise Linux 5, 6, and 7 and Windows XP client systems. HAWQ Load Tools include both a loader program and a parallel file distribution program. This topic presents the instructions to install the HAWQ Load Tools on your client machine. It also includes the information necessary to configure HAWQ databases to accept remote client connections. http://git-wip-us.apache.org/repos/asf/incubator-hawq-docs/blob/a42c05a9/markdown/datamgmt/load/creating-external-tables-examples.html.md.erb ---------------------------------------------------------------------- diff --git a/markdown/datamgmt/load/creating-external-tables-examples.html.md.erb b/markdown/datamgmt/load/creating-external-tables-examples.html.md.erb index 7d38971..4001968 100644 --- a/markdown/datamgmt/load/creating-external-tables-examples.html.md.erb +++ b/markdown/datamgmt/load/creating-external-tables-examples.html.md.erb @@ -2,6 +2,25 @@ title: Creating External Tables - Examples --- +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. +--> + The following examples show how to define external data with different protocols. Each `CREATE EXTERNAL TABLE` command can contain only one protocol. **Note:** When using IPv6, always enclose the numeric IP addresses in square brackets. http://git-wip-us.apache.org/repos/asf/incubator-hawq-docs/blob/a42c05a9/markdown/datamgmt/load/g-about-gpfdist-setup-and-performance.html.md.erb ---------------------------------------------------------------------- diff --git a/markdown/datamgmt/load/g-about-gpfdist-setup-and-performance.html.md.erb b/markdown/datamgmt/load/g-about-gpfdist-setup-and-performance.html.md.erb index 28a0bfe..9868850 100644 --- a/markdown/datamgmt/load/g-about-gpfdist-setup-and-performance.html.md.erb +++ b/markdown/datamgmt/load/g-about-gpfdist-setup-and-performance.html.md.erb @@ -2,6 +2,25 @@ title: About gpfdist Setup and Performance --- +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. +--> + Consider the following scenarios for optimizing your ETL network performance. - Allow network traffic to use all ETL host Network Interface Cards (NICs) simultaneously. Run one instance of `gpfdist` on the ETL host, then declare the host name of each NIC in the `LOCATION` clause of your external table definition (see [Creating External Tables - Examples](creating-external-tables-examples.html#topic44)). http://git-wip-us.apache.org/repos/asf/incubator-hawq-docs/blob/a42c05a9/markdown/datamgmt/load/g-character-encoding.html.md.erb ---------------------------------------------------------------------- diff --git a/markdown/datamgmt/load/g-character-encoding.html.md.erb b/markdown/datamgmt/load/g-character-encoding.html.md.erb index 9f3756d..ea52334 100644 --- a/markdown/datamgmt/load/g-character-encoding.html.md.erb +++ b/markdown/datamgmt/load/g-character-encoding.html.md.erb @@ -2,6 +2,25 @@ title: Character Encoding --- +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. +--> + Character encoding systems consist of a code that pairs each character from a character set with something else, such as a sequence of numbers or octets, to facilitate data stransmission and storage. HAWQ supports a variety of character sets, including single-byte character sets such as the ISO 8859 series and multiple-byte character sets such as EUC (Extended UNIX Code), UTF-8, and Mule internal code. Clients can use all supported character sets transparently, but a few are not supported for use within the server as a server-side encoding. Data files must be in a character encoding recognized by HAWQ. Data files that contain invalid or unsupported encoding sequences encounter errors when loading into HAWQ. http://git-wip-us.apache.org/repos/asf/incubator-hawq-docs/blob/a42c05a9/markdown/datamgmt/load/g-command-based-web-external-tables.html.md.erb ---------------------------------------------------------------------- diff --git a/markdown/datamgmt/load/g-command-based-web-external-tables.html.md.erb b/markdown/datamgmt/load/g-command-based-web-external-tables.html.md.erb index 7830cc3..4a6a5cb 100644 --- a/markdown/datamgmt/load/g-command-based-web-external-tables.html.md.erb +++ b/markdown/datamgmt/load/g-command-based-web-external-tables.html.md.erb @@ -2,6 +2,25 @@ title: Command-based Web External Tables --- +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. +--> + The output of a shell command or script defines command-based web table data. Specify the command in the `EXECUTE` clause of `CREATE EXTERNAL WEB TABLE`. The data is current as of the time the command runs. The `EXECUTE` clause runs the shell command or script on the specified master or virtual segments. The virtual segments run the command in parallel. Scripts must be executable by the gpadmin user and reside in the same location on the master or the hosts of virtual segments. The command that you specify in the external table definition executes from the database and cannot access environment variables from `.bashrc` or `.profile`. Set environment variables in the `EXECUTE` clause. The following external web table, for example, runs a command on the HAWQ master host: http://git-wip-us.apache.org/repos/asf/incubator-hawq-docs/blob/a42c05a9/markdown/datamgmt/load/g-configuration-file-format.html.md.erb ---------------------------------------------------------------------- diff --git a/markdown/datamgmt/load/g-configuration-file-format.html.md.erb b/markdown/datamgmt/load/g-configuration-file-format.html.md.erb index 73f51a9..f375e5c 100644 --- a/markdown/datamgmt/load/g-configuration-file-format.html.md.erb +++ b/markdown/datamgmt/load/g-configuration-file-format.html.md.erb @@ -2,6 +2,25 @@ title: Configuration File Format --- +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. +--> + The `gpfdist` configuration file uses the YAML 1.1 document format and implements a schema for defining the transformation parameters. The configuration file must be a valid YAML document. The `gpfdist` program processes the document in order and uses indentation (spaces) to determine the document hierarchy and relationships of the sections to one another. The use of white space is significant. Do not use white space for formatting and do not use tabs. http://git-wip-us.apache.org/repos/asf/incubator-hawq-docs/blob/a42c05a9/markdown/datamgmt/load/g-controlling-segment-parallelism.html.md.erb ---------------------------------------------------------------------- diff --git a/markdown/datamgmt/load/g-controlling-segment-parallelism.html.md.erb b/markdown/datamgmt/load/g-controlling-segment-parallelism.html.md.erb index 4e0096c..ff89ffb 100644 --- a/markdown/datamgmt/load/g-controlling-segment-parallelism.html.md.erb +++ b/markdown/datamgmt/load/g-controlling-segment-parallelism.html.md.erb @@ -2,6 +2,25 @@ title: Controlling Segment Parallelism --- +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. +--> + The `gp_external_max_segs` server configuration parameter controls the number of virtual segments that can simultaneously access a single `gpfdist` instance. The default is 64. You can set the number of segments such that some segments process external data files and some perform other database processing. Set this parameter in the `hawq-site.xml` file of your master instance. The number of segments in the `gpfdist` location list specify the minimum number of virtual segments required to serve data to a `gpfdist` external table. http://git-wip-us.apache.org/repos/asf/incubator-hawq-docs/blob/a42c05a9/markdown/datamgmt/load/g-create-an-error-table-and-declare-a-reject-limit.html.md.erb ---------------------------------------------------------------------- diff --git a/markdown/datamgmt/load/g-create-an-error-table-and-declare-a-reject-limit.html.md.erb b/markdown/datamgmt/load/g-create-an-error-table-and-declare-a-reject-limit.html.md.erb index ade14ea..6f001ea 100644 --- a/markdown/datamgmt/load/g-create-an-error-table-and-declare-a-reject-limit.html.md.erb +++ b/markdown/datamgmt/load/g-create-an-error-table-and-declare-a-reject-limit.html.md.erb @@ -2,6 +2,25 @@ title: Capture Row Formatting Errors and Declare a Reject Limit --- +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. +--> + The following SQL fragment captures formatting errors internally in HAWQ and declares a reject limit of 10 rows. ``` sql http://git-wip-us.apache.org/repos/asf/incubator-hawq-docs/blob/a42c05a9/markdown/datamgmt/load/g-creating-and-using-web-external-tables.html.md.erb ---------------------------------------------------------------------- diff --git a/markdown/datamgmt/load/g-creating-and-using-web-external-tables.html.md.erb b/markdown/datamgmt/load/g-creating-and-using-web-external-tables.html.md.erb index 4ef6cab..bdbe808 100644 --- a/markdown/datamgmt/load/g-creating-and-using-web-external-tables.html.md.erb +++ b/markdown/datamgmt/load/g-creating-and-using-web-external-tables.html.md.erb @@ -2,6 +2,25 @@ title: Creating and Using Web External Tables --- +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. +--> + `CREATE EXTERNAL WEB TABLE` creates a web table definition. Web external tables allow HAWQ to treat dynamic data sources like regular database tables. Because web table data can change as a query runs, the data is not rescannable. You can define command-based or URL-based web external tables. The definition forms are distinct: you cannot mix command-based and URL-based definitions. http://git-wip-us.apache.org/repos/asf/incubator-hawq-docs/blob/a42c05a9/markdown/datamgmt/load/g-define-an-external-table-with-single-row-error-isolation.html.md.erb ---------------------------------------------------------------------- diff --git a/markdown/datamgmt/load/g-define-an-external-table-with-single-row-error-isolation.html.md.erb b/markdown/datamgmt/load/g-define-an-external-table-with-single-row-error-isolation.html.md.erb index e0c3c17..54cde2a 100644 --- a/markdown/datamgmt/load/g-define-an-external-table-with-single-row-error-isolation.html.md.erb +++ b/markdown/datamgmt/load/g-define-an-external-table-with-single-row-error-isolation.html.md.erb @@ -2,6 +2,25 @@ title: Define an External Table with Single Row Error Isolation --- +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. +--> + The following example logs errors internally in HAWQ and sets an error threshold of 10 errors. ``` sql http://git-wip-us.apache.org/repos/asf/incubator-hawq-docs/blob/a42c05a9/markdown/datamgmt/load/g-defining-a-command-based-writable-external-web-table.html.md.erb ---------------------------------------------------------------------- diff --git a/markdown/datamgmt/load/g-defining-a-command-based-writable-external-web-table.html.md.erb b/markdown/datamgmt/load/g-defining-a-command-based-writable-external-web-table.html.md.erb index 8a24474..067fbfb 100644 --- a/markdown/datamgmt/load/g-defining-a-command-based-writable-external-web-table.html.md.erb +++ b/markdown/datamgmt/load/g-defining-a-command-based-writable-external-web-table.html.md.erb @@ -2,6 +2,25 @@ title: Defining a Command-Based Writable External Web Table --- +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. +--> + You can define writable external web tables to send output rows to an application or script. The application must accept an input stream, reside in the same location on all of the HAWQ segment hosts, and be executable by the `gpadmin` user. All segments in the HAWQ system run the application or script, whether or not a segment has output rows to process. Use `CREATE WRITABLE EXTERNAL WEB TABLE` to define the external table and specify the application or script to run on the segment hosts. Commands execute from within the database and cannot access environment variables (such as `$PATH`). Set environment variables in the `EXECUTE` clause of your writable external table definition. For example: http://git-wip-us.apache.org/repos/asf/incubator-hawq-docs/blob/a42c05a9/markdown/datamgmt/load/g-defining-a-file-based-writable-external-table.html.md.erb ---------------------------------------------------------------------- diff --git a/markdown/datamgmt/load/g-defining-a-file-based-writable-external-table.html.md.erb b/markdown/datamgmt/load/g-defining-a-file-based-writable-external-table.html.md.erb index fa1ddfa..5a7468b 100644 --- a/markdown/datamgmt/load/g-defining-a-file-based-writable-external-table.html.md.erb +++ b/markdown/datamgmt/load/g-defining-a-file-based-writable-external-table.html.md.erb @@ -2,6 +2,25 @@ title: Defining a File-Based Writable External Table --- +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. +--> + Writable external tables that output data to files use the HAWQ parallel file server program, `gpfdist`, or HAWQ Extensions Framework (PXF). Use the `CREATE WRITABLE EXTERNAL TABLE` command to define the external table and specify the location and format of the output files. http://git-wip-us.apache.org/repos/asf/incubator-hawq-docs/blob/a42c05a9/markdown/datamgmt/load/g-determine-the-transformation-schema.html.md.erb ---------------------------------------------------------------------- diff --git a/markdown/datamgmt/load/g-determine-the-transformation-schema.html.md.erb b/markdown/datamgmt/load/g-determine-the-transformation-schema.html.md.erb index 1a4eb9b..4376c67 100644 --- a/markdown/datamgmt/load/g-determine-the-transformation-schema.html.md.erb +++ b/markdown/datamgmt/load/g-determine-the-transformation-schema.html.md.erb @@ -2,6 +2,25 @@ title: Determine the Transformation Schema --- +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. +--> + To prepare for the transformation project: 1. <span class="ph">Determine the goal of the project, such as indexing data, analyzing data, combining data, and so on.</span> http://git-wip-us.apache.org/repos/asf/incubator-hawq-docs/blob/a42c05a9/markdown/datamgmt/load/g-disabling-execute-for-web-or-writable-external-tables.html.md.erb ---------------------------------------------------------------------- diff --git a/markdown/datamgmt/load/g-disabling-execute-for-web-or-writable-external-tables.html.md.erb b/markdown/datamgmt/load/g-disabling-execute-for-web-or-writable-external-tables.html.md.erb index f0332b5..a462807 100644 --- a/markdown/datamgmt/load/g-disabling-execute-for-web-or-writable-external-tables.html.md.erb +++ b/markdown/datamgmt/load/g-disabling-execute-for-web-or-writable-external-tables.html.md.erb @@ -2,6 +2,25 @@ title: Disabling EXECUTE for Web or Writable External Tables --- +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. +--> + There is a security risk associated with allowing external tables to execute OS commands or scripts. To disable the use of `EXECUTE` in web and writable external table definitions, set the `gp_external_enable_exec server` configuration parameter to off in your master `hawq-site.xml` file: ``` pre http://git-wip-us.apache.org/repos/asf/incubator-hawq-docs/blob/a42c05a9/markdown/datamgmt/load/g-escaping-in-csv-formatted-files.html.md.erb ---------------------------------------------------------------------- diff --git a/markdown/datamgmt/load/g-escaping-in-csv-formatted-files.html.md.erb b/markdown/datamgmt/load/g-escaping-in-csv-formatted-files.html.md.erb index d07b463..1ac32a2 100644 --- a/markdown/datamgmt/load/g-escaping-in-csv-formatted-files.html.md.erb +++ b/markdown/datamgmt/load/g-escaping-in-csv-formatted-files.html.md.erb @@ -2,6 +2,25 @@ title: Escaping in CSV Formatted Files --- +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. +--> + By default, the escape character is a `"` (double quote) for CSV-formatted files. If you want to use a different escape character, use the `ESCAPE` clause of `COPY`, `CREATE EXTERNAL TABLE` or the `hawq load` control file to declare a different escape character. In cases where your selected escape character is present in your data, you can use it to escape itself. For example, suppose you have a table with three columns and you want to load the following three fields: http://git-wip-us.apache.org/repos/asf/incubator-hawq-docs/blob/a42c05a9/markdown/datamgmt/load/g-escaping-in-text-formatted-files.html.md.erb ---------------------------------------------------------------------- diff --git a/markdown/datamgmt/load/g-escaping-in-text-formatted-files.html.md.erb b/markdown/datamgmt/load/g-escaping-in-text-formatted-files.html.md.erb index e24a2b7..110638e 100644 --- a/markdown/datamgmt/load/g-escaping-in-text-formatted-files.html.md.erb +++ b/markdown/datamgmt/load/g-escaping-in-text-formatted-files.html.md.erb @@ -2,6 +2,25 @@ title: Escaping in Text Formatted Files --- +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. +--> + By default, the escape character is a \\ (backslash) for text-formatted files. You can declare a different escape character in the `ESCAPE` clause of `COPY`, `CREATE EXTERNAL TABLE`, or the `hawq load` control file. If your escape character appears in your data, use it to escape itself. For example, suppose you have a table with three columns and you want to load the following three fields: http://git-wip-us.apache.org/repos/asf/incubator-hawq-docs/blob/a42c05a9/markdown/datamgmt/load/g-escaping.html.md.erb ---------------------------------------------------------------------- diff --git a/markdown/datamgmt/load/g-escaping.html.md.erb b/markdown/datamgmt/load/g-escaping.html.md.erb index 0a1e62a..d3a5aa7 100644 --- a/markdown/datamgmt/load/g-escaping.html.md.erb +++ b/markdown/datamgmt/load/g-escaping.html.md.erb @@ -2,6 +2,25 @@ title: Escaping --- +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. +--> + There are two reserved characters that have special meaning to HAWQ: - The designated delimiter character separates columns or fields in the data file. http://git-wip-us.apache.org/repos/asf/incubator-hawq-docs/blob/a42c05a9/markdown/datamgmt/load/g-example-1-dblp-database-publications-in-demo-directory.html.md.erb ---------------------------------------------------------------------- diff --git a/markdown/datamgmt/load/g-example-1-dblp-database-publications-in-demo-directory.html.md.erb b/markdown/datamgmt/load/g-example-1-dblp-database-publications-in-demo-directory.html.md.erb index 4f61396..0c27cd7 100644 --- a/markdown/datamgmt/load/g-example-1-dblp-database-publications-in-demo-directory.html.md.erb +++ b/markdown/datamgmt/load/g-example-1-dblp-database-publications-in-demo-directory.html.md.erb @@ -2,6 +2,25 @@ title: Command-based Web External Tables --- +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. +--> + The output of a shell command or script defines command-based web table data. Specify the command in the `EXECUTE` clause of `CREATE EXTERNAL WEB TABLE`. The data is current as of the time the command runs. The `EXECUTE` clause runs the shell command or script on the specified master, and/or segment host or hosts. The command or script must reside on the hosts corresponding to the host(s) defined in the `EXECUTE` clause. By default, the command is run on segment hosts when active segments have output rows to process. For example, if each segment host runs four primary segment instances that have output rows to process, the command runs four times per segment host. You can optionally limit the number of segment instances that execute the web table command. All segments included in the web table definition in the `ON` clause run the command in parallel. http://git-wip-us.apache.org/repos/asf/incubator-hawq-docs/blob/a42c05a9/markdown/datamgmt/load/g-example-hawq-file-server-gpfdist.html.md.erb ---------------------------------------------------------------------- diff --git a/markdown/datamgmt/load/g-example-hawq-file-server-gpfdist.html.md.erb b/markdown/datamgmt/load/g-example-hawq-file-server-gpfdist.html.md.erb index a0bf669..af57c44 100644 --- a/markdown/datamgmt/load/g-example-hawq-file-server-gpfdist.html.md.erb +++ b/markdown/datamgmt/load/g-example-hawq-file-server-gpfdist.html.md.erb @@ -2,6 +2,25 @@ title: Example - HAWQ file server (gpfdist) --- +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. +--> + ``` sql =# CREATE WRITABLE EXTERNAL TABLE unload_expenses ( LIKE expenses ) http://git-wip-us.apache.org/repos/asf/incubator-hawq-docs/blob/a42c05a9/markdown/datamgmt/load/g-example-irs-mef-xml-files-in-demo-directory.html.md.erb ---------------------------------------------------------------------- diff --git a/markdown/datamgmt/load/g-example-irs-mef-xml-files-in-demo-directory.html.md.erb b/markdown/datamgmt/load/g-example-irs-mef-xml-files-in-demo-directory.html.md.erb index 6f5b9e3..df225b2 100644 --- a/markdown/datamgmt/load/g-example-irs-mef-xml-files-in-demo-directory.html.md.erb +++ b/markdown/datamgmt/load/g-example-irs-mef-xml-files-in-demo-directory.html.md.erb @@ -2,6 +2,25 @@ title: Example using IRS MeF XML Files (In demo Directory) --- +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. +--> + This example demonstrates loading a sample IRS Modernized eFile tax return using a Joost STX transformation. The data is in the form of a complex XML file. The U.S. Internal Revenue Service (IRS) made a significant commitment to XML and specifies its use in its Modernized e-File (MeF) system. In MeF, each tax return is an XML document with a deep hierarchical structure that closely reflects the particular form of the underlying tax code. http://git-wip-us.apache.org/repos/asf/incubator-hawq-docs/blob/a42c05a9/markdown/datamgmt/load/g-example-witsml-files-in-demo-directory.html.md.erb ---------------------------------------------------------------------- diff --git a/markdown/datamgmt/load/g-example-witsml-files-in-demo-directory.html.md.erb b/markdown/datamgmt/load/g-example-witsml-files-in-demo-directory.html.md.erb index 0484523..e5de2b5 100644 --- a/markdown/datamgmt/load/g-example-witsml-files-in-demo-directory.html.md.erb +++ b/markdown/datamgmt/load/g-example-witsml-files-in-demo-directory.html.md.erb @@ -2,6 +2,25 @@ title: Example using WITSML⢠Files (In demo Directory) --- +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. +--> + This example demonstrates loading sample data describing an oil rig using a Joost STX transformation. The data is in the form of a complex XML file downloaded from energistics.org. The Wellsite Information Transfer Standard Markup Language (WITSMLâ¢) is an oil industry initiative to provide open, non-proprietary, standard interfaces for technology and software to share information among oil companies, service companies, drilling contractors, application vendors, and regulatory agencies. For more information about WITSMLâ¢, see [http://www.witsml.org](http://www.witsml.org). http://git-wip-us.apache.org/repos/asf/incubator-hawq-docs/blob/a42c05a9/markdown/datamgmt/load/g-examples-read-fixed-width-data.html.md.erb ---------------------------------------------------------------------- diff --git a/markdown/datamgmt/load/g-examples-read-fixed-width-data.html.md.erb b/markdown/datamgmt/load/g-examples-read-fixed-width-data.html.md.erb index 174529a..4303bad 100644 --- a/markdown/datamgmt/load/g-examples-read-fixed-width-data.html.md.erb +++ b/markdown/datamgmt/load/g-examples-read-fixed-width-data.html.md.erb @@ -2,6 +2,25 @@ title: Examples - Read Fixed-Width Data --- +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. +--> + The following examples show how to read fixed-width data. ## Example 1 â Loading a table with PRESERVED\_BLANKS on http://git-wip-us.apache.org/repos/asf/incubator-hawq-docs/blob/a42c05a9/markdown/datamgmt/load/g-external-tables.html.md.erb ---------------------------------------------------------------------- diff --git a/markdown/datamgmt/load/g-external-tables.html.md.erb b/markdown/datamgmt/load/g-external-tables.html.md.erb index 4142a07..e833756 100644 --- a/markdown/datamgmt/load/g-external-tables.html.md.erb +++ b/markdown/datamgmt/load/g-external-tables.html.md.erb @@ -2,6 +2,25 @@ title: Accessing File-Based External Tables --- +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. +--> + External tables enable accessing external files as if they are regular database tables. They are often used to move data into and out of a HAWQ database. To create an external table definition, you specify the format of your input files and the location of your external data sources. For information input file formats, see [Formatting Data Files](g-formatting-data-files.html#topic95). http://git-wip-us.apache.org/repos/asf/incubator-hawq-docs/blob/a42c05a9/markdown/datamgmt/load/g-formatting-columns.html.md.erb ---------------------------------------------------------------------- diff --git a/markdown/datamgmt/load/g-formatting-columns.html.md.erb b/markdown/datamgmt/load/g-formatting-columns.html.md.erb index b828212..6987fe2 100644 --- a/markdown/datamgmt/load/g-formatting-columns.html.md.erb +++ b/markdown/datamgmt/load/g-formatting-columns.html.md.erb @@ -2,6 +2,25 @@ title: Formatting Columns --- +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. +--> + The default column or field delimiter is the horizontal `TAB` character (`0x09`) for text files and the comma character (`0x2C`) for CSV files. You can declare a single character delimiter using the `DELIMITER` clause of `COPY`, `CREATE EXTERNAL TABLE` or the `hawq load` configuration table when you define your data format. The delimiter character must appear between any two data value fields. Do not place a delimiter at the beginning or end of a row. For example, if the pipe character ( | ) is your delimiter: ``` pre http://git-wip-us.apache.org/repos/asf/incubator-hawq-docs/blob/a42c05a9/markdown/datamgmt/load/g-formatting-data-files.html.md.erb ---------------------------------------------------------------------- diff --git a/markdown/datamgmt/load/g-formatting-data-files.html.md.erb b/markdown/datamgmt/load/g-formatting-data-files.html.md.erb index 6c929ad..28ffdfe 100644 --- a/markdown/datamgmt/load/g-formatting-data-files.html.md.erb +++ b/markdown/datamgmt/load/g-formatting-data-files.html.md.erb @@ -2,6 +2,25 @@ title: Formatting Data Files --- +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. +--> + When you use the HAWQ tools for loading and unloading data, you must specify how your data is formatted. `COPY`, `CREATE EXTERNAL TABLE, `and `hawq load` have clauses that allow you to specify how your data is formatted. Data can be delimited text (`TEXT`) or comma separated values (`CSV`) format. External data must be formatted correctly to be read by HAWQ. This topic explains the format of data files expected by HAWQ. - **[Formatting Rows](../../datamgmt/load/g-formatting-rows.html)** http://git-wip-us.apache.org/repos/asf/incubator-hawq-docs/blob/a42c05a9/markdown/datamgmt/load/g-formatting-rows.html.md.erb ---------------------------------------------------------------------- diff --git a/markdown/datamgmt/load/g-formatting-rows.html.md.erb b/markdown/datamgmt/load/g-formatting-rows.html.md.erb index ea9b416..0fac00c 100644 --- a/markdown/datamgmt/load/g-formatting-rows.html.md.erb +++ b/markdown/datamgmt/load/g-formatting-rows.html.md.erb @@ -2,6 +2,25 @@ title: Formatting Rows --- +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. +--> + HAWQ expects rows of data to be separated by the `LF` character (Line feed, `0x0A`), `CR` (Carriage return, `0x0D`), or `CR` followed by `LF` (`CR+LF`, `0x0D 0x0A`). `LF` is the standard newline representation on UNIX or UNIX-like operating systems. Operating systems such as Windows or Mac OS X use `CR` or `CR+LF`. All of these representations of a newline are supported by HAWQ as a row delimiter. For more information, see [Importing and Exporting Fixed Width Data](g-importing-and-exporting-fixed-width-data.html#topic37). http://git-wip-us.apache.org/repos/asf/incubator-hawq-docs/blob/a42c05a9/markdown/datamgmt/load/g-gpfdist-protocol.html.md.erb ---------------------------------------------------------------------- diff --git a/markdown/datamgmt/load/g-gpfdist-protocol.html.md.erb b/markdown/datamgmt/load/g-gpfdist-protocol.html.md.erb index ee98609..0e8d8e2 100644 --- a/markdown/datamgmt/load/g-gpfdist-protocol.html.md.erb +++ b/markdown/datamgmt/load/g-gpfdist-protocol.html.md.erb @@ -2,6 +2,25 @@ title: gpfdist Protocol --- +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. +--> + The `gpfdist://` protocol is used in a URI to reference a running `gpfdist` instance. The `gpfdist` utility serves external data files from a directory on a file host to all HAWQ segments in parallel. `gpfdist` is located in the `$GPHOME/bin` directory on your HAWQ master host and on each segment host. http://git-wip-us.apache.org/repos/asf/incubator-hawq-docs/blob/a42c05a9/markdown/datamgmt/load/g-gpfdists-protocol.html.md.erb ---------------------------------------------------------------------- diff --git a/markdown/datamgmt/load/g-gpfdists-protocol.html.md.erb b/markdown/datamgmt/load/g-gpfdists-protocol.html.md.erb index 2f5641d..17f1e00 100644 --- a/markdown/datamgmt/load/g-gpfdists-protocol.html.md.erb +++ b/markdown/datamgmt/load/g-gpfdists-protocol.html.md.erb @@ -2,6 +2,25 @@ title: gpfdists Protocol --- +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. +--> + The `gpfdists://` protocol is a secure version of the `gpfdist:// protocol`. To use it, you run the `gpfdist` utility with the `--ssl` option. When specified in a URI, the `gpfdists://` protocol enables encrypted communication and secure identification of the file server and the HAWQ to protect against attacks such as eavesdropping and man-in-the-middle attacks. `gpfdists` implements SSL security in a client/server scheme with the following attributes and limitations: http://git-wip-us.apache.org/repos/asf/incubator-hawq-docs/blob/a42c05a9/markdown/datamgmt/load/g-handling-errors-ext-table-data.html.md.erb ---------------------------------------------------------------------- diff --git a/markdown/datamgmt/load/g-handling-errors-ext-table-data.html.md.erb b/markdown/datamgmt/load/g-handling-errors-ext-table-data.html.md.erb index 2b8dc78..32e786f 100644 --- a/markdown/datamgmt/load/g-handling-errors-ext-table-data.html.md.erb +++ b/markdown/datamgmt/load/g-handling-errors-ext-table-data.html.md.erb @@ -2,6 +2,25 @@ title: Handling Errors in External Table Data --- +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. +--> + By default, if external table data contains an error, the command fails and no data loads into the target database table. Define the external table with single row error handling to enable loading correctly formatted rows and to isolate data errors in external table data. See [Handling Load Errors](g-handling-load-errors.html#topic55). The `gpfdist` file server uses the `HTTP` protocol. External table queries that use `LIMIT` end the connection after retrieving the rows, causing an HTTP socket error. If you use `LIMIT` in queries of external tables that use the `gpfdist://` or `http:// protocols`, ignore these errors â data is returned to the database as expected. http://git-wip-us.apache.org/repos/asf/incubator-hawq-docs/blob/a42c05a9/markdown/datamgmt/load/g-handling-load-errors.html.md.erb ---------------------------------------------------------------------- diff --git a/markdown/datamgmt/load/g-handling-load-errors.html.md.erb b/markdown/datamgmt/load/g-handling-load-errors.html.md.erb index 6faf7a5..3068889 100644 --- a/markdown/datamgmt/load/g-handling-load-errors.html.md.erb +++ b/markdown/datamgmt/load/g-handling-load-errors.html.md.erb @@ -2,6 +2,25 @@ title: Handling Load Errors --- +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. +--> + Readable external tables are most commonly used to select data to load into regular database tables. You use the `CREATE TABLE AS SELECT` or `INSERT INTO `commands to query the external table data. By default, if the data contains an error, the entire command fails and the data is not loaded into the target database table. The `SEGMENT REJECT LIMIT` clause allows you to isolate format errors in external table data and to continue loading correctly formatted rows. Use `SEGMENT REJECT LIMIT `to set an error threshold, specifying the reject limit `count` as number of `ROWS` (the default) or as a `PERCENT` of total rows (1-100). http://git-wip-us.apache.org/repos/asf/incubator-hawq-docs/blob/a42c05a9/markdown/datamgmt/load/g-identifying-invalid-csv-files-in-error-table-data.html.md.erb ---------------------------------------------------------------------- diff --git a/markdown/datamgmt/load/g-identifying-invalid-csv-files-in-error-table-data.html.md.erb b/markdown/datamgmt/load/g-identifying-invalid-csv-files-in-error-table-data.html.md.erb index 534d530..f13451f 100644 --- a/markdown/datamgmt/load/g-identifying-invalid-csv-files-in-error-table-data.html.md.erb +++ b/markdown/datamgmt/load/g-identifying-invalid-csv-files-in-error-table-data.html.md.erb @@ -2,6 +2,25 @@ title: Identifying Invalid CSV Files in Error Table Data --- +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. +--> + If a CSV file contains invalid formatting, the *rawdata* field in the error table can contain several combined rows. For example, if a closing quote for a specific field is missing, all the following newlines are treated as embedded newlines. When this happens, HAWQ stops parsing a row when it reaches 64K, puts that 64K of data into the error table as a single row, resets the quote flag, and continues. If this happens three times during load processing, the load file is considered invalid and the entire load fails with the message "`rejected ` `N` ` or more rows`". See [Escaping in CSV Formatted Files](g-escaping-in-csv-formatted-files.html#topic101) for more information on the correct use of quotes in CSV files. http://git-wip-us.apache.org/repos/asf/incubator-hawq-docs/blob/a42c05a9/markdown/datamgmt/load/g-importing-and-exporting-fixed-width-data.html.md.erb ---------------------------------------------------------------------- diff --git a/markdown/datamgmt/load/g-importing-and-exporting-fixed-width-data.html.md.erb b/markdown/datamgmt/load/g-importing-and-exporting-fixed-width-data.html.md.erb index f49cae0..8ac2797 100644 --- a/markdown/datamgmt/load/g-importing-and-exporting-fixed-width-data.html.md.erb +++ b/markdown/datamgmt/load/g-importing-and-exporting-fixed-width-data.html.md.erb @@ -2,6 +2,25 @@ title: Importing and Exporting Fixed Width Data --- +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. +--> + Specify custom formats for fixed-width data with the HAWQ functions `fixedwith_in` and `fixedwidth_out`. These functions already exist in the file `$GPHOME/share/postgresql/cdb_external_extensions.sql`. The following example declares a custom format, then calls the `fixedwidth_in` function to format the data. ``` sql http://git-wip-us.apache.org/repos/asf/incubator-hawq-docs/blob/a42c05a9/markdown/datamgmt/load/g-installing-gpfdist.html.md.erb ---------------------------------------------------------------------- diff --git a/markdown/datamgmt/load/g-installing-gpfdist.html.md.erb b/markdown/datamgmt/load/g-installing-gpfdist.html.md.erb index 85549df..704b392 100644 --- a/markdown/datamgmt/load/g-installing-gpfdist.html.md.erb +++ b/markdown/datamgmt/load/g-installing-gpfdist.html.md.erb @@ -2,6 +2,25 @@ title: Installing gpfdist --- +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. +--> + You may choose to run `gpfdist` from a machine other than the HAWQ master, such as on a machine devoted to ETL processing. To install `gpfdist` on your ETL server, refer to [Client-Based HAWQ Load Tools](client-loadtools.html) for information related to Linux and Windows load tools installation and configuration. http://git-wip-us.apache.org/repos/asf/incubator-hawq-docs/blob/a42c05a9/markdown/datamgmt/load/g-load-the-data.html.md.erb ---------------------------------------------------------------------- diff --git a/markdown/datamgmt/load/g-load-the-data.html.md.erb b/markdown/datamgmt/load/g-load-the-data.html.md.erb index 4c88c9f..51c7184 100644 --- a/markdown/datamgmt/load/g-load-the-data.html.md.erb +++ b/markdown/datamgmt/load/g-load-the-data.html.md.erb @@ -2,6 +2,25 @@ title: Load the Data --- +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. +--> + Create the tables with SQL statements based on the appropriate schema. There are no special requirements for the HAWQ tables that hold loaded data. In the prices example, the following command creates the appropriate table. http://git-wip-us.apache.org/repos/asf/incubator-hawq-docs/blob/a42c05a9/markdown/datamgmt/load/g-loading-and-unloading-data.html.md.erb ---------------------------------------------------------------------- diff --git a/markdown/datamgmt/load/g-loading-and-unloading-data.html.md.erb b/markdown/datamgmt/load/g-loading-and-unloading-data.html.md.erb index 8ea43d5..e687137 100644 --- a/markdown/datamgmt/load/g-loading-and-unloading-data.html.md.erb +++ b/markdown/datamgmt/load/g-loading-and-unloading-data.html.md.erb @@ -2,6 +2,25 @@ title: Loading and Unloading Data --- +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. +--> + The topics in this section describe methods for loading and writing data into and out of HAWQ, and how to format data files. It also covers registering HDFS files and folders directly into HAWQ internal tables. HAWQ supports high-performance parallel data loading and unloading, and for smaller amounts of data, single file, non-parallel data import and export. http://git-wip-us.apache.org/repos/asf/incubator-hawq-docs/blob/a42c05a9/markdown/datamgmt/load/g-loading-and-writing-non-hdfs-custom-data.html.md.erb ---------------------------------------------------------------------- diff --git a/markdown/datamgmt/load/g-loading-and-writing-non-hdfs-custom-data.html.md.erb b/markdown/datamgmt/load/g-loading-and-writing-non-hdfs-custom-data.html.md.erb index e826963..d21ad32 100644 --- a/markdown/datamgmt/load/g-loading-and-writing-non-hdfs-custom-data.html.md.erb +++ b/markdown/datamgmt/load/g-loading-and-writing-non-hdfs-custom-data.html.md.erb @@ -2,6 +2,25 @@ title: Loading and Writing Non-HDFS Custom Data --- +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. +--> + HAWQ supports `TEXT` and `CSV` formats for importing and exporting data. You can load and write the data in other formats by defining and using a custom format or custom protocol. - **[Using a Custom Format](../../datamgmt/load/g-using-a-custom-format.html)** http://git-wip-us.apache.org/repos/asf/incubator-hawq-docs/blob/a42c05a9/markdown/datamgmt/load/g-loading-data-using-an-external-table.html.md.erb ---------------------------------------------------------------------- diff --git a/markdown/datamgmt/load/g-loading-data-using-an-external-table.html.md.erb b/markdown/datamgmt/load/g-loading-data-using-an-external-table.html.md.erb index 32a741a..bd80880 100644 --- a/markdown/datamgmt/load/g-loading-data-using-an-external-table.html.md.erb +++ b/markdown/datamgmt/load/g-loading-data-using-an-external-table.html.md.erb @@ -2,6 +2,25 @@ title: Loading Data Using an External Table --- +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. +--> + Use SQL commands such as `INSERT` and `SELECT` to query a readable external table, the same way that you query a regular database table. For example, to load travel expense data from an external table, `ext_expenses`, into a database table,` expenses_travel`: ``` sql http://git-wip-us.apache.org/repos/asf/incubator-hawq-docs/blob/a42c05a9/markdown/datamgmt/load/g-loading-data-with-copy.html.md.erb ---------------------------------------------------------------------- diff --git a/markdown/datamgmt/load/g-loading-data-with-copy.html.md.erb b/markdown/datamgmt/load/g-loading-data-with-copy.html.md.erb index 72e5ac6..e47017b 100644 --- a/markdown/datamgmt/load/g-loading-data-with-copy.html.md.erb +++ b/markdown/datamgmt/load/g-loading-data-with-copy.html.md.erb @@ -2,6 +2,25 @@ title: Loading Data with COPY --- +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. +--> + `COPY FROM` copies data from a file or standard input into a table and appends the data to the table contents. `COPY` is non-parallel: data is loaded in a single process using the HAWQ master instance. Using `COPY` is only recommended for very small data files. The `COPY` source file must be accessible to the master host. Specify the `COPY` source file name relative to the master host location. http://git-wip-us.apache.org/repos/asf/incubator-hawq-docs/blob/a42c05a9/markdown/datamgmt/load/g-loading-data-with-hawqload.html.md.erb ---------------------------------------------------------------------- diff --git a/markdown/datamgmt/load/g-loading-data-with-hawqload.html.md.erb b/markdown/datamgmt/load/g-loading-data-with-hawqload.html.md.erb index 68e4459..9fa4f1c 100644 --- a/markdown/datamgmt/load/g-loading-data-with-hawqload.html.md.erb +++ b/markdown/datamgmt/load/g-loading-data-with-hawqload.html.md.erb @@ -2,6 +2,25 @@ title: Loading Data with hawq load --- +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. +--> + The HAWQ `hawq load` utility loads data using readable external tables and the HAWQ parallel file server ( `gpfdist` or `gpfdists`). It handles parallel file-based external table setup and allows users to configure their data format, external table definition, and `gpfdist` or `gpfdists` setup in a single configuration file. ## <a id="topic62__du168147"></a>To use hawq load http://git-wip-us.apache.org/repos/asf/incubator-hawq-docs/blob/a42c05a9/markdown/datamgmt/load/g-moving-data-between-tables.html.md.erb ---------------------------------------------------------------------- diff --git a/markdown/datamgmt/load/g-moving-data-between-tables.html.md.erb b/markdown/datamgmt/load/g-moving-data-between-tables.html.md.erb index 2603ae4..bba28cc 100644 --- a/markdown/datamgmt/load/g-moving-data-between-tables.html.md.erb +++ b/markdown/datamgmt/load/g-moving-data-between-tables.html.md.erb @@ -2,6 +2,25 @@ title: Moving Data between Tables --- +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. +--> + You can use `CREATE TABLE AS` or `INSERT...SELECT` to load external and web external table data into another (non-external) database table, and the data will be loaded in parallel according to the external or web external table definition. If an external table file or web external table data source has an error, one of the following will happen, depending on the isolation mode used: http://git-wip-us.apache.org/repos/asf/incubator-hawq-docs/blob/a42c05a9/markdown/datamgmt/load/g-optimizing-data-load-and-query-performance.html.md.erb ---------------------------------------------------------------------- diff --git a/markdown/datamgmt/load/g-optimizing-data-load-and-query-performance.html.md.erb b/markdown/datamgmt/load/g-optimizing-data-load-and-query-performance.html.md.erb index ff1c230..96083a5 100644 --- a/markdown/datamgmt/load/g-optimizing-data-load-and-query-performance.html.md.erb +++ b/markdown/datamgmt/load/g-optimizing-data-load-and-query-performance.html.md.erb @@ -2,6 +2,25 @@ title: Optimizing Data Load and Query Performance --- +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. +--> + Use the following tip to help optimize your data load and subsequent query performance. - Run `ANALYZE` after loading data. If you significantly altered the data in a table, run `ANALYZE` or `VACUUM ANALYZE` (system catalog tables only) to update table statistics for the query optimizer. Current statistics ensure that the optimizer makes the best decisions during query planning and avoids poor performance due to inaccurate or nonexistent statistics. http://git-wip-us.apache.org/repos/asf/incubator-hawq-docs/blob/a42c05a9/markdown/datamgmt/load/g-register_files.html.md.erb ---------------------------------------------------------------------- diff --git a/markdown/datamgmt/load/g-register_files.html.md.erb b/markdown/datamgmt/load/g-register_files.html.md.erb index 25c24ca..c3a70be 100644 --- a/markdown/datamgmt/load/g-register_files.html.md.erb +++ b/markdown/datamgmt/load/g-register_files.html.md.erb @@ -2,6 +2,25 @@ title: Registering Files into HAWQ Internal Tables --- +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. +--> + The `hawq register` utility loads and registers HDFS data files or folders into HAWQ internal tables. Files can be read directly, rather than having to be copied or loaded, resulting in higher performance and more efficient transaction processing. Data from the file or directory specified by \<hdfsfilepath\> is loaded into the appropriate HAWQ table directory in HDFS and the utility updates the corresponding HAWQ metadata for the files. Either AO or Parquet-formatted tables in HDFS can be loaded into a corresponding table in HAWQ. http://git-wip-us.apache.org/repos/asf/incubator-hawq-docs/blob/a42c05a9/markdown/datamgmt/load/g-representing-null-values.html.md.erb ---------------------------------------------------------------------- diff --git a/markdown/datamgmt/load/g-representing-null-values.html.md.erb b/markdown/datamgmt/load/g-representing-null-values.html.md.erb index 4d4ffdd..a6ca89d 100644 --- a/markdown/datamgmt/load/g-representing-null-values.html.md.erb +++ b/markdown/datamgmt/load/g-representing-null-values.html.md.erb @@ -2,6 +2,25 @@ title: Representing NULL Values --- +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. +--> + `NULL` represents an unknown piece of data in a column or field. Within your data files you can designate a string to represent null values. The default string is `\N` (backslash-N) in `TEXT` mode, or an empty value with no quotations in `CSV` mode. You can also declare a different string using the `NULL` clause of `COPY`, `CREATE EXTERNAL TABLE `or the `hawq load` control file when defining your data format. For example, you can use an empty string if you do not want to distinguish nulls from empty strings. When using the HAWQ loading tools, any data item that matches the designated null string is considered a null value. http://git-wip-us.apache.org/repos/asf/incubator-hawq-docs/blob/a42c05a9/markdown/datamgmt/load/g-running-copy-in-single-row-error-isolation-mode.html.md.erb ---------------------------------------------------------------------- diff --git a/markdown/datamgmt/load/g-running-copy-in-single-row-error-isolation-mode.html.md.erb b/markdown/datamgmt/load/g-running-copy-in-single-row-error-isolation-mode.html.md.erb index ba0603c..d08231f 100644 --- a/markdown/datamgmt/load/g-running-copy-in-single-row-error-isolation-mode.html.md.erb +++ b/markdown/datamgmt/load/g-running-copy-in-single-row-error-isolation-mode.html.md.erb @@ -2,6 +2,25 @@ title: Running COPY in Single Row Error Isolation Mode --- +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. +--> + By default, `COPY` stops an operation at the first error: if the data contains an error, the operation fails and no data loads. If you run `COPY FROM` in *single row error isolation mode*, HAWQ skips rows that contain format errors and loads properly formatted rows. Single row error isolation mode applies only to rows in the input file that contain format errors. If the data contains a contraint error such as violation of a `NOT NULL` or `CHECK` constraint, the operation fails and no data loads. Specifying `SEGMENT REJECT LIMIT` runs the `COPY` operation in single row error isolation mode. Specify the acceptable number of error rows on each segment, after which the entire `COPY FROM` operation fails and no rows load. The error row count is for each HAWQ segment, not for the entire load operation. http://git-wip-us.apache.org/repos/asf/incubator-hawq-docs/blob/a42c05a9/markdown/datamgmt/load/g-starting-and-stopping-gpfdist.html.md.erb ---------------------------------------------------------------------- diff --git a/markdown/datamgmt/load/g-starting-and-stopping-gpfdist.html.md.erb b/markdown/datamgmt/load/g-starting-and-stopping-gpfdist.html.md.erb index 7e2cca9..7d068fa 100644 --- a/markdown/datamgmt/load/g-starting-and-stopping-gpfdist.html.md.erb +++ b/markdown/datamgmt/load/g-starting-and-stopping-gpfdist.html.md.erb @@ -2,6 +2,25 @@ title: Starting and Stopping gpfdist --- +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. +--> + You can start `gpfdist` in your current directory location or in any directory that you specify. The default port is `8080`. From your current directory, type: http://git-wip-us.apache.org/repos/asf/incubator-hawq-docs/blob/a42c05a9/markdown/datamgmt/load/g-transfer-and-store-the-data.html.md.erb ---------------------------------------------------------------------- diff --git a/markdown/datamgmt/load/g-transfer-and-store-the-data.html.md.erb b/markdown/datamgmt/load/g-transfer-and-store-the-data.html.md.erb index 8a6d7ab..3155839 100644 --- a/markdown/datamgmt/load/g-transfer-and-store-the-data.html.md.erb +++ b/markdown/datamgmt/load/g-transfer-and-store-the-data.html.md.erb @@ -2,6 +2,25 @@ title: Transfer and Store the Data --- +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. +--> + Use one of the following approaches to transform the data with `gpfdist`. - `GPLOAD` supports only input transformations, but is easier to implement in many cases. http://git-wip-us.apache.org/repos/asf/incubator-hawq-docs/blob/a42c05a9/markdown/datamgmt/load/g-transforming-with-gpload.html.md.erb ---------------------------------------------------------------------- diff --git a/markdown/datamgmt/load/g-transforming-with-gpload.html.md.erb b/markdown/datamgmt/load/g-transforming-with-gpload.html.md.erb index 438fedb..4c2e146 100644 --- a/markdown/datamgmt/load/g-transforming-with-gpload.html.md.erb +++ b/markdown/datamgmt/load/g-transforming-with-gpload.html.md.erb @@ -2,6 +2,25 @@ title: Transforming with GPLOAD --- +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. +--> + To transform data using the `GPLOAD ` control file, you must specify both the file name for the `TRANSFORM_CONFIG` file and the name of the `TRANSFORM` operation in the `INPUT` section of the `GPLOAD` control file. - `TRANSFORM_CONFIG `specifies the name of the `gpfdist` configuration file. http://git-wip-us.apache.org/repos/asf/incubator-hawq-docs/blob/a42c05a9/markdown/datamgmt/load/g-transforming-with-insert-into-select-from.html.md.erb ---------------------------------------------------------------------- diff --git a/markdown/datamgmt/load/g-transforming-with-insert-into-select-from.html.md.erb b/markdown/datamgmt/load/g-transforming-with-insert-into-select-from.html.md.erb index d91cc93..342c907 100644 --- a/markdown/datamgmt/load/g-transforming-with-insert-into-select-from.html.md.erb +++ b/markdown/datamgmt/load/g-transforming-with-insert-into-select-from.html.md.erb @@ -2,6 +2,25 @@ title: Transforming with INSERT INTO SELECT FROM --- +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. +--> + Specify the transformation in the `CREATE EXTERNAL TABLE` definition's `LOCATION` clause. For example, the transform is shown in bold in the following command. (Run `gpfdist` first, using the command `gpfdist -c config.yaml`). ``` sql http://git-wip-us.apache.org/repos/asf/incubator-hawq-docs/blob/a42c05a9/markdown/datamgmt/load/g-transforming-xml-data.html.md.erb ---------------------------------------------------------------------- diff --git a/markdown/datamgmt/load/g-transforming-xml-data.html.md.erb b/markdown/datamgmt/load/g-transforming-xml-data.html.md.erb index f9520bb..ca183ae 100644 --- a/markdown/datamgmt/load/g-transforming-xml-data.html.md.erb +++ b/markdown/datamgmt/load/g-transforming-xml-data.html.md.erb @@ -2,6 +2,25 @@ title: Transforming XML Data --- +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. +--> + The HAWQ data loader *gpfdist* provides transformation features to load XML data into a table and to write data from the HAWQ to XML files. The following diagram shows *gpfdist* performing an XML transform. <a id="topic75__du185408"></a>
