IMPALA-3406: [DOCS] Empty the original Cloudera FAQ Almost all of the original Impala FAQ material was Cloudera-themed or commercially oriented. Lots of answers about the QuickStart VM, Cloudera discussion forums, CDH-based recommendations, etc. IMO it is not worth trying to adapt each FAQ entry to be generic. Better to start over from the ground up.
Phase 1 of making an Apache-friendly FAQ is to strip the original page "down to the studs" so new FAQ entries can be added with more of a developer theme, based on questions people have in the community. Change-Id: Ib81242f0981c04fff99e2c27e06a8d9f4da34c9f Reviewed-on: http://gerrit.cloudera.org:8080/6003 Reviewed-by: Jim Apple <[email protected]> Tested-by: Impala Public Jenkins Project: http://git-wip-us.apache.org/repos/asf/incubator-impala/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-impala/commit/49b407e9 Tree: http://git-wip-us.apache.org/repos/asf/incubator-impala/tree/49b407e9 Diff: http://git-wip-us.apache.org/repos/asf/incubator-impala/diff/49b407e9 Branch: refs/heads/master Commit: 49b407e9a7324741cd72db0e3cbcdda38fe77eba Parents: d07580c Author: John Russell <[email protected]> Authored: Tue Feb 14 12:14:36 2017 -0800 Committer: Impala Public Jenkins <[email protected]> Committed: Wed Feb 22 23:41:21 2017 +0000 ---------------------------------------------------------------------- docs/topics/impala_faq.xml | 1858 +-------------------------------------- 1 file changed, 6 insertions(+), 1852 deletions(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/49b407e9/docs/topics/impala_faq.xml ---------------------------------------------------------------------- diff --git a/docs/topics/impala_faq.xml b/docs/topics/impala_faq.xml index 6db1b03..bb7b098 100644 --- a/docs/topics/impala_faq.xml +++ b/docs/topics/impala_faq.xml @@ -36,1860 +36,14 @@ under the License. <conbody> <p> - Here are the categories of frequently asked questions for Impala, the interactive SQL engine included with CDH. + This section lists frequently asked questions for Apache Impala (incubating), + the interactive SQL engine for Hadoop. </p> - <p outputclass="toc inpage"/> - </conbody> - - <concept id="faq_eval"> - - <title>Trying Impala</title> - - <conbody> - - <p outputclass="toc inpage" audience="PDF"> - FAQs in this category: - </p> - - <section id="faq_tryout" audience="hidden"> - - <title>How do I try Impala out?</title> - - <sectiondiv id="faq_try_impala"> - - <p> - To look at the core features and functionality on Impala, the easiest way to try out Impala is to - download the Cloudera QuickStart VM and start the Impala service through Cloudera Manager, then use - <cmdname>impala-shell</cmdname> in a terminal window or the Impala Query UI in the Hue web interface. - </p> - - <p> - To do performance testing and try out the management features for Impala on a cluster, you need to move - beyond the QuickStart VM with its virtualized single-node environment. Ideally, download the Cloudera - Manager software to set up the cluster, then install the Impala software through Cloudera Manager. - </p> - - </sectiondiv> - </section> - - <section id="faq_demo_vm" audience="hidden"> - - <title>Does Cloudera offer a VM for demonstrating Impala?</title> - - <sectiondiv id="faq_demo_vm_sect"> - - <p> - Cloudera offers a demonstration VM called the QuickStart VM, available in VMWare, VirtualBox, and KVM - formats. For more information, see -<!-- Was: <xref href="cloudera-content/cloudera-docs/DemoVMs/Cloudera-QuickStart-VM/cloudera_impala.html" scope="external" format="html">Cloudera Impala Demo VM</xref> --> -<!-- Then was: <xref href="cloudera-content/cloudera-docs/DemoVMs/Cloudera-QuickStart-VM/cloudera_quickstart_vm.html" scope="external" format="html">the Cloudera QuickStart VM</xref>. --> -<!-- Finally(?) was: <xref href="https://ccp.cloudera.com/display/SUPPORT/Cloudera+QuickStart+VM"; scope="external" format="html">the Cloudera QuickStart VM</xref>. --> - <xref href="http://www.cloudera.com/content/support/en/downloads/quickstart_vms.html"; scope="external" format="html">the - Cloudera QuickStart VM</xref>. After booting the QuickStart VM, many services are turned off by - default; in the Cloudera Manager UI that appears automatically, turn on Impala and any other components - that you want to try out. - </p> - - </sectiondiv> - </section> - - <section id="faq_docs"> - - <title>Where can I find Impala documentation?</title> - - <sectiondiv id="faq_doc"> - - <p> - Starting with Impala 1.3.0, Impala documentation is integrated with the CDH 5 documentation, in - addition to the standalone Impala documentation for use with CDH 4. For CDH 5, the core Impala - developer and administrator information remains in the associated -<!-- Original URL: http://www.cloudera.com/content/cloudera-content/cloudera-docs/CDH5/latest/Impala/impala.html --> - <xref href="http://www.cloudera.com/documentation/enterprise/latest/topics/impala.html"; scope="external" format="html">Impala - documentation</xref> portion. Information about Impala release notes, installation, configuration, - startup, and security is embedded in the corresponding CDH 5 guides. - </p> - -<!-- Same list is in impala.xml and Impala FAQs. Conref in both places. --> - - <ul> - <li> - <xref href="impala_new_features.xml#new_features">New features</xref> - </li> - - <li> - <xref href="impala_known_issues.xml#known_issues">Known and fixed issues</xref> - </li> - - <li> - <xref href="impala_incompatible_changes.xml#incompatible_changes">Incompatible changes</xref> - </li> - - <li> - <xref href="impala_install.xml#install">Installing Impala</xref> - </li> - - <li> - <xref href="impala_upgrading.xml#upgrading">Upgrading Impala</xref> - </li> - - <li> - <xref href="impala_config.xml#config">Configuring Impala</xref> - </li> - - <li> - <xref href="impala_processes.xml#processes">Starting Impala</xref> - </li> - - <li> - <xref href="impala_security.xml#security">Security for Impala</xref> - </li> - - <li> -<!-- Original URL: http://www.cloudera.com/content/cloudera-content/cloudera-docs/CDH5/latest/CDH-Version-and-Packaging-Information/CDH-Version-and-Packaging-Information.html --> - <xref href="http://www.cloudera.com/documentation/enterprise/latest/topics/rg_vd.html"; scope="external" format="html">CDH - Version and Packaging Information</xref> - </li> - </ul> - - <p> - Information about the latest CDH 4-compatible Impala release remains at the -<!-- Original URL: updated this from a /v1/ URL. --> - <xref href="http://www.cloudera.com/content/cloudera/en/documentation/impala/latest.html"; scope="external" format="html">Impala - for CDH 4 Documentation</xref> page. - </p> - - </sectiondiv> - </section> - - <section id="faq_more_info"> - - <title>Where can I get more information about Impala?</title> - - <sectiondiv id="faq_more_info_sect"> - - <!-- JDR: Not changing these instances of 'Cloudera Impala' because those are the real titles of those books or blog posts. --> - <p> - More product information is available here: - </p> - - <ul> - <li> - O'Reilly introductory e-book: - <xref href="http://radar.oreilly.com/2013/10/cloudera-impala-bringing-the-sql-and-hadoop-worlds-together.html"; scope="external" format="html">Cloudera - Impala: Bringing the SQL and Hadoop Worlds Together</xref> - </li> - - <li> - O'Reilly getting started guide for developers: - <xref href="http://shop.oreilly.com/product/0636920033936.do"; scope="external" format="html">Getting - Started with Impala: Interactive SQL for Apache Hadoop</xref> - </li> - - <li> - Blog: - <xref href="http://blog.cloudera.com/blog/2012/10/cloudera-impala-real-time-queries-in-apache-hadoop-for-real"; scope="external" format="html">Cloudera - Impala: Real-Time Queries in Apache Hadoop, For Real</xref> - </li> - - <li> - Webinar: - <xref href="http://www.cloudera.com/content/cloudera/en/resources/library/recordedwebinar/impala-real-time-queries-in-hadoop-webinar-slides.html"; scope="external" format="html">Introduction - to Impala</xref> - </li> - - <li> - Product website page: - <xref href="http://www.cloudera.com/content/cloudera/en/products-and-services/cdh/impala.html"; scope="external" format="html">Cloudera - Enterprise RTQ</xref> - </li> - </ul> - - <p> - To see the latest release announcements for Impala, see the - <xref href="http://community.cloudera.com/t5/Release-Announcements/bd-p/RelAnnounce"; scope="external" format="html">Cloudera - Announcements</xref> forum. - </p> - - </sectiondiv> - </section> - - <section id="faq_community"> - - <title>How can I ask questions and provide feedback about Impala?</title> - - <sectiondiv id="faq_qanda"> - - <ul> - <li> - Join the - <xref href="http://community.cloudera.com/t5/Interactive-Short-cycle-SQL/bd-p/Impala"; scope="external" format="html">Impala - discussion forum</xref> and the - <xref href="https://groups.google.com/a/cloudera.org/forum/?fromgroups#!forum/impala-user"; scope="external" format="html">Impala - mailing list</xref> to ask questions and provide feedback. - </li> - - <li> - Use the <xref href="https://issues.cloudera.org/browse/IMPALA"; scope="external" format="html">Impala - Jira project</xref> to log bug reports and requests for features. - </li> - </ul> - - </sectiondiv> - </section> - - <section id="faq_tpcds"> - - <title>Where can I get sample data to try?</title> - - <p> - You can get scripts that produce data files and set up an environment for TPC-DS style benchmark tests - from <xref href="https://github.com/cloudera/impala-tpcds-kit"; scope="external" format="html">this Github - repository</xref>. In addition to being useful for experimenting with performance, the tables are suited - to experimenting with many aspects of SQL on Impala: they contain a good mixture of data types, data - distributions, partitioning, and relational data suitable for join queries. - </p> - </section> - </conbody> - </concept> - - <concept id="faq_prereq"> - - <title>Impala System Requirements</title> - <prolog> - <metadata> - <!-- Normally I don't categorize subtopics under FAQs. Making an exception to beef up the EC2 category, - and to judge whether it makes sense to relax that rule a bit. --> - <data name="Category" value="Amazon"/> - <data name="Category" value="EC2"/> - </metadata> - </prolog> - - <conbody> - - <p outputclass="toc inpage" audience="PDF"> - FAQs in this category: - </p> - - <section id="faq_prereqs"> - - <title>What are the software and hardware requirements for running Impala?</title> - - <sectiondiv id="faq_system_reqs"> - - <p> - For information on Impala requirements, see <xref href="impala_prereqs.xml#prereqs"/>. Note that there - is often a minimum required level of Cloudera Manager for any given Impala version. - </p> - - </sectiondiv> - </section> - - <section id="faq_memory_prereq"> - - <title>How much memory is required?</title> - - <sectiondiv id="faq_mem_req"> - - <!-- To do: - Prefer to have more examples / citations for larger memory sizes. What are the most - memory-intensive operations that require or benefit from large mem size? - Actually that info should go into impala_scalability.xml and be xref'ed from here. - --> - - <p> - Although Impala is not an in-memory database, when dealing with large tables and large result sets, you - should expect to dedicate a substantial portion of physical memory for the <cmdname>impalad</cmdname> - daemon. Recommended physical memory for an Impala node is 128 GB or higher. If practical, devote - approximately 80% of physical memory to Impala. -<!-- The machines we typically run on have approximately 32-48 GB. --> - </p> - - <p> - The amount of memory required for an Impala operation depends on several factors: - </p> - - <ul> - <li> - <p> - The file format of the table. Different file formats represent the same data in more or fewer data - files. The compression and encoding for each file format might require a different amount of - temporary memory to decompress the data for analysis. - </p> - </li> - - <li> - <p> - Whether the operation is a <codeph>SELECT</codeph> or an <codeph>INSERT</codeph>. For example, - Parquet tables require relatively little memory to query, because Impala reads and decompresses - data in 8 MB chunks. Inserting into a Parquet table is a more memory-intensive operation because - the data for each data file (potentially <ph rev="parquet_block_size">hundreds of megabytes, - depending on the value of the <codeph>PARQUET_FILE_SIZE</codeph> query option</ph>) is stored in - memory until encoded, compressed, and written to disk. -<!-- In 2.0, default might be smaller than maximum. --> - </p> - </li> - - <li> - <p> - Whether the table is partitioned or not, and whether a query against a partitioned table can take - advantage of partition pruning. - </p> - </li> - - <li> - <p> - Whether the final result set is sorted by the <codeph>ORDER BY</codeph> clause. -<!-- -<ph rev="obwl">Remember, Impala requires that all <codeph>ORDER BY</codeph> queries include a -<codeph>LIMIT</codeph> clause too, either in the query syntax or implicitly -through the <codeph>DEFAULT_ORDER_BY_LIMIT</codeph> query option.</ph> ---> - Each Impala node scans and filters a portion of the total data, and applies the - <codeph>LIMIT</codeph> to its own portion of the result set. <ph rev="1.4.0">In Impala 1.4.0 and - higher, if the sort operation requires more memory than is available on any particular host, Impala - uses a temporary disk work area to perform the sort.</ph> The intermediate result sets -<!-- (each with a maximum size of <codeph>LIMIT</codeph> rows) --> - are all sent back to the coordinator node, which does the final sorting and then applies the - <codeph>LIMIT</codeph> clause to the final result set. - </p> - <p> - For example, if you execute the query: -<codeblock>select * from giant_table order by some_column limit 1000;</codeblock> - and your cluster has 50 nodes, then each of those 50 nodes will transmit a maximum of 1000 rows - back to the coordinator node. The coordinator node needs enough memory to sort - (<codeph>LIMIT</codeph> * <varname>cluster_size</varname>) rows, although in the end the final - result set is at most <codeph>LIMIT</codeph> rows, 1000 in this case. - </p> - <p> - Likewise, if you execute the query: -<codeblock>select * from giant_table where test_val > 100 order by some_column;</codeblock> - then each node filters out a set of rows matching the <codeph>WHERE</codeph> conditions, sorts the - results (with no size limit), and sends the sorted intermediate rows back to the coordinator node. - The coordinator node might need substantial memory to sort the final result set, and so might use a - temporary disk work area for that final phase of the query. - </p> - </li> - - <li> - <p> - Whether the query contains any join clauses, <codeph>GROUP BY</codeph> clauses, analytic functions, - or <codeph>DISTINCT</codeph> operators. These operations all require some in-memory work areas that - vary depending on the volume and distribution of data. In Impala 2.0 and later, these kinds of - operations utilize temporary disk work areas if memory usage grows too large to handle. See - <xref href="impala_scalability.xml#spill_to_disk"/> for details. - </p> - </li> - - <li> - <p> - The size of the result set. When intermediate results are being passed around between nodes, the - amount of data depends on the number of columns returned by the query. For example, it is more - memory-efficient to query only the columns that are actually needed in the result set rather than - always issuing <codeph>SELECT *</codeph>. - </p> - </li> - - <li> - <p> - The mechanism by which work is divided for a join query. You use the <codeph>COMPUTE STATS</codeph> - statement, and query hints in the most difficult cases, to help Impala pick the most efficient - execution plan. See <xref href="impala_perf_joins.xml#perf_joins"/> for details. - </p> - </li> - </ul> - - <p> - See <xref href="impala_prereqs.xml#prereqs_hardware"/> for more details and recommendations about - Impala hardware prerequisites. - </p> - - </sectiondiv> - </section> - - <section id="faq_cpu_prereq"> - - <title>What processor type and speed work best for Impala?</title> - - <sectiondiv id="faq_cpu_req"> - - <p rev="CDH-24874"> - Impala makes use of SSE 4.1 instructions. -<!-- Commenting out of caution after IMPALA-160 and CDH-20937. - For best performance, use Nehalem or later for - Intel chips and Bulldozer or later for AMD chips. - Impala runs on older machines with the SSE3 instruction set, - but will not achieve the best performance. - --> - </p> - - </sectiondiv> - </section> - - <section id="faq_prereq_ec2"> - - <title>What EC2 instances are recommended for Impala?</title> - - <p> - For large storage capacity and large I/O bandwidth, consider the <codeph>hs1.8xlarge</codeph> and - <codeph>cc2.8xlarge</codeph> instance types. Impala I/O patterns typically do not benefit enough from SSD - storage to make up for the lower overall size. For performance and security considerations for deploying - CDH and its components on AWS, see - <xref href="http://www.cloudera.com/content/dam/cloudera/Resources/PDF/whitepaper/AWS_Reference_Architecture_Whitepaper.pdf"; scope="external" format="html">Cloudera - Enterprise Reference Architecture for AWS Deployments</xref>. - </p> - </section> - </conbody> - </concept> - - <concept id="faq_features"> - - <title>Supported and Unsupported Functionality In Impala</title> - - <conbody> - - <p outputclass="toc inpage" audience="PDF"> - FAQs in this category: - </p> - - <section id="features"> - - <title>What are the main features of Impala?</title> - - <sectiondiv id="faq_features_sql"> - - <ul> - <li> - A large set of SQL statements, including <xref href="impala_select.xml#select">SELECT</xref> and - <xref href="impala_insert.xml#insert">INSERT</xref>, with - <xref href="impala_joins.xml#joins">joins</xref>, <xref href="impala_subqueries.xml#subqueries"/>, - and <xref href="impala_analytic_functions.xml#analytic_functions"/>. Highly compatible with HiveQL, - and also including some vendor extensions. For more information, see - <xref href="impala_langref.xml#langref"/>. - </li> - - <li> - Distributed, high-performance queries. See <xref href="impala_performance.xml#performance"/> for - information about Impala performance optimizations and tuning techniques for queries. - </li> - - <li audience="hidden"> - Using Cloudera Manager, you can deploy and manage your Impala services. Cloudera Manager is the best - way to get started with Impala on your cluster. - </li> - - <li> - Using Hue for queries. - </li> - - <li> - Appending and inserting data into tables through the - <xref href="impala_insert.xml#insert">INSERT</xref> statement. See - <xref href="impala_file_formats.xml#file_formats"/> for the details about which operations are - supported for which file formats. - </li> - - <li> - ODBC: Impala is certified to run against MicroStrategy and Tableau, with restrictions. For more - information, see <xref href="impala_odbc.xml#impala_odbc"/>. - </li> - - <li> - Querying data stored in HDFS and HBase in a single query. See - <xref href="impala_hbase.xml#impala_hbase"/> for details. - </li> - - <li rev="2.2.0"> - In Impala 2.2.0 and higher, querying data stored in the Amazon Simple Storage Service (S3). See - <xref href="impala_s3.xml#s3"/> for details. - </li> - - <li> - Concurrent client requests. Each Impala daemon can handle multiple concurrent client requests. The - effects on performance depend on your particular hardware and workload. - </li> - - <li> - Kerberos authentication. For more information, see - <xref href="impala_security.xml#security"/>. - </li> - - <li> - Partitions. With Impala SQL, you can create partitioned tables with the <codeph>CREATE TABLE</codeph> - statement, and add and drop partitions with the <codeph>ALTER TABLE</codeph> statement. Impala also - takes advantage of the partitioning present in Hive tables. See - <xref href="impala_partitioning.xml#partitioning"/> for details. - </li> - </ul> - - </sectiondiv> - </section> - - <section id="faq_unsupported"> - - <title>What features from relational databases or Hive are not available in Impala?</title> - - <sectiondiv id="faq_unsupported_sql"> - - <!-- To do: - Good opportunity for a conref since there is a similar "unsupported" topic in the Language Reference section. - --> - - <ul> - <li> - Querying streaming data. - </li> - - <li> - Deleting individual rows. You delete data in bulk by overwriting an entire table or partition, or by - dropping a table. - </li> - - <li> - Indexing (not currently). LZO-compressed text files can be indexed outside of Impala, as described in - <xref href="impala_txtfile.xml#lzo"/>. - </li> - -<!-- - <li> - YARN integration (available when Impala is used with CDH 5). - </li> ---> - - <li> -<!-- Former URL disappeared: cloudera.comcloudera/en/products/cdh/search.html --> -<!-- Subscription URL doesn't seem appropriate: http://www.cloudera.com/content/cloudera/en/products-and-services/cloudera-enterprise/RTS-subscription.html --> - Full text search on text fields. The Cloudera Search product is appropriate for this use case. - </li> - - <li> - Custom Hive Serializer/Deserializer classes (SerDes). Impala supports a set of common native file - formats that have built-in SerDes in CDH. See <xref href="impala_file_formats.xml#file_formats"/> for - details. - </li> - - <li> - Checkpointing within a query. That is, Impala does not save intermediate results to disk during - long-running queries. Currently, Impala cancels a running query if any host on which that query is - executing fails. When one or more hosts are down, Impala reroutes future queries to only use the - available hosts, and Impala detects when the hosts come back up and begins using them again. Because - a query can be submitted through any Impala node, there is no single point of failure. In the future, - we will consider adding additional work allocation features to Impala, so that a running query would - complete even in the presence of host failures. - </li> - -<!-- - <li> - Transforms. - </li> ---> - - <li> - Encryption of data transmitted between Impala daemons. - </li> - -<!-- - <li> - Window functions. - </li> ---> - -<!-- - <li> - Hive UDFs. - </li> ---> - - <li> - Hive indexes. - </li> - - <li> - Non-Hadoop data stores, such as relational databases. - </li> - </ul> - - <p> - For the detailed list of features that are different between Impala and HiveQL, see - <xref href="impala_langref_unsupported.xml#langref_hiveql_delta"/>. - </p> - - </sectiondiv> - </section> - - <section id="faq_jdbc"> - - <title>Does Impala support generic JDBC?</title> - - <sectiondiv id="faq_jdbc_sect"> - - <p> - Impala supports the HiveServer2 JDBC driver. - </p> - - </sectiondiv> - </section> - - <section id="faq_avro"> - - <title>Is Avro supported?</title> - - <sectiondiv id="faq_avro_sect"> - - <p> - Yes, Avro is supported. Impala has always been able to query Avro tables. You can use the Impala - <codeph>LOAD DATA</codeph> statement to load existing Avro data files into a table. Starting with - Impala 1.4, you can create Avro tables with Impala. Currently, you still use the - <codeph>INSERT</codeph> statement in Hive to copy data from another table into an Avro table. See - <xref href="impala_avro.xml#avro"/> for details. - </p> - - </sectiondiv> - </section> - - <section audience="hidden" id="faq_roadmap"> - -<!-- Hidden to avoid RevRec implications. --> - - <title>What's next for Impala?</title> - - <sectiondiv id="faq_next"> - - <p> - See our blog post: - <xref href="http://blog.cloudera.com/blog/2013/09/whats-next-for-impala-after-release-1-1/"; scope="external" format="html">http://blog.cloudera.com/blog/2012/12/whats-next-for-cloudera-impala/</xref> - </p> - - </sectiondiv> - </section> - </conbody> - </concept> - - <concept id="faq_tasks"> - - <title>How do I?</title> - - <conbody> - - <p outputclass="toc inpage" audience="PDF"> - FAQs in this category: - </p> - - <section id="faq_secure_sql_text"> - - <title>How do I prevent users from seeing the text of SQL queries?</title> - - <p> - For instructions on making the Impala log files unreadable by unprivileged users, see - <xref href="impala_security_files.xml#secure_files"/>. - </p> - - <p> - For instructions on password-protecting the web interface to the Impala log files and other internal - server information, see <xref href="impala_security_webui.xml#security_webui"/>. - </p> - - <p rev="2.2.0"> - In <keyword keyref="impala22_full"/> and higher, you can use the log redaction feature - to obfuscate sensitive information in Impala log files. - See - <xref audience="integrated" href="sg_redaction.xml#log_redact"/><xref audience="standalone" href="http://www.cloudera.com/documentation/enterprise/latest/topics/sg_redaction.html"; scope="external" format="html"/> - for details. - </p> - - </section> - - <section id="faq_num_nodes"> - - <title>How do I know how many Impala nodes are in my cluster?</title> - - <p> - The Impala statestore keeps track of how many <cmdname>impalad</cmdname> nodes are currently available. - You can see this information through the statestore web interface. For example, at the URL - <codeph>http://<varname>statestore_host</varname>:25010/metrics</codeph> you might see lines like the - following: - </p> - -<codeblock>statestore.live-backends:3 -statestore.live-backends.list:[<varname>host1</varname>:22000, <varname>host1</varname>:26000, <varname>host2</varname>:22000]</codeblock> - - <p> - The number of <cmdname>impalad</cmdname> nodes is the number of list items referring to port 22000, in - this case two. (Typically, this number is one less than the number reported by the - <codeph>statestore.live-backends</codeph> line.) If an <cmdname>impalad</cmdname> node became unavailable - or came back after an outage, the information reported on this page would change appropriately. - </p> - - <!-- To do: - If there is a good CM technique, mention that here also. - --> - </section> - - </conbody> - </concept> - - <concept id="faq_performance"> - - <title>Impala Performance</title> - - <conbody> - -<!-- Template for new FAQ entries. - <section> - <title></title> - <sectiondiv id=""> - <p> - </p> - </sectiondiv> - </section> - ---> - - <p outputclass="toc inpage" audience="PDF"> - FAQs in this category: - </p> - - <section id="faq_streaming"> - - <title>Are results returned as they become available, or all at once when a query completes?</title> - - <sectiondiv id="faq_stream_results"> - - <p> - Impala streams results whenever they are available, when possible. Certain SQL operations (aggregation - or <codeph>ORDER BY</codeph>) require all of the input to be ready before Impala can return results. - </p> - - </sectiondiv> - </section> - - <section id="faq_slow_query"> - - <title>Why does my query run slowly?</title> - - <sectiondiv id="faq_slow_query_sect"> - - <p> - There are many possible reasons why a given query could be slow. Use the following checklist to - diagnose performance issues with existing queries, and to avoid such issues when writing new queries, - setting up new nodes, creating new tables, or loading data. - </p> - - <ul> - <li rev="1.4.0"> - Immediately after the query finishes, issue a <codeph>SUMMARY</codeph> command in - <cmdname>impala-shell</cmdname>. You can check which phases of execution took the longest, and - compare estimated values for memory usage and number of rows with the actual values. - </li> - - <li> - Immediately after the query finishes, issue a <codeph>PROFILE</codeph> command in - <cmdname>impala-shell</cmdname>. The numbers in the <codeph>BytesRead</codeph>, - <codeph>BytesReadLocal</codeph>, and <codeph>BytesReadShortCircuit</codeph> should be identical for a - specific node. For example: -<codeblock>- BytesRead: 180.33 MB -- BytesReadLocal: 180.33 MB -- BytesReadShortCircuit: 180.33 MB</codeblock> - If <codeph>BytesReadLocal</codeph> is lower than <codeph>BytesRead</codeph>, something in your - cluster is misconfigured, such as the <cmdname>impalad</cmdname> daemon not running on all the data - nodes. If <codeph>BytesReadShortCircuit</codeph> is lower than <codeph>BytesRead</codeph>, - short-circuit reads are not enabled properly on that node; see - <xref href="impala_config_performance.xml#config_performance"/> for instructions. - </li> - - <li> - If the table was just created, or this is the first query that accessed the table after an - <codeph>INVALIDATE METADATA</codeph> statement or after the <cmdname>impalad</cmdname> daemon was - restarted, there might be a one-time delay while the metadata for the table is loaded and cached. - Check whether the slowdown disappears when the query is run again. When doing performance - comparisons, consider issuing a <codeph>DESCRIBE <varname>table_name</varname></codeph> statement for - each table first, to make sure any timings only measure the actual query time and not the one-time - wait to load the table metadata. - </li> - - <li> - Is the table data in uncompressed text format? Check by issuing a <codeph>DESCRIBE FORMATTED - <varname>table_name</varname></codeph> statement. A text table is indicated by the line: -<codeblock>InputFormat: org.apache.hadoop.mapred.TextInputFormat</codeblock> - Although uncompressed text is the default format for a <codeph>CREATE TABLE</codeph> statement with - no <codeph>STORED AS</codeph> clauses, it is also the bulkiest format for disk storage and - consequently usually the slowest format for queries. For data where query performance is crucial, - particularly for tables that are frequently queried, consider starting with or converting to a - compact binary file format such as Parquet, Avro, RCFile, or SequenceFile. For details, see - <xref href="impala_file_formats.xml#file_formats"/>. - </li> - - <li> - If your table has many columns, but the query refers to only a few columns, consider using the - Parquet file format. Its data files are organized with a column-oriented layout that lets queries - minimize the amount of I/O needed to retrieve, filter, and aggregate the values for specific columns. - See <xref href="impala_parquet.xml#parquet"/> for details. - </li> - - <li> - If your query involves any joins, are the tables in the query ordered so that the tables or - subqueries are ordered with the one returning the largest number of rows on the left, followed by the - smallest (most selective), the second smallest, and so on? That ordering allows Impala to optimize - the way work is distributed among the nodes and how intermediate results are routed from one node to - another. For example, all other things being equal, the following join order results in an efficient - query: -<codeblock>select some_col from - huge_table join big_table join small_table join medium_table - where - huge_table.id = big_table.id - and big_table.id = medium_table.id - and medium_table.id = small_table.id;</codeblock> - See <xref href="impala_perf_joins.xml#perf_joins"/> for performance tips for join queries. - </li> - - <li> - Also for join queries, do you have table statistics for the table, and column statistics for the - columns used in the join clauses? Column statistics let Impala better choose how to distribute the - work for the various pieces of a join query. See <xref href="impala_perf_stats.xml#perf_stats"/> for - details about gathering statistics. - </li> - - <li> - Does your table consist of many small data files? Impala works most efficiently with data files in - the multi-megabyte range; Parquet, a format optimized for data warehouse-style queries, uses - <ph rev="parquet_block_size">large files (originally 1 GB, now 256 MB in Impala 2.0 and higher) with - a block size matching the file size</ph>. Use the <codeph>DESCRIBE FORMATTED - <varname>table_name</varname></codeph> statement in <cmdname>impala-shell</cmdname> to see where the - data for a table is located, and use the <cmdname>hadoop fs -ls</cmdname> or <cmdname>hdfs dfs - -ls</cmdname> Unix commands to see the files and their sizes. If you have thousands of small data - files, that is a signal that you should consolidate into a smaller number of large files. Use an - <codeph>INSERT ... SELECT</codeph> statement to copy the data to a new table, reorganizing into new - data files as part of the process. Prefer to construct large data files and import them in bulk - through the <codeph>LOAD DATA</codeph> or <codeph>CREATE EXTERNAL TABLE</codeph> statements, rather - than issuing many <codeph>INSERT ... VALUES</codeph> statements; each <codeph>INSERT ... - VALUES</codeph> statement creates a separate tiny data file. If you have thousands of files all in - the same directory, but each one is megabytes in size, consider using a partitioned table so that - each partition contains a smaller number of files. See the following point for more on partitioning. - </li> - - <li> - If your data is easy to group according to time or geographic region, have you partitioned your table - based on the corresponding columns such as <codeph>YEAR</codeph>, <codeph>MONTH</codeph>, and/or - <codeph>DAY</codeph>? Partitioning a table based on certain columns allows queries that filter based - on those same columns to avoid reading the data files for irrelevant years, postal codes, and so on. - (Do not partition down to too fine a level; try to structure the partitions so that there is still - sufficient data in each one to take advantage of the multi-megabyte HDFS block size.) See - <xref href="impala_partitioning.xml#partitioning"/> for details. - </li> - </ul> - - </sectiondiv> - </section> - - <section id="failed_query"> - - <title>Why does my SELECT statement fail?</title> - - <sectiondiv id="faq_select_fail"> - - <p> - When a <codeph>SELECT</codeph> statement fails, the cause usually falls into one of the following - categories: - </p> - - <ul> - <li> - A timeout because of a performance, capacity, or network issue affecting one particular node. - </li> - - <li> - Excessive memory use for a join query, resulting in automatic cancellation of the query. - </li> - - <li> - A low-level issue affecting how native code is generated on each node to handle particular - <codeph>WHERE</codeph> clauses in the query. For example, a machine instruction could be generated - that is not supported by the processor of a certain node. If the error message in the log suggests - the cause was an illegal instruction, consider turning off native code generation temporarily, and - trying the query again. - </li> - - <li> - Malformed input data, such as a text data file with an enormously long line, or with a delimiter that - does not match the character specified in the <codeph>FIELDS TERMINATED BY</codeph> clause of the - <codeph>CREATE TABLE</codeph> statement. - </li> - </ul> - - </sectiondiv> - </section> - - <section id="failed_insert"> - - <title>Why does my INSERT statement fail?</title> - - <sectiondiv id="faq_insert_fail"> - - <p> - When an <codeph>INSERT</codeph> statement fails, it is usually the result of exceeding some limit - within a Hadoop component, typically HDFS. - </p> - - <ul> - <li> - An <codeph>INSERT</codeph> into a partitioned table can be a strenuous operation due to the - possibility of opening many files and associated threads simultaneously in HDFS. Impala 1.1.1 - includes some improvements to distribute the work more efficiently, so that the values for each - partition are written by a single node, rather than as a separate data file from each node. - </li> - - <li> - Certain expressions in the <codeph>SELECT</codeph> part of the <codeph>INSERT</codeph> statement can - complicate the execution planning and result in an inefficient <codeph>INSERT</codeph> operation. Try - to make the column data types of the source and destination tables match up, for example by doing - <codeph>ALTER TABLE ... REPLACE COLUMNS</codeph> on the source table if necessary. Try to avoid - <codeph>CASE</codeph> expressions in the <codeph>SELECT</codeph> portion, because they make the - result values harder to predict than transferring a column unchanged or passing the column through a - built-in function. - </li> - - <li> - Be prepared to raise some limits in the HDFS configuration settings, either temporarily during the - <codeph>INSERT</codeph> or permanently if you frequently run such <codeph>INSERT</codeph> statements - as part of your ETL pipeline. - </li> - - <li> - The resource usage of an <codeph>INSERT</codeph> statement can vary depending on the file format of - the destination table. Inserting into a Parquet table is memory-intensive, because the data for each - partition is buffered in memory until it reaches 1 gigabyte, at which point the data file is written - to disk. Impala can distribute the work for an <codeph>INSERT</codeph> more efficiently when - statistics are available for the source table that is queried during the <codeph>INSERT</codeph> - statement. See <xref href="impala_perf_stats.xml#perf_stats"/> for details about gathering - statistics. - </li> - </ul> - - </sectiondiv> - </section> - - <section id="faq_scalability"> - - <title>Does Impala performance improve as it is deployed to more hosts in a cluster in much the same way that Hadoop performance does?</title> - - <sectiondiv id="faq_hosts"> - - <draft-comment translate="no"> -Like to combine this one with the DataNodes question a little later. -</draft-comment> - - <p> - Yes. Impala scales with the number of hosts. It is important to install Impala on all the DataNodes in - the cluster, because otherwise some of the nodes must do remote reads to retrieve data not available - for local reads. Data locality is an important architectural aspect for Impala performance. See - <xref href="http://blog.cloudera.com/blog/2014/01/impala-performance-dbms-class-speed/"; scope="external" format="html">this - Impala performance blog post</xref> for background. Note that this blog post refers to benchmarks with - Impala 1.1.1; Impala has added even more performance features in the 1.2.x series. - </p> - - </sectiondiv> - </section> - - <section id="faq_hdfs_block_size"> - - <title>Is the HDFS block size reduced to achieve faster query results?</title> - - <sectiondiv id="faq_block_size"> - - <p> - No. Impala does not make any changes to the HDFS or HBase data sets. - </p> - - <p> - The default Parquet block size is relatively large (<ph rev="parquet_block_size">256 MB in Impala 2.0 - and later; 1 GB in earlier releases</ph>). You can control the block size when creating Parquet files - using the <xref href="impala_parquet_file_size.xml#parquet_file_size">PARQUET_FILE_SIZE</xref> query - option. - </p> - - </sectiondiv> - </section> - - <section id="faq_caching"> - - <title>Does Impala use caching?</title> - - <sectiondiv> - - <p id="caching"> - Impala does not cache table data. It does cache some table and file metadata. Although queries might run - faster on subsequent iterations because the data set was cached in the OS buffer cache, Impala does not - explicitly control this. - </p> - - <p rev="1.4.0"> - Impala takes advantage of the HDFS caching feature in CDH 5. You can designate - which tables or partitions are cached through the <codeph>CACHED</codeph> - and <codeph>UNCACHED</codeph> clauses of the <codeph>CREATE TABLE</codeph> - and <codeph>ALTER TABLE</codeph> statements. - Impala can also take advantage of data that is pinned in the HDFS cache - through the <cmdname>hdfscacheadmin</cmdname> command. - See <xref href="impala_perf_hdfs_caching.xml#hdfs_caching"/> for details. - </p> - - </sectiondiv> - </section> - </conbody> - </concept> - - <concept id="faq_use_cases"> - - <title>Impala Use Cases</title> - <prolog> - <metadata> - <data name="Category" value="Use Cases"/> - </metadata> - </prolog> - - <conbody> - - <p outputclass="toc inpage" audience="PDF"> - FAQs in this category: - </p> - - <section id="faq_impala_hive_mr"> - - <title>What are good use cases for Impala as opposed to Hive or MapReduce?</title> - - <sectiondiv id="faq_impala_vs_hive"> - - <p> - Impala is well-suited to executing SQL queries for interactive exploratory analytics on large data - sets. Hive and MapReduce are appropriate for very long running, batch-oriented tasks such as ETL. - </p> - - </sectiondiv> - </section> - - <section id="faq_mapreduce"> - - <title>Is MapReduce required for Impala? Will Impala continue to work as expected if MapReduce is stopped?</title> - - <sectiondiv id="faq_mapreduce_sect"> - - <p> - Impala does not use MapReduce at all. - </p> - - </sectiondiv> - </section> - - <section id="faq_cep"> - - <title>Can Impala be used for complex event processing?</title> - - <sectiondiv id="faq_cep_sect"> - - <p> - For example, in an industrial environment, many agents may generate large amounts of data. Can Impala - be used to analyze this data, checking for notable changes in the environment? - </p> - - <p> - Complex Event Processing (CEP) is usually performed by dedicated stream-processing systems. Impala is - not a stream-processing system, as it most closely resembles a relational database. - </p> - - </sectiondiv> - </section> - - <section id="faq_ad_hoc"> - - <title>Is Impala intended to handle real time queries in low-latency applications or is it for ad hoc queries for the purpose of data exploration?</title> - - <sectiondiv id="faq_real_time"> - - <p> - Ad-hoc queries are the primary use case for Impala. We anticipate it being used in many other - situations where low-latency is required. Whether Impala is appropriate for any particular use-case - depends on the workload, data size and query volume. See <xref href="impala_intro.xml#benefits"/> for - the primary benefits you can expect when using Impala. - </p> - - </sectiondiv> - </section> - </conbody> - </concept> - - <concept id="faq_hive"> - - <title>Questions about Impala And Hive</title> - - <conbody> - - <p outputclass="toc inpage" audience="PDF"> - FAQs in this category: - </p> - - <draft-comment translate="no"> -Note: earlier question refers to Impala vs. Hive and MapReduce altogether. -Should consolidate since makes sense to have one faq_hive ID. -</draft-comment> - - <section id="faq_hive_pig"> - - <title>How does Impala compare to Hive and Pig?</title> - - <sectiondiv id="faq_hive_pig_sect"> - - <p> - Impala is different from Hive and Pig because it uses its own daemons that are spread across the - cluster for queries. Because Impala does not rely on MapReduce, it avoids the startup overhead of - MapReduce jobs, allowing Impala to return results in real time. - </p> - - </sectiondiv> - </section> - - <section id="faq_serdes"> - - <title>Can I do transforms or add new functionality?</title> - - <sectiondiv id="faq_udf"> - - <p> - Impala adds support for UDFs in Impala 1.2. You can write your own functions in C++, or reuse existing - Java-based Hive UDFs. The UDF support includes scalar functions and user-defined aggregate functions - (UDAs). User-defined table functions (UDTFs) are not currently supported. - </p> - - <p> - Impala does not currently support an extensible serialization-deserialization framework (SerDes), and - so adding extra functionality to Impala is not as straightforward as for Hive or Pig. - </p> - - </sectiondiv> - </section> - - <section id="faq_hive_compat"> - - <title>Can any Impala query also be executed in Hive?</title> - - <sectiondiv id="faq_hiveql"> - - <p> - Yes. There are some minor differences in how some queries are handled, but Impala queries can also be - completed in Hive. Impala SQL is a subset of HiveQL, with some functional limitations such as - transforms. For details of the Impala SQL dialect, see - <xref href="impala_langref_sql.xml#langref_sql"/>. For the Impala built-in functions, see - <xref href="impala_functions.xml#builtins"/>. For the detailed list of differences between Impala and - HiveQL, see <xref href="impala_langref_unsupported.xml#langref_hiveql_delta"/>. - </p> - - </sectiondiv> - </section> - - <section id="faq_hive_hbase_import"> - - <title>Can I use Impala to query data already loaded into Hive and HBase?</title> - - <sectiondiv id="faq_hive_hbase"> - - <p> - There are no additional steps to allow Impala to query tables managed by Hive, whether they are stored - in HDFS or HBase. Make sure that Impala is configured to access the Hive metastore correctly and you - should be ready to go. Keep in mind that <codeph>impalad</codeph>, by default, runs as the - <codeph>impala</codeph> user, so you might need to adjust some file permissions depending on how strict - your permissions are currently. - </p> - - <p> - See <xref href="impala_hbase.xml#impala_hbase"/> for details about querying data in HBase. - </p> - - </sectiondiv> - </section> - - <section id="faq_hive_prereq"> - - <title>Is Hive an Impala requirement?</title> - - <sectiondiv id="faq_hive_prereq_sect"> - - <p> - The Hive metastore service is a requirement. Impala shares the same metastore database as Hive, - allowing Impala and Hive to access the same tables transparently. - </p> - - <p> - Hive itself is optional, and does not need to be installed on the same nodes as Impala. Currently, - Impala supports a wider variety of read (query) operations than write (insert) operations; you use Hive - to insert data into tables that use certain file formats. See - <xref href="impala_file_formats.xml#file_formats"/> for details. - </p> - - </sectiondiv> - </section> - </conbody> - </concept> - - <concept id="faq_ha"> - - <title>Impala Availability</title> - - <conbody> - - <p outputclass="toc inpage" audience="PDF"> - FAQs in this category: - </p> - - <section id="faq_production"> - - <title>Is Impala production ready?</title> - - <sectiondiv id="faq_production_sect"> - - <p> - Impala has finished its beta release cycle, and the 1.0, 1.1, and 1.2 GA releases are production ready. - The 1.1.x series includes additional security features for authorization, an important requirement for - production use in many organizations. The 1.2.x series includes important performance features, - particularly for large join queries. Some Cloudera customers are already using Impala for large - workloads. - </p> - - <p rev="1.3.0"> - The Impala 1.3.0 and higher releases are bundled with corresponding levels of CDH 5. - The number of new features grows with each release. - See <xref href="impala_new_features.xml#new_features"/> for a full list. - </p> - - </sectiondiv> - </section> - - <section id="faq_ha_config"> - - <title>How do I configure Hadoop high availability (HA) for Impala?</title> - - <sectiondiv id="faq_ha_config_sect"> - - <p rev="1.2.0"> - You can set up a proxy server to relay requests back and forth to the Impala servers, for load - balancing and high availability. See <xref href="impala_proxy.xml#proxy"/> for details. - </p> - - <p> - You can enable HDFS HA for the Hive metastore. See the - <xref href="http://www.cloudera.com/documentation/enterprise/latest/topics/cdh_hag_cdh_other_ha.html"; scope="external" format="html">CDH5 High Availability Guide</xref> - for details. - </p> - - </sectiondiv> - </section> - - <section id="faq_spof"> - - <title>What happens if there is an error in Impala?</title> - - <sectiondiv id="faq_spof_sect"> - - <p> - There is not a single point of failure in Impala. All Impala daemons are fully able to handle incoming - queries. If a machine fails however, all queries with fragments running on that machine will fail. - Because queries are expected to return quickly, you can just rerun the query if there is a failure. See - <xref href="impala_concepts.xml#concepts"/> for details about the Impala architecture. - </p> - - <draft-comment translate="no"> -Clarify to what extent the catalog service could be seen as a single point of failure. -</draft-comment> - - <p> - The longer answer: Impala must be able to connect to the Hive metastore. Impala aggressively caches - metadata so the metastore host should have minimal load. Impala relies on the HDFS NameNode, and, in - CDH4, you can configure HA for HDFS. Impala also has centralized services, known as the - <xref href="impala_components.xml#intro_statestore">statestore</xref> and - <xref href="impala_components.xml#intro_catalogd">catalog</xref> services, that run on one host only. - Impala continues to execute queries if the statestore host is down, but it will not get state updates. - For example, if a host is added to the cluster while the statestore host is down, the existing - instances of <codeph>impalad</codeph> running on the other hosts will not find out about this new host. - Once the statestore process is restarted, all the information it serves is automatically reconstructed - from all running Impala daemons. - </p> - - </sectiondiv> - </section> - - <section id="faq_max_rows"> - - <title>What is the maximum number of rows in a table?</title> - - <sectiondiv id="faq_max_rows_sect"> - - <p> - There is no defined maximum. Some customers have used Impala to query a table with over a trillion - rows. - </p> - - </sectiondiv> - </section> - - <section id="faq_contention"> - - <title>Can Impala and MapReduce jobs run on the same cluster without resource contention?</title> - - <sectiondiv id="faq_mapreduce_contention"> - - <p> - Yes. See <xref href="impala_perf_resources.xml#mem_limits"/> for how to control Impala resource usage - using the Linux cgroup mechanism, and <xref href="impala_resource_management.xml#resource_management"/> - for how to use Impala with the YARN resource management framework. Impala is designed to run on the - DataNode hosts. Any contention depends mostly on the cluster setup and workload. - </p> - - <p conref="../shared/impala_common.xml#common/impala_mr"/> - - </sectiondiv> - </section> - </conbody> - </concept> - - <concept id="faq_internals"> - - <title>Impala Internals</title> - - <conbody> - - <p outputclass="toc inpage" audience="PDF"> - FAQs in this category: - </p> - - <section id="faq_impalad_hosts"> - - <title>On which hosts does Impala run?</title> - - <sectiondiv id="faq_data_nodes"> - - <p> - Cloudera strongly recommends running the <cmdname>impalad</cmdname> daemon on each DataNode for good - performance. Although this topology is not a hard requirement, if there are data blocks with no Impala - daemons running on any of the hosts containing replicas of those blocks, queries involving that data - could be very inefficient. In that case, the data must be transmitted from one host to another for - processing by <q>remote reads</q>, a condition Impala normally tries to avoid. See - <xref href="impala_concepts.xml#concepts"/> for details about the Impala architecture. Impala schedules - query fragments on all hosts holding data relevant to the query, if possible. - </p> - - <p> - In cases where some hosts in the cluster have much greater CPU and memory capacity than others, or - where some hosts have extra CPU capacity because some CPU-intensive phases are single-threaded, - some users have run multiple <cmdname>impalad</cmdname> daemons on a single host to take advantage - of the extra CPU capacity. This configuration is only practical for specific workloads that - rely heavily on aggregation, and the physical hosts must have sufficient memory to accomodate - the requirements for multiple <cmdname>impalad</cmdname> instances. - </p> - - </sectiondiv> - </section> - - <section id="faq_join_internals"> - - <title>How are joins performed in Impala?</title> - - <sectiondiv id="faq_joins"> - - <draft-comment translate="no"> -Will change with join order optimizations, now slated for 1.2.2. -</draft-comment> - - <p> - By default, Impala automatically determines the most efficient order in which to join tables using a - cost-based method, based on their overall size and number of rows. (This is a new feature in Impala - 1.2.2 and higher.) The <codeph>COMPUTE STATS</codeph> statement gathers information about each table - that is crucial for efficient join performance. -<!-- - The order in which tables are joined is the same order in which tables appear in the - <codeph>SELECT</codeph> statement's - <codeph>FROM</codeph> clause. That is, there is no join order optimization - taking place at the moment. It is usually optimal for the smallest table to appear as the right-most table in - a <codeph>JOIN</codeph> clause. - --> - Impala chooses between two techniques for join queries, known as <q>broadcast joins</q> and - <q>partitioned joins</q>. See <xref href="impala_joins.xml#joins"/> for syntax details and - <xref href="impala_perf_joins.xml#perf_joins"/> for performance considerations. - </p> - - </sectiondiv> - </section> - - <section id="faq_join_sizes"> - - <title>How does Impala process join queries for large tables?</title> - - <sectiondiv> - - <p> - Impala utilizes multiple strategies to allow joins between tables and result sets of various sizes. - When joining a large table with a small one, the data from the small table is transmitted to each node - for intermediate processing. When joining two large tables, the data from one of the tables is divided - into pieces, and each node processes only selected pieces. See <xref href="impala_joins.xml#joins"/> - for details about join processing, <xref href="impala_perf_joins.xml#perf_joins"/> for performance - considerations, and <xref href="impala_hints.xml#hints"/> for how to fine-tune the join strategy. - </p> - - </sectiondiv> - </section> - - <section id="faq_aggregation_implementation"> - - <title>What is Impala's aggregation strategy?</title> - - <sectiondiv id="faq_join_aggregation"> - - <p rev="2.0.0"> - Impala currently only supports in-memory hash aggregation. - In Impala 2.0 and higher, if the memory requirements for a - join or aggregation operation exceed the memory limit for - a particular host, Impala uses a temporary work area on disk - to help the query complete successfully. - </p> - - </sectiondiv> - </section> - - <section id="faq_metadata_management"> - - <title>How is Impala metadata managed?</title> - - <sectiondiv id="faq_metadata"> - - <draft-comment translate="no"> -Doesn't seem related to joins... -</draft-comment> - - <p> - Impala uses two pieces of metadata: the catalog information from the Hive metastore and the file - metadata from the NameNode. Currently, this metadata is lazily populated and cached when an - <codeph>impalad</codeph> needs it to plan a query. - </p> - - <p> - The <xref href="impala_refresh.xml#refresh">REFRESH</xref> statement updates the metadata for a - particular table after loading new data through Hive. The - <xref href="impala_invalidate_metadata.xml#invalidate_metadata"/> statement refreshes all metadata, so - that Impala recognizes new tables or other DDL and DML changes performed through Hive. - </p> - - <p rev="1.2.0"> - In Impala 1.2 and higher, a dedicated <cmdname>catalogd</cmdname> daemon broadcasts metadata changes - due to Impala DDL or DML statements to all nodes, reducing or eliminating the need to use the - <codeph>REFRESH</codeph> and <codeph>INVALIDATE METADATA</codeph> statements. - </p> - - </sectiondiv> - </section> - - <section id="faq_namenode_overhead"> - - <title>What load do concurrent queries produce on the NameNode?</title> - - <sectiondiv id="faq_namenode_load"> - - <p> - The load Impala generates is very similar to MapReduce. Impala contacts the NameNode during the - planning phase to get the file metadata (this is only run on the host the query was sent to). Every - <codeph>impalad</codeph> will read files as part of normal processing of the query. - </p> - - </sectiondiv> - </section> - - <section id="faq_perf_architecture"> - - <title>How does Impala achieve its performance improvements?</title> - - <sectiondiv id="faq_performance_features"> - - <p> - These are the main factors in the performance of Impala versus that of other Hadoop components and - related technologies. - </p> - - <p> - Impala avoids MapReduce. While MapReduce is a great general parallel processing model with many - benefits, it is not designed to execute SQL. Impala avoids the inefficiencies of MapReduce in these - ways: - </p> - - <ul> - <li> - Impala does not materialize intermediate results to disk. SQL queries often map to multiple MapReduce - jobs with all intermediate data sets written to disk. - </li> - - <li> - Impala avoids MapReduce start-up time. For interactive queries, the MapReduce start-up time becomes - very noticeable. Impala runs as a service and essentially has no start-up time. - </li> - - <li> - Impala can more naturally disperse query plans instead of having to fit them into a pipeline of map - and reduce jobs. This enables Impala to parallelize multiple stages of a query and avoid overheads - such as sort and shuffle when unnecessary. - </li> - </ul> - - <p> - Impala uses a more efficient execution engine by taking advantage of modern hardware and technologies: - </p> - - <ul> - <li> - Impala generates runtime code. Impala uses LLVM to generate assembly code for the query that is being - run. Individual queries do not have to pay the overhead of running on a system that needs to be able - to execute arbitrary queries. - </li> - - <li> - Impala uses available hardware instructions when possible. Impala uses the supplemental SSE3 (SSSE3) - instructions which can offer tremendous speedups in some cases. (Impala 2.0 and 2.1 required - the SSE4.1 instruction set; Impala 2.2 and higher relax the restriction again so only - SSSE3 is required.) - </li> - - <li> - Impala uses better I/O scheduling. Impala is aware of the disk location of blocks and is able to - schedule the order to process blocks to keep all disks busy. - </li> - - <li> - Impala is designed for performance. A lot of time has been spent in designing Impala with sound - performance-oriented fundamentals, such as tight inner loops, inlined function calls, minimal - branching, better use of cache, and minimal memory usage. - </li> - </ul> - - </sectiondiv> - </section> - - <section id="faq_memory_exceeded"> - - <title>What happens when the data set exceeds available memory?</title> - - <sectiondiv id="faq_mem_limit_exceeded"> - - <p> - Currently, if the memory required to process intermediate results on a node exceeds the amount - available to Impala on that node, the query is cancelled. You can adjust the memory available to Impala - on each node, and you can fine-tune the join strategy to reduce the memory required for the biggest - queries. We do plan on supporting external joins and sorting in the future. - </p> - - <p> - Keep in mind though that the memory usage is not directly based on the input data set size. For - aggregations, the memory usage is the number of rows <i>after</i> grouping. For joins, the memory usage - is the combined size of the tables <i>excluding</i> the biggest table, and Impala can use join - strategies that divide up large joined tables among the various nodes rather than transmitting the - entire table to each node. - </p> - - </sectiondiv> - </section> - - <section id="faq_memory_pressure"> - - <title>What are the most memory-intensive operations?</title> - - <sectiondiv id="faq_memory_fail"> - - <p> - If a query fails with an error indicating <q>memory limit exceeded</q>, you might suspect a memory - leak. The problem could actually be a query that is structured in a way that causes Impala to allocate - more memory than you expect, exceeded the memory allocated for Impala on a particular node. Some - examples of query or table structures that are especially memory-intensive are: - </p> - - <ul> - <li> - <codeph>INSERT</codeph> statements using dynamic partitioning, into a table with many different - partitions. (Particularly for tables using Parquet format, where the data for each partition is held - in memory until it reaches <ph rev="parquet_block_size">the full block size</ph> in size before it is - written to disk.) Consider breaking up such operations into several different <codeph>INSERT</codeph> - statements, for example to load data one year at a time rather than for all years at once. - </li> - - <li> - <codeph>GROUP BY</codeph> on a unique or high-cardinality column. Impala allocates some handler - structures for each different value in a <codeph>GROUP BY</codeph> query. Having millions of - different <codeph>GROUP BY</codeph> values could exceed the memory limit. - </li> - - <li> - Queries involving very wide tables, with thousands of columns, particularly with many - <codeph>STRING</codeph> columns. Because Impala allows a <codeph>STRING</codeph> value to be up to 32 - KB, the intermediate results during such queries could require substantial memory allocation. - </li> - </ul> - - </sectiondiv> - </section> - - <section id="faq_memory_dealloc"> - - <title>When does Impala hold on to or return memory?</title> - - <p> - Impala allocates memory using - <codeph><xref href="http://goog-perftools.sourceforge.net/doc/tcmalloc.html"; scope="external" format="html">tcmalloc</xref></codeph>, - a memory allocator that is optimized for high concurrency. Once Impala allocates memory, it keeps that - memory reserved to use for future queries. Thus, it is normal for Impala to show high memory usage when - idle. If Impala detects that it is about to exceed its memory limit (defined by the - <codeph>-mem_limit</codeph> startup option or the <codeph>MEM_LIMIT</codeph> query option), it - deallocates memory not needed by the current queries. - </p> - - <p> - When issuing queries through the JDBC or ODBC interfaces, make sure to call the appropriate close method - afterwards. Otherwise, some memory associated with the query is not freed. - </p> - </section> - </conbody> - </concept> - - <concept id="faq_sql"> - - <title>SQL</title> - - <conbody> - - <p outputclass="toc inpage" audience="PDF"> - FAQs in this category: - </p> - - <section id="faq_update"> - - <title>Is there an UPDATE statement?</title> - - <sectiondiv id="faq_update_sect"> - - <p> - Impala does not currently have an <codeph>UPDATE</codeph> statement, which would typically be used to - change a single row, a small group of rows, or a specific column. The HDFS-based files used by typical - Impala queries are optimized for bulk operations across many megabytes of data at a time, making - traditional <codeph>UPDATE</codeph> operations inefficient or impractical. - </p> - - <p> - You can use the following techniques to achieve the same goals as the familiar <codeph>UPDATE</codeph> - statement, in a way that preserves efficient file layouts for subsequent queries: - </p> - - <ul> - <li> - Replace the entire contents of a table or partition with updated data that you have already staged in - a different location, either using <codeph>INSERT OVERWRITE</codeph>, <codeph>LOAD DATA</codeph>, or - manual HDFS file operations followed by a <codeph>REFRESH</codeph> statement for the table. - Optionally, you can use built-in functions and expressions in the <codeph>INSERT</codeph> statement - to transform the copied data in the same way you would normally do in an <codeph>UPDATE</codeph> - statement, for example to turn a mixed-case string into all uppercase or all lowercase. - </li> - - <li> - To update a single row, use an HBase table, and issue an <codeph>INSERT ... VALUES</codeph> statement - using the same key as the original row. Because HBase handles duplicate keys by only returning the - latest row with a particular key value, the newly inserted row effectively hides the previous one. - </li> - </ul> - - </sectiondiv> - </section> - - <section id="faq_udfs"> - - <title>Can Impala do user-defined functions (UDFs)?</title> - - <p> - Impala 1.2 and higher does support UDFs and UDAs. You can either write native Impala UDFs and UDAs in - C++, or reuse UDFs (but not UDAs) originally written in Java for use with Hive. See - <xref href="impala_udf.xml#udfs"/> for details. - </p> - </section> - - <section id="faq_refresh"> - - <title>Why do I have to use REFRESH and INVALIDATE METADATA, what do they do?</title> - - <p> - In Impala 1.2 and higher, there is much less need to use the <codeph>REFRESH</codeph> and - <codeph>INVALIDATE METADATA</codeph> statements: - </p> - - <ul> - <li> - The new <codeph>impala-catalog</codeph> service, represented by the <cmdname>catalogd</cmdname> daemon, - broadcasts the results of Impala DDL statements to all Impala nodes. Thus, if you do a <codeph>CREATE - TABLE</codeph> statement in Impala while connected to one node, you do not need to do - <codeph>INVALIDATE METADATA</codeph> before issuing queries through a different node. - </li> - - <li> - The catalog service only recognizes changes made through Impala, so you must still issue a - <codeph>REFRESH</codeph> statement if you load data through Hive or by manipulating files in HDFS, and - you must issue an <codeph>INVALIDATE METADATA</codeph> statement if you create a table, alter a table, - add or drop partitions, or do other DDL statements in Hive. - </li> - - <li> - Because the catalog service broadcasts the results of <codeph>REFRESH</codeph> and <codeph>INVALIDATE - METADATA</codeph> statements to all nodes, in the cases where you do still need to issue those - statements, you can do that on a single node rather than on every node, and the changes will be - automatically recognized across the cluster, making it more convenient to load balance by issuing - queries through arbitrary Impala nodes rather than always using the same coordinator node. - </li> - </ul> - </section> - - <section id="faq_drop_table_space"> - - <title>Why is space not freed up when I issue DROP TABLE?</title> - - <p> - Impala deletes data files when you issue a <codeph>DROP TABLE</codeph> on an internal table, but not an - external one. By default, the <codeph>CREATE TABLE</codeph> statement creates internal tables, where the - files are managed by Impala. An external table is created with a <codeph>CREATE EXTERNAL TABLE</codeph> - statement, where the files reside in a location outside the control of Impala. Issue a <codeph>DESCRIBE - FORMATTED</codeph> statement to check whether a table is internal or external. The keyword - <codeph>MANAGED_TABLE</codeph> indicates an internal table, from which Impala can delete the data files. - The keyword <codeph>EXTERNAL_TABLE</codeph> indicates an external table, where Impala will leave the data - files untouched when you drop the table. - </p> - - <p> - Even when you drop an internal table and the files are removed from their original location, you might - not get the hard drive space back immediately. By default, files that are deleted in HDFS go into a - special trashcan directory, from which they are purged after a period of time (by default, 6 hours). For - background information on the trashcan mechanism, see - <xref href="https://archive.cloudera.com/cdh4/cdh/4/hadoop/hadoop-project-dist/hadoop-hdfs/HdfsDesign.html"; scope="external" format="html"/>. - For information on purging files from the trashcan, see - <xref href="https://archive.cloudera.com/cdh4/cdh/4/hadoop/hadoop-project-dist/hadoop-common/FileSystemShell.html"; scope="external" format="html"/>. - </p> - - <p> - When Impala deletes files and they are moved to the HDFS trashcan, they go into an HDFS directory owned - by the <codeph>impala</codeph> user. If the <codeph>impala</codeph> user does not have an HDFS home - directory where a trashcan can be created, the files are not deleted or moved, as a safety measure. If - you issue a <codeph>DROP TABLE</codeph> statement and find that the table data files are left in their - original location, create an HDFS directory <filepath>/user/impala</filepath>, owned and writeable by - the <codeph>impala</codeph> user. For example, you might find that <filepath>/user/impala</filepath> is - owned by the <codeph>hdfs</codeph> user, in which case you would switch to the <codeph>hdfs</codeph> user - and issue a command such as: - </p> - -<codeblock>hdfs dfs -chown -R impala /user/impala</codeblock> - </section> - - <section id="faq_dual"> - - <title>Is there a DUAL table?</title> - - <p> - You might be used to running queries against a single-row table named <codeph>DUAL</codeph> to try out - expressions, built-in functions, and UDFs. Impala does not have a <codeph>DUAL</codeph> table. To achieve - the same result, you can issue a <codeph>SELECT</codeph> statement without any table name: - </p> - -<codeblock>select 2+2; -select substr('hello',2,1); -select pow(10,6); -</codeblock> - </section> - </conbody> - </concept> - - <concept id="faq_partitioning"> - - <title>Partitioned Tables</title> - - <conbody> - - <p outputclass="toc inpage" audience="PDF"> - FAQs in this category: - </p> - - <section id="faq_partition_csv_etl"> - - <title>How do I load a big CSV file into a partitioned table?</title> - - <p> - To load a data file into a partitioned table, when the data file includes fields like year, month, and so - on that correspond to the partition key columns, use a two-stage process. First, use the <codeph>LOAD - DATA</codeph> or <codeph>CREATE EXTERNAL TABLE</codeph> statement to bring the data into an unpartitioned - text table. Then use an <codeph>INSERT ... SELECT</codeph> statement to copy the data from the - unpartitioned table to a partitioned one. Include a <codeph>PARTITION</codeph> clause in the - <codeph>INSERT</codeph> statement to specify the partition key columns. The <codeph>INSERT</codeph> - operation splits up the data into separate data files for each partition. For examples, see - <xref href="impala_partitioning.xml#partitioning"/>. For details about loading data into partitioned - Parquet tables, a popular choice for high-volume data, see <xref href="impala_parquet.xml#parquet_etl"/>. - </p> - </section> - - <section id="faq_partition_select_star"> - - <title>Can I do INSERT ... SELECT * into a partitioned table?</title> - - <p> - When you use the <codeph>INSERT ... SELECT *</codeph> syntax to copy data into a partitioned table, the - columns corresponding to the partition key columns must appear last in the columns returned by the - <codeph>SELECT *</codeph>. You can create the table with the partition key columns defined last. Or, you - can use the <codeph>CREATE VIEW</codeph> statement to create a view that reorders the columns: put the - partition key columns last, then do the <codeph>INSERT ... SELECT *</codeph> from the view. - </p> - </section> - </conbody> - </concept> - - <concept id="faq_hbase"> - - <title>HBase</title> - - <conbody> - - <p outputclass="toc inpage" audience="PDF"> - FAQs in this category: - </p> - - <section id="faq_hbase_use_cases"> - - <title>What kinds of Impala queries or data are best suited for HBase?</title> - - <p> - HBase tables are ideal for queries where normally you would use a key-value store. That is, where you - retrieve a single row or a few rows, by testing a special unique key column using the <codeph>=</codeph> - or <codeph>IN</codeph> operators. - </p> - - <p> - HBase tables are not suitable for queries that produce large result sets with thousands of rows. HBase - tables are also not suitable for queries that perform full table scans because the <codeph>WHERE</codeph> - clause does not request specific values from the unique key column. - </p> - - <p> - Use HBase tables for data that is inserted one row or a few rows at a time, such as by the <codeph>INSERT - ... VALUES</codeph> syntax. Loading data piecemeal like this into an HDFS-backed table produces many tiny - files, which is a very inefficient layout for HDFS data files. - </p> + <p> + This section is under construction. + </p> - <p> - If the lack of an <codeph>UPDATE</codeph> statement in Impala is a problem for you, you can simulate - single-row updates by doing an <codeph>INSERT ... VALUES</codeph> statement using an existing value for - the key column. The old row value is hidden; only the new row value is seen by queries. - </p> + </conbody> - <p> - HBase tables are often wide (containing many columns) and sparse (with most column values - <codeph>NULL</codeph>). For example, you might record hundreds of different data points for each user of - an online service, such as whether the user had registered for an online game or enabled particular - account features. With Impala and HBase, you could look up all the information for a specific customer - efficiently in a single query. For any given customer, most of these columns might be - <codeph>NULL</codeph>, because a typical customer might not make use of most features of an online - service. - </p> - </section> - </conbody> - </concept> </concept>
