http://git-wip-us.apache.org/repos/asf/incubator-hawq-docs/blob/7514e193/install/aws-config.html.md.erb ---------------------------------------------------------------------- diff --git a/install/aws-config.html.md.erb b/install/aws-config.html.md.erb new file mode 100644 index 0000000..395970b --- /dev/null +++ b/install/aws-config.html.md.erb @@ -0,0 +1,135 @@ +--- +title: Amazon EC2 Configuration +--- + +Amazon Elastic Compute Cloud (EC2) is a service provided by Amazon Web Services (AWS). You can install and configure HAWQ on virtual servers provided by Amazon EC2. The following information describes some considerations when deploying a HAWQ cluster in an Amazon EC2 environment. + +## <a id="topic_wqv_yfx_y5"></a>About Amazon EC2 + +Amazon EC2 can be used to launch as many virtual servers as you need, configure security and networking, and manage storage. An EC2 *instance* is a virtual server in the AWS cloud virtual computing environment. + +EC2 instances are managed by AWS. AWS isolates your EC2 instances from other users in a virtual private cloud (VPC) and lets you control access to the instances. You can configure instance features such as operating system, network connectivity (network ports and protocols, IP addresses), access to the Internet, and size and type of disk storage. + +For information about Amazon EC2, see the [EC2 User Guide](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/concepts.html). + +## <a id="topic_nhk_df4_2v"></a>Create and Launch HAWQ Instances + +Use the *Amazon EC2 Console* to launch instances and configure, start, stop, and terminate (delete) virtual servers. When you launch a HAWQ instance, you select and configure key attributes via the EC2 Console. + + +### <a id="topic_amitype"></a>Choose AMI Type + +An Amazon Machine Image (AMI) is a template that contains a software configuration including the operating system, application server, and applications that best suit your purpose. When configuring a HAWQ virtual instance, we recommend you use a *hardware virtualized* AMI running 64-bit Red Hat Enterprise Linux version 6.4 or 6.5 or 64-bit CentOS 6.4 or 6.5. Obtain the licenses and instances directly from the OS provider. + +### <a id="topic_selcfgstorage"></a>Consider Storage +EC2 instances can be launched as either Elastic Block Store (EBS)-backed or instance store-backed. + +Instance store-backed storage is generally better performing than EBS and recommended for HAWQ's large data workloads. SSD (solid state) instance store is preferred over magnetic drives. + +**Note** EC2 *instance store* provides temporary block-level storage. This storage is located on disks that are physically attached to the host computer. While instance store provides high performance, powering off the instance causes data loss. Soft reboots preserve instance store data. + +Virtual devices for instance store volumes for HAWQ EC2 instance store instances are named `ephemeralN` (where *N* varies based on instance type). CentOS instance store block device are named `/dev/xvdletter` (where *letter* is a lower case letter of the alphabet). + +?? DO WE RECOMMEND EBS-backed VOLUMES FOR ANYTHING? I.E. BACKUPS, ...??) + + +### <a id="topic_cfgplacegrp"></a>Configure Placement Group + +A placement group is a logical grouping of instances within a single availability zone that together participate in a low-latency, 10 Gbps network. Your HAWQ master and segment cluster instances should support enhanced networking and reside in a single placement group (and subnet) for optimal network performance. + +If your Ambari node is not a data node, locating the Ambari node instance in a subnet separate from the HAWQ master/segment placement group enables you to manage multiple HAWQ clusters from the single Ambari instance. + +Amazon recommends that you use the same instance type for all instances in the placement group and that you launch all instances within the placement group at the same time. + +Membership in a placement group has some implications on your HAWQ cluster. Specifically, growing the cluster over capacity may require shutting down all HAWQ instances in the current placement group and restarting the instances to a new placement group. Instance store volumes are lost in this scenario. + +### <a id="topic_selinsttype"></a>Select EC2 Instance Type + +An EC2 instance type is a specific combination of CPU, memory, default storage, and networking capacity. + +Several instance store-backed EC2 instance types have shown acceptable performance for HAWQ nodes in development and production environments: + +| Instance Type | Env | vCPUs | Memory (GB) | Disk Capacity (GB) | Storage Type | +|-------|-----|------|--------|----------|--------| +| cc2.8xlarge | Dev | 32 | 60.5 | 4 x 840 | HDD | +| d2.2xlarge | Dev | 8 | 60 | 6 x 2000 | HDD | +| d2.4xlarge | Dev/QA | 16 | 122 | 12 x 2000 | HDD | +| i2.8xlarge | Prod | 32 | 244 | 8 x 800 | SSD | +| hs1.8xlarge | Prod | 16 | 117 | 24 x 2000 | HDD | +| d2.8xlarge | Prod | 36 | 244 | 24 x 2000 | HDD | + +For optimal network performance, the chosen HAWQ instance type should support EC2 enhanced networking. Enhanced networking results in higher performance, lower latency, and lower jitter. Refer to [Enhanced Networking on Linux Instances](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/enhanced-networking.html) for detailed information on enabling enhanced networking in your instances. + +All instance types identified in the table above support enhanced networking. + +### <a id="topic_cfgnetw"></a>Configure Networking + +Your HAWQ cluster instances should be in a single VPC and on the same subnet. Instances are always assigned a VPC internal IP address. This internal IP address should be used for HAWQ communication between hosts. You can also use the internal IP address to access an instance from another instance within the HAWQ VPC. + +You may choose to locate your Ambari node on a separate subnet in the VPC. Both a public IP address for the instance and an Internet gateway configured for the EC2 VPC are required to access the Ambari instance from an external source and for the instance to access the Internet. + +Ensure your Ambari and HAWQ master instances are each assigned a public IP address for external and internet access. We recommend you also assign an Elastic IP Address to the HAWQ master instance. + + +###Configure Security Groups<a id="topic_cfgsecgrp"></a> + +A security group is a set of rules that control network traffic to and from your HAWQ instance. One or more rules may be associated with a security group, and one or more security groups may be associated with an instance. + +To configure HAWQ communication between nodes in the HAWQ cluster, include and open the following ports in the appropriate security group for the HAWQ master and segment nodes: + +?? PROVIDE SPECIFIC type/protocol/portrange/source settings from EC2 console ?? + +| Port | Application | +|-------|-------------------------------------| +| 22 | ssh - secure connect to other hosts | +| XX | XX | + +UDP access within the VPC should be enabled by default. + + +?? DIFFENT SECURITY GROUP FOR AMBARI NODE ?? + +To allow access to/from a source external to the Ambari management node, include and open the following ports in an appropriate security group for your Ambari node: + +?? PROVIDE SPECIFIC type/protocol/portrange/source settings from EC2 console ?? + +| Port | Application | +|-------|-------------------------------------| +| 22 | ssh - secure connect to other hosts | +| 8080 | Ambari - HAWQ admin/config web console | + + +###Generate Key Pair<a id="topic_cfgkeypair"></a> +AWS uses public-key cryptography to secure the login information for your instance. You use the EC2 console to generate and name a key pair when you launch your instance. + +A key pair for an EC2 instance consists of a *public key* that AWS stores, and a *private key file* that you maintain. Together, they allow you to connect to your instance securely. The private key file name typically has a `.pem` suffix. + +This example logs into an into EC2 instance from an external location with the private key file `my-test.pem` as user `user1`. In this example, the instance is configured with the public IP address `192.0.2.0` and the private key file resides in the current directory. + +```shell +$ ssh -i my-test.pem [email protected] +``` + +##Additional HAWQ Considerations <a id="topic_mj4_524_2v"></a> + +After launching your HAWQ instance, you will connect to and configure the instance. The *Instances* page of the EC2 Console lists the running instances and their associated network access information. + +Before installing HAWQ, set up the EC2 instances as you would local host server machines. Configure the host operating system, configure host network information (for example, update the `/etc/hosts` file), set operating system parameters, and install operating system packages. For information about how to prepare your operating system environment for HAWQ, see [Select HAWQ Host Machines](../install/select-hosts.html). + +?? PACKAGES, KERNEL VERSION, KERNEL SETTINGS, BEST PRACTICES HERE OR WILL THEY BE THE SAME AS FOR PHYSICAL DEVICE?? + + +##References<a id="topic_hgz_zwy_bv"></a> + +Links to related Amazon Web Services and EC2 features and information. + +- [Amazon Web Services](https://aws.amazon.com) +- [Amazon Machine Image \(AMI\)](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/AMIs.html) +- [EC2 Instance Store](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/InstanceStorage.html) +- [Elastic Block Store](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/EBSOptimized.html) +- [EC2 Key Pairs](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-key-pairs.html) +- [Elastic IP Address](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/elastic-ip-addresses-eip.html) +- [Enhanced Networking on Linux Instances](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/enhanced-networking.html) +- [Internet Gateways] (http://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/VPC_Internet_Gateway.html) +- [Subnet Public IP Addressing](http://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/vpc-ip-addressing.html#subnet-public-ip) +- [Virtual Private Cloud](http://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/VPC_Introduction.html)
http://git-wip-us.apache.org/repos/asf/incubator-hawq-docs/blob/7514e193/install/select-hosts.html.md.erb ---------------------------------------------------------------------- diff --git a/install/select-hosts.html.md.erb b/install/select-hosts.html.md.erb new file mode 100644 index 0000000..c49f184 --- /dev/null +++ b/install/select-hosts.html.md.erb @@ -0,0 +1,19 @@ +--- +title: Select HAWQ Host Machines +--- + +Before you begin to install HAWQ, follow these steps to select and prepare the host machines. + +Complete this procedure for all HAWQ deployments: + +1. **Choose the host machines that will host a HAWQ segment.** Keep in mind these restrictions and requirements: + - Each host must meet the system requirements for the version of HAWQ you are installing. + - Each HAWQ segment must be co-located on a host that runs an HDFS data node. + - The HAWQ master segment and standby master segment must be hosted on separate machines. +2. **Choose the host machines that will run PXF.** Keep in mind these restrictions and requirements: + - PXF must be installed on the HDFS NameNode *and* on all HDFS data nodes. + - If you have configured Hadoop with high availability, PXF must also be installed on all HDFS nodes including all NameNode services. + - If you want to use PXF with HBase or Hive, you must first install the HBase client \(hbase-client\) and/or Hive client \(hive-client\) on each machine where you intend to install PXF. See the [HDP installation documentation](http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.3.4/index.html) for more information. +3. **Verify that required ports on all machines are unused.** By default, a HAWQ master or standby master service configuration uses port 5432. Hosts that run other PostgreSQL instances cannot be used to run a default HAWQ master or standby service configuration because the default PostgreSQL port \(5432\) conflicts with the default HAWQ port. You must either change the default port configuration of the running PostgreSQL instance or change the HAWQ master port setting during the HAWQ service installation to avoid port conflicts. + + **Note:** The Ambari server node uses PostgreSQL as the default metadata database. The Hive Metastore uses MySQL as the default metadata database. \ No newline at end of file http://git-wip-us.apache.org/repos/asf/incubator-hawq-docs/blob/7514e193/master_middleman/source/images/favicon.ico ---------------------------------------------------------------------- diff --git a/master_middleman/source/images/favicon.ico b/master_middleman/source/images/favicon.ico new file mode 100644 index 0000000..b2c3a0c Binary files /dev/null and b/master_middleman/source/images/favicon.ico differ http://git-wip-us.apache.org/repos/asf/incubator-hawq-docs/blob/7514e193/master_middleman/source/javascripts/book.js ---------------------------------------------------------------------- diff --git a/master_middleman/source/javascripts/book.js b/master_middleman/source/javascripts/book.js new file mode 100644 index 0000000..90879c4 --- /dev/null +++ b/master_middleman/source/javascripts/book.js @@ -0,0 +1,16 @@ +// Declare your book-specific javascript overrides in this file. +//= require 'waypoints/waypoint' +//= require 'waypoints/context' +//= require 'waypoints/group' +//= require 'waypoints/noframeworkAdapter' +//= require 'waypoints/sticky' + +window.onload = function() { + Bookbinder.boot(); + var sticky = new Waypoint.Sticky({ + element: document.querySelector('#js-to-top'), + wrapper: '<div class="sticky-wrapper" />', + stuckClass: 'sticky', + offset: 100 + }); +} http://git-wip-us.apache.org/repos/asf/incubator-hawq-docs/blob/7514e193/master_middleman/source/javascripts/waypoints/context.js ---------------------------------------------------------------------- diff --git a/master_middleman/source/javascripts/waypoints/context.js b/master_middleman/source/javascripts/waypoints/context.js new file mode 100644 index 0000000..5e3551b --- /dev/null +++ b/master_middleman/source/javascripts/waypoints/context.js @@ -0,0 +1,300 @@ +(function() { + 'use strict' + + function requestAnimationFrameShim(callback) { + window.setTimeout(callback, 1000 / 60) + } + + var keyCounter = 0 + var contexts = {} + var Waypoint = window.Waypoint + var oldWindowLoad = window.onload + + /* http://imakewebthings.com/waypoints/api/context */ + function Context(element) { + this.element = element + this.Adapter = Waypoint.Adapter + this.adapter = new this.Adapter(element) + this.key = 'waypoint-context-' + keyCounter + this.didScroll = false + this.didResize = false + this.oldScroll = { + x: this.adapter.scrollLeft(), + y: this.adapter.scrollTop() + } + this.waypoints = { + vertical: {}, + horizontal: {} + } + + element.waypointContextKey = this.key + contexts[element.waypointContextKey] = this + keyCounter += 1 + + this.createThrottledScrollHandler() + this.createThrottledResizeHandler() + } + + /* Private */ + Context.prototype.add = function(waypoint) { + var axis = waypoint.options.horizontal ? 'horizontal' : 'vertical' + this.waypoints[axis][waypoint.key] = waypoint + this.refresh() + } + + /* Private */ + Context.prototype.checkEmpty = function() { + var horizontalEmpty = this.Adapter.isEmptyObject(this.waypoints.horizontal) + var verticalEmpty = this.Adapter.isEmptyObject(this.waypoints.vertical) + if (horizontalEmpty && verticalEmpty) { + this.adapter.off('.waypoints') + delete contexts[this.key] + } + } + + /* Private */ + Context.prototype.createThrottledResizeHandler = function() { + var self = this + + function resizeHandler() { + self.handleResize() + self.didResize = false + } + + this.adapter.on('resize.waypoints', function() { + if (!self.didResize) { + self.didResize = true + Waypoint.requestAnimationFrame(resizeHandler) + } + }) + } + + /* Private */ + Context.prototype.createThrottledScrollHandler = function() { + var self = this + function scrollHandler() { + self.handleScroll() + self.didScroll = false + } + + this.adapter.on('scroll.waypoints', function() { + if (!self.didScroll || Waypoint.isTouch) { + self.didScroll = true + Waypoint.requestAnimationFrame(scrollHandler) + } + }) + } + + /* Private */ + Context.prototype.handleResize = function() { + Waypoint.Context.refreshAll() + } + + /* Private */ + Context.prototype.handleScroll = function() { + var triggeredGroups = {} + var axes = { + horizontal: { + newScroll: this.adapter.scrollLeft(), + oldScroll: this.oldScroll.x, + forward: 'right', + backward: 'left' + }, + vertical: { + newScroll: this.adapter.scrollTop(), + oldScroll: this.oldScroll.y, + forward: 'down', + backward: 'up' + } + } + + for (var axisKey in axes) { + var axis = axes[axisKey] + var isForward = axis.newScroll > axis.oldScroll + var direction = isForward ? axis.forward : axis.backward + + for (var waypointKey in this.waypoints[axisKey]) { + var waypoint = this.waypoints[axisKey][waypointKey] + var wasBeforeTriggerPoint = axis.oldScroll < waypoint.triggerPoint + var nowAfterTriggerPoint = axis.newScroll >= waypoint.triggerPoint + var crossedForward = wasBeforeTriggerPoint && nowAfterTriggerPoint + var crossedBackward = !wasBeforeTriggerPoint && !nowAfterTriggerPoint + if (crossedForward || crossedBackward) { + waypoint.queueTrigger(direction) + triggeredGroups[waypoint.group.id] = waypoint.group + } + } + } + + for (var groupKey in triggeredGroups) { + triggeredGroups[groupKey].flushTriggers() + } + + this.oldScroll = { + x: axes.horizontal.newScroll, + y: axes.vertical.newScroll + } + } + + /* Private */ + Context.prototype.innerHeight = function() { + /*eslint-disable eqeqeq */ + if (this.element == this.element.window) { + return Waypoint.viewportHeight() + } + /*eslint-enable eqeqeq */ + return this.adapter.innerHeight() + } + + /* Private */ + Context.prototype.remove = function(waypoint) { + delete this.waypoints[waypoint.axis][waypoint.key] + this.checkEmpty() + } + + /* Private */ + Context.prototype.innerWidth = function() { + /*eslint-disable eqeqeq */ + if (this.element == this.element.window) { + return Waypoint.viewportWidth() + } + /*eslint-enable eqeqeq */ + return this.adapter.innerWidth() + } + + /* Public */ + /* http://imakewebthings.com/waypoints/api/context-destroy */ + Context.prototype.destroy = function() { + var allWaypoints = [] + for (var axis in this.waypoints) { + for (var waypointKey in this.waypoints[axis]) { + allWaypoints.push(this.waypoints[axis][waypointKey]) + } + } + for (var i = 0, end = allWaypoints.length; i < end; i++) { + allWaypoints[i].destroy() + } + } + + /* Public */ + /* http://imakewebthings.com/waypoints/api/context-refresh */ + Context.prototype.refresh = function() { + /*eslint-disable eqeqeq */ + var isWindow = this.element == this.element.window + /*eslint-enable eqeqeq */ + var contextOffset = isWindow ? undefined : this.adapter.offset() + var triggeredGroups = {} + var axes + + this.handleScroll() + axes = { + horizontal: { + contextOffset: isWindow ? 0 : contextOffset.left, + contextScroll: isWindow ? 0 : this.oldScroll.x, + contextDimension: this.innerWidth(), + oldScroll: this.oldScroll.x, + forward: 'right', + backward: 'left', + offsetProp: 'left' + }, + vertical: { + contextOffset: isWindow ? 0 : contextOffset.top, + contextScroll: isWindow ? 0 : this.oldScroll.y, + contextDimension: this.innerHeight(), + oldScroll: this.oldScroll.y, + forward: 'down', + backward: 'up', + offsetProp: 'top' + } + } + + for (var axisKey in axes) { + var axis = axes[axisKey] + for (var waypointKey in this.waypoints[axisKey]) { + var waypoint = this.waypoints[axisKey][waypointKey] + var adjustment = waypoint.options.offset + var oldTriggerPoint = waypoint.triggerPoint + var elementOffset = 0 + var freshWaypoint = oldTriggerPoint == null + var contextModifier, wasBeforeScroll, nowAfterScroll + var triggeredBackward, triggeredForward + + if (waypoint.element !== waypoint.element.window) { + elementOffset = waypoint.adapter.offset()[axis.offsetProp] + } + + if (typeof adjustment === 'function') { + adjustment = adjustment.apply(waypoint) + } + else if (typeof adjustment === 'string') { + adjustment = parseFloat(adjustment) + if (waypoint.options.offset.indexOf('%') > - 1) { + adjustment = Math.ceil(axis.contextDimension * adjustment / 100) + } + } + + contextModifier = axis.contextScroll - axis.contextOffset + waypoint.triggerPoint = elementOffset + contextModifier - adjustment + wasBeforeScroll = oldTriggerPoint < axis.oldScroll + nowAfterScroll = waypoint.triggerPoint >= axis.oldScroll + triggeredBackward = wasBeforeScroll && nowAfterScroll + triggeredForward = !wasBeforeScroll && !nowAfterScroll + + if (!freshWaypoint && triggeredBackward) { + waypoint.queueTrigger(axis.backward) + triggeredGroups[waypoint.group.id] = waypoint.group + } + else if (!freshWaypoint && triggeredForward) { + waypoint.queueTrigger(axis.forward) + triggeredGroups[waypoint.group.id] = waypoint.group + } + else if (freshWaypoint && axis.oldScroll >= waypoint.triggerPoint) { + waypoint.queueTrigger(axis.forward) + triggeredGroups[waypoint.group.id] = waypoint.group + } + } + } + + Waypoint.requestAnimationFrame(function() { + for (var groupKey in triggeredGroups) { + triggeredGroups[groupKey].flushTriggers() + } + }) + + return this + } + + /* Private */ + Context.findOrCreateByElement = function(element) { + return Context.findByElement(element) || new Context(element) + } + + /* Private */ + Context.refreshAll = function() { + for (var contextId in contexts) { + contexts[contextId].refresh() + } + } + + /* Public */ + /* http://imakewebthings.com/waypoints/api/context-find-by-element */ + Context.findByElement = function(element) { + return contexts[element.waypointContextKey] + } + + window.onload = function() { + if (oldWindowLoad) { + oldWindowLoad() + } + Context.refreshAll() + } + + Waypoint.requestAnimationFrame = function(callback) { + var requestFn = window.requestAnimationFrame || + window.mozRequestAnimationFrame || + window.webkitRequestAnimationFrame || + requestAnimationFrameShim + requestFn.call(window, callback) + } + Waypoint.Context = Context +}()) http://git-wip-us.apache.org/repos/asf/incubator-hawq-docs/blob/7514e193/master_middleman/source/javascripts/waypoints/group.js ---------------------------------------------------------------------- diff --git a/master_middleman/source/javascripts/waypoints/group.js b/master_middleman/source/javascripts/waypoints/group.js new file mode 100644 index 0000000..57c3038 --- /dev/null +++ b/master_middleman/source/javascripts/waypoints/group.js @@ -0,0 +1,105 @@ +(function() { + 'use strict' + + function byTriggerPoint(a, b) { + return a.triggerPoint - b.triggerPoint + } + + function byReverseTriggerPoint(a, b) { + return b.triggerPoint - a.triggerPoint + } + + var groups = { + vertical: {}, + horizontal: {} + } + var Waypoint = window.Waypoint + + /* http://imakewebthings.com/waypoints/api/group */ + function Group(options) { + this.name = options.name + this.axis = options.axis + this.id = this.name + '-' + this.axis + this.waypoints = [] + this.clearTriggerQueues() + groups[this.axis][this.name] = this + } + + /* Private */ + Group.prototype.add = function(waypoint) { + this.waypoints.push(waypoint) + } + + /* Private */ + Group.prototype.clearTriggerQueues = function() { + this.triggerQueues = { + up: [], + down: [], + left: [], + right: [] + } + } + + /* Private */ + Group.prototype.flushTriggers = function() { + for (var direction in this.triggerQueues) { + var waypoints = this.triggerQueues[direction] + var reverse = direction === 'up' || direction === 'left' + waypoints.sort(reverse ? byReverseTriggerPoint : byTriggerPoint) + for (var i = 0, end = waypoints.length; i < end; i += 1) { + var waypoint = waypoints[i] + if (waypoint.options.continuous || i === waypoints.length - 1) { + waypoint.trigger([direction]) + } + } + } + this.clearTriggerQueues() + } + + /* Private */ + Group.prototype.next = function(waypoint) { + this.waypoints.sort(byTriggerPoint) + var index = Waypoint.Adapter.inArray(waypoint, this.waypoints) + var isLast = index === this.waypoints.length - 1 + return isLast ? null : this.waypoints[index + 1] + } + + /* Private */ + Group.prototype.previous = function(waypoint) { + this.waypoints.sort(byTriggerPoint) + var index = Waypoint.Adapter.inArray(waypoint, this.waypoints) + return index ? this.waypoints[index - 1] : null + } + + /* Private */ + Group.prototype.queueTrigger = function(waypoint, direction) { + this.triggerQueues[direction].push(waypoint) + } + + /* Private */ + Group.prototype.remove = function(waypoint) { + var index = Waypoint.Adapter.inArray(waypoint, this.waypoints) + if (index > -1) { + this.waypoints.splice(index, 1) + } + } + + /* Public */ + /* http://imakewebthings.com/waypoints/api/first */ + Group.prototype.first = function() { + return this.waypoints[0] + } + + /* Public */ + /* http://imakewebthings.com/waypoints/api/last */ + Group.prototype.last = function() { + return this.waypoints[this.waypoints.length - 1] + } + + /* Private */ + Group.findOrCreate = function(options) { + return groups[options.axis][options.name] || new Group(options) + } + + Waypoint.Group = Group +}()) http://git-wip-us.apache.org/repos/asf/incubator-hawq-docs/blob/7514e193/master_middleman/source/javascripts/waypoints/noframeworkAdapter.js ---------------------------------------------------------------------- diff --git a/master_middleman/source/javascripts/waypoints/noframeworkAdapter.js b/master_middleman/source/javascripts/waypoints/noframeworkAdapter.js new file mode 100644 index 0000000..99abcb5 --- /dev/null +++ b/master_middleman/source/javascripts/waypoints/noframeworkAdapter.js @@ -0,0 +1,213 @@ +(function() { + 'use strict' + + var Waypoint = window.Waypoint + + function isWindow(element) { + return element === element.window + } + + function getWindow(element) { + if (isWindow(element)) { + return element + } + return element.defaultView + } + + function classNameRegExp(className) { + return new RegExp("\\b" + className + "\\b"); + } + + function NoFrameworkAdapter(element) { + this.element = element + this.handlers = {} + } + + NoFrameworkAdapter.prototype.innerHeight = function() { + var isWin = isWindow(this.element) + return isWin ? this.element.innerHeight : this.element.clientHeight + } + + NoFrameworkAdapter.prototype.innerWidth = function() { + var isWin = isWindow(this.element) + return isWin ? this.element.innerWidth : this.element.clientWidth + } + + NoFrameworkAdapter.prototype.off = function(event, handler) { + function removeListeners(element, listeners, handler) { + for (var i = 0, end = listeners.length - 1; i < end; i++) { + var listener = listeners[i] + if (!handler || handler === listener) { + element.removeEventListener(listener) + } + } + } + + var eventParts = event.split('.') + var eventType = eventParts[0] + var namespace = eventParts[1] + var element = this.element + + if (namespace && this.handlers[namespace] && eventType) { + removeListeners(element, this.handlers[namespace][eventType], handler) + this.handlers[namespace][eventType] = [] + } + else if (eventType) { + for (var ns in this.handlers) { + removeListeners(element, this.handlers[ns][eventType] || [], handler) + this.handlers[ns][eventType] = [] + } + } + else if (namespace && this.handlers[namespace]) { + for (var type in this.handlers[namespace]) { + removeListeners(element, this.handlers[namespace][type], handler) + } + this.handlers[namespace] = {} + } + } + + /* Adapted from jQuery 1.x offset() */ + NoFrameworkAdapter.prototype.offset = function() { + if (!this.element.ownerDocument) { + return null + } + + var documentElement = this.element.ownerDocument.documentElement + var win = getWindow(this.element.ownerDocument) + var rect = { + top: 0, + left: 0 + } + + if (this.element.getBoundingClientRect) { + rect = this.element.getBoundingClientRect() + } + + return { + top: rect.top + win.pageYOffset - documentElement.clientTop, + left: rect.left + win.pageXOffset - documentElement.clientLeft + } + } + + NoFrameworkAdapter.prototype.on = function(event, handler) { + var eventParts = event.split('.') + var eventType = eventParts[0] + var namespace = eventParts[1] || '__default' + var nsHandlers = this.handlers[namespace] = this.handlers[namespace] || {} + var nsTypeList = nsHandlers[eventType] = nsHandlers[eventType] || [] + + nsTypeList.push(handler) + this.element.addEventListener(eventType, handler) + } + + NoFrameworkAdapter.prototype.outerHeight = function(includeMargin) { + var height = this.innerHeight() + var computedStyle + + if (includeMargin && !isWindow(this.element)) { + computedStyle = window.getComputedStyle(this.element) + height += parseInt(computedStyle.marginTop, 10) + height += parseInt(computedStyle.marginBottom, 10) + } + + return height + } + + NoFrameworkAdapter.prototype.outerWidth = function(includeMargin) { + var width = this.innerWidth() + var computedStyle + + if (includeMargin && !isWindow(this.element)) { + computedStyle = window.getComputedStyle(this.element) + width += parseInt(computedStyle.marginLeft, 10) + width += parseInt(computedStyle.marginRight, 10) + } + + return width + } + + NoFrameworkAdapter.prototype.scrollLeft = function() { + var win = getWindow(this.element) + return win ? win.pageXOffset : this.element.scrollLeft + } + + NoFrameworkAdapter.prototype.scrollTop = function() { + var win = getWindow(this.element) + return win ? win.pageYOffset : this.element.scrollTop + } + + NoFrameworkAdapter.prototype.height = function(newHeight) { + this.element.style.height = newHeight; + } + + NoFrameworkAdapter.prototype.removeClass = function(className) { + this.element.className = this.element.className.replace(classNameRegExp(className), ''); + } + + NoFrameworkAdapter.prototype.toggleClass = function(className, addClass) { + var check = classNameRegExp(className); + if (check.test(this.element.className)) { + if (!addClass) { + this.removeClass(className); + } + } else { + this.element.className += ' ' + className; + } + } + + NoFrameworkAdapter.prototype.parent = function() { + return new NoFrameworkAdapter(this.element.parentNode); + } + + NoFrameworkAdapter.prototype.wrap = function(wrapper) { + this.element.insertAdjacentHTML('beforebegin', wrapper) + var wrapperNode = this.element.previousSibling + this.element.parentNode.removeChild(this.element) + wrapperNode.appendChild(this.element) + } + + NoFrameworkAdapter.extend = function() { + var args = Array.prototype.slice.call(arguments) + + function merge(target, obj) { + if (typeof target === 'object' && typeof obj === 'object') { + for (var key in obj) { + if (obj.hasOwnProperty(key)) { + target[key] = obj[key] + } + } + } + + return target + } + + for (var i = 1, end = args.length; i < end; i++) { + merge(args[0], args[i]) + } + return args[0] + } + + NoFrameworkAdapter.inArray = function(element, array, i) { + return array == null ? -1 : array.indexOf(element, i) + } + + NoFrameworkAdapter.isEmptyObject = function(obj) { + /* eslint no-unused-vars: 0 */ + for (var name in obj) { + return false + } + return true + } + + NoFrameworkAdapter.proxy = function(func, obj) { + return function() { + return func.apply(obj, arguments); + } + } + + Waypoint.adapters.push({ + name: 'noframework', + Adapter: NoFrameworkAdapter + }) + Waypoint.Adapter = NoFrameworkAdapter +}()) http://git-wip-us.apache.org/repos/asf/incubator-hawq-docs/blob/7514e193/master_middleman/source/javascripts/waypoints/sticky.js ---------------------------------------------------------------------- diff --git a/master_middleman/source/javascripts/waypoints/sticky.js b/master_middleman/source/javascripts/waypoints/sticky.js new file mode 100644 index 0000000..569fcdb --- /dev/null +++ b/master_middleman/source/javascripts/waypoints/sticky.js @@ -0,0 +1,63 @@ +(function() { + 'use strict' + + var Waypoint = window.Waypoint; + var adapter = Waypoint.Adapter; + + /* http://imakewebthings.com/waypoints/shortcuts/sticky-elements */ + function Sticky(options) { + this.options = adapter.extend({}, Waypoint.defaults, Sticky.defaults, options) + this.element = this.options.element + this.$element = new adapter(this.element) + this.createWrapper() + this.createWaypoint() + } + + /* Private */ + Sticky.prototype.createWaypoint = function() { + var originalHandler = this.options.handler + + this.waypoint = new Waypoint(adapter.extend({}, this.options, { + element: this.wrapper, + handler: adapter.proxy(function(direction) { + var shouldBeStuck = this.options.direction.indexOf(direction) > -1 + var wrapperHeight = shouldBeStuck ? this.$element.outerHeight(true) : '' + + this.$wrapper.height(wrapperHeight) + this.$element.toggleClass(this.options.stuckClass, shouldBeStuck) + + if (originalHandler) { + originalHandler.call(this, direction) + } + }, this) + })) + } + + /* Private */ + Sticky.prototype.createWrapper = function() { + if (this.options.wrapper) { + this.$element.wrap(this.options.wrapper) + } + this.$wrapper = this.$element.parent() + this.wrapper = this.$wrapper.element + } + + /* Public */ + Sticky.prototype.destroy = function() { + if (this.$element.parent().element === this.wrapper) { + this.waypoint.destroy() + this.$element.removeClass(this.options.stuckClass) + if (this.options.wrapper) { + this.$element.unwrap() + } + } + } + + Sticky.defaults = { + wrapper: '<div class="sticky-wrapper" />', + stuckClass: 'stuck', + direction: 'down right' + } + + Waypoint.Sticky = Sticky +}()) http://git-wip-us.apache.org/repos/asf/incubator-hawq-docs/blob/7514e193/master_middleman/source/javascripts/waypoints/waypoint.js ---------------------------------------------------------------------- diff --git a/master_middleman/source/javascripts/waypoints/waypoint.js b/master_middleman/source/javascripts/waypoints/waypoint.js new file mode 100644 index 0000000..7f76f1d --- /dev/null +++ b/master_middleman/source/javascripts/waypoints/waypoint.js @@ -0,0 +1,160 @@ +(function() { + 'use strict' + + var keyCounter = 0 + var allWaypoints = {} + + /* http://imakewebthings.com/waypoints/api/waypoint */ + function Waypoint(options) { + if (!options) { + throw new Error('No options passed to Waypoint constructor') + } + if (!options.element) { + throw new Error('No element option passed to Waypoint constructor') + } + if (!options.handler) { + throw new Error('No handler option passed to Waypoint constructor') + } + + this.key = 'waypoint-' + keyCounter + this.options = Waypoint.Adapter.extend({}, Waypoint.defaults, options) + this.element = this.options.element + this.adapter = new Waypoint.Adapter(this.element) + this.callback = options.handler + this.axis = this.options.horizontal ? 'horizontal' : 'vertical' + this.enabled = this.options.enabled + this.triggerPoint = null + this.group = Waypoint.Group.findOrCreate({ + name: this.options.group, + axis: this.axis + }) + this.context = Waypoint.Context.findOrCreateByElement(this.options.context) + + if (Waypoint.offsetAliases[this.options.offset]) { + this.options.offset = Waypoint.offsetAliases[this.options.offset] + } + this.group.add(this) + this.context.add(this) + allWaypoints[this.key] = this + keyCounter += 1 + } + + /* Private */ + Waypoint.prototype.queueTrigger = function(direction) { + this.group.queueTrigger(this, direction) + } + + /* Private */ + Waypoint.prototype.trigger = function(args) { + if (!this.enabled) { + return + } + if (this.callback) { + this.callback.apply(this, args) + } + } + + /* Public */ + /* http://imakewebthings.com/waypoints/api/destroy */ + Waypoint.prototype.destroy = function() { + this.context.remove(this) + this.group.remove(this) + delete allWaypoints[this.key] + } + + /* Public */ + /* http://imakewebthings.com/waypoints/api/disable */ + Waypoint.prototype.disable = function() { + this.enabled = false + return this + } + + /* Public */ + /* http://imakewebthings.com/waypoints/api/enable */ + Waypoint.prototype.enable = function() { + this.context.refresh() + this.enabled = true + return this + } + + /* Public */ + /* http://imakewebthings.com/waypoints/api/next */ + Waypoint.prototype.next = function() { + return this.group.next(this) + } + + /* Public */ + /* http://imakewebthings.com/waypoints/api/previous */ + Waypoint.prototype.previous = function() { + return this.group.previous(this) + } + + /* Private */ + Waypoint.invokeAll = function(method) { + var allWaypointsArray = [] + for (var waypointKey in allWaypoints) { + allWaypointsArray.push(allWaypoints[waypointKey]) + } + for (var i = 0, end = allWaypointsArray.length; i < end; i++) { + allWaypointsArray[i][method]() + } + } + + /* Public */ + /* http://imakewebthings.com/waypoints/api/destroy-all */ + Waypoint.destroyAll = function() { + Waypoint.invokeAll('destroy') + } + + /* Public */ + /* http://imakewebthings.com/waypoints/api/disable-all */ + Waypoint.disableAll = function() { + Waypoint.invokeAll('disable') + } + + /* Public */ + /* http://imakewebthings.com/waypoints/api/enable-all */ + Waypoint.enableAll = function() { + Waypoint.invokeAll('enable') + } + + /* Public */ + /* http://imakewebthings.com/waypoints/api/refresh-all */ + Waypoint.refreshAll = function() { + Waypoint.Context.refreshAll() + } + + /* Public */ + /* http://imakewebthings.com/waypoints/api/viewport-height */ + Waypoint.viewportHeight = function() { + return window.innerHeight || document.documentElement.clientHeight + } + + /* Public */ + /* http://imakewebthings.com/waypoints/api/viewport-width */ + Waypoint.viewportWidth = function() { + return document.documentElement.clientWidth + } + + Waypoint.adapters = [] + + Waypoint.defaults = { + context: window, + continuous: true, + enabled: true, + group: 'default', + horizontal: false, + offset: 0 + } + + Waypoint.offsetAliases = { + 'bottom-in-view': function() { + return this.context.innerHeight() - this.adapter.outerHeight() + }, + 'right-in-view': function() { + return this.context.innerWidth() - this.adapter.outerWidth() + } + } + + window.Waypoint = Waypoint +}()) http://git-wip-us.apache.org/repos/asf/incubator-hawq-docs/blob/7514e193/master_middleman/source/layouts/_title.erb ---------------------------------------------------------------------- diff --git a/master_middleman/source/layouts/_title.erb b/master_middleman/source/layouts/_title.erb new file mode 100644 index 0000000..ea744d9 --- /dev/null +++ b/master_middleman/source/layouts/_title.erb @@ -0,0 +1,6 @@ +<% if current_page.data.title %> + <h1 class="title-container" <%= current_page.data.dita ? 'style="display: none;"' : '' %>> + <%= current_page.data.title %> + </h1> +<% end %> + http://git-wip-us.apache.org/repos/asf/incubator-hawq-docs/blob/7514e193/master_middleman/source/patch/dynamic_variable_interpretation.py ---------------------------------------------------------------------- diff --git a/master_middleman/source/patch/dynamic_variable_interpretation.py b/master_middleman/source/patch/dynamic_variable_interpretation.py new file mode 100644 index 0000000..66df9ff --- /dev/null +++ b/master_middleman/source/patch/dynamic_variable_interpretation.py @@ -0,0 +1,192 @@ +#!/usr/bin/env python +""" +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. + +""" + +__all__ = ["copy_tarballs_to_hdfs", ] +import os +import glob +import re +import tempfile +from resource_management.libraries.functions.default import default +from resource_management.libraries.functions.format import format +from resource_management.libraries.resources.copy_from_local import CopyFromLocal +from resource_management.libraries.resources.execute_hadoop import ExecuteHadoop +from resource_management.core.resources.system import Execute +from resource_management.core.exceptions import Fail +from resource_management.core.logger import Logger +from resource_management.core import shell + +""" +This file provides helper methods needed for the versioning of RPMs. Specifically, it does dynamic variable +interpretation to replace strings like {{ hdp_stack_version }} where the value of the +variables cannot be determined ahead of time, but rather, depends on what files are found. + +It assumes that {{ hdp_stack_version }} is constructed as ${major.minor.patch.rev}-${build_number} +E.g., 998.2.2.1.0-998 +Please note that "-${build_number}" is optional. +""" + +# These values must be the suffix of the properties in cluster-env.xml +TAR_SOURCE_SUFFIX = "_tar_source" +TAR_DESTINATION_FOLDER_SUFFIX = "_tar_destination_folder" + + +def _get_tar_source_and_dest_folder(tarball_prefix): + """ + :param tarball_prefix: Prefix of the tarball must be one of tez, hive, mr, pig + :return: Returns a tuple of (x, y) after verifying the properties + """ + component_tar_source_file = default("/configurations/cluster-env/%s%s" % (tarball_prefix.lower(), TAR_SOURCE_SUFFIX), None) + # E.g., /usr/hdp/current/hadoop-client/tez-{{ hdp_stack_version }}.tar.gz + + component_tar_destination_folder = default("/configurations/cluster-env/%s%s" % (tarball_prefix.lower(), TAR_DESTINATION_FOLDER_SUFFIX), None) + # E.g., hdfs:///hdp/apps/{{ hdp_stack_version }}/mapreduce/ + + if not component_tar_source_file or not component_tar_destination_folder: + Logger.warning("Did not find %s tar source file and destination folder properties in cluster-env.xml" % + tarball_prefix) + return None, None + + if component_tar_source_file.find("/") == -1: + Logger.warning("The tar file path %s is not valid" % str(component_tar_source_file)) + return None, None + + if not component_tar_destination_folder.endswith("/"): + component_tar_destination_folder = component_tar_destination_folder + "/" + + if not component_tar_destination_folder.startswith("hdfs://"): + return None, None + + return component_tar_source_file, component_tar_destination_folder + + +def _copy_files(source_and_dest_pairs, file_owner, group_owner, kinit_if_needed): + """ + :param source_and_dest_pairs: List of tuples (x, y), where x is the source file in the local file system, + and y is the destination file path in HDFS + :param file_owner: Owner to set for the file copied to HDFS (typically hdfs account) + :param group_owner: Owning group to set for the file copied to HDFS (typically hadoop group) + :param kinit_if_needed: kinit command if it is needed, otherwise an empty string + :return: Returns 0 if at least one file was copied and no exceptions occurred, and 1 otherwise. + + Must kinit before calling this function. + """ + import params + + return_value = 1 + if source_and_dest_pairs and len(source_and_dest_pairs) > 0: + return_value = 0 + for (source, destination) in source_and_dest_pairs: + try: + destination_dir = os.path.dirname(destination) + + params.HdfsDirectory(destination_dir, + action="create", + owner=file_owner, + mode=0555 + ) + + CopyFromLocal(source, + mode=0444, + owner=file_owner, + group=group_owner, + dest_dir=destination_dir, + kinnit_if_needed=kinit_if_needed, + hdfs_user=params.hdfs_user, + hadoop_bin_dir=params.hadoop_bin_dir, + hadoop_conf_dir=params.hadoop_conf_dir + ) + except: + return_value = 1 + return return_value + + +def copy_tarballs_to_hdfs(tarball_prefix, component_user, file_owner, group_owner): + """ + :param tarball_prefix: Prefix of the tarball must be one of tez, hive, mr, pig + :param component_user: User that will execute the Hadoop commands + :param file_owner: Owner of the files copied to HDFS (typically hdfs account) + :param group_owner: Group owner of the files copied to HDFS (typically hadoop group) + :return: Returns 0 on success, 1 if no files were copied, and in some cases may raise an exception. + + In order to call this function, params.py must have all of the following, + hdp_stack_version, kinit_path_local, security_enabled, hdfs_user, hdfs_principal_name, hdfs_user_keytab, + hadoop_bin_dir, hadoop_conf_dir, and HdfsDirectory as a partial function. + """ + import params + + if not hasattr(params, "hdp_stack_version") or params.hdp_stack_version is None: + Logger.warning("Could not find hdp_stack_version") + return 1 + + component_tar_source_file, component_tar_destination_folder = _get_tar_source_and_dest_folder(tarball_prefix) + if not component_tar_source_file or not component_tar_destination_folder: + Logger.warning("Could not retrieve properties for tarball with prefix: %s" % str(tarball_prefix)) + return 1 + + if not os.path.exists(component_tar_source_file): + Logger.warning("Could not find file: %s" % str(component_tar_source_file)) + return 1 + + # Ubuntu returns: "stdin: is not a tty", as subprocess output. + tmpfile = tempfile.NamedTemporaryFile() + with open(tmpfile.name, 'r+') as file: + get_hdp_version_cmd = '/usr/bin/hdp-select versions > %s' % tmpfile.name + code, stdoutdata = shell.call(get_hdp_version_cmd) + out = file.read() + pass + if code != 0 or out is None: + Logger.warning("Could not verify HDP version by calling '%s'. Return Code: %s, Output: %s." % + (get_hdp_version_cmd, str(code), str(out))) + return 1 + + hdp_version = out.strip() # this should include the build number + + file_name = os.path.basename(component_tar_source_file) + destination_file = os.path.join(component_tar_destination_folder, file_name) + destination_file = destination_file.replace("{{ hdp_stack_version }}", hdp_version) + + does_hdfs_file_exist_cmd = "fs -ls %s" % destination_file + + kinit_if_needed = "" + if params.security_enabled: + kinit_if_needed = format("{kinit_path_local} -kt {hdfs_user_keytab} {hdfs_principal_name};") + + if kinit_if_needed: + Execute(kinit_if_needed, + user=component_user, + path='/bin' + ) + + does_hdfs_file_exist = False + try: + ExecuteHadoop(does_hdfs_file_exist_cmd, + user=component_user, + logoutput=True, + conf_dir=params.hadoop_conf_dir, + bin_dir=params.hadoop_bin_dir + ) + does_hdfs_file_exist = True + except Fail: + pass + + if not does_hdfs_file_exist: + source_and_dest_pairs = [(component_tar_source_file, destination_file), ] + return _copy_files(source_and_dest_pairs, file_owner, group_owner, kinit_if_needed) + return 1 http://git-wip-us.apache.org/repos/asf/incubator-hawq-docs/blob/7514e193/master_middleman/source/stylesheets/book-styles.css.scss ---------------------------------------------------------------------- diff --git a/master_middleman/source/stylesheets/book-styles.css.scss b/master_middleman/source/stylesheets/book-styles.css.scss new file mode 100644 index 0000000..1236d8e --- /dev/null +++ b/master_middleman/source/stylesheets/book-styles.css.scss @@ -0,0 +1,3 @@ +* { + box-sizing: border-box; +} http://git-wip-us.apache.org/repos/asf/incubator-hawq-docs/blob/7514e193/master_middleman/source/stylesheets/partials/_book-base-values.scss ---------------------------------------------------------------------- diff --git a/master_middleman/source/stylesheets/partials/_book-base-values.scss b/master_middleman/source/stylesheets/partials/_book-base-values.scss new file mode 100644 index 0000000..e69de29 http://git-wip-us.apache.org/repos/asf/incubator-hawq-docs/blob/7514e193/master_middleman/source/stylesheets/partials/_book-vars.scss ---------------------------------------------------------------------- diff --git a/master_middleman/source/stylesheets/partials/_book-vars.scss b/master_middleman/source/stylesheets/partials/_book-vars.scss new file mode 100644 index 0000000..4245d57 --- /dev/null +++ b/master_middleman/source/stylesheets/partials/_book-vars.scss @@ -0,0 +1,19 @@ +$navy: #243640; +$blue1: #2185c5; +$blue2: #a7cae1; +$bluegray1: #4b6475; +$teal1: #03786D; +$teal2: #00a79d; + +$color-accent: $teal1; +$color-accent-bright: $teal2; + +// link colors +$color-link: $blue1; +$color-link-border: $blue2; + +$color-border-tip: $blue2; + +$color-bg-header: $navy; +$color-bg-dark: $bluegray1; +
