Markdown for Apex Application package & Application configuration package
Project: http://git-wip-us.apache.org/repos/asf/incubator-apex-core/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-apex-core/commit/c689142d Tree: http://git-wip-us.apache.org/repos/asf/incubator-apex-core/tree/c689142d Diff: http://git-wip-us.apache.org/repos/asf/incubator-apex-core/diff/c689142d Branch: refs/heads/APEXCORE-293 Commit: c689142dc0a38ea9f54fe0c016d1723f3a3cf868 Parents: bd13fbb Author: sandeshh <[email protected]> Authored: Tue Oct 27 16:34:05 2015 -0700 Committer: Thomas Weise <[email protected]> Committed: Sun Feb 28 22:46:31 2016 -0800 ---------------------------------------------------------------------- ApexPackage.md | 700 +++++++++++++++++++++++++++++++++++++++++++++++++ AppConfPackage.md | 246 +++++++++++++++++ 2 files changed, 946 insertions(+) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/incubator-apex-core/blob/c689142d/ApexPackage.md ---------------------------------------------------------------------- diff --git a/ApexPackage.md b/ApexPackage.md new file mode 100644 index 0000000..6e0729c --- /dev/null +++ b/ApexPackage.md @@ -0,0 +1,700 @@ +#Apex Application Packages + +# 1. Introduction + +An Apex Application Package is a zip file that contains all the +necessary files to launch an application in Project Apex. It is the +standard way for assembling and sharing an Apex application. + +# 2. Requirements + +You will need have the following installed: + +1. Apache Maven 3.0 or later (for assembling the App Package) +2. Apex 3.0.0 or later (for launching the App Package in your cluster) + +#3. Creating Your First Apex App Package + + +You can create an Apex Application Package using your Linux command +line, or using your favorite IDE. + +##Using Command Line + +First, change to the directory where you put your projects, and create +an Apex application project using Maven by running the following +command. Replace "com.example", "mydtapp" and "1.0-SNAPSHOT" with the +appropriate values (make sure this is all on one line): + +``` + $ mvn archetype:generate + -DarchetypeRepository=https://www.datatorrent.com/maven/content/reposito + ries/releases + -DarchetypeGroupId=com.datatorrent + -DarchetypeArtifactId=apex-app-archetype -DarchetypeVersion=3.0.0 + -DgroupId=com.example -Dpackage=com.example.mydtapp -DartifactId=mydtapp + -Dversion=1.0-SNAPSHOT +``` + +This creates a Maven project named "mydtapp". Open it with your favorite +IDE (e.g. NetBeans, Eclipse, IntelliJ IDEA). In the project, there is a +sample DAG that generates a number of tuples with a random number and +prints out "hello world" and the random number in the tuples. The code +that builds the DAG is in +src/main/java/com/example/mydtapp/Application.java, and the code that +runs the unit test for the DAG is in +src/test/java/com/example/mydtapp/ApplicationTest.java. Try it out by +running the following command: + +``` + $cd mydtapp; mvn package +``` +This builds the App Package runs the unit test of the DAG. You should +be getting test output similar to this: + +``` + ------------------------------------------------------- + + TESTS + + ------------------------------------------------------- + + Running com.example.mydtapp.ApplicationTest + + hello world: 0.8015370953286478 + + hello world: 0.9785359225545481 + + hello world: 0.6322611586644047 + + hello world: 0.8460953663451775 + + hello world: 0.5719372906929072 + + hello world: 0.6361174312337172 + + hello world: 0.14873007534816318 + + hello world: 0.8866986277418261 + + hello world: 0.6346526809866057 + hello world: 0.48587295703904465 + + hello world: 0.6436832429676687 + + ... + + Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 11.863 + sec + + Results : + + Tests run: 1, Failures: 0, Errors: 0, Skipped: 0 +``` + +The "mvn package" command creates the App Package file in target +directory as target/mydtapp-1.0-SNAPSHOT.apa. You will be able to use +that App Package file to launch this sample application in your actual +Apex installation. + +## Using IDE + +Alternatively, you can do the above steps all within your IDE. For +example, in NetBeans, select File -\> New Project. Then choose âMavenâ +and âProject from Archetypeâ in the dialog box, as shown. + + + +Then fill the Group ID, Artifact ID, Version and Repository entries as shown below. + + + +Group ID: com.datatorrent +Artifact ID: apex-app-archetype +Version: 3.0.0 (or any later version) + +Repository: +[https://www.datatorrent.com/maven/content/repositories/releases](https://www.datatorrent.com/maven/content/repositories/releases) + +Press Next and fill out the rest of the required information. For +example: + + + +Click Finish, and now you have created your own DataTorrent App Package +project, with a default unit test. You can run the unit test, make code +changes or make dependency changes within your IDE. The procedure for +other IDEs, like Eclipse or IntelliJ, is similar. + +# 4. Writing Your Own App Package + + +Please refer to the [Application Developer Guide](https://www.datatorrent.com/docs/guides/ApplicationDeveloperGuide.html) on the basics on how to write a DataTorrent application. In your AppPackage project, you can add custom operators (refer to [Operator Developer Guide](https://www.google.com/url?q=https://www.datatorrent.com/docs/guides/OperatorDeveloperGuide.html&sa=D&usg=AFQjCNHMFBKWyQlg6VtvBzuqlQSM78rTFw)), project dependencies, default and required configuration properties, pre-set configurations and other metadata. + +## Adding (and removing) project dependencies + +Under the project, you can add project dependencies in pom.xml, or do it +through your IDE. Hereâs the section that describes the dependencies in +the default pom.xml: +``` + <dependencies> + <!-- add your dependencies here --> + <dependency> + <groupId>com.datatorrent</groupId> + <artifactId>malhar-library</artifactId> + <version>${datatorrent.version}</version> + <!-- + If you know your application do not need the transitive dependencies that are pulled in by malhar-library, + Uncomment the following to reduce the size of your app package. + --> + <!-- + <exclusions> + <exclusion> + <groupId>*</groupId> + <artifactId>*</artifactId> + </exclusion> + </exclusions> + --> + </dependency> + <dependency> + <groupId>com.datatorrent</groupId> + <artifactId>dt-engine</artifactId> + <version>${datatorrent.version}</version> + <scope>provided</scope> + </dependency> + <dependency> + <groupId>junit</groupId> + <artifactId>junit</artifactId> + <version>4.10</version> + <scope>test</scope> + </dependency> + </dependencies> +``` + +By default, as shown above, the default dependencies include +malhar-library in compile scope, dt-engine in provided scope, and junit +in test scope. Do not remove these three dependencies since they are +necessary for any Apex application. You can, however, exclude +transitive dependencies from malhar-library to reduce the size of your +App Package, provided that none of the operators in malhar-library that +need the transitive dependencies will be used in your application. + +In the sample application, it is safe to remove the transitive +dependencies from malhar-library, by uncommenting the "exclusions" +section. It will reduce the size of the sample App Package from 8MB to +700KB. + +Note that if we exclude \*, in some versions of Maven, you may get +warnings similar to the following: + +``` + + [WARNING] 'dependencies.dependency.exclusions.exclusion.groupId' for + com.datatorrent:malhar-library:jar with value '*' does not match a + valid id pattern. + + [WARNING] + [WARNING] It is highly recommended to fix these problems because they + threaten the stability of your build. + [WARNING] + [WARNING] For this reason, future Maven versions might no longer support + building such malformed projects. + [WARNING] + +``` +This is a bug in early versions of Maven 3. The dependency exclusion is +still valid and it is safe to ignore these warnings. + +## Application Configuration + +A configuration file can be used to configure an application. Different +kinds of configuration parameters can be specified. They are application +attributes, operator attributes and properties, port attributes, stream +properties and application specific properties. They are all specified +as name value pairs, in XML format, like the following. + +``` +<?xml version="1.0"?> +<configuration> + <property> + <name>some_name_1</name> + <value>some_default_value</value> + </property> + <property> + <name>some_name_2</name> + <value>some_default_value</value> + </property> +</configuration> +``` + +## Application attributes {.c13 .c14 .c30} + +Application attributes are used to specify the platform behavior for the +application. They can be specified using the parameter +```dt.attr.<attribute>```. The prefix âdtâ is a constant, âattrâ is a +constant denoting an attribute is being specified and ```<attribute>``` +specifies the name of the attribute. Below is an example snippet setting +the streaming windows size of the application to be 1000 milliseconds. + +``` + <property> + <name>dt.attr.STREAMING_WINDOW_SIZE_MILLIS</name> + <value>1000</value> + </property> +``` + +The name tag specifies the attribute and value tag specifies the +attribute value. The name of the attribute is a JAVA constant name +identifying the attribute. The constants are defined in +com.datatorrent.api.Context.DAGContext and the different attributes can +be specified in the format described above. + +## Operator attributes + +Operator attributes are used to specify the platform behavior for the +operator. They can be specified using the parameter +```dt.operator.<operator-name>.attr.<attribute>```. The prefix âdtâ is a +constant, âoperatorâ is a constant denoting that an operator is being +specified, ```<operator-name>``` denotes the name of the operator, âattrâ is +the constant denoting that an attribute is being specified and +```<attribute>``` is the name of the attribute. The operator name is the +same name that is specified when the operator is added to the DAG using +the addOperator method. An example illustrating the specification is +shown below. It specifies the number of streaming windows for one +application window of an operator named âinputâ to be 10 + +``` +<property> + <name>dt.operator.input.attr.APPLICATION_WINDOW_COUNT</name> + <value>10</value> +</property> +``` + +The name tag specifies the attribute and value tag specifies the +attribute value. The name of the attribute is a JAVA constant name +identifying the attribute. The constants are defined in +com.datatorrent.api.Context.OperatorContext and the different attributes +can be specified in the format described above. + +## Operator properties + +Operators can be configured using operator specific properties. The +properties can be specified using the parameter +```dt.operator.<operator-name>.prop.<property-name>```. The difference +between this and the operator attribute specification described above is +that the keyword âpropâ is used to denote that it is a property and +```<property-name>``` specifies the property name. An example illustrating +this is specified below. It specifies the property âhostnameâ of the +redis server for a âredisâ output operator. + +``` + <property> + <name>dt.operator.redis.prop.host</name> + <value>127.0.0.1</value> + </property> +``` + +The name tag specifies the property and the value specifies the property +value. The property name is converted to a setter method which is called +on the actual operator. The method name is composed by appending the +word âsetâ and the property name with the first character of the name +capitalized. In the above example the setter method would become +setHost. The method is called using JAVA reflection and the property +value is passed as an argument. In the above example the method setHost +will be called on the âredisâ operator with â127.0.0.1â as the argument. + +## Port attributes + +Port attributes are used to specify the platform behavior for input and +output ports. They can be specified using the parameter +```dt.operator.<operator-name>.inputport.<port-name>.attr.<attribute>``` +for input port and +```dt.operator.<operator-name>.outputport.<port-name>.attr.<attribute>``` +for output port. The keyword âinputportâ is used to denote an input port +and âoutputportâ to denote an output port. The rest of the specification +follows the conventions described in other specifications above. An +example illustrating this is specified below. It specifies the queue +capacity for an input port named âinputâ of an operator named ârangeâ to +be 4k. + +``` +<property> +<name>dt.operator.range.inputport.input.attr.QUEUE_CAPACITY</name> + <value>4000</value> +</property> +``` + +The name tag specifies the attribute and value tag specifies the +attribute value. The name of the attribute is a JAVA constant name +identifying the attribute. The constants are defined in +com.datatorrent.api.Context.PortContext and the different attributes can +be specified in the format described above. + +The attributes for an output port can also be specified in a similar way +as described above with a change that keyword âoutputportâ is used +instead of âintputportâ. A generic keyword âportâ can be used to specify +either an input or an output port. It is useful in the wildcard +specification described below. + +## Stream properties + +Streams can be configured using stream properties. The properties can be +specified using the parameter +```dt.stream.<stream-name>.prop.<property-name>``` The constant âstreamâ +specifies that it is a stream, ```<stream-name>``` specifies the name of the +stream and ```<property-name>``` the name of the property. The name of the +stream is the same name that is passed when the stream is added to the +DAG using the addStream method. An example illustrating the +specification is shown below. It sets the locality of the stream named +âstream1â to container local indicating that the operators the stream is +connecting be run in the same container. + +``` + <property> + <name>dt.stream.stream1.prop.locality</name> + <value>CONTAINER_LOCAL</value> + </property> +``` + +The property name is converted into a set method on the stream in the +same way as described in operator properties section above. In this case +the method would be setLocality and it will be called in the stream +âstream1â with the value as the argument. + +Along with the above system defined parameters, the applications can +define their own specific parameters they can be specified in the +configuration file. The only condition is that the names of these +parameters donât conflict with the system defined parameters or similar +application parameters defined by other applications. To this end, it is +recommended that the application parameters have the format +```<full-application-class-name>.<param-name>.``` The +full-application-class-name is the full JAVA class name of the +application including the package path and param-name is the name of the +parameter within the application. The application will still have to +still read the parameter in using the configuration API of the +configuration object that is passed in populateDAG. + +## Wildcards + +Wildcards and regular expressions can be used in place of names to +specify a group for applications, operators, ports or streams. For +example, to specify an attribute for all ports of an operator it can be +done as follows +``` +<property> + <name>dt.operator.range.port.*.attr.QUEUE_CAPACITY</name> + <value>4000</value> +</property> +``` + +The wildcard â\*â was used instead of the name of the port. Wildcard can +also be used for operator name, stream name or application name. Regular +expressions can also be used for names to specify attributes or +properties for a specific set. + +##Adding configuration properties + +It is common for applications to require configuration parameters to +run. For example, the address and port of the database, the location of +a file for ingestion, etc. You can specify them in +src/main/resources/META-INF/properties.xml under the App Package +project. The properties.xml may look like: + +``` +<?xml version="1.0"?> +<configuration> + <property> + <name>some_name_1</name> + </property> + <property> + <name>some_name_2</name> + <value>some_default_value</value> + </property> +</configuration> +``` + +The name of an application-specific property takes the form of: + +```dt.operator.{opName}.prop.{propName}``` + +The first represents the property with name propName of operator opName. + Or you can set the application name at run time by setting this +property: + + dt.attr.APPLICATION_NAME + +There are also other properties that can be set. For details on +properties, refer to the [Operation and Installation Guide](https://www.datatorrent.com/docs/guides/OperationandInstallationGuide.html). + +In this example, property some_name_1 is a required property which +must be set at launch time, or it must be set by a pre-set configuration +(see next section). Property some\_name\_2 is a property that is +assigned with value some\_default\_value unless it is overridden at +launch time. + +##Adding pre-set configurations + + +At build time, you can add pre-set configurations to the App Package by +adding configuration XML files under ```src/site/conf/<conf>.xml```in your +project. You can then specify which configuration to use at launch +time. The configuration XML is of the same format of the properties.xml +file. + +##Application-specific properties file + +You can also specify properties.xml per application in the application +package. Just create a file with the name properties-{appName}.xml and +it will be picked up when you launch the application with the specified +name within the application package. In short: + + properties.xml: Properties that are global to the Configuration +Package + + properties-{appName}.xml: Properties that are specific when launching +an application with the specified appName. + +## Properties source precedence + +If properties with the same key appear in multiple sources (e.g. from +app package default configuration as META-INF/properties.xml, from app +package configuration in the conf directory, from launch time defines, +etc), the precedence of sources, from highest to lowest, is as follows: + +1. Launch time defines (using -D option in CLI, or the POST payload + with the Gateway REST APIâs launch call) +2. Launch time specified configuration file in file system (using -conf + option in CLI) +3. Launch time specified package configuration (using -apconf option in + CLI or the conf={confname} with Gateway REST APIâs launch call) +4. Configuration from \$HOME/.dt/dt-site.xml +5. Application defaults within the package as + META-INF/properties-{appname}.xml +6. Package defaults as META-INF/properties.xml +7. dt-site.xml in local DT installation +8. dt-site.xml stored in HDFS + +## Other meta-data + +In a Apex App Package project, the pom.xml file contains a +section that looks like: + +``` +<properties> <datatorrent.version>3.0.0</datatorrent.version> <datatorrent.apppackage.classpath\>lib*.jar</datatorrent.apppackage.classpath> +</properties> +``` +datatorrent.version is the DataTorrent RTS version that are to be used +with this Application Package. + +datatorrent.apppackage.classpath is the classpath that is used when +launching the application in the Application Package. The default is +lib/\*.jar, where lib is where all the dependency jars are kept within +the Application Package. One reason to change this field is when your +Application Package needs the classpath in a specific order. + +## Logging configuration + +Just like other Java projects, you can change the logging configuration +by having your log4j.properties under src/main/resources. For example, +if you have the following in src/main/resources/log4j.properties: +``` + log4j.rootLogger=WARN,CONSOLE + log4j.appender.CONSOLE=org.apache.log4j.ConsoleAppender + log4j.appender.CONSOLE.layout=org.apache.log4j.PatternLayout + log4j.appender.CONSOLE.layout.ConversionPattern=%d{ISO8601} [%t] %-5p + %c{2} %M - %m%n +``` + +The root loggerâs level is set to WARN and the output is set to the console (stdout). + +Note that by default from project created from the maven archetype, +there is already a log4j.properties file under src/test/resources and +that file is only used for the unit test. + +#5. Zip Structure of Application Package + + +DataTorrent Application Package files are zip files. You can examine the content of any Application Package by using unzip -t on your Linux command line. + +There are four top level directories in an Application Package: + +1. "app" contains the jar files of the DAG code and any custom operators. +2. "lib" contains all dependency jars +3. "conf" contains all the pre-set configuration XML files. +4. "META-INF" contains the MANIFEST.MF file and the properties.xml file. +5. âresourcesâ contains other files that are to be served by the Gateway on behalf of the app package. + + +#6. Managing Application Packages Through DT Gateway + +The DT Gateway provides storing and retrieving Application Packages to +and from your distributed file system, e.g. HDFS. + +##Storing an Application Package + +You can store your Application Packages through DT Gateway using this +REST call: + +``` + POST /ws/v2/appPackages +``` + +The payload is the raw content of your Application Package. For +example, you can issue this request using curl on your Linux command +line like this, assuming your DT Gateway is accepting requests at +localhost:9090: + +``` +$ curl -XPOST -T <app-package-file> http://localhost:9090/ws/v2/appPackages +``` + +##Getting Meta Information on Application Packages + + +You can get the meta information on Application Packages stored through +DT Gateway using this call. The information includes the logical plan +of each application within the Application Package. + +``` + GET /ws/v2/appPackages/{owner}/{pkgName}/{pkgVersion} +``` + +## Getting Available Operators In Application Package + +You can get the list of available operators in the Application Package +using this call. + +``` +GET /ws/v2/appPackages/{owner}/{pkgName}/{pkgVersion}/operators?parent={parent} +``` + +The parent parameter is optional. If given, parent should be the fully +qualified class name. It will only return operators that derive from +that class or interface. For example, if parent is +com.datatorrent.api.InputOperator, this call will only return input +operators provided by the Application Package. + +## Getting Properties of Operators in Application Package + +You can get the list of properties of any operator in the Application +Package using this call. + +``` +GET /ws/v2/appPackages/{owner}/{pkgName}/{pkgVersion}/operators/{className} +``` + +## Getting List of Pre-Set Configurations in Application Package + +You can get a list of pre-set configurations within the Application +Package using this call. + +``` +GET /ws/v2/appPackages/{owner}/{pkgName}/{packageVersion}/configs +``` + +You can also get the content of a specific pre-set configuration within +the Application Package. + +``` + GET /ws/v2/appPackages/{owner}/{pkgName}/{pkgVersion}/configs/{configName} +``` + +## Changing Pre-Set Configurations in Application Package + +You can create or replace pre-set configurations within the Application +Package +``` + PUT /ws/v2/appPackages/{owner}/{pkgName}/{pkgVersion}/configs/{configName} +``` +The payload of this PUT call is the XML file that represents the pre-set configuration. The Content-Type of the payload is "application/xml" and you can delete a pre-set configuration within the Application Package. +``` + DELETE /ws/v2/appPackages/{owner}/{pkgName}/{pkgVersion}/configs/{configName} +``` + +## Retrieving an Application Package + +You can download the Application Package file. This Application Package +is not necessarily the same file as the one that was originally uploaded +since the pre-set configurations may have been modified. + +``` + GET /ws/v2/appPackages/{owner}/{pkgName}/{pkgVersion}/download +``` + +## Launching an Application Package + +You can launch an application within an Application Package. +``` +POST /ws/v2/appPackages/{owner}/{pkgName}/{pkgVersion}/applications/{appName}/launch?config={configName} +``` + +The config parameter is optional. If given, it must be one of the +pre-set configuration within the given Application Package. The +Content-Type of the payload of the POST request is "application/json" +and should contain the properties to be launched with the application. + It is of the form: + +``` + {"property-name":"property-value", ... } +``` + +Here is an example of launching an application through curl: + +``` + $ curl -XPOST -d'{"dt.operator.console.prop.stringFormat":"xyz %s"}' + http://localhost:9090/ws/v2/appPackages/dtadmin/mydtapp/1.0-SNAPSHOT/app + lications/MyFirstApplication/launch +``` + +Please refer to the [Gateway API reference](https://www.google.com/url?q=https://www.datatorrent.com/docs/guides/DTGatewayAPISpecification.html&sa=D&usg=AFQjCNEWfN7-e7fd6MoWZjmJUE3GW7UwdQ) for the complete specification of the REST API. + +# 7. Examining and Launching Application Packages Through DT CLI + +If you are working with Application Packages in the local filesystem and +do not want to deal with the DT Gateway, you can use the DataTorrent +Command Line Interface (DT CLI). Please refer to the [Gateway API reference](https://www.datatorrent.com/docs/guides/DTGatewayAPISpecification.html) +to see samples for these commands. + +##Getting Application Package Meta Information + +You can get the meta information about the Application Package using +this DT CLI command. + +``` + dt> get-app-package-info <app-package-file> +``` + +## Getting Available Operators In Application Package + +You can get the list of available operators in the Application Package +using this command. + +``` + dt> get-app-package-operators <app-package-file> <package-prefix> + [parent-class] +``` + +## Getting Properties of Operators in Application Package + +You can get the list of properties of any operator in the Application +Package using this command. + + dt> get-app-package-operator-properties <app-package-file> <operator-class> + + +##Launching an Application Package + +You can launch an application within an Application Package. +``` +dt> launch [-D property-name=property-value, ...] [-conf config-name] + [-apconf config-file-within-app-package] <app-package-file> + [matching-app-name] +``` +Note that -conf expects a configuration file in the file system, while -apconf expects a configuration file within the app package. + +© 2014-2015 DataTorrent Inc. + + + + + http://git-wip-us.apache.org/repos/asf/incubator-apex-core/blob/c689142d/AppConfPackage.md ---------------------------------------------------------------------- diff --git a/AppConfPackage.md b/AppConfPackage.md new file mode 100644 index 0000000..004f9e3 --- /dev/null +++ b/AppConfPackage.md @@ -0,0 +1,246 @@ +#Application Configuration Packages + +# Introduction + +An Apex Application Configuration Package is a zip file that contains +configuration files and additional files to be launched with an +[Application Package](https://www.datatorrent.com/docs/guides/ApplicationPackages.html) using +DTCLI or REST API. This guide assumes the readerâs familiarity of +Application Package. Please read the Application Package document to +get yourself familiar with the concept first if you have not done so. + +#Requirements + +You will need have the following installed: + +1. Apache Maven 3.0 or later (for assembling the Config Package) +2. Apex 3.0.0 or later (for launching the App Package with the Config + Package in your cluster) + +#Creating Your First Configuration Package + +You can create a Configuration Package using your Linux command line, or +using your favorite IDE. + +## Using Command Line + +First, change to the directory where you put your projects, and create a +DT configuration project using Maven by running the following command. + Replace "com.example", "mydtconfig" and "1.0-SNAPSHOT" with the +appropriate values: + +``` + $ mvn archetype:generate + -DarchetypeRepository=https://www.datatorrent.com/maven/content/reposito + ries/releases + -DarchetypeGroupId=com.datatorrent + -DarchetypeArtifactId=apex-conf-archetype -DarchetypeVersion=3.0.0 + -DgroupId=com.example -Dpackage=com.example.mydtconfig + -DartifactId=mydtconfig -Dversion=1.0-SNAPSHOT + +``` + +This creates a Maven project named "mydtconfig". Open it with your +favorite IDE (e.g. NetBeans, Eclipse, IntelliJ IDEA). Try it out by +running the following command: +``` +$ mvn package +``` + +The "mvn package" command creates the Config Package file in target +directory as target/mydtconfig.apc. You will be able to use that +Configuration Package file to launch an application in your actual +DataTorrent RTS installation. + +## Using IDE + +Alternatively, you can do the above steps all within your IDE. For +example, in NetBeans, select File -\> New Project. Then choose âMavenâ +and âProject from Archetypeâ in the dialog box, as shown. + + + +Then fill the Group ID, Artifact ID, Version and Repository entries as +shown below. + + + +Group ID: com.datatorrent +Artifact ID: apex-conf-archetype +Version: 3.0.0 (or any later version) + +Repository: +[https://www.datatorrent.com/maven/content/repositories/releases](https://www.datatorrent.com/maven/content/repositories/releases) + +[](https://www.datatorrent.com/maven/content/repositories/releases) + +Press Next and fill out the rest of the required information. For +example: + + + +Click Finish, and now you have created your own Apex +Configuration Package project. The procedure for other IDEs, like +Eclipse or IntelliJ, is similar. + +#Assembling your own configuration package + +Inside the project created by the archetype, these are the files that +you should know about when assembling your own configuration package: + + ./pom.xml + ./src/main/resources/classpath + ./src/main/resources/files + +./src/main/resources/META-INF/properties.xml +./src/main/resources/META-INF/properties-{appname}.xml + +##pom.xml + +Example: + +``` + <groupId>com.example</groupId> + <version>1.0.0</version> + <artifactId>mydtconf</artifactId> + <packaging>jar</packaging> + <!-- change these to the appropriate values --> + <name>My DataTorrent Application Configuration</name> + <description>My DataTorrent Application Configuration Description</description> + <properties> + <datatorrent.apppackage.name>mydtapp</datatorrent.apppackage.name> + <datatorrent.apppackage.minversion>1.0.0</datatorrent.apppackage.minversion> + <datatorrent.apppackage.maxversion>1.9999.9999</datatorrent.apppackage.maxversion> + <datatorrent.appconf.classpath>classpath/*</datatorrent.appconf.classpath> + <datatorrent.appconf.files>files/*</datatorrent.appconf.files> + </properties> + +``` +In pom.xml, you can change the values of ```<groupId>, <version>, +<artifactId>, <name> ```and ```<description>``` to your desired values. + +You can also change the values of ```<datatorrent.apppackage.name>, <datatorrent.apppackage.minversion> <datatorrent.apppackage.maxversion>``` +to reflect what app packages should be used with this configuration +package. Apex will use this information to check whether a +configuration package is compatible with the application package when +you issue a launch command. + +## ./src/main/resources/classpath + +Place any file in this directory that youâd like to be copied to the +compute machines when launching an application and included in the +classpath of the application. Example of such files are Java properties +files and jar files. + +## ./src/main/resources/files + +Place any file in this directory that youâd like to be copied to the +compute machines when launching an application but not included in the +classpath of the application. + +## Properties XML file + +A properties xml file consists of a set of key-value pairs. The set of +key-value pairs specifies the configuration options the application +should be launched with. + +Example: +``` +<configuration> + <property> + <name>some-property-name</name> + <value>some-property-value</value> + </property> + ... +</configuration> +``` +Names of properties XML file: + + **properties.xml:** Properties that are global to the Configuration +Package + + **properties-{appName}.xml:** Properties that are specific when launching +an application with the specified appName within the Application +Package. + +After you are done with the above, remember to do mvn package to +generate a new configuration package, which will be located in the +target directory in your project. + +## Zip structure of configuration package +Apex Application Configuration Package files are zip files. You +can examine the content of any Application Configuration Package by +using unzip -t on your Linux command line. The structure of the zip +file is as follow: + +``` +META-INF + MANIFEST.MF + properties.xml + properties-{appname}.xml +classpath + {classpath files} +files + {files} +``` + +#Launching with CLI + +-conf of the launch command in CLI supports specifying configuration +package in the local filesystem. Example: + +dt\> launch DTApp-mydtapp-1.0.0.jar -conf DTConfig-mydtconfig-1.0.0.jar + +This command expects both the application package and the configuration +package to be in the local file system. + +#Related REST API + +##POST /ws/v2/configPackages + +Payload: Raw content of configuration package zip +Function: Creates or replace a configuration package zip file in HDFS + +Curl example: +$ curl -XPOST -T DTConfig-{name}.jar http://{yourhost:port}/ws/v2/configPackages + +## GET /ws/v2/configPackages?appPackageName=...&appPackageVersion=... + +All query parameters are optional + +Function: Returns the configuration packages that the user is authorized +to use and that are compatible with the specified appPackageName, +appPackageVersion and appName. + +## GET /ws/v2/configPackages/``<user>``?appPackageName=...&appPackageVersion=... + +All query parameters are optional + +Function: Returns the configuration packages under the specified user +and that are compatible with the specified appPackageName, +appPackageVersion and appName. + +## GET /ws/v2/configPackages/```<user>```/```<name>``` + +Function: Returns the information of the specified configuration package + +## GET /ws/v2/configPackages/```<user>```/```<name>```/download + +Function: Returns the raw config package file + +Curl example: +$ curl http://{yourhost:port}/ws/v2/configPackages/{user}/{name}/download \> DTConfig-xyz.jar + +\$ unzip -t DTConfig-xyz.jar + +## POST /ws/v2/appPackages/```<user>```/```<app-pkg-name>```/```<app-pkg-version>```/applications/{app-name}/launch?configPackage=```<user>```/```<confpkgname>``` + +Function: Launches the app package with the specified configuration +package stored in HDFS. + +Curl example: + +$ curl -XPOST -d â{}â http://{yourhost:port}/ws/v2/appPackages/{user}/{app-pkg-name}/{app-pkg-version}/applications/{app-name}/launch?configPackage={user}/{confpkgname} + +© 2014-2015 DataTorrent Inc. +
