METAMODEL-247: Fixed Closes #97
Project: http://git-wip-us.apache.org/repos/asf/metamodel/repo Commit: http://git-wip-us.apache.org/repos/asf/metamodel/commit/47c6d56c Tree: http://git-wip-us.apache.org/repos/asf/metamodel/tree/47c6d56c Diff: http://git-wip-us.apache.org/repos/asf/metamodel/diff/47c6d56c Branch: refs/heads/5.x Commit: 47c6d56cc7dbc3e6772b94d3816b94252622cbef Parents: 025d07f Author: kaspersorensen <[email protected]> Authored: Thu Apr 28 16:39:41 2016 -0700 Committer: kaspersorensen <[email protected]> Committed: Thu Apr 28 16:39:41 2016 -0700 ---------------------------------------------------------------------- CHANGES.md | 335 ++++++++--------- .../schema/naming/ColumnNamingStrategies.java | 10 + .../naming/CustomColumnNamingStrategy.java | 62 ++++ fixedwidth/pom.xml | 98 ++--- .../fixedwidth/FixedWidthColumnSpec.java | 45 +++ .../fixedwidth/FixedWidthConfiguration.java | 358 ++++++++++--------- .../FixedWidthConfigurationReader.java | 180 ++++++++++ .../FixedWidthConfigurationReaderTest.java | 89 +++++ .../src/test/resources/metadata_spec1/data.txt | 5 + .../metadata_spec1/sas-formatfile-metadata.txt | 4 + .../metadata_spec1/sas-input-metadata.txt | 19 + 11 files changed, 823 insertions(+), 382 deletions(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/metamodel/blob/47c6d56c/CHANGES.md ---------------------------------------------------------------------- diff --git a/CHANGES.md b/CHANGES.md index 4356071..dd03195 100644 --- a/CHANGES.md +++ b/CHANGES.md @@ -1,167 +1,168 @@ -### Apache MetaModel 4.5.3 (work in progress) - - * [METAMODEL-235] - Fixed a bug related to handling of null or missing values in ElasticSearch using REST client. - * [METAMODEL-225] - Fixed support for nested objects and arrays in ElasticSearch using REST client. - * [METAMODEL-244] - Added ColumnNamingStrategies concept which allows custom column naming and column name overriding. - * [METAMODEL-242] - Fixed issue when de-serializing old enum-instances of FunctionType. - -### Apache MetaModel 4.5.2 - - * [METAMODEL-236] - Made OperatorType and FunctionType Serializable to ensure that serialization of Query is possible. - -### Apache MetaModel 4.5.1 - - * [METAMODEL-227] - Fix for respecting CSV escape character also when no quote character is set. - * [METAMODEL-183] - MongoDB module split into three: common, Mongo2 and Mongo3 to allow use of either old or new MongoDB API. - * [METAMODEL-231] - Fixed a bug causing the Neo4j to represent the same table multiple times within a schema. - * [METAMODEL-228] - Fixed a bug causing Number.class to not be converted to ColumnType.NUMBER. - -### Apache MetaModel 4.5.0 - - * [METAMODEL-212] - New module for ElasticSearch via REST client. - * [METAMODEL-99] - New module for Neo4j connectivity with MetaModel. - * [METAMODEL-207] - Ensured the serializability of the SingleLineCsvRow class. - * [METAMODEL-211] - Fixed a bug related to lookup by primary key (_id) on MongoDB. - * [METAMODEL-216] - Added new aggregate functions: FIRST, LAST and RANDOM. - * [METAMODEL-195] - Added new function MAP_VALUE which allows extracting a nested value from within a key/value map field. - * [METAMODEL-15] - Query parser support for table names with space. Delimitters can be double quote or square brackets. - * [METAMODEL-215] - Improved the capability of NumberComparator to support Integer, Long, Double, BigInteger and other built-in Number classes. - * [METAMODEL-218] - Fixed conversion of STRING and NUMBER types to database-specific types in JDBC module. - * [METAMODEL-205] - Added validation of Excel sheet name before attempting to create table (sheet). - * [METAMODEL-219] - Made HdfsResource capable of incorporating Hadoop configuration files core-site.xml and hdfs-site.xml - * [METAMODEL-220] - Made HdfsResource capable of working with other URI schemes than 'hdfs'. - -### Apache MetaModel 4.4.1 - - * [METAMODEL-198] - Fixed support for JDBC TIMESTAMP precision to match the underlying database's precision. - * [METAMODEL-200] - Added optional "APPROXIMATE" keyword to query syntax for aggregate functions. - * [METAMODEL-144] - Automated binary packaging of the MetaModel project. - * [METAMODEL-197] - ElasticSearch schema update/change after CREATE TABLE statements. - * [METAMODEL-199] - Fixed a bug in query parser when parsing two consecutive WHERE items with parentheses around them. - * [METAMODEL-203] - Upgraded MongoDB dependency version and API to the 3.x line. - -### Apache MetaModel 4.4.0 - - * [METAMODEL-192] - Added support for Scalar functions. We have a basic set of datatype conversion functions as well as support for UDF via implementing the ScalarFunction interface. - * [METAMODEL-194] - Added support for setting the "Max rows" flag of a query to 0. This will always return an empty dataset. - * [METAMODEL-173] - Improved CSV writing to non-file destinations. Added .write() and .append() methods to Resource interface. - * [METAMODEL-170] - Dropped support for Java 6. - * [METAMODEL-176] - Trimmed the transient dependencies of the JDBC module. - * [METAMODEL-178] - Added AggregateFunction and ScalarFunction interfaces. Changed FunctionType enum to be super-interface of those. Compatibility is retained but a recompile of code using FunctionType is needed. - * [METAMODEL-188] - Changed OperatorType enum to be an interface. Compatibility is retained but a recompile of code is needed. - * [METAMODEL-179] - Ensured that HdfsResource is not closing a shared HDFS file system reference. - * [METAMODEL-171] - Made integration tests for Cassandra module function properly in all environments. - * [METAMODEL-177] - Fixed a bug pertaining to the serializability of HdfsResource. - * [METAMODEL-172] - ElasticSearch Date types should be converted properly. - * [METAMODEL-184] - ElasticSearch querying with "IS NULL" and "IS NOT NULL" now uses MissingFilter and ExistsFilter. - * [METAMODEL-190] - Improved decimal number support in Excel module. - * [METAMODEL-187] - Improved memory consumption of Excel module by passing random-access-file handles to POI when possible. - * [METAMODEL-191] - Resolved a number of dependency conflicts/overlaps when combining multiple MetaModel modules. - * [METAMODEL-157] - Fixed an issue in DELETE FROM statements with WHERE clauses requiring client-side data type conversion on JDBC databases. - * [METAMODEL-182] - Improved HdfsResource and FileResource directory-based implementations by adding also getSize() and getLastModified() directory-based implementations. - -### Apache MetaModel 4.3.6 - - * [METAMODEL-161] - Upgraded HBase client API to version 1.1.1 - * [METAMODEL-160] - Added support for Apache Hive via the JDBC module of MetaModel. - * [METAMODEL-162] - Made HdfsResource Serializable and added property getters - * [METAMODEL-163] - Made FileResource and HdfsResource work with folders containing file chunks instead of only single files - * [METAMODEL-104] - Added 'hadoop' and 'hbase' modules to dependencies of 'MetaModel-full'. - * [METAMODEL-164] - Fixed a bug in data type parsing of ElasticSearch mapping document. - -### Apache MetaModel 4.3.5 - - * [METAMODEL-148] - Added a 'hadoop' module with a HdfsResource class to allow CSV, Excel and Fixed-width file access on HDFS. - * [METAMODEL-152] - Fixed an issue of not clearing schema cache when refreshSchemas() is invoked. - * [METAMODEL-149] - Added support for COUNTER data type in Cassandra. - * [METAMODEL-151] - Added support for DOUBLE data type mapping in PostgreSQL - * [METAMODEL-154] - Use embedded Cassandra server for integration tests. - -### Apache MetaModel 4.3.4 - - * [METAMODEL-136] - Added LIKE operator native support (using conversion to regex) for MongoDB. - * [METAMODEL-138] - Allow empty characters before AS keyword in query parsing. - * [METAMODEL-141] - Improved mapping of ColumnType to SQL data types for Oracle, SQL Server, MySQL, DB2 and PostgreSQL - * [METAMODEL-142] - Ensured that JDBC schema refreshes in an UpdateScript is using same Connection/Transaction as rest of operations - * [METAMODEL-133] - Improved query parser support for multiple JOINs in same query. - * [METAMODEL-140] - Fixed support for ElasticSearch mappings with additional property attributes. - -### Apache MetaModel 4.3.3 - - * [METAMODEL-123] - Added compatibility with ElasticSearch version 1.4.x - * [METAMODEL-93] - Added compatibility with Apache HBase version 1.0.0 - * [METAMODEL-124] - Invoked ElasticSearch cross-version incompatible methods via reflection - * [METAMODEL-125] - Added support for comma-separated select items in Query.select(String) method argument. - * [METAMODEL-128] - Fixed bug in DataSet ordering when aggregation functions are applied to non-JDBC modules. - * [METAMODEL-131] - Added support for composite primary keys in JDBC CREATE TABLE statements. - -### Apache MetaModel 4.3.2 - - * [METAMODEL-78] - Fixed a bug that caused SELECT DISTINCT to sometimes return duplicates records on certain DataContext implementations. - * [METAMODEL-106] - Improved handling of invalid or non-existing index names passed to ElasticSearchDataContext - * [METAMODEL-79] - Added update execution support on ElasticSearch module. Increased capability of pushing down WHERE items to ElasticSearch searches. - * [METAMODEL-115] - Improved query parsing to allow lower-case function names, operators etc. - -### Apache MetaModel 4.3.1 - - * [METAMODEL-100] - Fixed bug when having multiple columns of same name. Added column no. comparison when calling Column.equals(...). - -### Apache MetaModel 4.3.0-incubating - - * [METAMODEL-77] - New module 'elasticsearch' for connecting and modeling ElasticSearch indexes through MetaModel. - * [METAMODEL-18] - New module 'cassandra' for connecting and modelling Apache Cassandra databases through MetaModel. - * [METAMODEL-83] - Added new operator types: GREATER_THAN_OR_EQUAL ('>=') and LESS_THAN_OR_EQUAL ('<='). - * [METAMODEL-92] - For JSON, MongoDB and CouchDB: Made it possible to specify column names referring nested fields such as "name.first" or "addresses[0].city". - * [METAMODEL-76] - Query parser improved to handle filters without spaces inbetween operator and operands. - * [METAMODEL-95] - Fixed a critical bug in the Salesforce.com module which caused all number values to be interpreted as '1'. - * [METAMODEL-74] - Fixed a bug related to skipping blank values when applying an aggregate function (SUM, AVG etc.) - * [METAMODEL-85] - Fixed a bug that caused NULL values to be evaluated with equal-sign in JDBC update and delete statements instead of 'IS NULL'. - -### Apache MetaModel 4.2.0-incubating - - * [METAMODEL-38] - New module 'json' for handling json files (containing JSON arrays of documents or line-delimited JSON documents) - * [METAMODEL-54] - ColumnType converted from enum to interface to allow for further specialization in modules. - * [METAMODEL-57] - Changed the column type VARCHAR into STRING for the modules: CSV, Fixedwidth, Excel and XML. - * [METAMODEL-56] - Made separate column types for converted JDBC LOBs - "CLOB as String" and "BLOB as bytes". - * [METAMODEL-46] - Improved row-lookup by primary key (ID) in CouchDB - * [METAMODEL-58] - Fixed a bug related to using CreateTable class and primary keys not getting created. - * [METAMODEL-3] - Improved writing of Byte-Order-Mark (BOM) for various encoding spelling variants. - * [METAMODEL-70] - Made the build compatible with both JDK versions 6 and 7. - * [METAMODEL-59] - Fixed a bug related to handling of date/time literals in MS SQL Server queries. - * [METAMODEL-60] - Fixed a bug related to DISTINCT and TOP keywords in MS SQL Server queries. - * [METAMODEL-45] - Improved and standardized way of handling integration test connection information towards external databases. - * [METAMODEL-62] - Fixed a bug related to fault-tolerant handling of malformed CSV lines when reading CSVs in single-line mode - * [METAMODEL-68] - Made it possible to create a CSV table without a header line in the file, if the user configures it. - * [METAMODEL-67] - Upgraded Jackson (JSON library) dependency from org.codehaus namespace to the newer com.fasterxml namespace. - * [METAMODEL-69] - Fixed issue with deserialization of ColumnType into the new interface instead of the old enum. - -### Apache MetaModel 4.1.0-incubating - - * [METAMODEL-13] - Added support for Apache HBase via the new module "MetaModel-hbase" - * [METAMODEL-41] - Added a parser for SimpleTableDef objects (SimpleTableDefParser). It parses statements similar to CREATE TABLE statements, although without the "CREATE TABLE" prefix. For example: foo (bar INTEGER, baz VARCHAR) - * [METAMODEL-11] - New module "MetaModel-spring" which adds a convenient FactoryBean to produce various types of DataContext objects based on externalizable parameters, for Spring framework users. - * [METAMODEL-32] - Fixed thread-safety issue in Excel module when tables (sheets) metadata is updated. - * [METAMODEL-47] - Fixed issue in Excel of loading schema if query is fired based on metadata from a previous DataContext instance. - * [METAMODEL-35] - Improved query rewriting for DB2 when paged queries contain ORDER BY clause. - * [METAMODEL-44] - Added an optional method for QueryPostprocessDataContext implementations to do a row-lookup by primary key value. - * [METAMODEL-43] - Made CSV datastores skip empty lines in file instead of treating them of rows with null values. - * [METAMODEL-39] - Added pooling of active/used Connections and PreparedStatements in JDBC compiled queries. - * [METAMODEL-34] - Updated LICENSE file to not include bundled dependencies' licenses. - * [METAMODEL-33] - Ensured that Apache Rat plugin for Maven is properly activated. - * [METAMODEL-37] - Removed old site sources from project. - -### Apache MetaModel 4.0.0-incubating - - * [METAMODEL-9] - SalesforceDataSet is throwing exception for insert sql of record having date/time. - * [METAMODEL-4] - Use folder name as schema name for file based DataContexts - * [METAMODEL-5] - Faster CsvDataContext implementation for single-line values - * [METAMODEL-26] - Provide endpoint URL with SalesforceDataContext - * Upgraded Apache POI dependency to v. 3.9 - * Improved fluent Query builder API by adding string parameter based methods for joining tables - * Added a utility ObjectInputStream for deserializing legacy MetaModel objects - * Performance improvement to CSV reading when values are only single line based - * Setting up the project on apache infrastructure - * [METAMODEL-10] - Exclude Jackcess dependency (Access module) from MetaModel - * Renaming the package hierarchy from org.eobjects.metamodel to org.apache.metamodel - * [METAMODEL-29] - Fixed issue in CreateTable builder class, causing it to only support a single column definition - * [METAMODEL-30] - Fixed issue with count(*) queries on CSV resources that does not provide a byte stream length +### Apache MetaModel 4.5.3 (work in progress) + + * [METAMODEL-235] - Fixed a bug related to handling of null or missing values in ElasticSearch using REST client. + * [METAMODEL-225] - Fixed support for nested objects and arrays in ElasticSearch using REST client. + * [METAMODEL-244] - Added ColumnNamingStrategies concept which allows custom column naming and column name overriding. + * [METAMODEL-242] - Fixed issue when de-serializing old enum-instances of FunctionType. + * [METAMODEL-247] - Added FixedWidthConfigurationReader for reading fixed width file metadata from external files. + +### Apache MetaModel 4.5.2 + + * [METAMODEL-236] - Made OperatorType and FunctionType Serializable to ensure that serialization of Query is possible. + +### Apache MetaModel 4.5.1 + + * [METAMODEL-227] - Fix for respecting CSV escape character also when no quote character is set. + * [METAMODEL-183] - MongoDB module split into three: common, Mongo2 and Mongo3 to allow use of either old or new MongoDB API. + * [METAMODEL-231] - Fixed a bug causing the Neo4j to represent the same table multiple times within a schema. + * [METAMODEL-228] - Fixed a bug causing Number.class to not be converted to ColumnType.NUMBER. + +### Apache MetaModel 4.5.0 + + * [METAMODEL-212] - New module for ElasticSearch via REST client. + * [METAMODEL-99] - New module for Neo4j connectivity with MetaModel. + * [METAMODEL-207] - Ensured the serializability of the SingleLineCsvRow class. + * [METAMODEL-211] - Fixed a bug related to lookup by primary key (_id) on MongoDB. + * [METAMODEL-216] - Added new aggregate functions: FIRST, LAST and RANDOM. + * [METAMODEL-195] - Added new function MAP_VALUE which allows extracting a nested value from within a key/value map field. + * [METAMODEL-15] - Query parser support for table names with space. Delimitters can be double quote or square brackets. + * [METAMODEL-215] - Improved the capability of NumberComparator to support Integer, Long, Double, BigInteger and other built-in Number classes. + * [METAMODEL-218] - Fixed conversion of STRING and NUMBER types to database-specific types in JDBC module. + * [METAMODEL-205] - Added validation of Excel sheet name before attempting to create table (sheet). + * [METAMODEL-219] - Made HdfsResource capable of incorporating Hadoop configuration files core-site.xml and hdfs-site.xml + * [METAMODEL-220] - Made HdfsResource capable of working with other URI schemes than 'hdfs'. + +### Apache MetaModel 4.4.1 + + * [METAMODEL-198] - Fixed support for JDBC TIMESTAMP precision to match the underlying database's precision. + * [METAMODEL-200] - Added optional "APPROXIMATE" keyword to query syntax for aggregate functions. + * [METAMODEL-144] - Automated binary packaging of the MetaModel project. + * [METAMODEL-197] - ElasticSearch schema update/change after CREATE TABLE statements. + * [METAMODEL-199] - Fixed a bug in query parser when parsing two consecutive WHERE items with parentheses around them. + * [METAMODEL-203] - Upgraded MongoDB dependency version and API to the 3.x line. + +### Apache MetaModel 4.4.0 + + * [METAMODEL-192] - Added support for Scalar functions. We have a basic set of datatype conversion functions as well as support for UDF via implementing the ScalarFunction interface. + * [METAMODEL-194] - Added support for setting the "Max rows" flag of a query to 0. This will always return an empty dataset. + * [METAMODEL-173] - Improved CSV writing to non-file destinations. Added .write() and .append() methods to Resource interface. + * [METAMODEL-170] - Dropped support for Java 6. + * [METAMODEL-176] - Trimmed the transient dependencies of the JDBC module. + * [METAMODEL-178] - Added AggregateFunction and ScalarFunction interfaces. Changed FunctionType enum to be super-interface of those. Compatibility is retained but a recompile of code using FunctionType is needed. + * [METAMODEL-188] - Changed OperatorType enum to be an interface. Compatibility is retained but a recompile of code is needed. + * [METAMODEL-179] - Ensured that HdfsResource is not closing a shared HDFS file system reference. + * [METAMODEL-171] - Made integration tests for Cassandra module function properly in all environments. + * [METAMODEL-177] - Fixed a bug pertaining to the serializability of HdfsResource. + * [METAMODEL-172] - ElasticSearch Date types should be converted properly. + * [METAMODEL-184] - ElasticSearch querying with "IS NULL" and "IS NOT NULL" now uses MissingFilter and ExistsFilter. + * [METAMODEL-190] - Improved decimal number support in Excel module. + * [METAMODEL-187] - Improved memory consumption of Excel module by passing random-access-file handles to POI when possible. + * [METAMODEL-191] - Resolved a number of dependency conflicts/overlaps when combining multiple MetaModel modules. + * [METAMODEL-157] - Fixed an issue in DELETE FROM statements with WHERE clauses requiring client-side data type conversion on JDBC databases. + * [METAMODEL-182] - Improved HdfsResource and FileResource directory-based implementations by adding also getSize() and getLastModified() directory-based implementations. + +### Apache MetaModel 4.3.6 + + * [METAMODEL-161] - Upgraded HBase client API to version 1.1.1 + * [METAMODEL-160] - Added support for Apache Hive via the JDBC module of MetaModel. + * [METAMODEL-162] - Made HdfsResource Serializable and added property getters + * [METAMODEL-163] - Made FileResource and HdfsResource work with folders containing file chunks instead of only single files + * [METAMODEL-104] - Added 'hadoop' and 'hbase' modules to dependencies of 'MetaModel-full'. + * [METAMODEL-164] - Fixed a bug in data type parsing of ElasticSearch mapping document. + +### Apache MetaModel 4.3.5 + + * [METAMODEL-148] - Added a 'hadoop' module with a HdfsResource class to allow CSV, Excel and Fixed-width file access on HDFS. + * [METAMODEL-152] - Fixed an issue of not clearing schema cache when refreshSchemas() is invoked. + * [METAMODEL-149] - Added support for COUNTER data type in Cassandra. + * [METAMODEL-151] - Added support for DOUBLE data type mapping in PostgreSQL + * [METAMODEL-154] - Use embedded Cassandra server for integration tests. + +### Apache MetaModel 4.3.4 + + * [METAMODEL-136] - Added LIKE operator native support (using conversion to regex) for MongoDB. + * [METAMODEL-138] - Allow empty characters before AS keyword in query parsing. + * [METAMODEL-141] - Improved mapping of ColumnType to SQL data types for Oracle, SQL Server, MySQL, DB2 and PostgreSQL + * [METAMODEL-142] - Ensured that JDBC schema refreshes in an UpdateScript is using same Connection/Transaction as rest of operations + * [METAMODEL-133] - Improved query parser support for multiple JOINs in same query. + * [METAMODEL-140] - Fixed support for ElasticSearch mappings with additional property attributes. + +### Apache MetaModel 4.3.3 + + * [METAMODEL-123] - Added compatibility with ElasticSearch version 1.4.x + * [METAMODEL-93] - Added compatibility with Apache HBase version 1.0.0 + * [METAMODEL-124] - Invoked ElasticSearch cross-version incompatible methods via reflection + * [METAMODEL-125] - Added support for comma-separated select items in Query.select(String) method argument. + * [METAMODEL-128] - Fixed bug in DataSet ordering when aggregation functions are applied to non-JDBC modules. + * [METAMODEL-131] - Added support for composite primary keys in JDBC CREATE TABLE statements. + +### Apache MetaModel 4.3.2 + + * [METAMODEL-78] - Fixed a bug that caused SELECT DISTINCT to sometimes return duplicates records on certain DataContext implementations. + * [METAMODEL-106] - Improved handling of invalid or non-existing index names passed to ElasticSearchDataContext + * [METAMODEL-79] - Added update execution support on ElasticSearch module. Increased capability of pushing down WHERE items to ElasticSearch searches. + * [METAMODEL-115] - Improved query parsing to allow lower-case function names, operators etc. + +### Apache MetaModel 4.3.1 + + * [METAMODEL-100] - Fixed bug when having multiple columns of same name. Added column no. comparison when calling Column.equals(...). + +### Apache MetaModel 4.3.0-incubating + + * [METAMODEL-77] - New module 'elasticsearch' for connecting and modeling ElasticSearch indexes through MetaModel. + * [METAMODEL-18] - New module 'cassandra' for connecting and modelling Apache Cassandra databases through MetaModel. + * [METAMODEL-83] - Added new operator types: GREATER_THAN_OR_EQUAL ('>=') and LESS_THAN_OR_EQUAL ('<='). + * [METAMODEL-92] - For JSON, MongoDB and CouchDB: Made it possible to specify column names referring nested fields such as "name.first" or "addresses[0].city". + * [METAMODEL-76] - Query parser improved to handle filters without spaces inbetween operator and operands. + * [METAMODEL-95] - Fixed a critical bug in the Salesforce.com module which caused all number values to be interpreted as '1'. + * [METAMODEL-74] - Fixed a bug related to skipping blank values when applying an aggregate function (SUM, AVG etc.) + * [METAMODEL-85] - Fixed a bug that caused NULL values to be evaluated with equal-sign in JDBC update and delete statements instead of 'IS NULL'. + +### Apache MetaModel 4.2.0-incubating + + * [METAMODEL-38] - New module 'json' for handling json files (containing JSON arrays of documents or line-delimited JSON documents) + * [METAMODEL-54] - ColumnType converted from enum to interface to allow for further specialization in modules. + * [METAMODEL-57] - Changed the column type VARCHAR into STRING for the modules: CSV, Fixedwidth, Excel and XML. + * [METAMODEL-56] - Made separate column types for converted JDBC LOBs - "CLOB as String" and "BLOB as bytes". + * [METAMODEL-46] - Improved row-lookup by primary key (ID) in CouchDB + * [METAMODEL-58] - Fixed a bug related to using CreateTable class and primary keys not getting created. + * [METAMODEL-3] - Improved writing of Byte-Order-Mark (BOM) for various encoding spelling variants. + * [METAMODEL-70] - Made the build compatible with both JDK versions 6 and 7. + * [METAMODEL-59] - Fixed a bug related to handling of date/time literals in MS SQL Server queries. + * [METAMODEL-60] - Fixed a bug related to DISTINCT and TOP keywords in MS SQL Server queries. + * [METAMODEL-45] - Improved and standardized way of handling integration test connection information towards external databases. + * [METAMODEL-62] - Fixed a bug related to fault-tolerant handling of malformed CSV lines when reading CSVs in single-line mode + * [METAMODEL-68] - Made it possible to create a CSV table without a header line in the file, if the user configures it. + * [METAMODEL-67] - Upgraded Jackson (JSON library) dependency from org.codehaus namespace to the newer com.fasterxml namespace. + * [METAMODEL-69] - Fixed issue with deserialization of ColumnType into the new interface instead of the old enum. + +### Apache MetaModel 4.1.0-incubating + + * [METAMODEL-13] - Added support for Apache HBase via the new module "MetaModel-hbase" + * [METAMODEL-41] - Added a parser for SimpleTableDef objects (SimpleTableDefParser). It parses statements similar to CREATE TABLE statements, although without the "CREATE TABLE" prefix. For example: foo (bar INTEGER, baz VARCHAR) + * [METAMODEL-11] - New module "MetaModel-spring" which adds a convenient FactoryBean to produce various types of DataContext objects based on externalizable parameters, for Spring framework users. + * [METAMODEL-32] - Fixed thread-safety issue in Excel module when tables (sheets) metadata is updated. + * [METAMODEL-47] - Fixed issue in Excel of loading schema if query is fired based on metadata from a previous DataContext instance. + * [METAMODEL-35] - Improved query rewriting for DB2 when paged queries contain ORDER BY clause. + * [METAMODEL-44] - Added an optional method for QueryPostprocessDataContext implementations to do a row-lookup by primary key value. + * [METAMODEL-43] - Made CSV datastores skip empty lines in file instead of treating them of rows with null values. + * [METAMODEL-39] - Added pooling of active/used Connections and PreparedStatements in JDBC compiled queries. + * [METAMODEL-34] - Updated LICENSE file to not include bundled dependencies' licenses. + * [METAMODEL-33] - Ensured that Apache Rat plugin for Maven is properly activated. + * [METAMODEL-37] - Removed old site sources from project. + +### Apache MetaModel 4.0.0-incubating + + * [METAMODEL-9] - SalesforceDataSet is throwing exception for insert sql of record having date/time. + * [METAMODEL-4] - Use folder name as schema name for file based DataContexts + * [METAMODEL-5] - Faster CsvDataContext implementation for single-line values + * [METAMODEL-26] - Provide endpoint URL with SalesforceDataContext + * Upgraded Apache POI dependency to v. 3.9 + * Improved fluent Query builder API by adding string parameter based methods for joining tables + * Added a utility ObjectInputStream for deserializing legacy MetaModel objects + * Performance improvement to CSV reading when values are only single line based + * Setting up the project on apache infrastructure + * [METAMODEL-10] - Exclude Jackcess dependency (Access module) from MetaModel + * Renaming the package hierarchy from org.eobjects.metamodel to org.apache.metamodel + * [METAMODEL-29] - Fixed issue in CreateTable builder class, causing it to only support a single column definition + * [METAMODEL-30] - Fixed issue with count(*) queries on CSV resources that does not provide a byte stream length http://git-wip-us.apache.org/repos/asf/metamodel/blob/47c6d56c/core/src/main/java/org/apache/metamodel/schema/naming/ColumnNamingStrategies.java ---------------------------------------------------------------------- diff --git a/core/src/main/java/org/apache/metamodel/schema/naming/ColumnNamingStrategies.java b/core/src/main/java/org/apache/metamodel/schema/naming/ColumnNamingStrategies.java index de78338..0696376 100644 --- a/core/src/main/java/org/apache/metamodel/schema/naming/ColumnNamingStrategies.java +++ b/core/src/main/java/org/apache/metamodel/schema/naming/ColumnNamingStrategies.java @@ -18,6 +18,8 @@ */ package org.apache.metamodel.schema.naming; +import java.util.List; + /** * Constructors and common utilities for {@link ColumnNamingStrategy} objects. */ @@ -32,4 +34,12 @@ public class ColumnNamingStrategies { public static ColumnNamingStrategy defaultStrategy() { return DEFAULT_STRATEGY; } + + public static ColumnNamingStrategy customNames(List<String> columnNames) { + return new CustomColumnNamingStrategy(columnNames); + } + + public static ColumnNamingStrategy customNames(String ... columnNames) { + return new CustomColumnNamingStrategy(columnNames); + } } http://git-wip-us.apache.org/repos/asf/metamodel/blob/47c6d56c/core/src/main/java/org/apache/metamodel/schema/naming/CustomColumnNamingStrategy.java ---------------------------------------------------------------------- diff --git a/core/src/main/java/org/apache/metamodel/schema/naming/CustomColumnNamingStrategy.java b/core/src/main/java/org/apache/metamodel/schema/naming/CustomColumnNamingStrategy.java new file mode 100644 index 0000000..e6cc706 --- /dev/null +++ b/core/src/main/java/org/apache/metamodel/schema/naming/CustomColumnNamingStrategy.java @@ -0,0 +1,62 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ +package org.apache.metamodel.schema.naming; + +import java.util.Arrays; +import java.util.Iterator; +import java.util.List; + +/** + * A {@link ColumnNamingStrategy} that allows the user to supply his own list of + * custom column names. + */ +public class CustomColumnNamingStrategy implements ColumnNamingStrategy { + + private static final long serialVersionUID = 1L; + + private final List<String> columnNames; + + public CustomColumnNamingStrategy(List<String> columnNames) { + this.columnNames = columnNames; + } + + public CustomColumnNamingStrategy(String... columnNames) { + this(Arrays.asList(columnNames)); + } + + @Override + public ColumnNamingSession startColumnNamingSession() { + final Iterator<String> iterator = columnNames.iterator(); + return new ColumnNamingSession() { + + @Override + public String getNextColumnName(ColumnNamingContext ctx) { + if (iterator.hasNext()) { + return iterator.next(); + } + return null; + } + + @Override + public void close() { + } + }; + } + +} http://git-wip-us.apache.org/repos/asf/metamodel/blob/47c6d56c/fixedwidth/pom.xml ---------------------------------------------------------------------- diff --git a/fixedwidth/pom.xml b/fixedwidth/pom.xml index e667a10..43e4021 100644 --- a/fixedwidth/pom.xml +++ b/fixedwidth/pom.xml @@ -1,46 +1,52 @@ -<?xml version="1.0" encoding="UTF-8" ?> -<!-- -Licensed to the Apache Software Foundation (ASF) under one -or more contributor license agreements. See the NOTICE file -distributed with this work for additional information -regarding copyright ownership. The ASF licenses this file -to you under the Apache License, Version 2.0 (the -"License"); you may not use this file except in compliance -with the License. You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - -Unless required by applicable law or agreed to in writing, -software distributed under the License is distributed on an -"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY -KIND, either express or implied. See the License for the -specific language governing permissions and limitations -under the License. ---> -<project xmlns="http://maven.apache.org/POM/4.0.0"; xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"; xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd";> - <parent> - <artifactId>MetaModel</artifactId> - <groupId>org.apache.metamodel</groupId> - <version>4.5.3-SNAPSHOT</version> - </parent> - <modelVersion>4.0.0</modelVersion> - <artifactId>MetaModel-fixedwidth</artifactId> - <name>MetaModel module for fixed width value files</name> - <dependencies> - <dependency> - <groupId>org.apache.metamodel</groupId> - <artifactId>MetaModel-core</artifactId> - <version>${project.version}</version> - </dependency> - <dependency> - <groupId>org.slf4j</groupId> - <artifactId>slf4j-nop</artifactId> - <scope>test</scope> - </dependency> - <dependency> - <groupId>junit</groupId> - <artifactId>junit</artifactId> - <scope>test</scope> - </dependency> - </dependencies> -</project> +<?xml version="1.0" encoding="UTF-8" ?> +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. +--> +<project xmlns="http://maven.apache.org/POM/4.0.0"; xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"; xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd";> + <parent> + <artifactId>MetaModel</artifactId> + <groupId>org.apache.metamodel</groupId> + <version>4.5.3-SNAPSHOT</version> + </parent> + <modelVersion>4.0.0</modelVersion> + <artifactId>MetaModel-fixedwidth</artifactId> + <name>MetaModel module for fixed width value files</name> + <dependencies> + <dependency> + <groupId>org.apache.metamodel</groupId> + <artifactId>MetaModel-core</artifactId> + <version>${project.version}</version> + </dependency> + <dependency> + <groupId>org.apache.metamodel</groupId> + <artifactId>MetaModel-csv</artifactId> + <version>${project.version}</version> + <optional>true</optional> + </dependency> + <dependency> + <groupId>org.slf4j</groupId> + <artifactId>slf4j-nop</artifactId> + <scope>test</scope> + </dependency> + <dependency> + <groupId>junit</groupId> + <artifactId>junit</artifactId> + <scope>test</scope> + </dependency> + </dependencies> +</project> http://git-wip-us.apache.org/repos/asf/metamodel/blob/47c6d56c/fixedwidth/src/main/java/org/apache/metamodel/fixedwidth/FixedWidthColumnSpec.java ---------------------------------------------------------------------- diff --git a/fixedwidth/src/main/java/org/apache/metamodel/fixedwidth/FixedWidthColumnSpec.java b/fixedwidth/src/main/java/org/apache/metamodel/fixedwidth/FixedWidthColumnSpec.java new file mode 100644 index 0000000..65ec219 --- /dev/null +++ b/fixedwidth/src/main/java/org/apache/metamodel/fixedwidth/FixedWidthColumnSpec.java @@ -0,0 +1,45 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ +package org.apache.metamodel.fixedwidth; + +import org.apache.metamodel.util.HasName; + +/** + * Represents the specification of a single column for a + * {@link FixedWidthDataContext}. + */ +public final class FixedWidthColumnSpec implements HasName { + + private final String name; + private final int width; + + public FixedWidthColumnSpec(String name, int width) { + this.name = name; + this.width = width; + } + + @Override + public String getName() { + return name; + } + + public int getWidth() { + return width; + } +} http://git-wip-us.apache.org/repos/asf/metamodel/blob/47c6d56c/fixedwidth/src/main/java/org/apache/metamodel/fixedwidth/FixedWidthConfiguration.java ---------------------------------------------------------------------- diff --git a/fixedwidth/src/main/java/org/apache/metamodel/fixedwidth/FixedWidthConfiguration.java b/fixedwidth/src/main/java/org/apache/metamodel/fixedwidth/FixedWidthConfiguration.java index 6e9f0f1..2b2cae5 100644 --- a/fixedwidth/src/main/java/org/apache/metamodel/fixedwidth/FixedWidthConfiguration.java +++ b/fixedwidth/src/main/java/org/apache/metamodel/fixedwidth/FixedWidthConfiguration.java @@ -1,169 +1,189 @@ -/** - * Licensed to the Apache Software Foundation (ASF) under one - * or more contributor license agreements. See the NOTICE file - * distributed with this work for additional information - * regarding copyright ownership. The ASF licenses this file - * to you under the Apache License, Version 2.0 (the - * "License"); you may not use this file except in compliance - * with the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, - * software distributed under the License is distributed on an - * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY - * KIND, either express or implied. See the License for the - * specific language governing permissions and limitations - * under the License. - */ -package org.apache.metamodel.fixedwidth; - -import java.io.Serializable; -import java.util.Arrays; -import java.util.List; - -import org.apache.metamodel.data.DataSet; -import org.apache.metamodel.schema.naming.ColumnNamingStrategies; -import org.apache.metamodel.schema.naming.ColumnNamingStrategy; -import org.apache.metamodel.util.BaseObject; -import org.apache.metamodel.util.FileHelper; - -/** - * Configuration of metadata about a fixed width values datacontext. - */ -public final class FixedWidthConfiguration extends BaseObject implements - Serializable { - - private static final long serialVersionUID = 1L; - - public static final int NO_COLUMN_NAME_LINE = 0; - public static final int DEFAULT_COLUMN_NAME_LINE = 1; - - private final String encoding; - private final int fixedValueWidth; - private final int[] valueWidths; - private final int columnNameLineNumber; - private final boolean failOnInconsistentLineWidth; - private final ColumnNamingStrategy columnNamingStrategy; - - public FixedWidthConfiguration(int fixedValueWidth) { - this(DEFAULT_COLUMN_NAME_LINE, FileHelper.DEFAULT_ENCODING, - fixedValueWidth); - } - - public FixedWidthConfiguration(int[] valueWidth) { - this(DEFAULT_COLUMN_NAME_LINE, FileHelper.DEFAULT_ENCODING, valueWidth, - false); - } - - public FixedWidthConfiguration(int columnNameLineNumber, String encoding, int fixedValueWidth) { - this(columnNameLineNumber, encoding, fixedValueWidth, false); - } - - public FixedWidthConfiguration(int columnNameLineNumber, String encoding, int fixedValueWidth, - boolean failOnInconsistentLineWidth) { - this.encoding = encoding; - this.fixedValueWidth = fixedValueWidth; - this.columnNameLineNumber = columnNameLineNumber; - this.failOnInconsistentLineWidth = failOnInconsistentLineWidth; - this.columnNamingStrategy = null; - this.valueWidths = new int[0]; - } - - public FixedWidthConfiguration(int columnNameLineNumber, String encoding, - int[] valueWidths, boolean failOnInconsistentLineWidth) { - this(columnNameLineNumber, null, encoding, valueWidths, failOnInconsistentLineWidth); - } - - public FixedWidthConfiguration(int columnNameLineNumber, ColumnNamingStrategy columnNamingStrategy, String encoding, - int[] valueWidths, boolean failOnInconsistentLineWidth) { - this.encoding = encoding; - this.fixedValueWidth = -1; - this.columnNameLineNumber = columnNameLineNumber; - this.failOnInconsistentLineWidth = failOnInconsistentLineWidth; - this.columnNamingStrategy = columnNamingStrategy; - this.valueWidths = valueWidths; - } - - /** - * The line number (1 based) from which to get the names of the columns. - * - * @return an int representing the line number of the column headers/names. - */ - public int getColumnNameLineNumber() { - return columnNameLineNumber; - } - - /** - * Gets a {@link ColumnNamingStrategy} to use if needed. - * @return - */ - public ColumnNamingStrategy getColumnNamingStrategy() { - if (columnNamingStrategy == null) { - return ColumnNamingStrategies.defaultStrategy(); - } - return columnNamingStrategy; - } - - /** - * Gets the file encoding to use for reading the file. - * - * @return the text encoding to use for reading the file. - */ - public String getEncoding() { - return encoding; - } - - /** - * Gets the width of each value within the fixed width value file. - * - * @return the fixed width to use when parsing the file. - */ - public int getFixedValueWidth() { - return fixedValueWidth; - } - - public int[] getValueWidths() { - return valueWidths; - } - - /** - * Determines if the {@link DataSet#next()} should throw an exception in - * case of inconsistent line width in the fixed width value file. - * - * @return a boolean indicating whether or not to fail on inconsistent line - * widths. - */ - public boolean isFailOnInconsistentLineWidth() { - return failOnInconsistentLineWidth; - } - - @Override - protected void decorateIdentity(List<Object> identifiers) { - identifiers.add(columnNameLineNumber); - identifiers.add(encoding); - identifiers.add(fixedValueWidth); - identifiers.add(valueWidths); - identifiers.add(failOnInconsistentLineWidth); - } - - @Override - public String toString() { - return "FixedWidthConfiguration[encoding=" + encoding - + ", fixedValueWidth=" + fixedValueWidth + ", valueWidths=" - + Arrays.toString(valueWidths) + ", columnNameLineNumber=" - + columnNameLineNumber + ", failOnInconsistentLineWidth=" - + failOnInconsistentLineWidth + "]"; - } - - public boolean isConstantValueWidth() { - return fixedValueWidth != -1; - } - - public int getValueWidth(int columnIndex) { - if (isConstantValueWidth()) { - return fixedValueWidth; - } - return valueWidths[columnIndex]; - } -} +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ +package org.apache.metamodel.fixedwidth; + +import java.io.Serializable; +import java.util.Arrays; +import java.util.List; + +import org.apache.metamodel.data.DataSet; +import org.apache.metamodel.schema.naming.ColumnNamingStrategies; +import org.apache.metamodel.schema.naming.ColumnNamingStrategy; +import org.apache.metamodel.util.BaseObject; +import org.apache.metamodel.util.CollectionUtils; +import org.apache.metamodel.util.FileHelper; +import org.apache.metamodel.util.HasNameMapper; + +/** + * Configuration of metadata about a fixed width values datacontext. + */ +public final class FixedWidthConfiguration extends BaseObject implements + Serializable { + + private static final long serialVersionUID = 1L; + + public static final int NO_COLUMN_NAME_LINE = 0; + public static final int DEFAULT_COLUMN_NAME_LINE = 1; + + private final String encoding; + private final int fixedValueWidth; + private final int[] valueWidths; + private final int columnNameLineNumber; + private final boolean failOnInconsistentLineWidth; + private final ColumnNamingStrategy columnNamingStrategy; + + public FixedWidthConfiguration(int fixedValueWidth) { + this(DEFAULT_COLUMN_NAME_LINE, FileHelper.DEFAULT_ENCODING, + fixedValueWidth); + } + + public FixedWidthConfiguration(int[] valueWidth) { + this(DEFAULT_COLUMN_NAME_LINE, FileHelper.DEFAULT_ENCODING, valueWidth, + false); + } + + public FixedWidthConfiguration(int columnNameLineNumber, String encoding, int fixedValueWidth) { + this(columnNameLineNumber, encoding, fixedValueWidth, false); + } + + public FixedWidthConfiguration(int columnNameLineNumber, String encoding, int fixedValueWidth, + boolean failOnInconsistentLineWidth) { + this.encoding = encoding; + this.fixedValueWidth = fixedValueWidth; + this.columnNameLineNumber = columnNameLineNumber; + this.failOnInconsistentLineWidth = failOnInconsistentLineWidth; + this.columnNamingStrategy = null; + this.valueWidths = new int[0]; + } + + public FixedWidthConfiguration(int columnNameLineNumber, String encoding, + int[] valueWidths, boolean failOnInconsistentLineWidth) { + this(columnNameLineNumber, null, encoding, valueWidths, failOnInconsistentLineWidth); + } + + public FixedWidthConfiguration(int columnNameLineNumber, ColumnNamingStrategy columnNamingStrategy, String encoding, + int[] valueWidths, boolean failOnInconsistentLineWidth) { + this.encoding = encoding; + this.fixedValueWidth = -1; + this.columnNameLineNumber = columnNameLineNumber; + this.failOnInconsistentLineWidth = failOnInconsistentLineWidth; + this.columnNamingStrategy = columnNamingStrategy; + this.valueWidths = valueWidths; + } + + public FixedWidthConfiguration(String encoding, List<FixedWidthColumnSpec> columnSpecs) { + this(encoding, columnSpecs, false); + } + + public FixedWidthConfiguration(String encoding, List<FixedWidthColumnSpec> columnSpecs, + boolean failOnInconsistentLineWidth) { + this.encoding = encoding; + this.fixedValueWidth = -1; + this.columnNameLineNumber = NO_COLUMN_NAME_LINE; + this.columnNamingStrategy = ColumnNamingStrategies.customNames(CollectionUtils.map(columnSpecs, + new HasNameMapper())); + this.valueWidths = new int[columnSpecs.size()]; + for (int i = 0; i < valueWidths.length; i++) { + valueWidths[i] = columnSpecs.get(i).getWidth(); + } + this.failOnInconsistentLineWidth = failOnInconsistentLineWidth; + } + + /** + * The line number (1 based) from which to get the names of the columns. + * + * @return an int representing the line number of the column headers/names. + */ + public int getColumnNameLineNumber() { + return columnNameLineNumber; + } + + /** + * Gets a {@link ColumnNamingStrategy} to use if needed. + * @return + */ + public ColumnNamingStrategy getColumnNamingStrategy() { + if (columnNamingStrategy == null) { + return ColumnNamingStrategies.defaultStrategy(); + } + return columnNamingStrategy; + } + + /** + * Gets the file encoding to use for reading the file. + * + * @return the text encoding to use for reading the file. + */ + public String getEncoding() { + return encoding; + } + + /** + * Gets the width of each value within the fixed width value file. + * + * @return the fixed width to use when parsing the file. + */ + public int getFixedValueWidth() { + return fixedValueWidth; + } + + public int[] getValueWidths() { + return valueWidths; + } + + /** + * Determines if the {@link DataSet#next()} should throw an exception in + * case of inconsistent line width in the fixed width value file. + * + * @return a boolean indicating whether or not to fail on inconsistent line + * widths. + */ + public boolean isFailOnInconsistentLineWidth() { + return failOnInconsistentLineWidth; + } + + @Override + protected void decorateIdentity(List<Object> identifiers) { + identifiers.add(columnNameLineNumber); + identifiers.add(encoding); + identifiers.add(fixedValueWidth); + identifiers.add(valueWidths); + identifiers.add(failOnInconsistentLineWidth); + } + + @Override + public String toString() { + return "FixedWidthConfiguration[encoding=" + encoding + + ", fixedValueWidth=" + fixedValueWidth + ", valueWidths=" + + Arrays.toString(valueWidths) + ", columnNameLineNumber=" + + columnNameLineNumber + ", failOnInconsistentLineWidth=" + + failOnInconsistentLineWidth + "]"; + } + + public boolean isConstantValueWidth() { + return fixedValueWidth != -1; + } + + public int getValueWidth(int columnIndex) { + if (isConstantValueWidth()) { + return fixedValueWidth; + } + return valueWidths[columnIndex]; + } +} http://git-wip-us.apache.org/repos/asf/metamodel/blob/47c6d56c/fixedwidth/src/main/java/org/apache/metamodel/fixedwidth/FixedWidthConfigurationReader.java ---------------------------------------------------------------------- diff --git a/fixedwidth/src/main/java/org/apache/metamodel/fixedwidth/FixedWidthConfigurationReader.java b/fixedwidth/src/main/java/org/apache/metamodel/fixedwidth/FixedWidthConfigurationReader.java new file mode 100644 index 0000000..9154e5e --- /dev/null +++ b/fixedwidth/src/main/java/org/apache/metamodel/fixedwidth/FixedWidthConfigurationReader.java @@ -0,0 +1,180 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ +package org.apache.metamodel.fixedwidth; + +import java.io.BufferedReader; +import java.io.InputStream; +import java.io.InputStreamReader; +import java.util.ArrayList; +import java.util.HashMap; +import java.util.LinkedHashMap; +import java.util.List; +import java.util.Map; +import java.util.Map.Entry; +import java.util.regex.Matcher; +import java.util.regex.Pattern; + +import org.apache.metamodel.csv.CsvConfiguration; +import org.apache.metamodel.csv.CsvDataContext; +import org.apache.metamodel.data.DataSet; +import org.apache.metamodel.schema.Table; +import org.apache.metamodel.util.Action; +import org.apache.metamodel.util.Resource; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +/** + * Object capable of reading fixed width metadata from external sources and + * thereby producing an appropriate {@link FixedWidthConfiguration} to use with + * a {@link FixedWidthDataContext}. + */ +public class FixedWidthConfigurationReader { + + private static final Logger logger = LoggerFactory.getLogger(FixedWidthConfigurationReader.class); + + // example: @1 COL1 $char1. + private final Pattern PATTERN_SAS_INPUT_LINE = Pattern.compile("\\@(\\d+) (.+) .*?(\\d+)\\."); + + // example: COL1 "Record type" + private final Pattern PATTERN_SAS_LABEL_LINE = Pattern.compile("(.+) \\\"(.+)\\\""); + + /** + * Reads a {@link FixedWidthConfiguration} based on a SAS 'format file', + * <a href= + * "http://support.sas.com/documentation/cdl/en/etlug/67323/HTML/default/viewer.htm#p0h03yig7fp1qan1arghp3lwjqi6.htm";> + * described here</a>. + * + * @param encoding + * @param resource + * the format file resource + * @param failOnInconsistentLineWidth + * @return a {@link FixedWidthConfiguration} object to use + */ + public FixedWidthConfiguration readFromSasFormatFile(String encoding, Resource resource, + boolean failOnInconsistentLineWidth) { + final List<FixedWidthColumnSpec> columnSpecs = new ArrayList<>(); + + final CsvDataContext dataContext = new CsvDataContext(resource, new CsvConfiguration()); + final Table table = dataContext.getDefaultSchema().getTable(0); + try (final DataSet dataSet = dataContext.query().from(table).select("Name", "BeginPosition", "EndPosition") + .execute()) { + while (dataSet.next()) { + final String name = (String) dataSet.getRow().getValue(0); + final int beginPosition = Integer.parseInt((String) dataSet.getRow().getValue(1)); + final int endPosition = Integer.parseInt((String) dataSet.getRow().getValue(2)); + final int width = 1 + endPosition - beginPosition; + columnSpecs.add(new FixedWidthColumnSpec(name, width)); + } + } + + return new FixedWidthConfiguration(encoding, columnSpecs, failOnInconsistentLineWidth); + } + + /** + * Reads a {@link FixedWidthConfiguration} based on a SAS INPUT declaration. + * The reader method also optionally will look for a LABEL defintion for + * column naming. + * + * @param encoding + * @param resource + * the format file resource + * @param failOnInconsistentLineWidth + * @return a {@link FixedWidthConfiguration} object to use + */ + public FixedWidthConfiguration readFromSasInputDefinition(String encoding, Resource resource, + boolean failOnInconsistentLineWidth) { + + final Map<String, Integer> inputWidthDeclarations = new LinkedHashMap<>(); + final Map<String, String> labelDeclarations = new HashMap<>(); + + resource.read(new Action<InputStream>() { + + private boolean inInputSection = false; + private boolean inLabelSection = false; + + @Override + public void run(InputStream in) throws Exception { + try (final BufferedReader reader = new BufferedReader(new InputStreamReader(in))) { + for (String line = reader.readLine(); line != null; line = reader.readLine()) { + processLine(line); + } + } + } + + private void processLine(String line) { + line = line.trim(); + if (line.isEmpty()) { + return; + } + if (";".equals(line)) { + inInputSection = false; + inLabelSection = false; + return; + } else if ("INPUT".equals(line)) { + inInputSection = true; + return; + } else if ("LABEL".equals(line)) { + inLabelSection = true; + return; + } + + if (inInputSection) { + final Matcher matcher = PATTERN_SAS_INPUT_LINE.matcher(line); + if (matcher.matches()) { + final String positionSpec = matcher.group(1); + final String nameSpec = matcher.group(2); + final int width = Integer.parseInt(matcher.group(3)); + logger.debug("Parsed INPUT line \"{}\": position={}, name={}, width={}", line, positionSpec, + nameSpec, width); + inputWidthDeclarations.put(nameSpec, width); + } else { + logger.debug("Failed to parse/recognize INPUT line \"{}\"", line); + } + } else if (inLabelSection) { + final Matcher matcher = PATTERN_SAS_LABEL_LINE.matcher(line); + if (matcher.matches()) { + final String nameSpec = matcher.group(1); + final String labelSpec = matcher.group(2); + logger.debug("Parsed LABEL line \"{}\": name={}, label={}", line, nameSpec, labelSpec); + labelDeclarations.put(nameSpec, labelSpec); + } else { + logger.debug("Failed to parse/recognize LABEL line \"{}\"", line); + } + } + + if (line.endsWith(";")) { + inInputSection = false; + inLabelSection = false; + } + } + }); + + final List<FixedWidthColumnSpec> columnSpecs = new ArrayList<>(); + for (Entry<String, Integer> entry : inputWidthDeclarations.entrySet()) { + final String columnKey = entry.getKey(); + final Integer columnWidth = entry.getValue(); + final String columnLabel = labelDeclarations.get(columnKey); + final String columnName = columnLabel == null ? columnKey : columnLabel; + columnSpecs.add(new FixedWidthColumnSpec(columnName, columnWidth)); + } + + return new FixedWidthConfiguration(encoding, columnSpecs, failOnInconsistentLineWidth); + } + +} http://git-wip-us.apache.org/repos/asf/metamodel/blob/47c6d56c/fixedwidth/src/test/java/org/apache/metamodel/fixedwidth/FixedWidthConfigurationReaderTest.java ---------------------------------------------------------------------- diff --git a/fixedwidth/src/test/java/org/apache/metamodel/fixedwidth/FixedWidthConfigurationReaderTest.java b/fixedwidth/src/test/java/org/apache/metamodel/fixedwidth/FixedWidthConfigurationReaderTest.java new file mode 100644 index 0000000..eb57233 --- /dev/null +++ b/fixedwidth/src/test/java/org/apache/metamodel/fixedwidth/FixedWidthConfigurationReaderTest.java @@ -0,0 +1,89 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ +package org.apache.metamodel.fixedwidth; + +import static org.junit.Assert.*; + +import java.util.Arrays; + +import org.apache.metamodel.DataContext; +import org.apache.metamodel.data.DataSet; +import org.apache.metamodel.schema.Table; +import org.apache.metamodel.util.FileResource; +import org.apache.metamodel.util.Resource; +import org.junit.Test; + +public class FixedWidthConfigurationReaderTest { + + private final FileResource dataResource = new FileResource("src/test/resources/metadata_spec1/data.txt"); + + @Test + public void testReadConfigurationFromSasFormatFile() throws Exception { + final FixedWidthConfigurationReader reader = new FixedWidthConfigurationReader(); + final Resource resource = new FileResource("src/test/resources/metadata_spec1/sas-formatfile-metadata.txt"); + assertTrue(resource.isExists()); + + final FixedWidthConfiguration configuration = reader.readFromSasFormatFile("UTF8", resource, false); + assertEquals("[1, 20, 2]", Arrays.toString(configuration.getValueWidths())); + + final FixedWidthDataContext dataContext = new FixedWidthDataContext(dataResource, configuration); + + performAssertionsOnSpec1(dataContext); + } + + @Test + public void testReadConfigurationFromSasInputMetadata() throws Exception { + final FixedWidthConfigurationReader reader = new FixedWidthConfigurationReader(); + final Resource resource = new FileResource("src/test/resources/metadata_spec1/sas-input-metadata.txt"); + assertTrue(resource.isExists()); + + final FixedWidthConfiguration configuration = reader.readFromSasInputDefinition("UTF8", resource, false); + assertEquals("[1, 20, 2]", Arrays.toString(configuration.getValueWidths())); + + final FixedWidthDataContext dataContext = new FixedWidthDataContext(dataResource, configuration); + + performAssertionsOnSpec1(dataContext); + } + + /** + * Shared assertions section once the 'metadata_spec1' {@link DataContext} + * has been loaded. + * + * @param dataContext + */ + private void performAssertionsOnSpec1(FixedWidthDataContext dataContext) { + final Table table = dataContext.getDefaultSchema().getTable(0); + final String[] columnNames = table.getColumnNames(); + assertEquals("[Record type, Description, Initials]", Arrays.toString(columnNames)); + + try (final DataSet dataSet = dataContext.query().from(table).selectAll().execute()) { + assertTrue(dataSet.next()); + assertEquals("Row[values=[P, Kasper Sorensen, KS]]", dataSet.getRow().toString()); + assertTrue(dataSet.next()); + assertEquals("Row[values=[C, Human Inference, HI]]", dataSet.getRow().toString()); + assertTrue(dataSet.next()); + assertEquals("Row[values=[P, Ankit Kumar, AK]]", dataSet.getRow().toString()); + assertTrue(dataSet.next()); + assertEquals("Row[values=[C, Stratio, S]]", dataSet.getRow().toString()); + assertTrue(dataSet.next()); + assertEquals("Row[values=[U, Unknown, ]]", dataSet.getRow().toString()); + assertFalse(dataSet.next()); + } + } +} http://git-wip-us.apache.org/repos/asf/metamodel/blob/47c6d56c/fixedwidth/src/test/resources/metadata_spec1/data.txt ---------------------------------------------------------------------- diff --git a/fixedwidth/src/test/resources/metadata_spec1/data.txt b/fixedwidth/src/test/resources/metadata_spec1/data.txt new file mode 100644 index 0000000..785a539 --- /dev/null +++ b/fixedwidth/src/test/resources/metadata_spec1/data.txt @@ -0,0 +1,5 @@ +PKasper Sorensen KS +CHuman Inference HI +PAnkit Kumar AK +CStratio S +UUnknown http://git-wip-us.apache.org/repos/asf/metamodel/blob/47c6d56c/fixedwidth/src/test/resources/metadata_spec1/sas-formatfile-metadata.txt ---------------------------------------------------------------------- diff --git a/fixedwidth/src/test/resources/metadata_spec1/sas-formatfile-metadata.txt b/fixedwidth/src/test/resources/metadata_spec1/sas-formatfile-metadata.txt new file mode 100644 index 0000000..9bbe411 --- /dev/null +++ b/fixedwidth/src/test/resources/metadata_spec1/sas-formatfile-metadata.txt @@ -0,0 +1,4 @@ +Name,SASColumnType,BeginPosition,EndPosition,ReadFlag,Desc,SASFormat,SASInformat +Record type,C,1,1,y,Record Type,$char.,$char. +Description,C,2,21,y,Description of record,$char.,$char. +Initials,C,22,23,y,Initials of record,, http://git-wip-us.apache.org/repos/asf/metamodel/blob/47c6d56c/fixedwidth/src/test/resources/metadata_spec1/sas-input-metadata.txt ---------------------------------------------------------------------- diff --git a/fixedwidth/src/test/resources/metadata_spec1/sas-input-metadata.txt b/fixedwidth/src/test/resources/metadata_spec1/sas-input-metadata.txt new file mode 100644 index 0000000..f12e418 --- /dev/null +++ b/fixedwidth/src/test/resources/metadata_spec1/sas-input-metadata.txt @@ -0,0 +1,19 @@ +INPUT + + @1 COL1 $char1. + + @2 COL2 $char20. + + @22 COL3 $char2. + +; + +LABEL + + COL1 "Record type" + + COL2 "Description" + + COL3 "Initials" + +;
