[jira] [Commented] (DRILL-8207) Fix Username Typo in JDBC SerDe
[ https://issues.apache.org/jira/browse/DRILL-8207?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17532534#comment-17532534 ] ASF GitHub Bot commented on DRILL-8207: --- cgivre merged PR #2530: URL: https://github.com/apache/drill/pull/2530 > Fix Username Typo in JDBC SerDe > --- > > Key: DRILL-8207 > URL: https://issues.apache.org/jira/browse/DRILL-8207 > Project: Apache Drill > Issue Type: Bug >Reporter: Charles Givre >Assignee: Charles Givre >Priority: Blocker > > Fixes SerDe error with default JDBC plugin. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Commented] (DRILL-8207) Fix Username Typo in JDBC SerDe
[ https://issues.apache.org/jira/browse/DRILL-8207?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17532449#comment-17532449 ] ASF GitHub Bot commented on DRILL-8207: --- cgivre opened a new pull request, #2530: URL: https://github.com/apache/drill/pull/2530 # [DRILL-8207](https://issues.apache.org/jira/browse/DRILL-8207): Fix Username Typo in JDBC SerDe ## Description Fixes a Serialization/Deserialization bug in the JDBC plugin ## Documentation No user facing changes. ## Testing Ran unit tests > Fix Username Typo in JDBC SerDe > --- > > Key: DRILL-8207 > URL: https://issues.apache.org/jira/browse/DRILL-8207 > Project: Apache Drill > Issue Type: Bug >Reporter: Charles Givre >Assignee: Charles Givre >Priority: Blocker > > Fixes SerDe error with default JDBC plugin. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Created] (DRILL-8207) Fix Username Typo in JDBC SerDe
Charles Givre created DRILL-8207: Summary: Fix Username Typo in JDBC SerDe Key: DRILL-8207 URL: https://issues.apache.org/jira/browse/DRILL-8207 Project: Apache Drill Issue Type: Bug Reporter: Charles Givre Assignee: Charles Givre Fixes SerDe error with default JDBC plugin. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Resolved] (DRILL-8204) Allow Provided Schema for HTTP Plugin in JSON Mode
[
https://issues.apache.org/jira/browse/DRILL-8204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vitalii Diravka resolved DRILL-8204.
Resolution: Fixed
> Allow Provided Schema for HTTP Plugin in JSON Mode
> --
>
> Key: DRILL-8204
> URL: https://issues.apache.org/jira/browse/DRILL-8204
> Project: Apache Drill
> Issue Type: Improvement
> Components: Storage - Other
>Affects Versions: 1.20.0
>Reporter: Charles Givre
>Assignee: Charles Givre
>Priority: Major
> Fix For: 2.0.0
>
>
> One of the challenges of querying APIs is inconsistent data. Drill allows you
> to provide a schema for individual endpoints. You can do this in one of two
> ways: either by
> providing a serialized TupleMetadata of the desired schema. This is an
> advanced functionality and should only be used by advanced Drill users.
> The schema provisioning currently supports complex types of Arrays and Maps
> at any nesting level.
> ### Example Schema Provisioning:
> ```json
> "jsonOptions": {
> "providedSchema": [
> {
> "fieldName": "int_field",
> "fieldType": "bigint"
> }, {
> "fieldName": "jsonField",
> "fieldType": "varchar",
> "properties": {
> "drill.json-mode":"json"
> }
> },{
> // Array field
> "fieldName": "stringField",
> "fieldType": "varchar",
> "isArray": true
> }, {
> // Map field
> "fieldName": "mapField",
> "fieldType": "map",
> "fields": [
> {
> "fieldName": "nestedField",
> "fieldType": "int"
> },{
> "fieldName": "nestedField2",
> "fieldType": "varchar"
> }
> ]
> }
> ]
> }
> ```
> ### Example Provisioning the Schema with a JSON String
> ```json
> "jsonOptions": {
> "jsonSchema":
> "\{\"type\":\"tuple_schema\",\"columns\":[{\"name\":\"outer_map\",\"type\":\"STRUCT<`int_field`
> BIGINT, `int_array` ARRAY>\",\"mode\":\"REQUIRED\"}]}"
> }
> ```
> You can print out a JSON string of a schema with the Java code below.
> ```java
> TupleMetadata schema = new SchemaBuilder()
> .addNullable("a", MinorType.BIGINT)
> .addNullable("m", MinorType.VARCHAR)
> .build();
> ColumnMetadata m = schema.metadata("m");
> m.setProperty(JsonLoader.JSON_MODE, JsonLoader.JSON_LITERAL_MODE);
> System.out.println(schema.jsonString());
> ```
> This will generate something like the JSON string below:
> ```json
> {
> "type":"tuple_schema",
> "columns":[
> {"name":"a","type":"BIGINT","mode":"OPTIONAL"},
> {"name":"m","type":"VARCHAR","mode":"OPTIONAL","properties":\{"drill.json-mode":"json"}
> }
> ]
> }
> ```
> ## Dealing With Inconsistent Schemas
> One of the major challenges of interacting with JSON data is when the schema
> is inconsistent. Drill has a `UNION` data type which is marked as
> experimental. At the time of
> writing, the HTTP plugin does not support the `UNION`, however supplying a
> schema can solve a lot of those issues.
> ### Json Mode
> Drill offers the option of reading all JSON values as a string. While this
> can complicate downstream analytics, it can also be a more memory-efficient
> way of reading data with
> inconsistent schema. Unfortunately, at the time of writing, JSON-mode is only
> available with a provided schema. However, future work will allow this mode
> to be enabled for
> any JSON data.
> Enabling JSON Mode:
> You can enable JSON mode simply by adding the `drill.json-mode` property with
> a value of `json` to a field, as shown below:
> ```json
> {
> "fieldName": "jsonField",
> "fieldType": "varchar",
> "properties": {
> "drill.json-mode": "json"
> }
> }
> ```
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[jira] [Resolved] (DRILL-8178) Bump S3 SDK to Lastest Version
[ https://issues.apache.org/jira/browse/DRILL-8178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vitalii Diravka resolved DRILL-8178. Resolution: Fixed > Bump S3 SDK to Lastest Version > -- > > Key: DRILL-8178 > URL: https://issues.apache.org/jira/browse/DRILL-8178 > Project: Apache Drill > Issue Type: Task > Components: Storage - Other >Affects Versions: 1.20.0 >Reporter: Charles Givre >Assignee: Charles Givre >Priority: Minor > Fix For: 2.0.0 > > -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Commented] (DRILL-8172) Use the specified memory usage for Travis CI
[ https://issues.apache.org/jira/browse/DRILL-8172?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17532212#comment-17532212 ] ASF GitHub Bot commented on DRILL-8172: --- vdiravka commented on code in PR #2500: URL: https://github.com/apache/drill/pull/2500#discussion_r865819857 ## .travis.yml: ## @@ -45,6 +45,9 @@ cache: before_install: - export JAVA_HOME="/usr/lib/jvm/java-8-openjdk-arm64" - export PATH="$JAVA_HOME/bin:$PATH" + - export MEMORYMB=2048 Review Comment: I think 2048 is small value. The default one is 2500M. Now Travis build hangs: ``` [INFO] Running org.apache.drill.exec.fn.impl.TestAggregateFunctions No output has been received in the last 10m0s, this potentially indicates a stalled build or something wrong with the build itself. ``` Possibly it is due to lack of heap memory. @luocooong What do you think about reverting this value to the default one? ## .travis.yml: ## @@ -45,6 +45,9 @@ cache: before_install: - export JAVA_HOME="/usr/lib/jvm/java-8-openjdk-arm64" - export PATH="$JAVA_HOME/bin:$PATH" + - export MEMORYMB=2048 + - export DIRECTMEMORYMB=5120 Review Comment: One note about `DIRECTMEMORYMB`. Do you think `4500M` is really small? GitHub Actions has the same [7Gb](https://docs.github.com/en/actions/using-github-hosted-runners/about-github-hosted-runners#supported-runners-and-hardware-resources) memory limit and these builds pass amost always. > Use the specified memory usage for Travis CI > > > Key: DRILL-8172 > URL: https://issues.apache.org/jira/browse/DRILL-8172 > Project: Apache Drill > Issue Type: Bug > Components: Tools, Build & Test >Reporter: Cong Luo >Assignee: Cong Luo >Priority: Major > > Unlike GitHub CI, Travis CI may have more memory. If so, we can run unit > tests using various policies to prevent failure. -- This message was sent by Atlassian Jira (v8.20.7#820007)
