[jira] [Commented] (DRILL-7657) Invalid internal link
[ https://issues.apache.org/jira/browse/DRILL-7657?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17066324#comment-17066324 ] Aaron-Mhs commented on DRILL-7657: -- [~volodymyr] , I am thinking about a question, how do I open this new pull request on the branch you mentioned([https://github.com/apache/drill/tree/gh-pages]), because my current problem is that I cannot open the link address in the document, but I do n’t know what the content of the link really is. > Invalid internal link > - > > Key: DRILL-7657 > URL: https://issues.apache.org/jira/browse/DRILL-7657 > Project: Apache Drill > Issue Type: Bug > Components: Security >Affects Versions: 1.17.0 >Reporter: Aaron-Mhs >Assignee: Aaron-Mhs >Priority: Major > Attachments: image-2020-03-24-14-57-53-568.png, > image-2020-03-24-14-58-17-865.png > > > There is an invalid link (Prerequisites-> See Enabling Authentication and > Encryption.) In the Configuring Kerberos Security module in the document. > After opening, it displays 404 (The requested URL was not found on this > server.), And I hope to fix it as soon as possible. > !image-2020-03-24-14-57-53-568.png! > !image-2020-03-24-14-58-17-865.png! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (DRILL-7660) Modify Drill Dockerfiles
Abhishek Girish created DRILL-7660: -- Summary: Modify Drill Dockerfiles Key: DRILL-7660 URL: https://issues.apache.org/jira/browse/DRILL-7660 Project: Apache Drill Issue Type: Sub-task Reporter: Abhishek Girish Assignee: Abhishek Girish -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (DRILL-7659) Add support for Helm Charts based deployment on Kubernetes
Abhishek Girish created DRILL-7659: -- Summary: Add support for Helm Charts based deployment on Kubernetes Key: DRILL-7659 URL: https://issues.apache.org/jira/browse/DRILL-7659 Project: Apache Drill Issue Type: Sub-task Reporter: Abhishek Girish Assignee: Abhishek Girish -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (DRILL-7658) Vector allocateNew() has poor error reporting
Paul Rogers created DRILL-7658:
--
Summary: Vector allocateNew() has poor error reporting
Key: DRILL-7658
URL: https://issues.apache.org/jira/browse/DRILL-7658
Project: Apache Drill
Issue Type: Bug
Affects Versions: 1.17.0
Reporter: Paul Rogers
See posting by Charles on 2020-03-24 on the user and dev lists of a message
forwarded from another user where a query ran out of memory. Stack trace:
{noformat}
Caused by: org.apache.drill.exec.exception.OutOfMemoryException: null
at
org.apache.drill.exec.vector.complex.AbstractContainerVector.allocateNew(AbstractContainerVector.java:59)
at
org.apache.drill.exec.test.generated.PartitionerGen5$OutgoingRecordBatch.allocateOutgoingRecordBatch(PartitionerTemplate.
{noformat}
Notice the complete lack of context. The method in question:
{code:java}
public void allocateNew() throws OutOfMemoryException {
if (!allocateNewSafe()) {
throw new OutOfMemoryException();
}
}
{code}
A generated implementation of the {{allocateNewSafe()}} method:
{code:java}
@Override
public boolean allocateNewSafe() {
long curAllocationSize = allocationSizeInBytes;
if (allocationMonitor > 10) {
curAllocationSize = Math.max(8, curAllocationSize / 2);
allocationMonitor = 0;
} else if (allocationMonitor < -2) {
curAllocationSize = allocationSizeInBytes * 2L;
allocationMonitor = 0;
}
try{
allocateBytes(curAllocationSize);
} catch (DrillRuntimeException ex) {
return false;
}
return true;
}
{code}
Note that the {{allocateNew()}} method is not "safe" (it throws an exception),
but it does so by discarding the underlying exception. What should happen is
that the "non-safe" {{allocateNew()}} should call the {{allocateBytes()}}
method and simply forward the {{DrillRuntimeException}}. It probably does not
do so because the author wanted to reuse the extra size calcs in
{{allocateNewSafe()}}.
The solution is to put the calcs and the call to {{allocateBytes()}} in a
"non-safe" method, and call that entire method from {{allocateNew()}} and
{{allocateNewSafe()}}. Or, better, generate {{allocateNew()}} using the above
code, but have the base class define {{allocateNewSafe()}} as a wrapper.
Note an extra complexity: although the base class provides the method shown
above, each generated vector also provides:
{code:java}
@Override
public void allocateNew() {
if (!allocateNewSafe()) {
throw new OutOfMemoryException("Failure while allocating buffer.");
}
}
{code}
Which is both redundant and inconsistent (one has a message, the other does
not.)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (DRILL-7641) Convert Excel Reader to Use Streaming Reader
[ https://issues.apache.org/jira/browse/DRILL-7641?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17066212#comment-17066212 ] Arina Ielchiieva commented on DRILL-7641: - [~cgivre] sorry made this by mistake. > Convert Excel Reader to Use Streaming Reader > > > Key: DRILL-7641 > URL: https://issues.apache.org/jira/browse/DRILL-7641 > Project: Apache Drill > Issue Type: Improvement > Components: Storage - Text CSV >Affects Versions: 1.17.0 >Reporter: Charles Givre >Assignee: Charles Givre >Priority: Major > Fix For: 1.18.0 > > > The current implementation of the Excel reader uses the Apache POI reader, > which uses excessive amounts of memory. As a result, attempting to read large > Excel files will cause out of memory errors. > This PR converts the format plugin to use a streaming reader, based still on > the POI library. The documentation for the streaming reader can be found > here. [1] > All unit tests pass and I tested the plugin with some large Excel files on my > computer. > [1]: [https://github.com/pjfanning/excel-streaming-reader] > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (DRILL-7641) Convert Excel Reader to Use Streaming Reader
[ https://issues.apache.org/jira/browse/DRILL-7641?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arina Ielchiieva updated DRILL-7641: Fix Version/s: 1.18.0 > Convert Excel Reader to Use Streaming Reader > > > Key: DRILL-7641 > URL: https://issues.apache.org/jira/browse/DRILL-7641 > Project: Apache Drill > Issue Type: Improvement > Components: Storage - Text CSV >Affects Versions: 1.17.0 >Reporter: Charles Givre >Assignee: Charles Givre >Priority: Major > Fix For: 1.18.0 > > > The current implementation of the Excel reader uses the Apache POI reader, > which uses excessive amounts of memory. As a result, attempting to read large > Excel files will cause out of memory errors. > This PR converts the format plugin to use a streaming reader, based still on > the POI library. The documentation for the streaming reader can be found > here. [1] > All unit tests pass and I tested the plugin with some large Excel files on my > computer. > [1]: [https://github.com/pjfanning/excel-streaming-reader] > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (DRILL-7641) Convert Excel Reader to Use Streaming Reader
[ https://issues.apache.org/jira/browse/DRILL-7641?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17066199#comment-17066199 ] Charles Givre commented on DRILL-7641: -- [~arina], Why was the fix version changed? I believe this PR is reviewable and is not a major change. I don't think this will be a major task to get this into Drill 1.18. > Convert Excel Reader to Use Streaming Reader > > > Key: DRILL-7641 > URL: https://issues.apache.org/jira/browse/DRILL-7641 > Project: Apache Drill > Issue Type: Improvement > Components: Storage - Text CSV >Affects Versions: 1.17.0 >Reporter: Charles Givre >Assignee: Charles Givre >Priority: Major > > The current implementation of the Excel reader uses the Apache POI reader, > which uses excessive amounts of memory. As a result, attempting to read large > Excel files will cause out of memory errors. > This PR converts the format plugin to use a streaming reader, based still on > the POI library. The documentation for the streaming reader can be found > here. [1] > All unit tests pass and I tested the plugin with some large Excel files on my > computer. > [1]: [https://github.com/pjfanning/excel-streaming-reader] > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (DRILL-6851) Create Drill Operator for Kubernetes
[ https://issues.apache.org/jira/browse/DRILL-6851?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Abhishek Girish updated DRILL-6851: --- Parent: DRILL-6598 Issue Type: Sub-task (was: Task) > Create Drill Operator for Kubernetes > > > Key: DRILL-6851 > URL: https://issues.apache.org/jira/browse/DRILL-6851 > Project: Apache Drill > Issue Type: Sub-task >Reporter: Abhishek Girish >Assignee: Abhishek Girish >Priority: Major > > This task is to track creating an initial version of the Drill Operator for > Kubernetes. I'll shortly update the JIRA on background, details on Operator, > and what's planned for the first version. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (DRILL-7563) Docker & Kubernetes Drill server container
[
https://issues.apache.org/jira/browse/DRILL-7563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17066097#comment-17066097
]
Abhishek Girish commented on DRILL-7563:
[~cgivre], once we get my changes in to Drill GitHub repo, I think we can work
on the docs / tutorials. I also have some material which can help for it.
> Docker & Kubernetes Drill server container
> --
>
> Key: DRILL-7563
> URL: https://issues.apache.org/jira/browse/DRILL-7563
> Project: Apache Drill
> Issue Type: Improvement
>Affects Versions: 1.17.0
>Reporter: Paul Rogers
>Assignee: Paul Rogers
>Priority: Major
>
> Drill provides two Docker containers:
> * [Build Drill from
> sources|https://github.com/apache/drill/blob/master/Dockerfile]
> * [Run Drill in interactive embedded
> mode|https://github.com/apache/drill/blob/master/distribution/Dockerfile]
> User feedback suggests that these are not quite the right solutions to run
> Drill in a K8s (or OpenShift) cluster. In addition, we need a container to
> run a Drill server. This ticket summarizes the tasks involved.
> h4. Container Image
> The container image should:
> * Start with the OpenJDK base image with minimal extra packages.
> * Download and install an official Drill release.
> We may then want to provide two derived images:
> The Drillbit image which:
> * Configures Drill for production and as needed in the following steps.
> * Provides entry points for the Drillbit and for Sqlline
> * Exposes Drill's four ports
> * Accept as parameters things like the ZK host IP(s).
> The Sqlline image, meant to be run in interactive mode (like the current
> embedded image) and which:
> * Accept as parameters the ZK host IP(s).
> Both should be published to the official Drill DockerHub account:
> https://hub.docker.com/r/apache/drill
> h4. Runtime Environment
> Drill has very few dependencies, but it must have a running ZK.
> * Start a [ZK container|https://hub.docker.com/_/zookeeper/].
> * A place to store logs, which can be in the container by default, stored on
> the host file system via a volume mount.
> * Access to a data source, which can be configured via a storage plugin
> stored in ZK.
> * Ensure graceful shutdown integration with the Docker shutdown mechanism.
> h4. Running Drill in Docker
> Users must run at least one Drillbit, and may run more. Users may want to run
> Sqlline.
> * The Drillbit container requires, at a minimum, the IP address of the ZK
> instance(s).
> * The Sqlline container requires only the ZK instances, from which it can
> find the Drillbit.
> Uses will want to customize some parts of Drill: at least memory, perhaps any
> of the other options. Provide a way to pass this information into the
> container to avoid the need to rebuild the container to change configuration.
> h4. Running Drill in K8s
> The containers should be usable in "plain" Docker. Today, however, many
> people use K8s to orchestrate Docker. Thus, the Drillbit (but probably not
> the Sqlline) container should be designed to work with K8s. An example set of
> K8s YAML files should illustrate:
> * Create a host-mount file system to capture Drill logs and query profiles.
> * Optionally write Drill logs to stdout, to be captured by {{fluentd}} or
> similar tools.
> * Pass Drill configuration (both HOCON and envrironment) as config maps.
> * Pass ZK as an environment variable (the value of which would, one presumes,
> come from some kind of service discovery system.)
> The result is that the user should be able to manually tinker with the YAML
> files, then use {{kubeadm}} to launch, monitor and stop Drill. The user sets
> cluster size manually by launching the desired number of Drill pods.
> h4. Helm Chart for Drill
> The next step is to wrap the YAML files in a Helm chart, with parameters
> exposed for the config options noted above.
> h4. Drill Operator for K8s
>
> Full K8s integration will require an operator to manage the Drill cluster.
> K8s operators are often written in Go, though doing so is not necessary.
> Drill already includes Drill-on-YARN which is, essential a "YARN operator."
> Repurpose this code to work with K8s as the target cluster manager rather
> than YARN. Reuse the same operations from DoY: configure, start, resize and
> stop a cluster.
> h4. Security
> The above steps provide an "MVP": minimum viable project - it will run Drill
> with standard options in the various environments. Users who chose to run
> Drill in production will likely require additional security settings. Enable
> SSL? Control ingress? We need to understand what is needed, what Drill
> offers, and how to enable Drill's security features in a containerized
> environment.
> h4. Production Deployments
> With Docker and K8s the old maxim "the devil is in the details" is true
[jira] [Commented] (DRILL-7563) Docker & Kubernetes Drill server container
[
https://issues.apache.org/jira/browse/DRILL-7563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17066095#comment-17066095
]
Dobes Vandermeer commented on DRILL-7563:
-
[~cgivre]
I don't have time to write a tutorial, sorry.
[~rjaimes]
Those yaml files were enough to get drill running, but I haven't tested them
much. A few times I had problems with zookeeper not coming back online, I think
it was some kind of DNS resolution issue, I tried to fix it but I can't be sure
I was successful yet. I never reached the point that I was confident that I
could run a zookeeper & drill setup on kubernetes without worrying it would
fail on me at an inopportune moment.
> Docker & Kubernetes Drill server container
> --
>
> Key: DRILL-7563
> URL: https://issues.apache.org/jira/browse/DRILL-7563
> Project: Apache Drill
> Issue Type: Improvement
>Affects Versions: 1.17.0
>Reporter: Paul Rogers
>Assignee: Paul Rogers
>Priority: Major
>
> Drill provides two Docker containers:
> * [Build Drill from
> sources|https://github.com/apache/drill/blob/master/Dockerfile]
> * [Run Drill in interactive embedded
> mode|https://github.com/apache/drill/blob/master/distribution/Dockerfile]
> User feedback suggests that these are not quite the right solutions to run
> Drill in a K8s (or OpenShift) cluster. In addition, we need a container to
> run a Drill server. This ticket summarizes the tasks involved.
> h4. Container Image
> The container image should:
> * Start with the OpenJDK base image with minimal extra packages.
> * Download and install an official Drill release.
> We may then want to provide two derived images:
> The Drillbit image which:
> * Configures Drill for production and as needed in the following steps.
> * Provides entry points for the Drillbit and for Sqlline
> * Exposes Drill's four ports
> * Accept as parameters things like the ZK host IP(s).
> The Sqlline image, meant to be run in interactive mode (like the current
> embedded image) and which:
> * Accept as parameters the ZK host IP(s).
> Both should be published to the official Drill DockerHub account:
> https://hub.docker.com/r/apache/drill
> h4. Runtime Environment
> Drill has very few dependencies, but it must have a running ZK.
> * Start a [ZK container|https://hub.docker.com/_/zookeeper/].
> * A place to store logs, which can be in the container by default, stored on
> the host file system via a volume mount.
> * Access to a data source, which can be configured via a storage plugin
> stored in ZK.
> * Ensure graceful shutdown integration with the Docker shutdown mechanism.
> h4. Running Drill in Docker
> Users must run at least one Drillbit, and may run more. Users may want to run
> Sqlline.
> * The Drillbit container requires, at a minimum, the IP address of the ZK
> instance(s).
> * The Sqlline container requires only the ZK instances, from which it can
> find the Drillbit.
> Uses will want to customize some parts of Drill: at least memory, perhaps any
> of the other options. Provide a way to pass this information into the
> container to avoid the need to rebuild the container to change configuration.
> h4. Running Drill in K8s
> The containers should be usable in "plain" Docker. Today, however, many
> people use K8s to orchestrate Docker. Thus, the Drillbit (but probably not
> the Sqlline) container should be designed to work with K8s. An example set of
> K8s YAML files should illustrate:
> * Create a host-mount file system to capture Drill logs and query profiles.
> * Optionally write Drill logs to stdout, to be captured by {{fluentd}} or
> similar tools.
> * Pass Drill configuration (both HOCON and envrironment) as config maps.
> * Pass ZK as an environment variable (the value of which would, one presumes,
> come from some kind of service discovery system.)
> The result is that the user should be able to manually tinker with the YAML
> files, then use {{kubeadm}} to launch, monitor and stop Drill. The user sets
> cluster size manually by launching the desired number of Drill pods.
> h4. Helm Chart for Drill
> The next step is to wrap the YAML files in a Helm chart, with parameters
> exposed for the config options noted above.
> h4. Drill Operator for K8s
>
> Full K8s integration will require an operator to manage the Drill cluster.
> K8s operators are often written in Go, though doing so is not necessary.
> Drill already includes Drill-on-YARN which is, essential a "YARN operator."
> Repurpose this code to work with K8s as the target cluster manager rather
> than YARN. Reuse the same operations from DoY: configure, start, resize and
> stop a cluster.
> h4. Security
> The above steps provide an "MVP": minimum viable project - it will run Drill
> with standard options in the various environments. Users who chose
[jira] [Commented] (DRILL-7633) Fixes for union and repeated list accessors
[ https://issues.apache.org/jira/browse/DRILL-7633?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17066089#comment-17066089 ] ASF GitHub Bot commented on DRILL-7633: --- paul-rogers commented on issue #2018: DRILL-7633: Fixes for union and repeated list accessors URL: https://github.com/apache/drill/pull/2018#issuecomment-603434987 @arina-ielchiieva, thanks for the suggestions, they solved the problem. I suspect we have some confusion with those methods based on earlier testing, but we'll fix those issues later. Squashed commits and reran all unit tests. We should be good to go. Thanks again for your help. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Fixes for union and repeated list accessors > --- > > Key: DRILL-7633 > URL: https://issues.apache.org/jira/browse/DRILL-7633 > Project: Apache Drill > Issue Type: Improvement >Affects Versions: 1.17.0 >Reporter: Paul Rogers >Assignee: Paul Rogers >Priority: Minor > Fix For: 1.18.0 > > > Minor fixes for repeated list and Union type support in column accessors -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (DRILL-7563) Docker & Kubernetes Drill server container
[
https://issues.apache.org/jira/browse/DRILL-7563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17066086#comment-17066086
]
Abhishek Girish commented on DRILL-7563:
I've added support for auto-scaling and I've tested that it works well. Please
see: https://github.com/Agirish/drill-helm-charts#autoscaling-drill-clusters
I have a script to test this:
https://github.com/Agirish/drill-helm-charts/blob/master/scripts/runCPULoadTest.sh
> Docker & Kubernetes Drill server container
> --
>
> Key: DRILL-7563
> URL: https://issues.apache.org/jira/browse/DRILL-7563
> Project: Apache Drill
> Issue Type: Improvement
>Affects Versions: 1.17.0
>Reporter: Paul Rogers
>Assignee: Paul Rogers
>Priority: Major
>
> Drill provides two Docker containers:
> * [Build Drill from
> sources|https://github.com/apache/drill/blob/master/Dockerfile]
> * [Run Drill in interactive embedded
> mode|https://github.com/apache/drill/blob/master/distribution/Dockerfile]
> User feedback suggests that these are not quite the right solutions to run
> Drill in a K8s (or OpenShift) cluster. In addition, we need a container to
> run a Drill server. This ticket summarizes the tasks involved.
> h4. Container Image
> The container image should:
> * Start with the OpenJDK base image with minimal extra packages.
> * Download and install an official Drill release.
> We may then want to provide two derived images:
> The Drillbit image which:
> * Configures Drill for production and as needed in the following steps.
> * Provides entry points for the Drillbit and for Sqlline
> * Exposes Drill's four ports
> * Accept as parameters things like the ZK host IP(s).
> The Sqlline image, meant to be run in interactive mode (like the current
> embedded image) and which:
> * Accept as parameters the ZK host IP(s).
> Both should be published to the official Drill DockerHub account:
> https://hub.docker.com/r/apache/drill
> h4. Runtime Environment
> Drill has very few dependencies, but it must have a running ZK.
> * Start a [ZK container|https://hub.docker.com/_/zookeeper/].
> * A place to store logs, which can be in the container by default, stored on
> the host file system via a volume mount.
> * Access to a data source, which can be configured via a storage plugin
> stored in ZK.
> * Ensure graceful shutdown integration with the Docker shutdown mechanism.
> h4. Running Drill in Docker
> Users must run at least one Drillbit, and may run more. Users may want to run
> Sqlline.
> * The Drillbit container requires, at a minimum, the IP address of the ZK
> instance(s).
> * The Sqlline container requires only the ZK instances, from which it can
> find the Drillbit.
> Uses will want to customize some parts of Drill: at least memory, perhaps any
> of the other options. Provide a way to pass this information into the
> container to avoid the need to rebuild the container to change configuration.
> h4. Running Drill in K8s
> The containers should be usable in "plain" Docker. Today, however, many
> people use K8s to orchestrate Docker. Thus, the Drillbit (but probably not
> the Sqlline) container should be designed to work with K8s. An example set of
> K8s YAML files should illustrate:
> * Create a host-mount file system to capture Drill logs and query profiles.
> * Optionally write Drill logs to stdout, to be captured by {{fluentd}} or
> similar tools.
> * Pass Drill configuration (both HOCON and envrironment) as config maps.
> * Pass ZK as an environment variable (the value of which would, one presumes,
> come from some kind of service discovery system.)
> The result is that the user should be able to manually tinker with the YAML
> files, then use {{kubeadm}} to launch, monitor and stop Drill. The user sets
> cluster size manually by launching the desired number of Drill pods.
> h4. Helm Chart for Drill
> The next step is to wrap the YAML files in a Helm chart, with parameters
> exposed for the config options noted above.
> h4. Drill Operator for K8s
>
> Full K8s integration will require an operator to manage the Drill cluster.
> K8s operators are often written in Go, though doing so is not necessary.
> Drill already includes Drill-on-YARN which is, essential a "YARN operator."
> Repurpose this code to work with K8s as the target cluster manager rather
> than YARN. Reuse the same operations from DoY: configure, start, resize and
> stop a cluster.
> h4. Security
> The above steps provide an "MVP": minimum viable project - it will run Drill
> with standard options in the various environments. Users who chose to run
> Drill in production will likely require additional security settings. Enable
> SSL? Control ingress? We need to understand what is needed, what Drill
> offers, and how to enable Drill's security features in a containerized
[jira] [Closed] (DRILL-7648) Scrypt j_security_check works without security headers
[
https://issues.apache.org/jira/browse/DRILL-7648?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dmytro Kondriukov closed DRILL-7648.
verified. fixed
> Scrypt j_security_check works without security headers
> ---
>
> Key: DRILL-7648
> URL: https://issues.apache.org/jira/browse/DRILL-7648
> Project: Apache Drill
> Issue Type: Bug
>Affects Versions: 1.17.0
>Reporter: Dmytro Kondriukov
>Assignee: Igor Guzenko
>Priority: Major
> Labels: ready-to-commit
> Fix For: 1.18.0
>
>
> *Preconditions:*
> drill-override.conf
> {noformat}
> drill.exec: {
> cluster-id: "drillbits1",
> zk.connect: "localhost:5181"
> impersonation: {
> enabled: true,
> max_chained_user_hops: 3
> },
> security: {
> auth.mechanisms : ["PLAIN"],
> },
> security.user.auth: {
> enabled: true,
> packages += "org.apache.drill.exec.rpc.user.security",
> impl: "pam4j",
> pam_profiles: [ "sudo", "login" ]
> }
> http: {
> ssl_enabled: true,.
> jetty.server.response.headers: {
> "X-XSS-Protection": "1; mode=block",
> "X-Content-Type-Options": "nosniff",
> "Strict-Transport-Security": "max-age=31536000;includeSubDomains",
> "Content-Security-Policy": "default-src https:; script-src
> 'unsafe-inline' https:; style-src 'unsafe-inline' https:; font-src data:
> https:; img-src data: https:"
> }
> }
> }
> {noformat}
> *Steps:*
> 1. Perform login to drillbit webUI
> 2. Check in browser console in tab "network" headers of resource
> https://node1.cluster.com:8047/j_security_check
> 3. Check section "response headers"
> *Expected result:* security headers are present
> *Actual result:* security headers are absent
> 4. Check section "Form Data"
> *Expected result:* parameter "j_password" content is hidden
> *Actual result:* parameter "j_password" content is visible
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (DRILL-7648) Scrypt j_security_check works without security headers
[
https://issues.apache.org/jira/browse/DRILL-7648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17066026#comment-17066026
]
Dmytro Kondriukov commented on DRILL-7648:
--
tested on
apache-drill-1.18.0-SNAPSHOT commit 3b3c4af39fdc26f255cc17d66c55eb7565552a7d
fixed
security headers are added
> Scrypt j_security_check works without security headers
> ---
>
> Key: DRILL-7648
> URL: https://issues.apache.org/jira/browse/DRILL-7648
> Project: Apache Drill
> Issue Type: Bug
>Affects Versions: 1.17.0
>Reporter: Dmytro Kondriukov
>Assignee: Igor Guzenko
>Priority: Major
> Labels: ready-to-commit
> Fix For: 1.18.0
>
>
> *Preconditions:*
> drill-override.conf
> {noformat}
> drill.exec: {
> cluster-id: "drillbits1",
> zk.connect: "localhost:5181"
> impersonation: {
> enabled: true,
> max_chained_user_hops: 3
> },
> security: {
> auth.mechanisms : ["PLAIN"],
> },
> security.user.auth: {
> enabled: true,
> packages += "org.apache.drill.exec.rpc.user.security",
> impl: "pam4j",
> pam_profiles: [ "sudo", "login" ]
> }
> http: {
> ssl_enabled: true,.
> jetty.server.response.headers: {
> "X-XSS-Protection": "1; mode=block",
> "X-Content-Type-Options": "nosniff",
> "Strict-Transport-Security": "max-age=31536000;includeSubDomains",
> "Content-Security-Policy": "default-src https:; script-src
> 'unsafe-inline' https:; style-src 'unsafe-inline' https:; font-src data:
> https:; img-src data: https:"
> }
> }
> }
> {noformat}
> *Steps:*
> 1. Perform login to drillbit webUI
> 2. Check in browser console in tab "network" headers of resource
> https://node1.cluster.com:8047/j_security_check
> 3. Check section "response headers"
> *Expected result:* security headers are present
> *Actual result:* security headers are absent
> 4. Check section "Form Data"
> *Expected result:* parameter "j_password" content is hidden
> *Actual result:* parameter "j_password" content is visible
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (DRILL-7563) Docker & Kubernetes Drill server container
[
https://issues.apache.org/jira/browse/DRILL-7563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17066022#comment-17066022
]
Rafael Jaimes commented on DRILL-7563:
--
Dobes - thanks. I am looking through your work now. Have you tested this with
scaling up drillbit pods?
> Docker & Kubernetes Drill server container
> --
>
> Key: DRILL-7563
> URL: https://issues.apache.org/jira/browse/DRILL-7563
> Project: Apache Drill
> Issue Type: Improvement
>Affects Versions: 1.17.0
>Reporter: Paul Rogers
>Assignee: Paul Rogers
>Priority: Major
>
> Drill provides two Docker containers:
> * [Build Drill from
> sources|https://github.com/apache/drill/blob/master/Dockerfile]
> * [Run Drill in interactive embedded
> mode|https://github.com/apache/drill/blob/master/distribution/Dockerfile]
> User feedback suggests that these are not quite the right solutions to run
> Drill in a K8s (or OpenShift) cluster. In addition, we need a container to
> run a Drill server. This ticket summarizes the tasks involved.
> h4. Container Image
> The container image should:
> * Start with the OpenJDK base image with minimal extra packages.
> * Download and install an official Drill release.
> We may then want to provide two derived images:
> The Drillbit image which:
> * Configures Drill for production and as needed in the following steps.
> * Provides entry points for the Drillbit and for Sqlline
> * Exposes Drill's four ports
> * Accept as parameters things like the ZK host IP(s).
> The Sqlline image, meant to be run in interactive mode (like the current
> embedded image) and which:
> * Accept as parameters the ZK host IP(s).
> Both should be published to the official Drill DockerHub account:
> https://hub.docker.com/r/apache/drill
> h4. Runtime Environment
> Drill has very few dependencies, but it must have a running ZK.
> * Start a [ZK container|https://hub.docker.com/_/zookeeper/].
> * A place to store logs, which can be in the container by default, stored on
> the host file system via a volume mount.
> * Access to a data source, which can be configured via a storage plugin
> stored in ZK.
> * Ensure graceful shutdown integration with the Docker shutdown mechanism.
> h4. Running Drill in Docker
> Users must run at least one Drillbit, and may run more. Users may want to run
> Sqlline.
> * The Drillbit container requires, at a minimum, the IP address of the ZK
> instance(s).
> * The Sqlline container requires only the ZK instances, from which it can
> find the Drillbit.
> Uses will want to customize some parts of Drill: at least memory, perhaps any
> of the other options. Provide a way to pass this information into the
> container to avoid the need to rebuild the container to change configuration.
> h4. Running Drill in K8s
> The containers should be usable in "plain" Docker. Today, however, many
> people use K8s to orchestrate Docker. Thus, the Drillbit (but probably not
> the Sqlline) container should be designed to work with K8s. An example set of
> K8s YAML files should illustrate:
> * Create a host-mount file system to capture Drill logs and query profiles.
> * Optionally write Drill logs to stdout, to be captured by {{fluentd}} or
> similar tools.
> * Pass Drill configuration (both HOCON and envrironment) as config maps.
> * Pass ZK as an environment variable (the value of which would, one presumes,
> come from some kind of service discovery system.)
> The result is that the user should be able to manually tinker with the YAML
> files, then use {{kubeadm}} to launch, monitor and stop Drill. The user sets
> cluster size manually by launching the desired number of Drill pods.
> h4. Helm Chart for Drill
> The next step is to wrap the YAML files in a Helm chart, with parameters
> exposed for the config options noted above.
> h4. Drill Operator for K8s
>
> Full K8s integration will require an operator to manage the Drill cluster.
> K8s operators are often written in Go, though doing so is not necessary.
> Drill already includes Drill-on-YARN which is, essential a "YARN operator."
> Repurpose this code to work with K8s as the target cluster manager rather
> than YARN. Reuse the same operations from DoY: configure, start, resize and
> stop a cluster.
> h4. Security
> The above steps provide an "MVP": minimum viable project - it will run Drill
> with standard options in the various environments. Users who chose to run
> Drill in production will likely require additional security settings. Enable
> SSL? Control ingress? We need to understand what is needed, what Drill
> offers, and how to enable Drill's security features in a containerized
> environment.
> h4. Production Deployments
> With Docker and K8s the old maxim "the devil is in the details" is true in
> spades. There are dozens of ways that Drill can
[jira] [Commented] (DRILL-7563) Docker & Kubernetes Drill server container
[
https://issues.apache.org/jira/browse/DRILL-7563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17065990#comment-17065990
]
Charles Givre commented on DRILL-7563:
--
@Dobes,
Would you be willing to write up a tutorial or something that we could post on
the documentation on the main Drill website?
Thanks,
-- C
> Docker & Kubernetes Drill server container
> --
>
> Key: DRILL-7563
> URL: https://issues.apache.org/jira/browse/DRILL-7563
> Project: Apache Drill
> Issue Type: Improvement
>Affects Versions: 1.17.0
>Reporter: Paul Rogers
>Assignee: Paul Rogers
>Priority: Major
>
> Drill provides two Docker containers:
> * [Build Drill from
> sources|https://github.com/apache/drill/blob/master/Dockerfile]
> * [Run Drill in interactive embedded
> mode|https://github.com/apache/drill/blob/master/distribution/Dockerfile]
> User feedback suggests that these are not quite the right solutions to run
> Drill in a K8s (or OpenShift) cluster. In addition, we need a container to
> run a Drill server. This ticket summarizes the tasks involved.
> h4. Container Image
> The container image should:
> * Start with the OpenJDK base image with minimal extra packages.
> * Download and install an official Drill release.
> We may then want to provide two derived images:
> The Drillbit image which:
> * Configures Drill for production and as needed in the following steps.
> * Provides entry points for the Drillbit and for Sqlline
> * Exposes Drill's four ports
> * Accept as parameters things like the ZK host IP(s).
> The Sqlline image, meant to be run in interactive mode (like the current
> embedded image) and which:
> * Accept as parameters the ZK host IP(s).
> Both should be published to the official Drill DockerHub account:
> https://hub.docker.com/r/apache/drill
> h4. Runtime Environment
> Drill has very few dependencies, but it must have a running ZK.
> * Start a [ZK container|https://hub.docker.com/_/zookeeper/].
> * A place to store logs, which can be in the container by default, stored on
> the host file system via a volume mount.
> * Access to a data source, which can be configured via a storage plugin
> stored in ZK.
> * Ensure graceful shutdown integration with the Docker shutdown mechanism.
> h4. Running Drill in Docker
> Users must run at least one Drillbit, and may run more. Users may want to run
> Sqlline.
> * The Drillbit container requires, at a minimum, the IP address of the ZK
> instance(s).
> * The Sqlline container requires only the ZK instances, from which it can
> find the Drillbit.
> Uses will want to customize some parts of Drill: at least memory, perhaps any
> of the other options. Provide a way to pass this information into the
> container to avoid the need to rebuild the container to change configuration.
> h4. Running Drill in K8s
> The containers should be usable in "plain" Docker. Today, however, many
> people use K8s to orchestrate Docker. Thus, the Drillbit (but probably not
> the Sqlline) container should be designed to work with K8s. An example set of
> K8s YAML files should illustrate:
> * Create a host-mount file system to capture Drill logs and query profiles.
> * Optionally write Drill logs to stdout, to be captured by {{fluentd}} or
> similar tools.
> * Pass Drill configuration (both HOCON and envrironment) as config maps.
> * Pass ZK as an environment variable (the value of which would, one presumes,
> come from some kind of service discovery system.)
> The result is that the user should be able to manually tinker with the YAML
> files, then use {{kubeadm}} to launch, monitor and stop Drill. The user sets
> cluster size manually by launching the desired number of Drill pods.
> h4. Helm Chart for Drill
> The next step is to wrap the YAML files in a Helm chart, with parameters
> exposed for the config options noted above.
> h4. Drill Operator for K8s
>
> Full K8s integration will require an operator to manage the Drill cluster.
> K8s operators are often written in Go, though doing so is not necessary.
> Drill already includes Drill-on-YARN which is, essential a "YARN operator."
> Repurpose this code to work with K8s as the target cluster manager rather
> than YARN. Reuse the same operations from DoY: configure, start, resize and
> stop a cluster.
> h4. Security
> The above steps provide an "MVP": minimum viable project - it will run Drill
> with standard options in the various environments. Users who chose to run
> Drill in production will likely require additional security settings. Enable
> SSL? Control ingress? We need to understand what is needed, what Drill
> offers, and how to enable Drill's security features in a containerized
> environment.
> h4. Production Deployments
> With Docker and K8s the old maxim "the devil is in the details" is true in
>
[jira] [Commented] (DRILL-7563) Docker & Kubernetes Drill server container
[
https://issues.apache.org/jira/browse/DRILL-7563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17065973#comment-17065973
]
Dobes Vandermeer commented on DRILL-7563:
-
I have been able to get drill to run in kubernetes using the existing drill
docker images currently published.
See here for example configs:
[https://gist.github.com/dobesv/98d85b18ee8566891c5122e2b990f0c5]
[https://gist.github.com/dobesv/be5aa3e6e5830e54c0e77b73884333cc]
> Docker & Kubernetes Drill server container
> --
>
> Key: DRILL-7563
> URL: https://issues.apache.org/jira/browse/DRILL-7563
> Project: Apache Drill
> Issue Type: Improvement
>Affects Versions: 1.17.0
>Reporter: Paul Rogers
>Assignee: Paul Rogers
>Priority: Major
>
> Drill provides two Docker containers:
> * [Build Drill from
> sources|https://github.com/apache/drill/blob/master/Dockerfile]
> * [Run Drill in interactive embedded
> mode|https://github.com/apache/drill/blob/master/distribution/Dockerfile]
> User feedback suggests that these are not quite the right solutions to run
> Drill in a K8s (or OpenShift) cluster. In addition, we need a container to
> run a Drill server. This ticket summarizes the tasks involved.
> h4. Container Image
> The container image should:
> * Start with the OpenJDK base image with minimal extra packages.
> * Download and install an official Drill release.
> We may then want to provide two derived images:
> The Drillbit image which:
> * Configures Drill for production and as needed in the following steps.
> * Provides entry points for the Drillbit and for Sqlline
> * Exposes Drill's four ports
> * Accept as parameters things like the ZK host IP(s).
> The Sqlline image, meant to be run in interactive mode (like the current
> embedded image) and which:
> * Accept as parameters the ZK host IP(s).
> Both should be published to the official Drill DockerHub account:
> https://hub.docker.com/r/apache/drill
> h4. Runtime Environment
> Drill has very few dependencies, but it must have a running ZK.
> * Start a [ZK container|https://hub.docker.com/_/zookeeper/].
> * A place to store logs, which can be in the container by default, stored on
> the host file system via a volume mount.
> * Access to a data source, which can be configured via a storage plugin
> stored in ZK.
> * Ensure graceful shutdown integration with the Docker shutdown mechanism.
> h4. Running Drill in Docker
> Users must run at least one Drillbit, and may run more. Users may want to run
> Sqlline.
> * The Drillbit container requires, at a minimum, the IP address of the ZK
> instance(s).
> * The Sqlline container requires only the ZK instances, from which it can
> find the Drillbit.
> Uses will want to customize some parts of Drill: at least memory, perhaps any
> of the other options. Provide a way to pass this information into the
> container to avoid the need to rebuild the container to change configuration.
> h4. Running Drill in K8s
> The containers should be usable in "plain" Docker. Today, however, many
> people use K8s to orchestrate Docker. Thus, the Drillbit (but probably not
> the Sqlline) container should be designed to work with K8s. An example set of
> K8s YAML files should illustrate:
> * Create a host-mount file system to capture Drill logs and query profiles.
> * Optionally write Drill logs to stdout, to be captured by {{fluentd}} or
> similar tools.
> * Pass Drill configuration (both HOCON and envrironment) as config maps.
> * Pass ZK as an environment variable (the value of which would, one presumes,
> come from some kind of service discovery system.)
> The result is that the user should be able to manually tinker with the YAML
> files, then use {{kubeadm}} to launch, monitor and stop Drill. The user sets
> cluster size manually by launching the desired number of Drill pods.
> h4. Helm Chart for Drill
> The next step is to wrap the YAML files in a Helm chart, with parameters
> exposed for the config options noted above.
> h4. Drill Operator for K8s
>
> Full K8s integration will require an operator to manage the Drill cluster.
> K8s operators are often written in Go, though doing so is not necessary.
> Drill already includes Drill-on-YARN which is, essential a "YARN operator."
> Repurpose this code to work with K8s as the target cluster manager rather
> than YARN. Reuse the same operations from DoY: configure, start, resize and
> stop a cluster.
> h4. Security
> The above steps provide an "MVP": minimum viable project - it will run Drill
> with standard options in the various environments. Users who chose to run
> Drill in production will likely require additional security settings. Enable
> SSL? Control ingress? We need to understand what is needed, what Drill
> offers, and how to enable Drill's security features in
[jira] [Updated] (DRILL-7602) Possibility to force repartition on read/select
[
https://issues.apache.org/jira/browse/DRILL-7602?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
benj updated DRILL-7602:
Description:
It will be nice and usefull in certain situations to have the capacity to do
repartition like in spark
([https://spark.apache.org/docs/latest/rdd-programming-guide.html])
either an automatically repartition in certain limit or possibility to indicate
the desired repartition or both options.
The only way (that I know now) to do that with Drill is to change
_store.parquet.block-size_ and regenerate the input file then do the request
(but it will be nice to have the ability to do that on read)
illustration : with 2 Parquets files _file1_ of 50Mo (1 million rows) and
_file2_ of 1Mo (5000 rows)
{code:sql}
CREATE TABLE dfs.test.`result_from_1_parquet` AS
(SELECT * FROM dfs.test.`file2` INNER JOIN dfs.test.`file1` ON ...)
=> ~ 50min
-- Today we have to change the parquet block size to force multiple parquet
files
ALTER SESSION SET `store.parquet.block-size` = 1048576;
-- Repartition data
CREATE TABLE dfs.test.`file1_bis` AS (SELECT * FROM dfs.test.`file1`);
-- then Launch the request
CREATE TABLE dfs.test.`result_from_1_parquet` AS
(SELECT * FROM dfs.test.`file2` INNER JOIN dfs.test.`file1_bis` ON ...)
=> ~ 1min
{code}
So it's possible to save a lot of time (depending of configuration of cluster)
by simply forcing more input file.
it would be useful not to have to regenerate the files with the ideal
fragmentation before request.
This situation easily appears when making inequality JOIN (to lookup ip in ip
range for example) on not so big dataset:
{code:java}
ALTER SESSION SET `planner.enable_nljoin_for_scalar_only` = false;
SELECT *
FROM dfs.test.`a_pqt` AS a
INNER JOIN dfs.test.`b_pqt` AS b
ON inet_aton(b.ip) >= inet_aton(a.ip_first) AND inet_aton(b.ip) <=
inet_aton(a.ip_last);
{code}
was:
It will be nice and usefull ion certain situations to have the capacity to do
repartition like in spark
(https://spark.apache.org/docs/latest/rdd-programming-guide.html)
either an automatically repartition in certain limit or possibility to indicate
the desired repartition or both options.
The only way (that I know now) to do that with Drill is to change
_store.parquet.block-size_ and regenerate the input file then do the request
(but it will be nice to have the ability to do that on read)
illustration : with 2 Parquets files _file1_ of 50Mo (1 million rows) and
_file2_ of 1Mo (5000 rows)
{code:sql}
CREATE TABLE dfs.test.`result_from_1_parquet` AS
(SELECT * FROM dfs.test.`file2` INNER JOIN dfs.test.`file1` ON ...)
=> ~ 50min
-- Tody we have to change the parquet block size to force multiple parquet files
ALTER SESSION SET `store.parquet.block-size` = 1048576;
-- Repartition data
CREATE TABLE dfs.test.`file1_bis` AS (SELECT * FROM dfs.test.`file1`);
-- then Launch the request
CREATE TABLE dfs.test.`result_from_1_parquet` AS
(SELECT * FROM dfs.test.`file2` INNER JOIN dfs.test.`file1_bis` ON ...)
=> ~ 1min
{code}
So it's possible to save a lot of time (depending of configuration of cluster)
by simply forcing more input file.
it would be useful not to have to regenerate the files with the ideal
fragmentation before request.
> Possibility to force repartition on read/select
> ---
>
> Key: DRILL-7602
> URL: https://issues.apache.org/jira/browse/DRILL-7602
> Project: Apache Drill
> Issue Type: Improvement
> Components: Execution - Flow, Query Planning Optimization
>Affects Versions: 1.17.0
>Reporter: benj
>Priority: Major
>
> It will be nice and usefull in certain situations to have the capacity to do
> repartition like in spark
> ([https://spark.apache.org/docs/latest/rdd-programming-guide.html])
> either an automatically repartition in certain limit or possibility to
> indicate the desired repartition or both options.
> The only way (that I know now) to do that with Drill is to change
> _store.parquet.block-size_ and regenerate the input file then do the request
> (but it will be nice to have the ability to do that on read)
> illustration : with 2 Parquets files _file1_ of 50Mo (1 million rows) and
> _file2_ of 1Mo (5000 rows)
> {code:sql}
> CREATE TABLE dfs.test.`result_from_1_parquet` AS
> (SELECT * FROM dfs.test.`file2` INNER JOIN dfs.test.`file1` ON ...)
> => ~ 50min
> -- Today we have to change the parquet block size to force multiple parquet
> files
> ALTER SESSION SET `store.parquet.block-size` = 1048576;
> -- Repartition data
> CREATE TABLE dfs.test.`file1_bis` AS (SELECT * FROM dfs.test.`file1`);
> -- then Launch the request
> CREATE TABLE dfs.test.`result_from_1_parquet` AS
> (SELECT * FROM dfs.test.`file2` INNER JOIN dfs.test.`file1_bis` ON ...)
> => ~ 1min
> {code}
> So it's possible to save a lot of time (depending of configuration of
>
[jira] [Commented] (DRILL-6604) Upgrade Drill Hive client to Hive3.1 version
[
https://issues.apache.org/jira/browse/DRILL-6604?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17065785#comment-17065785
]
ASF GitHub Bot commented on DRILL-6604:
---
cgivre commented on issue #2038: DRILL-6604: Upgrade Drill Hive client to
Hive3.1 version
URL: https://github.com/apache/drill/pull/2038#issuecomment-603251228
@vvysotskyi
Thanks for the response. The reason I ask is that for many enterprises, they
are at the mercy of IT teams and in many cases they are forced to use older
versions of tools like Hive. Often, the upgrade schedules are years behind the
most current version. (I'm speaking from experience here..;-))
Obviously we should do our best to support the most current version of
common platforms like Hive, however, I think you would be surprised at how many
large enterprises still use very old versions of these tools.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Upgrade Drill Hive client to Hive3.1 version
>
>
> Key: DRILL-6604
> URL: https://issues.apache.org/jira/browse/DRILL-6604
> Project: Apache Drill
> Issue Type: Improvement
>Affects Versions: 1.14.0
>Reporter: Vitalii Diravka
>Assignee: Vova Vysotskyi
>Priority: Major
> Fix For: 1.18.0
>
>
> Recently the Hive3.1 version is released.
> 3.1 versions of {{hive-exec}} and {{hive-metastore}} (main Drill Hive client
> libraries) are the last available artifacts in maven repo for now:
> [https://mvnrepository.com/artifact/org.apache.hive/hive-exec/3.1.0]
> [https://mvnrepository.com/artifact/org.apache.hive/hive-metastore/3.1.0]
> It is necessary to investigate what impact of update onto the newer major
> Hive client version.
> The initial conflicts after changing the {{hive.version}} in Drill root pom
> file from 2.3.2 to 3.1.0 are:
> {code}
> [ERROR] COMPILATION ERROR :
> [INFO] -
> [ERROR]
> /home/vitalii/IdeaProjects/drill-fork/contrib/storage-hive/core/src/main/java/org/apache/drill/exec/store/hive/HiveUtilities.java:[58,39]
> error: cannot find symbol
> [ERROR] symbol: class MetaStoreUtils
> location: package org.apache.hadoop.hive.metastore
> /home/vitalii/IdeaProjects/drill-fork/contrib/storage-hive/core/src/main/java/org/apache/drill/exec/store/hive/HiveMetadataProvider.java:[34,39]
> error: cannot find symbol
> [INFO] 2 errors
> {code}
> {code}
> [ERROR] COMPILATION ERROR :
> [INFO] -
> [ERROR]
> /home/vitalii/IdeaProjects/drill-fork/contrib/storage-hive/core/src/main/java/org/apache/drill/exec/store/hive/HiveUtilities.java:[575,15]
> error: cannot find symbol
> [ERROR] symbol: method setTransactionalTableScan(JobConf,boolean)
> location: class AcidUtils
> /home/vitalii/IdeaProjects/drill-fork/contrib/storage-hive/core/src/main/java/org/apache/drill/exec/store/hive/HiveFieldConverter.java:[216,92]
> error: incompatible types: org.apache.hadoop.hive.common.type.Timestamp
> cannot be converted to java.sql.Timestamp
> [ERROR]
> /home/vitalii/IdeaProjects/drill-fork/contrib/storage-hive/core/src/main/java/org/apache/drill/exec/store/hive/HiveFieldConverter.java:[225,82]
> error: incompatible types: org.apache.hadoop.hive.common.type.Date cannot be
> converted to java.sql.Date
> [ERROR]
> /home/vitalii/IdeaProjects/drill-fork/contrib/storage-hive/core/src/main/java/org/apache/drill/exec/store/hive/DrillHiveMetaStoreClient.java:[100,17]
> error: cannot find symbol
> [ERROR] symbol: method setTokenStr(UserGroupInformation,String,String)
> location: class Utils
> /home/vitalii/IdeaProjects/drill-fork/contrib/storage-hive/core/target/generated-sources/org/apache/drill/exec/expr/fn/impl/hive/DrillTimeStampTimestampObjectInspector.java:[29,16]
> error: Required is not abstract and does not override abstract method
> getPrimitiveJavaObject(Object) in TimestampObjectInspector
> [ERROR]
> /home/vitalii/IdeaProjects/drill-fork/contrib/storage-hive/core/target/generated-sources/org/apache/drill/exec/expr/fn/impl/hive/DrillTimeStampTimestampObjectInspector.java:[35,30]
> error: getPrimitiveJavaObject(Object) in Required cannot implement
> getPrimitiveJavaObject(Object) in TimestampObjectInspector
> [ERROR] return type java.sql.Timestamp is not compatible with
> org.apache.hadoop.hive.common.type.Timestamp
>
[jira] [Commented] (DRILL-6604) Upgrade Drill Hive client to Hive3.1 version
[
https://issues.apache.org/jira/browse/DRILL-6604?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17065771#comment-17065771
]
ASF GitHub Bot commented on DRILL-6604:
---
vvysotskyi commented on issue #2038: DRILL-6604: Upgrade Drill Hive client to
Hive3.1 version
URL: https://github.com/apache/drill/pull/2038#issuecomment-603237847
@cgivre,
1. Info about building Drill for different Hive versions is a workaround, we
shouldn't recommend doing it for users if other alternatives available, so I
don't think that we should document it. Also, when all supported profiles are
moved to this version of Hive, code will be cleaned up to remove hacks which
allowed using older versions.
2. I don't see a way to achieve this. The issue here is in the interaction
between Hive Jars with different versions. I assume besides API changes, also
thrift format updates.
3. I don't think so. For updating versions to 2.3.2, code was changed, so I
think it still incompatible.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Upgrade Drill Hive client to Hive3.1 version
>
>
> Key: DRILL-6604
> URL: https://issues.apache.org/jira/browse/DRILL-6604
> Project: Apache Drill
> Issue Type: Improvement
>Affects Versions: 1.14.0
>Reporter: Vitalii Diravka
>Assignee: Vova Vysotskyi
>Priority: Major
> Fix For: 1.18.0
>
>
> Recently the Hive3.1 version is released.
> 3.1 versions of {{hive-exec}} and {{hive-metastore}} (main Drill Hive client
> libraries) are the last available artifacts in maven repo for now:
> [https://mvnrepository.com/artifact/org.apache.hive/hive-exec/3.1.0]
> [https://mvnrepository.com/artifact/org.apache.hive/hive-metastore/3.1.0]
> It is necessary to investigate what impact of update onto the newer major
> Hive client version.
> The initial conflicts after changing the {{hive.version}} in Drill root pom
> file from 2.3.2 to 3.1.0 are:
> {code}
> [ERROR] COMPILATION ERROR :
> [INFO] -
> [ERROR]
> /home/vitalii/IdeaProjects/drill-fork/contrib/storage-hive/core/src/main/java/org/apache/drill/exec/store/hive/HiveUtilities.java:[58,39]
> error: cannot find symbol
> [ERROR] symbol: class MetaStoreUtils
> location: package org.apache.hadoop.hive.metastore
> /home/vitalii/IdeaProjects/drill-fork/contrib/storage-hive/core/src/main/java/org/apache/drill/exec/store/hive/HiveMetadataProvider.java:[34,39]
> error: cannot find symbol
> [INFO] 2 errors
> {code}
> {code}
> [ERROR] COMPILATION ERROR :
> [INFO] -
> [ERROR]
> /home/vitalii/IdeaProjects/drill-fork/contrib/storage-hive/core/src/main/java/org/apache/drill/exec/store/hive/HiveUtilities.java:[575,15]
> error: cannot find symbol
> [ERROR] symbol: method setTransactionalTableScan(JobConf,boolean)
> location: class AcidUtils
> /home/vitalii/IdeaProjects/drill-fork/contrib/storage-hive/core/src/main/java/org/apache/drill/exec/store/hive/HiveFieldConverter.java:[216,92]
> error: incompatible types: org.apache.hadoop.hive.common.type.Timestamp
> cannot be converted to java.sql.Timestamp
> [ERROR]
> /home/vitalii/IdeaProjects/drill-fork/contrib/storage-hive/core/src/main/java/org/apache/drill/exec/store/hive/HiveFieldConverter.java:[225,82]
> error: incompatible types: org.apache.hadoop.hive.common.type.Date cannot be
> converted to java.sql.Date
> [ERROR]
> /home/vitalii/IdeaProjects/drill-fork/contrib/storage-hive/core/src/main/java/org/apache/drill/exec/store/hive/DrillHiveMetaStoreClient.java:[100,17]
> error: cannot find symbol
> [ERROR] symbol: method setTokenStr(UserGroupInformation,String,String)
> location: class Utils
> /home/vitalii/IdeaProjects/drill-fork/contrib/storage-hive/core/target/generated-sources/org/apache/drill/exec/expr/fn/impl/hive/DrillTimeStampTimestampObjectInspector.java:[29,16]
> error: Required is not abstract and does not override abstract method
> getPrimitiveJavaObject(Object) in TimestampObjectInspector
> [ERROR]
> /home/vitalii/IdeaProjects/drill-fork/contrib/storage-hive/core/target/generated-sources/org/apache/drill/exec/expr/fn/impl/hive/DrillTimeStampTimestampObjectInspector.java:[35,30]
> error: getPrimitiveJavaObject(Object) in Required cannot implement
> getPrimitiveJavaObject(Object) in TimestampObjectInspector
> [ERROR] return type java.sql.Timestamp is not compatible with
> org.apache.hadoop.hive.common.type.Timestamp
>
[jira] [Updated] (DRILL-6604) Upgrade Drill Hive client to Hive3.1 version
[
https://issues.apache.org/jira/browse/DRILL-6604?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arina Ielchiieva updated DRILL-6604:
Reviewer: Igor Guzenko
> Upgrade Drill Hive client to Hive3.1 version
>
>
> Key: DRILL-6604
> URL: https://issues.apache.org/jira/browse/DRILL-6604
> Project: Apache Drill
> Issue Type: Improvement
>Affects Versions: 1.14.0
>Reporter: Vitalii Diravka
>Assignee: Vova Vysotskyi
>Priority: Major
> Fix For: 1.18.0
>
>
> Recently the Hive3.1 version is released.
> 3.1 versions of {{hive-exec}} and {{hive-metastore}} (main Drill Hive client
> libraries) are the last available artifacts in maven repo for now:
> [https://mvnrepository.com/artifact/org.apache.hive/hive-exec/3.1.0]
> [https://mvnrepository.com/artifact/org.apache.hive/hive-metastore/3.1.0]
> It is necessary to investigate what impact of update onto the newer major
> Hive client version.
> The initial conflicts after changing the {{hive.version}} in Drill root pom
> file from 2.3.2 to 3.1.0 are:
> {code}
> [ERROR] COMPILATION ERROR :
> [INFO] -
> [ERROR]
> /home/vitalii/IdeaProjects/drill-fork/contrib/storage-hive/core/src/main/java/org/apache/drill/exec/store/hive/HiveUtilities.java:[58,39]
> error: cannot find symbol
> [ERROR] symbol: class MetaStoreUtils
> location: package org.apache.hadoop.hive.metastore
> /home/vitalii/IdeaProjects/drill-fork/contrib/storage-hive/core/src/main/java/org/apache/drill/exec/store/hive/HiveMetadataProvider.java:[34,39]
> error: cannot find symbol
> [INFO] 2 errors
> {code}
> {code}
> [ERROR] COMPILATION ERROR :
> [INFO] -
> [ERROR]
> /home/vitalii/IdeaProjects/drill-fork/contrib/storage-hive/core/src/main/java/org/apache/drill/exec/store/hive/HiveUtilities.java:[575,15]
> error: cannot find symbol
> [ERROR] symbol: method setTransactionalTableScan(JobConf,boolean)
> location: class AcidUtils
> /home/vitalii/IdeaProjects/drill-fork/contrib/storage-hive/core/src/main/java/org/apache/drill/exec/store/hive/HiveFieldConverter.java:[216,92]
> error: incompatible types: org.apache.hadoop.hive.common.type.Timestamp
> cannot be converted to java.sql.Timestamp
> [ERROR]
> /home/vitalii/IdeaProjects/drill-fork/contrib/storage-hive/core/src/main/java/org/apache/drill/exec/store/hive/HiveFieldConverter.java:[225,82]
> error: incompatible types: org.apache.hadoop.hive.common.type.Date cannot be
> converted to java.sql.Date
> [ERROR]
> /home/vitalii/IdeaProjects/drill-fork/contrib/storage-hive/core/src/main/java/org/apache/drill/exec/store/hive/DrillHiveMetaStoreClient.java:[100,17]
> error: cannot find symbol
> [ERROR] symbol: method setTokenStr(UserGroupInformation,String,String)
> location: class Utils
> /home/vitalii/IdeaProjects/drill-fork/contrib/storage-hive/core/target/generated-sources/org/apache/drill/exec/expr/fn/impl/hive/DrillTimeStampTimestampObjectInspector.java:[29,16]
> error: Required is not abstract and does not override abstract method
> getPrimitiveJavaObject(Object) in TimestampObjectInspector
> [ERROR]
> /home/vitalii/IdeaProjects/drill-fork/contrib/storage-hive/core/target/generated-sources/org/apache/drill/exec/expr/fn/impl/hive/DrillTimeStampTimestampObjectInspector.java:[35,30]
> error: getPrimitiveJavaObject(Object) in Required cannot implement
> getPrimitiveJavaObject(Object) in TimestampObjectInspector
> [ERROR] return type java.sql.Timestamp is not compatible with
> org.apache.hadoop.hive.common.type.Timestamp
> /home/vitalii/IdeaProjects/drill-fork/contrib/storage-hive/core/target/generated-sources/org/apache/drill/exec/expr/fn/impl/hive/DrillTimeStampTimestampObjectInspector.java:[44,29]
> error: getPrimitiveWritableObject(Object) in Required cannot implement
> getPrimitiveWritableObject(Object) in TimestampObjectInspector
> [ERROR] return type TimestampWritable is not compatible with
> TimestampWritableV2
> /home/vitalii/IdeaProjects/drill-fork/contrib/storage-hive/core/target/generated-sources/org/apache/drill/exec/expr/fn/impl/hive/DrillTimeStampTimestampObjectInspector.java:[54,16]
> error: Optional is not abstract and does not override abstract method
> getPrimitiveJavaObject(Object) in TimestampObjectInspector
> [ERROR]
> /home/vitalii/IdeaProjects/drill-fork/contrib/storage-hive/core/target/generated-sources/org/apache/drill/exec/expr/fn/impl/hive/DrillTimeStampTimestampObjectInspector.java:[60,30]
> error: getPrimitiveJavaObject(Object) in Optional cannot implement
> getPrimitiveJavaObject(Object) in TimestampObjectInspector
> [ERROR] return type java.sql.Timestamp is not compatible with
> org.apache.hadoop.hive.common.type.Timestamp
>
[jira] [Commented] (DRILL-6604) Upgrade Drill Hive client to Hive3.1 version
[
https://issues.apache.org/jira/browse/DRILL-6604?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17065752#comment-17065752
]
ASF GitHub Bot commented on DRILL-6604:
---
cgivre commented on issue #2038: DRILL-6604: Upgrade Drill Hive client to
Hive3.1 version
URL: https://github.com/apache/drill/pull/2038#issuecomment-603223087
@vvysotskyi
Thanks for submitting this. A few questions:
1. Can we please include the bit about building Drill for different Hive
versions in the documentation?
2. Is there any way to write this such that the users will not have to
build a dedicated version of Drill for each version of Hive or are the Hive
APIs so different that it is not practical?
3. If we are requiring that Drill is built specially for a given version of
Hive, is it possible to support older versions than `2.3.2`?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Upgrade Drill Hive client to Hive3.1 version
>
>
> Key: DRILL-6604
> URL: https://issues.apache.org/jira/browse/DRILL-6604
> Project: Apache Drill
> Issue Type: Improvement
>Affects Versions: 1.14.0
>Reporter: Vitalii Diravka
>Assignee: Vova Vysotskyi
>Priority: Major
> Fix For: 1.18.0
>
>
> Recently the Hive3.1 version is released.
> 3.1 versions of {{hive-exec}} and {{hive-metastore}} (main Drill Hive client
> libraries) are the last available artifacts in maven repo for now:
> [https://mvnrepository.com/artifact/org.apache.hive/hive-exec/3.1.0]
> [https://mvnrepository.com/artifact/org.apache.hive/hive-metastore/3.1.0]
> It is necessary to investigate what impact of update onto the newer major
> Hive client version.
> The initial conflicts after changing the {{hive.version}} in Drill root pom
> file from 2.3.2 to 3.1.0 are:
> {code}
> [ERROR] COMPILATION ERROR :
> [INFO] -
> [ERROR]
> /home/vitalii/IdeaProjects/drill-fork/contrib/storage-hive/core/src/main/java/org/apache/drill/exec/store/hive/HiveUtilities.java:[58,39]
> error: cannot find symbol
> [ERROR] symbol: class MetaStoreUtils
> location: package org.apache.hadoop.hive.metastore
> /home/vitalii/IdeaProjects/drill-fork/contrib/storage-hive/core/src/main/java/org/apache/drill/exec/store/hive/HiveMetadataProvider.java:[34,39]
> error: cannot find symbol
> [INFO] 2 errors
> {code}
> {code}
> [ERROR] COMPILATION ERROR :
> [INFO] -
> [ERROR]
> /home/vitalii/IdeaProjects/drill-fork/contrib/storage-hive/core/src/main/java/org/apache/drill/exec/store/hive/HiveUtilities.java:[575,15]
> error: cannot find symbol
> [ERROR] symbol: method setTransactionalTableScan(JobConf,boolean)
> location: class AcidUtils
> /home/vitalii/IdeaProjects/drill-fork/contrib/storage-hive/core/src/main/java/org/apache/drill/exec/store/hive/HiveFieldConverter.java:[216,92]
> error: incompatible types: org.apache.hadoop.hive.common.type.Timestamp
> cannot be converted to java.sql.Timestamp
> [ERROR]
> /home/vitalii/IdeaProjects/drill-fork/contrib/storage-hive/core/src/main/java/org/apache/drill/exec/store/hive/HiveFieldConverter.java:[225,82]
> error: incompatible types: org.apache.hadoop.hive.common.type.Date cannot be
> converted to java.sql.Date
> [ERROR]
> /home/vitalii/IdeaProjects/drill-fork/contrib/storage-hive/core/src/main/java/org/apache/drill/exec/store/hive/DrillHiveMetaStoreClient.java:[100,17]
> error: cannot find symbol
> [ERROR] symbol: method setTokenStr(UserGroupInformation,String,String)
> location: class Utils
> /home/vitalii/IdeaProjects/drill-fork/contrib/storage-hive/core/target/generated-sources/org/apache/drill/exec/expr/fn/impl/hive/DrillTimeStampTimestampObjectInspector.java:[29,16]
> error: Required is not abstract and does not override abstract method
> getPrimitiveJavaObject(Object) in TimestampObjectInspector
> [ERROR]
> /home/vitalii/IdeaProjects/drill-fork/contrib/storage-hive/core/target/generated-sources/org/apache/drill/exec/expr/fn/impl/hive/DrillTimeStampTimestampObjectInspector.java:[35,30]
> error: getPrimitiveJavaObject(Object) in Required cannot implement
> getPrimitiveJavaObject(Object) in TimestampObjectInspector
> [ERROR] return type java.sql.Timestamp is not compatible with
> org.apache.hadoop.hive.common.type.Timestamp
>
[jira] [Commented] (DRILL-7633) Fixes for union and repeated list accessors
[ https://issues.apache.org/jira/browse/DRILL-7633?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17065560#comment-17065560 ] ASF GitHub Bot commented on DRILL-7633: --- arina-ielchiieva commented on issue #2018: DRILL-7633: Fixes for union and repeated list accessors URL: https://github.com/apache/drill/pull/2018#issuecomment-603206563 @paul-rogers thanks for addressing code review comment, I have left comment how to fix unit test failures, please apply them and squash the commits. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Fixes for union and repeated list accessors > --- > > Key: DRILL-7633 > URL: https://issues.apache.org/jira/browse/DRILL-7633 > Project: Apache Drill > Issue Type: Improvement >Affects Versions: 1.17.0 >Reporter: Paul Rogers >Assignee: Paul Rogers >Priority: Minor > Fix For: 1.18.0 > > > Minor fixes for repeated list and Union type support in column accessors -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (DRILL-7633) Fixes for union and repeated list accessors
[
https://issues.apache.org/jira/browse/DRILL-7633?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17065558#comment-17065558
]
ASF GitHub Bot commented on DRILL-7633:
---
arina-ielchiieva commented on pull request #2018: DRILL-7633: Fixes for union
and repeated list accessors
URL: https://github.com/apache/drill/pull/2018#discussion_r397109112
##
File path:
exec/vector/src/main/java/org/apache/drill/exec/record/metadata/AbstractColumnMetadata.java
##
@@ -295,12 +295,6 @@ public String toString() {
.toString();
}
- @JsonProperty("type")
- @Override
- public String typeString() {
-return majorType().toString();
- }
-
Review comment:
Please don't remove this method but leave it as abstract with json property
annotation:
```
@JsonProperty("type")
@Override
public abstract String typeString();
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Fixes for union and repeated list accessors
> ---
>
> Key: DRILL-7633
> URL: https://issues.apache.org/jira/browse/DRILL-7633
> Project: Apache Drill
> Issue Type: Improvement
>Affects Versions: 1.17.0
>Reporter: Paul Rogers
>Assignee: Paul Rogers
>Priority: Minor
> Fix For: 1.18.0
>
>
> Minor fixes for repeated list and Union type support in column accessors
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (DRILL-7633) Fixes for union and repeated list accessors
[ https://issues.apache.org/jira/browse/DRILL-7633?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17065561#comment-17065561 ] ASF GitHub Bot commented on DRILL-7633: --- arina-ielchiieva commented on issue #2018: DRILL-7633: Fixes for union and repeated list accessors URL: https://github.com/apache/drill/pull/2018#issuecomment-603206563 @paul-rogers thanks for addressing code review comment, I have left comments how to fix unit test failures, please apply them and squash the commits. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Fixes for union and repeated list accessors > --- > > Key: DRILL-7633 > URL: https://issues.apache.org/jira/browse/DRILL-7633 > Project: Apache Drill > Issue Type: Improvement >Affects Versions: 1.17.0 >Reporter: Paul Rogers >Assignee: Paul Rogers >Priority: Minor > Fix For: 1.18.0 > > > Minor fixes for repeated list and Union type support in column accessors -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (DRILL-7633) Fixes for union and repeated list accessors
[
https://issues.apache.org/jira/browse/DRILL-7633?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17065559#comment-17065559
]
ASF GitHub Bot commented on DRILL-7633:
---
arina-ielchiieva commented on pull request #2018: DRILL-7633: Fixes for union
and repeated list accessors
URL: https://github.com/apache/drill/pull/2018#discussion_r397109336
##
File path:
exec/vector/src/main/java/org/apache/drill/exec/record/metadata/VariantColumnMetadata.java
##
@@ -145,16 +147,28 @@ public ColumnMetadata cloneEmpty() {
@Override
public ColumnMetadata copy() {
-// TODO Auto-generated method stub
-assert false;
-return null;
+return new VariantColumnMetadata(name, type, mode, variantSchema.copy());
}
@Override
public VariantMetadata variantSchema() {
return variantSchema;
}
+ @JsonProperty("type")
Review comment:
Remove json property annotation as it will be present in parent class.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Fixes for union and repeated list accessors
> ---
>
> Key: DRILL-7633
> URL: https://issues.apache.org/jira/browse/DRILL-7633
> Project: Apache Drill
> Issue Type: Improvement
>Affects Versions: 1.17.0
>Reporter: Paul Rogers
>Assignee: Paul Rogers
>Priority: Minor
> Fix For: 1.18.0
>
>
> Minor fixes for repeated list and Union type support in column accessors
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (DRILL-6604) Upgrade Drill Hive client to Hive3.1 version
[
https://issues.apache.org/jira/browse/DRILL-6604?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17065747#comment-17065747
]
ASF GitHub Bot commented on DRILL-6604:
---
cgivre commented on issue #2038: DRILL-6604: Upgrade Drill Hive client to
Hive3.1 version
URL: https://github.com/apache/drill/pull/2038#issuecomment-603219947
@vvysotskyi
Thank you for submitting this! I am wondering whether we test for backwards
compatibility with older versions of Hive as we upgrade? What is the earliest
version we currently support?
Thanks!
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Upgrade Drill Hive client to Hive3.1 version
>
>
> Key: DRILL-6604
> URL: https://issues.apache.org/jira/browse/DRILL-6604
> Project: Apache Drill
> Issue Type: Improvement
>Affects Versions: 1.14.0
>Reporter: Vitalii Diravka
>Assignee: Vova Vysotskyi
>Priority: Major
> Fix For: 1.18.0
>
>
> Recently the Hive3.1 version is released.
> 3.1 versions of {{hive-exec}} and {{hive-metastore}} (main Drill Hive client
> libraries) are the last available artifacts in maven repo for now:
> [https://mvnrepository.com/artifact/org.apache.hive/hive-exec/3.1.0]
> [https://mvnrepository.com/artifact/org.apache.hive/hive-metastore/3.1.0]
> It is necessary to investigate what impact of update onto the newer major
> Hive client version.
> The initial conflicts after changing the {{hive.version}} in Drill root pom
> file from 2.3.2 to 3.1.0 are:
> {code}
> [ERROR] COMPILATION ERROR :
> [INFO] -
> [ERROR]
> /home/vitalii/IdeaProjects/drill-fork/contrib/storage-hive/core/src/main/java/org/apache/drill/exec/store/hive/HiveUtilities.java:[58,39]
> error: cannot find symbol
> [ERROR] symbol: class MetaStoreUtils
> location: package org.apache.hadoop.hive.metastore
> /home/vitalii/IdeaProjects/drill-fork/contrib/storage-hive/core/src/main/java/org/apache/drill/exec/store/hive/HiveMetadataProvider.java:[34,39]
> error: cannot find symbol
> [INFO] 2 errors
> {code}
> {code}
> [ERROR] COMPILATION ERROR :
> [INFO] -
> [ERROR]
> /home/vitalii/IdeaProjects/drill-fork/contrib/storage-hive/core/src/main/java/org/apache/drill/exec/store/hive/HiveUtilities.java:[575,15]
> error: cannot find symbol
> [ERROR] symbol: method setTransactionalTableScan(JobConf,boolean)
> location: class AcidUtils
> /home/vitalii/IdeaProjects/drill-fork/contrib/storage-hive/core/src/main/java/org/apache/drill/exec/store/hive/HiveFieldConverter.java:[216,92]
> error: incompatible types: org.apache.hadoop.hive.common.type.Timestamp
> cannot be converted to java.sql.Timestamp
> [ERROR]
> /home/vitalii/IdeaProjects/drill-fork/contrib/storage-hive/core/src/main/java/org/apache/drill/exec/store/hive/HiveFieldConverter.java:[225,82]
> error: incompatible types: org.apache.hadoop.hive.common.type.Date cannot be
> converted to java.sql.Date
> [ERROR]
> /home/vitalii/IdeaProjects/drill-fork/contrib/storage-hive/core/src/main/java/org/apache/drill/exec/store/hive/DrillHiveMetaStoreClient.java:[100,17]
> error: cannot find symbol
> [ERROR] symbol: method setTokenStr(UserGroupInformation,String,String)
> location: class Utils
> /home/vitalii/IdeaProjects/drill-fork/contrib/storage-hive/core/target/generated-sources/org/apache/drill/exec/expr/fn/impl/hive/DrillTimeStampTimestampObjectInspector.java:[29,16]
> error: Required is not abstract and does not override abstract method
> getPrimitiveJavaObject(Object) in TimestampObjectInspector
> [ERROR]
> /home/vitalii/IdeaProjects/drill-fork/contrib/storage-hive/core/target/generated-sources/org/apache/drill/exec/expr/fn/impl/hive/DrillTimeStampTimestampObjectInspector.java:[35,30]
> error: getPrimitiveJavaObject(Object) in Required cannot implement
> getPrimitiveJavaObject(Object) in TimestampObjectInspector
> [ERROR] return type java.sql.Timestamp is not compatible with
> org.apache.hadoop.hive.common.type.Timestamp
> /home/vitalii/IdeaProjects/drill-fork/contrib/storage-hive/core/target/generated-sources/org/apache/drill/exec/expr/fn/impl/hive/DrillTimeStampTimestampObjectInspector.java:[44,29]
> error: getPrimitiveWritableObject(Object) in Required cannot implement
> getPrimitiveWritableObject(Object) in TimestampObjectInspector
> [ERROR] return type TimestampWritable is not compatible with
> TimestampWritableV2
>
[jira] [Commented] (DRILL-6604) Upgrade Drill Hive client to Hive3.1 version
[
https://issues.apache.org/jira/browse/DRILL-6604?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17065745#comment-17065745
]
ASF GitHub Bot commented on DRILL-6604:
---
cgivre commented on issue #2038: DRILL-6604: Upgrade Drill Hive client to
Hive3.1 version
URL: https://github.com/apache/drill/pull/2038#issuecomment-603219947
@vvysotskyi
Thank you for submitting this! I am wondering whether we test for backwards
compatibility with older versions of Hive as we upgrade? What is the earliest
version we currently support?
Thanks!
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Upgrade Drill Hive client to Hive3.1 version
>
>
> Key: DRILL-6604
> URL: https://issues.apache.org/jira/browse/DRILL-6604
> Project: Apache Drill
> Issue Type: Improvement
>Affects Versions: 1.14.0
>Reporter: Vitalii Diravka
>Assignee: Vova Vysotskyi
>Priority: Major
> Fix For: 1.18.0
>
>
> Recently the Hive3.1 version is released.
> 3.1 versions of {{hive-exec}} and {{hive-metastore}} (main Drill Hive client
> libraries) are the last available artifacts in maven repo for now:
> [https://mvnrepository.com/artifact/org.apache.hive/hive-exec/3.1.0]
> [https://mvnrepository.com/artifact/org.apache.hive/hive-metastore/3.1.0]
> It is necessary to investigate what impact of update onto the newer major
> Hive client version.
> The initial conflicts after changing the {{hive.version}} in Drill root pom
> file from 2.3.2 to 3.1.0 are:
> {code}
> [ERROR] COMPILATION ERROR :
> [INFO] -
> [ERROR]
> /home/vitalii/IdeaProjects/drill-fork/contrib/storage-hive/core/src/main/java/org/apache/drill/exec/store/hive/HiveUtilities.java:[58,39]
> error: cannot find symbol
> [ERROR] symbol: class MetaStoreUtils
> location: package org.apache.hadoop.hive.metastore
> /home/vitalii/IdeaProjects/drill-fork/contrib/storage-hive/core/src/main/java/org/apache/drill/exec/store/hive/HiveMetadataProvider.java:[34,39]
> error: cannot find symbol
> [INFO] 2 errors
> {code}
> {code}
> [ERROR] COMPILATION ERROR :
> [INFO] -
> [ERROR]
> /home/vitalii/IdeaProjects/drill-fork/contrib/storage-hive/core/src/main/java/org/apache/drill/exec/store/hive/HiveUtilities.java:[575,15]
> error: cannot find symbol
> [ERROR] symbol: method setTransactionalTableScan(JobConf,boolean)
> location: class AcidUtils
> /home/vitalii/IdeaProjects/drill-fork/contrib/storage-hive/core/src/main/java/org/apache/drill/exec/store/hive/HiveFieldConverter.java:[216,92]
> error: incompatible types: org.apache.hadoop.hive.common.type.Timestamp
> cannot be converted to java.sql.Timestamp
> [ERROR]
> /home/vitalii/IdeaProjects/drill-fork/contrib/storage-hive/core/src/main/java/org/apache/drill/exec/store/hive/HiveFieldConverter.java:[225,82]
> error: incompatible types: org.apache.hadoop.hive.common.type.Date cannot be
> converted to java.sql.Date
> [ERROR]
> /home/vitalii/IdeaProjects/drill-fork/contrib/storage-hive/core/src/main/java/org/apache/drill/exec/store/hive/DrillHiveMetaStoreClient.java:[100,17]
> error: cannot find symbol
> [ERROR] symbol: method setTokenStr(UserGroupInformation,String,String)
> location: class Utils
> /home/vitalii/IdeaProjects/drill-fork/contrib/storage-hive/core/target/generated-sources/org/apache/drill/exec/expr/fn/impl/hive/DrillTimeStampTimestampObjectInspector.java:[29,16]
> error: Required is not abstract and does not override abstract method
> getPrimitiveJavaObject(Object) in TimestampObjectInspector
> [ERROR]
> /home/vitalii/IdeaProjects/drill-fork/contrib/storage-hive/core/target/generated-sources/org/apache/drill/exec/expr/fn/impl/hive/DrillTimeStampTimestampObjectInspector.java:[35,30]
> error: getPrimitiveJavaObject(Object) in Required cannot implement
> getPrimitiveJavaObject(Object) in TimestampObjectInspector
> [ERROR] return type java.sql.Timestamp is not compatible with
> org.apache.hadoop.hive.common.type.Timestamp
> /home/vitalii/IdeaProjects/drill-fork/contrib/storage-hive/core/target/generated-sources/org/apache/drill/exec/expr/fn/impl/hive/DrillTimeStampTimestampObjectInspector.java:[44,29]
> error: getPrimitiveWritableObject(Object) in Required cannot implement
> getPrimitiveWritableObject(Object) in TimestampObjectInspector
> [ERROR] return type TimestampWritable is not compatible with
> TimestampWritableV2
>
[jira] [Commented] (DRILL-6604) Upgrade Drill Hive client to Hive3.1 version
[
https://issues.apache.org/jira/browse/DRILL-6604?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17065483#comment-17065483
]
ASF GitHub Bot commented on DRILL-6604:
---
vvysotskyi commented on pull request #2038: DRILL-6604: Upgrade Drill Hive
client to Hive3.1 version
URL: https://github.com/apache/drill/pull/2038
# [DRILL-6604](https://issues.apache.org/jira/browse/DRILL-6604): Upgrade
Drill Hive client to Hive3.1 version
## Description
One of the major changes in this PR is cleaning up the output for Hive
tests. Now, almost all hive-related stuff is not printed to the stdout.
Additionally, the Hive version was updated to the latest (at the current
time) version 3.1.2.
New Hive version introduced new `ObjectInspector` classes for date and
timestamp values, so to be able to compile with other versions, code was
updated to use correct class names (see changes in `tdd` files and related
changes.
As for any other Hive update, Dill wouldn't be able to work with previous
Hive versions, but for users, who still want to do it, it is possible to
compile manually Drill with setting the following properties:
`hive.version=2.3.2` (or any other version in range [2.3.2-3.1.2]),
`freemarker.conf.file=src/main/codegen/config.fmpp` (or
`src/main/codegen/configHive3.fmpp` for versions where new date / timestamp
classes were introduced).
Example of usage:
```
mvn clean install -DskipTests -Dhive.version=2.3.2
-Dfreemarker.conf.file=src/main/codegen/config.fmpp
```
## Documentation
A new supported Hive version should be documented.
## Testing
Ran all tests suite, checked manually that Drill is able to select from new
Hive.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Upgrade Drill Hive client to Hive3.1 version
>
>
> Key: DRILL-6604
> URL: https://issues.apache.org/jira/browse/DRILL-6604
> Project: Apache Drill
> Issue Type: Improvement
>Affects Versions: 1.14.0
>Reporter: Vitalii Diravka
>Assignee: Vova Vysotskyi
>Priority: Major
> Fix For: 1.18.0
>
>
> Recently the Hive3.1 version is released.
> 3.1 versions of {{hive-exec}} and {{hive-metastore}} (main Drill Hive client
> libraries) are the last available artifacts in maven repo for now:
> [https://mvnrepository.com/artifact/org.apache.hive/hive-exec/3.1.0]
> [https://mvnrepository.com/artifact/org.apache.hive/hive-metastore/3.1.0]
> It is necessary to investigate what impact of update onto the newer major
> Hive client version.
> The initial conflicts after changing the {{hive.version}} in Drill root pom
> file from 2.3.2 to 3.1.0 are:
> {code}
> [ERROR] COMPILATION ERROR :
> [INFO] -
> [ERROR]
> /home/vitalii/IdeaProjects/drill-fork/contrib/storage-hive/core/src/main/java/org/apache/drill/exec/store/hive/HiveUtilities.java:[58,39]
> error: cannot find symbol
> [ERROR] symbol: class MetaStoreUtils
> location: package org.apache.hadoop.hive.metastore
> /home/vitalii/IdeaProjects/drill-fork/contrib/storage-hive/core/src/main/java/org/apache/drill/exec/store/hive/HiveMetadataProvider.java:[34,39]
> error: cannot find symbol
> [INFO] 2 errors
> {code}
> {code}
> [ERROR] COMPILATION ERROR :
> [INFO] -
> [ERROR]
> /home/vitalii/IdeaProjects/drill-fork/contrib/storage-hive/core/src/main/java/org/apache/drill/exec/store/hive/HiveUtilities.java:[575,15]
> error: cannot find symbol
> [ERROR] symbol: method setTransactionalTableScan(JobConf,boolean)
> location: class AcidUtils
> /home/vitalii/IdeaProjects/drill-fork/contrib/storage-hive/core/src/main/java/org/apache/drill/exec/store/hive/HiveFieldConverter.java:[216,92]
> error: incompatible types: org.apache.hadoop.hive.common.type.Timestamp
> cannot be converted to java.sql.Timestamp
> [ERROR]
> /home/vitalii/IdeaProjects/drill-fork/contrib/storage-hive/core/src/main/java/org/apache/drill/exec/store/hive/HiveFieldConverter.java:[225,82]
> error: incompatible types: org.apache.hadoop.hive.common.type.Date cannot be
> converted to java.sql.Date
> [ERROR]
> /home/vitalii/IdeaProjects/drill-fork/contrib/storage-hive/core/src/main/java/org/apache/drill/exec/store/hive/DrillHiveMetaStoreClient.java:[100,17]
> error: cannot find symbol
> [ERROR] symbol: method setTokenStr(UserGroupInformation,String,String)
> location: class Utils
>
[jira] [Assigned] (DRILL-7657) Invalid internal link
[ https://issues.apache.org/jira/browse/DRILL-7657?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vova Vysotskyi reassigned DRILL-7657: - Assignee: Aaron-Mhs > Invalid internal link > - > > Key: DRILL-7657 > URL: https://issues.apache.org/jira/browse/DRILL-7657 > Project: Apache Drill > Issue Type: Bug > Components: Security >Affects Versions: 1.17.0 >Reporter: Aaron-Mhs >Assignee: Aaron-Mhs >Priority: Major > Attachments: image-2020-03-24-14-57-53-568.png, > image-2020-03-24-14-58-17-865.png > > > There is an invalid link (Prerequisites-> See Enabling Authentication and > Encryption.) In the Configuring Kerberos Security module in the document. > After opening, it displays 404 (The requested URL was not found on this > server.), And I hope to fix it as soon as possible. > !image-2020-03-24-14-57-53-568.png! > !image-2020-03-24-14-58-17-865.png! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (DRILL-7657) Invalid internal link
[ https://issues.apache.org/jira/browse/DRILL-7657?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17065456#comment-17065456 ] Aaron-Mhs commented on DRILL-7657: -- [~volodymyr] , thanks for your prompt reply, I will submit a pull request on this branch later... > Invalid internal link > - > > Key: DRILL-7657 > URL: https://issues.apache.org/jira/browse/DRILL-7657 > Project: Apache Drill > Issue Type: Bug > Components: Security >Affects Versions: 1.17.0 >Reporter: Aaron-Mhs >Priority: Major > Attachments: image-2020-03-24-14-57-53-568.png, > image-2020-03-24-14-58-17-865.png > > > There is an invalid link (Prerequisites-> See Enabling Authentication and > Encryption.) In the Configuring Kerberos Security module in the document. > After opening, it displays 404 (The requested URL was not found on this > server.), And I hope to fix it as soon as possible. > !image-2020-03-24-14-57-53-568.png! > !image-2020-03-24-14-58-17-865.png! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (DRILL-7648) Scrypt j_security_check works without security headers
[
https://issues.apache.org/jira/browse/DRILL-7648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17065453#comment-17065453
]
ASF GitHub Bot commented on DRILL-7648:
---
arina-ielchiieva commented on pull request #2037: DRILL-7648: Scrypt
j_security_check works without security headers
URL: https://github.com/apache/drill/pull/2037
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Scrypt j_security_check works without security headers
> ---
>
> Key: DRILL-7648
> URL: https://issues.apache.org/jira/browse/DRILL-7648
> Project: Apache Drill
> Issue Type: Bug
>Affects Versions: 1.17.0
>Reporter: Dmytro Kondriukov
>Assignee: Igor Guzenko
>Priority: Major
> Labels: ready-to-commit
> Fix For: 1.18.0
>
>
> *Preconditions:*
> drill-override.conf
> {noformat}
> drill.exec: {
> cluster-id: "drillbits1",
> zk.connect: "localhost:5181"
> impersonation: {
> enabled: true,
> max_chained_user_hops: 3
> },
> security: {
> auth.mechanisms : ["PLAIN"],
> },
> security.user.auth: {
> enabled: true,
> packages += "org.apache.drill.exec.rpc.user.security",
> impl: "pam4j",
> pam_profiles: [ "sudo", "login" ]
> }
> http: {
> ssl_enabled: true,.
> jetty.server.response.headers: {
> "X-XSS-Protection": "1; mode=block",
> "X-Content-Type-Options": "nosniff",
> "Strict-Transport-Security": "max-age=31536000;includeSubDomains",
> "Content-Security-Policy": "default-src https:; script-src
> 'unsafe-inline' https:; style-src 'unsafe-inline' https:; font-src data:
> https:; img-src data: https:"
> }
> }
> }
> {noformat}
> *Steps:*
> 1. Perform login to drillbit webUI
> 2. Check in browser console in tab "network" headers of resource
> https://node1.cluster.com:8047/j_security_check
> 3. Check section "response headers"
> *Expected result:* security headers are present
> *Actual result:* security headers are absent
> 4. Check section "Form Data"
> *Expected result:* parameter "j_password" content is hidden
> *Actual result:* parameter "j_password" content is visible
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (DRILL-6604) Upgrade Drill Hive client to Hive3.1 version
[
https://issues.apache.org/jira/browse/DRILL-6604?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vova Vysotskyi updated DRILL-6604:
--
Fix Version/s: (was: Future)
1.18.0
> Upgrade Drill Hive client to Hive3.1 version
>
>
> Key: DRILL-6604
> URL: https://issues.apache.org/jira/browse/DRILL-6604
> Project: Apache Drill
> Issue Type: Improvement
>Affects Versions: 1.14.0
>Reporter: Vitalii Diravka
>Assignee: Vova Vysotskyi
>Priority: Major
> Fix For: 1.18.0
>
>
> Recently the Hive3.1 version is released.
> 3.1 versions of {{hive-exec}} and {{hive-metastore}} (main Drill Hive client
> libraries) are the last available artifacts in maven repo for now:
> [https://mvnrepository.com/artifact/org.apache.hive/hive-exec/3.1.0]
> [https://mvnrepository.com/artifact/org.apache.hive/hive-metastore/3.1.0]
> It is necessary to investigate what impact of update onto the newer major
> Hive client version.
> The initial conflicts after changing the {{hive.version}} in Drill root pom
> file from 2.3.2 to 3.1.0 are:
> {code}
> [ERROR] COMPILATION ERROR :
> [INFO] -
> [ERROR]
> /home/vitalii/IdeaProjects/drill-fork/contrib/storage-hive/core/src/main/java/org/apache/drill/exec/store/hive/HiveUtilities.java:[58,39]
> error: cannot find symbol
> [ERROR] symbol: class MetaStoreUtils
> location: package org.apache.hadoop.hive.metastore
> /home/vitalii/IdeaProjects/drill-fork/contrib/storage-hive/core/src/main/java/org/apache/drill/exec/store/hive/HiveMetadataProvider.java:[34,39]
> error: cannot find symbol
> [INFO] 2 errors
> {code}
> {code}
> [ERROR] COMPILATION ERROR :
> [INFO] -
> [ERROR]
> /home/vitalii/IdeaProjects/drill-fork/contrib/storage-hive/core/src/main/java/org/apache/drill/exec/store/hive/HiveUtilities.java:[575,15]
> error: cannot find symbol
> [ERROR] symbol: method setTransactionalTableScan(JobConf,boolean)
> location: class AcidUtils
> /home/vitalii/IdeaProjects/drill-fork/contrib/storage-hive/core/src/main/java/org/apache/drill/exec/store/hive/HiveFieldConverter.java:[216,92]
> error: incompatible types: org.apache.hadoop.hive.common.type.Timestamp
> cannot be converted to java.sql.Timestamp
> [ERROR]
> /home/vitalii/IdeaProjects/drill-fork/contrib/storage-hive/core/src/main/java/org/apache/drill/exec/store/hive/HiveFieldConverter.java:[225,82]
> error: incompatible types: org.apache.hadoop.hive.common.type.Date cannot be
> converted to java.sql.Date
> [ERROR]
> /home/vitalii/IdeaProjects/drill-fork/contrib/storage-hive/core/src/main/java/org/apache/drill/exec/store/hive/DrillHiveMetaStoreClient.java:[100,17]
> error: cannot find symbol
> [ERROR] symbol: method setTokenStr(UserGroupInformation,String,String)
> location: class Utils
> /home/vitalii/IdeaProjects/drill-fork/contrib/storage-hive/core/target/generated-sources/org/apache/drill/exec/expr/fn/impl/hive/DrillTimeStampTimestampObjectInspector.java:[29,16]
> error: Required is not abstract and does not override abstract method
> getPrimitiveJavaObject(Object) in TimestampObjectInspector
> [ERROR]
> /home/vitalii/IdeaProjects/drill-fork/contrib/storage-hive/core/target/generated-sources/org/apache/drill/exec/expr/fn/impl/hive/DrillTimeStampTimestampObjectInspector.java:[35,30]
> error: getPrimitiveJavaObject(Object) in Required cannot implement
> getPrimitiveJavaObject(Object) in TimestampObjectInspector
> [ERROR] return type java.sql.Timestamp is not compatible with
> org.apache.hadoop.hive.common.type.Timestamp
> /home/vitalii/IdeaProjects/drill-fork/contrib/storage-hive/core/target/generated-sources/org/apache/drill/exec/expr/fn/impl/hive/DrillTimeStampTimestampObjectInspector.java:[44,29]
> error: getPrimitiveWritableObject(Object) in Required cannot implement
> getPrimitiveWritableObject(Object) in TimestampObjectInspector
> [ERROR] return type TimestampWritable is not compatible with
> TimestampWritableV2
> /home/vitalii/IdeaProjects/drill-fork/contrib/storage-hive/core/target/generated-sources/org/apache/drill/exec/expr/fn/impl/hive/DrillTimeStampTimestampObjectInspector.java:[54,16]
> error: Optional is not abstract and does not override abstract method
> getPrimitiveJavaObject(Object) in TimestampObjectInspector
> [ERROR]
> /home/vitalii/IdeaProjects/drill-fork/contrib/storage-hive/core/target/generated-sources/org/apache/drill/exec/expr/fn/impl/hive/DrillTimeStampTimestampObjectInspector.java:[60,30]
> error: getPrimitiveJavaObject(Object) in Optional cannot implement
> getPrimitiveJavaObject(Object) in TimestampObjectInspector
> [ERROR] return type java.sql.Timestamp is not compatible with
>
[jira] [Assigned] (DRILL-6604) Upgrade Drill Hive client to Hive3.1 version
[
https://issues.apache.org/jira/browse/DRILL-6604?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vova Vysotskyi reassigned DRILL-6604:
-
Assignee: Vova Vysotskyi (was: Vitalii Diravka)
> Upgrade Drill Hive client to Hive3.1 version
>
>
> Key: DRILL-6604
> URL: https://issues.apache.org/jira/browse/DRILL-6604
> Project: Apache Drill
> Issue Type: Improvement
>Affects Versions: 1.14.0
>Reporter: Vitalii Diravka
>Assignee: Vova Vysotskyi
>Priority: Major
> Fix For: Future
>
>
> Recently the Hive3.1 version is released.
> 3.1 versions of {{hive-exec}} and {{hive-metastore}} (main Drill Hive client
> libraries) are the last available artifacts in maven repo for now:
> [https://mvnrepository.com/artifact/org.apache.hive/hive-exec/3.1.0]
> [https://mvnrepository.com/artifact/org.apache.hive/hive-metastore/3.1.0]
> It is necessary to investigate what impact of update onto the newer major
> Hive client version.
> The initial conflicts after changing the {{hive.version}} in Drill root pom
> file from 2.3.2 to 3.1.0 are:
> {code}
> [ERROR] COMPILATION ERROR :
> [INFO] -
> [ERROR]
> /home/vitalii/IdeaProjects/drill-fork/contrib/storage-hive/core/src/main/java/org/apache/drill/exec/store/hive/HiveUtilities.java:[58,39]
> error: cannot find symbol
> [ERROR] symbol: class MetaStoreUtils
> location: package org.apache.hadoop.hive.metastore
> /home/vitalii/IdeaProjects/drill-fork/contrib/storage-hive/core/src/main/java/org/apache/drill/exec/store/hive/HiveMetadataProvider.java:[34,39]
> error: cannot find symbol
> [INFO] 2 errors
> {code}
> {code}
> [ERROR] COMPILATION ERROR :
> [INFO] -
> [ERROR]
> /home/vitalii/IdeaProjects/drill-fork/contrib/storage-hive/core/src/main/java/org/apache/drill/exec/store/hive/HiveUtilities.java:[575,15]
> error: cannot find symbol
> [ERROR] symbol: method setTransactionalTableScan(JobConf,boolean)
> location: class AcidUtils
> /home/vitalii/IdeaProjects/drill-fork/contrib/storage-hive/core/src/main/java/org/apache/drill/exec/store/hive/HiveFieldConverter.java:[216,92]
> error: incompatible types: org.apache.hadoop.hive.common.type.Timestamp
> cannot be converted to java.sql.Timestamp
> [ERROR]
> /home/vitalii/IdeaProjects/drill-fork/contrib/storage-hive/core/src/main/java/org/apache/drill/exec/store/hive/HiveFieldConverter.java:[225,82]
> error: incompatible types: org.apache.hadoop.hive.common.type.Date cannot be
> converted to java.sql.Date
> [ERROR]
> /home/vitalii/IdeaProjects/drill-fork/contrib/storage-hive/core/src/main/java/org/apache/drill/exec/store/hive/DrillHiveMetaStoreClient.java:[100,17]
> error: cannot find symbol
> [ERROR] symbol: method setTokenStr(UserGroupInformation,String,String)
> location: class Utils
> /home/vitalii/IdeaProjects/drill-fork/contrib/storage-hive/core/target/generated-sources/org/apache/drill/exec/expr/fn/impl/hive/DrillTimeStampTimestampObjectInspector.java:[29,16]
> error: Required is not abstract and does not override abstract method
> getPrimitiveJavaObject(Object) in TimestampObjectInspector
> [ERROR]
> /home/vitalii/IdeaProjects/drill-fork/contrib/storage-hive/core/target/generated-sources/org/apache/drill/exec/expr/fn/impl/hive/DrillTimeStampTimestampObjectInspector.java:[35,30]
> error: getPrimitiveJavaObject(Object) in Required cannot implement
> getPrimitiveJavaObject(Object) in TimestampObjectInspector
> [ERROR] return type java.sql.Timestamp is not compatible with
> org.apache.hadoop.hive.common.type.Timestamp
> /home/vitalii/IdeaProjects/drill-fork/contrib/storage-hive/core/target/generated-sources/org/apache/drill/exec/expr/fn/impl/hive/DrillTimeStampTimestampObjectInspector.java:[44,29]
> error: getPrimitiveWritableObject(Object) in Required cannot implement
> getPrimitiveWritableObject(Object) in TimestampObjectInspector
> [ERROR] return type TimestampWritable is not compatible with
> TimestampWritableV2
> /home/vitalii/IdeaProjects/drill-fork/contrib/storage-hive/core/target/generated-sources/org/apache/drill/exec/expr/fn/impl/hive/DrillTimeStampTimestampObjectInspector.java:[54,16]
> error: Optional is not abstract and does not override abstract method
> getPrimitiveJavaObject(Object) in TimestampObjectInspector
> [ERROR]
> /home/vitalii/IdeaProjects/drill-fork/contrib/storage-hive/core/target/generated-sources/org/apache/drill/exec/expr/fn/impl/hive/DrillTimeStampTimestampObjectInspector.java:[60,30]
> error: getPrimitiveJavaObject(Object) in Optional cannot implement
> getPrimitiveJavaObject(Object) in TimestampObjectInspector
> [ERROR] return type java.sql.Timestamp is not compatible with
> org.apache.hadoop.hive.common.type.Timestamp
[jira] [Commented] (DRILL-7657) Invalid internal link
[ https://issues.apache.org/jira/browse/DRILL-7657?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17065423#comment-17065423 ] Vova Vysotskyi commented on DRILL-7657: --- [~扎啤], thanks for pointing to this issue. Sources for Drill Web site are placed in this branch: [https://github.com/apache/drill/tree/gh-pages]. Fell free to contribute! > Invalid internal link > - > > Key: DRILL-7657 > URL: https://issues.apache.org/jira/browse/DRILL-7657 > Project: Apache Drill > Issue Type: Bug > Components: Security >Affects Versions: 1.17.0 >Reporter: Aaron-Mhs >Priority: Major > Attachments: image-2020-03-24-14-57-53-568.png, > image-2020-03-24-14-58-17-865.png > > > There is an invalid link (Prerequisites-> See Enabling Authentication and > Encryption.) In the Configuring Kerberos Security module in the document. > After opening, it displays 404 (The requested URL was not found on this > server.), And I hope to fix it as soon as possible. > !image-2020-03-24-14-57-53-568.png! > !image-2020-03-24-14-58-17-865.png! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (DRILL-7633) Fixes for union and repeated list accessors
[ https://issues.apache.org/jira/browse/DRILL-7633?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17065394#comment-17065394 ] ASF GitHub Bot commented on DRILL-7633: --- paul-rogers commented on issue #2018: DRILL-7633: Fixes for union and repeated list accessors URL: https://github.com/apache/drill/pull/2018#issuecomment-603078801 After rebasing I'm seeing a number of metastore test failures: ``` [ERROR] Errors: [ERROR] TestTableMetadataUnitConversion.testBaseTableMetadata:145 » IllegalArgument Un... [ERROR] TestTableMetadataUnitConversion.testFileMetadata:280 » IllegalArgument Unable ... [ERROR] TestTableMetadataUnitConversion.testPartitionMetadata:383 » IllegalArgument Un... [ERROR] TestTableMetadataUnitConversion.testRowGroupMetadata:332 » IllegalArgument Una... [ERROR] TestTableMetadataUnitConversion.testSegmentMetadata:237 » IllegalArgument Unab... ``` It is not immediately obvious what went wrong; I'll poke around tomorrow to see if I can find the cause. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Fixes for union and repeated list accessors > --- > > Key: DRILL-7633 > URL: https://issues.apache.org/jira/browse/DRILL-7633 > Project: Apache Drill > Issue Type: Improvement >Affects Versions: 1.17.0 >Reporter: Paul Rogers >Assignee: Paul Rogers >Priority: Minor > Fix For: 1.18.0 > > > Minor fixes for repeated list and Union type support in column accessors -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (DRILL-7633) Fixes for union and repeated list accessors
[ https://issues.apache.org/jira/browse/DRILL-7633?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17065392#comment-17065392 ] ASF GitHub Bot commented on DRILL-7633: --- paul-rogers commented on issue #2018: DRILL-7633: Fixes for union and repeated list accessors URL: https://github.com/apache/drill/pull/2018#issuecomment-603075717 Squashed commits and rebased on the latest master. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Fixes for union and repeated list accessors > --- > > Key: DRILL-7633 > URL: https://issues.apache.org/jira/browse/DRILL-7633 > Project: Apache Drill > Issue Type: Improvement >Affects Versions: 1.17.0 >Reporter: Paul Rogers >Assignee: Paul Rogers >Priority: Minor > Fix For: 1.18.0 > > > Minor fixes for repeated list and Union type support in column accessors -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (DRILL-7633) Fixes for union and repeated list accessors
[
https://issues.apache.org/jira/browse/DRILL-7633?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17065389#comment-17065389
]
ASF GitHub Bot commented on DRILL-7633:
---
paul-rogers commented on pull request #2018: DRILL-7633: Fixes for union and
repeated list accessors
URL: https://github.com/apache/drill/pull/2018#discussion_r396947139
##
File path:
exec/vector/src/main/java/org/apache/drill/exec/record/metadata/VariantColumnMetadata.java
##
@@ -96,11 +134,13 @@ public StructureType structureType() {
public boolean isVariant() { return true; }
@Override
- public boolean isArray() { return type() == MinorType.LIST; }
+ public boolean isArray() {
+return super.isArray() || type() == MinorType.LIST;
+ }
@Override
public ColumnMetadata cloneEmpty() {
-return new VariantColumnMetadata(name, type, variantSchema.cloneEmpty());
+return new VariantColumnMetadata(name, type, mode, new VariantSchema());
Review comment:
@arina-ielchiieva, thanks for the reminder; I did miss your response.
I think I see where we had a misunderstanding. I thought the `typeString()`
code already worked because I saw this in `AbstractColumnMetadata`:
```
public String typeString() {
return majorType().toString();
}
```
`PrimitiveColumnMetadata` uses a different set of type names.
It seems this area has gotten a bit muddled: the `AbstractColumnMetadata`
version seems to never be called, except for `VariantColumnMetadata`. (This
confused the heck out of me on an earlier attempt to clean up this area, but I
didn't clue into the problem then.)
So, now that I understand what you are asking for, I did this:
* Removed the `AbstractColumnMetadata` version.
* Added a `VariantColumnMetadata` that produces either `UNION` or
`ARRAY`.
Until I see a good reason otherwise, I think we should treat `UNION` (what I
call a "variant") as an opaque type. I guess I don't see the contents of a
variant as something we want to specify, either in the metastore or in a
provided schema. For example, can the metastore compute an NDV across an `INT`,
`VARCHAR` and `MAP`? Probably not. Nor do the min/max values or other stats
make sense for a variant. As a result, a variant is an opaque type for the
metastore.
Similarly, in a provided schema, I can't convince myself that the user wants
not only to say, "the type of this field can vary", but also that "the type can
be an `INT`, `DOUBLE` or `VARCHAR`, but not a `BIGINT`.) Just does not seem
super-useful.
An additional concern, which I think I mentioned somewhere else, is that as
soon as we start serializing complex types, the SQL-like text format becomes
far too cumbersome. We'd be much better of with a JSON format that can capture
the complex tree structure. Compare:
```
(A UNION)>)
```
With something like:
```
{ schema: [
{name: "a",
type: {
name: "UNION",
nullable: true,
subtypes: [
{name: INT, nullable: false},
{name: MAP, nullable: false},
members: [ ...
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Fixes for union and repeated list accessors
> ---
>
> Key: DRILL-7633
> URL: https://issues.apache.org/jira/browse/DRILL-7633
> Project: Apache Drill
> Issue Type: Improvement
>Affects Versions: 1.17.0
>Reporter: Paul Rogers
>Assignee: Paul Rogers
>Priority: Minor
> Fix For: 1.18.0
>
>
> Minor fixes for repeated list and Union type support in column accessors
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (DRILL-7657) Invalid internal link
Aaron-Mhs created DRILL-7657: Summary: Invalid internal link Key: DRILL-7657 URL: https://issues.apache.org/jira/browse/DRILL-7657 Project: Apache Drill Issue Type: Bug Components: Security Affects Versions: 1.17.0 Reporter: Aaron-Mhs Attachments: image-2020-03-24-14-57-53-568.png, image-2020-03-24-14-58-17-865.png There is an invalid link (Prerequisites-> See Enabling Authentication and Encryption.) In the Configuring Kerberos Security module in the document. After opening, it displays 404 (The requested URL was not found on this server.), And I hope to fix it as soon as possible. !image-2020-03-24-14-57-53-568.png! !image-2020-03-24-14-58-17-865.png! -- This message was sent by Atlassian Jira (v8.3.4#803005)
