[jira] [Updated] (HIVE-19307) Support ArrowOutputStream in LlapOutputFormatService
[ https://issues.apache.org/jira/browse/HIVE-19307?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Wohlstadter updated HIVE-19307: Status: Patch Available (was: Open) > Support ArrowOutputStream in LlapOutputFormatService > > > Key: HIVE-19307 > URL: https://issues.apache.org/jira/browse/HIVE-19307 > Project: Hive > Issue Type: Task > Components: llap >Reporter: Eric Wohlstadter >Assignee: Eric Wohlstadter >Priority: Major > Attachments: HIVE-19307.3.patch, HIVE-19307.4.patch, > HIVE-19307.5.patch, HIVE-19307.6.patch, HIVE-19307.7.patch, HIVE-19307.8.patch > > > Support pushing arrow batches through > org.apache.arrow.vector.ipc.ArrowOutputStream in LllapOutputFormatService. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19307) Support ArrowOutputStream in LlapOutputFormatService
[ https://issues.apache.org/jira/browse/HIVE-19307?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Wohlstadter updated HIVE-19307: Attachment: HIVE-19307.8.patch > Support ArrowOutputStream in LlapOutputFormatService > > > Key: HIVE-19307 > URL: https://issues.apache.org/jira/browse/HIVE-19307 > Project: Hive > Issue Type: Task > Components: llap >Reporter: Eric Wohlstadter >Assignee: Eric Wohlstadter >Priority: Major > Attachments: HIVE-19307.3.patch, HIVE-19307.4.patch, > HIVE-19307.5.patch, HIVE-19307.6.patch, HIVE-19307.7.patch, HIVE-19307.8.patch > > > Support pushing arrow batches through > org.apache.arrow.vector.ipc.ArrowOutputStream in LllapOutputFormatService. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19307) Support ArrowOutputStream in LlapOutputFormatService
[ https://issues.apache.org/jira/browse/HIVE-19307?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Wohlstadter updated HIVE-19307: Status: Open (was: Patch Available) > Support ArrowOutputStream in LlapOutputFormatService > > > Key: HIVE-19307 > URL: https://issues.apache.org/jira/browse/HIVE-19307 > Project: Hive > Issue Type: Task > Components: llap >Reporter: Eric Wohlstadter >Assignee: Eric Wohlstadter >Priority: Major > Attachments: HIVE-19307.3.patch, HIVE-19307.4.patch, > HIVE-19307.5.patch, HIVE-19307.6.patch, HIVE-19307.7.patch > > > Support pushing arrow batches through > org.apache.arrow.vector.ipc.ArrowOutputStream in LllapOutputFormatService. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19307) Support ArrowOutputStream in LlapOutputFormatService
[ https://issues.apache.org/jira/browse/HIVE-19307?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Wohlstadter updated HIVE-19307: Attachment: HIVE-19307.4.patch > Support ArrowOutputStream in LlapOutputFormatService > > > Key: HIVE-19307 > URL: https://issues.apache.org/jira/browse/HIVE-19307 > Project: Hive > Issue Type: Task > Components: llap >Reporter: Eric Wohlstadter >Assignee: Eric Wohlstadter >Priority: Major > Attachments: HIVE-19307.3.patch, HIVE-19307.4.patch > > > Support pushing arrow batches through > org.apache.arrow.vector.ipc.ArrowOutputStream in LllapOutputFormatService. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19307) Support ArrowOutputStream in LlapOutputFormatService
[ https://issues.apache.org/jira/browse/HIVE-19307?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Wohlstadter updated HIVE-19307: Status: Open (was: Patch Available) > Support ArrowOutputStream in LlapOutputFormatService > > > Key: HIVE-19307 > URL: https://issues.apache.org/jira/browse/HIVE-19307 > Project: Hive > Issue Type: Task > Components: llap >Reporter: Eric Wohlstadter >Assignee: Eric Wohlstadter >Priority: Major > Attachments: HIVE-19307.3.patch, HIVE-19307.4.patch > > > Support pushing arrow batches through > org.apache.arrow.vector.ipc.ArrowOutputStream in LllapOutputFormatService. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19445) Graceful handling of "close" in WritableByteChannelAdapter
[
https://issues.apache.org/jira/browse/HIVE-19445?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Wohlstadter updated HIVE-19445:
Issue Type: Sub-task (was: Bug)
Parent: HIVE-19305
> Graceful handling of "close" in WritableByteChannelAdapter
> --
>
> Key: HIVE-19445
> URL: https://issues.apache.org/jira/browse/HIVE-19445
> Project: Hive
> Issue Type: Sub-task
>Reporter: Eric Wohlstadter
>Priority: Major
>
> org.apache.hadoop.hive.llap.WritableByteChannelAdapter
> {quote}"I see now that the writeListener could be implemented in such a way
> as to propagate a write error back to the writer (so we can possibly throw an
> exception and fail the current operation rather than just log and ignore the
> error). Plus on close I'm wondering if it is better just to wait for the
> close future to complete so we can check the status."
> {quote}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19307) Support ArrowOutputStream in LlapOutputFormatService
[
https://issues.apache.org/jira/browse/HIVE-19307?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16466520#comment-16466520
]
Eric Wohlstadter commented on HIVE-19307:
-
[~jdere]
I created HIVE-19445 to followup on:
{quote}I see now that the writeListener could be implemented in such a way as
to propagate a write error back to the writer (so we can possibly throw an
exception and fail the current operation rather than just log and ignore the
error). Plus on close I'm wondering if it is better just to wait for the close
future to complete so we can check the status.
{quote}
Will upload a new patch to fix the other suggestions.
> Support ArrowOutputStream in LlapOutputFormatService
>
>
> Key: HIVE-19307
> URL: https://issues.apache.org/jira/browse/HIVE-19307
> Project: Hive
> Issue Type: Task
> Components: llap
>Reporter: Eric Wohlstadter
>Assignee: Eric Wohlstadter
>Priority: Major
> Attachments: HIVE-19307.3.patch
>
>
> Support pushing arrow batches through
> org.apache.arrow.vector.ipc.ArrowOutputStream in LllapOutputFormatService.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work started] (HIVE-19307) Support ArrowOutputStream in LlapOutputFormatService
[ https://issues.apache.org/jira/browse/HIVE-19307?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HIVE-19307 started by Eric Wohlstadter. --- > Support ArrowOutputStream in LlapOutputFormatService > > > Key: HIVE-19307 > URL: https://issues.apache.org/jira/browse/HIVE-19307 > Project: Hive > Issue Type: Task > Components: llap >Reporter: Eric Wohlstadter >Assignee: Eric Wohlstadter >Priority: Major > Attachments: HIVE-19307.3.patch, HIVE-19307.4.patch > > > Support pushing arrow batches through > org.apache.arrow.vector.ipc.ArrowOutputStream in LllapOutputFormatService. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19307) Support ArrowOutputStream in LlapOutputFormatService
[ https://issues.apache.org/jira/browse/HIVE-19307?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16469172#comment-16469172 ] Eric Wohlstadter commented on HIVE-19307: - [~jdere] That makes sense. I'll make that change in the next patch. > Support ArrowOutputStream in LlapOutputFormatService > > > Key: HIVE-19307 > URL: https://issues.apache.org/jira/browse/HIVE-19307 > Project: Hive > Issue Type: Task > Components: llap >Reporter: Eric Wohlstadter >Assignee: Eric Wohlstadter >Priority: Major > Attachments: HIVE-19307.3.patch, HIVE-19307.4.patch > > > Support pushing arrow batches through > org.apache.arrow.vector.ipc.ArrowOutputStream in LllapOutputFormatService. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19307) Support ArrowOutputStream in LlapOutputFormatService
[ https://issues.apache.org/jira/browse/HIVE-19307?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Wohlstadter updated HIVE-19307: Attachment: HIVE-19307.2.patch > Support ArrowOutputStream in LlapOutputFormatService > > > Key: HIVE-19307 > URL: https://issues.apache.org/jira/browse/HIVE-19307 > Project: Hive > Issue Type: Task > Components: llap >Reporter: Eric Wohlstadter >Assignee: Eric Wohlstadter >Priority: Major > Attachments: HIVE-19307.2.patch > > > Support pushing arrow batches through > org.apache.arrow.vector.ipc.ArrowOutputStream in LllapOutputFormatService. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19307) Support ArrowOutputStream in LlapOutputFormatService
[ https://issues.apache.org/jira/browse/HIVE-19307?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Wohlstadter updated HIVE-19307: Attachment: (was: HIVE-19307.1.patch) > Support ArrowOutputStream in LlapOutputFormatService > > > Key: HIVE-19307 > URL: https://issues.apache.org/jira/browse/HIVE-19307 > Project: Hive > Issue Type: Task > Components: llap >Reporter: Eric Wohlstadter >Assignee: Eric Wohlstadter >Priority: Major > > Support pushing arrow batches through > org.apache.arrow.vector.ipc.ArrowOutputStream in LllapOutputFormatService. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-19495) Arrow SerDe itest failure
[ https://issues.apache.org/jira/browse/HIVE-19495?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Wohlstadter reassigned HIVE-19495: --- > Arrow SerDe itest failure > - > > Key: HIVE-19495 > URL: https://issues.apache.org/jira/browse/HIVE-19495 > Project: Hive > Issue Type: Sub-task > Components: Serializers/Deserializers >Reporter: Eric Wohlstadter >Assignee: Teddy Choi >Priority: Major > Fix For: 3.1.0 > > > "You tried to write a Bit type when you are using a ValueWriter of type > NullableMapWriter." -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19307) Support ArrowOutputStream in LlapOutputFormatService
[ https://issues.apache.org/jira/browse/HIVE-19307?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Wohlstadter updated HIVE-19307: Attachment: HIVE-19307.5.patch > Support ArrowOutputStream in LlapOutputFormatService > > > Key: HIVE-19307 > URL: https://issues.apache.org/jira/browse/HIVE-19307 > Project: Hive > Issue Type: Task > Components: llap >Reporter: Eric Wohlstadter >Assignee: Eric Wohlstadter >Priority: Major > Attachments: HIVE-19307.3.patch, HIVE-19307.4.patch, > HIVE-19307.5.patch > > > Support pushing arrow batches through > org.apache.arrow.vector.ipc.ArrowOutputStream in LllapOutputFormatService. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19307) Support ArrowOutputStream in LlapOutputFormatService
[ https://issues.apache.org/jira/browse/HIVE-19307?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Wohlstadter updated HIVE-19307: Status: Patch Available (was: In Progress) > Support ArrowOutputStream in LlapOutputFormatService > > > Key: HIVE-19307 > URL: https://issues.apache.org/jira/browse/HIVE-19307 > Project: Hive > Issue Type: Task > Components: llap >Reporter: Eric Wohlstadter >Assignee: Eric Wohlstadter >Priority: Major > Attachments: HIVE-19307.3.patch, HIVE-19307.4.patch, > HIVE-19307.5.patch > > > Support pushing arrow batches through > org.apache.arrow.vector.ipc.ArrowOutputStream in LllapOutputFormatService. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19307) Support ArrowOutputStream in LlapOutputFormatService
[ https://issues.apache.org/jira/browse/HIVE-19307?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Wohlstadter updated HIVE-19307: Attachment: HIVE-19307.6.patch > Support ArrowOutputStream in LlapOutputFormatService > > > Key: HIVE-19307 > URL: https://issues.apache.org/jira/browse/HIVE-19307 > Project: Hive > Issue Type: Task > Components: llap >Reporter: Eric Wohlstadter >Assignee: Eric Wohlstadter >Priority: Major > Attachments: HIVE-19307.3.patch, HIVE-19307.4.patch, > HIVE-19307.5.patch, HIVE-19307.6.patch > > > Support pushing arrow batches through > org.apache.arrow.vector.ipc.ArrowOutputStream in LllapOutputFormatService. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19307) Support ArrowOutputStream in LlapOutputFormatService
[ https://issues.apache.org/jira/browse/HIVE-19307?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Wohlstadter updated HIVE-19307: Status: Open (was: Patch Available) > Support ArrowOutputStream in LlapOutputFormatService > > > Key: HIVE-19307 > URL: https://issues.apache.org/jira/browse/HIVE-19307 > Project: Hive > Issue Type: Task > Components: llap >Reporter: Eric Wohlstadter >Assignee: Eric Wohlstadter >Priority: Major > Attachments: HIVE-19307.3.patch, HIVE-19307.4.patch, > HIVE-19307.5.patch > > > Support pushing arrow batches through > org.apache.arrow.vector.ipc.ArrowOutputStream in LllapOutputFormatService. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19307) Support ArrowOutputStream in LlapOutputFormatService
[ https://issues.apache.org/jira/browse/HIVE-19307?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Wohlstadter updated HIVE-19307: Status: Patch Available (was: Open) Fixed some formatting and added more comments > Support ArrowOutputStream in LlapOutputFormatService > > > Key: HIVE-19307 > URL: https://issues.apache.org/jira/browse/HIVE-19307 > Project: Hive > Issue Type: Task > Components: llap >Reporter: Eric Wohlstadter >Assignee: Eric Wohlstadter >Priority: Major > Attachments: HIVE-19307.3.patch, HIVE-19307.4.patch, > HIVE-19307.5.patch, HIVE-19307.6.patch > > > Support pushing arrow batches through > org.apache.arrow.vector.ipc.ArrowOutputStream in LllapOutputFormatService. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19307) Support ArrowOutputStream in LlapOutputFormatService
[ https://issues.apache.org/jira/browse/HIVE-19307?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Wohlstadter updated HIVE-19307: Attachment: HIVE-19307.7.patch > Support ArrowOutputStream in LlapOutputFormatService > > > Key: HIVE-19307 > URL: https://issues.apache.org/jira/browse/HIVE-19307 > Project: Hive > Issue Type: Task > Components: llap >Reporter: Eric Wohlstadter >Assignee: Eric Wohlstadter >Priority: Major > Attachments: HIVE-19307.3.patch, HIVE-19307.4.patch, > HIVE-19307.5.patch, HIVE-19307.6.patch, HIVE-19307.7.patch > > > Support pushing arrow batches through > org.apache.arrow.vector.ipc.ArrowOutputStream in LllapOutputFormatService. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19307) Support ArrowOutputStream in LlapOutputFormatService
[ https://issues.apache.org/jira/browse/HIVE-19307?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Wohlstadter updated HIVE-19307: Status: Open (was: Patch Available) > Support ArrowOutputStream in LlapOutputFormatService > > > Key: HIVE-19307 > URL: https://issues.apache.org/jira/browse/HIVE-19307 > Project: Hive > Issue Type: Task > Components: llap >Reporter: Eric Wohlstadter >Assignee: Eric Wohlstadter >Priority: Major > Attachments: HIVE-19307.3.patch, HIVE-19307.4.patch, > HIVE-19307.5.patch, HIVE-19307.6.patch > > > Support pushing arrow batches through > org.apache.arrow.vector.ipc.ArrowOutputStream in LllapOutputFormatService. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19307) Support ArrowOutputStream in LlapOutputFormatService
[ https://issues.apache.org/jira/browse/HIVE-19307?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Wohlstadter updated HIVE-19307: Status: Patch Available (was: Open) > Support ArrowOutputStream in LlapOutputFormatService > > > Key: HIVE-19307 > URL: https://issues.apache.org/jira/browse/HIVE-19307 > Project: Hive > Issue Type: Task > Components: llap >Reporter: Eric Wohlstadter >Assignee: Eric Wohlstadter >Priority: Major > Attachments: HIVE-19307.3.patch, HIVE-19307.4.patch, > HIVE-19307.5.patch, HIVE-19307.6.patch, HIVE-19307.7.patch > > > Support pushing arrow batches through > org.apache.arrow.vector.ipc.ArrowOutputStream in LllapOutputFormatService. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19306) Arrow batch serializer
[ https://issues.apache.org/jira/browse/HIVE-19306?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16466188#comment-16466188 ] Eric Wohlstadter commented on HIVE-19306: - +1 lgtm > Arrow batch serializer > -- > > Key: HIVE-19306 > URL: https://issues.apache.org/jira/browse/HIVE-19306 > Project: Hive > Issue Type: Task > Components: Serializers/Deserializers >Reporter: Eric Wohlstadter >Assignee: Teddy Choi >Priority: Major > Attachments: HIVE-19306.2.patch > > > Leverage the ThriftJDBCBinarySerDe code path that already exists in > SemanticAnalyzer/FileSinkOperator to create a serializer that batches rows > into Arrow vector batches. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19306) Arrow batch serializer
[ https://issues.apache.org/jira/browse/HIVE-19306?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Wohlstadter updated HIVE-19306: Status: Open (was: Patch Available) > Arrow batch serializer > -- > > Key: HIVE-19306 > URL: https://issues.apache.org/jira/browse/HIVE-19306 > Project: Hive > Issue Type: Task > Components: Serializers/Deserializers >Reporter: Eric Wohlstadter >Assignee: Teddy Choi >Priority: Major > Attachments: HIVE-19306.2.patch, HIVE-19306.3.patch > > > Leverage the ThriftJDBCBinarySerDe code path that already exists in > SemanticAnalyzer/FileSinkOperator to create a serializer that batches rows > into Arrow vector batches. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19306) Arrow batch serializer
[ https://issues.apache.org/jira/browse/HIVE-19306?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Wohlstadter updated HIVE-19306: Status: Patch Available (was: Open) > Arrow batch serializer > -- > > Key: HIVE-19306 > URL: https://issues.apache.org/jira/browse/HIVE-19306 > Project: Hive > Issue Type: Task > Components: Serializers/Deserializers >Reporter: Eric Wohlstadter >Assignee: Teddy Choi >Priority: Major > Attachments: HIVE-19306.2.patch, HIVE-19306.3.patch > > > Leverage the ThriftJDBCBinarySerDe code path that already exists in > SemanticAnalyzer/FileSinkOperator to create a serializer that batches rows > into Arrow vector batches. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19306) Arrow batch serializer
[ https://issues.apache.org/jira/browse/HIVE-19306?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Wohlstadter updated HIVE-19306: Attachment: HIVE-19306.3.patch > Arrow batch serializer > -- > > Key: HIVE-19306 > URL: https://issues.apache.org/jira/browse/HIVE-19306 > Project: Hive > Issue Type: Task > Components: Serializers/Deserializers >Reporter: Eric Wohlstadter >Assignee: Teddy Choi >Priority: Major > Attachments: HIVE-19306.2.patch, HIVE-19306.3.patch > > > Leverage the ThriftJDBCBinarySerDe code path that already exists in > SemanticAnalyzer/FileSinkOperator to create a serializer that batches rows > into Arrow vector batches. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19308) Provide an Arrow stream reader for external LLAP clients
[ https://issues.apache.org/jira/browse/HIVE-19308?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Wohlstadter updated HIVE-19308: Attachment: HIVE-19308.6.patch > Provide an Arrow stream reader for external LLAP clients > - > > Key: HIVE-19308 > URL: https://issues.apache.org/jira/browse/HIVE-19308 > Project: Hive > Issue Type: Task > Components: llap >Reporter: Eric Wohlstadter >Assignee: Eric Wohlstadter >Priority: Major > Attachments: HIVE-19308.1.patch, HIVE-19308.2.patch, > HIVE-19308.3.patch, HIVE-19308.4.patch, HIVE-19308.5.patch, HIVE-19308.6.patch > > > This is a sub-class of LlapBaseRecordReader that wraps the socket inputStream > and produces Arrow batches for an external client. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19308) Provide an Arrow stream reader for external LLAP clients
[ https://issues.apache.org/jira/browse/HIVE-19308?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Wohlstadter updated HIVE-19308: Status: Patch Available (was: Open) > Provide an Arrow stream reader for external LLAP clients > - > > Key: HIVE-19308 > URL: https://issues.apache.org/jira/browse/HIVE-19308 > Project: Hive > Issue Type: Task > Components: llap >Reporter: Eric Wohlstadter >Assignee: Eric Wohlstadter >Priority: Major > Attachments: HIVE-19308.1.patch, HIVE-19308.2.patch, > HIVE-19308.3.patch, HIVE-19308.4.patch, HIVE-19308.5.patch, HIVE-19308.6.patch > > > This is a sub-class of LlapBaseRecordReader that wraps the socket inputStream > and produces Arrow batches for an external client. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19308) Provide an Arrow stream reader for external LLAP clients
[
https://issues.apache.org/jira/browse/HIVE-19308?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16480042#comment-16480042
]
Eric Wohlstadter commented on HIVE-19308:
-
[~jdere]
For the current test failure ... interestingly:
1. All the tests pass for both Row and Arrow, except when
{{testNonAsciiStrings}} is run for the second test.
2. It doesn't matter whether Row or Arrow runs first. {{testNonAsciiStrings}}
will fail for the second.
3. There doesn't seem to be anything special about {{testNonAsciiStrings}},
e.g. it isn't the first test case run, it reuses the same create table function
as the other test cases, etc.
I'll keep digging. If you think of anything special about
{{testNonAsciiStrings}} that I might be missing, please let me know.
> Provide an Arrow stream reader for external LLAP clients
> -
>
> Key: HIVE-19308
> URL: https://issues.apache.org/jira/browse/HIVE-19308
> Project: Hive
> Issue Type: Task
> Components: llap
>Reporter: Eric Wohlstadter
>Assignee: Eric Wohlstadter
>Priority: Major
> Attachments: HIVE-19308.1.patch, HIVE-19308.2.patch,
> HIVE-19308.3.patch, HIVE-19308.4.patch, HIVE-19308.5.patch
>
>
> This is a sub-class of LlapBaseRecordReader that wraps the socket inputStream
> and produces Arrow batches for an external client.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19308) Provide an Arrow stream reader for external LLAP clients
[ https://issues.apache.org/jira/browse/HIVE-19308?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Wohlstadter updated HIVE-19308: Status: Open (was: Patch Available) > Provide an Arrow stream reader for external LLAP clients > - > > Key: HIVE-19308 > URL: https://issues.apache.org/jira/browse/HIVE-19308 > Project: Hive > Issue Type: Task > Components: llap >Reporter: Eric Wohlstadter >Assignee: Eric Wohlstadter >Priority: Major > Attachments: HIVE-19308.1.patch, HIVE-19308.2.patch, > HIVE-19308.3.patch, HIVE-19308.4.patch, HIVE-19308.5.patch > > > This is a sub-class of LlapBaseRecordReader that wraps the socket inputStream > and produces Arrow batches for an external client. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19308) Provide an Arrow stream reader for external LLAP clients
[
https://issues.apache.org/jira/browse/HIVE-19308?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16480229#comment-16480229
]
Eric Wohlstadter commented on HIVE-19308:
-
Fixed the test failure by not-specifying a specific database name in
{{testNonAsciiStrings}}.
> Provide an Arrow stream reader for external LLAP clients
> -
>
> Key: HIVE-19308
> URL: https://issues.apache.org/jira/browse/HIVE-19308
> Project: Hive
> Issue Type: Task
> Components: llap
>Reporter: Eric Wohlstadter
>Assignee: Eric Wohlstadter
>Priority: Major
> Attachments: HIVE-19308.1.patch, HIVE-19308.2.patch,
> HIVE-19308.3.patch, HIVE-19308.4.patch, HIVE-19308.5.patch
>
>
> This is a sub-class of LlapBaseRecordReader that wraps the socket inputStream
> and produces Arrow batches for an external client.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19308) Provide an Arrow stream reader for external LLAP clients
[ https://issues.apache.org/jira/browse/HIVE-19308?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Wohlstadter updated HIVE-19308: Attachment: HIVE-19308.2.patch > Provide an Arrow stream reader for external LLAP clients > - > > Key: HIVE-19308 > URL: https://issues.apache.org/jira/browse/HIVE-19308 > Project: Hive > Issue Type: Task > Components: llap >Reporter: Eric Wohlstadter >Assignee: Eric Wohlstadter >Priority: Major > Attachments: HIVE-19308.1.patch, HIVE-19308.2.patch > > > This is a sub-class of LlapBaseRecordReader that wraps the socket inputStream > and produces Arrow batches for an external client. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19308) Provide an Arrow stream reader for external LLAP clients
[ https://issues.apache.org/jira/browse/HIVE-19308?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Wohlstadter updated HIVE-19308: Status: Patch Available (was: In Progress) > Provide an Arrow stream reader for external LLAP clients > - > > Key: HIVE-19308 > URL: https://issues.apache.org/jira/browse/HIVE-19308 > Project: Hive > Issue Type: Task > Components: llap >Reporter: Eric Wohlstadter >Assignee: Eric Wohlstadter >Priority: Major > Attachments: HIVE-19308.1.patch, HIVE-19308.2.patch > > > This is a sub-class of LlapBaseRecordReader that wraps the socket inputStream > and produces Arrow batches for an external client. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19308) Provide an Arrow stream reader for external LLAP clients
[ https://issues.apache.org/jira/browse/HIVE-19308?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Wohlstadter updated HIVE-19308: Status: Open (was: Patch Available) > Provide an Arrow stream reader for external LLAP clients > - > > Key: HIVE-19308 > URL: https://issues.apache.org/jira/browse/HIVE-19308 > Project: Hive > Issue Type: Task > Components: llap >Reporter: Eric Wohlstadter >Assignee: Eric Wohlstadter >Priority: Major > Attachments: HIVE-19308.1.patch, HIVE-19308.2.patch, > HIVE-19308.3.patch, HIVE-19308.4.patch, HIVE-19308.5.patch, > HIVE-19308.6.patch, HIVE-19308.7.patch, HIVE-19308.8.patch > > > This is a sub-class of LlapBaseRecordReader that wraps the socket inputStream > and produces Arrow batches for an external client. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19308) Provide an Arrow stream reader for external LLAP clients
[ https://issues.apache.org/jira/browse/HIVE-19308?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Wohlstadter updated HIVE-19308: Attachment: HIVE-19308.8.patch > Provide an Arrow stream reader for external LLAP clients > - > > Key: HIVE-19308 > URL: https://issues.apache.org/jira/browse/HIVE-19308 > Project: Hive > Issue Type: Task > Components: llap >Reporter: Eric Wohlstadter >Assignee: Eric Wohlstadter >Priority: Major > Attachments: HIVE-19308.1.patch, HIVE-19308.2.patch, > HIVE-19308.3.patch, HIVE-19308.4.patch, HIVE-19308.5.patch, > HIVE-19308.6.patch, HIVE-19308.7.patch, HIVE-19308.8.patch > > > This is a sub-class of LlapBaseRecordReader that wraps the socket inputStream > and produces Arrow batches for an external client. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19308) Provide an Arrow stream reader for external LLAP clients
[ https://issues.apache.org/jira/browse/HIVE-19308?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Wohlstadter updated HIVE-19308: Status: Patch Available (was: Open) > Provide an Arrow stream reader for external LLAP clients > - > > Key: HIVE-19308 > URL: https://issues.apache.org/jira/browse/HIVE-19308 > Project: Hive > Issue Type: Task > Components: llap >Reporter: Eric Wohlstadter >Assignee: Eric Wohlstadter >Priority: Major > Attachments: HIVE-19308.1.patch, HIVE-19308.2.patch, > HIVE-19308.3.patch, HIVE-19308.4.patch, HIVE-19308.5.patch, > HIVE-19308.6.patch, HIVE-19308.7.patch, HIVE-19308.8.patch > > > This is a sub-class of LlapBaseRecordReader that wraps the socket inputStream > and produces Arrow batches for an external client. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19308) Provide an Arrow stream reader for external LLAP clients
[ https://issues.apache.org/jira/browse/HIVE-19308?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Wohlstadter updated HIVE-19308: Attachment: HIVE-19308.10.patch > Provide an Arrow stream reader for external LLAP clients > - > > Key: HIVE-19308 > URL: https://issues.apache.org/jira/browse/HIVE-19308 > Project: Hive > Issue Type: Task > Components: llap >Reporter: Eric Wohlstadter >Assignee: Eric Wohlstadter >Priority: Major > Attachments: HIVE-19308.1.patch, HIVE-19308.10.patch, > HIVE-19308.2.patch, HIVE-19308.3.patch, HIVE-19308.4.patch, > HIVE-19308.5.patch, HIVE-19308.6.patch, HIVE-19308.7.patch, > HIVE-19308.8.patch, HIVE-19308.9.patch > > > This is a sub-class of LlapBaseRecordReader that wraps the socket inputStream > and produces Arrow batches for an external client. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19308) Provide an Arrow stream reader for external LLAP clients
[ https://issues.apache.org/jira/browse/HIVE-19308?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Wohlstadter updated HIVE-19308: Status: Patch Available (was: Open) > Provide an Arrow stream reader for external LLAP clients > - > > Key: HIVE-19308 > URL: https://issues.apache.org/jira/browse/HIVE-19308 > Project: Hive > Issue Type: Task > Components: llap >Reporter: Eric Wohlstadter >Assignee: Eric Wohlstadter >Priority: Major > Attachments: HIVE-19308.1.patch, HIVE-19308.10.patch, > HIVE-19308.2.patch, HIVE-19308.3.patch, HIVE-19308.4.patch, > HIVE-19308.5.patch, HIVE-19308.6.patch, HIVE-19308.7.patch, > HIVE-19308.8.patch, HIVE-19308.9.patch > > > This is a sub-class of LlapBaseRecordReader that wraps the socket inputStream > and produces Arrow batches for an external client. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19308) Provide an Arrow stream reader for external LLAP clients
[ https://issues.apache.org/jira/browse/HIVE-19308?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Wohlstadter updated HIVE-19308: Status: Open (was: Patch Available) > Provide an Arrow stream reader for external LLAP clients > - > > Key: HIVE-19308 > URL: https://issues.apache.org/jira/browse/HIVE-19308 > Project: Hive > Issue Type: Task > Components: llap >Reporter: Eric Wohlstadter >Assignee: Eric Wohlstadter >Priority: Major > Attachments: HIVE-19308.1.patch, HIVE-19308.10.patch, > HIVE-19308.2.patch, HIVE-19308.3.patch, HIVE-19308.4.patch, > HIVE-19308.5.patch, HIVE-19308.6.patch, HIVE-19308.7.patch, > HIVE-19308.8.patch, HIVE-19308.9.patch > > > This is a sub-class of LlapBaseRecordReader that wraps the socket inputStream > and produces Arrow batches for an external client. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19308) Provide an Arrow stream reader for external LLAP clients
[ https://issues.apache.org/jira/browse/HIVE-19308?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Wohlstadter updated HIVE-19308: Status: Patch Available (was: Open) > Provide an Arrow stream reader for external LLAP clients > - > > Key: HIVE-19308 > URL: https://issues.apache.org/jira/browse/HIVE-19308 > Project: Hive > Issue Type: Task > Components: llap >Reporter: Eric Wohlstadter >Assignee: Eric Wohlstadter >Priority: Major > Attachments: HIVE-19308.1.patch, HIVE-19308.2.patch, > HIVE-19308.3.patch, HIVE-19308.4.patch, HIVE-19308.5.patch, > HIVE-19308.6.patch, HIVE-19308.7.patch, HIVE-19308.8.patch, HIVE-19308.9.patch > > > This is a sub-class of LlapBaseRecordReader that wraps the socket inputStream > and produces Arrow batches for an external client. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19308) Provide an Arrow stream reader for external LLAP clients
[ https://issues.apache.org/jira/browse/HIVE-19308?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Wohlstadter updated HIVE-19308: Status: Open (was: Patch Available) > Provide an Arrow stream reader for external LLAP clients > - > > Key: HIVE-19308 > URL: https://issues.apache.org/jira/browse/HIVE-19308 > Project: Hive > Issue Type: Task > Components: llap >Reporter: Eric Wohlstadter >Assignee: Eric Wohlstadter >Priority: Major > Attachments: HIVE-19308.1.patch, HIVE-19308.2.patch, > HIVE-19308.3.patch, HIVE-19308.4.patch, HIVE-19308.5.patch, > HIVE-19308.6.patch, HIVE-19308.7.patch, HIVE-19308.8.patch, HIVE-19308.9.patch > > > This is a sub-class of LlapBaseRecordReader that wraps the socket inputStream > and produces Arrow batches for an external client. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19308) Provide an Arrow stream reader for external LLAP clients
[ https://issues.apache.org/jira/browse/HIVE-19308?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Wohlstadter updated HIVE-19308: Attachment: HIVE-19308.9.patch > Provide an Arrow stream reader for external LLAP clients > - > > Key: HIVE-19308 > URL: https://issues.apache.org/jira/browse/HIVE-19308 > Project: Hive > Issue Type: Task > Components: llap >Reporter: Eric Wohlstadter >Assignee: Eric Wohlstadter >Priority: Major > Attachments: HIVE-19308.1.patch, HIVE-19308.2.patch, > HIVE-19308.3.patch, HIVE-19308.4.patch, HIVE-19308.5.patch, > HIVE-19308.6.patch, HIVE-19308.7.patch, HIVE-19308.8.patch, HIVE-19308.9.patch > > > This is a sub-class of LlapBaseRecordReader that wraps the socket inputStream > and produces Arrow batches for an external client. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19308) Provide an Arrow stream reader for external LLAP clients
[ https://issues.apache.org/jira/browse/HIVE-19308?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16482081#comment-16482081 ] Eric Wohlstadter commented on HIVE-19308: - [~jdere] Can you merge if you are ready? Got a green build/test run now. > Provide an Arrow stream reader for external LLAP clients > - > > Key: HIVE-19308 > URL: https://issues.apache.org/jira/browse/HIVE-19308 > Project: Hive > Issue Type: Task > Components: llap >Reporter: Eric Wohlstadter >Assignee: Eric Wohlstadter >Priority: Major > Attachments: HIVE-19308.1.patch, HIVE-19308.10.patch, > HIVE-19308.2.patch, HIVE-19308.3.patch, HIVE-19308.4.patch, > HIVE-19308.5.patch, HIVE-19308.6.patch, HIVE-19308.7.patch, > HIVE-19308.8.patch, HIVE-19308.9.patch > > > This is a sub-class of LlapBaseRecordReader that wraps the socket inputStream > and produces Arrow batches for an external client. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19308) Provide an Arrow stream reader for external LLAP clients
[ https://issues.apache.org/jira/browse/HIVE-19308?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Wohlstadter updated HIVE-19308: Status: Open (was: Patch Available) > Provide an Arrow stream reader for external LLAP clients > - > > Key: HIVE-19308 > URL: https://issues.apache.org/jira/browse/HIVE-19308 > Project: Hive > Issue Type: Task > Components: llap >Reporter: Eric Wohlstadter >Assignee: Eric Wohlstadter >Priority: Major > Attachments: HIVE-19308.1.patch, HIVE-19308.2.patch, > HIVE-19308.3.patch, HIVE-19308.4.patch > > > This is a sub-class of LlapBaseRecordReader that wraps the socket inputStream > and produces Arrow batches for an external client. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19308) Provide an Arrow stream reader for external LLAP clients
[ https://issues.apache.org/jira/browse/HIVE-19308?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Wohlstadter updated HIVE-19308: Attachment: HIVE-19308.4.patch > Provide an Arrow stream reader for external LLAP clients > - > > Key: HIVE-19308 > URL: https://issues.apache.org/jira/browse/HIVE-19308 > Project: Hive > Issue Type: Task > Components: llap >Reporter: Eric Wohlstadter >Assignee: Eric Wohlstadter >Priority: Major > Attachments: HIVE-19308.1.patch, HIVE-19308.2.patch, > HIVE-19308.3.patch, HIVE-19308.4.patch > > > This is a sub-class of LlapBaseRecordReader that wraps the socket inputStream > and produces Arrow batches for an external client. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19308) Provide an Arrow stream reader for external LLAP clients
[ https://issues.apache.org/jira/browse/HIVE-19308?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Wohlstadter updated HIVE-19308: Status: Patch Available (was: Open) > Provide an Arrow stream reader for external LLAP clients > - > > Key: HIVE-19308 > URL: https://issues.apache.org/jira/browse/HIVE-19308 > Project: Hive > Issue Type: Task > Components: llap >Reporter: Eric Wohlstadter >Assignee: Eric Wohlstadter >Priority: Major > Attachments: HIVE-19308.1.patch, HIVE-19308.2.patch, > HIVE-19308.3.patch, HIVE-19308.4.patch > > > This is a sub-class of LlapBaseRecordReader that wraps the socket inputStream > and produces Arrow batches for an external client. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-19627) Add support for LlapArrowBatchRecordReader to be used through a Hadoop InputFormat
[ https://issues.apache.org/jira/browse/HIVE-19627?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Wohlstadter reassigned HIVE-19627: --- > Add support for LlapArrowBatchRecordReader to be used through a Hadoop > InputFormat > -- > > Key: HIVE-19627 > URL: https://issues.apache.org/jira/browse/HIVE-19627 > Project: Hive > Issue Type: Sub-task >Reporter: Eric Wohlstadter >Assignee: Eric Wohlstadter >Priority: Major > > LlapArrowBatchRecordReader would need to support configuration through > JobConf, rather than, or in addition to, the external client's native > configuration. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19305) Arrow format for LlapOutputFormatService (umbrella)
[ https://issues.apache.org/jira/browse/HIVE-19305?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Wohlstadter updated HIVE-19305: Status: Patch Available (was: In Progress) > Arrow format for LlapOutputFormatService (umbrella) > --- > > Key: HIVE-19305 > URL: https://issues.apache.org/jira/browse/HIVE-19305 > Project: Hive > Issue Type: Improvement > Components: llap >Reporter: Eric Wohlstadter >Assignee: Eric Wohlstadter >Priority: Major > Attachments: HIVE-19305.1-branch-3.patch > > > Allows external clients to consume output from LLAP daemons in Arrow stream > format. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19305) Arrow format for LlapOutputFormatService (umbrella)
[ https://issues.apache.org/jira/browse/HIVE-19305?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Wohlstadter updated HIVE-19305: Attachment: HIVE-19305.1-branch-3.patch > Arrow format for LlapOutputFormatService (umbrella) > --- > > Key: HIVE-19305 > URL: https://issues.apache.org/jira/browse/HIVE-19305 > Project: Hive > Issue Type: Improvement > Components: llap >Reporter: Eric Wohlstadter >Assignee: Eric Wohlstadter >Priority: Major > Attachments: HIVE-19305.1-branch-3.patch > > > Allows external clients to consume output from LLAP daemons in Arrow stream > format. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19305) Arrow format for LlapOutputFormatService (umbrella)
[ https://issues.apache.org/jira/browse/HIVE-19305?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Wohlstadter updated HIVE-19305: Status: Open (was: Patch Available) > Arrow format for LlapOutputFormatService (umbrella) > --- > > Key: HIVE-19305 > URL: https://issues.apache.org/jira/browse/HIVE-19305 > Project: Hive > Issue Type: Improvement > Components: llap >Reporter: Eric Wohlstadter >Assignee: Eric Wohlstadter >Priority: Major > Attachments: HIVE-19305.1-branch-3.patch, HIVE-19305.2-branch-3.patch > > > Allows external clients to consume output from LLAP daemons in Arrow stream > format. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19305) Arrow format for LlapOutputFormatService (umbrella)
[ https://issues.apache.org/jira/browse/HIVE-19305?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Wohlstadter updated HIVE-19305: Attachment: HIVE-19305.2-branch-3.patch > Arrow format for LlapOutputFormatService (umbrella) > --- > > Key: HIVE-19305 > URL: https://issues.apache.org/jira/browse/HIVE-19305 > Project: Hive > Issue Type: Improvement > Components: llap >Reporter: Eric Wohlstadter >Assignee: Eric Wohlstadter >Priority: Major > Attachments: HIVE-19305.1-branch-3.patch, HIVE-19305.2-branch-3.patch > > > Allows external clients to consume output from LLAP daemons in Arrow stream > format. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19305) Arrow format for LlapOutputFormatService (umbrella)
[ https://issues.apache.org/jira/browse/HIVE-19305?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Wohlstadter updated HIVE-19305: Status: Patch Available (was: Open) > Arrow format for LlapOutputFormatService (umbrella) > --- > > Key: HIVE-19305 > URL: https://issues.apache.org/jira/browse/HIVE-19305 > Project: Hive > Issue Type: Improvement > Components: llap >Reporter: Eric Wohlstadter >Assignee: Eric Wohlstadter >Priority: Major > Attachments: HIVE-19305.1-branch-3.patch, HIVE-19305.2-branch-3.patch > > > Allows external clients to consume output from LLAP daemons in Arrow stream > format. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19682) Provide option for GenericUDTFGetSplits to return only schema metadata
[
https://issues.apache.org/jira/browse/HIVE-19682?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16489970#comment-16489970
]
Eric Wohlstadter commented on HIVE-19682:
-
[~prasanth_j]
Can you check?
> Provide option for GenericUDTFGetSplits to return only schema metadata
> --
>
> Key: HIVE-19682
> URL: https://issues.apache.org/jira/browse/HIVE-19682
> Project: Hive
> Issue Type: Improvement
>Reporter: Eric Wohlstadter
>Assignee: Eric Wohlstadter
>Priority: Major
> Attachments: HIVE-19682.1.patch
>
>
> For some uses cases it is necessary to know the output schema for a HiveQL
> before executing the query. But there is no existing client API that provides
> this information.
> Hive JDBC doesn't provide the schema for parametric types in
> {{ResultSetMetaData}}.
> GenericUDTFGetSplits bundles the proper schema metadata with the fragments
> for input splits. An option can be added to return only the schema metadata
> from compilation, and the generation of input splits can be skipped.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work started] (HIVE-19682) Provide option for GenericUDTFGetSplits to return only schema metadata
[
https://issues.apache.org/jira/browse/HIVE-19682?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on HIVE-19682 started by Eric Wohlstadter.
---
> Provide option for GenericUDTFGetSplits to return only schema metadata
> --
>
> Key: HIVE-19682
> URL: https://issues.apache.org/jira/browse/HIVE-19682
> Project: Hive
> Issue Type: Improvement
>Reporter: Eric Wohlstadter
>Assignee: Eric Wohlstadter
>Priority: Major
>
> For some uses cases it is necessary to know the output schema for a HiveQL
> before executing the query. But there is no existing client API that provides
> this information.
> Hive JDBC doesn't provide the schema for parametric types in
> {{ResultSetMetaData}}.
> GenericUDTFGetSplits bundles the proper schema metadata with the fragments
> for input splits. An option can be added to return only the schema metadata
> from compilation, and the generation of input splits can be skipped.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19682) Provide option for GenericUDTFGetSplits to return only schema metadata
[
https://issues.apache.org/jira/browse/HIVE-19682?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Wohlstadter updated HIVE-19682:
Attachment: HIVE-19682.1.patch
> Provide option for GenericUDTFGetSplits to return only schema metadata
> --
>
> Key: HIVE-19682
> URL: https://issues.apache.org/jira/browse/HIVE-19682
> Project: Hive
> Issue Type: Improvement
>Reporter: Eric Wohlstadter
>Assignee: Eric Wohlstadter
>Priority: Major
> Attachments: HIVE-19682.1.patch
>
>
> For some uses cases it is necessary to know the output schema for a HiveQL
> before executing the query. But there is no existing client API that provides
> this information.
> Hive JDBC doesn't provide the schema for parametric types in
> {{ResultSetMetaData}}.
> GenericUDTFGetSplits bundles the proper schema metadata with the fragments
> for input splits. An option can be added to return only the schema metadata
> from compilation, and the generation of input splits can be skipped.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19682) Provide option for GenericUDTFGetSplits to return only schema metadata
[
https://issues.apache.org/jira/browse/HIVE-19682?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Wohlstadter updated HIVE-19682:
Status: Patch Available (was: In Progress)
> Provide option for GenericUDTFGetSplits to return only schema metadata
> --
>
> Key: HIVE-19682
> URL: https://issues.apache.org/jira/browse/HIVE-19682
> Project: Hive
> Issue Type: Improvement
>Reporter: Eric Wohlstadter
>Assignee: Eric Wohlstadter
>Priority: Major
> Attachments: HIVE-19682.1.patch
>
>
> For some uses cases it is necessary to know the output schema for a HiveQL
> before executing the query. But there is no existing client API that provides
> this information.
> Hive JDBC doesn't provide the schema for parametric types in
> {{ResultSetMetaData}}.
> GenericUDTFGetSplits bundles the proper schema metadata with the fragments
> for input splits. An option can be added to return only the schema metadata
> from compilation, and the generation of input splits can be skipped.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19308) Provide an Arrow stream reader for external LLAP clients
[ https://issues.apache.org/jira/browse/HIVE-19308?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Wohlstadter updated HIVE-19308: Status: Patch Available (was: Open) > Provide an Arrow stream reader for external LLAP clients > - > > Key: HIVE-19308 > URL: https://issues.apache.org/jira/browse/HIVE-19308 > Project: Hive > Issue Type: Task > Components: llap >Reporter: Eric Wohlstadter >Assignee: Eric Wohlstadter >Priority: Major > Attachments: HIVE-19308.1.patch, HIVE-19308.2.patch, > HIVE-19308.3.patch > > > This is a sub-class of LlapBaseRecordReader that wraps the socket inputStream > and produces Arrow batches for an external client. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19308) Provide an Arrow stream reader for external LLAP clients
[ https://issues.apache.org/jira/browse/HIVE-19308?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Wohlstadter updated HIVE-19308: Status: Open (was: Patch Available) > Provide an Arrow stream reader for external LLAP clients > - > > Key: HIVE-19308 > URL: https://issues.apache.org/jira/browse/HIVE-19308 > Project: Hive > Issue Type: Task > Components: llap >Reporter: Eric Wohlstadter >Assignee: Eric Wohlstadter >Priority: Major > Attachments: HIVE-19308.1.patch, HIVE-19308.2.patch, > HIVE-19308.3.patch > > > This is a sub-class of LlapBaseRecordReader that wraps the socket inputStream > and produces Arrow batches for an external client. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19308) Provide an Arrow stream reader for external LLAP clients
[ https://issues.apache.org/jira/browse/HIVE-19308?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Wohlstadter updated HIVE-19308: Attachment: HIVE-19308.3.patch > Provide an Arrow stream reader for external LLAP clients > - > > Key: HIVE-19308 > URL: https://issues.apache.org/jira/browse/HIVE-19308 > Project: Hive > Issue Type: Task > Components: llap >Reporter: Eric Wohlstadter >Assignee: Eric Wohlstadter >Priority: Major > Attachments: HIVE-19308.1.patch, HIVE-19308.2.patch, > HIVE-19308.3.patch > > > This is a sub-class of LlapBaseRecordReader that wraps the socket inputStream > and produces Arrow batches for an external client. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19308) Provide an Arrow stream reader for external LLAP clients
[ https://issues.apache.org/jira/browse/HIVE-19308?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16477982#comment-16477982 ] Eric Wohlstadter commented on HIVE-19308: - https://reviews.apache.org/r/67159/ > Provide an Arrow stream reader for external LLAP clients > - > > Key: HIVE-19308 > URL: https://issues.apache.org/jira/browse/HIVE-19308 > Project: Hive > Issue Type: Task > Components: llap >Reporter: Eric Wohlstadter >Assignee: Eric Wohlstadter >Priority: Major > Attachments: HIVE-19308.1.patch, HIVE-19308.2.patch > > > This is a sub-class of LlapBaseRecordReader that wraps the socket inputStream > and produces Arrow batches for an external client. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19308) Provide an Arrow stream reader for external LLAP clients
[ https://issues.apache.org/jira/browse/HIVE-19308?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Wohlstadter updated HIVE-19308: Attachment: HIVE-19308.5.patch > Provide an Arrow stream reader for external LLAP clients > - > > Key: HIVE-19308 > URL: https://issues.apache.org/jira/browse/HIVE-19308 > Project: Hive > Issue Type: Task > Components: llap >Reporter: Eric Wohlstadter >Assignee: Eric Wohlstadter >Priority: Major > Attachments: HIVE-19308.1.patch, HIVE-19308.2.patch, > HIVE-19308.3.patch, HIVE-19308.4.patch, HIVE-19308.5.patch > > > This is a sub-class of LlapBaseRecordReader that wraps the socket inputStream > and produces Arrow batches for an external client. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19308) Provide an Arrow stream reader for external LLAP clients
[ https://issues.apache.org/jira/browse/HIVE-19308?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Wohlstadter updated HIVE-19308: Status: Patch Available (was: Open) > Provide an Arrow stream reader for external LLAP clients > - > > Key: HIVE-19308 > URL: https://issues.apache.org/jira/browse/HIVE-19308 > Project: Hive > Issue Type: Task > Components: llap >Reporter: Eric Wohlstadter >Assignee: Eric Wohlstadter >Priority: Major > Attachments: HIVE-19308.1.patch, HIVE-19308.2.patch, > HIVE-19308.3.patch, HIVE-19308.4.patch, HIVE-19308.5.patch > > > This is a sub-class of LlapBaseRecordReader that wraps the socket inputStream > and produces Arrow batches for an external client. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19308) Provide an Arrow stream reader for external LLAP clients
[ https://issues.apache.org/jira/browse/HIVE-19308?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Wohlstadter updated HIVE-19308: Status: Open (was: Patch Available) > Provide an Arrow stream reader for external LLAP clients > - > > Key: HIVE-19308 > URL: https://issues.apache.org/jira/browse/HIVE-19308 > Project: Hive > Issue Type: Task > Components: llap >Reporter: Eric Wohlstadter >Assignee: Eric Wohlstadter >Priority: Major > Attachments: HIVE-19308.1.patch, HIVE-19308.2.patch, > HIVE-19308.3.patch, HIVE-19308.4.patch, HIVE-19308.5.patch > > > This is a sub-class of LlapBaseRecordReader that wraps the socket inputStream > and produces Arrow batches for an external client. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (HIVE-19359) itest for Arrow LLAP OutputFormat
[ https://issues.apache.org/jira/browse/HIVE-19359?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Wohlstadter resolved HIVE-19359. - Resolution: Duplicate > itest for Arrow LLAP OutputFormat > - > > Key: HIVE-19359 > URL: https://issues.apache.org/jira/browse/HIVE-19359 > Project: Hive > Issue Type: Task > Components: Tests >Reporter: Eric Wohlstadter >Assignee: Eric Wohlstadter >Priority: Major > Attachments: HIVE-19359.1.patch > > > Modified version of TestJdbcWithMiniLlap > Exercises HIVE-19306, HIVE-19307, and HIVE-19308. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-19682) Provide option for GenericUDTFGetSplits to return only schema metadata
[
https://issues.apache.org/jira/browse/HIVE-19682?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Wohlstadter reassigned HIVE-19682:
---
> Provide option for GenericUDTFGetSplits to return only schema metadata
> --
>
> Key: HIVE-19682
> URL: https://issues.apache.org/jira/browse/HIVE-19682
> Project: Hive
> Issue Type: Improvement
>Reporter: Eric Wohlstadter
>Assignee: Eric Wohlstadter
>Priority: Major
>
> For some uses cases it is necessary to know the output schema for a HiveQL
> before executing the query. But there is no existing client API that provides
> this information.
> Hive JDBC doesn't provide the schema for parametric types in
> {{ResultSetMetaData}}.
> GenericUDTFGetSplits bundles the proper schema metadata with the fragments
> for input splits. An option can be added to return only the schema metadata
> from compilation, and the generation of input splits can be skipped.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19308) Provide an Arrow stream reader for external LLAP clients
[ https://issues.apache.org/jira/browse/HIVE-19308?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Wohlstadter updated HIVE-19308: Status: Patch Available (was: Open) > Provide an Arrow stream reader for external LLAP clients > - > > Key: HIVE-19308 > URL: https://issues.apache.org/jira/browse/HIVE-19308 > Project: Hive > Issue Type: Task > Components: llap >Reporter: Eric Wohlstadter >Assignee: Eric Wohlstadter >Priority: Major > Attachments: HIVE-19308.1.patch, HIVE-19308.2.patch, > HIVE-19308.3.patch, HIVE-19308.4.patch, HIVE-19308.5.patch, > HIVE-19308.6.patch, HIVE-19308.7.patch > > > This is a sub-class of LlapBaseRecordReader that wraps the socket inputStream > and produces Arrow batches for an external client. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19308) Provide an Arrow stream reader for external LLAP clients
[ https://issues.apache.org/jira/browse/HIVE-19308?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Wohlstadter updated HIVE-19308: Status: Open (was: Patch Available) > Provide an Arrow stream reader for external LLAP clients > - > > Key: HIVE-19308 > URL: https://issues.apache.org/jira/browse/HIVE-19308 > Project: Hive > Issue Type: Task > Components: llap >Reporter: Eric Wohlstadter >Assignee: Eric Wohlstadter >Priority: Major > Attachments: HIVE-19308.1.patch, HIVE-19308.2.patch, > HIVE-19308.3.patch, HIVE-19308.4.patch, HIVE-19308.5.patch, HIVE-19308.6.patch > > > This is a sub-class of LlapBaseRecordReader that wraps the socket inputStream > and produces Arrow batches for an external client. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19308) Provide an Arrow stream reader for external LLAP clients
[ https://issues.apache.org/jira/browse/HIVE-19308?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Wohlstadter updated HIVE-19308: Attachment: HIVE-19308.7.patch > Provide an Arrow stream reader for external LLAP clients > - > > Key: HIVE-19308 > URL: https://issues.apache.org/jira/browse/HIVE-19308 > Project: Hive > Issue Type: Task > Components: llap >Reporter: Eric Wohlstadter >Assignee: Eric Wohlstadter >Priority: Major > Attachments: HIVE-19308.1.patch, HIVE-19308.2.patch, > HIVE-19308.3.patch, HIVE-19308.4.patch, HIVE-19308.5.patch, > HIVE-19308.6.patch, HIVE-19308.7.patch > > > This is a sub-class of LlapBaseRecordReader that wraps the socket inputStream > and produces Arrow batches for an external client. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19853) Arrow serializer needs to create a TimeStampMicroTZVector instead of TimeStampMicroVector
[ https://issues.apache.org/jira/browse/HIVE-19853?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16514855#comment-16514855 ] Eric Wohlstadter commented on HIVE-19853: - The test failure was previously reported here, I assume it is unrelated: https://issues.apache.org/jira/browse/HIVE-19922 [https://builds.apache.org/job/PreCommit-HIVE-Build/11826/testReport/] > Arrow serializer needs to create a TimeStampMicroTZVector instead of > TimeStampMicroVector > - > > Key: HIVE-19853 > URL: https://issues.apache.org/jira/browse/HIVE-19853 > Project: Hive > Issue Type: Bug >Reporter: Teddy Choi >Assignee: Teddy Choi >Priority: Major > Labels: pull-request-available > Attachments: HIVE-19853.1.patch, HIVE-19853.2.patch > > > HIVE-19723 changed nanosecond to microsecond in Arrow serialization. However, > it needs to be microsecond with time zone. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19853) Arrow serializer needs to create a TimeStampMicroTZVector instead of TimeStampMicroVector
[ https://issues.apache.org/jira/browse/HIVE-19853?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16514396#comment-16514396 ] Eric Wohlstadter commented on HIVE-19853: - lgtm [~mmccline], can you merge to master and branch-3? Thanks! > Arrow serializer needs to create a TimeStampMicroTZVector instead of > TimeStampMicroVector > - > > Key: HIVE-19853 > URL: https://issues.apache.org/jira/browse/HIVE-19853 > Project: Hive > Issue Type: Bug >Reporter: Teddy Choi >Assignee: Teddy Choi >Priority: Major > Labels: pull-request-available > Attachments: HIVE-19853.1.patch, HIVE-19853.2.patch > > > HIVE-19723 changed nanosecond to microsecond in Arrow serialization. However, > it needs to be microsecond with time zone. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19723) Arrow serde: "Unsupported data type: Timestamp(NANOSECOND, null)"
[
https://issues.apache.org/jira/browse/HIVE-19723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16506809#comment-16506809
]
Eric Wohlstadter commented on HIVE-19723:
-
[~teddy.choi]
Serializer needs to create a {{TimeStampMicroTZVector}} instead of
{{TimeStampMicroVector}}.
See:
{{org.apache.spark.sql.vectorized.ArrowColumnVector.ArrowColumnVector(ValueVector
vector)}}
> Arrow serde: "Unsupported data type: Timestamp(NANOSECOND, null)"
> -
>
> Key: HIVE-19723
> URL: https://issues.apache.org/jira/browse/HIVE-19723
> Project: Hive
> Issue Type: Bug
>Reporter: Teddy Choi
>Assignee: Teddy Choi
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.1.0
>
> Attachments: HIVE-19723.1.patch, HIVE-19723.3.patch,
> HIVE-19723.4.patch, HIVE-19732.2.patch
>
>
> Spark's Arrow support only provides Timestamp at MICROSECOND granularity.
> Spark 2.3.0 won't accept NANOSECOND. Switch it back to MICROSECOND.
> The unit test org.apache.hive.jdbc.TestJdbcWithMiniLlapArrow will just need
> to change the assertion to test microsecond. And we'll need to add this to
> documentation on supported datatypes.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (HIVE-19723) Arrow serde: "Unsupported data type: Timestamp(NANOSECOND, null)"
[
https://issues.apache.org/jira/browse/HIVE-19723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16506809#comment-16506809
]
Eric Wohlstadter edited comment on HIVE-19723 at 6/9/18 4:24 AM:
-
[~teddy.choi]
Serializer needs to create a {{TimeStampMicroTZVector}} instead of
{{TimeStampMicroVector}}.
See:
{{org.apache.spark.sql.vectorized.ArrowColumnVector.ArrowColumnVector(ValueVector
vector)}}
Can you create a new JIRA for that?
was (Author: ewohlstadter):
[~teddy.choi]
Serializer needs to create a {{TimeStampMicroTZVector}} instead of
{{TimeStampMicroVector}}.
See:
{{org.apache.spark.sql.vectorized.ArrowColumnVector.ArrowColumnVector(ValueVector
vector)}}
> Arrow serde: "Unsupported data type: Timestamp(NANOSECOND, null)"
> -
>
> Key: HIVE-19723
> URL: https://issues.apache.org/jira/browse/HIVE-19723
> Project: Hive
> Issue Type: Bug
>Reporter: Teddy Choi
>Assignee: Teddy Choi
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.1.0
>
> Attachments: HIVE-19723.1.patch, HIVE-19723.3.patch,
> HIVE-19723.4.patch, HIVE-19732.2.patch
>
>
> Spark's Arrow support only provides Timestamp at MICROSECOND granularity.
> Spark 2.3.0 won't accept NANOSECOND. Switch it back to MICROSECOND.
> The unit test org.apache.hive.jdbc.TestJdbcWithMiniLlapArrow will just need
> to change the assertion to test microsecond. And we'll need to add this to
> documentation on supported datatypes.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (HIVE-19839) sssssssssssss
[ https://issues.apache.org/jira/browse/HIVE-19839?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Wohlstadter resolved HIVE-19839. - Resolution: Invalid > s > - > > Key: HIVE-19839 > URL: https://issues.apache.org/jira/browse/HIVE-19839 > Project: Hive > Issue Type: Bug > Components: Hive, UDF >Affects Versions: 2.3.1 >Reporter: sadashiv >Priority: Major > Fix For: 0.10.1 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19808) GenericUDTFGetSplits should support ACID reads in the temp. table read path
[
https://issues.apache.org/jira/browse/HIVE-19808?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16505656#comment-16505656
]
Eric Wohlstadter commented on HIVE-19808:
-
[~jdere]
Got a green run. Can you help merge to master and branch-3?
> GenericUDTFGetSplits should support ACID reads in the temp. table read path
> ---
>
> Key: HIVE-19808
> URL: https://issues.apache.org/jira/browse/HIVE-19808
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Reporter: Eric Wohlstadter
>Assignee: Eric Wohlstadter
>Priority: Major
> Attachments: HIVE-19808.1.patch, HIVE-19808.2.patch
>
>
> 1. Map-only reads work on ACID tables.
> 2. Temp. table reads (for multi-vertex queries) work on non-ACID tables.
> 3. But temp. table reads don't work on ACID tables.
> {code}
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Failed to create
> temp table: java.lang.IllegalStateException: calling recordValidTxn() more
> than once in the same txnid:420
> at
> org.apache.hadoop.hive.ql.udf.generic.GenericUDTFGetSplits.createPlanFragment(GenericUDTFGetSplits.java:303)
> at
> org.apache.hadoop.hive.ql.udf.generic.GenericUDTFGetSplits.process(GenericUDTFGetSplits.java:202)
> at
> org.apache.hadoop.hive.ql.exec.UDTFOperator.process(UDTFOperator.java:116)
> at
> org.apache.hadoop.hive.ql.exec.Operator.baseForward(Operator.java:985)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:931)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:918)
> at
> org.apache.hadoop.hive.ql.exec.SelectOperator.process(SelectOperator.java:95)
> at
> org.apache.hadoop.hive.ql.exec.Operator.baseForward(Operator.java:985)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:931)
> at
> org.apache.hadoop.hive.ql.exec.TableScanOperator.process(TableScanOperator.java:125)
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator.pushRow(FetchOperator.java:492)
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator.pushRow(FetchOperator.java:484)
> at org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:145)
> ... 16 more
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19305) Arrow format for LlapOutputFormatService (umbrella)
[ https://issues.apache.org/jira/browse/HIVE-19305?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Wohlstadter updated HIVE-19305: Status: Patch Available (was: Open) > Arrow format for LlapOutputFormatService (umbrella) > --- > > Key: HIVE-19305 > URL: https://issues.apache.org/jira/browse/HIVE-19305 > Project: Hive > Issue Type: Improvement > Components: llap >Reporter: Eric Wohlstadter >Assignee: Eric Wohlstadter >Priority: Major > Attachments: HIVE-19305.1-branch-3.patch, > HIVE-19305.2-branch-3.patch, HIVE-19305.3-branch-3.patch, > HIVE-19305.4-branch-3.patch > > > Allows external clients to consume output from LLAP daemons in Arrow stream > format. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19305) Arrow format for LlapOutputFormatService (umbrella)
[ https://issues.apache.org/jira/browse/HIVE-19305?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Wohlstadter updated HIVE-19305: Attachment: HIVE-19305.4-branch-3.patch > Arrow format for LlapOutputFormatService (umbrella) > --- > > Key: HIVE-19305 > URL: https://issues.apache.org/jira/browse/HIVE-19305 > Project: Hive > Issue Type: Improvement > Components: llap >Reporter: Eric Wohlstadter >Assignee: Eric Wohlstadter >Priority: Major > Attachments: HIVE-19305.1-branch-3.patch, > HIVE-19305.2-branch-3.patch, HIVE-19305.3-branch-3.patch, > HIVE-19305.4-branch-3.patch > > > Allows external clients to consume output from LLAP daemons in Arrow stream > format. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19305) Arrow format for LlapOutputFormatService (umbrella)
[ https://issues.apache.org/jira/browse/HIVE-19305?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Wohlstadter updated HIVE-19305: Status: Open (was: Patch Available) > Arrow format for LlapOutputFormatService (umbrella) > --- > > Key: HIVE-19305 > URL: https://issues.apache.org/jira/browse/HIVE-19305 > Project: Hive > Issue Type: Improvement > Components: llap >Reporter: Eric Wohlstadter >Assignee: Eric Wohlstadter >Priority: Major > Attachments: HIVE-19305.1-branch-3.patch, > HIVE-19305.2-branch-3.patch, HIVE-19305.3-branch-3.patch, > HIVE-19305.4-branch-3.patch > > > Allows external clients to consume output from LLAP daemons in Arrow stream > format. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19682) Provide option for GenericUDTFGetSplits to return only schema metadata
[
https://issues.apache.org/jira/browse/HIVE-19682?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Wohlstadter updated HIVE-19682:
Status: Patch Available (was: Open)
> Provide option for GenericUDTFGetSplits to return only schema metadata
> --
>
> Key: HIVE-19682
> URL: https://issues.apache.org/jira/browse/HIVE-19682
> Project: Hive
> Issue Type: Improvement
>Reporter: Eric Wohlstadter
>Assignee: Eric Wohlstadter
>Priority: Major
> Attachments: HIVE-19682.1.patch, HIVE-19682.2.patch,
> HIVE-19682.3.patch
>
>
> For some uses cases it is necessary to know the output schema for a HiveQL
> before executing the query. But there is no existing client API that provides
> this information.
> Hive JDBC doesn't provide the schema for parametric types in
> {{ResultSetMetaData}}.
> GenericUDTFGetSplits bundles the proper schema metadata with the fragments
> for input splits. An option can be added to return only the schema metadata
> from compilation, and the generation of input splits can be skipped.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19682) Provide option for GenericUDTFGetSplits to return only schema metadata
[
https://issues.apache.org/jira/browse/HIVE-19682?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Wohlstadter updated HIVE-19682:
Status: Open (was: Patch Available)
> Provide option for GenericUDTFGetSplits to return only schema metadata
> --
>
> Key: HIVE-19682
> URL: https://issues.apache.org/jira/browse/HIVE-19682
> Project: Hive
> Issue Type: Improvement
>Reporter: Eric Wohlstadter
>Assignee: Eric Wohlstadter
>Priority: Major
> Attachments: HIVE-19682.1.patch, HIVE-19682.2.patch,
> HIVE-19682.3.patch
>
>
> For some uses cases it is necessary to know the output schema for a HiveQL
> before executing the query. But there is no existing client API that provides
> this information.
> Hive JDBC doesn't provide the schema for parametric types in
> {{ResultSetMetaData}}.
> GenericUDTFGetSplits bundles the proper schema metadata with the fragments
> for input splits. An option can be added to return only the schema metadata
> from compilation, and the generation of input splits can be skipped.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19682) Provide option for GenericUDTFGetSplits to return only schema metadata
[
https://issues.apache.org/jira/browse/HIVE-19682?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Wohlstadter updated HIVE-19682:
Attachment: HIVE-19682.3.patch
> Provide option for GenericUDTFGetSplits to return only schema metadata
> --
>
> Key: HIVE-19682
> URL: https://issues.apache.org/jira/browse/HIVE-19682
> Project: Hive
> Issue Type: Improvement
>Reporter: Eric Wohlstadter
>Assignee: Eric Wohlstadter
>Priority: Major
> Attachments: HIVE-19682.1.patch, HIVE-19682.2.patch,
> HIVE-19682.3.patch
>
>
> For some uses cases it is necessary to know the output schema for a HiveQL
> before executing the query. But there is no existing client API that provides
> this information.
> Hive JDBC doesn't provide the schema for parametric types in
> {{ResultSetMetaData}}.
> GenericUDTFGetSplits bundles the proper schema metadata with the fragments
> for input splits. An option can be added to return only the schema metadata
> from compilation, and the generation of input splits can be skipped.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19305) Arrow format for LlapOutputFormatService (umbrella)
[ https://issues.apache.org/jira/browse/HIVE-19305?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Wohlstadter updated HIVE-19305: Status: Patch Available (was: Open) > Arrow format for LlapOutputFormatService (umbrella) > --- > > Key: HIVE-19305 > URL: https://issues.apache.org/jira/browse/HIVE-19305 > Project: Hive > Issue Type: Improvement > Components: llap >Reporter: Eric Wohlstadter >Assignee: Eric Wohlstadter >Priority: Major > Attachments: HIVE-19305.1-branch-3.patch, > HIVE-19305.2-branch-3.patch, HIVE-19305.3-branch-3.patch > > > Allows external clients to consume output from LLAP daemons in Arrow stream > format. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19682) Provide option for GenericUDTFGetSplits to return only schema metadata
[
https://issues.apache.org/jira/browse/HIVE-19682?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Wohlstadter updated HIVE-19682:
Attachment: HIVE-19682.2.patch
> Provide option for GenericUDTFGetSplits to return only schema metadata
> --
>
> Key: HIVE-19682
> URL: https://issues.apache.org/jira/browse/HIVE-19682
> Project: Hive
> Issue Type: Improvement
>Reporter: Eric Wohlstadter
>Assignee: Eric Wohlstadter
>Priority: Major
> Attachments: HIVE-19682.1.patch, HIVE-19682.2.patch
>
>
> For some uses cases it is necessary to know the output schema for a HiveQL
> before executing the query. But there is no existing client API that provides
> this information.
> Hive JDBC doesn't provide the schema for parametric types in
> {{ResultSetMetaData}}.
> GenericUDTFGetSplits bundles the proper schema metadata with the fragments
> for input splits. An option can be added to return only the schema metadata
> from compilation, and the generation of input splits can be skipped.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19682) Provide option for GenericUDTFGetSplits to return only schema metadata
[
https://issues.apache.org/jira/browse/HIVE-19682?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Wohlstadter updated HIVE-19682:
Status: Patch Available (was: Open)
> Provide option for GenericUDTFGetSplits to return only schema metadata
> --
>
> Key: HIVE-19682
> URL: https://issues.apache.org/jira/browse/HIVE-19682
> Project: Hive
> Issue Type: Improvement
>Reporter: Eric Wohlstadter
>Assignee: Eric Wohlstadter
>Priority: Major
> Attachments: HIVE-19682.1.patch, HIVE-19682.2.patch
>
>
> For some uses cases it is necessary to know the output schema for a HiveQL
> before executing the query. But there is no existing client API that provides
> this information.
> Hive JDBC doesn't provide the schema for parametric types in
> {{ResultSetMetaData}}.
> GenericUDTFGetSplits bundles the proper schema metadata with the fragments
> for input splits. An option can be added to return only the schema metadata
> from compilation, and the generation of input splits can be skipped.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19305) Arrow format for LlapOutputFormatService (umbrella)
[ https://issues.apache.org/jira/browse/HIVE-19305?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16494347#comment-16494347 ] Eric Wohlstadter commented on HIVE-19305: - [~ashutoshc] Yes, except HIVE-19713 was also added to pass the storage-handler version problem. > Arrow format for LlapOutputFormatService (umbrella) > --- > > Key: HIVE-19305 > URL: https://issues.apache.org/jira/browse/HIVE-19305 > Project: Hive > Issue Type: Improvement > Components: llap >Reporter: Eric Wohlstadter >Assignee: Eric Wohlstadter >Priority: Major > Attachments: HIVE-19305.1-branch-3.patch, > HIVE-19305.2-branch-3.patch, HIVE-19305.3-branch-3.patch, > HIVE-19305.4-branch-3.patch > > > Allows external clients to consume output from LLAP daemons in Arrow stream > format. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19682) Provide option for GenericUDTFGetSplits to return only schema metadata
[
https://issues.apache.org/jira/browse/HIVE-19682?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Wohlstadter updated HIVE-19682:
Status: Open (was: Patch Available)
> Provide option for GenericUDTFGetSplits to return only schema metadata
> --
>
> Key: HIVE-19682
> URL: https://issues.apache.org/jira/browse/HIVE-19682
> Project: Hive
> Issue Type: Improvement
>Reporter: Eric Wohlstadter
>Assignee: Eric Wohlstadter
>Priority: Major
> Attachments: HIVE-19682.1.patch
>
>
> For some uses cases it is necessary to know the output schema for a HiveQL
> before executing the query. But there is no existing client API that provides
> this information.
> Hive JDBC doesn't provide the schema for parametric types in
> {{ResultSetMetaData}}.
> GenericUDTFGetSplits bundles the proper schema metadata with the fragments
> for input splits. An option can be added to return only the schema metadata
> from compilation, and the generation of input splits can be skipped.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (HIVE-19723) Arrow serde: "Unsupported data type: Timestamp(NANOSECOND, null)"
[
https://issues.apache.org/jira/browse/HIVE-19723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16498593#comment-16498593
]
Eric Wohlstadter edited comment on HIVE-19723 at 6/1/18 9:20 PM:
-
[~teddy.choi]
Hive's Arrow serializer appears to truncate down to MILLISECONDS, but the Jira
description calls for MICROSECONDS.
This is motivated by {{org.apache.spark.sql.execution.arrow.ArrowUtils.scala}}
{code:java}
case ts: ArrowType.Timestamp if ts.getUnit == TimeUnit.MICROSECOND =>
TimestampType{code}
My understanding is that since the primary use-case for {{ArrowUtils}} is
Python integration, some of the conversions are currently somewhat particular
for Python. Perhaps Python/Pandas only supports MICROSECOND timestamps.
FYI: [~hyukjin.kwon] [~bryanc]
was (Author: ewohlstadter):
[~teddy.choi]
The Arrow serializer appears to truncate down to MILLISECONDS, but the Jira
description calls for MICROSECONDS.
This is motivated by {{org.apache.spark.sql.execution.arrow.ArrowUtils.scala}}
{code:java}
case ts: ArrowType.Timestamp if ts.getUnit == TimeUnit.MICROSECOND =>
TimestampType{code}
My understanding is that since the primary use-case for {{ArrowUtils}} is
Python integration, some of the conversions are currently somewhat particular
for Python. Perhaps Python/Pandas only supports MICROSECOND timestamps.
FYI: [~hyukjin.kwon] [~bryanc]
> Arrow serde: "Unsupported data type: Timestamp(NANOSECOND, null)"
> -
>
> Key: HIVE-19723
> URL: https://issues.apache.org/jira/browse/HIVE-19723
> Project: Hive
> Issue Type: Bug
>Reporter: Teddy Choi
>Assignee: Teddy Choi
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.1.0, 4.0.0
>
> Attachments: HIVE-19723.1.patch, HIVE-19732.2.patch
>
>
> Spark's Arrow support only provides Timestamp at MICROSECOND granularity.
> Spark 2.3.0 won't accept NANOSECOND. Switch it back to MICROSECOND.
> The unit test org.apache.hive.jdbc.TestJdbcWithMiniLlapArrow will just need
> to change the assertion to test microsecond. And we'll need to add this to
> documentation on supported datatypes.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19723) Arrow serde: "Unsupported data type: Timestamp(NANOSECOND, null)"
[
https://issues.apache.org/jira/browse/HIVE-19723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16498593#comment-16498593
]
Eric Wohlstadter commented on HIVE-19723:
-
[~teddy.choi]
The Arrow serializer appears to truncate down to MILLISECONDS, but the Jira
description calls for MICROSECONDS.
This is motivated by {{org.apache.spark.sql.execution.arrow.ArrowUtils.scala}}
{code:java}
case ts: ArrowType.Timestamp if ts.getUnit == TimeUnit.MICROSECOND =>
TimestampType{code}
My understanding is that since the primary use-case for {{ArrowUtils}} is
Python integration, some of the conversions are currently somewhat particular
for Python. Perhaps Python/Pandas only supports MICROSECOND timestamps.
FYI: [~hyukjin.kwon] [~bryanc]
> Arrow serde: "Unsupported data type: Timestamp(NANOSECOND, null)"
> -
>
> Key: HIVE-19723
> URL: https://issues.apache.org/jira/browse/HIVE-19723
> Project: Hive
> Issue Type: Bug
>Reporter: Teddy Choi
>Assignee: Teddy Choi
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.1.0, 4.0.0
>
> Attachments: HIVE-19723.1.patch, HIVE-19732.2.patch
>
>
> Spark's Arrow support only provides Timestamp at MICROSECOND granularity.
> Spark 2.3.0 won't accept NANOSECOND. Switch it back to MICROSECOND.
> The unit test org.apache.hive.jdbc.TestJdbcWithMiniLlapArrow will just need
> to change the assertion to test microsecond. And we'll need to add this to
> documentation on supported datatypes.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (HIVE-19808) GenericUDTFGetSplits should support ACID reads in the temp. table read path
[
https://issues.apache.org/jira/browse/HIVE-19808?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Wohlstadter reassigned HIVE-19808:
---
> GenericUDTFGetSplits should support ACID reads in the temp. table read path
> ---
>
> Key: HIVE-19808
> URL: https://issues.apache.org/jira/browse/HIVE-19808
> Project: Hive
> Issue Type: Bug
>Reporter: Eric Wohlstadter
>Assignee: Eric Wohlstadter
>Priority: Major
>
> 1. Map-only reads work on ACID tables.
> 2. Temp. table reads (for multi-vertex queries) work on non-ACID tables.
> 3. But temp. table reads don't work on ACID tables.
> {code}
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Failed to create
> temp table: java.lang.IllegalStateException: calling recordValidTxn() more
> than once in the same txnid:420
> at
> org.apache.hadoop.hive.ql.udf.generic.GenericUDTFGetSplits.createPlanFragment(GenericUDTFGetSplits.java:303)
> at
> org.apache.hadoop.hive.ql.udf.generic.GenericUDTFGetSplits.process(GenericUDTFGetSplits.java:202)
> at
> org.apache.hadoop.hive.ql.exec.UDTFOperator.process(UDTFOperator.java:116)
> at
> org.apache.hadoop.hive.ql.exec.Operator.baseForward(Operator.java:985)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:931)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:918)
> at
> org.apache.hadoop.hive.ql.exec.SelectOperator.process(SelectOperator.java:95)
> at
> org.apache.hadoop.hive.ql.exec.Operator.baseForward(Operator.java:985)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:931)
> at
> org.apache.hadoop.hive.ql.exec.TableScanOperator.process(TableScanOperator.java:125)
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator.pushRow(FetchOperator.java:492)
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator.pushRow(FetchOperator.java:484)
> at org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:145)
> ... 16 more
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19808) GenericUDTFGetSplits should support ACID reads in the temp. table read path
[
https://issues.apache.org/jira/browse/HIVE-19808?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Wohlstadter updated HIVE-19808:
Attachment: HIVE-19808.1.patch
> GenericUDTFGetSplits should support ACID reads in the temp. table read path
> ---
>
> Key: HIVE-19808
> URL: https://issues.apache.org/jira/browse/HIVE-19808
> Project: Hive
> Issue Type: Bug
>Reporter: Eric Wohlstadter
>Assignee: Eric Wohlstadter
>Priority: Major
> Attachments: HIVE-19808.1.patch
>
>
> 1. Map-only reads work on ACID tables.
> 2. Temp. table reads (for multi-vertex queries) work on non-ACID tables.
> 3. But temp. table reads don't work on ACID tables.
> {code}
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Failed to create
> temp table: java.lang.IllegalStateException: calling recordValidTxn() more
> than once in the same txnid:420
> at
> org.apache.hadoop.hive.ql.udf.generic.GenericUDTFGetSplits.createPlanFragment(GenericUDTFGetSplits.java:303)
> at
> org.apache.hadoop.hive.ql.udf.generic.GenericUDTFGetSplits.process(GenericUDTFGetSplits.java:202)
> at
> org.apache.hadoop.hive.ql.exec.UDTFOperator.process(UDTFOperator.java:116)
> at
> org.apache.hadoop.hive.ql.exec.Operator.baseForward(Operator.java:985)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:931)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:918)
> at
> org.apache.hadoop.hive.ql.exec.SelectOperator.process(SelectOperator.java:95)
> at
> org.apache.hadoop.hive.ql.exec.Operator.baseForward(Operator.java:985)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:931)
> at
> org.apache.hadoop.hive.ql.exec.TableScanOperator.process(TableScanOperator.java:125)
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator.pushRow(FetchOperator.java:492)
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator.pushRow(FetchOperator.java:484)
> at org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:145)
> ... 16 more
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19808) GenericUDTFGetSplits should support ACID reads in the temp. table read path
[
https://issues.apache.org/jira/browse/HIVE-19808?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Wohlstadter updated HIVE-19808:
Status: Patch Available (was: Open)
> GenericUDTFGetSplits should support ACID reads in the temp. table read path
> ---
>
> Key: HIVE-19808
> URL: https://issues.apache.org/jira/browse/HIVE-19808
> Project: Hive
> Issue Type: Bug
>Reporter: Eric Wohlstadter
>Assignee: Eric Wohlstadter
>Priority: Major
> Attachments: HIVE-19808.1.patch
>
>
> 1. Map-only reads work on ACID tables.
> 2. Temp. table reads (for multi-vertex queries) work on non-ACID tables.
> 3. But temp. table reads don't work on ACID tables.
> {code}
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Failed to create
> temp table: java.lang.IllegalStateException: calling recordValidTxn() more
> than once in the same txnid:420
> at
> org.apache.hadoop.hive.ql.udf.generic.GenericUDTFGetSplits.createPlanFragment(GenericUDTFGetSplits.java:303)
> at
> org.apache.hadoop.hive.ql.udf.generic.GenericUDTFGetSplits.process(GenericUDTFGetSplits.java:202)
> at
> org.apache.hadoop.hive.ql.exec.UDTFOperator.process(UDTFOperator.java:116)
> at
> org.apache.hadoop.hive.ql.exec.Operator.baseForward(Operator.java:985)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:931)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:918)
> at
> org.apache.hadoop.hive.ql.exec.SelectOperator.process(SelectOperator.java:95)
> at
> org.apache.hadoop.hive.ql.exec.Operator.baseForward(Operator.java:985)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:931)
> at
> org.apache.hadoop.hive.ql.exec.TableScanOperator.process(TableScanOperator.java:125)
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator.pushRow(FetchOperator.java:492)
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator.pushRow(FetchOperator.java:484)
> at org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:145)
> ... 16 more
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19808) GenericUDTFGetSplits should support ACID reads in the temp. table read path
[
https://issues.apache.org/jira/browse/HIVE-19808?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Wohlstadter updated HIVE-19808:
Status: Open (was: Patch Available)
> GenericUDTFGetSplits should support ACID reads in the temp. table read path
> ---
>
> Key: HIVE-19808
> URL: https://issues.apache.org/jira/browse/HIVE-19808
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Reporter: Eric Wohlstadter
>Assignee: Eric Wohlstadter
>Priority: Major
> Attachments: HIVE-19808.1.patch
>
>
> 1. Map-only reads work on ACID tables.
> 2. Temp. table reads (for multi-vertex queries) work on non-ACID tables.
> 3. But temp. table reads don't work on ACID tables.
> {code}
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Failed to create
> temp table: java.lang.IllegalStateException: calling recordValidTxn() more
> than once in the same txnid:420
> at
> org.apache.hadoop.hive.ql.udf.generic.GenericUDTFGetSplits.createPlanFragment(GenericUDTFGetSplits.java:303)
> at
> org.apache.hadoop.hive.ql.udf.generic.GenericUDTFGetSplits.process(GenericUDTFGetSplits.java:202)
> at
> org.apache.hadoop.hive.ql.exec.UDTFOperator.process(UDTFOperator.java:116)
> at
> org.apache.hadoop.hive.ql.exec.Operator.baseForward(Operator.java:985)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:931)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:918)
> at
> org.apache.hadoop.hive.ql.exec.SelectOperator.process(SelectOperator.java:95)
> at
> org.apache.hadoop.hive.ql.exec.Operator.baseForward(Operator.java:985)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:931)
> at
> org.apache.hadoop.hive.ql.exec.TableScanOperator.process(TableScanOperator.java:125)
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator.pushRow(FetchOperator.java:492)
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator.pushRow(FetchOperator.java:484)
> at org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:145)
> ... 16 more
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19808) GenericUDTFGetSplits should support ACID reads in the temp. table read path
[
https://issues.apache.org/jira/browse/HIVE-19808?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16502563#comment-16502563
]
Eric Wohlstadter commented on HIVE-19808:
-
[~ekoifman]
Yeah.
What happens in GenericUDTFGetSplits is:
{code:java}
"create temporary table " + tableName + " as " + query
{code}
and then the temp table is read into LLAP and exported by the
{{LlapOutputFormatService}}.
Currently if {{query}} references an ACID table, then it fails.
The temp table itself is not ACID, the issue occurs when any of the source
tables are ACID.
> GenericUDTFGetSplits should support ACID reads in the temp. table read path
> ---
>
> Key: HIVE-19808
> URL: https://issues.apache.org/jira/browse/HIVE-19808
> Project: Hive
> Issue Type: Bug
>Reporter: Eric Wohlstadter
>Assignee: Eric Wohlstadter
>Priority: Major
>
> 1. Map-only reads work on ACID tables.
> 2. Temp. table reads (for multi-vertex queries) work on non-ACID tables.
> 3. But temp. table reads don't work on ACID tables.
> {code}
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Failed to create
> temp table: java.lang.IllegalStateException: calling recordValidTxn() more
> than once in the same txnid:420
> at
> org.apache.hadoop.hive.ql.udf.generic.GenericUDTFGetSplits.createPlanFragment(GenericUDTFGetSplits.java:303)
> at
> org.apache.hadoop.hive.ql.udf.generic.GenericUDTFGetSplits.process(GenericUDTFGetSplits.java:202)
> at
> org.apache.hadoop.hive.ql.exec.UDTFOperator.process(UDTFOperator.java:116)
> at
> org.apache.hadoop.hive.ql.exec.Operator.baseForward(Operator.java:985)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:931)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:918)
> at
> org.apache.hadoop.hive.ql.exec.SelectOperator.process(SelectOperator.java:95)
> at
> org.apache.hadoop.hive.ql.exec.Operator.baseForward(Operator.java:985)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:931)
> at
> org.apache.hadoop.hive.ql.exec.TableScanOperator.process(TableScanOperator.java:125)
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator.pushRow(FetchOperator.java:492)
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator.pushRow(FetchOperator.java:484)
> at org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:145)
> ... 16 more
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (HIVE-19808) GenericUDTFGetSplits should support ACID reads in the temp. table read path
[
https://issues.apache.org/jira/browse/HIVE-19808?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16502563#comment-16502563
]
Eric Wohlstadter edited comment on HIVE-19808 at 6/5/18 9:59 PM:
-
[~ekoifman] [~jdere]
Yeah.
What happens in GenericUDTFGetSplits is:
{code:java}
"create temporary table " + tableName + " as " + query
{code}
and then the temp table is read into LLAP and exported by the
{{LlapOutputFormatService}}.
Currently if {{query}} references an ACID table, then it fails.
The temp table itself is not ACID, the issue occurs when any of the source
tables are ACID.
was (Author: ewohlstadter):
[~ekoifman]
Yeah.
What happens in GenericUDTFGetSplits is:
{code:java}
"create temporary table " + tableName + " as " + query
{code}
and then the temp table is read into LLAP and exported by the
{{LlapOutputFormatService}}.
Currently if {{query}} references an ACID table, then it fails.
The temp table itself is not ACID, the issue occurs when any of the source
tables are ACID.
> GenericUDTFGetSplits should support ACID reads in the temp. table read path
> ---
>
> Key: HIVE-19808
> URL: https://issues.apache.org/jira/browse/HIVE-19808
> Project: Hive
> Issue Type: Bug
>Reporter: Eric Wohlstadter
>Assignee: Eric Wohlstadter
>Priority: Major
>
> 1. Map-only reads work on ACID tables.
> 2. Temp. table reads (for multi-vertex queries) work on non-ACID tables.
> 3. But temp. table reads don't work on ACID tables.
> {code}
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Failed to create
> temp table: java.lang.IllegalStateException: calling recordValidTxn() more
> than once in the same txnid:420
> at
> org.apache.hadoop.hive.ql.udf.generic.GenericUDTFGetSplits.createPlanFragment(GenericUDTFGetSplits.java:303)
> at
> org.apache.hadoop.hive.ql.udf.generic.GenericUDTFGetSplits.process(GenericUDTFGetSplits.java:202)
> at
> org.apache.hadoop.hive.ql.exec.UDTFOperator.process(UDTFOperator.java:116)
> at
> org.apache.hadoop.hive.ql.exec.Operator.baseForward(Operator.java:985)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:931)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:918)
> at
> org.apache.hadoop.hive.ql.exec.SelectOperator.process(SelectOperator.java:95)
> at
> org.apache.hadoop.hive.ql.exec.Operator.baseForward(Operator.java:985)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:931)
> at
> org.apache.hadoop.hive.ql.exec.TableScanOperator.process(TableScanOperator.java:125)
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator.pushRow(FetchOperator.java:492)
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator.pushRow(FetchOperator.java:484)
> at org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:145)
> ... 16 more
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19808) GenericUDTFGetSplits should support ACID reads in the temp. table read path
[
https://issues.apache.org/jira/browse/HIVE-19808?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16502579#comment-16502579
]
Eric Wohlstadter commented on HIVE-19808:
-
[~jdere] [~prasanth_j]
[https://reviews.apache.org/r/67462/]
> GenericUDTFGetSplits should support ACID reads in the temp. table read path
> ---
>
> Key: HIVE-19808
> URL: https://issues.apache.org/jira/browse/HIVE-19808
> Project: Hive
> Issue Type: Bug
>Reporter: Eric Wohlstadter
>Assignee: Eric Wohlstadter
>Priority: Major
> Attachments: HIVE-19808.1.patch
>
>
> 1. Map-only reads work on ACID tables.
> 2. Temp. table reads (for multi-vertex queries) work on non-ACID tables.
> 3. But temp. table reads don't work on ACID tables.
> {code}
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Failed to create
> temp table: java.lang.IllegalStateException: calling recordValidTxn() more
> than once in the same txnid:420
> at
> org.apache.hadoop.hive.ql.udf.generic.GenericUDTFGetSplits.createPlanFragment(GenericUDTFGetSplits.java:303)
> at
> org.apache.hadoop.hive.ql.udf.generic.GenericUDTFGetSplits.process(GenericUDTFGetSplits.java:202)
> at
> org.apache.hadoop.hive.ql.exec.UDTFOperator.process(UDTFOperator.java:116)
> at
> org.apache.hadoop.hive.ql.exec.Operator.baseForward(Operator.java:985)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:931)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:918)
> at
> org.apache.hadoop.hive.ql.exec.SelectOperator.process(SelectOperator.java:95)
> at
> org.apache.hadoop.hive.ql.exec.Operator.baseForward(Operator.java:985)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:931)
> at
> org.apache.hadoop.hive.ql.exec.TableScanOperator.process(TableScanOperator.java:125)
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator.pushRow(FetchOperator.java:492)
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator.pushRow(FetchOperator.java:484)
> at org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:145)
> ... 16 more
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19808) GenericUDTFGetSplits should support ACID reads in the temp. table read path
[
https://issues.apache.org/jira/browse/HIVE-19808?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Wohlstadter updated HIVE-19808:
Status: Patch Available (was: Open)
> GenericUDTFGetSplits should support ACID reads in the temp. table read path
> ---
>
> Key: HIVE-19808
> URL: https://issues.apache.org/jira/browse/HIVE-19808
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Reporter: Eric Wohlstadter
>Assignee: Eric Wohlstadter
>Priority: Major
> Attachments: HIVE-19808.1.patch, HIVE-19808.2.patch
>
>
> 1. Map-only reads work on ACID tables.
> 2. Temp. table reads (for multi-vertex queries) work on non-ACID tables.
> 3. But temp. table reads don't work on ACID tables.
> {code}
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Failed to create
> temp table: java.lang.IllegalStateException: calling recordValidTxn() more
> than once in the same txnid:420
> at
> org.apache.hadoop.hive.ql.udf.generic.GenericUDTFGetSplits.createPlanFragment(GenericUDTFGetSplits.java:303)
> at
> org.apache.hadoop.hive.ql.udf.generic.GenericUDTFGetSplits.process(GenericUDTFGetSplits.java:202)
> at
> org.apache.hadoop.hive.ql.exec.UDTFOperator.process(UDTFOperator.java:116)
> at
> org.apache.hadoop.hive.ql.exec.Operator.baseForward(Operator.java:985)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:931)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:918)
> at
> org.apache.hadoop.hive.ql.exec.SelectOperator.process(SelectOperator.java:95)
> at
> org.apache.hadoop.hive.ql.exec.Operator.baseForward(Operator.java:985)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:931)
> at
> org.apache.hadoop.hive.ql.exec.TableScanOperator.process(TableScanOperator.java:125)
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator.pushRow(FetchOperator.java:492)
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator.pushRow(FetchOperator.java:484)
> at org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:145)
> ... 16 more
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19808) GenericUDTFGetSplits should support ACID reads in the temp. table read path
[
https://issues.apache.org/jira/browse/HIVE-19808?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Wohlstadter updated HIVE-19808:
Attachment: HIVE-19808.2.patch
> GenericUDTFGetSplits should support ACID reads in the temp. table read path
> ---
>
> Key: HIVE-19808
> URL: https://issues.apache.org/jira/browse/HIVE-19808
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Reporter: Eric Wohlstadter
>Assignee: Eric Wohlstadter
>Priority: Major
> Attachments: HIVE-19808.1.patch, HIVE-19808.2.patch
>
>
> 1. Map-only reads work on ACID tables.
> 2. Temp. table reads (for multi-vertex queries) work on non-ACID tables.
> 3. But temp. table reads don't work on ACID tables.
> {code}
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Failed to create
> temp table: java.lang.IllegalStateException: calling recordValidTxn() more
> than once in the same txnid:420
> at
> org.apache.hadoop.hive.ql.udf.generic.GenericUDTFGetSplits.createPlanFragment(GenericUDTFGetSplits.java:303)
> at
> org.apache.hadoop.hive.ql.udf.generic.GenericUDTFGetSplits.process(GenericUDTFGetSplits.java:202)
> at
> org.apache.hadoop.hive.ql.exec.UDTFOperator.process(UDTFOperator.java:116)
> at
> org.apache.hadoop.hive.ql.exec.Operator.baseForward(Operator.java:985)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:931)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:918)
> at
> org.apache.hadoop.hive.ql.exec.SelectOperator.process(SelectOperator.java:95)
> at
> org.apache.hadoop.hive.ql.exec.Operator.baseForward(Operator.java:985)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:931)
> at
> org.apache.hadoop.hive.ql.exec.TableScanOperator.process(TableScanOperator.java:125)
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator.pushRow(FetchOperator.java:492)
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator.pushRow(FetchOperator.java:484)
> at org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:145)
> ... 16 more
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19305) Arrow format for LlapOutputFormatService (umbrella)
[ https://issues.apache.org/jira/browse/HIVE-19305?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Wohlstadter updated HIVE-19305: Attachment: HIVE-19305.3-branch-3.patch > Arrow format for LlapOutputFormatService (umbrella) > --- > > Key: HIVE-19305 > URL: https://issues.apache.org/jira/browse/HIVE-19305 > Project: Hive > Issue Type: Improvement > Components: llap >Reporter: Eric Wohlstadter >Assignee: Eric Wohlstadter >Priority: Major > Attachments: HIVE-19305.1-branch-3.patch, > HIVE-19305.2-branch-3.patch, HIVE-19305.3-branch-3.patch > > > Allows external clients to consume output from LLAP daemons in Arrow stream > format. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19305) Arrow format for LlapOutputFormatService (umbrella)
[ https://issues.apache.org/jira/browse/HIVE-19305?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Wohlstadter updated HIVE-19305: Status: Open (was: Patch Available) > Arrow format for LlapOutputFormatService (umbrella) > --- > > Key: HIVE-19305 > URL: https://issues.apache.org/jira/browse/HIVE-19305 > Project: Hive > Issue Type: Improvement > Components: llap >Reporter: Eric Wohlstadter >Assignee: Eric Wohlstadter >Priority: Major > Attachments: HIVE-19305.1-branch-3.patch, > HIVE-19305.2-branch-3.patch, HIVE-19305.3-branch-3.patch > > > Allows external clients to consume output from LLAP daemons in Arrow stream > format. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (HIVE-19309) Add Arrow dependencies to LlapServiceDriver
[ https://issues.apache.org/jira/browse/HIVE-19309?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16458990#comment-16458990 ] Eric Wohlstadter edited comment on HIVE-19309 at 4/30/18 8:18 PM: -- [~sershe] All the dependencies are covered in this JIRA. No additional jars need to be added/configured by the user. Can you go ahead and merge (unless you have any other concerns)? was (Author: ewohlstadter): [~sershe] Can you go ahead and merge? > Add Arrow dependencies to LlapServiceDriver > --- > > Key: HIVE-19309 > URL: https://issues.apache.org/jira/browse/HIVE-19309 > Project: Hive > Issue Type: Task > Components: llap >Reporter: Eric Wohlstadter >Assignee: Eric Wohlstadter >Priority: Major > Attachments: HIVE-19309.1.patch, HIVE-19309.2.patch > > > Need to make arrow jars available to daemons. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (HIVE-19309) Add Arrow dependencies to LlapServiceDriver
[ https://issues.apache.org/jira/browse/HIVE-19309?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16458990#comment-16458990 ] Eric Wohlstadter edited comment on HIVE-19309 at 4/30/18 8:18 PM: -- [~sershe] All the dependencies are covered in this JIRA (standard Hive build). No additional jars need to be added/configured by the user. Can you go ahead and merge (unless you have any other concerns)? was (Author: ewohlstadter): [~sershe] All the dependencies are covered in this JIRA. No additional jars need to be added/configured by the user. Can you go ahead and merge (unless you have any other concerns)? > Add Arrow dependencies to LlapServiceDriver > --- > > Key: HIVE-19309 > URL: https://issues.apache.org/jira/browse/HIVE-19309 > Project: Hive > Issue Type: Task > Components: llap >Reporter: Eric Wohlstadter >Assignee: Eric Wohlstadter >Priority: Major > Attachments: HIVE-19309.1.patch, HIVE-19309.2.patch > > > Need to make arrow jars available to daemons. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19309) Add Arrow dependencies to LlapServiceDriver
[ https://issues.apache.org/jira/browse/HIVE-19309?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16458990#comment-16458990 ] Eric Wohlstadter commented on HIVE-19309: - [~sershe] Can you go ahead and merge? > Add Arrow dependencies to LlapServiceDriver > --- > > Key: HIVE-19309 > URL: https://issues.apache.org/jira/browse/HIVE-19309 > Project: Hive > Issue Type: Task > Components: llap >Reporter: Eric Wohlstadter >Assignee: Eric Wohlstadter >Priority: Major > Attachments: HIVE-19309.1.patch, HIVE-19309.2.patch > > > Need to make arrow jars available to daemons. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19307) Support ArrowOutputStream in LlapOutputFormatService
[ https://issues.apache.org/jira/browse/HIVE-19307?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Wohlstadter updated HIVE-19307: Status: Patch Available (was: In Progress) > Support ArrowOutputStream in LlapOutputFormatService > > > Key: HIVE-19307 > URL: https://issues.apache.org/jira/browse/HIVE-19307 > Project: Hive > Issue Type: Task > Components: llap >Reporter: Eric Wohlstadter >Assignee: Eric Wohlstadter >Priority: Major > Attachments: HIVE-19307.3.patch > > > Support pushing arrow batches through > org.apache.arrow.vector.ipc.ArrowOutputStream in LllapOutputFormatService. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
