[GitHub] hive pull request #506: HIVE-20911: External Table Replication for Hive

[anishek]

GitHub user anishek opened a pull request:

https://github.com/apache/hive/pull/506

HIVE-20911: External Table Replication for Hive



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/anishek/hive HIVE-20911

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/hive/pull/506.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #506


commit 57fa9f0c2c3e00e92b8ba472b05668ff3767c5f4
Author: Anishek Agarwal 
Date:   2018-10-30T08:24:41Z

HIVE-20911: External Table Replication for Hive




---

[GitHub] hive pull request #505: update Readme to fix style issue with Md

[b-slim]

Github user b-slim closed the pull request at:

https://github.com/apache/hive/pull/505


---

Re: Review Request 69462: HIVE-20936

[Jaume Marhuenda]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69462/
---

(Updated Dec. 11, 2018, 11:45 p.m.)


Review request for hive.


Repository: hive-git


Description
---

Allow the Worker thread in the metastore to run outside of it


Diffs (updated)
-

  
hcatalog/streaming/src/test/org/apache/hive/hcatalog/streaming/TestStreaming.java
 b290a40734 
  
itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/txn/compactor/TestCompactor.java
 beb36d7674 
  ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Cleaner.java 18253c9bab 
  ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/CompactorMR.java 
c6cb7c5254 
  ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/CompactorThread.java 
f5b901d6e8 
  ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Initiator.java cdcc0e9548 
  
ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/MetaStoreCompactorThread.java
 PRE-CREATION 
  
ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/RemoteCompactorThread.java 
PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Worker.java 4a1cac123c 
  ql/src/test/org/apache/hadoop/hive/ql/TestTxnCommands2.java dc39f5ef61 
  ql/src/test/org/apache/hadoop/hive/ql/TxnCommandsBaseForTests.java 52453a2ec4 
  service/src/java/org/apache/hive/service/server/HiveServer2.java 0c55654475 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/CompactionInfoStruct.java
 PRE-CREATION 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/ThriftHiveMetastore.java
 b6a0893524 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-php/metastore/ThriftHiveMetastore.php
 3170798663 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-php/metastore/Types.php
 39f8b1f05a 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-py/hive_metastore/ThriftHiveMetastore-remote
 d57de353c6 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-py/hive_metastore/ThriftHiveMetastore.py
 a896849989 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-py/hive_metastore/ttypes.py
 4ef4aadfee 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-rb/hive_metastore_types.rb
 97dc0696b7 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-rb/thrift_hive_metastore.rb
 a5f976bc5c 
  
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java
 9eb1193a27 
  
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/IMetaStoreClient.java
 fa19440ba2 
  
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/conf/MetastoreConf.java
 e25a8cf9a1 
  standalone-metastore/metastore-common/src/main/thrift/hive_metastore.thrift 
cb899d791f 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
 598847df03 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/MetaStoreThread.java
 6ef2e3560d 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/CompactionInfo.java
 b34b7d70de 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/utils/MetaStoreServerUtils.java
 3f611d6270 
  
standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClientPreCatalog.java
 9fe9a65677 
  streaming/src/test/org/apache/hive/streaming/TestStreaming.java 2170178a81 


Diff: https://reviews.apache.org/r/69462/diff/2/

Changes: https://reviews.apache.org/r/69462/diff/1-2/


Testing
---


Thanks,

Jaume Marhuenda

[GitHub] hive pull request #505: update Readme to fix style issue with Md

[b-slim]

GitHub user b-slim opened a pull request:

https://github.com/apache/hive/pull/505

update Readme to fix style issue with Md



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/b-slim/hive patch-1

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/hive/pull/505.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #505


commit 50f6c47ca0022ea445855a897f57458f118b048d
Author: Slim Bouguerra 
Date:   2018-12-11T23:35:58Z

update Readme to fix style issue with Md




---

[jira] [Created] (HIVE-21031) Array with one empty string is inserted as an empty array

[Patrick Byrnes (JIRA)]

Patrick Byrnes created HIVE-21031:
-

 Summary: Array with one empty string is inserted as an empty array
 Key: HIVE-21031
 URL: https://issues.apache.org/jira/browse/HIVE-21031
 Project: Hive
  Issue Type: Bug
Affects Versions: 2.3.2
Reporter: Patrick Byrnes


In beeline the output of
{code:java}
select array("");{code}
is:
{code:java}
[""]
{code}
However, the output of
{code:java}
insert into table a select array("");select * from a;{code}
is one row of:
{code:java}
[]{code}
 

 

Similarly, the output of
{code:java}
select array(array()){code}
is:
{code:java}
[[]]{code}
However, the output of
{code:java}
insert into table b select array(array());select a,size(a) from b;{code}
is one row of:
{code:java}
[]{code}
 

Is there a way to insert an array whose only element is an empty string or an 
array whose only element is an empty array into a table?



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [NOTICE] Mandatory relocation of Apache git repositories on git-wip-us.apache.org

[Slim Bouguerra]

Then next step will be to formalize/define the process within the community
thought.

On Tue, Dec 11, 2018 at 12:55 PM Alan Gates  wrote:

> IIUC it's the only step necessary to have one place where we could review
> and merge pull requests.  There'd be no more need for attaching patches to
> JIRA.  Of course we'd need to update our CI processes to match.
>
> Alan.
>
> On Tue, Dec 11, 2018 at 12:46 PM Slim Bouguerra  wrote:
>
> > +1 on Early move.
> >
> > Having direct write access to Github is a good first step to move toward
> > one place that can be used to review and merge pull requests IMO.
> >
> > On Dec 11, 2018, at 12:06 PM, Alan Gates  wrote:
> >
> > This includes the hive project, as we're on the older git-wip-us.  Do we
> > want to move sooner or later?
> >
> > Alan.
> >
> > -- Forwarded message -
> > From: Daniel Gruno 
> > Date: Fri, Dec 7, 2018 at 8:52 AM
> > Subject: [NOTICE] Mandatory relocation of Apache git repositories on
> > git-wip-us.apache.org
> > To: us...@infra.apache.org 
> >
> >
> > [IF YOUR PROJECT DOES NOT HAVE GIT REPOSITORIES ON GIT-WIP-US PLEASE
> >  DISREGARD THIS EMAIL; IT WAS MASS-MAILED TO ALL APACHE PROJECTS]
> >
> > Hello Apache projects,
> >
> > I am writing to you because you may have git repositories on the
> > git-wip-us server, which is slated to be decommissioned in the coming
> > months. All repositories will be moved to the new gitbox service which
> > includes direct write access on github as well as the standard ASF
> > commit access via gitbox.apache.org.
> >
> > ## Why this move? ##
> > The move comes as a result of retiring the git-wip service, as the
> > hardware it runs on is longing for retirement. In lieu of this, we
> > have decided to consolidate the two services (git-wip and gitbox), to
> > ease the management of our repository systems and future-proof the
> > underlying hardware. The move is fully automated, and ideally, nothing
> > will change in your workflow other than added features and access to
> > GitHub.
> >
> > ## Timeframe for relocation ##
> > Initially, we are asking that projects voluntarily request to move
> > their repositories to gitbox, hence this email. The voluntary
> > timeframe is between now and January 9th 2019, during which projects
> > are free to either move over to gitbox or stay put on git-wip. After
> > this phase, we will be requiring the remaining projects to move within
> > one month, after which we will move the remaining projects over.
> >
> > To have your project moved in this initial phase, you will need:
> >
> > - Consensus in the project (documented via the mailing list)
> > - File a JIRA ticket with INFRA to voluntarily move your project repos
> >   over to gitbox (as stated, this is highly automated and will take
> >   between a minute and an hour, depending on the size and number of
> >   your repositories)
> >
> > To sum up the preliminary timeline;
> >
> > - December 9th 2018 -> January 9th 2019: Voluntary (coordinated)
> >   relocation
> > - January 9th -> February 6th: Mandated (coordinated) relocation
> > - February 7th: All remaining repositories are mass migrated.
> >
> > This timeline may change to accommodate various scenarios.
> >
> > ## Using GitHub with ASF repositories ##
> > When your project has moved, you are free to use either the ASF
> > repository system (gitbox.apache.org) OR GitHub for your development
> > and code pushes. To be able to use GitHub, please follow the primer
> > at: https://reference.apache.org/committer/github
> >
> >
> > We appreciate your understanding of this issue, and hope that your
> > project can coordinate voluntarily moving your repositories in a
> > timely manner.
> >
> > All settings, such as commit mail targets, issue linking, PR
> > notification schemes etc will automatically be migrated to gitbox as
> > well.
> >
> > With regards, Daniel on behalf of ASF Infra.
> >
> > PS:For inquiries, please reply to us...@infra.apache.org, not your
> > project's dev list :-).
> >
>

Re: [NOTICE] Mandatory relocation of Apache git repositories on git-wip-us.apache.org

[Alan Gates]

IIUC it's the only step necessary to have one place where we could review
and merge pull requests.  There'd be no more need for attaching patches to
JIRA.  Of course we'd need to update our CI processes to match.

Alan.

On Tue, Dec 11, 2018 at 12:46 PM Slim Bouguerra  wrote:

> +1 on Early move.
>
> Having direct write access to Github is a good first step to move toward
> one place that can be used to review and merge pull requests IMO.
>
> On Dec 11, 2018, at 12:06 PM, Alan Gates  wrote:
>
> This includes the hive project, as we're on the older git-wip-us.  Do we
> want to move sooner or later?
>
> Alan.
>
> -- Forwarded message -
> From: Daniel Gruno 
> Date: Fri, Dec 7, 2018 at 8:52 AM
> Subject: [NOTICE] Mandatory relocation of Apache git repositories on
> git-wip-us.apache.org
> To: us...@infra.apache.org 
>
>
> [IF YOUR PROJECT DOES NOT HAVE GIT REPOSITORIES ON GIT-WIP-US PLEASE
>  DISREGARD THIS EMAIL; IT WAS MASS-MAILED TO ALL APACHE PROJECTS]
>
> Hello Apache projects,
>
> I am writing to you because you may have git repositories on the
> git-wip-us server, which is slated to be decommissioned in the coming
> months. All repositories will be moved to the new gitbox service which
> includes direct write access on github as well as the standard ASF
> commit access via gitbox.apache.org.
>
> ## Why this move? ##
> The move comes as a result of retiring the git-wip service, as the
> hardware it runs on is longing for retirement. In lieu of this, we
> have decided to consolidate the two services (git-wip and gitbox), to
> ease the management of our repository systems and future-proof the
> underlying hardware. The move is fully automated, and ideally, nothing
> will change in your workflow other than added features and access to
> GitHub.
>
> ## Timeframe for relocation ##
> Initially, we are asking that projects voluntarily request to move
> their repositories to gitbox, hence this email. The voluntary
> timeframe is between now and January 9th 2019, during which projects
> are free to either move over to gitbox or stay put on git-wip. After
> this phase, we will be requiring the remaining projects to move within
> one month, after which we will move the remaining projects over.
>
> To have your project moved in this initial phase, you will need:
>
> - Consensus in the project (documented via the mailing list)
> - File a JIRA ticket with INFRA to voluntarily move your project repos
>   over to gitbox (as stated, this is highly automated and will take
>   between a minute and an hour, depending on the size and number of
>   your repositories)
>
> To sum up the preliminary timeline;
>
> - December 9th 2018 -> January 9th 2019: Voluntary (coordinated)
>   relocation
> - January 9th -> February 6th: Mandated (coordinated) relocation
> - February 7th: All remaining repositories are mass migrated.
>
> This timeline may change to accommodate various scenarios.
>
> ## Using GitHub with ASF repositories ##
> When your project has moved, you are free to use either the ASF
> repository system (gitbox.apache.org) OR GitHub for your development
> and code pushes. To be able to use GitHub, please follow the primer
> at: https://reference.apache.org/committer/github
>
>
> We appreciate your understanding of this issue, and hope that your
> project can coordinate voluntarily moving your repositories in a
> timely manner.
>
> All settings, such as commit mail targets, issue linking, PR
> notification schemes etc will automatically be migrated to gitbox as
> well.
>
> With regards, Daniel on behalf of ASF Infra.
>
> PS:For inquiries, please reply to us...@infra.apache.org, not your
> project's dev list :-).
>

Re: [NOTICE] Mandatory relocation of Apache git repositories on git-wip-us.apache.org

[Vineet Garg]

+1 on early move.

> On Dec 11, 2018, at 12:46 PM, Slim Bouguerra  wrote:
> 
> +1 on Early move.
> 
> Having direct write access to Github is a good first step to move toward
> one place that can be used to review and merge pull requests IMO.
> 
> On Dec 11, 2018, at 12:06 PM, Alan Gates  wrote:
> 
> This includes the hive project, as we're on the older git-wip-us.  Do we
> want to move sooner or later?
> 
> Alan.
> 
> -- Forwarded message -
> From: Daniel Gruno 
> Date: Fri, Dec 7, 2018 at 8:52 AM
> Subject: [NOTICE] Mandatory relocation of Apache git repositories on
> git-wip-us.apache.org
> To: us...@infra.apache.org 
> 
> 
> [IF YOUR PROJECT DOES NOT HAVE GIT REPOSITORIES ON GIT-WIP-US PLEASE
> DISREGARD THIS EMAIL; IT WAS MASS-MAILED TO ALL APACHE PROJECTS]
> 
> Hello Apache projects,
> 
> I am writing to you because you may have git repositories on the
> git-wip-us server, which is slated to be decommissioned in the coming
> months. All repositories will be moved to the new gitbox service which
> includes direct write access on github as well as the standard ASF
> commit access via gitbox.apache.org.
> 
> ## Why this move? ##
> The move comes as a result of retiring the git-wip service, as the
> hardware it runs on is longing for retirement. In lieu of this, we
> have decided to consolidate the two services (git-wip and gitbox), to
> ease the management of our repository systems and future-proof the
> underlying hardware. The move is fully automated, and ideally, nothing
> will change in your workflow other than added features and access to
> GitHub.
> 
> ## Timeframe for relocation ##
> Initially, we are asking that projects voluntarily request to move
> their repositories to gitbox, hence this email. The voluntary
> timeframe is between now and January 9th 2019, during which projects
> are free to either move over to gitbox or stay put on git-wip. After
> this phase, we will be requiring the remaining projects to move within
> one month, after which we will move the remaining projects over.
> 
> To have your project moved in this initial phase, you will need:
> 
> - Consensus in the project (documented via the mailing list)
> - File a JIRA ticket with INFRA to voluntarily move your project repos
>  over to gitbox (as stated, this is highly automated and will take
>  between a minute and an hour, depending on the size and number of
>  your repositories)
> 
> To sum up the preliminary timeline;
> 
> - December 9th 2018 -> January 9th 2019: Voluntary (coordinated)
>  relocation
> - January 9th -> February 6th: Mandated (coordinated) relocation
> - February 7th: All remaining repositories are mass migrated.
> 
> This timeline may change to accommodate various scenarios.
> 
> ## Using GitHub with ASF repositories ##
> When your project has moved, you are free to use either the ASF
> repository system (gitbox.apache.org) OR GitHub for your development
> and code pushes. To be able to use GitHub, please follow the primer
> at: https://reference.apache.org/committer/github
> 
> 
> We appreciate your understanding of this issue, and hope that your
> project can coordinate voluntarily moving your repositories in a
> timely manner.
> 
> All settings, such as commit mail targets, issue linking, PR
> notification schemes etc will automatically be migrated to gitbox as
> well.
> 
> With regards, Daniel on behalf of ASF Infra.
> 
> PS:For inquiries, please reply to us...@infra.apache.org, not your
> project's dev list :-).

Re: [NOTICE] Mandatory relocation of Apache git repositories on git-wip-us.apache.org

[Slim Bouguerra]

+1 on Early move.

Having direct write access to Github is a good first step to move toward
one place that can be used to review and merge pull requests IMO.

On Dec 11, 2018, at 12:06 PM, Alan Gates  wrote:

This includes the hive project, as we're on the older git-wip-us.  Do we
want to move sooner or later?

Alan.

-- Forwarded message -
From: Daniel Gruno 
Date: Fri, Dec 7, 2018 at 8:52 AM
Subject: [NOTICE] Mandatory relocation of Apache git repositories on
git-wip-us.apache.org
To: us...@infra.apache.org 


[IF YOUR PROJECT DOES NOT HAVE GIT REPOSITORIES ON GIT-WIP-US PLEASE
 DISREGARD THIS EMAIL; IT WAS MASS-MAILED TO ALL APACHE PROJECTS]

Hello Apache projects,

I am writing to you because you may have git repositories on the
git-wip-us server, which is slated to be decommissioned in the coming
months. All repositories will be moved to the new gitbox service which
includes direct write access on github as well as the standard ASF
commit access via gitbox.apache.org.

## Why this move? ##
The move comes as a result of retiring the git-wip service, as the
hardware it runs on is longing for retirement. In lieu of this, we
have decided to consolidate the two services (git-wip and gitbox), to
ease the management of our repository systems and future-proof the
underlying hardware. The move is fully automated, and ideally, nothing
will change in your workflow other than added features and access to
GitHub.

## Timeframe for relocation ##
Initially, we are asking that projects voluntarily request to move
their repositories to gitbox, hence this email. The voluntary
timeframe is between now and January 9th 2019, during which projects
are free to either move over to gitbox or stay put on git-wip. After
this phase, we will be requiring the remaining projects to move within
one month, after which we will move the remaining projects over.

To have your project moved in this initial phase, you will need:

- Consensus in the project (documented via the mailing list)
- File a JIRA ticket with INFRA to voluntarily move your project repos
  over to gitbox (as stated, this is highly automated and will take
  between a minute and an hour, depending on the size and number of
  your repositories)

To sum up the preliminary timeline;

- December 9th 2018 -> January 9th 2019: Voluntary (coordinated)
  relocation
- January 9th -> February 6th: Mandated (coordinated) relocation
- February 7th: All remaining repositories are mass migrated.

This timeline may change to accommodate various scenarios.

## Using GitHub with ASF repositories ##
When your project has moved, you are free to use either the ASF
repository system (gitbox.apache.org) OR GitHub for your development
and code pushes. To be able to use GitHub, please follow the primer
at: https://reference.apache.org/committer/github


We appreciate your understanding of this issue, and hope that your
project can coordinate voluntarily moving your repositories in a
timely manner.

All settings, such as commit mail targets, issue linking, PR
notification schemes etc will automatically be migrated to gitbox as
well.

With regards, Daniel on behalf of ASF Infra.

PS:For inquiries, please reply to us...@infra.apache.org, not your
project's dev list :-).

Fwd: [NOTICE] Mandatory relocation of Apache git repositories on git-wip-us.apache.org

[Alan Gates]

This includes the hive project, as we're on the older git-wip-us.  Do we
want to move sooner or later?

Alan.

-- Forwarded message -
From: Daniel Gruno 
Date: Fri, Dec 7, 2018 at 8:52 AM
Subject: [NOTICE] Mandatory relocation of Apache git repositories on
git-wip-us.apache.org
To: us...@infra.apache.org 


[IF YOUR PROJECT DOES NOT HAVE GIT REPOSITORIES ON GIT-WIP-US PLEASE
  DISREGARD THIS EMAIL; IT WAS MASS-MAILED TO ALL APACHE PROJECTS]

Hello Apache projects,

I am writing to you because you may have git repositories on the
git-wip-us server, which is slated to be decommissioned in the coming
months. All repositories will be moved to the new gitbox service which
includes direct write access on github as well as the standard ASF
commit access via gitbox.apache.org.

## Why this move? ##
The move comes as a result of retiring the git-wip service, as the
hardware it runs on is longing for retirement. In lieu of this, we
have decided to consolidate the two services (git-wip and gitbox), to
ease the management of our repository systems and future-proof the
underlying hardware. The move is fully automated, and ideally, nothing
will change in your workflow other than added features and access to
GitHub.

## Timeframe for relocation ##
Initially, we are asking that projects voluntarily request to move
their repositories to gitbox, hence this email. The voluntary
timeframe is between now and January 9th 2019, during which projects
are free to either move over to gitbox or stay put on git-wip. After
this phase, we will be requiring the remaining projects to move within
one month, after which we will move the remaining projects over.

To have your project moved in this initial phase, you will need:

- Consensus in the project (documented via the mailing list)
- File a JIRA ticket with INFRA to voluntarily move your project repos
   over to gitbox (as stated, this is highly automated and will take
   between a minute and an hour, depending on the size and number of
   your repositories)

To sum up the preliminary timeline;

- December 9th 2018 -> January 9th 2019: Voluntary (coordinated)
   relocation
- January 9th -> February 6th: Mandated (coordinated) relocation
- February 7th: All remaining repositories are mass migrated.

This timeline may change to accommodate various scenarios.

## Using GitHub with ASF repositories ##
When your project has moved, you are free to use either the ASF
repository system (gitbox.apache.org) OR GitHub for your development
and code pushes. To be able to use GitHub, please follow the primer
at: https://reference.apache.org/committer/github


We appreciate your understanding of this issue, and hope that your
project can coordinate voluntarily moving your repositories in a
timely manner.

All settings, such as commit mail targets, issue linking, PR
notification schemes etc will automatically be migrated to gitbox as
well.

With regards, Daniel on behalf of ASF Infra.

PS:For inquiries, please reply to us...@infra.apache.org, not your
project's dev list :-).

Re: Review Request 69550: Add credential store env properties redaction in JobConf

[Vihang Karajgaonkar via Review Board]

> On Dec. 11, 2018, 4:30 p.m., Vihang Karajgaonkar wrote:
> > common/src/java/org/apache/hadoop/hive/conf/HiveConfUtil.java
> > Lines 217 (patched)
> > 
> >
> > Do we need to do redact for the Spark config as well?
> 
> Denys Kuzmenko wrote:
> no, it's already handled in Spark:
> 
> https://github.com/apache/spark/commit/66636ef0b046e5d1f340c3b8153d7213fa9d19c7

Okay. Thanks for checking.


- Vihang


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69550/#review211199
---


On Dec. 11, 2018, 12:43 p.m., Denys Kuzmenko wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/69550/
> ---
> 
> (Updated Dec. 11, 2018, 12:43 p.m.)
> 
> 
> Review request for hive, Peter Vary and Vihang Karajgaonkar.
> 
> 
> Bugs: HIVE-21030
> https://issues.apache.org/jira/browse/HIVE-21030
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Credstore decryption password should be redacted in JobConf
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConfUtil.java 
> 2ad5f9ee39f376d3466994a24cc9f7850be902ae 
> 
> 
> Diff: https://reviews.apache.org/r/69550/diff/1/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Denys Kuzmenko
> 
>

Re: Review Request 69550: Add credential store env properties redaction in JobConf

[Vihang Karajgaonkar via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69550/#review211205
---


Ship it!




Ship It!

- Vihang Karajgaonkar


On Dec. 11, 2018, 12:43 p.m., Denys Kuzmenko wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/69550/
> ---
> 
> (Updated Dec. 11, 2018, 12:43 p.m.)
> 
> 
> Review request for hive, Peter Vary and Vihang Karajgaonkar.
> 
> 
> Bugs: HIVE-21030
> https://issues.apache.org/jira/browse/HIVE-21030
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Credstore decryption password should be redacted in JobConf
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConfUtil.java 
> 2ad5f9ee39f376d3466994a24cc9f7850be902ae 
> 
> 
> Diff: https://reviews.apache.org/r/69550/diff/1/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Denys Kuzmenko
> 
>

Re: [DISCUSS] Consistent Timestamps across Hadoop

[Zoltan Ivanfi]

Hi Owen,

Thanks, I think your email contains a great summary of the problems tackled
in the proposal. I would like highlight two particular topics from the
discussion that we are having in the comments (details can be read in the
document

):

   - It seems that we have agreement on the desired semantics of the more
   explicit SQL types. In particular, I was glad to hear that the TIMESTAMP
   WITH LOCAL TIME ZONE type that is already implemented in Hive is supposed
   to have Instant semantics. (In fact, it already does have Instant
   semantics, but it also has additional time zone information that is unused
   at this moment, and I wasn't sure whether that will be utilized, changing
   the semantics, or whether the semantics will remain and the superflous time
   zone data will be removed.)
   - We are still discussing what is the best course of action to take with
   the plain TIMESTAMP type, which behaved differently in different file
   formats in Hive 2 and was made to behave the same way in a
   compatibility-breaking manner in Hive 3. My take on this type is that it
   has already been used to write huge amounts of data and for this reason we
   should restore its Avro- and Parquet-specific incosistent behaviour
   (possibly controlled by a feature flag), so that legacy data remains
   readable and legacy workarounds remain functional. The new, more explicit
   SQL types will provide a clear migration path away from the messy TIMESTAMP
   type.

All in all, I feel that we are converging towards a common goal and I have
high hopes that the more explicit timestamp types will have much better
interoperability and consistency across different Hadoop SQL engines.

Thanks,

Zoltan


On Mon, Dec 10, 2018 at 7:54 PM Owen O'Malley 
wrote:

> Thank you for starting this discussion. Clearly the Hive semantics on
> timestamp are very messed up, but has been moving in the right direction of
> becoming more SQL standard compliant. I'm pulling this discussion back to
> the list rather than the personal GoogleDoc, which isn't very
> collaborative.
>
> I like your breakdown of the semantics:
>
>- Instant - point in time that will appear different depending on the
>reader time zone
>- LocalDateTime - consistent hour and minute regardless of the reader
>time zone.
>- OffsetDateTime - consistent hour and minute with the offset of the
>writer time zone
>
> The SQL standard has:
>
>- Timestamp & Timestamp without time zone = LocalDateTime
>- Timestamp with time zone = OffsetDateTime
>
> Hive 2 had very confused semantics for timestamp:
>
>- When storage was ORC, text, or RCFile with a text serde it was
>LocalDateTime
>- When storage was Avro, Parquet, or RCFile with a binary serde it was
>Instant
>
> Hive 3.1 has moved toward the SQL standard extended with Oracles'
> timestamp with local time zone:
>
>- Timestamp = LocalDateTime
>- Timestamp with local time zone = Instant
>
> This leaves us with a few problems:
>
>- The Hive bindings to Parquet and Avro don't handle timestamps
>correctly.
>- ORC doesn't support timestamps with local time zone. I start working
>on it in ORC-189.
>- We don't have timestamp with time zone support.
>
> .. Owen
>
> On Thu, Dec 6, 2018 at 7:55 AM Marta Kuczora 
> wrote:
>
>> Hi Hive Community,
>>
>> I would like to share the following document on our "Consistent Timestamp
>> types in Hadoop" plans for review.
>>
>> https://docs.google.com/document/d/1gNRww9mZJcHvUDCXklzjFEQGpefsuR_akCDfWsdE35Q/edit
>>
>> With this plan we would like to get an agreement on consistent timestamp
>> behavior on Hive, Spark and Impala and in order to achieve this, we are
>> sharing this document with all three communities.
>>
>> Please review and comment, any feedback is much appreciated!
>>
>> Regards,
>> Marta
>>
>

Re: Review Request 69550: Add credential store env properties redaction in JobConf

[Denys Kuzmenko via Review Board]

> On Dec. 11, 2018, 4:30 p.m., Vihang Karajgaonkar wrote:
> > common/src/java/org/apache/hadoop/hive/conf/HiveConfUtil.java
> > Lines 217 (patched)
> > 
> >
> > Do we need to do redact for the Spark config as well?

no, it's already handled in Spark:
https://github.com/apache/spark/commit/66636ef0b046e5d1f340c3b8153d7213fa9d19c7


- Denys


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69550/#review211199
---


On Dec. 11, 2018, 12:43 p.m., Denys Kuzmenko wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/69550/
> ---
> 
> (Updated Dec. 11, 2018, 12:43 p.m.)
> 
> 
> Review request for hive, Peter Vary and Vihang Karajgaonkar.
> 
> 
> Bugs: HIVE-21030
> https://issues.apache.org/jira/browse/HIVE-21030
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Credstore decryption password should be redacted in JobConf
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConfUtil.java 
> 2ad5f9ee39f376d3466994a24cc9f7850be902ae 
> 
> 
> Diff: https://reviews.apache.org/r/69550/diff/1/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Denys Kuzmenko
> 
>

Re: Review Request 69550: Add credential store env properties redaction in JobConf

[Vihang Karajgaonkar via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69550/#review211199
---




common/src/java/org/apache/hadoop/hive/conf/HiveConfUtil.java
Lines 217 (patched)


Do we need to do redact for the Spark config as well?


- Vihang Karajgaonkar


On Dec. 11, 2018, 12:43 p.m., Denys Kuzmenko wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/69550/
> ---
> 
> (Updated Dec. 11, 2018, 12:43 p.m.)
> 
> 
> Review request for hive, Peter Vary and Vihang Karajgaonkar.
> 
> 
> Bugs: HIVE-21030
> https://issues.apache.org/jira/browse/HIVE-21030
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Credstore decryption password should be redacted in JobConf
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConfUtil.java 
> 2ad5f9ee39f376d3466994a24cc9f7850be902ae 
> 
> 
> Diff: https://reviews.apache.org/r/69550/diff/1/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Denys Kuzmenko
> 
>

Re: Review Request 69550: Add credential store env properties redaction in JobConf

[Denys Kuzmenko via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69550/
---

(Updated Dec. 11, 2018, 12:43 p.m.)


Review request for hive, Peter Vary and Vihang Karajgaonkar.


Bugs: HIVE-21030
https://issues.apache.org/jira/browse/HIVE-21030


Repository: hive-git


Description
---

Credstore decryption password should be redacted in JobConf


Diffs
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConfUtil.java 
2ad5f9ee39f376d3466994a24cc9f7850be902ae 


Diff: https://reviews.apache.org/r/69550/diff/1/


Testing
---


Thanks,

Denys Kuzmenko

Review Request 69550: Add credential store env properties redaction in JobConf

[Denys Kuzmenko via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69550/
---

Review request for hive, Peter Vary and Vihang Karajgaonkar.


Repository: hive-git


Description
---

Credstore decryption password should be redacted in JobConf


Diffs
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConfUtil.java 
2ad5f9ee39f376d3466994a24cc9f7850be902ae 


Diff: https://reviews.apache.org/r/69550/diff/1/


Testing
---


Thanks,

Denys Kuzmenko

[jira] [Created] (HIVE-21030) Add credential store env properties redaction in JobConf

[Denys Kuzmenko (JIRA)]

Denys Kuzmenko created HIVE-21030:
-

 Summary: Add credential store env properties redaction in JobConf
 Key: HIVE-21030
 URL: https://issues.apache.org/jira/browse/HIVE-21030
 Project: Hive
  Issue Type: Bug
Reporter: Denys Kuzmenko






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)