[GitHub] [incubator-iceberg] rdsr commented on a change in pull request #526: Add Baseline to iceberg-parquet

2019-10-09 Thread GitBox
rdsr commented on a change in pull request #526: Add Baseline to iceberg-parquet URL: https://github.com/apache/incubator-iceberg/pull/526#discussion_r03586 ## File path: build.gradle ## @@ -165,7 +165,7 @@ task deploySite(type: Exec) { // Baseline style guide. def

[GitHub] [incubator-iceberg] rdsr commented on issue #525: Apply Baseline to iceberg-pig

2019-10-09 Thread GitBox
rdsr commented on issue #525: Apply Baseline to iceberg-pig URL: https://github.com/apache/incubator-iceberg/pull/525#issuecomment-540294316 It seems there are a few more checkstyle errors being reported by `iceberg-pig` module in the travis build

[GitHub] [incubator-iceberg] manishmalhotrawork commented on a change in pull request #524: respect commit.manifest.min.count

2019-10-09 Thread GitBox
manishmalhotrawork commented on a change in pull request #524: respect commit.manifest.min.count URL: https://github.com/apache/incubator-iceberg/pull/524#discussion_r333281246 ## File path: core/src/main/java/org/apache/iceberg/MergingSnapshotProducer.java ## @@ -595,6

[GitHub] [incubator-iceberg] ppadma commented on issue #491: Use relative path for manifest_path and file_path

2019-10-09 Thread GitBox
ppadma commented on issue #491: Use relative path for manifest_path and file_path URL: https://github.com/apache/incubator-iceberg/pull/491#issuecomment-540242391 @rdblue @jun-ma-0 We have a use case for some complex migration scenarios where table data could be in multiple locations. We

[GitHub] [incubator-iceberg] aokolnychyi commented on a change in pull request #524: respect commit.manifest.min.count

2019-10-09 Thread GitBox
aokolnychyi commented on a change in pull request #524: respect commit.manifest.min.count URL: https://github.com/apache/incubator-iceberg/pull/524#discussion_r333245819 ## File path: core/src/main/java/org/apache/iceberg/MergingSnapshotProducer.java ## @@ -595,6 +595,9

[GitHub] [incubator-iceberg] aokolnychyi commented on a change in pull request #524: respect commit.manifest.min.count

2019-10-09 Thread GitBox
aokolnychyi commented on a change in pull request #524: respect commit.manifest.min.count URL: https://github.com/apache/incubator-iceberg/pull/524#discussion_r333245244 ## File path: core/src/main/java/org/apache/iceberg/MergingSnapshotProducer.java ## @@ -595,6 +595,9

[GitHub] [incubator-iceberg] rdsr opened a new issue #528: Infer name mapping from expected schema

2019-10-09 Thread GitBox
rdsr opened a new issue #528: Infer name mapping from expected schema URL: https://github.com/apache/incubator-iceberg/issues/528 We can infer name mapping in certain scenarios. As per @rdblue > 1. If there is a name mapping in table metadata, pass it in > 1. Otherwise, if the

[GitHub] [incubator-iceberg] rdblue commented on issue #527: Update docs to Gradle 5.4.1

2019-10-09 Thread GitBox
rdblue commented on issue #527: Update docs to Gradle 5.4.1 URL: https://github.com/apache/incubator-iceberg/pull/527#issuecomment-540183270 Thanks @Fokko! This is an automated message from the Apache Git Service. To respond

[GitHub] [incubator-iceberg] rdblue closed issue #167: Update gradle version in README.md

2019-10-09 Thread GitBox
rdblue closed issue #167: Update gradle version in README.md URL: https://github.com/apache/incubator-iceberg/issues/167 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [incubator-iceberg] Fokko opened a new pull request #527: Update docs to Gradle 5.4.1

2019-10-09 Thread GitBox
Fokko opened a new pull request #527: Update docs to Gradle 5.4.1 URL: https://github.com/apache/incubator-iceberg/pull/527 Fixes https://github.com/apache/incubator-iceberg/issues/167 This is an automated message from the

[GitHub] [incubator-iceberg] rdblue commented on issue #156: [Baseline] Apply Baseline to iceberg-data

2019-10-09 Thread GitBox
rdblue commented on issue #156: [Baseline] Apply Baseline to iceberg-data URL: https://github.com/apache/incubator-iceberg/issues/156#issuecomment-540175400 Fixed by #198. This is an automated message from the Apache Git

[GitHub] [incubator-iceberg] rdblue closed issue #156: [Baseline] Apply Baseline to iceberg-data

2019-10-09 Thread GitBox
rdblue closed issue #156: [Baseline] Apply Baseline to iceberg-data URL: https://github.com/apache/incubator-iceberg/issues/156 This is an automated message from the Apache Git Service. To respond to the message, please log

[GitHub] [incubator-iceberg] Fokko commented on issue #33: Add Avro support to Pig reader

2019-10-09 Thread GitBox
Fokko commented on issue #33: Add Avro support to Pig reader URL: https://github.com/apache/incubator-iceberg/issues/33#issuecomment-540174195 I can pick this up This is an automated message from the Apache Git Service. To

[GitHub] [incubator-iceberg] Fokko commented on issue #156: [Baseline] Apply Baseline to iceberg-data

2019-10-09 Thread GitBox
Fokko commented on issue #156: [Baseline] Apply Baseline to iceberg-data URL: https://github.com/apache/incubator-iceberg/issues/156#issuecomment-540173778 @rdsr @mccheah @rdblue I think we can close this one: https://github.com/apache/incubator-iceberg/blob/master/build.gradle#L167

[GitHub] [incubator-iceberg] rdblue commented on issue #491: Use relative path for manifest_path and file_path

2019-10-09 Thread GitBox
rdblue commented on issue #491: Use relative path for manifest_path and file_path URL: https://github.com/apache/incubator-iceberg/pull/491#issuecomment-540173189 I agree with the approach of making relative paths optional. If a path is a full path, then use it. But if it is a relative

[GitHub] [incubator-iceberg] Fokko opened a new pull request #526: Add Baseline to iceberg-parquet

2019-10-09 Thread GitBox
Fokko opened a new pull request #526: Add Baseline to iceberg-parquet URL: https://github.com/apache/incubator-iceberg/pull/526 Fixes https://github.com/apache/incubator-iceberg/issues/155 This is an automated message from

[GitHub] [incubator-iceberg] rdblue commented on issue #514: Fix for cannot update an Iceberg dataset from a Parquet file (#510)

2019-10-09 Thread GitBox
rdblue commented on issue #514: Fix for cannot update an Iceberg dataset from a Parquet file (#510) URL: https://github.com/apache/incubator-iceberg/pull/514#issuecomment-540168081 This looks like a great start, but I think this should be controlled by an option (possibly a table option

[GitHub] [incubator-iceberg] rdblue commented on issue #510: Cannot update an Iceberg dataset from a Parquet file due to "field should be required, but is optional"

2019-10-09 Thread GitBox
rdblue commented on issue #510: Cannot update an Iceberg dataset from a Parquet file due to "field should be required, but is optional" URL: https://github.com/apache/incubator-iceberg/issues/510#issuecomment-540167159 @andrei-ionescu, sorry for the delay, I had fallen behind with

[GitHub] [incubator-iceberg] rdblue commented on a change in pull request #207: Add external schema mappings for files written with name-based schemas #40

2019-10-09 Thread GitBox
rdblue commented on a change in pull request #207: Add external schema mappings for files written with name-based schemas #40 URL: https://github.com/apache/incubator-iceberg/pull/207#discussion_r333203489 ## File path:

[GitHub] [incubator-iceberg] rdsr commented on a change in pull request #207: Add external schema mappings for files written with name-based schemas #40

2019-10-09 Thread GitBox
rdsr commented on a change in pull request #207: Add external schema mappings for files written with name-based schemas #40 URL: https://github.com/apache/incubator-iceberg/pull/207#discussion_r333202728 ## File path: core/src/main/java/org/apache/iceberg/avro/PruneColumns.java

[GitHub] [incubator-iceberg] rdsr commented on a change in pull request #207: Add external schema mappings for files written with name-based schemas #40

2019-10-09 Thread GitBox
rdsr commented on a change in pull request #207: Add external schema mappings for files written with name-based schemas #40 URL: https://github.com/apache/incubator-iceberg/pull/207#discussion_r333201960 ## File path:

[GitHub] [incubator-iceberg] rdblue commented on issue #524: respect commit.manifest.min.count

2019-10-09 Thread GitBox
rdblue commented on issue #524: respect commit.manifest.min.count URL: https://github.com/apache/incubator-iceberg/pull/524#issuecomment-540155136 @manishmalhotrawork, can you describe what this change does? This is an

[GitHub] [incubator-iceberg] rdblue commented on issue #499: Add persistent IDs to partition fields (WIP)

2019-10-09 Thread GitBox
rdblue commented on issue #499: Add persistent IDs to partition fields (WIP) URL: https://github.com/apache/incubator-iceberg/pull/499#issuecomment-540154278 It looks like this is trying to assign the same IDs for a spec each time it is created, but I think the approach should be to assign

[GitHub] [incubator-iceberg] rdblue commented on a change in pull request #499: Add persistent IDs to partition fields (WIP)

2019-10-09 Thread GitBox
rdblue commented on a change in pull request #499: Add persistent IDs to partition fields (WIP) URL: https://github.com/apache/incubator-iceberg/pull/499#discussion_r333186407 ## File path: core/src/main/java/org/apache/iceberg/PartitionSpecParser.java ## @@ -33,14 +33,19

[GitHub] [incubator-iceberg] rdblue commented on a change in pull request #499: Add persistent IDs to partition fields (WIP)

2019-10-09 Thread GitBox
rdblue commented on a change in pull request #499: Add persistent IDs to partition fields (WIP) URL: https://github.com/apache/incubator-iceberg/pull/499#discussion_r333186632 ## File path: core/src/main/java/org/apache/iceberg/PartitionSpecParser.java ## @@ -125,18

[GitHub] [incubator-iceberg] Fokko opened a new pull request #525: Apply Baseline to iceberg-pig

2019-10-09 Thread GitBox
Fokko opened a new pull request #525: Apply Baseline to iceberg-pig URL: https://github.com/apache/incubator-iceberg/pull/525 Fixes https://github.com/apache/incubator-iceberg/issues/159 This is an automated message from the

[GitHub] [incubator-iceberg] rdblue commented on a change in pull request #499: Add persistent IDs to partition fields (WIP)

2019-10-09 Thread GitBox
rdblue commented on a change in pull request #499: Add persistent IDs to partition fields (WIP) URL: https://github.com/apache/incubator-iceberg/pull/499#discussion_r333162767 ## File path: api/src/main/java/org/apache/iceberg/PartitionSpec.java ## @@ -48,23 +48,24 @@ *

[GitHub] [incubator-iceberg] manishmalhotrawork opened a new pull request #524: respect commit.manifest.min.count

2019-10-09 Thread GitBox
manishmalhotrawork opened a new pull request #524: respect commit.manifest.min.count URL: https://github.com/apache/incubator-iceberg/pull/524 related to issue #490 @aokolnychyi can you please review. thanks ! This

[GitHub] [incubator-iceberg] aokolnychyi merged pull request #513: Fix concurrency issue in HiveTableOperations when Table is reused

2019-10-09 Thread GitBox
aokolnychyi merged pull request #513: Fix concurrency issue in HiveTableOperations when Table is reused URL: https://github.com/apache/incubator-iceberg/pull/513 This is an automated message from the Apache Git Service. To

[GitHub] [incubator-iceberg] rdblue commented on issue #513: Fix concurrency issue in HiveTableOperations when Table is reused

2019-10-09 Thread GitBox
rdblue commented on issue #513: Fix concurrency issue in HiveTableOperations when Table is reused URL: https://github.com/apache/incubator-iceberg/pull/513#issuecomment-540125233 Go for it! This is an automated message from

[GitHub] [incubator-iceberg] rdblue commented on a change in pull request #497: Support retaining last N snapshots

2019-10-09 Thread GitBox
rdblue commented on a change in pull request #497: Support retaining last N snapshots URL: https://github.com/apache/incubator-iceberg/pull/497#discussion_r333161128 ## File path: core/src/main/java/org/apache/iceberg/RemoveSnapshots.java ## @@ -77,8 +79,34 @@ public

[GitHub] [incubator-iceberg] aokolnychyi commented on issue #513: Fix concurrency issue in HiveTableOperations when Table is reused

2019-10-09 Thread GitBox
aokolnychyi commented on issue #513: Fix concurrency issue in HiveTableOperations when Table is reused URL: https://github.com/apache/incubator-iceberg/pull/513#issuecomment-540124989 @rdblue, I am going to merge this one if there are no objections.

[GitHub] [incubator-iceberg] yathindranath commented on a change in pull request #497: Support retaining last N snapshots

2019-10-09 Thread GitBox
yathindranath commented on a change in pull request #497: Support retaining last N snapshots URL: https://github.com/apache/incubator-iceberg/pull/497#discussion_r333158423 ## File path: core/src/main/java/org/apache/iceberg/RemoveSnapshots.java ## @@ -77,8 +79,34 @@

[GitHub] [incubator-iceberg] rdsr commented on a change in pull request #207: Add external schema mappings for files written with name-based schemas #40

2019-10-09 Thread GitBox
rdsr commented on a change in pull request #207: Add external schema mappings for files written with name-based schemas #40 URL: https://github.com/apache/incubator-iceberg/pull/207#discussion_r333149905 ## File path: core/src/main/java/org/apache/iceberg/avro/PruneColumns.java

[GitHub] [incubator-iceberg] rdblue commented on a change in pull request #207: Add external schema mappings for files written with name-based schemas #40

2019-10-09 Thread GitBox
rdblue commented on a change in pull request #207: Add external schema mappings for files written with name-based schemas #40 URL: https://github.com/apache/incubator-iceberg/pull/207#discussion_r333148519 ## File path: core/src/main/java/org/apache/iceberg/avro/PruneColumns.java

[GitHub] [incubator-iceberg] rdblue commented on a change in pull request #207: Add external schema mappings for files written with name-based schemas #40

2019-10-09 Thread GitBox
rdblue commented on a change in pull request #207: Add external schema mappings for files written with name-based schemas #40 URL: https://github.com/apache/incubator-iceberg/pull/207#discussion_r333147885 ## File path: core/src/main/java/org/apache/iceberg/avro/PruneColumns.java

[GitHub] [incubator-iceberg] rdsr commented on a change in pull request #207: Add external schema mappings for files written with name-based schemas #40

2019-10-09 Thread GitBox
rdsr commented on a change in pull request #207: Add external schema mappings for files written with name-based schemas #40 URL: https://github.com/apache/incubator-iceberg/pull/207#discussion_r333145829 ## File path: core/src/main/java/org/apache/iceberg/avro/PruneColumns.java

[GitHub] [incubator-iceberg] rdblue commented on a change in pull request #207: Add external schema mappings for files written with name-based schemas #40

2019-10-09 Thread GitBox
rdblue commented on a change in pull request #207: Add external schema mappings for files written with name-based schemas #40 URL: https://github.com/apache/incubator-iceberg/pull/207#discussion_r333144511 ## File path: core/src/main/java/org/apache/iceberg/avro/PruneColumns.java

[GitHub] [incubator-iceberg] rdblue commented on a change in pull request #207: Add external schema mappings for files written with name-based schemas #40

2019-10-09 Thread GitBox
rdblue commented on a change in pull request #207: Add external schema mappings for files written with name-based schemas #40 URL: https://github.com/apache/incubator-iceberg/pull/207#discussion_r333144120 ## File path:

[GitHub] [incubator-iceberg] rdblue commented on a change in pull request #207: Add external schema mappings for files written with name-based schemas #40

2019-10-09 Thread GitBox
rdblue commented on a change in pull request #207: Add external schema mappings for files written with name-based schemas #40 URL: https://github.com/apache/incubator-iceberg/pull/207#discussion_r333141603 ## File path: core/src/main/java/org/apache/iceberg/avro/AvroSchemaUtil.java

[GitHub] [incubator-iceberg] rdblue commented on a change in pull request #207: Add external schema mappings for files written with name-based schemas #40

2019-10-09 Thread GitBox
rdblue commented on a change in pull request #207: Add external schema mappings for files written with name-based schemas #40 URL: https://github.com/apache/incubator-iceberg/pull/207#discussion_r333140669 ## File path: core/src/main/java/org/apache/iceberg/avro/AvroSchemaUtil.java

[GitHub] [incubator-iceberg] rdblue commented on a change in pull request #207: Add external schema mappings for files written with name-based schemas #40

2019-10-09 Thread GitBox
rdblue commented on a change in pull request #207: Add external schema mappings for files written with name-based schemas #40 URL: https://github.com/apache/incubator-iceberg/pull/207#discussion_r333140157 ## File path: core/src/main/java/org/apache/iceberg/avro/AvroSchemaUtil.java

[GitHub] [incubator-iceberg] rdblue commented on a change in pull request #207: Add external schema mappings for files written with name-based schemas #40

2019-10-09 Thread GitBox
rdblue commented on a change in pull request #207: Add external schema mappings for files written with name-based schemas #40 URL: https://github.com/apache/incubator-iceberg/pull/207#discussion_r333140039 ## File path: core/src/main/java/org/apache/iceberg/avro/AvroSchemaUtil.java

[GitHub] [incubator-iceberg] rdblue commented on a change in pull request #207: Add external schema mappings for files written with name-based schemas #40

2019-10-09 Thread GitBox
rdblue commented on a change in pull request #207: Add external schema mappings for files written with name-based schemas #40 URL: https://github.com/apache/incubator-iceberg/pull/207#discussion_r333139806 ## File path: core/src/main/java/org/apache/iceberg/avro/AvroSchemaUtil.java

[GitHub] [incubator-iceberg] rdblue commented on a change in pull request #207: Add external schema mappings for files written with name-based schemas #40

2019-10-09 Thread GitBox
rdblue commented on a change in pull request #207: Add external schema mappings for files written with name-based schemas #40 URL: https://github.com/apache/incubator-iceberg/pull/207#discussion_r333139717 ## File path: core/src/main/java/org/apache/iceberg/avro/AvroSchemaUtil.java

[GitHub] [incubator-iceberg] rdblue commented on issue #31: Add startsWith predicate

2019-10-09 Thread GitBox
rdblue commented on issue #31: Add startsWith predicate URL: https://github.com/apache/incubator-iceberg/issues/31#issuecomment-540083687 I didn't realize that was locked down. Good to know, thanks! This is an automated

[GitHub] [incubator-iceberg] xabriel commented on issue #31: Add startsWith predicate

2019-10-09 Thread GitBox
xabriel commented on issue #31: Add startsWith predicate URL: https://github.com/apache/incubator-iceberg/issues/31#issuecomment-540079827 I only have comment privileges :) This is an automated message from the Apache Git

[GitHub] [incubator-iceberg] rdblue commented on a change in pull request #512: Extend RewriteManifests with a way to add/delete manifests

2019-10-09 Thread GitBox
rdblue commented on a change in pull request #512: Extend RewriteManifests with a way to add/delete manifests URL: https://github.com/apache/incubator-iceberg/pull/512#discussion_r333109080 ## File path: core/src/main/java/org/apache/iceberg/BaseRewriteManifests.java ##

[GitHub] [incubator-iceberg] rdblue merged pull request #511: Expose partition spec info

2019-10-09 Thread GitBox
rdblue merged pull request #511: Expose partition spec info URL: https://github.com/apache/incubator-iceberg/pull/511 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [incubator-iceberg] rdblue merged pull request #523: Fix typos in documentation.

2019-10-09 Thread GitBox
rdblue merged pull request #523: Fix typos in documentation. URL: https://github.com/apache/incubator-iceberg/pull/523 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [incubator-iceberg] rdblue commented on issue #523: Fix typos in documentation.

2019-10-09 Thread GitBox
rdblue commented on issue #523: Fix typos in documentation. URL: https://github.com/apache/incubator-iceberg/pull/523#issuecomment-540068617 Looks good to me. Thanks for catching these! This is an automated message from the

[GitHub] [incubator-iceberg] rdblue commented on issue #31: Add startsWith predicate

2019-10-09 Thread GitBox
rdblue commented on issue #31: Add startsWith predicate URL: https://github.com/apache/incubator-iceberg/issues/31#issuecomment-540067535 Feel free to close issues like this. It's easy to reopen them if we need to. This is

[GitHub] [incubator-iceberg] rdblue closed issue #31: Add startsWith predicate

2019-10-09 Thread GitBox
rdblue closed issue #31: Add startsWith predicate URL: https://github.com/apache/incubator-iceberg/issues/31 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and

[GitHub] [incubator-iceberg] prodeezy edited a comment on issue #522: Separate Arrow work into its own sub-module in Iceberg

2019-10-09 Thread GitBox
prodeezy edited a comment on issue #522: Separate Arrow work into its own sub-module in Iceberg URL: https://github.com/apache/incubator-iceberg/issues/522#issuecomment-539840666 @anjalinorwood isn't this same as https://github.com/apache/incubator-iceberg/issues/311 ? I can close that