[GitHub] [incubator-iceberg] rdblue commented on issue #391: Fix create transaction metadata locations

rdblue commented on issue #391: Fix create transaction metadata locations URL: https://github.com/apache/incubator-iceberg/pull/391#issuecomment-521463182 @Parth-Brahmbhatt and @mccheah, could you take a look at this? This is

[GitHub] [incubator-iceberg] rdblue commented on a change in pull request #377: Add FindFiles helper API

rdblue commented on a change in pull request #377: Add FindFiles helper API URL: https://github.com/apache/incubator-iceberg/pull/377#discussion_r314128772 ## File path: core/src/main/java/org/apache/iceberg/FindFiles.java ## @@ -0,0 +1,195 @@ +/* + * Licensed to the

[GitHub] [incubator-iceberg] rdblue commented on issue #392: Update README.md

rdblue commented on issue #392: Update README.md URL: https://github.com/apache/incubator-iceberg/pull/392#issuecomment-521459116 Looks good to me. Thanks for fixing this! This is an automated message from the Apache Git

[GitHub] [incubator-iceberg] rdblue merged pull request #392: Update README.md

rdblue merged pull request #392: Update README.md URL: https://github.com/apache/incubator-iceberg/pull/392 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use

[GitHub] [incubator-iceberg] frankcash opened a new pull request #392: Update README.md

frankcash opened a new pull request #392: Update README.md URL: https://github.com/apache/incubator-iceberg/pull/392 Fixes top level header for README # Iceberg vs #Iceberg This is an automated message from the

[GitHub] [incubator-iceberg] rdblue opened a new pull request #391: Fix create transaction metadata locations

rdblue opened a new pull request #391: Fix create transaction metadata locations URL: https://github.com/apache/incubator-iceberg/pull/391 When a create transaction creates table metadata, it calls the underlying table operations to construct a location for the new metadata file. For

[GitHub] [incubator-iceberg] rdblue opened a new pull request #390: Minor updates to BaseMetastoreTableOperations

rdblue opened a new pull request #390: Minor updates to BaseMetastoreTableOperations URL: https://github.com/apache/incubator-iceberg/pull/390 This includes 2 updates to BaseMetastoreTableOperations: * Use overwrite to create metadata JSON files to avoid S3 negative caching * Add

[GitHub] [incubator-iceberg] rdblue opened a new pull request #389: Add test cases

rdblue opened a new pull request #389: Add test cases URL: https://github.com/apache/incubator-iceberg/pull/389 This adds a couple of test cases that have been left out of other PRs. This is an automated message from the

[GitHub] [incubator-iceberg] rdblue commented on a change in pull request #388: Handle rollback in snapshot expiration

rdblue commented on a change in pull request #388: Handle rollback in snapshot expiration URL: https://github.com/apache/incubator-iceberg/pull/388#discussion_r314106205 ## File path: core/src/main/java/org/apache/iceberg/RemoveSnapshots.java ## @@ -120,56 +123,128 @@

[GitHub] [incubator-iceberg] rdblue commented on a change in pull request #388: Handle rollback in snapshot expiration

rdblue commented on a change in pull request #388: Handle rollback in snapshot expiration URL: https://github.com/apache/incubator-iceberg/pull/388#discussion_r314106205 ## File path: core/src/main/java/org/apache/iceberg/RemoveSnapshots.java ## @@ -120,56 +123,128 @@

[GitHub] [incubator-iceberg] rdblue opened a new pull request #388: Handle rollback in snapshot expiration

rdblue opened a new pull request #388: Handle rollback in snapshot expiration URL: https://github.com/apache/incubator-iceberg/pull/388 This fixes snapshot expiration to handle rollbacks. When a table's state is rolled back, manifests and data files from the commit must be handled

[GitHub] [incubator-iceberg] rdblue opened a new pull request #387: Optimize overwrite and delete commits.

rdblue opened a new pull request #387: Optimize overwrite and delete commits. URL: https://github.com/apache/incubator-iceberg/pull/387 This uses manifest metadata to avoid filtering manifests that cannot contain deleted files in overwrite and delete operations.

[GitHub] [incubator-iceberg] rdblue opened a new pull request #386: Fix copies in partition data and field summaries

rdblue opened a new pull request #386: Fix copies in partition data and field summaries URL: https://github.com/apache/incubator-iceberg/pull/386 This is an automated message from the Apache Git Service. To respond to the

[GitHub] [incubator-iceberg] rdblue opened a new pull request #385: Make original ManifestReader factory method public

rdblue opened a new pull request #385: Make original ManifestReader factory method public URL: https://github.com/apache/incubator-iceberg/pull/385 This is easier to use when reading a manifest directly, even though the partition spec will use the schema from when the manifest was

[GitHub] [incubator-iceberg] manishmalhotrawork commented on issue #365: test cases for parquetMetrics with multiple rowgroup

manishmalhotrawork commented on issue #365: test cases for parquetMetrics with multiple rowgroup URL: https://github.com/apache/incubator-iceberg/pull/365#issuecomment-521432398 sure @rdblue, thanks for the reviews and getting it merged !!

[GitHub] [incubator-iceberg] danielcweeks merged pull request #383: Handle nulls in Conversions

danielcweeks merged pull request #383: Handle nulls in Conversions URL: https://github.com/apache/incubator-iceberg/pull/383 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [incubator-iceberg] xabriel commented on a change in pull request #377: Add FindFiles helper API

xabriel commented on a change in pull request #377: Add FindFiles helper API URL: https://github.com/apache/incubator-iceberg/pull/377#discussion_r314027026 ## File path: core/src/main/java/org/apache/iceberg/FindFiles.java ## @@ -0,0 +1,195 @@ +/* + * Licensed to the

[GitHub] [incubator-iceberg] rdblue opened a new pull request #384: Add BaseCombinedScanTask.toString

rdblue opened a new pull request #384: Add BaseCombinedScanTask.toString URL: https://github.com/apache/incubator-iceberg/pull/384 This is an automated message from the Apache Git Service. To respond to the message, please

[GitHub] [incubator-iceberg] danielcweeks merged pull request #376: Do not use original manifest list when adding filters

danielcweeks merged pull request #376: Do not use original manifest list when adding filters URL: https://github.com/apache/incubator-iceberg/pull/376 This is an automated message from the Apache Git Service. To respond to

[GitHub] [incubator-iceberg] rdblue commented on a change in pull request #377: Add FindFiles helper API

rdblue commented on a change in pull request #377: Add FindFiles helper API URL: https://github.com/apache/incubator-iceberg/pull/377#discussion_r314017383 ## File path: core/src/main/java/org/apache/iceberg/FindFiles.java ## @@ -0,0 +1,195 @@ +/* + * Licensed to the

[GitHub] [incubator-iceberg] rdblue merged pull request #379: Return full stats for added files in snapshots

rdblue merged pull request #379: Return full stats for added files in snapshots URL: https://github.com/apache/incubator-iceberg/pull/379 This is an automated message from the Apache Git Service. To respond to the message,

[GitHub] [incubator-iceberg] rdblue commented on issue #379: Return full stats for added files in snapshots

rdblue commented on issue #379: Return full stats for added files in snapshots URL: https://github.com/apache/incubator-iceberg/pull/379#issuecomment-521332016 Good catch, thanks for fixing this! This is an automated message

[GitHub] [incubator-iceberg] rdblue opened a new pull request #383: Handle nulls in Conversions

rdblue opened a new pull request #383: Handle nulls in Conversions URL: https://github.com/apache/incubator-iceberg/pull/383 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [incubator-iceberg] rdblue commented on a change in pull request #357: Add in and not in predicates

rdblue commented on a change in pull request #357: Add in and not in predicates URL: https://github.com/apache/incubator-iceberg/pull/357#discussion_r313978735 ## File path: api/src/main/java/org/apache/iceberg/expressions/Expressions.java ## @@ -109,16 +111,46 @@ public

[GitHub] [incubator-iceberg] rdblue commented on a change in pull request #357: Add in and not in predicates

rdblue commented on a change in pull request #357: Add in and not in predicates URL: https://github.com/apache/incubator-iceberg/pull/357#discussion_r313978864 ## File path: api/src/main/java/org/apache/iceberg/expressions/InclusiveMetricsEvaluator.java ## @@ -246,12

[GitHub] [incubator-iceberg] rdblue commented on issue #279: Transforming timestamp to date should produce date

rdblue commented on issue #279: Transforming timestamp to date should produce date URL: https://github.com/apache/incubator-iceberg/issues/279#issuecomment-521322801 Sorry for the confusion. I'm talking about the Iceberg date type, `Types.DateType.get()`. The transform implementation is

[GitHub] [incubator-iceberg] rdblue commented on a change in pull request #357: Add in and not in predicates

rdblue commented on a change in pull request #357: Add in and not in predicates URL: https://github.com/apache/incubator-iceberg/pull/357#discussion_r313970266 ## File path: parquet/src/main/java/org/apache/iceberg/parquet/ParquetMetricsRowGroupFilter.java ## @@ -330,12

[GitHub] [incubator-iceberg] rdblue commented on a change in pull request #357: Add in and not in predicates

rdblue commented on a change in pull request #357: Add in and not in predicates URL: https://github.com/apache/incubator-iceberg/pull/357#discussion_r313970180 ## File path: parquet/src/main/java/org/apache/iceberg/parquet/ParquetDictionaryRowGroupFilter.java ## @@

[GitHub] [incubator-iceberg] rdblue commented on a change in pull request #357: Add in and not in predicates

rdblue commented on a change in pull request #357: Add in and not in predicates URL: https://github.com/apache/incubator-iceberg/pull/357#discussion_r313966518 ## File path: api/src/main/java/org/apache/iceberg/expressions/UnboundPredicate.java ## @@ -125,13 +154,38 @@

[GitHub] [incubator-iceberg] rdblue commented on a change in pull request #357: Add in and not in predicates

rdblue commented on a change in pull request #357: Add in and not in predicates URL: https://github.com/apache/incubator-iceberg/pull/357#discussion_r313964026 ## File path: api/src/main/java/org/apache/iceberg/expressions/Predicate.java ## @@ -19,15 +19,49 @@ package

[GitHub] [incubator-iceberg] rdblue commented on a change in pull request #357: Add in and not in predicates

rdblue commented on a change in pull request #357: Add in and not in predicates URL: https://github.com/apache/incubator-iceberg/pull/357#discussion_r313963715 ## File path: api/src/main/java/org/apache/iceberg/expressions/Predicate.java ## @@ -19,15 +19,49 @@ package

[GitHub] [incubator-iceberg] rdblue commented on a change in pull request #357: Add in and not in predicates

rdblue commented on a change in pull request #357: Add in and not in predicates URL: https://github.com/apache/incubator-iceberg/pull/357#discussion_r313963009 ## File path: api/src/main/java/org/apache/iceberg/expressions/ManifestEvaluator.java ## @@ -245,12 +245,12 @@

[GitHub] [incubator-iceberg] rdblue commented on a change in pull request #357: Add in and not in predicates

rdblue commented on a change in pull request #357: Add in and not in predicates URL: https://github.com/apache/incubator-iceberg/pull/357#discussion_r313961855 ## File path: api/src/main/java/org/apache/iceberg/expressions/LiteralSet.java ## @@ -0,0 +1,212 @@ +/* + *

[GitHub] [incubator-iceberg] rdblue commented on a change in pull request #357: Add in and not in predicates

rdblue commented on a change in pull request #357: Add in and not in predicates URL: https://github.com/apache/incubator-iceberg/pull/357#discussion_r313961395 ## File path: api/src/main/java/org/apache/iceberg/expressions/ExpressionVisitors.java ## @@ -89,12 +93,12 @@

[GitHub] [incubator-iceberg] rdblue commented on a change in pull request #357: Add in and not in predicates

rdblue commented on a change in pull request #357: Add in and not in predicates URL: https://github.com/apache/incubator-iceberg/pull/357#discussion_r313961073 ## File path: api/src/main/java/org/apache/iceberg/expressions/Literals.java ## @@ -109,6 +110,26 @@ public T

[GitHub] [incubator-iceberg] rdblue commented on a change in pull request #357: Add in and not in predicates

rdblue commented on a change in pull request #357: Add in and not in predicates URL: https://github.com/apache/incubator-iceberg/pull/357#discussion_r313959744 ## File path: api/src/main/java/org/apache/iceberg/expressions/LiteralSet.java ## @@ -0,0 +1,212 @@ +/* + *

[GitHub] [incubator-iceberg] rdblue commented on a change in pull request #357: Add in and not in predicates

rdblue commented on a change in pull request #357: Add in and not in predicates URL: https://github.com/apache/incubator-iceberg/pull/357#discussion_r313959518 ## File path: api/src/main/java/org/apache/iceberg/expressions/LiteralSet.java ## @@ -0,0 +1,212 @@ +/* + *

[GitHub] [incubator-iceberg] rdblue commented on a change in pull request #357: Add in and not in predicates

rdblue commented on a change in pull request #357: Add in and not in predicates URL: https://github.com/apache/incubator-iceberg/pull/357#discussion_r313958393 ## File path: api/src/main/java/org/apache/iceberg/expressions/LiteralSet.java ## @@ -0,0 +1,212 @@ +/* + *

[GitHub] [incubator-iceberg] rdblue commented on a change in pull request #357: Add in and not in predicates

rdblue commented on a change in pull request #357: Add in and not in predicates URL: https://github.com/apache/incubator-iceberg/pull/357#discussion_r313957904 ## File path: api/src/main/java/org/apache/iceberg/expressions/LiteralSet.java ## @@ -0,0 +1,212 @@ +/* + *

[GitHub] [incubator-iceberg] rdblue commented on a change in pull request #357: Add in and not in predicates

rdblue commented on a change in pull request #357: Add in and not in predicates URL: https://github.com/apache/incubator-iceberg/pull/357#discussion_r313957552 ## File path: api/src/main/java/org/apache/iceberg/expressions/LiteralSet.java ## @@ -0,0 +1,212 @@ +/* + *

[GitHub] [incubator-iceberg] TGooch44 opened a new pull request #382: Bringing type module into sync

TGooch44 opened a new pull request #382: Bringing type module into sync URL: https://github.com/apache/incubator-iceberg/pull/382 Some minor code refactors and fixing some bugs uncovered during read path implementation. @danielcweeks

[GitHub] [incubator-iceberg] TGooch44 opened a new pull request #381: Bringing expression implementations back into synchronization

TGooch44 opened a new pull request #381: Bringing expression implementations back into synchronization URL: https://github.com/apache/incubator-iceberg/pull/381 Some of the files in expressions modules have diverged between our internal project and the open source. This commit has some

[GitHub] [incubator-iceberg] aokolnychyi opened a new pull request #380: Rename some implementations for consistency

aokolnychyi opened a new pull request #380: Rename some implementations for consistency URL: https://github.com/apache/incubator-iceberg/pull/380 This PR renames some classes to be consistent with the rest of the codebase.

[GitHub] [incubator-iceberg] aokolnychyi opened a new pull request #379: Return full stats for added files in snapshots

aokolnychyi opened a new pull request #379: Return full stats for added files in snapshots URL: https://github.com/apache/incubator-iceberg/pull/379 As a follow-up to #369, we need to include stats columns in the projection while reading manifest.

[GitHub] [incubator-iceberg] chenjunjiedada commented on issue #374: Migrate spark table to iceberg table

chenjunjiedada commented on issue #374: Migrate spark table to iceberg table URL: https://github.com/apache/incubator-iceberg/pull/374#issuecomment-521232521 @rdblue @aokolnychyi, could you please help to take an early look? I 'd like to get some basic feedback firstly.

[GitHub] [incubator-iceberg] aokolnychyi commented on issue #369: Return full stats for added files from Snapshot

aokolnychyi commented on issue #369: Return full stats for added files from Snapshot URL: https://github.com/apache/incubator-iceberg/pull/369#issuecomment-521227185 Let me submit a PR later today. This is an automated

[GitHub] [incubator-iceberg] aokolnychyi commented on issue #369: Return full stats for added files from Snapshot

aokolnychyi commented on issue #369: Return full stats for added files from Snapshot URL: https://github.com/apache/incubator-iceberg/pull/369#issuecomment-521217399 @rdblue I am afraid this change is not enough to get stats back as we have a projection as well that discards stats anyway.