[GitHub] [incubator-iceberg] waterlx edited a comment on issue #279: Transforming timestamp to date should produce date

waterlx edited a comment on issue #279: Transforming timestamp to date should produce date URL: https://github.com/apache/incubator-iceberg/issues/279#issuecomment-521086258 @rdblue, regarding > @manishmalhotrawork, only the `day` transform should return a date. Years, months, and

[GitHub] [incubator-iceberg] waterlx commented on issue #279: Transforming timestamp to date should produce date

waterlx commented on issue #279: Transforming timestamp to date should produce date URL: https://github.com/apache/incubator-iceberg/issues/279#issuecomment-521086258 @rdblue > @manishmalhotrawork, only the `day` transform should return a date. Years, months, and hours should

[GitHub] [incubator-iceberg] fbocse commented on a change in pull request #244: [WIP] Add new Update schema API to support adding of all new fields detected based on a companion schema

fbocse commented on a change in pull request #244: [WIP] Add new Update schema API to support adding of all new fields detected based on a companion schema URL: https://github.com/apache/incubator-iceberg/pull/244#discussion_r313681588 ## File path:

[GitHub] [incubator-iceberg] fbocse commented on a change in pull request #244: [WIP] Add new Update schema API to support adding of all new fields detected based on a companion schema

fbocse commented on a change in pull request #244: [WIP] Add new Update schema API to support adding of all new fields detected based on a companion schema URL: https://github.com/apache/incubator-iceberg/pull/244#discussion_r313681188 ## File path:

[GitHub] [incubator-iceberg] xabriel commented on a change in pull request #377: Add FindFiles helper API

xabriel commented on a change in pull request #377: Add FindFiles helper API URL: https://github.com/apache/incubator-iceberg/pull/377#discussion_r313667733 ## File path: core/src/main/java/org/apache/iceberg/FindFiles.java ## @@ -0,0 +1,195 @@ +/* + * Licensed to the

[GitHub] [incubator-iceberg] jun-he commented on a change in pull request #357: Add in and not in predicates

jun-he commented on a change in pull request #357: Add in and not in predicates URL: https://github.com/apache/incubator-iceberg/pull/357#discussion_r313653184 ## File path: api/src/main/java/org/apache/iceberg/expressions/Evaluator.java ## @@ -134,18 +134,18 @@ public

[GitHub] [incubator-iceberg] rdblue commented on issue #372: Allow read split behavior overrides

rdblue commented on issue #372: Allow read split behavior overrides URL: https://github.com/apache/incubator-iceberg/pull/372#issuecomment-521056199 Merged. Thanks @xabriel! This is an automated message from the Apache Git

[GitHub] [incubator-iceberg] rdblue merged pull request #372: Allow read split behavior overrides

rdblue merged pull request #372: Allow read split behavior overrides URL: https://github.com/apache/incubator-iceberg/pull/372 This is an automated message from the Apache Git Service. To respond to the message, please log

[GitHub] [incubator-iceberg] rdblue opened a new pull request #378: Handle null values in transforms

rdblue opened a new pull request #378: Handle null values in transforms URL: https://github.com/apache/incubator-iceberg/pull/378 This is an automated message from the Apache Git Service. To respond to the message, please

[GitHub] [incubator-iceberg] xabriel commented on a change in pull request #372: Allow read split behavior overrides

xabriel commented on a change in pull request #372: Allow read split behavior overrides URL: https://github.com/apache/incubator-iceberg/pull/372#discussion_r313655207 ## File path: core/src/main/java/org/apache/iceberg/BaseTableScan.java ## @@ -54,22 +56,24 @@ private

[GitHub] [incubator-iceberg] xabriel commented on a change in pull request #372: Allow read split behavior overrides

xabriel commented on a change in pull request #372: Allow read split behavior overrides URL: https://github.com/apache/incubator-iceberg/pull/372#discussion_r313654409 ## File path: core/src/main/java/org/apache/iceberg/BaseTableScan.java ## @@ -54,22 +56,24 @@ private

[GitHub] [incubator-iceberg] xabriel commented on a change in pull request #372: Allow read split behavior overrides

xabriel commented on a change in pull request #372: Allow read split behavior overrides URL: https://github.com/apache/incubator-iceberg/pull/372#discussion_r313654308 ## File path: core/src/main/java/org/apache/iceberg/BaseTableScan.java ## @@ -123,31 +128,46 @@ public

[GitHub] [incubator-iceberg] jun-he commented on a change in pull request #357: Add in and not in predicates

jun-he commented on a change in pull request #357: Add in and not in predicates URL: https://github.com/apache/incubator-iceberg/pull/357#discussion_r313653184 ## File path: api/src/main/java/org/apache/iceberg/expressions/Evaluator.java ## @@ -134,18 +134,18 @@ public

[GitHub] [incubator-iceberg] jun-he commented on a change in pull request #357: Add in and not in predicates

jun-he commented on a change in pull request #357: Add in and not in predicates URL: https://github.com/apache/incubator-iceberg/pull/357#discussion_r313653184 ## File path: api/src/main/java/org/apache/iceberg/expressions/Evaluator.java ## @@ -134,18 +134,18 @@ public

[GitHub] [incubator-iceberg] jun-he commented on a change in pull request #357: Add in and not in predicates

jun-he commented on a change in pull request #357: Add in and not in predicates URL: https://github.com/apache/incubator-iceberg/pull/357#discussion_r313653184 ## File path: api/src/main/java/org/apache/iceberg/expressions/Evaluator.java ## @@ -134,18 +134,18 @@ public

[GitHub] [incubator-iceberg] rdblue opened a new pull request #377: Add FindFiles helper API

rdblue opened a new pull request #377: Add FindFiles helper API URL: https://github.com/apache/incubator-iceberg/pull/377 This adds `FindFiles`, a helper API for working with files in a table. This API is an alternative to building a scan and calling `planFiles`. Using a scan returns

[GitHub] [incubator-iceberg] rdblue commented on a change in pull request #351: Extend Iceberg with a way to overwrite files for eager updates/deletes

rdblue commented on a change in pull request #351: Extend Iceberg with a way to overwrite files for eager updates/deletes URL: https://github.com/apache/incubator-iceberg/pull/351#discussion_r313647841 ## File path: core/src/main/java/org/apache/iceberg/OverwriteData.java

[GitHub] [incubator-iceberg] rdblue commented on a change in pull request #351: Extend Iceberg with a way to overwrite files for eager updates/deletes

rdblue commented on a change in pull request #351: Extend Iceberg with a way to overwrite files for eager updates/deletes URL: https://github.com/apache/incubator-iceberg/pull/351#discussion_r313646140 ## File path: core/src/main/java/org/apache/iceberg/OverwriteData.java

[GitHub] [incubator-iceberg] rdblue commented on a change in pull request #351: Extend Iceberg with a way to overwrite files for eager updates/deletes

rdblue commented on a change in pull request #351: Extend Iceberg with a way to overwrite files for eager updates/deletes URL: https://github.com/apache/incubator-iceberg/pull/351#discussion_r313645766 ## File path: api/src/main/java/org/apache/iceberg/OverwriteFiles.java

[GitHub] [incubator-iceberg] rdblue commented on a change in pull request #351: Extend Iceberg with a way to overwrite files for eager updates/deletes

rdblue commented on a change in pull request #351: Extend Iceberg with a way to overwrite files for eager updates/deletes URL: https://github.com/apache/incubator-iceberg/pull/351#discussion_r313645460 ## File path: api/src/main/java/org/apache/iceberg/OverwriteFiles.java

[GitHub] [incubator-iceberg] rdblue commented on a change in pull request #351: Extend Iceberg with a way to overwrite files for eager updates/deletes

rdblue commented on a change in pull request #351: Extend Iceberg with a way to overwrite files for eager updates/deletes URL: https://github.com/apache/incubator-iceberg/pull/351#discussion_r313644916 ## File path: api/src/main/java/org/apache/iceberg/OverwriteFiles.java

[GitHub] [incubator-iceberg] rdblue commented on a change in pull request #351: Extend Iceberg with a way to overwrite files for eager updates/deletes

rdblue commented on a change in pull request #351: Extend Iceberg with a way to overwrite files for eager updates/deletes URL: https://github.com/apache/incubator-iceberg/pull/351#discussion_r313643617 ## File path: api/src/main/java/org/apache/iceberg/OverwriteFiles.java

[GitHub] [incubator-iceberg] rdblue commented on a change in pull request #351: Extend Iceberg with a way to overwrite files for eager updates/deletes

rdblue commented on a change in pull request #351: Extend Iceberg with a way to overwrite files for eager updates/deletes URL: https://github.com/apache/incubator-iceberg/pull/351#discussion_r313643529 ## File path: api/src/main/java/org/apache/iceberg/OverwriteFiles.java

[GitHub] [incubator-iceberg] rdblue commented on a change in pull request #372: Allow read split behavior overrides

rdblue commented on a change in pull request #372: Allow read split behavior overrides URL: https://github.com/apache/incubator-iceberg/pull/372#discussion_r313640101 ## File path: site/docs/configuration.md ## @@ -10,7 +10,7 @@ Iceberg tables support table properties to

[GitHub] [incubator-iceberg] rdblue commented on a change in pull request #372: Allow read split behavior overrides

rdblue commented on a change in pull request #372: Allow read split behavior overrides URL: https://github.com/apache/incubator-iceberg/pull/372#discussion_r313639861 ## File path: core/src/main/java/org/apache/iceberg/BaseTableScan.java ## @@ -54,22 +56,24 @@ private

[GitHub] [incubator-iceberg] rdblue commented on a change in pull request #372: Allow read split behavior overrides

rdblue commented on a change in pull request #372: Allow read split behavior overrides URL: https://github.com/apache/incubator-iceberg/pull/372#discussion_r313639689 ## File path: core/src/main/java/org/apache/iceberg/BaseTableScan.java ## @@ -54,22 +56,24 @@ private

[GitHub] [incubator-iceberg] rdblue commented on a change in pull request #372: Allow read split behavior overrides

rdblue commented on a change in pull request #372: Allow read split behavior overrides URL: https://github.com/apache/incubator-iceberg/pull/372#discussion_r313639543 ## File path: core/src/main/java/org/apache/iceberg/BaseTableScan.java ## @@ -123,31 +128,46 @@ public

[GitHub] [incubator-iceberg] rdblue commented on a change in pull request #372: Allow read split behavior overrides

rdblue commented on a change in pull request #372: Allow read split behavior overrides URL: https://github.com/apache/incubator-iceberg/pull/372#discussion_r313638753 ## File path: core/src/main/java/org/apache/iceberg/BaseTableScan.java ## @@ -176,11 +188,26 @@ public

[GitHub] [incubator-iceberg] rdblue commented on a change in pull request #372: Allow read split behavior overrides

rdblue commented on a change in pull request #372: Allow read split behavior overrides URL: https://github.com/apache/incubator-iceberg/pull/372#discussion_r313638753 ## File path: core/src/main/java/org/apache/iceberg/BaseTableScan.java ## @@ -176,11 +188,26 @@ public

[GitHub] [incubator-iceberg] rdblue commented on a change in pull request #357: Add in and not in predicates

rdblue commented on a change in pull request #357: Add in and not in predicates URL: https://github.com/apache/incubator-iceberg/pull/357#discussion_r313633929 ## File path: api/src/main/java/org/apache/iceberg/expressions/Evaluator.java ## @@ -134,18 +134,18 @@ public

[GitHub] [incubator-iceberg] rdblue opened a new pull request #376: Do not use original manifest list when adding filters

rdblue opened a new pull request #376: Do not use original manifest list when adding filters URL: https://github.com/apache/incubator-iceberg/pull/376 This fixes a bug in `ManifestGroup` that caused all manifests to be read instead of correctly filtering when either `ignoreExisting` or

[GitHub] [incubator-iceberg] jun-he commented on a change in pull request #357: Add in and not in predicates

jun-he commented on a change in pull request #357: Add in and not in predicates URL: https://github.com/apache/incubator-iceberg/pull/357#discussion_r313593900 ## File path: api/src/main/java/org/apache/iceberg/expressions/Predicate.java ## @@ -19,15 +19,44 @@ package

[GitHub] [incubator-iceberg] jun-he commented on a change in pull request #357: Add in and not in predicates

jun-he commented on a change in pull request #357: Add in and not in predicates URL: https://github.com/apache/incubator-iceberg/pull/357#discussion_r313593900 ## File path: api/src/main/java/org/apache/iceberg/expressions/Predicate.java ## @@ -19,15 +19,44 @@ package

[GitHub] [incubator-iceberg] xabriel commented on a change in pull request #372: Allow read split behavior overrides

xabriel commented on a change in pull request #372: Allow read split behavior overrides URL: https://github.com/apache/incubator-iceberg/pull/372#discussion_r313589124 ## File path: core/src/main/java/org/apache/iceberg/BaseTableScan.java ## @@ -176,11 +188,26 @@ public

[GitHub] [incubator-iceberg] jun-he commented on a change in pull request #357: Add in and not in predicates

jun-he commented on a change in pull request #357: Add in and not in predicates URL: https://github.com/apache/incubator-iceberg/pull/357#discussion_r313576493 ## File path: api/src/main/java/org/apache/iceberg/expressions/Evaluator.java ## @@ -134,18 +134,18 @@ public

[GitHub] [incubator-iceberg] xabriel commented on a change in pull request #372: Allow read split behavior overrides

xabriel commented on a change in pull request #372: Allow read split behavior overrides URL: https://github.com/apache/incubator-iceberg/pull/372#discussion_r313525003 ## File path: core/src/main/java/org/apache/iceberg/BaseTableScan.java ## @@ -176,11 +188,26 @@ public

[GitHub] [incubator-iceberg] xabriel commented on issue #371: Fix reading of old .metadata.json.gz files

xabriel commented on issue #371: Fix reading of old .metadata.json.gz files URL: https://github.com/apache/incubator-iceberg/pull/371#issuecomment-520931166 I think the guarantee should match what we have for table evolution: If Iceberg generated it, then Iceberg should be able to read it.

[GitHub] [incubator-iceberg] rdblue commented on a change in pull request #372: Allow read split behavior overrides

rdblue commented on a change in pull request #372: Allow read split behavior overrides URL: https://github.com/apache/incubator-iceberg/pull/372#discussion_r313520937 ## File path: core/src/main/java/org/apache/iceberg/BaseTableScan.java ## @@ -176,11 +188,26 @@ public

[GitHub] [incubator-iceberg] rdblue commented on issue #365: test cases for parquetMetrics with multiple rowgroup

rdblue commented on issue #365: test cases for parquetMetrics with multiple rowgroup URL: https://github.com/apache/incubator-iceberg/pull/365#issuecomment-520927967 Thanks for contributing this, @manishmalhotrawork! I merged it.

[GitHub] [incubator-iceberg] rdblue merged pull request #365: test cases for parquetMetrics with multiple rowgroup

rdblue merged pull request #365: test cases for parquetMetrics with multiple rowgroup URL: https://github.com/apache/incubator-iceberg/pull/365 This is an automated message from the Apache Git Service. To respond to the

[GitHub] [incubator-iceberg] rdblue commented on a change in pull request #372: Allow read split behavior overrides

rdblue commented on a change in pull request #372: Allow read split behavior overrides URL: https://github.com/apache/incubator-iceberg/pull/372#discussion_r313517386 ## File path: api/src/main/java/org/apache/iceberg/TableScan.java ## @@ -58,6 +58,15 @@ */

[GitHub] [incubator-iceberg] rdblue commented on a change in pull request #372: Allow read split behavior overrides

rdblue commented on a change in pull request #372: Allow read split behavior overrides URL: https://github.com/apache/incubator-iceberg/pull/372#discussion_r313516781 ## File path: core/src/main/java/org/apache/iceberg/BaseTableScan.java ## @@ -176,11 +188,26 @@ public

[GitHub] [incubator-iceberg] rdblue commented on a change in pull request #372: Allow read split behavior overrides

rdblue commented on a change in pull request #372: Allow read split behavior overrides URL: https://github.com/apache/incubator-iceberg/pull/372#discussion_r313515860 ## File path: api/src/main/java/org/apache/iceberg/TableScan.java ## @@ -58,6 +58,15 @@ */

[GitHub] [incubator-iceberg] rdblue commented on issue #372: Allow read split behavior overrides

rdblue commented on issue #372: Allow read split behavior overrides URL: https://github.com/apache/incubator-iceberg/pull/372#issuecomment-520925050 I restarted the tests. The Hive tests are a little flaky right now. This is

[GitHub] [incubator-iceberg] rdblue merged pull request #375: Vectorized iceberg reader

rdblue merged pull request #375: Vectorized iceberg reader URL: https://github.com/apache/incubator-iceberg/pull/375 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [incubator-iceberg] rdblue merged pull request #371: Fix reading of old .metadata.json.gz files

rdblue merged pull request #371: Fix reading of old .metadata.json.gz files URL: https://github.com/apache/incubator-iceberg/pull/371 This is an automated message from the Apache Git Service. To respond to the message,

[GitHub] [incubator-iceberg] rdblue commented on issue #371: Fix reading of old .metadata.json.gz files

rdblue commented on issue #371: Fix reading of old .metadata.json.gz files URL: https://github.com/apache/incubator-iceberg/pull/371#issuecomment-520924428 Thanks for fixing this, @xabriel! I didn't realize that #258 had broken the old extension. I'm a little concerned about needing

[GitHub] [incubator-iceberg] manishmalhotrawork commented on a change in pull request #365: test cases for parquetMetrics with multiple rowgroup

manishmalhotrawork commented on a change in pull request #365: test cases for parquetMetrics with multiple rowgroup URL: https://github.com/apache/incubator-iceberg/pull/365#discussion_r313506603 ## File path: core/src/test/java/org/apache/iceberg/TestMetrics.java ## @@

[GitHub] [incubator-iceberg] manishmalhotrawork commented on a change in pull request #365: test cases for parquetMetrics with multiple rowgroup

manishmalhotrawork commented on a change in pull request #365: test cases for parquetMetrics with multiple rowgroup URL: https://github.com/apache/incubator-iceberg/pull/365#discussion_r313506252 ## File path: core/src/test/java/org/apache/iceberg/TestMetrics.java ## @@

[GitHub] [incubator-iceberg] manishmalhotrawork commented on a change in pull request #365: test cases for parquetMetrics with multiple rowgroup

manishmalhotrawork commented on a change in pull request #365: test cases for parquetMetrics with multiple rowgroup URL: https://github.com/apache/incubator-iceberg/pull/365#discussion_r313506183 ## File path: core/src/test/java/org/apache/iceberg/TestMetrics.java ## @@

[GitHub] [incubator-iceberg] xabriel commented on issue #372: Allow read split behavior overrides

xabriel commented on issue #372: Allow read split behavior overrides URL: https://github.com/apache/incubator-iceberg/pull/372#issuecomment-520912826 Not sure how to retest. Failed tests do look unrelated, caused by a `java.net.SocketException`.

[GitHub] [incubator-iceberg] rdblue commented on a change in pull request #357: Add in and not in predicates

rdblue commented on a change in pull request #357: Add in and not in predicates URL: https://github.com/apache/incubator-iceberg/pull/357#discussion_r313499455 ## File path: api/src/main/java/org/apache/iceberg/expressions/Evaluator.java ## @@ -134,18 +134,18 @@ public

[GitHub] [incubator-iceberg] rdblue commented on a change in pull request #357: Add in and not in predicates

rdblue commented on a change in pull request #357: Add in and not in predicates URL: https://github.com/apache/incubator-iceberg/pull/357#discussion_r313498614 ## File path: api/src/main/java/org/apache/iceberg/expressions/Predicate.java ## @@ -19,15 +19,44 @@ package

[GitHub] [incubator-iceberg] anjalinorwood opened a new pull request #375: Vectorized iceberg reader

anjalinorwood opened a new pull request #375: Vectorized iceberg reader URL: https://github.com/apache/incubator-iceberg/pull/375 This commit: + implements batched version of iterators (BatchedPageReader, ColumnarBatchReader etc) as the value-based iterator abstractions were

[GitHub] [incubator-iceberg] xabriel commented on issue #372: Allow read split behavior overrides

xabriel commented on issue #372: Allow read split behavior overrides URL: https://github.com/apache/incubator-iceberg/pull/372#issuecomment-520893129 retest This is an automated message from the Apache Git Service. To respond

[GitHub] [incubator-iceberg] jun-he commented on a change in pull request #357: Add in and not in predicates

jun-he commented on a change in pull request #357: Add in and not in predicates URL: https://github.com/apache/incubator-iceberg/pull/357#discussion_r313463995 ## File path: api/src/main/java/org/apache/iceberg/expressions/ExpressionVisitors.java ## @@ -89,11 +91,11 @@

[GitHub] [incubator-iceberg] chenjunjiedada opened a new pull request #374: Migrate spark table to iceberg table

chenjunjiedada opened a new pull request #374: Migrate spark table to iceberg table URL: https://github.com/apache/incubator-iceberg/pull/374 This PR converts a spark table to iceberg table. It assumes no parallel operation is going on the original table and target table and thus is not

[GitHub] [incubator-iceberg] chenjunjiedada closed pull request #373: Migrate spark table to iceberg table

chenjunjiedada closed pull request #373: Migrate spark table to iceberg table URL: https://github.com/apache/incubator-iceberg/pull/373 This is an automated message from the Apache Git Service. To respond to the message,

[GitHub] [incubator-iceberg] chenjunjiedada opened a new pull request #373: Migrate spark table to iceberg table

chenjunjiedada opened a new pull request #373: Migrate spark table to iceberg table URL: https://github.com/apache/incubator-iceberg/pull/373 This PR converts a spark table to iceberg table. It assumes no parallel operation is going on the original table and target table and thus is not

[GitHub] [incubator-iceberg] jun-he commented on a change in pull request #357: Add in and not in predicates

jun-he commented on a change in pull request #357: Add in and not in predicates URL: https://github.com/apache/incubator-iceberg/pull/357#discussion_r313259192 ## File path: api/src/main/java/org/apache/iceberg/expressions/Evaluator.java ## @@ -134,13 +135,13 @@ public

[GitHub] [incubator-iceberg] manishmalhotrawork commented on a change in pull request #365: test cases for parquetMetrics with multiple rowgroup

manishmalhotrawork commented on a change in pull request #365: test cases for parquetMetrics with multiple rowgroup URL: https://github.com/apache/incubator-iceberg/pull/365#discussion_r313238208 ## File path: core/src/test/java/org/apache/iceberg/TestMetrics.java ## @@

[GitHub] [incubator-iceberg] manishmalhotrawork commented on a change in pull request #365: test cases for parquetMetrics with multiple rowgroup

manishmalhotrawork commented on a change in pull request #365: test cases for parquetMetrics with multiple rowgroup URL: https://github.com/apache/incubator-iceberg/pull/365#discussion_r313238331 ## File path: core/src/test/java/org/apache/iceberg/TestMetrics.java ## @@

[GitHub] [incubator-iceberg] manishmalhotrawork commented on a change in pull request #365: test cases for parquetMetrics with multiple rowgroup

manishmalhotrawork commented on a change in pull request #365: test cases for parquetMetrics with multiple rowgroup URL: https://github.com/apache/incubator-iceberg/pull/365#discussion_r313238232 ## File path: core/src/test/java/org/apache/iceberg/TestMetrics.java ## @@