prodeezy commented on issue #312: Add batch sizing by num rows as config for
Vectorized reads
URL:
https://github.com/apache/incubator-iceberg/issues/312#issuecomment-518967213
yes. thank you @rdblue
This is an automated
jun-he opened a new pull request #357: Jun/in and not in predicates
URL: https://github.com/apache/incubator-iceberg/pull/357
Implements #39 .
This is an automated message from the Apache Git Service.
To respond to the
jun-he commented on issue #357: Jun/in and not in predicates
URL: https://github.com/apache/incubator-iceberg/pull/357#issuecomment-518973038
I took a different approach than the previous incomplete PR
(Netflix/iceberg#81).
Instead of adding `CollectionLiteral` and `CollectionPredicate`
aokolnychyi commented on a change in pull request #351: Provide an API to
modify records within files
URL: https://github.com/apache/incubator-iceberg/pull/351#discussion_r311501300
##
File path: core/src/main/java/org/apache/iceberg/DefaultModifyFiles.java
##
@@ -0,0
prabcs commented on issue #179: Use Iceberg tables as sources for Spark
Structured Streaming
URL:
https://github.com/apache/incubator-iceberg/issues/179#issuecomment-519074048
Ok, take it up. Thanks
On Wed, Aug 7, 2019 at 5:27 PM Li Cheng wrote:
> No work done yet. Feel
aokolnychyi edited a comment on issue #351: Provide an API to modify records
within files
URL: https://github.com/apache/incubator-iceberg/pull/351#issuecomment-519054168
I see the benefit of reusing `OverwriteFiles`, so let's think it through.
We have the following methods in
aokolnychyi edited a comment on issue #351: Provide an API to modify records
within files
URL: https://github.com/apache/incubator-iceberg/pull/351#issuecomment-519054168
I see the benefit of reusing `OverwriteFiles`, so let's think it through.
We have the following methods in
timmylicheng commented on issue #179: Use Iceberg tables as sources for Spark
Structured Streaming
URL:
https://github.com/apache/incubator-iceberg/issues/179#issuecomment-519064936
No work done yet. Feel free to take this up if you already have something
done. Just keep us posted. :)
rdblue opened a new issue #358: Add sequence numbers to metadata
URL: https://github.com/apache/incubator-iceberg/issues/358
Row-level deletes require a way to determine whether a given delete file
applies to a data file.
The scope of a delete file will be limited in 2 ways: by time
rdblue commented on a change in pull request #357: Jun/in and not in predicates
URL: https://github.com/apache/incubator-iceberg/pull/357#discussion_r311678614
##
File path:
api/src/main/java/org/apache/iceberg/expressions/UnboundPredicate.java
##
@@ -125,13 +132,26 @@
rdblue commented on a change in pull request #351: Provide an API to modify
records within files
URL: https://github.com/apache/incubator-iceberg/pull/351#discussion_r311633613
##
File path: core/src/main/java/org/apache/iceberg/DefaultModifyFiles.java
##
@@ -0,0 +1,114
rdblue opened a new issue #359: Spec: Add file and position delete files
URL: https://github.com/apache/incubator-iceberg/issues/359
One option for encoding row-level deletes is to track rows by file and row
position in the file. This format should be documented in the spec.
rdblue commented on a change in pull request #357: Jun/in and not in predicates
URL: https://github.com/apache/incubator-iceberg/pull/357#discussion_r311676526
##
File path: api/src/main/java/org/apache/iceberg/expressions/Expressions.java
##
@@ -105,6 +105,16 @@ public
rdblue commented on a change in pull request #357: Jun/in and not in predicates
URL: https://github.com/apache/incubator-iceberg/pull/357#discussion_r311681044
##
File path: api/src/main/java/org/apache/iceberg/expressions/Predicate.java
##
@@ -19,15 +19,44 @@
package
rdblue commented on issue #351: Provide an API to modify records within files
URL: https://github.com/apache/incubator-iceberg/pull/351#issuecomment-519167629
> we need to track the base snapshot id when the data was read and it should
not change during retries
Do we need to add
rdblue opened a new issue #361: Spec: Add delete file partition and sequence
number scope rules
URL: https://github.com/apache/incubator-iceberg/issues/361
This is an automated message from the Apache Git Service.
To
rdblue commented on a change in pull request #357: Jun/in and not in predicates
URL: https://github.com/apache/incubator-iceberg/pull/357#discussion_r311674751
##
File path:
api/src/main/java/org/apache/iceberg/expressions/ExpressionVisitors.java
##
@@ -89,11 +91,11 @@
rdblue commented on a change in pull request #357: Jun/in and not in predicates
URL: https://github.com/apache/incubator-iceberg/pull/357#discussion_r311680177
##
File path: api/src/main/java/org/apache/iceberg/expressions/Evaluator.java
##
@@ -134,13 +135,13 @@ public
rdblue commented on a change in pull request #357: Jun/in and not in predicates
URL: https://github.com/apache/incubator-iceberg/pull/357#discussion_r311673642
##
File path: api/src/main/java/org/apache/iceberg/expressions/Predicate.java
##
@@ -62,10 +95,10 @@ public
TGooch44 commented on issue #355: python: Remove unused deps
URL: https://github.com/apache/incubator-iceberg/pull/355#issuecomment-519192573
Should be fine we talked about it over slack. Six isn't needed anymore,
pytest isn't an install dependency and the bandit command was referencing
rdblue commented on issue #355: python: Remove unused deps
URL: https://github.com/apache/incubator-iceberg/pull/355#issuecomment-519204487
Thanks for reviewing, @TGooch44! I'm merging this.
This is an automated message from
rdblue commented on issue #357: Jun/in and not in predicates
URL: https://github.com/apache/incubator-iceberg/pull/357#issuecomment-519204167
Nice work, @jun-he! I think this is looking good, but there are a few areas
to fix. Thanks for working on this!
rdblue merged pull request #355: python: Remove unused deps
URL: https://github.com/apache/incubator-iceberg/pull/355
This is an automated message from the Apache Git Service.
To respond to the message, please log on to
rdsr commented on issue #170: Add support for Iceberg MR / InputFormat and
OutputFormat APIs
URL:
https://github.com/apache/incubator-iceberg/issues/170#issuecomment-519263693
All good points.
> I'd use Iceberg's Record interface instead. GenericRecord is the
implementation class.
rdblue commented on a change in pull request #357: Jun/in and not in predicates
URL: https://github.com/apache/incubator-iceberg/pull/357#discussion_r311684316
##
File path: api/src/main/java/org/apache/iceberg/expressions/Expressions.java
##
@@ -105,6 +105,16 @@ public
aokolnychyi opened a new pull request #362: Support create and replace
transactions in Catalog
URL: https://github.com/apache/incubator-iceberg/pull/362
This PR adds support for create and replace transactions in Catalog and
resolves #261.
rdblue commented on a change in pull request #350: Add dropTable purge option
to Catalog API
URL: https://github.com/apache/incubator-iceberg/pull/350#discussion_r311708146
##
File path: api/src/main/java/org/apache/iceberg/catalog/Catalog.java
##
@@ -120,6 +120,14 @@
moulimukherjee commented on a change in pull request #327: Introduce startsWith
Predicate
URL: https://github.com/apache/incubator-iceberg/pull/327#discussion_r311725364
##
File path: api/src/main/java/org/apache/iceberg/transforms/Truncate.java
##
@@ -213,20 +213,35 @@
aokolnychyi commented on issue #351: Provide an API to modify records within
files
URL: https://github.com/apache/incubator-iceberg/pull/351#issuecomment-519237696
> Do we need to add this to the constructor? I think it should be possible
to start an overwrite, then scan the same table
johnclara commented on a change in pull request #351: Provide an API to modify
records within files
URL: https://github.com/apache/incubator-iceberg/pull/351#discussion_r311741878
##
File path: core/src/main/java/org/apache/iceberg/DefaultModifyFiles.java
##
@@ -0,0
johnclara commented on a change in pull request #351: Provide an API to modify
records within files
URL: https://github.com/apache/incubator-iceberg/pull/351#discussion_r311741878
##
File path: core/src/main/java/org/apache/iceberg/DefaultModifyFiles.java
##
@@ -0,0
rdblue commented on issue #170: Add support for Iceberg MR / InputFormat and
OutputFormat APIs
URL:
https://github.com/apache/incubator-iceberg/issues/170#issuecomment-519258569
> IcebergPigInputFormat relies on Java serialization via
org.apache.pig.impl.util.ObjectSerializer, shall I
rdblue commented on a change in pull request #357: Jun/in and not in predicates
URL: https://github.com/apache/incubator-iceberg/pull/357#discussion_r311682632
##
File path: api/src/main/java/org/apache/iceberg/expressions/Predicate.java
##
@@ -19,15 +19,44 @@
package
aokolnychyi commented on a change in pull request #362: Support create and
replace transactions in Catalog
URL: https://github.com/apache/incubator-iceberg/pull/362#discussion_r311696942
##
File path: api/src/main/java/org/apache/iceberg/catalog/Catalog.java
##
@@ -97,6
sujithjay commented on a change in pull request #327: Introduce startsWith
Predicate
URL: https://github.com/apache/incubator-iceberg/pull/327#discussion_r311713190
##
File path: api/src/main/java/org/apache/iceberg/transforms/Truncate.java
##
@@ -213,20 +213,35 @@ public
sujithjay commented on a change in pull request #327: Introduce startsWith
Predicate
URL: https://github.com/apache/incubator-iceberg/pull/327#discussion_r311715547
##
File path: api/src/main/java/org/apache/iceberg/transforms/Truncate.java
##
@@ -243,17 +243,35 @@ public
moulimukherjee commented on issue #327: Introduce startsWith Predicate
URL: https://github.com/apache/incubator-iceberg/pull/327#issuecomment-519242836
Some nits, but the changes look good to me .
This is an automated
johnclara commented on a change in pull request #351: Provide an API to modify
records within files
URL: https://github.com/apache/incubator-iceberg/pull/351#discussion_r311741878
##
File path: core/src/main/java/org/apache/iceberg/DefaultModifyFiles.java
##
@@ -0,0
johnclara commented on a change in pull request #351: Provide an API to modify
records within files
URL: https://github.com/apache/incubator-iceberg/pull/351#discussion_r311741878
##
File path: core/src/main/java/org/apache/iceberg/DefaultModifyFiles.java
##
@@ -0,0
sujithjay commented on a change in pull request #327: Introduce startsWith
Predicate
URL: https://github.com/apache/incubator-iceberg/pull/327#discussion_r311752998
##
File path: api/src/test/java/org/apache/iceberg/transforms/TestStartsWith.java
##
@@ -0,0 +1,99 @@
+/*
+
rdblue commented on a change in pull request #357: Jun/in and not in predicates
URL: https://github.com/apache/incubator-iceberg/pull/357#discussion_r311685596
##
File path: api/src/main/java/org/apache/iceberg/expressions/Evaluator.java
##
@@ -134,13 +135,13 @@ public
sujithjay commented on a change in pull request #327: Introduce startsWith
Predicate
URL: https://github.com/apache/incubator-iceberg/pull/327#discussion_r311715547
##
File path: api/src/main/java/org/apache/iceberg/transforms/Truncate.java
##
@@ -243,17 +243,35 @@ public
aokolnychyi commented on a change in pull request #351: Provide an API to
modify records within files
URL: https://github.com/apache/incubator-iceberg/pull/351#discussion_r311756153
##
File path: core/src/main/java/org/apache/iceberg/DefaultModifyFiles.java
##
@@ -0,0
rdsr commented on a change in pull request #350: Add dropTable purge option to
Catalog API
URL: https://github.com/apache/incubator-iceberg/pull/350#discussion_r311762522
##
File path: core/src/main/java/org/apache/iceberg/BaseMetastoreCatalog.java
##
@@ -119,4 +133,81 @@
rdsr commented on a change in pull request #350: Add dropTable purge option to
Catalog API
URL: https://github.com/apache/incubator-iceberg/pull/350#discussion_r311761154
##
File path: core/src/main/java/org/apache/iceberg/BaseMetastoreCatalog.java
##
@@ -119,4 +133,81 @@
rdsr commented on a change in pull request #350: Add dropTable purge option to
Catalog API
URL: https://github.com/apache/incubator-iceberg/pull/350#discussion_r311761500
##
File path: core/src/main/java/org/apache/iceberg/BaseMetastoreCatalog.java
##
@@ -119,4 +133,81 @@
rdsr commented on a change in pull request #350: Add dropTable purge option to
Catalog API
URL: https://github.com/apache/incubator-iceberg/pull/350#discussion_r311763069
##
File path: core/src/main/java/org/apache/iceberg/BaseMetastoreCatalog.java
##
@@ -119,4 +133,81 @@
rdsr commented on a change in pull request #350: Add dropTable purge option to
Catalog API
URL: https://github.com/apache/incubator-iceberg/pull/350#discussion_r311762522
##
File path: core/src/main/java/org/apache/iceberg/BaseMetastoreCatalog.java
##
@@ -119,4 +133,81 @@
rdsr commented on a change in pull request #350: Add dropTable purge option to
Catalog API
URL: https://github.com/apache/incubator-iceberg/pull/350#discussion_r311762522
##
File path: core/src/main/java/org/apache/iceberg/BaseMetastoreCatalog.java
##
@@ -119,4 +133,81 @@
rdblue opened a new issue #363: Python: Add Quickstart doc to iceberg.apache.org
URL: https://github.com/apache/incubator-iceberg/issues/363
This is an automated message from the Apache Git Service.
To respond to the
rdblue commented on issue #351: Provide an API to modify records within files
URL: https://github.com/apache/incubator-iceberg/pull/351#issuecomment-519261508
I think that plan sounds reasonable.
> Could you elaborate a bit? I am not sure I got the use case.
We need to make
rdblue commented on a change in pull request #357: Jun/in and not in predicates
URL: https://github.com/apache/incubator-iceberg/pull/357#discussion_r311683906
##
File path:
api/src/main/java/org/apache/iceberg/expressions/UnboundPredicate.java
##
@@ -125,13 +132,26 @@
moulimukherjee commented on a change in pull request #327: Introduce startsWith
Predicate
URL: https://github.com/apache/incubator-iceberg/pull/327#discussion_r311725953
##
File path: api/src/test/java/org/apache/iceberg/transforms/TestStartsWith.java
##
@@ -0,0 +1,99 @@
guilload commented on issue #170: Add support for Iceberg MR / InputFormat and
OutputFormat APIs
URL:
https://github.com/apache/incubator-iceberg/issues/170#issuecomment-519238245
Hey,
I started working on an MR InputFormat for Iceberg since there’s some
overlap with the
moulimukherjee commented on a change in pull request #327: Introduce startsWith
Predicate
URL: https://github.com/apache/incubator-iceberg/pull/327#discussion_r311731335
##
File path: api/src/test/java/org/apache/iceberg/transforms/TestStartsWith.java
##
@@ -0,0 +1,99 @@
TGooch44 commented on a change in pull request #277: Moving/Renaming hadoop
module to filesystem
URL: https://github.com/apache/incubator-iceberg/pull/277#discussion_r311749843
##
File path: python/iceberg/core/filesystem/local_filesystem.py
##
@@ -51,34 +63,19 @@ def
rdblue commented on a change in pull request #327: Introduce startsWith
Predicate
URL: https://github.com/apache/incubator-iceberg/pull/327#discussion_r311758102
##
File path: api/src/test/java/org/apache/iceberg/transforms/TestStartsWith.java
##
@@ -0,0 +1,99 @@
+/*
+ *
rdblue commented on a change in pull request #327: Introduce startsWith
Predicate
URL: https://github.com/apache/incubator-iceberg/pull/327#discussion_r311758230
##
File path: api/src/test/java/org/apache/iceberg/transforms/TestStartsWith.java
##
@@ -0,0 +1,99 @@
+/*
+ *
rdblue commented on a change in pull request #350: Add dropTable purge option
to Catalog API
URL: https://github.com/apache/incubator-iceberg/pull/350#discussion_r311780357
##
File path: core/src/main/java/org/apache/iceberg/BaseMetastoreCatalog.java
##
@@ -119,4 +133,81
rdblue commented on a change in pull request #362: Support create and replace
transactions in Catalog
URL: https://github.com/apache/incubator-iceberg/pull/362#discussion_r311807833
##
File path:
hive/src/test/java/org/apache/iceberg/hive/HiveCreateReplaceTableTest.java
rdblue opened a new pull request #364: Clean up after create and replace
transaction failures.
URL: https://github.com/apache/incubator-iceberg/pull/364
This is a follow-up to #352. The create and replace transactions did not
delete deleted files. This also calls `cleanAll()` for each
rdblue commented on a change in pull request #350: Add dropTable purge option
to Catalog API
URL: https://github.com/apache/incubator-iceberg/pull/350#discussion_r311780233
##
File path: core/src/main/java/org/apache/iceberg/BaseMetastoreCatalog.java
##
@@ -119,4 +133,81
rdblue commented on a change in pull request #362: Support create and replace
transactions in Catalog
URL: https://github.com/apache/incubator-iceberg/pull/362#discussion_r311807434
##
File path:
hive/src/test/java/org/apache/iceberg/hive/HiveCreateReplaceTableTest.java
rdblue commented on a change in pull request #362: Support create and replace
transactions in Catalog
URL: https://github.com/apache/incubator-iceberg/pull/362#discussion_r311807710
##
File path:
hive/src/test/java/org/apache/iceberg/hive/HiveCreateReplaceTableTest.java
rdblue commented on a change in pull request #362: Support create and replace
transactions in Catalog
URL: https://github.com/apache/incubator-iceberg/pull/362#discussion_r311807151
##
File path:
hive/src/test/java/org/apache/iceberg/hive/HiveCreateReplaceTableTest.java
guilload commented on issue #170: Add support for Iceberg MR / InputFormat and
OutputFormat APIs
URL:
https://github.com/apache/incubator-iceberg/issues/170#issuecomment-519342060
Ok, thanks for the pointers. That's helpful. I'll start with `HiveCatalogs`
and we can revisit later.
guilload edited a comment on issue #170: Add support for Iceberg MR /
InputFormat and OutputFormat APIs
URL:
https://github.com/apache/incubator-iceberg/issues/170#issuecomment-519342060
Ok, thanks for the pointers. That's helpful. I'll start with `HiveCatalogs`
and we can revisit later.
guilload edited a comment on issue #170: Add support for Iceberg MR /
InputFormat and OutputFormat APIs
URL:
https://github.com/apache/incubator-iceberg/issues/170#issuecomment-519342060
Ok, thanks for the pointers. That's helpful. I'll start with `HiveCatalogs`
and we can revisit later.
rdsr edited a comment on issue #170: Add support for Iceberg MR / InputFormat
and OutputFormat APIs
URL:
https://github.com/apache/incubator-iceberg/issues/170#issuecomment-519263693
All good points.
> I'd use Iceberg's Record interface instead. GenericRecord is the
implementation
rdblue commented on a change in pull request #362: Support create and replace
transactions in Catalog
URL: https://github.com/apache/incubator-iceberg/pull/362#discussion_r311806870
##
File path: api/src/main/java/org/apache/iceberg/catalog/Catalog.java
##
@@ -97,6 +98,42
70 matches
Mail list logo