[jira] [Updated] (HUDI-389) Updates sent to diff partition for a given key with Global Index
[
https://issues.apache.org/jira/browse/HUDI-389?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
leesf updated HUDI-389:
---
Fix Version/s: (was: 0.5.2)
0.5.1
> Updates sent to diff partition for a given key with Global Index
> -
>
> Key: HUDI-389
> URL: https://issues.apache.org/jira/browse/HUDI-389
> Project: Apache Hudi (incubating)
> Issue Type: Bug
> Components: Index
>Reporter: sivabalan narayanan
>Assignee: sivabalan narayanan
>Priority: Major
> Labels: pull-request-available
> Fix For: 0.5.1
>
> Original Estimate: 48h
> Time Spent: 20m
> Remaining Estimate: 47h 40m
>
> Updates sent to diff partition for a given key with Global Index should
> succeed by updating the record under original partition. As of now, it throws
> exception.
> [https://github.com/apache/incubator-hudi/issues/1021]
>
>
> error log:
> {code:java}
> 14738 [Executor task launch worker-0] INFO
> com.uber.hoodie.common.table.timeline.HoodieActiveTimeline - Loaded instants
> java.util.stream.ReferencePipeline$Head@d02b1c7
> 14738 [Executor task launch worker-0] INFO
> com.uber.hoodie.common.table.view.AbstractTableFileSystemView - Building file
> system view for partition (2016/04/15)
> 14738 [Executor task launch worker-0] INFO
> com.uber.hoodie.common.table.view.AbstractTableFileSystemView - #files found
> in partition (2016/04/15) =0, Time taken =0
> 14738 [Executor task launch worker-0] INFO
> com.uber.hoodie.common.table.view.AbstractTableFileSystemView -
> addFilesToView: NumFiles=0, FileGroupsCreationTime=0, StoreTimeTaken=0
> 14738 [Executor task launch worker-0] INFO
> com.uber.hoodie.common.table.view.HoodieTableFileSystemView - Adding
> file-groups for partition :2016/04/15, #FileGroups=0
> 14738 [Executor task launch worker-0] INFO
> com.uber.hoodie.common.table.view.AbstractTableFileSystemView - Time to load
> partition (2016/04/15) =0
> 14754 [Executor task launch worker-0] ERROR
> com.uber.hoodie.table.HoodieCopyOnWriteTable - Error upserting bucketType
> UPDATE for partition :0
> java.util.NoSuchElementException: No value present
> at com.uber.hoodie.common.util.Option.get(Option.java:112)
> at com.uber.hoodie.io.HoodieMergeHandle.(HoodieMergeHandle.java:71)
> at
> com.uber.hoodie.table.HoodieCopyOnWriteTable.getUpdateHandle(HoodieCopyOnWriteTable.java:226)
> at
> com.uber.hoodie.table.HoodieCopyOnWriteTable.handleUpdate(HoodieCopyOnWriteTable.java:180)
> at
> com.uber.hoodie.table.HoodieCopyOnWriteTable.handleUpsertPartition(HoodieCopyOnWriteTable.java:263)
> at
> com.uber.hoodie.HoodieWriteClient.lambda$upsertRecordsInternal$7ef77fd$1(HoodieWriteClient.java:442)
> at
> org.apache.spark.api.java.JavaRDDLike$$anonfun$mapPartitionsWithIndex$1.apply(JavaRDDLike.scala:102)
> at
> org.apache.spark.api.java.JavaRDDLike$$anonfun$mapPartitionsWithIndex$1.apply(JavaRDDLike.scala:102)
> at
> org.apache.spark.rdd.RDD$$anonfun$mapPartitionsWithIndex$1$$anonfun$apply$26.apply(RDD.scala:843)
> at
> org.apache.spark.rdd.RDD$$anonfun$mapPartitionsWithIndex$1$$anonfun$apply$26.apply(RDD.scala:843)
> at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
> at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
> at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
> at org.apache.spark.rdd.RDD$$anonfun$8.apply(RDD.scala:336)
> at org.apache.spark.rdd.RDD$$anonfun$8.apply(RDD.scala:334)
> at
> org.apache.spark.storage.BlockManager$$anonfun$doPutIterator$1.apply(BlockManager.scala:973)
> at
> org.apache.spark.storage.BlockManager$$anonfun$doPutIterator$1.apply(BlockManager.scala:948)
> at org.apache.spark.storage.BlockManager.doPut(BlockManager.scala:888)
> at
> org.apache.spark.storage.BlockManager.doPutIterator(BlockManager.scala:948)
> at
> org.apache.spark.storage.BlockManager.getOrElseUpdate(BlockManager.scala:694)
> at org.apache.spark.rdd.RDD.getOrCompute(RDD.scala:334)
> at org.apache.spark.rdd.RDD.iterator(RDD.scala:285)
> at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
> at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
> at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
> at org.apache.spark.scheduler.Task.run(Task.scala:99)
> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:282)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at
[jira] [Updated] (HUDI-389) Updates sent to diff partition for a given key with Global Index
[
https://issues.apache.org/jira/browse/HUDI-389?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Suneel Marthi updated HUDI-389:
---
Fix Version/s: (was: 0.5.1)
0.5.2
> Updates sent to diff partition for a given key with Global Index
> -
>
> Key: HUDI-389
> URL: https://issues.apache.org/jira/browse/HUDI-389
> Project: Apache Hudi (incubating)
> Issue Type: Bug
> Components: Index
>Reporter: sivabalan narayanan
>Assignee: sivabalan narayanan
>Priority: Major
> Labels: pull-request-available
> Fix For: 0.5.2
>
> Original Estimate: 48h
> Time Spent: 20m
> Remaining Estimate: 47h 40m
>
> Updates sent to diff partition for a given key with Global Index should
> succeed by updating the record under original partition. As of now, it throws
> exception.
> [https://github.com/apache/incubator-hudi/issues/1021]
>
>
> error log:
> {code:java}
> 14738 [Executor task launch worker-0] INFO
> com.uber.hoodie.common.table.timeline.HoodieActiveTimeline - Loaded instants
> java.util.stream.ReferencePipeline$Head@d02b1c7
> 14738 [Executor task launch worker-0] INFO
> com.uber.hoodie.common.table.view.AbstractTableFileSystemView - Building file
> system view for partition (2016/04/15)
> 14738 [Executor task launch worker-0] INFO
> com.uber.hoodie.common.table.view.AbstractTableFileSystemView - #files found
> in partition (2016/04/15) =0, Time taken =0
> 14738 [Executor task launch worker-0] INFO
> com.uber.hoodie.common.table.view.AbstractTableFileSystemView -
> addFilesToView: NumFiles=0, FileGroupsCreationTime=0, StoreTimeTaken=0
> 14738 [Executor task launch worker-0] INFO
> com.uber.hoodie.common.table.view.HoodieTableFileSystemView - Adding
> file-groups for partition :2016/04/15, #FileGroups=0
> 14738 [Executor task launch worker-0] INFO
> com.uber.hoodie.common.table.view.AbstractTableFileSystemView - Time to load
> partition (2016/04/15) =0
> 14754 [Executor task launch worker-0] ERROR
> com.uber.hoodie.table.HoodieCopyOnWriteTable - Error upserting bucketType
> UPDATE for partition :0
> java.util.NoSuchElementException: No value present
> at com.uber.hoodie.common.util.Option.get(Option.java:112)
> at com.uber.hoodie.io.HoodieMergeHandle.(HoodieMergeHandle.java:71)
> at
> com.uber.hoodie.table.HoodieCopyOnWriteTable.getUpdateHandle(HoodieCopyOnWriteTable.java:226)
> at
> com.uber.hoodie.table.HoodieCopyOnWriteTable.handleUpdate(HoodieCopyOnWriteTable.java:180)
> at
> com.uber.hoodie.table.HoodieCopyOnWriteTable.handleUpsertPartition(HoodieCopyOnWriteTable.java:263)
> at
> com.uber.hoodie.HoodieWriteClient.lambda$upsertRecordsInternal$7ef77fd$1(HoodieWriteClient.java:442)
> at
> org.apache.spark.api.java.JavaRDDLike$$anonfun$mapPartitionsWithIndex$1.apply(JavaRDDLike.scala:102)
> at
> org.apache.spark.api.java.JavaRDDLike$$anonfun$mapPartitionsWithIndex$1.apply(JavaRDDLike.scala:102)
> at
> org.apache.spark.rdd.RDD$$anonfun$mapPartitionsWithIndex$1$$anonfun$apply$26.apply(RDD.scala:843)
> at
> org.apache.spark.rdd.RDD$$anonfun$mapPartitionsWithIndex$1$$anonfun$apply$26.apply(RDD.scala:843)
> at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
> at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
> at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
> at org.apache.spark.rdd.RDD$$anonfun$8.apply(RDD.scala:336)
> at org.apache.spark.rdd.RDD$$anonfun$8.apply(RDD.scala:334)
> at
> org.apache.spark.storage.BlockManager$$anonfun$doPutIterator$1.apply(BlockManager.scala:973)
> at
> org.apache.spark.storage.BlockManager$$anonfun$doPutIterator$1.apply(BlockManager.scala:948)
> at org.apache.spark.storage.BlockManager.doPut(BlockManager.scala:888)
> at
> org.apache.spark.storage.BlockManager.doPutIterator(BlockManager.scala:948)
> at
> org.apache.spark.storage.BlockManager.getOrElseUpdate(BlockManager.scala:694)
> at org.apache.spark.rdd.RDD.getOrCompute(RDD.scala:334)
> at org.apache.spark.rdd.RDD.iterator(RDD.scala:285)
> at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
> at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
> at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
> at org.apache.spark.scheduler.Task.run(Task.scala:99)
> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:282)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[jira] [Updated] (HUDI-389) Updates sent to diff partition for a given key with Global Index
[
https://issues.apache.org/jira/browse/HUDI-389?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vinoth Chandar updated HUDI-389:
Status: Closed (was: Patch Available)
> Updates sent to diff partition for a given key with Global Index
> -
>
> Key: HUDI-389
> URL: https://issues.apache.org/jira/browse/HUDI-389
> Project: Apache Hudi (incubating)
> Issue Type: Bug
> Components: Index
>Reporter: sivabalan narayanan
>Assignee: sivabalan narayanan
>Priority: Major
> Labels: pull-request-available
> Fix For: 0.5.1
>
> Original Estimate: 48h
> Time Spent: 20m
> Remaining Estimate: 47h 40m
>
> Updates sent to diff partition for a given key with Global Index should
> succeed by updating the record under original partition. As of now, it throws
> exception.
> [https://github.com/apache/incubator-hudi/issues/1021]
>
>
> error log:
> {code:java}
> 14738 [Executor task launch worker-0] INFO
> com.uber.hoodie.common.table.timeline.HoodieActiveTimeline - Loaded instants
> java.util.stream.ReferencePipeline$Head@d02b1c7
> 14738 [Executor task launch worker-0] INFO
> com.uber.hoodie.common.table.view.AbstractTableFileSystemView - Building file
> system view for partition (2016/04/15)
> 14738 [Executor task launch worker-0] INFO
> com.uber.hoodie.common.table.view.AbstractTableFileSystemView - #files found
> in partition (2016/04/15) =0, Time taken =0
> 14738 [Executor task launch worker-0] INFO
> com.uber.hoodie.common.table.view.AbstractTableFileSystemView -
> addFilesToView: NumFiles=0, FileGroupsCreationTime=0, StoreTimeTaken=0
> 14738 [Executor task launch worker-0] INFO
> com.uber.hoodie.common.table.view.HoodieTableFileSystemView - Adding
> file-groups for partition :2016/04/15, #FileGroups=0
> 14738 [Executor task launch worker-0] INFO
> com.uber.hoodie.common.table.view.AbstractTableFileSystemView - Time to load
> partition (2016/04/15) =0
> 14754 [Executor task launch worker-0] ERROR
> com.uber.hoodie.table.HoodieCopyOnWriteTable - Error upserting bucketType
> UPDATE for partition :0
> java.util.NoSuchElementException: No value present
> at com.uber.hoodie.common.util.Option.get(Option.java:112)
> at com.uber.hoodie.io.HoodieMergeHandle.(HoodieMergeHandle.java:71)
> at
> com.uber.hoodie.table.HoodieCopyOnWriteTable.getUpdateHandle(HoodieCopyOnWriteTable.java:226)
> at
> com.uber.hoodie.table.HoodieCopyOnWriteTable.handleUpdate(HoodieCopyOnWriteTable.java:180)
> at
> com.uber.hoodie.table.HoodieCopyOnWriteTable.handleUpsertPartition(HoodieCopyOnWriteTable.java:263)
> at
> com.uber.hoodie.HoodieWriteClient.lambda$upsertRecordsInternal$7ef77fd$1(HoodieWriteClient.java:442)
> at
> org.apache.spark.api.java.JavaRDDLike$$anonfun$mapPartitionsWithIndex$1.apply(JavaRDDLike.scala:102)
> at
> org.apache.spark.api.java.JavaRDDLike$$anonfun$mapPartitionsWithIndex$1.apply(JavaRDDLike.scala:102)
> at
> org.apache.spark.rdd.RDD$$anonfun$mapPartitionsWithIndex$1$$anonfun$apply$26.apply(RDD.scala:843)
> at
> org.apache.spark.rdd.RDD$$anonfun$mapPartitionsWithIndex$1$$anonfun$apply$26.apply(RDD.scala:843)
> at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
> at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
> at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
> at org.apache.spark.rdd.RDD$$anonfun$8.apply(RDD.scala:336)
> at org.apache.spark.rdd.RDD$$anonfun$8.apply(RDD.scala:334)
> at
> org.apache.spark.storage.BlockManager$$anonfun$doPutIterator$1.apply(BlockManager.scala:973)
> at
> org.apache.spark.storage.BlockManager$$anonfun$doPutIterator$1.apply(BlockManager.scala:948)
> at org.apache.spark.storage.BlockManager.doPut(BlockManager.scala:888)
> at
> org.apache.spark.storage.BlockManager.doPutIterator(BlockManager.scala:948)
> at
> org.apache.spark.storage.BlockManager.getOrElseUpdate(BlockManager.scala:694)
> at org.apache.spark.rdd.RDD.getOrCompute(RDD.scala:334)
> at org.apache.spark.rdd.RDD.iterator(RDD.scala:285)
> at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
> at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
> at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
> at org.apache.spark.scheduler.Task.run(Task.scala:99)
> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:282)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at
[jira] [Updated] (HUDI-389) Updates sent to diff partition for a given key with Global Index
[
https://issues.apache.org/jira/browse/HUDI-389?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vinoth Chandar updated HUDI-389:
Status: Patch Available (was: In Progress)
> Updates sent to diff partition for a given key with Global Index
> -
>
> Key: HUDI-389
> URL: https://issues.apache.org/jira/browse/HUDI-389
> Project: Apache Hudi (incubating)
> Issue Type: Bug
> Components: Index
>Reporter: sivabalan narayanan
>Assignee: sivabalan narayanan
>Priority: Major
> Labels: pull-request-available
> Fix For: 0.5.1
>
> Original Estimate: 48h
> Time Spent: 10m
> Remaining Estimate: 47h 50m
>
> Updates sent to diff partition for a given key with Global Index should
> succeed by updating the record under original partition. As of now, it throws
> exception.
> [https://github.com/apache/incubator-hudi/issues/1021]
>
>
> error log:
> {code:java}
> 14738 [Executor task launch worker-0] INFO
> com.uber.hoodie.common.table.timeline.HoodieActiveTimeline - Loaded instants
> java.util.stream.ReferencePipeline$Head@d02b1c7
> 14738 [Executor task launch worker-0] INFO
> com.uber.hoodie.common.table.view.AbstractTableFileSystemView - Building file
> system view for partition (2016/04/15)
> 14738 [Executor task launch worker-0] INFO
> com.uber.hoodie.common.table.view.AbstractTableFileSystemView - #files found
> in partition (2016/04/15) =0, Time taken =0
> 14738 [Executor task launch worker-0] INFO
> com.uber.hoodie.common.table.view.AbstractTableFileSystemView -
> addFilesToView: NumFiles=0, FileGroupsCreationTime=0, StoreTimeTaken=0
> 14738 [Executor task launch worker-0] INFO
> com.uber.hoodie.common.table.view.HoodieTableFileSystemView - Adding
> file-groups for partition :2016/04/15, #FileGroups=0
> 14738 [Executor task launch worker-0] INFO
> com.uber.hoodie.common.table.view.AbstractTableFileSystemView - Time to load
> partition (2016/04/15) =0
> 14754 [Executor task launch worker-0] ERROR
> com.uber.hoodie.table.HoodieCopyOnWriteTable - Error upserting bucketType
> UPDATE for partition :0
> java.util.NoSuchElementException: No value present
> at com.uber.hoodie.common.util.Option.get(Option.java:112)
> at com.uber.hoodie.io.HoodieMergeHandle.(HoodieMergeHandle.java:71)
> at
> com.uber.hoodie.table.HoodieCopyOnWriteTable.getUpdateHandle(HoodieCopyOnWriteTable.java:226)
> at
> com.uber.hoodie.table.HoodieCopyOnWriteTable.handleUpdate(HoodieCopyOnWriteTable.java:180)
> at
> com.uber.hoodie.table.HoodieCopyOnWriteTable.handleUpsertPartition(HoodieCopyOnWriteTable.java:263)
> at
> com.uber.hoodie.HoodieWriteClient.lambda$upsertRecordsInternal$7ef77fd$1(HoodieWriteClient.java:442)
> at
> org.apache.spark.api.java.JavaRDDLike$$anonfun$mapPartitionsWithIndex$1.apply(JavaRDDLike.scala:102)
> at
> org.apache.spark.api.java.JavaRDDLike$$anonfun$mapPartitionsWithIndex$1.apply(JavaRDDLike.scala:102)
> at
> org.apache.spark.rdd.RDD$$anonfun$mapPartitionsWithIndex$1$$anonfun$apply$26.apply(RDD.scala:843)
> at
> org.apache.spark.rdd.RDD$$anonfun$mapPartitionsWithIndex$1$$anonfun$apply$26.apply(RDD.scala:843)
> at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
> at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
> at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
> at org.apache.spark.rdd.RDD$$anonfun$8.apply(RDD.scala:336)
> at org.apache.spark.rdd.RDD$$anonfun$8.apply(RDD.scala:334)
> at
> org.apache.spark.storage.BlockManager$$anonfun$doPutIterator$1.apply(BlockManager.scala:973)
> at
> org.apache.spark.storage.BlockManager$$anonfun$doPutIterator$1.apply(BlockManager.scala:948)
> at org.apache.spark.storage.BlockManager.doPut(BlockManager.scala:888)
> at
> org.apache.spark.storage.BlockManager.doPutIterator(BlockManager.scala:948)
> at
> org.apache.spark.storage.BlockManager.getOrElseUpdate(BlockManager.scala:694)
> at org.apache.spark.rdd.RDD.getOrCompute(RDD.scala:334)
> at org.apache.spark.rdd.RDD.iterator(RDD.scala:285)
> at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
> at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
> at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
> at org.apache.spark.scheduler.Task.run(Task.scala:99)
> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:282)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at
[jira] [Updated] (HUDI-389) Updates sent to diff partition for a given key with Global Index
[
https://issues.apache.org/jira/browse/HUDI-389?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vinoth Chandar updated HUDI-389:
Status: In Progress (was: Open)
> Updates sent to diff partition for a given key with Global Index
> -
>
> Key: HUDI-389
> URL: https://issues.apache.org/jira/browse/HUDI-389
> Project: Apache Hudi (incubating)
> Issue Type: Bug
> Components: Index
>Reporter: sivabalan narayanan
>Assignee: sivabalan narayanan
>Priority: Major
> Labels: pull-request-available
> Fix For: 0.5.1
>
> Original Estimate: 48h
> Time Spent: 10m
> Remaining Estimate: 47h 50m
>
> Updates sent to diff partition for a given key with Global Index should
> succeed by updating the record under original partition. As of now, it throws
> exception.
> [https://github.com/apache/incubator-hudi/issues/1021]
>
>
> error log:
> {code:java}
> 14738 [Executor task launch worker-0] INFO
> com.uber.hoodie.common.table.timeline.HoodieActiveTimeline - Loaded instants
> java.util.stream.ReferencePipeline$Head@d02b1c7
> 14738 [Executor task launch worker-0] INFO
> com.uber.hoodie.common.table.view.AbstractTableFileSystemView - Building file
> system view for partition (2016/04/15)
> 14738 [Executor task launch worker-0] INFO
> com.uber.hoodie.common.table.view.AbstractTableFileSystemView - #files found
> in partition (2016/04/15) =0, Time taken =0
> 14738 [Executor task launch worker-0] INFO
> com.uber.hoodie.common.table.view.AbstractTableFileSystemView -
> addFilesToView: NumFiles=0, FileGroupsCreationTime=0, StoreTimeTaken=0
> 14738 [Executor task launch worker-0] INFO
> com.uber.hoodie.common.table.view.HoodieTableFileSystemView - Adding
> file-groups for partition :2016/04/15, #FileGroups=0
> 14738 [Executor task launch worker-0] INFO
> com.uber.hoodie.common.table.view.AbstractTableFileSystemView - Time to load
> partition (2016/04/15) =0
> 14754 [Executor task launch worker-0] ERROR
> com.uber.hoodie.table.HoodieCopyOnWriteTable - Error upserting bucketType
> UPDATE for partition :0
> java.util.NoSuchElementException: No value present
> at com.uber.hoodie.common.util.Option.get(Option.java:112)
> at com.uber.hoodie.io.HoodieMergeHandle.(HoodieMergeHandle.java:71)
> at
> com.uber.hoodie.table.HoodieCopyOnWriteTable.getUpdateHandle(HoodieCopyOnWriteTable.java:226)
> at
> com.uber.hoodie.table.HoodieCopyOnWriteTable.handleUpdate(HoodieCopyOnWriteTable.java:180)
> at
> com.uber.hoodie.table.HoodieCopyOnWriteTable.handleUpsertPartition(HoodieCopyOnWriteTable.java:263)
> at
> com.uber.hoodie.HoodieWriteClient.lambda$upsertRecordsInternal$7ef77fd$1(HoodieWriteClient.java:442)
> at
> org.apache.spark.api.java.JavaRDDLike$$anonfun$mapPartitionsWithIndex$1.apply(JavaRDDLike.scala:102)
> at
> org.apache.spark.api.java.JavaRDDLike$$anonfun$mapPartitionsWithIndex$1.apply(JavaRDDLike.scala:102)
> at
> org.apache.spark.rdd.RDD$$anonfun$mapPartitionsWithIndex$1$$anonfun$apply$26.apply(RDD.scala:843)
> at
> org.apache.spark.rdd.RDD$$anonfun$mapPartitionsWithIndex$1$$anonfun$apply$26.apply(RDD.scala:843)
> at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
> at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
> at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
> at org.apache.spark.rdd.RDD$$anonfun$8.apply(RDD.scala:336)
> at org.apache.spark.rdd.RDD$$anonfun$8.apply(RDD.scala:334)
> at
> org.apache.spark.storage.BlockManager$$anonfun$doPutIterator$1.apply(BlockManager.scala:973)
> at
> org.apache.spark.storage.BlockManager$$anonfun$doPutIterator$1.apply(BlockManager.scala:948)
> at org.apache.spark.storage.BlockManager.doPut(BlockManager.scala:888)
> at
> org.apache.spark.storage.BlockManager.doPutIterator(BlockManager.scala:948)
> at
> org.apache.spark.storage.BlockManager.getOrElseUpdate(BlockManager.scala:694)
> at org.apache.spark.rdd.RDD.getOrCompute(RDD.scala:334)
> at org.apache.spark.rdd.RDD.iterator(RDD.scala:285)
> at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
> at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
> at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
> at org.apache.spark.scheduler.Task.run(Task.scala:99)
> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:282)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at
[jira] [Updated] (HUDI-389) Updates sent to diff partition for a given key with Global Index
[
https://issues.apache.org/jira/browse/HUDI-389?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated HUDI-389:
Labels: pull-request-available (was: )
> Updates sent to diff partition for a given key with Global Index
> -
>
> Key: HUDI-389
> URL: https://issues.apache.org/jira/browse/HUDI-389
> Project: Apache Hudi (incubating)
> Issue Type: Bug
> Components: Index
>Reporter: sivabalan narayanan
>Assignee: sivabalan narayanan
>Priority: Major
> Labels: pull-request-available
> Fix For: 0.5.1
>
> Original Estimate: 48h
> Remaining Estimate: 48h
>
> Updates sent to diff partition for a given key with Global Index should
> succeed by updating the record under original partition. As of now, it throws
> exception.
> [https://github.com/apache/incubator-hudi/issues/1021]
>
>
> error log:
> {code:java}
> 14738 [Executor task launch worker-0] INFO
> com.uber.hoodie.common.table.timeline.HoodieActiveTimeline - Loaded instants
> java.util.stream.ReferencePipeline$Head@d02b1c7
> 14738 [Executor task launch worker-0] INFO
> com.uber.hoodie.common.table.view.AbstractTableFileSystemView - Building file
> system view for partition (2016/04/15)
> 14738 [Executor task launch worker-0] INFO
> com.uber.hoodie.common.table.view.AbstractTableFileSystemView - #files found
> in partition (2016/04/15) =0, Time taken =0
> 14738 [Executor task launch worker-0] INFO
> com.uber.hoodie.common.table.view.AbstractTableFileSystemView -
> addFilesToView: NumFiles=0, FileGroupsCreationTime=0, StoreTimeTaken=0
> 14738 [Executor task launch worker-0] INFO
> com.uber.hoodie.common.table.view.HoodieTableFileSystemView - Adding
> file-groups for partition :2016/04/15, #FileGroups=0
> 14738 [Executor task launch worker-0] INFO
> com.uber.hoodie.common.table.view.AbstractTableFileSystemView - Time to load
> partition (2016/04/15) =0
> 14754 [Executor task launch worker-0] ERROR
> com.uber.hoodie.table.HoodieCopyOnWriteTable - Error upserting bucketType
> UPDATE for partition :0
> java.util.NoSuchElementException: No value present
> at com.uber.hoodie.common.util.Option.get(Option.java:112)
> at com.uber.hoodie.io.HoodieMergeHandle.(HoodieMergeHandle.java:71)
> at
> com.uber.hoodie.table.HoodieCopyOnWriteTable.getUpdateHandle(HoodieCopyOnWriteTable.java:226)
> at
> com.uber.hoodie.table.HoodieCopyOnWriteTable.handleUpdate(HoodieCopyOnWriteTable.java:180)
> at
> com.uber.hoodie.table.HoodieCopyOnWriteTable.handleUpsertPartition(HoodieCopyOnWriteTable.java:263)
> at
> com.uber.hoodie.HoodieWriteClient.lambda$upsertRecordsInternal$7ef77fd$1(HoodieWriteClient.java:442)
> at
> org.apache.spark.api.java.JavaRDDLike$$anonfun$mapPartitionsWithIndex$1.apply(JavaRDDLike.scala:102)
> at
> org.apache.spark.api.java.JavaRDDLike$$anonfun$mapPartitionsWithIndex$1.apply(JavaRDDLike.scala:102)
> at
> org.apache.spark.rdd.RDD$$anonfun$mapPartitionsWithIndex$1$$anonfun$apply$26.apply(RDD.scala:843)
> at
> org.apache.spark.rdd.RDD$$anonfun$mapPartitionsWithIndex$1$$anonfun$apply$26.apply(RDD.scala:843)
> at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
> at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
> at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
> at org.apache.spark.rdd.RDD$$anonfun$8.apply(RDD.scala:336)
> at org.apache.spark.rdd.RDD$$anonfun$8.apply(RDD.scala:334)
> at
> org.apache.spark.storage.BlockManager$$anonfun$doPutIterator$1.apply(BlockManager.scala:973)
> at
> org.apache.spark.storage.BlockManager$$anonfun$doPutIterator$1.apply(BlockManager.scala:948)
> at org.apache.spark.storage.BlockManager.doPut(BlockManager.scala:888)
> at
> org.apache.spark.storage.BlockManager.doPutIterator(BlockManager.scala:948)
> at
> org.apache.spark.storage.BlockManager.getOrElseUpdate(BlockManager.scala:694)
> at org.apache.spark.rdd.RDD.getOrCompute(RDD.scala:334)
> at org.apache.spark.rdd.RDD.iterator(RDD.scala:285)
> at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
> at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
> at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
> at org.apache.spark.scheduler.Task.run(Task.scala:99)
> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:282)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
>
[jira] [Updated] (HUDI-389) Updates sent to diff partition for a given key with Global Index
[
https://issues.apache.org/jira/browse/HUDI-389?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vinoth Chandar updated HUDI-389:
Description:
Updates sent to diff partition for a given key with Global Index should succeed
by updating the record under original partition. As of now, it throws
exception.
[https://github.com/apache/incubator-hudi/issues/1021]
error log:
{code:java}
14738 [Executor task launch worker-0] INFO
com.uber.hoodie.common.table.timeline.HoodieActiveTimeline - Loaded instants
java.util.stream.ReferencePipeline$Head@d02b1c7
14738 [Executor task launch worker-0] INFO
com.uber.hoodie.common.table.view.AbstractTableFileSystemView - Building file
system view for partition (2016/04/15)
14738 [Executor task launch worker-0] INFO
com.uber.hoodie.common.table.view.AbstractTableFileSystemView - #files found in
partition (2016/04/15) =0, Time taken =0
14738 [Executor task launch worker-0] INFO
com.uber.hoodie.common.table.view.AbstractTableFileSystemView - addFilesToView:
NumFiles=0, FileGroupsCreationTime=0, StoreTimeTaken=0
14738 [Executor task launch worker-0] INFO
com.uber.hoodie.common.table.view.HoodieTableFileSystemView - Adding
file-groups for partition :2016/04/15, #FileGroups=0
14738 [Executor task launch worker-0] INFO
com.uber.hoodie.common.table.view.AbstractTableFileSystemView - Time to load
partition (2016/04/15) =0
14754 [Executor task launch worker-0] ERROR
com.uber.hoodie.table.HoodieCopyOnWriteTable - Error upserting bucketType
UPDATE for partition :0
java.util.NoSuchElementException: No value present
at com.uber.hoodie.common.util.Option.get(Option.java:112)
at com.uber.hoodie.io.HoodieMergeHandle.(HoodieMergeHandle.java:71)
at
com.uber.hoodie.table.HoodieCopyOnWriteTable.getUpdateHandle(HoodieCopyOnWriteTable.java:226)
at
com.uber.hoodie.table.HoodieCopyOnWriteTable.handleUpdate(HoodieCopyOnWriteTable.java:180)
at
com.uber.hoodie.table.HoodieCopyOnWriteTable.handleUpsertPartition(HoodieCopyOnWriteTable.java:263)
at
com.uber.hoodie.HoodieWriteClient.lambda$upsertRecordsInternal$7ef77fd$1(HoodieWriteClient.java:442)
at
org.apache.spark.api.java.JavaRDDLike$$anonfun$mapPartitionsWithIndex$1.apply(JavaRDDLike.scala:102)
at
org.apache.spark.api.java.JavaRDDLike$$anonfun$mapPartitionsWithIndex$1.apply(JavaRDDLike.scala:102)
at
org.apache.spark.rdd.RDD$$anonfun$mapPartitionsWithIndex$1$$anonfun$apply$26.apply(RDD.scala:843)
at
org.apache.spark.rdd.RDD$$anonfun$mapPartitionsWithIndex$1$$anonfun$apply$26.apply(RDD.scala:843)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
at org.apache.spark.rdd.RDD$$anonfun$8.apply(RDD.scala:336)
at org.apache.spark.rdd.RDD$$anonfun$8.apply(RDD.scala:334)
at
org.apache.spark.storage.BlockManager$$anonfun$doPutIterator$1.apply(BlockManager.scala:973)
at
org.apache.spark.storage.BlockManager$$anonfun$doPutIterator$1.apply(BlockManager.scala:948)
at org.apache.spark.storage.BlockManager.doPut(BlockManager.scala:888)
at org.apache.spark.storage.BlockManager.doPutIterator(BlockManager.scala:948)
at
org.apache.spark.storage.BlockManager.getOrElseUpdate(BlockManager.scala:694)
at org.apache.spark.rdd.RDD.getOrCompute(RDD.scala:334)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:285)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
at org.apache.spark.scheduler.Task.run(Task.scala:99)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:282)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
{code}
Refer [here|[https://github.com/apache/incubator-hudi/issues/1021]] for some
context.
was:
Updates sent to diff partition for a given key with Global Index should succeed
by updating the record under original partition. As of now, it throws
exception.
error log:
14738 [Executor task launch worker-0] INFO
com.uber.hoodie.common.table.timeline.HoodieActiveTimeline - Loaded instants
java.util.stream.ReferencePipeline$Head@d02b1c7
14738 [Executor task launch worker-0] INFO
com.uber.hoodie.common.table.view.AbstractTableFileSystemView - Building file
system view for partition (2016/04/15)
14738 [Executor task launch worker-0] INFO
com.uber.hoodie.common.table.view.AbstractTableFileSystemView - #files found in
partition
[jira] [Updated] (HUDI-389) Updates sent to diff partition for a given key with Global Index
[
https://issues.apache.org/jira/browse/HUDI-389?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vinoth Chandar updated HUDI-389:
Fix Version/s: 0.5.1
> Updates sent to diff partition for a given key with Global Index
> -
>
> Key: HUDI-389
> URL: https://issues.apache.org/jira/browse/HUDI-389
> Project: Apache Hudi (incubating)
> Issue Type: Bug
> Components: Index
>Reporter: sivabalan narayanan
>Assignee: sivabalan narayanan
>Priority: Major
> Fix For: 0.5.1
>
> Original Estimate: 48h
> Remaining Estimate: 48h
>
> Updates sent to diff partition for a given key with Global Index should
> succeed by updating the record under original partition. As of now, it throws
> exception.
> [https://github.com/apache/incubator-hudi/issues/1021]
>
>
> error log:
> {code:java}
> 14738 [Executor task launch worker-0] INFO
> com.uber.hoodie.common.table.timeline.HoodieActiveTimeline - Loaded instants
> java.util.stream.ReferencePipeline$Head@d02b1c7
> 14738 [Executor task launch worker-0] INFO
> com.uber.hoodie.common.table.view.AbstractTableFileSystemView - Building file
> system view for partition (2016/04/15)
> 14738 [Executor task launch worker-0] INFO
> com.uber.hoodie.common.table.view.AbstractTableFileSystemView - #files found
> in partition (2016/04/15) =0, Time taken =0
> 14738 [Executor task launch worker-0] INFO
> com.uber.hoodie.common.table.view.AbstractTableFileSystemView -
> addFilesToView: NumFiles=0, FileGroupsCreationTime=0, StoreTimeTaken=0
> 14738 [Executor task launch worker-0] INFO
> com.uber.hoodie.common.table.view.HoodieTableFileSystemView - Adding
> file-groups for partition :2016/04/15, #FileGroups=0
> 14738 [Executor task launch worker-0] INFO
> com.uber.hoodie.common.table.view.AbstractTableFileSystemView - Time to load
> partition (2016/04/15) =0
> 14754 [Executor task launch worker-0] ERROR
> com.uber.hoodie.table.HoodieCopyOnWriteTable - Error upserting bucketType
> UPDATE for partition :0
> java.util.NoSuchElementException: No value present
> at com.uber.hoodie.common.util.Option.get(Option.java:112)
> at com.uber.hoodie.io.HoodieMergeHandle.(HoodieMergeHandle.java:71)
> at
> com.uber.hoodie.table.HoodieCopyOnWriteTable.getUpdateHandle(HoodieCopyOnWriteTable.java:226)
> at
> com.uber.hoodie.table.HoodieCopyOnWriteTable.handleUpdate(HoodieCopyOnWriteTable.java:180)
> at
> com.uber.hoodie.table.HoodieCopyOnWriteTable.handleUpsertPartition(HoodieCopyOnWriteTable.java:263)
> at
> com.uber.hoodie.HoodieWriteClient.lambda$upsertRecordsInternal$7ef77fd$1(HoodieWriteClient.java:442)
> at
> org.apache.spark.api.java.JavaRDDLike$$anonfun$mapPartitionsWithIndex$1.apply(JavaRDDLike.scala:102)
> at
> org.apache.spark.api.java.JavaRDDLike$$anonfun$mapPartitionsWithIndex$1.apply(JavaRDDLike.scala:102)
> at
> org.apache.spark.rdd.RDD$$anonfun$mapPartitionsWithIndex$1$$anonfun$apply$26.apply(RDD.scala:843)
> at
> org.apache.spark.rdd.RDD$$anonfun$mapPartitionsWithIndex$1$$anonfun$apply$26.apply(RDD.scala:843)
> at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
> at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
> at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
> at org.apache.spark.rdd.RDD$$anonfun$8.apply(RDD.scala:336)
> at org.apache.spark.rdd.RDD$$anonfun$8.apply(RDD.scala:334)
> at
> org.apache.spark.storage.BlockManager$$anonfun$doPutIterator$1.apply(BlockManager.scala:973)
> at
> org.apache.spark.storage.BlockManager$$anonfun$doPutIterator$1.apply(BlockManager.scala:948)
> at org.apache.spark.storage.BlockManager.doPut(BlockManager.scala:888)
> at
> org.apache.spark.storage.BlockManager.doPutIterator(BlockManager.scala:948)
> at
> org.apache.spark.storage.BlockManager.getOrElseUpdate(BlockManager.scala:694)
> at org.apache.spark.rdd.RDD.getOrCompute(RDD.scala:334)
> at org.apache.spark.rdd.RDD.iterator(RDD.scala:285)
> at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
> at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
> at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
> at org.apache.spark.scheduler.Task.run(Task.scala:99)
> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:282)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> {code}
> Refer
