[jira] [Commented] (CRUNCH-606) Create a KafkaSource

[ https://issues.apache.org/jira/browse/CRUNCH-606?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15289096#comment-15289096 ] Andrew Olson commented on CRUNCH-606: - +1 > Create a KafkaSource > --

[jira] [Commented] (CRUNCH-621) Kafka no data to consume could get stuck

[ https://issues.apache.org/jira/browse/CRUNCH-621?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15536158#comment-15536158 ] Andrew Olson commented on CRUNCH-621: - This patch looks good. +1 > Kafka no data to c

[jira] [Commented] (CRUNCH-620) Reduce "isn't a known config" warnings by slimming down ConsumerConfig properties

[ https://issues.apache.org/jira/browse/CRUNCH-620?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15536209#comment-15536209 ] Andrew Olson commented on CRUNCH-620: - This patch looks good. +1 > Reduce "isn't a kn

[jira] [Commented] (CRUNCH-630) KafkaRecordReader keeps retrying to poll data when the offset is reset to latest offset

[ https://issues.apache.org/jira/browse/CRUNCH-630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15742438#comment-15742438 ] Andrew Olson commented on CRUNCH-630: - The current workaround for this bug is to set a

[jira] [Commented] (CRUNCH-630) KafkaRecordReader keeps retrying to poll data when the offset is reset to latest offset

[ https://issues.apache.org/jira/browse/CRUNCH-630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15795661#comment-15795661 ] Andrew Olson commented on CRUNCH-630: - Yes that looks good. > KafkaRecordReader keeps

[jira] [Created] (CRUNCH-654) KafkaSource should use new Kafka Consumer API instead of Simple Consumer

Andrew Olson created CRUNCH-654: --- Summary: KafkaSource should use new Kafka Consumer API instead of Simple Consumer Key: CRUNCH-654 URL: https://issues.apache.org/jira/browse/CRUNCH-654 Project: Crunch

[jira] [Updated] (CRUNCH-667) Documentation should mention that cogrouping requires a common type family

[ https://issues.apache.org/jira/browse/CRUNCH-667?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Olson updated CRUNCH-667: Description: Attempting to cogroup an Avro-based table with a Writable-based table produces an exce

[jira] [Created] (CRUNCH-667) Documentation should mention that cogrouping requires a common type family

Andrew Olson created CRUNCH-667: --- Summary: Documentation should mention that cogrouping requires a common type family Key: CRUNCH-667 URL: https://issues.apache.org/jira/browse/CRUNCH-667 Project: Crunc

[jira] [Commented] (CRUNCH-660) FileTargetImpl uses Distcp vs FileUtils.copy

[ https://issues.apache.org/jira/browse/CRUNCH-660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16740686#comment-16740686 ] Andrew Olson commented on CRUNCH-660: - I've implemented a basic version of this solut

[jira] [Created] (CRUNCH-675) HFileTarget should use DistCp when source and destination are in different filesystems

Andrew Olson created CRUNCH-675: --- Summary: HFileTarget should use DistCp when source and destination are in different filesystems Key: CRUNCH-675 URL: https://issues.apache.org/jira/browse/CRUNCH-675 Pr

[jira] [Commented] (CRUNCH-619) Run on HBase 2

[ https://issues.apache.org/jira/browse/CRUNCH-619?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16742190#comment-16742190 ] Andrew Olson commented on CRUNCH-619: - One helpful bug fix contained in this uplift w

[jira] [Commented] (CRUNCH-660) FileTargetImpl uses Distcp vs FileUtils.copy

[ https://issues.apache.org/jira/browse/CRUNCH-660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16743122#comment-16743122 ] Andrew Olson commented on CRUNCH-660: - On a side note, even DistCp can be pretty seve

[jira] [Commented] (CRUNCH-660) FileTargetImpl uses Distcp vs FileUtils.copy

[ https://issues.apache.org/jira/browse/CRUNCH-660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16750326#comment-16750326 ] Andrew Olson commented on CRUNCH-660: - Pull request, https://github.com/apache/crunch

[jira] [Commented] (CRUNCH-675) HFileTarget should use DistCp when source and destination are in different filesystems

[ https://issues.apache.org/jira/browse/CRUNCH-675?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16750327#comment-16750327 ] Andrew Olson commented on CRUNCH-675: - Pull request, https://github.com/apache/crunch

[jira] [Updated] (CRUNCH-675) HFileTarget should use DistCp when source and destination are in different filesystems

[ https://issues.apache.org/jira/browse/CRUNCH-675?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Olson updated CRUNCH-675: Fix Version/s: (was: 0.15.0) 1.0.0 > HFileTarget should use DistCp when sour

[jira] [Commented] (CRUNCH-658) Add a way to skip the getSize checks for Sources from object stores

[ https://issues.apache.org/jira/browse/CRUNCH-658?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16772200#comment-16772200 ] Andrew Olson commented on CRUNCH-658: - Looks like skipping the getLastModifiedAt for

[jira] [Commented] (CRUNCH-658) Add a way to skip the getSize checks for Sources from object stores

[ https://issues.apache.org/jira/browse/CRUNCH-658?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16772417#comment-16772417 ] Andrew Olson commented on CRUNCH-658: - Created CRUNCH-678 for addressing getLastModif

[jira] [Created] (CRUNCH-678) Avoid unnecessary retrieval of last modified time

Andrew Olson created CRUNCH-678: --- Summary: Avoid unnecessary retrieval of last modified time Key: CRUNCH-678 URL: https://issues.apache.org/jira/browse/CRUNCH-678 Project: Crunch Issue Type: Im

[jira] [Commented] (CRUNCH-678) Avoid unnecessary retrieval of last modified time

[ https://issues.apache.org/jira/browse/CRUNCH-678?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16772424#comment-16772424 ] Andrew Olson commented on CRUNCH-678: - [~jwills] I'm working on it, will have a pull

[jira] [Commented] (CRUNCH-678) Avoid unnecessary retrieval of last modified time

[ https://issues.apache.org/jira/browse/CRUNCH-678?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16772440#comment-16772440 ] Andrew Olson commented on CRUNCH-678: - Here's a pull request: https://github.com/apac

[jira] [Commented] (CRUNCH-658) Add a way to skip the getSize checks for Sources from object stores

[ https://issues.apache.org/jira/browse/CRUNCH-658?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16773492#comment-16773492 ] Andrew Olson commented on CRUNCH-658: - The getSize() == 0 check for "Materializing an

[jira] [Commented] (CRUNCH-679) Improvements for usage of DistCp

[ https://issues.apache.org/jira/browse/CRUNCH-679?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16778424#comment-16778424 ] Andrew Olson commented on CRUNCH-679: - I will open a pull request for these changes l

[jira] [Created] (CRUNCH-679) Improvements for usage of DistCp

Andrew Olson created CRUNCH-679: --- Summary: Improvements for usage of DistCp Key: CRUNCH-679 URL: https://issues.apache.org/jira/browse/CRUNCH-679 Project: Crunch Issue Type: Improvement

[jira] [Commented] (CRUNCH-679) Improvements for usage of DistCp

[ https://issues.apache.org/jira/browse/CRUNCH-679?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16778487#comment-16778487 ] Andrew Olson commented on CRUNCH-679: - Pull request, https://github.com/apache/crunch

[jira] [Created] (CRUNCH-680) Kafka Source should split very large partitions

Andrew Olson created CRUNCH-680: --- Summary: Kafka Source should split very large partitions Key: CRUNCH-680 URL: https://issues.apache.org/jira/browse/CRUNCH-680 Project: Crunch Issue Type: Impr

[jira] [Commented] (CRUNCH-680) Kafka Source should split very large partitions

[ https://issues.apache.org/jira/browse/CRUNCH-680?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16778672#comment-16778672 ] Andrew Olson commented on CRUNCH-680: - Pull request, https://github.com/apache/crunch

[jira] [Created] (CRUNCH-682) Update HFileUtils to support targets and sources in remote filesystems

Andrew Olson created CRUNCH-682: --- Summary: Update HFileUtils to support targets and sources in remote filesystems Key: CRUNCH-682 URL: https://issues.apache.org/jira/browse/CRUNCH-682 Project: Crunch

[jira] [Commented] (CRUNCH-681) HFileUtils. writeToHFilesForIncrementalLoad() should accept a FileSystem parameter

[ https://issues.apache.org/jira/browse/CRUNCH-681?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16821144#comment-16821144 ] Andrew Olson commented on CRUNCH-681: - This change also needs to be made for writePut

[jira] [Closed] (CRUNCH-682) Update HFileUtils to support targets and sources in remote filesystems

[ https://issues.apache.org/jira/browse/CRUNCH-682?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Olson closed CRUNCH-682. --- Resolution: Duplicate > Update HFileUtils to support targets and sources in remote filesystems >

[jira] [Commented] (CRUNCH-682) Update HFileUtils to support targets and sources in remote filesystems

[ https://issues.apache.org/jira/browse/CRUNCH-682?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16821146#comment-16821146 ] Andrew Olson commented on CRUNCH-682: - Closing as duplicate of CRUNCH-681 > Update H

[jira] [Commented] (CRUNCH-681) HFileUtils. writeToHFilesForIncrementalLoad() should accept a FileSystem parameter

[ https://issues.apache.org/jira/browse/CRUNCH-681?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16821147#comment-16821147 ] Andrew Olson commented on CRUNCH-681: - The component for this issue should be "IO".

[jira] [Commented] (CRUNCH-681) HFileUtils. writeToHFilesForIncrementalLoad() should accept a FileSystem parameter

[ https://issues.apache.org/jira/browse/CRUNCH-681?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16821224#comment-16821224 ] Andrew Olson commented on CRUNCH-681: - Pull request, https://github.com/apache/crunch

[jira] [Commented] (CRUNCH-684) [crunch-hbase] HbaseTarget getting ignored even if configuration is different

[ https://issues.apache.org/jira/browse/CRUNCH-684?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16831145#comment-16831145 ] Andrew Olson commented on CRUNCH-684: - Yes I think your proposal makes sense. A quick

[jira] [Commented] (CRUNCH-684) [crunch-hbase] HbaseTarget getting ignored even if configuration is different

[ https://issues.apache.org/jira/browse/CRUNCH-684?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16831248#comment-16831248 ] Andrew Olson commented on CRUNCH-684: - I will submit a pull request for this correcti

[jira] [Commented] (CRUNCH-684) [crunch-hbase] HbaseTarget getting ignored even if configuration is different

[ https://issues.apache.org/jira/browse/CRUNCH-684?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16831262#comment-16831262 ] Andrew Olson commented on CRUNCH-684: - Pull request, https://github.com/apache/crunch

[jira] [Commented] (CRUNCH-685) Limit Target#fileSystem(FileSystem) to only apply filesystem specific configurations to the FormatBundle

[ https://issues.apache.org/jira/browse/CRUNCH-685?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16836646#comment-16836646 ] Andrew Olson commented on CRUNCH-685: - This appears similar to a problem we encounter

[jira] [Updated] (CRUNCH-679) Improvements for usage of DistCp

[ https://issues.apache.org/jira/browse/CRUNCH-679?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Olson updated CRUNCH-679: Description: As a follow-up to CRUNCH-660 and CRUNCH-675, a handful of corrections and improvement

[jira] [Updated] (CRUNCH-679) Improvements for usage of DistCp

[ https://issues.apache.org/jira/browse/CRUNCH-679?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Olson updated CRUNCH-679: Description: As a follow-up to CRUNCH-660 and CRUNCH-675, a handful of corrections and improvement

[jira] [Created] (CRUNCH-688) HFile node affinity only works with default namespace HBase tables

Andrew Olson created CRUNCH-688: --- Summary: HFile node affinity only works with default namespace HBase tables Key: CRUNCH-688 URL: https://issues.apache.org/jira/browse/CRUNCH-688 Project: Crunch

[jira] [Commented] (CRUNCH-644) Set HDFS node affinity on created HFiles to improve locality

[ https://issues.apache.org/jira/browse/CRUNCH-644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16899072#comment-16899072 ] Andrew Olson commented on CRUNCH-644: - A problem was found with this when using a non

[jira] [Commented] (CRUNCH-688) HFile node affinity only works with default namespace HBase tables

[ https://issues.apache.org/jira/browse/CRUNCH-688?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16899227#comment-16899227 ] Andrew Olson commented on CRUNCH-688: - Pull request: https://github.com/apache/crunch

[jira] [Updated] (CRUNCH-679) Improvements for usage of DistCp

[ https://issues.apache.org/jira/browse/CRUNCH-679?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Olson updated CRUNCH-679: Fix Version/s: 1.0.0 > Improvements for usage of DistCp > > >

[jira] [Commented] (CRUNCH-686) Fix FormatBundle to hide redacted properties

[ https://issues.apache.org/jira/browse/CRUNCH-686?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16979388#comment-16979388 ] Andrew Olson commented on CRUNCH-686: - This should have fix version of 1.0.0 > Fix F

[jira] [Commented] (CRUNCH-633) Remove the commons-httpclient:commons-httpclient dependency

[ https://issues.apache.org/jira/browse/CRUNCH-633?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16979390#comment-16979390 ] Andrew Olson commented on CRUNCH-633: - This should have fix version of 1.0.0 > Remov

[jira] [Created] (CRUNCH-695) NullPointerException in RegionLocationTable

Andrew Olson created CRUNCH-695: --- Summary: NullPointerException in RegionLocationTable Key: CRUNCH-695 URL: https://issues.apache.org/jira/browse/CRUNCH-695 Project: Crunch Issue Type: Bug

[jira] [Commented] (CRUNCH-695) NullPointerException in RegionLocationTable

[ https://issues.apache.org/jira/browse/CRUNCH-695?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17066787#comment-17066787 ] Andrew Olson commented on CRUNCH-695: - Appears the region info may potentially be nul

[jira] [Commented] (CRUNCH-695) NullPointerException in RegionLocationTable

[ https://issues.apache.org/jira/browse/CRUNCH-695?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17066788#comment-17066788 ] Andrew Olson commented on CRUNCH-695: - Pull request, [https://github.com/apache/crunc

[jira] [Updated] (CRUNCH-695) NullPointerException in RegionLocationTable

[ https://issues.apache.org/jira/browse/CRUNCH-695?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Olson updated CRUNCH-695: Component/s: IO > NullPointerException in RegionLocationTable > ---

[jira] [Updated] (CRUNCH-695) NullPointerException in RegionLocationTable

[ https://issues.apache.org/jira/browse/CRUNCH-695?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Olson updated CRUNCH-695: Fix Version/s: 1.1.0 > NullPointerException in RegionLocationTable > --

[jira] [Created] (CRUNCH-696) EOFException in App Master from FormatBundle#fromSerialized

Andrew Olson created CRUNCH-696: --- Summary: EOFException in App Master from FormatBundle#fromSerialized Key: CRUNCH-696 URL: https://issues.apache.org/jira/browse/CRUNCH-696 Project: Crunch Iss

[jira] [Created] (CRUNCH-698) Avro DataFileReader creation can hang

Andrew Olson created CRUNCH-698: --- Summary: Avro DataFileReader creation can hang Key: CRUNCH-698 URL: https://issues.apache.org/jira/browse/CRUNCH-698 Project: Crunch Issue Type: Bug

[jira] [Updated] (CRUNCH-698) Avro DataFileReader creation can hang

[ https://issues.apache.org/jira/browse/CRUNCH-698?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Olson updated CRUNCH-698: Component/s: IO > Avro DataFileReader creation can hang > - > >

[jira] [Resolved] (CRUNCH-698) Avro DataFileReader creation can hang

[ https://issues.apache.org/jira/browse/CRUNCH-698?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Olson resolved CRUNCH-698. - Fix Version/s: 1.1.0 Resolution: Fixed Pull request has been merged. > Avro DataFileReade

[jira] [Updated] (CRUNCH-698) Avro DataFileReader creation can hang

[ https://issues.apache.org/jira/browse/CRUNCH-698?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Olson updated CRUNCH-698: Description: A severe Avro bug [AVRO-2944|https://issues.apache.org/jira/browse/AVRO-2944] was rec