[GitHub] apex-malhar pull request #603: APEXMALHAR-2476-Fix-tupleSeperator-override.
GitHub user yogidevendra reopened a pull request: https://github.com/apache/apex-malhar/pull/603 APEXMALHAR-2476-Fix-tupleSeperator-override. You can merge this pull request into a Git repository by running: $ git pull https://github.com/yogidevendra/apex-malhar APEXMALHAR-2476-tupleSeparator-GenericFileOutputOperator Alternatively you can review and apply these changes as the patch at: https://github.com/apache/apex-malhar/pull/603.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #603 commit c26e07e37aaaeb533e50aecdc1d07ac850e75640 Author: yogidevendra <yogideven...@apache.org> Date: 2017-04-06T22:40:07Z APEXMALHAR-2476-Fix-tupleSeperator-override. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] apex-malhar pull request #603: APEXMALHAR-2476-Fix-tupleSeperator-override.
Github user yogidevendra closed the pull request at: https://github.com/apache/apex-malhar/pull/603 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] apex-malhar pull request #603: APEXMALHAR-2476-Fix-tupleSeperator-override.
GitHub user yogidevendra opened a pull request: https://github.com/apache/apex-malhar/pull/603 APEXMALHAR-2476-Fix-tupleSeperator-override. You can merge this pull request into a Git repository by running: $ git pull https://github.com/yogidevendra/apex-malhar APEXMALHAR-2476-tupleSeparator-GenericFileOutputOperator Alternatively you can review and apply these changes as the patch at: https://github.com/apache/apex-malhar/pull/603.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #603 commit c26e07e37aaaeb533e50aecdc1d07ac850e75640 Author: yogidevendra <yogideven...@apache.org> Date: 2017-04-06T22:40:07Z APEXMALHAR-2476-Fix-tupleSeperator-override. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] apex-malhar pull request #542: APEXMALHAR-2369 S3 output-tuple-based
GitHub user yogidevendra opened a pull request: https://github.com/apache/apex-malhar/pull/542 APEXMALHAR-2369 S3 output-tuple-based You can merge this pull request into a Git repository by running: $ git pull https://github.com/yogidevendra/apex-malhar APEXMALHAR-2369-s3-output-tuple-based-PR Alternatively you can review and apply these changes as the patch at: https://github.com/apache/apex-malhar/pull/542.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #542 commit affa5877516b2319d21789e810d39fdcdf6f9862 Author: yogidevendra <yogideven...@apache.org> Date: 2016-12-20T10:08:51Z APEXMALHAR-2369 S3 output-tuple-based --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] apex-malhar pull request #439: APEXMALHAR-2272 : Fixed sequentialFileRead on...
Github user yogidevendra closed the pull request at: https://github.com/apache/apex-malhar/pull/439 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] apex-malhar pull request #439: APEXMALHAR-2272 : Fixed sequentialFileRead on...
GitHub user yogidevendra reopened a pull request: https://github.com/apache/apex-malhar/pull/439 APEXMALHAR-2272 : Fixed sequentialFileRead on FSInputModule You can merge this pull request into a Git repository by running: $ git pull https://github.com/yogidevendra/apex-malhar APEXMALHAR-2272-sequencialFileRead Alternatively you can review and apply these changes as the patch at: https://github.com/apache/apex-malhar/pull/439.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #439 commit 58be0b5986e653682bb3c3421dc7f2cb748babef Author: yogidevendra <yogideven...@apache.org> Date: 2016-10-01T02:49:56Z APEXMALHAR-2272 : Fixed sequencialFileRead on FSInputModule 1. Fixed StreamCodec to route all blocks for a file to same partition. 2. Fixed spelling for sequentialFileRead in the code, javadocs. 3.marked FSInputModule as evolving 4.added japicmp exclusion for FSInputModule --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] apex-malhar pull request #439: APEXMALHAR-2272 : Fixed sequentialFileRead on...
Github user yogidevendra closed the pull request at: https://github.com/apache/apex-malhar/pull/439 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] apex-malhar pull request #409: APEXMALHAR-2234 FilterOperator documentation
Github user yogidevendra closed the pull request at: https://github.com/apache/apex-malhar/pull/409 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] apex-malhar pull request #409: APEXMALHAR-2234 FilterOperator documentation
GitHub user yogidevendra reopened a pull request: https://github.com/apache/apex-malhar/pull/409 APEXMALHAR-2234 FilterOperator documentation You can merge this pull request into a Git repository by running: $ git pull https://github.com/yogidevendra/apex-malhar APEXMALHAR-2234-filter-doc Alternatively you can review and apply these changes as the patch at: https://github.com/apache/apex-malhar/pull/409.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #409 commit 5aa6250d3381f2922a3ce13a93a9bbaa8864b672 Author: yogidevendra <yogideven...@apache.org> Date: 2016-08-17T10:08:50Z APEXMALHAR-2234 FilterOperator documentation` FilterOperator documentation`` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] apex-malhar pull request #409: APEXMALHAR-2234 FilterOperator documentation
Github user yogidevendra closed the pull request at: https://github.com/apache/apex-malhar/pull/409 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] apex-malhar pull request #409: APEXMALHAR-2234 FilterOperator documentation
GitHub user yogidevendra reopened a pull request: https://github.com/apache/apex-malhar/pull/409 APEXMALHAR-2234 FilterOperator documentation You can merge this pull request into a Git repository by running: $ git pull https://github.com/yogidevendra/apex-malhar APEXMALHAR-2234-filter-doc Alternatively you can review and apply these changes as the patch at: https://github.com/apache/apex-malhar/pull/409.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #409 commit 5aa6250d3381f2922a3ce13a93a9bbaa8864b672 Author: yogidevendra <yogideven...@apache.org> Date: 2016-08-17T10:08:50Z APEXMALHAR-2234 FilterOperator documentation` FilterOperator documentation`` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] apex-malhar pull request #439: APEXMALHAR-2272 : Fixed sequentialFileRead on...
GitHub user yogidevendra opened a pull request: https://github.com/apache/apex-malhar/pull/439 APEXMALHAR-2272 : Fixed sequentialFileRead on FSInputModule You can merge this pull request into a Git repository by running: $ git pull https://github.com/yogidevendra/apex-malhar APEXMALHAR-2272-sequencialFileRead Alternatively you can review and apply these changes as the patch at: https://github.com/apache/apex-malhar/pull/439.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #439 commit f01b56b5a671572f37e487f422db4cbe5537 Author: yogidevendra <yogideven...@apache.org> Date: 2016-10-01T02:49:56Z APEXMALHAR-2272 : Fixed sequencialFileRead on FSInputModule --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] apex-malhar pull request #409: APEXMALHAR-2234 FilterOperator documentation
Github user yogidevendra closed the pull request at: https://github.com/apache/apex-malhar/pull/409 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] apex-malhar pull request #409: APEXMALHAR-2234 FilterOperator documentation
GitHub user yogidevendra reopened a pull request: https://github.com/apache/apex-malhar/pull/409 APEXMALHAR-2234 FilterOperator documentation You can merge this pull request into a Git repository by running: $ git pull https://github.com/yogidevendra/apex-malhar APEXMALHAR-2234-filter-doc Alternatively you can review and apply these changes as the patch at: https://github.com/apache/apex-malhar/pull/409.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #409 commit 9ca4e7aae16ef1f2bc49b8d9194fff52c082e270 Author: yogidevendra <yogideven...@apache.org> Date: 2016-08-17T10:08:50Z FilterOperator documentation FilterOperator documentation --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] apex-malhar pull request #409: APEXMALHAR-2234 FilterOperator documentation
GitHub user yogidevendra reopened a pull request: https://github.com/apache/apex-malhar/pull/409 APEXMALHAR-2234 FilterOperator documentation You can merge this pull request into a Git repository by running: $ git pull https://github.com/yogidevendra/apex-malhar APEXMALHAR-2234-filter-doc Alternatively you can review and apply these changes as the patch at: https://github.com/apache/apex-malhar/pull/409.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #409 commit 9ca4e7aae16ef1f2bc49b8d9194fff52c082e270 Author: yogidevendra <yogideven...@apache.org> Date: 2016-08-17T10:08:50Z FilterOperator documentation FilterOperator documentation --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] apex-malhar pull request #428: APEXMALHAR-2237 : Changes in FSInputModule to...
GitHub user yogidevendra opened a pull request: https://github.com/apache/apex-malhar/pull/428 APEXMALHAR-2237 : Changes in FSInputModule to support Dynamic partitioning. You can merge this pull request into a Git repository by running: $ git pull https://github.com/yogidevendra/apex-malhar APEXMALHAR-2237-fsinput-dynamic Alternatively you can review and apply these changes as the patch at: https://github.com/apache/apex-malhar/pull/428.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #428 commit 4163cd5b0dfd11f17a96995510d03a9a8097694e Author: yogidevendra <yogideven...@apache.org> Date: 2016-09-21T06:59:06Z Changes in FSInputModule to support Dynamic partitioning. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] apex-malhar pull request #409: APEXMALHAR-2234 FilterOperator documentation
GitHub user yogidevendra opened a pull request: https://github.com/apache/apex-malhar/pull/409 APEXMALHAR-2234 FilterOperator documentation You can merge this pull request into a Git repository by running: $ git pull https://github.com/yogidevendra/apex-malhar APEXMALHAR-2234-filter-doc Alternatively you can review and apply these changes as the patch at: https://github.com/apache/apex-malhar/pull/409.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #409 commit 79d40d53afe6434fec33cbb8fcfd8baa8f019b44 Author: yogidevendra <yogideven...@apache.org> Date: 2016-08-17T10:08:50Z FilterOperator documentation --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] apex-malhar pull request #384: APEXMALHAR-2206 : Application tests fixes
GitHub user yogidevendra reopened a pull request: https://github.com/apache/apex-malhar/pull/384 APEXMALHAR-2206 : Application tests fixes 1. Fixing placement for lc.shutdown() 2. Disabling log messages for console output in JsonFormatterTest. You can merge this pull request into a Git repository by running: $ git pull https://github.com/yogidevendra/apex-malhar APEXMALHAR-2206-test-fixes-polishing Alternatively you can review and apply these changes as the patch at: https://github.com/apache/apex-malhar/pull/384.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #384 commit edf9635e005a6366341438970cbcd8a21416f106 Author: yogidevendra <yogideven...@apache.org> Date: 2016-08-26T16:14:04Z APEXMALHAR-2206 : Application tests fixes 1. Fixing placement for lc.shutdown() 2. Disabling log messages for console output in JsonFormatterTest. 3. Incorporating review comments --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] apex-malhar pull request #384: APEXMALHAR-2206 : Application tests fixes
Github user yogidevendra closed the pull request at: https://github.com/apache/apex-malhar/pull/384 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] apex-malhar pull request #384: APEXMALHAR-2206 : Application tests fixes
GitHub user yogidevendra opened a pull request: https://github.com/apache/apex-malhar/pull/384 APEXMALHAR-2206 : Application tests fixes 1. Fixing placement for lc.shutdown() 2. Disabling log messages for console output in JsonFormatterTest. You can merge this pull request into a Git repository by running: $ git pull https://github.com/yogidevendra/apex-malhar APEXMALHAR-2206-test-fixes-polishing Alternatively you can review and apply these changes as the patch at: https://github.com/apache/apex-malhar/pull/384.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #384 commit d6232d7d4a5d247a86f7124d52f8f1f3fee6587b Author: yogidevendra <yogideven...@apache.org> Date: 2016-08-26T16:14:04Z APEXMALHAR-2206 : Application tests fixes 1. Fixing placement for lc.shutdown() 2. Disabling log messages for console output in JsonFormatterTest. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] apex-malhar pull request #372: APEXMALHAR-2195 - Fix LineReaderContext last ...
GitHub user yogidevendra reopened a pull request: https://github.com/apache/apex-malhar/pull/372 APEXMALHAR-2195 - Fix LineReaderContext last record 2. Changes in the test app You can merge this pull request into a Git repository by running: $ git pull https://github.com/yogidevendra/apex-malhar APEXMALHAR-2195-LineReaderContext-last-record Alternatively you can review and apply these changes as the patch at: https://github.com/apache/apex-malhar/pull/372.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #372 commit 6ace0b2c72faf5b884cb349fb3ec02003576a5b8 Author: yogidevendra <yogideven...@apache.org> Date: 2016-08-14T17:45:30Z APEXMALHAR-2195 Fix LineReaderContext last record 1. Fixing ReaderContext Issue 2. Changes in the test app 3. Incorporating review comments 4. Graceful handling of test termination --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] apex-malhar pull request #372: APEXMALHAR-2195 - Fix LineReaderContext last ...
Github user yogidevendra closed the pull request at: https://github.com/apache/apex-malhar/pull/372 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] apex-malhar pull request #361: APEXMALHAR-2176 expressionFunctions for Filte...
GitHub user yogidevendra reopened a pull request: https://github.com/apache/apex-malhar/pull/361 APEXMALHAR-2176 expressionFunctions for FilterOperator 1. added setExpressionFunctionsItem method You can merge this pull request into a Git repository by running: $ git pull https://github.com/yogidevendra/apex-malhar APEXMALHAR-2176-filter-expressionFunctions Alternatively you can review and apply these changes as the patch at: https://github.com/apache/apex-malhar/pull/361.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #361 commit 083bef378df3839513b49fd91f975a4824042b98 Author: yogidevendra <yogideven...@apache.org> Date: 2016-08-04T14:31:11Z APEXMALHAR-2176 expressionFunctions for FilterOperator 1. added setExpressionFunctionsItem method 2. incorporating review comments. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] apex-malhar pull request #361: APEXMALHAR-2176 expressionFunctions for Filte...
GitHub user yogidevendra reopened a pull request: https://github.com/apache/apex-malhar/pull/361 APEXMALHAR-2176 expressionFunctions for FilterOperator 1. added setExpressionFunctionsItem method You can merge this pull request into a Git repository by running: $ git pull https://github.com/yogidevendra/apex-malhar APEXMALHAR-2176-filter-expressionFunctions Alternatively you can review and apply these changes as the patch at: https://github.com/apache/apex-malhar/pull/361.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #361 commit 083bef378df3839513b49fd91f975a4824042b98 Author: yogidevendra <yogideven...@apache.org> Date: 2016-08-04T14:31:11Z APEXMALHAR-2176 expressionFunctions for FilterOperator 1. added setExpressionFunctionsItem method 2. incorporating review comments. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] apex-malhar pull request #361: APEXMALHAR-2176 expressionFunctions for Filte...
Github user yogidevendra closed the pull request at: https://github.com/apache/apex-malhar/pull/361 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] apex-malhar pull request #372: APEXMALHAR-2195 - Fixing LineReaderContext Is...
GitHub user yogidevendra opened a pull request: https://github.com/apache/apex-malhar/pull/372 APEXMALHAR-2195 - Fixing LineReaderContext Issue 2. Changes in the test app You can merge this pull request into a Git repository by running: $ git pull https://github.com/yogidevendra/apex-malhar APEXMALHAR-2195-LineReaderContext-last-record Alternatively you can review and apply these changes as the patch at: https://github.com/apache/apex-malhar/pull/372.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #372 commit accc093ddb70faecc5b6a7342ad52bbaf010405b Author: yogidevendra <yogideven...@apache.org> Date: 2016-08-14T17:45:30Z Fixing ReaderContext Issue Changes in the test app --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] apex-malhar pull request #359: APEXMALHAR-2161: Add tests for AbstractThroug...
Github user yogidevendra commented on a diff in the pull request:
https://github.com/apache/apex-malhar/pull/359#discussion_r73849743
--- Diff:
library/src/test/java/com/datatorrent/lib/io/fs/AbstractThroughputFileInputOperatorTest.java
---
@@ -0,0 +1,129 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package com.datatorrent.lib.io.fs;
+
+import java.io.IOException;
+import java.util.Collections;
+
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.Test;
+import org.mockito.Mock;
+import org.mockito.MockitoAnnotations;
+
+import org.apache.commons.lang.mutable.MutableLong;
+
+import com.datatorrent.api.Partitioner.Partition;
+import com.datatorrent.api.Stats.OperatorStats;
+import com.datatorrent.api.StatsListener.BatchedOperatorStats;
+import com.datatorrent.api.StatsListener.Response;
+import com.datatorrent.lib.counters.BasicCounters;
+import com.datatorrent.lib.io.fs.AbstractFileInputOperator.FileCounters;
+
+import static org.mockito.Matchers.any;
+import static org.mockito.Mockito.when;
+
+public class AbstractThroughputFileInputOperatorTest
+{
+ private AbstractThroughputFileInputOperator underTest;
+ @Mock
+ private Partition<AbstractFileInputOperator> mockPartition;
+ @Mock
+ private BatchedOperatorStats mockBatchStats;
+ @Mock
+ private OperatorStats mockOperatorStats;
+ @Mock
+ private BasicCounters fileCountersMock;
+ @Mock
+ private MutableLong fileCounterMock;
+
+ @Before
+ public void setup()
+ {
+underTest = new ThroughputFileInputOperator();
+MockitoAnnotations.initMocks(this);
+
+when(mockPartition.getPartitionedInstance()).thenReturn(underTest);
+ }
+
+ @Test
+ public void testInitialPartitioning()
+ {
+underTest.setPartitionCount(4);
+underTest.setPreferredMaxPendingFilesPerOperator(6);
+
+for (int i = 0; i < 74; i++) {
+ underTest.pendingFiles.add("file-" + i);
+}
+
+int partitioncount =
underTest.getNewPartitionCount(Collections.singleton(mockPartition), null);
+Assert.assertEquals(4, partitioncount);
+ }
+
+ @Test
+ public void testProcessStats() throws Exception
+ {
+underTest.setPartitionCount(4);
+underTest.setPreferredMaxPendingFilesPerOperator(10);
+
+for (int i = 0; i < 21; i++) {
+ underTest.pendingFiles.add("file-" + i);
+}
+
+mockOperatorStats.counters = fileCountersMock;
+when(mockPartition.getStats()).thenReturn(mockBatchStats);
+
when(mockBatchStats.getLastWindowedStats()).thenReturn(Collections.singletonList(mockOperatorStats));
+
when(fileCountersMock.getCounter(any(FileCounters.class))).thenReturn(fileCounterMock);
+when(fileCounterMock.getValue()).thenReturn(20L);
+
+Response response = underTest.processStats(mockBatchStats);
+
+Assert.assertTrue(response.repartitionRequired);
+ }
+
+ @Test
+ public void testRepartitioning()
+ {
+underTest.setPartitionCount(4);
--- End diff --
Please add more scenarios for repartitioning.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] apex-malhar pull request #361: APEXMALHAR-2176 expressionFunctions for Filte...
GitHub user yogidevendra opened a pull request: https://github.com/apache/apex-malhar/pull/361 APEXMALHAR-2176 expressionFunctions for FilterOperator 1. added setExpressionFunctionsItem method You can merge this pull request into a Git repository by running: $ git pull https://github.com/yogidevendra/apex-malhar APEXMALHAR-2176-filter-expressionFunctions Alternatively you can review and apply these changes as the patch at: https://github.com/apache/apex-malhar/pull/361.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #361 commit 058c422149596f5e471b200498801860971fc37a Author: yogidevendra <yogideven...@apache.org> Date: 2016-08-04T14:31:11Z APEXMALHAR-2176 expressionFunctions for FilterOperator 1. added setExpressionFunctionsItem method --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] apex-malhar pull request #360: APEXMALHAR-2174-S3-ReaderIssue Fixed the S3 r...
Github user yogidevendra commented on a diff in the pull request:
https://github.com/apache/apex-malhar/pull/360#discussion_r73466833
--- Diff:
library/src/main/java/com/datatorrent/lib/io/fs/S3BlockReader.java ---
@@ -47,63 +56,144 @@ public S3BlockReader()
public void setup(Context.OperatorContext context)
{
super.setup(context);
-s3bucketUri = fs.getScheme() + "://" + bucketName;
+s3Client = new AmazonS3Client(new BasicAWSCredentials(accessKey,
secretAccessKey));
+((S3BlockReaderContext)readerContext).setBucketName(bucketName);
+((S3BlockReaderContext)readerContext).setS3Client(s3Client);
}
/**
* Extracts the bucket name from the given uri
* @param s3uri s3 uri
* @return name of the bucket
*/
- @VisibleForTesting
protected static String extractBucket(String s3uri)
{
return s3uri.substring(s3uri.indexOf('@') + 1, s3uri.indexOf("/",
s3uri.indexOf('@')));
}
/**
- * Create the stream from the bucket uri and block path.
+ * Extracts the accessKey from the given uri
+ * @param s3uri given s3 uri
+ * @return the accessKey
+ */
+ protected static String extractAccessKey(String s3uri)
+ {
+return s3uri.substring(s3uri.indexOf("://") + 3, s3uri.indexOf(':',
s3uri.indexOf("://") + 3));
+ }
+
+ /**
+ * Extracts the secretAccessKey from the given uri
+ * @param s3uri given s3uri
+ * @return the secretAccessKey
+ */
+ protected static String extractSecretAccessKey(String s3uri)
+ {
+return s3uri.substring(s3uri.indexOf(':', s3uri.indexOf("://") + 1) +
1, s3uri.indexOf('@'));
+ }
+
+ /**
+ * Extract the file path from given block and set it to the readerContext
* @param block block metadata
* @return stream
* @throws IOException
*/
@Override
protected FSDataInputStream setupStream(BlockMetadata.FileBlockMetadata
block) throws IOException
{
-FSDataInputStream ins = fs.open(new Path(s3bucketUri +
block.getFilePath()));
-ins.seek(block.getOffset());
-return ins;
+String filePath = block.getFilePath();
+if (filePath.startsWith("/")) {
--- End diff --
Please add comments mentioning why this is needed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] apex-malhar pull request #326: APEXMALHAR-2116 Added FS record reader operat...
Github user yogidevendra commented on a diff in the pull request:
https://github.com/apache/apex-malhar/pull/326#discussion_r71698302
--- Diff:
library/src/main/java/org/apache/apex/malhar/lib/fs/FSRecordReaderModule.java
---
@@ -0,0 +1,344 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.apex.malhar.lib.fs;
+
+import javax.validation.constraints.Min;
+
+import javax.validation.constraints.NotNull;
+import javax.validation.constraints.Size;
+
+import org.apache.apex.malhar.lib.fs.FSRecordReader.RECORD_READER_MODE;
+import org.apache.hadoop.conf.Configuration;
+
+import com.datatorrent.api.Context;
+import com.datatorrent.api.DAG;
+import com.datatorrent.api.Module;
+import com.datatorrent.common.partitioner.StatelessPartitioner;
+import com.datatorrent.lib.codec.KryoSerializableStreamCodec;
+import com.datatorrent.lib.io.block.BlockMetadata;
+import com.datatorrent.lib.io.block.FSSliceReader;
+import com.datatorrent.lib.io.fs.FileSplitterInput;
+
+/**
+ * This module is used for reading records/tuples from FileSystem. Records
can
+ * be read in parallel using multiple partitions of record reader operator.
+ * (Ordering is not guaranteed when records are read in parallel)
+ *
+ * Input directory is scanned at specified interval to poll for new data.
+ *
+ * The module reads data in parallel, following parameters can be
configured
+ *
+ * 1. files: list of file(s)/directories to read
+ * 2. filePatternRegularExp: Files with names matching given regex will be
read
+ * 3. scanIntervalMillis: interval between two scans to discover new files
in
+ * input directory
+ * 4. recursive: if true, scan input directories recursively
+ * 5. blockSize: block size used to read input blocks of file
+ * 6. readersCount: count of readers to read input file
+ * 7. sequentialFileRead: if true, then each reader partition will read
different file.
+ *instead of reading different offsets of the same file.
+ *(File level parallelism instead of block level parallelism)
+ * 8. blocksThreshold: number of blocks emitted per window
+ */
+@org.apache.hadoop.classification.InterfaceStability.Evolving
+public class FSRecordReaderModule implements Module
+{
+ @NotNull
+ @Size(min = 1)
+ private String files;
+ private String filePatternRegularExp;
+ @Min(1)
+ private long scanIntervalMillis = 5000;
+ private boolean recursive = true;
+ private boolean sequentialFileRead = false;
+ @Min(1)
+ private int readersCount = 1;
+ @Min(1)
+ protected int blocksThreshold = 1;
+
+ public final transient ProxyOutputPort<byte[]> records = new
ProxyOutputPort<byte[]>();
+
+ /**
+ * Criteria for record split
+ */
+ private RECORD_READER_MODE mode = RECORD_READER_MODE.DELIMITED_RECORD;
+
+ /**
+ * Length for fixed width record
+ */
+ @Min(1)
+ private int recordLength;
--- End diff --
@amberarrow As per your suggestion, removed @ min annotation for record
length.
Added validation in the code to ensure that the value is positive if
FIXED_WIDTH_RECORD.
Also, added test case for validating this.
Could you have a look and merge this if it looks OK.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] apex-malhar pull request #326: APEXMALHAR-2116 Added FS record reader operat...
Github user yogidevendra commented on a diff in the pull request:
https://github.com/apache/apex-malhar/pull/326#discussion_r70930388
--- Diff:
library/src/main/java/org/apache/apex/malhar/lib/fs/FSRecordReaderModule.java
---
@@ -0,0 +1,344 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.apex.malhar.lib.fs;
+
+import javax.validation.constraints.Min;
+
+import javax.validation.constraints.NotNull;
+import javax.validation.constraints.Size;
+
+import org.apache.apex.malhar.lib.fs.FSRecordReader.RECORD_READER_MODE;
+import org.apache.hadoop.conf.Configuration;
+
+import com.datatorrent.api.Context;
+import com.datatorrent.api.DAG;
+import com.datatorrent.api.Module;
+import com.datatorrent.common.partitioner.StatelessPartitioner;
+import com.datatorrent.lib.codec.KryoSerializableStreamCodec;
+import com.datatorrent.lib.io.block.BlockMetadata;
+import com.datatorrent.lib.io.block.FSSliceReader;
+import com.datatorrent.lib.io.fs.FileSplitterInput;
+
+/**
+ * This module is used for reading records/tuples from FileSystem. Records
can
+ * be read in parallel using multiple partitions of record reader operator.
+ * (Ordering is not guaranteed when records are read in parallel)
+ *
+ * Input directory is scanned at specified interval to poll for new data.
+ *
+ * The module reads data in parallel, following parameters can be
configured
+ *
+ * 1. files: list of file(s)/directories to read
+ * 2. filePatternRegularExp: Files with names matching given regex will be
read
+ * 3. scanIntervalMillis: interval between two scans to discover new files
in
+ * input directory
+ * 4. recursive: if true, scan input directories recursively
+ * 5. blockSize: block size used to read input blocks of file
+ * 6. readersCount: count of readers to read input file
+ * 7. sequentialFileRead: if true, then each reader partition will read
different file.
+ *instead of reading different offsets of the same file.
+ *(File level parallelism instead of block level parallelism)
+ * 8. blocksThreshold: number of blocks emitted per window
+ */
+@org.apache.hadoop.classification.InterfaceStability.Evolving
+public class FSRecordReaderModule implements Module
+{
+ @NotNull
+ @Size(min = 1)
+ private String files;
+ private String filePatternRegularExp;
+ @Min(1)
+ private long scanIntervalMillis = 5000;
+ private boolean recursive = true;
+ private boolean sequentialFileRead = false;
+ @Min(1)
+ private int readersCount = 1;
+ @Min(1)
+ protected int blocksThreshold = 1;
+
+ public final transient ProxyOutputPort<byte[]> records = new
ProxyOutputPort<byte[]>();
+
+ /**
+ * Criteria for record split
+ */
+ private RECORD_READER_MODE mode = RECORD_READER_MODE.DELIMITED_RECORD;
+
+ /**
+ * Length for fixed width record
+ */
+ @Min(1)
+ private int recordLength;
--- End diff --
Oh. Good point.
Should we separate this into two different classes. FixedWidthRecordReader,
DelimitedRecordReader? That will make configuration clean.
@amberarrow Any thoughts?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] apex-malhar pull request #263: APEXMALHAR-2019 S3-Input Implemented S3 Input...
Github user yogidevendra commented on a diff in the pull request:
https://github.com/apache/apex-malhar/pull/263#discussion_r70375602
--- Diff:
library/src/main/java/com/datatorrent/lib/io/fs/S3BlockReader.java ---
@@ -0,0 +1,89 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package com.datatorrent.lib.io.fs;
+
+import java.io.IOException;
+import org.apache.hadoop.fs.FSDataInputStream;
+import org.apache.hadoop.fs.Path;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.datatorrent.api.Context;
+import com.datatorrent.lib.io.block.BlockMetadata;
+import com.datatorrent.lib.io.block.BlockReader;
+import com.datatorrent.lib.io.block.ReaderContext;
+
+/**
+ * S3BlockReader extends from BlockReader and serves the functionality of
read objects and
+ * parse Block metadata
+ */
+public class S3BlockReader extends BlockReader
+{
+ protected transient String s3bucketUri;
+
+ public S3BlockReader()
+ {
+this.readerContext = new S3BlockReaderContext();
+ }
+
+ @Override
+ public void setup(Context.OperatorContext context)
+ {
+super.setup(context);
+s3bucketUri = fs.getScheme() + "://" + extractBucket(uri);
+ }
+
+ @VisibleForTesting
+ protected String extractBucket(String s3uri)
+ {
+return s3uri.substring(s3uri.indexOf('@') + 1, s3uri.indexOf("/",
s3uri.indexOf('@')));
+ }
+
+ @Override
+ protected FSDataInputStream setupStream(BlockMetadata.FileBlockMetadata
block) throws IOException
+ {
+FSDataInputStream ins = fs.open(new Path(s3bucketUri +
block.getFilePath()));
+ins.seek(block.getOffset());
+return ins;
+ }
+
+ /**
+ * BlockReadeContext for reading S3 Blocks.
+ * This should use read API without offset.
+ */
+ private static class S3BlockReaderContext extends
ReaderContext.FixedBytesReaderContext
+ {
+@Override
+protected Entity readEntity() throws IOException
--- End diff --
@chaithu14 Could you please add javadoc to readEntity() mentioning the
difference between the super implementation?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] apex-malhar pull request #326: APEXMALHAR-2116 Added FS record reader operat...
Github user yogidevendra closed the pull request at: https://github.com/apache/apex-malhar/pull/326 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] apex-malhar pull request #326: APEXMALHAR-2116 Added FS record reader operat...
GitHub user yogidevendra reopened a pull request: https://github.com/apache/apex-malhar/pull/326 APEXMALHAR-2116 Added FS record reader operator, module, test You can merge this pull request into a Git repository by running: $ git pull https://github.com/yogidevendra/apex-malhar APEXMALHAR-2116-record-reader Alternatively you can review and apply these changes as the patch at: https://github.com/apache/apex-malhar/pull/326.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #326 commit 506ab58d382f6f6b338486a49bad0838d77272f9 Author: yogidevendra <yogideven...@apache.org> Date: 2016-06-20T05:47:08Z Added FS record reader operator, module, test 2. incorporated review comments 3. javadoc improvements. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] apex-malhar pull request #326: APEXMALHAR-2116 Added FS record reader operat...
Github user yogidevendra commented on a diff in the pull request:
https://github.com/apache/apex-malhar/pull/326#discussion_r69111489
--- Diff:
library/src/main/java/org/apache/apex/malhar/lib/fs/FSRecordReaderModule.java
---
@@ -0,0 +1,332 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.apex.malhar.lib.fs;
+
+import javax.validation.constraints.Min;
+
+import javax.validation.constraints.NotNull;
+import javax.validation.constraints.Size;
+
+import org.apache.apex.malhar.lib.fs.FSRecordReader.RECORD_READER_MODE;
+import org.apache.hadoop.conf.Configuration;
+
+import com.datatorrent.api.Context;
+import com.datatorrent.api.DAG;
+import com.datatorrent.api.Module;
+import com.datatorrent.common.partitioner.StatelessPartitioner;
+import com.datatorrent.lib.codec.KryoSerializableStreamCodec;
+import com.datatorrent.lib.io.block.BlockMetadata;
+import com.datatorrent.lib.io.block.FSSliceReader;
+import com.datatorrent.lib.io.fs.FileSplitterInput;
+
+/**
+ * This module is used for reading records/tuples from FileSystem. Records
can
+ * be read in parallel using multiple partitions of record reader operator.
+ * (Ordering is not guaranteed when records are read in parallel)
+ *
+ * Input directory is scanned at specified interval to poll for new data.
+ *
+ * The module reads data in parallel, following parameters can be
configured
+ *
+ * 1. files: list of file(s)/directories to read
+ * 2. filePatternRegularExp: Files names matching given regex will be
read
+ * 3. scanIntervalMillis: interval between two scans to discover new files
in
+ * input directory
+ * 4. recursive: if scan recursively input directories
+ * 5. blockSize: block size used to read input blocks of file
+ * 6. readersCount: count of readers to read input file
+ * 7. sequentialFileRead: If emit file blocks in sequence?
+ * 8. blocksThreshold: number of blocks emitted per window
+ */
+@org.apache.hadoop.classification.InterfaceStability.Evolving
+public class FSRecordReaderModule implements Module
+{
+ @NotNull
+ @Size(min = 1)
+ private String files;
+ private String filePatternRegularExp;
+ @Min(0)
+ private long scanIntervalMillis;
+ private boolean recursive = true;
+ private boolean sequentialFileRead = false;
+ private int readersCount;
+ @Min(1)
+ protected int blocksThreshold;
+
+ public final transient ProxyOutputPort<byte[]> records = new
ProxyOutputPort<byte[]>();
+
+ /**
+ * Criteria for record split
+ */
+ private RECORD_READER_MODE mode;
+
+ /**
+ * Length for fixed width record
+ */
+ private int recordLength;
+
+ public FileSplitterInput createFileSplitter()
+ {
+return new FileSplitterInput();
+ }
+
+ public FSRecordReader createBlockReader()
+ {
+FSRecordReader recordReader = new FSRecordReader();
+recordReader.setMode(mode);
+recordReader.setRecordLength(recordLength);
+
+return recordReader;
+ }
+
+ @Override
+ public void populateDAG(DAG dag, Configuration configuration)
+ {
+FileSplitterInput fileSplitter = dag.addOperator("FileSplitter",
createFileSplitter());
+FSRecordReader recordReader = dag.addOperator("BlockReader",
createBlockReader());
+
+dag.addStream("BlockMetadata", fileSplitter.blocksMetadataOutput,
recordReader.blocksMetadataInput);
+
+if (sequentialFileRead) {
+ dag.setInputPortAttribute(recordReader.blocksMetadataInput,
Context.PortContext.STREAM_CODEC,
+ new SequentialFileBlockMetadataCodec());
+}
+
+FileSplitterInput.TimeBasedDirectoryScanner fileScanner =
fileSplitter.getScanner();
+fileScanner.setFiles(files);
+if (scanIntervalMillis != 0) {
[GitHub] apex-malhar pull request #326: APEXMALHAR-2116 Added FS record reader operat...
GitHub user yogidevendra reopened a pull request: https://github.com/apache/apex-malhar/pull/326 APEXMALHAR-2116 Added FS record reader operator, module, test You can merge this pull request into a Git repository by running: $ git pull https://github.com/yogidevendra/apex-malhar APEXMALHAR-2116-record-reader Alternatively you can review and apply these changes as the patch at: https://github.com/apache/apex-malhar/pull/326.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #326 commit e5dba39f1aa5390275faf5c5938d89dd3cd78598 Author: yogidevendra <yogideven...@apache.org> Date: 2016-06-20T05:47:08Z Added FS record reader operator, module, test 2. incorporated review comments --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] apex-malhar pull request #326: APEXMALHAR-2116 Added FS record reader operat...
Github user yogidevendra closed the pull request at: https://github.com/apache/apex-malhar/pull/326 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] apex-malhar pull request #326: APEXMALHAR-2116 Added FS record reader operat...
GitHub user yogidevendra reopened a pull request: https://github.com/apache/apex-malhar/pull/326 APEXMALHAR-2116 Added FS record reader operator, module, test You can merge this pull request into a Git repository by running: $ git pull https://github.com/yogidevendra/apex-malhar APEXMALHAR-2116-record-reader Alternatively you can review and apply these changes as the patch at: https://github.com/apache/apex-malhar/pull/326.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #326 commit c4a6d552c5b0c6b8ede672086f3548847311ab70 Author: yogidevendra <yogideven...@apache.org> Date: 2016-06-20T05:47:08Z Added FS record reader operator, module, test 2. incorporated review comments --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] apex-malhar pull request #326: APEXMALHAR-2116 Added FS record reader operat...
Github user yogidevendra closed the pull request at: https://github.com/apache/apex-malhar/pull/326 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] apex-malhar pull request #326: APEXMALHAR-2116 Added FS record reader operat...
GitHub user yogidevendra opened a pull request: https://github.com/apache/apex-malhar/pull/326 APEXMALHAR-2116 Added FS record reader operator, module, test You can merge this pull request into a Git repository by running: $ git pull https://github.com/yogidevendra/apex-malhar APEXMALHAR-2116-record-reader Alternatively you can review and apply these changes as the patch at: https://github.com/apache/apex-malhar/pull/326.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #326 commit aaeb7fe34dc69de71dae120ea01452149389854f Author: yogidevendra <yogideven...@apache.org> Date: 2016-06-20T05:47:08Z Added FS record reader operator, module, test --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-apex-malhar pull request: APEXMALHAR-2104-making input ports optio...
GitHub user yogidevendra opened a pull request: https://github.com/apache/incubator-apex-malhar/pull/304 APEXMALHAR-2104-making input ports optional 1. Using variable shadowing for making port optional. 2. Marking BytesFileOutputOperator as @evolving You can merge this pull request into a Git repository by running: $ git pull https://github.com/yogidevendra/incubator-apex-malhar APEXMALHAR-2104-BytesFileOutput-optional-port Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-apex-malhar/pull/304.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #304 commit 85bdecf5d66b907f7f17173668afd6e24d88406c Author: yogidevendra <deven...@datatorrent.com> Date: 2016-06-01T06:35:31Z APEXMALHAR-2104-making input ports optional 1. Using variable shadowing for making port optional. 2. Marking this at @evolving --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-apex-malhar pull request: APEXMALHAR-2087 Hive output mo...
GitHub user yogidevendra reopened a pull request: https://github.com/apache/incubator-apex-malhar/pull/289 APEXMALHAR-2087 Hive output module 1. Added Hive output module 2. Minor enhancements for getters, setters 3. Fixing some import order, checkstyle violations You can merge this pull request into a Git repository by running: $ git pull https://github.com/yogidevendra/incubator-apex-malhar APEXMALHAR-2087-hive-output-module-PR1 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-apex-malhar/pull/289.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #289 commit bb0a93d02a91723a323fc09c23150cc5f1a4feb6 Author: yogidevendra <deven...@datatorrent.com> Date: 2016-05-10T09:58:17Z APEXMALHAR-2087 Hive output module 1. Added Hive output module 2. Minor enhancements for getters, setters 3. Fixing some import order, checkstyle violations 4. Moving hive support under separate pom project 5. replacing deprecated calls with recommended calls in the test cases. 6. making test case compatible to parentclass (junit 3) 7. added hive dependency to benchmark 8. added hive to all-modules profile 9. Incorporating review comments --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-apex-malhar pull request: APEXMALHAR-2087 Hive output mo...
Github user yogidevendra closed the pull request at: https://github.com/apache/incubator-apex-malhar/pull/289 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-apex-malhar pull request: APEXMALHAR-2087 Hive output mo...
Github user yogidevendra closed the pull request at: https://github.com/apache/incubator-apex-malhar/pull/289 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-apex-malhar pull request: APEXMALHAR-2087 Hive output mo...
GitHub user yogidevendra reopened a pull request: https://github.com/apache/incubator-apex-malhar/pull/289 APEXMALHAR-2087 Hive output module 1. Added Hive output module 2. Minor enhancements for getters, setters 3. Fixing some import order, checkstyle violations You can merge this pull request into a Git repository by running: $ git pull https://github.com/yogidevendra/incubator-apex-malhar APEXMALHAR-2087-hive-output-module-PR1 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-apex-malhar/pull/289.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #289 commit 8b23cc9552fc514d20640bce313c42a6cbebee11 Author: yogidevendra <deven...@datatorrent.com> Date: 2016-05-10T09:58:17Z APEXMALHAR-2087 Hive output module 1. Added Hive output module 2. Minor enhancements for getters, setters 3. Fixing some import order, checkstyle violations 4. Moving hive support under separate pom project 5. Incorporating review comments 6. replacing deprecated calls with recommended calls in the test cases. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-apex-malhar pull request: APEXMALHAR-2087 Hive output mo...
GitHub user yogidevendra opened a pull request: https://github.com/apache/incubator-apex-malhar/pull/289 APEXMALHAR-2087 Hive output module 1. Added Hive output module 2. Minor enhancements for getters, setters 3. Fixing some import order, checkstyle violations You can merge this pull request into a Git repository by running: $ git pull https://github.com/yogidevendra/incubator-apex-malhar APEXMALHAR-2087-hive-output-module-PR1 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-apex-malhar/pull/289.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #289 commit b4d8ec5c312f81b9fc9ab3abd6e4cb148a1f3e12 Author: yogidevendra <deven...@datatorrent.com> Date: 2016-05-10T09:58:17Z APEXMALHAR-2087 Hive output module 1. Added Hive output module 2. Minor enhancements for getters, setters 3. Fixing some import order, checkstyle violations --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
